Files
minstrel/internal/gc/worker.go
T
bvandeusen 258bc1f75c
test-go / test (push) Successful in 29s
test-go / integration (push) Failing after 11m57s
feat(server): drift audit batch 7 — periodic GC worker for 5 lifecycle gaps
New `internal/gc` package with a single Worker that runs all five
lifecycle / retention sweeps from the 2026-06-02 drift audit on a
1-hour tick. Each sweep is small, idempotent (re-running on
already-clean rows is a no-op), and logs its affected-row count.

Sweeps (Scribe parent #552):

- **#566** GcCloseStalePlayEvents — play_events rows opened > 24h
  ago that never got a play_ended (client crash, network drop).
  Synthesizes ended_at from duration_played_ms when known, falls
  back to now() so the row stops looking "open" to downstream
  filters (ended_at IS NULL).

- **#565** GcClosePlaySessionsWithNoRecentEvents — play_sessions
  with last_event_at older than 6h get ended_at = last_event_at
  ("user moved on"); empty sessions older than 1h get closed
  too (stale handshakes from clients that never recorded a play).
  The audit caught that the column was added but never populated
  by any writer — every session row was "open" forever, breaking
  downstream dedup queries that assume closed semantics.

- **#567** GcExpireScrobbleQueueFailedRows — drops scrobble_queue
  rows in status='failed' older than 14 days. The worker stops
  retrying after maxAttempts so these otherwise accumulate
  forever on a persistent ListenBrainz outage / revoked token.

- **#574** GcResetStuckSystemPlaylistRuns — flips
  system_playlist_runs.in_flight back to false on rows whose
  last_run_at is older than 10 minutes. Catches goroutine-panic
  wedges where the generator died between SET in_flight=true and
  SET in_flight=false; the duplicate-prevention check refuses to
  start a fresh regen while in_flight, so a stuck row would
  otherwise deadlock all future regens for that user. Records
  "stuck-row auto-reset by gc" in last_error so the operator can
  tell auto-reset from a recent real failure.

- **#575** GcDeleteExpiredPasswordResets — deletes expired
  password_resets rows. Unused expired rows go after a 1h grace
  (gives the operator time to debug an active reset attempt);
  used rows are kept 7 days for audit.

Wiring:
- main.go `go gcWorker.Run(ctx)` alongside the other periodic
  workers (scrobble, similarity, lidarr).
- tickOnce fires once at start so a freshly-deployed server does
  its initial sweep without waiting a full tick, matching the
  scrobble worker pattern.
- Errors per sweep are logged but do NOT abort the remaining
  ones — a transient pgx error from one query shouldn't prevent
  the others from running.

Tests:
- 4 integration tests, one per UPDATE/DELETE sweep, that seed
  rows-to-sweep + rows-to-leave-alone and assert the right rows
  changed state. Skip unless MINSTREL_TEST_DATABASE_URL is set
  (mirrors the api package pattern).
- Empty-tables no-op smoke test.
- Run() cancellation honoured (no spinning goroutine at
  test-runner exit).

That's all five remaining server-side lifecycle findings from the
audit. The Android LOCAL_USER_ID hardcode (#576) is a separate
refactor that needs auth-store wiring and stays in the queue.
2026-06-02 18:32:22 -04:00

102 lines
3.7 KiB
Go

// Package gc runs periodic garbage-collection / lifecycle sweeps
// against tables that have NO writer-side close path or NO retention
// policy. Each sweep addresses a drift finding from the 2026-06-02
// audit (Scribe parent #552) and is idempotent — re-running it on
// already-clean rows is a no-op.
//
// One Worker handles all sweeps so a single long-tick goroutine
// amortises the per-tick fixed cost. Each individual sweep is small
// (single UPDATE / DELETE with a time-bounded WHERE) and emits a
// log line with the affected-row count so the sweep cadence is
// visible in the application log without an explicit metrics layer.
//
// Sweeps:
// - GcCloseStalePlayEvents (#566)
// - GcClosePlaySessionsWithNoRecentEvents (#565)
// - GcExpireScrobbleQueueFailedRows (#567)
// - GcResetStuckSystemPlaylistRuns (#574)
// - GcDeleteExpiredPasswordResets (#575)
package gc
import (
"context"
"log/slog"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq"
)
// defaultTick is the production sweep cadence. 1 hour is generous
// since each sweep's WHERE clause uses a multi-hour staleness
// threshold; the worst-case delay between a row becoming sweepable
// and the worker noticing is bounded by tick + threshold.
const defaultTick = 1 * time.Hour
// Worker holds the pool + logger + tick interval. Construct with
// [NewWorker]; pass the returned Worker to a goroutine that calls
// [Worker.Run] with a context that's cancelled on shutdown.
type Worker struct {
pool *pgxpool.Pool
logger *slog.Logger
tick time.Duration
}
// NewWorker builds a Worker with the production tick (1h). Tests can
// reach into the Worker after construction to override `tick` for
// faster iteration.
func NewWorker(pool *pgxpool.Pool, logger *slog.Logger) *Worker {
return &Worker{pool: pool, logger: logger, tick: defaultTick}
}
// Run blocks until ctx is cancelled, running every sweep on each
// tick. Sweeps fire in fixed order; an error in one does NOT abort
// the rest (the panic-vs-just-failed distinction matters here — a
// pgx transient error from one query shouldn't prevent the others
// from running).
func (w *Worker) Run(ctx context.Context) {
// Fire once at start so a freshly-deployed server doesn't wait a
// full tick before doing the initial sweep. Matches the scrobble
// + similarity workers' "sweep then tick" pattern.
w.tickOnce(ctx)
t := time.NewTicker(w.tick)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
w.tickOnce(ctx)
}
}
}
// tickOnce runs each sweep once, logging the affected-row count.
// Errors are logged per-sweep but do NOT abort the remaining ones —
// each sweep is independent.
func (w *Worker) tickOnce(ctx context.Context) {
q := dbq.New(w.pool)
w.runSweep(ctx, "close_stale_play_events", q.GcCloseStalePlayEvents)
w.runSweep(ctx, "close_play_sessions", q.GcClosePlaySessionsWithNoRecentEvents)
w.runSweep(ctx, "expire_scrobble_failed", q.GcExpireScrobbleQueueFailedRows)
w.runSweep(ctx, "reset_stuck_system_runs", q.GcResetStuckSystemPlaylistRuns)
w.runSweep(ctx, "delete_expired_password_resets", q.GcDeleteExpiredPasswordResets)
}
// runSweep is a small adapter so each sweep call site is a one-liner
// in tickOnce. Logs at info on rows>0 and debug on rows=0 to keep
// the normal-case (nothing-to-do) noise out of operator logs.
func (w *Worker) runSweep(ctx context.Context, name string, fn func(context.Context) (int64, error)) {
rows, err := fn(ctx)
if err != nil {
w.logger.Error("gc sweep failed", "sweep", name, "err", err)
return
}
if rows > 0 {
w.logger.Info("gc sweep", "sweep", name, "rows_affected", rows)
} else {
w.logger.Debug("gc sweep", "sweep", name, "rows_affected", 0)
}
}