diff --git a/docs/superpowers/specs/2026-04-28-m4a-scrobble-design.md b/docs/superpowers/specs/2026-04-28-m4a-scrobble-design.md new file mode 100644 index 00000000..fb1c8fc8 --- /dev/null +++ b/docs/superpowers/specs/2026-04-28-m4a-scrobble-design.md @@ -0,0 +1,403 @@ +# M4a — ListenBrainz outbound scrobble worker + +**Status:** Spec draft, 2026-04-28 +**Tracking:** Fable #345 +**Milestone:** M4 — ListenBrainz scrobble + similarity + radio + +## 1. Goal + +Send the user's qualifying plays to ListenBrainz with batching and exponential +retry. Reads from the existing M2 `play_events` stream; backed by a new +`scrobble_queue` table that the worker drains every 30 seconds. Per-user token +configuration via a new minimal `/settings` page in the SPA. + +When this slice ships, a play that meets ListenBrainz's "≥240s OR ≥50% +completion" threshold appears in the user's LB profile within ~30 seconds, and +the system survives LB outages, bad networks, and process restarts without +losing or duplicating scrobbles. + +## 2. Non-goals (explicit) + +- **`now-playing` endpoint** — real-time "user is currently listening" status. + Requires push architecture; defer until a UI surface needs it. +- **Listen import** — backfilling historical plays into LB. Useful for migration + but not part of the core flow. +- **MusicBrainz ID resolution at scan time** — payload uses MBIDs only when + already populated in `tracks.mbid`/`albums.mbid`/`artists.mbid`. Better + scanner-side MBID coverage is a future improvement. +- **Encrypted-at-rest token storage** — chose plaintext for v1 (industry norm + for self-hosted scrobble apps). +- **Scrobble status indicator in PlayerBar** — latency-sensitive UX; bundle + with the future now-playing push. +- **Multi-instance queue safety** — single worker assumes single Minstrel + process. ROW LOCK FOR UPDATE SKIP LOCKED is a future change if/when Minstrel + goes multi-instance. +- **CLI admin token rotation** — operators can `UPDATE users …` for + break-glass; CLI tooling lands with M6 packaging. +- **Proactive rate limiting** — we honor LB's `Retry-After` headers but don't + throttle ourselves. Human-scale play rate is well below LB's limits. +- **Multiple scrobble services** (Last.fm, Maloja, etc.) — LB-only by design. + The `internal/scrobble/listenbrainz/` package structure leaves room for + siblings later if demand emerges. + +## 3. Architecture overview + +``` + ┌──────────────────────────┐ +play_event closed ──► │ scrobble.MaybeEnqueue │ (called from +(M2 RecordPlayEnded) │ - apply LB threshold │ Writer hooks) + │ - INSERT scrobble_queue │ + └────────────┬─────────────┘ + │ + ▼ + ┌────────────────────────┐ + │ scrobble_queue table │ status: pending | failed + │ (work list, not log) │ + └────────────┬───────────┘ + │ + ▼ + tick 30s ─────► ┌────────────────────────────────────┐ + (goroutine) │ scrobble.Worker │ + │ - SELECT pending, next_attempt<= now │ + │ - batch POST to LB submit-listens │ + │ - on 2xx: DELETE row + │ + │ UPDATE play_events.scrobbled_at │ + │ - on 5xx/network: increment │ + │ attempts, schedule next │ + │ - on 4xx: mark failed, log │ + │ - on 429 Retry-After: respect │ + │ header │ + └────────────────────────────────────┘ + │ + ▼ + ListenBrainz https://api.listenbrainz.org +``` + +### 3.1 New Go packages + +- **`internal/scrobble/listenbrainz/`** — pure HTTP client. One method: + `SubmitListens(ctx, token, listens)`. No DB, no worker plumbing. Tested + against `httptest.Server`. +- **`internal/scrobble/threshold.go`** — `Qualifies(durationPlayedMs, + completionRatio) bool`. Pure function. +- **`internal/scrobble/queue.go`** — `MaybeEnqueue(ctx, q, playEventID)`. + Reads user config + threshold, inserts row. +- **`internal/scrobble/worker.go`** — the goroutine. Started from + `cmd/minstrel/main.go` alongside `playevents.Writer`. + +### 3.2 Hooked into existing code + +- **`playevents.Writer.RecordPlayEnded`** — after the close transaction + commits, call `scrobble.MaybeEnqueue(ctx, q, playEvent.ID)` as a + best-effort post-commit step. Errors are logged and swallowed (matches + the M3 `CaptureContextualLikeIfPlaying` pattern: scrobble delivery is + enrichment, not a precondition for the canonical play history). +- **`playevents.Writer.RecordSyntheticCompletedPlay`** — same hook (Subsonic + `submission=true` plays from `/rest/scrobble`). +- **`internal/api/me.go`** — two new handlers: `handleGetListenBrainz` and + `handlePutListenBrainz`. Wired in `Mount(...)` under `/api/me/listenbrainz`. +- **`cmd/minstrel/main.go`** — start the worker: + ```go + worker := scrobble.NewWorker(pool, listenbrainz.NewClient(), logger) + go worker.Run(ctx) + ``` + +## 4. Database schema + +New migration `0008_scrobble.up.sql`: + +```sql +ALTER TABLE users + ADD COLUMN listenbrainz_token TEXT NULL, + ADD COLUMN listenbrainz_enabled BOOLEAN NOT NULL DEFAULT FALSE; + +CREATE TABLE scrobble_queue ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, + play_event_id uuid NOT NULL REFERENCES play_events(id) ON DELETE CASCADE, + status TEXT NOT NULL CHECK (status IN ('pending', 'failed')), + attempts INTEGER NOT NULL DEFAULT 0, + next_attempt_at timestamptz NOT NULL DEFAULT now(), + last_error TEXT NULL, + enqueued_at timestamptz NOT NULL DEFAULT now(), + UNIQUE (play_event_id) +); + +CREATE INDEX scrobble_queue_pending_idx + ON scrobble_queue (next_attempt_at) + WHERE status = 'pending'; +``` + +Down migration drops the table and the user columns. + +**Reused (no schema changes):** +- `play_events.scrobbled_at` — already exists from M2 migration 0005, + currently always NULL. Worker stamps it on successful submit. +- `play_events.duration_played_ms`, `completion_ratio` — already populated + by M2; used for the LB eligibility threshold. + +**Lifecycle:** +- Successful submit → `DELETE FROM scrobble_queue WHERE id = $1` AND + `UPDATE play_events SET scrobbled_at = now() WHERE id = $1`. Canonical + record stays in `play_events`; queue is purely a work list. +- `UNIQUE(play_event_id)` makes `MaybeEnqueue` idempotent: re-running the + close path on a play_event with an existing queue row is a no-op via + `ON CONFLICT DO NOTHING`. +- No `'sent'` status enum value — the simpler binary `pending`/`failed` + state machine is sufficient because successful rows are deleted. + +## 5. Backend components + +### 5.1 ListenBrainz client (`internal/scrobble/listenbrainz/client.go`) + +```go +type Listen struct { + ListenedAt int64 + Track Track +} + +type Track struct { + ArtistName string + TrackName string + ReleaseName string + DurationMs int + RecordingMBID string + ArtistMBIDs []string + ReleaseMBID string +} + +type Client struct { + BaseURL string // default https://api.listenbrainz.org + HTTP *http.Client +} + +// Errors are typed so the worker can branch on response semantics. +var ( + ErrAuth = errors.New("listenbrainz: auth rejected") // 401 + ErrPermanent = errors.New("listenbrainz: permanent error") // other 4xx + ErrTransient = errors.New("listenbrainz: transient error") // 5xx, network +) + +type RetryAfterError struct{ Wait time.Duration } +func (e *RetryAfterError) Error() string { ... } + +func (c *Client) SubmitListens(ctx context.Context, token string, listens []Listen) error +``` + +Sets `Authorization: Token `. POSTs to `/1/submit-listens` with +`payload_type=import` for batches >1, `single` for batches of 1 (LB convention). +On 429, parses the `Retry-After` header and returns `*RetryAfterError`. + +### 5.2 Threshold (`internal/scrobble/threshold.go`) + +```go +// Qualifies returns true if a closed play_event meets ListenBrainz's +// recommended scrobble threshold. +func Qualifies(durationPlayedMs int, completionRatio float64) bool { + return durationPlayedMs >= 240_000 || completionRatio >= 0.5 +} +``` + +### 5.3 Enqueue (`internal/scrobble/queue.go`) + +```go +// MaybeEnqueue inserts a scrobble_queue row for the given play_event if: +// 1. The user has listenbrainz_enabled = true with a non-empty token. +// 2. The play_event passes Qualifies(). +// Idempotent via UNIQUE(play_event_id) — re-runs are no-ops. +// Returns nil even when the row is skipped (no-op is not an error). +// Best-effort: callers (the playevents.Writer hooks) should log returned +// errors but not propagate them, since scrobble delivery is enrichment +// and must not block the canonical play history. +func MaybeEnqueue(ctx context.Context, q *dbq.Queries, playEventID pgtype.UUID) error +``` + +### 5.4 Worker (`internal/scrobble/worker.go`) + +```go +type Worker struct { + pool *pgxpool.Pool + client *listenbrainz.Client + logger *slog.Logger + tick time.Duration // 30s in production, injectable for tests + now func() time.Time // injectable for tests + batch int // up to 50 rows per tick +} + +func NewWorker(pool *pgxpool.Pool, client *listenbrainz.Client, logger *slog.Logger) *Worker + +// Run blocks until ctx is cancelled. +func (w *Worker) Run(ctx context.Context) + +// tickOnce drains up to `batch` pending rows. Exposed for tests. +func (w *Worker) tickOnce(ctx context.Context) error +``` + +Backoff schedule (constants in `worker.go`): + +```go +var backoffSchedule = []time.Duration{ + 1 * time.Minute, + 5 * time.Minute, + 30 * time.Minute, + 2 * time.Hour, + 6 * time.Hour, +} +const maxAttempts = 5 + +// backoffDelay maps a failure count (the value of the `attempts` column +// AFTER the failure has been recorded) to the delay before the next +// attempt. attempts=1 (first failure just happened) → backoffSchedule[0] +// = 1m. attempts=5 → backoffSchedule[4] = 6h. attempts >= 6 → give up. +// Returns (_, false) when attempts > maxAttempts. +func backoffDelay(attempts int) (time.Duration, bool) +``` + +Per-row outcome handling: +- 2xx → `DELETE FROM scrobble_queue WHERE id = $1` + `UPDATE play_events SET + scrobbled_at = now() WHERE id = $play_event_id`. +- `*RetryAfterError` (429) → `UPDATE … SET next_attempt_at = now() + Wait` + WITHOUT incrementing `attempts` (server told us to wait, not that we failed). +- `ErrTransient` (5xx, network) → increment `attempts`, look up + `backoffDelay(attempts)`. If `false`, mark `failed`. Otherwise schedule next. +- `ErrPermanent` (4xx) → mark `failed` immediately, no retry. +- `ErrAuth` (401) → mark `failed` immediately AND set + `users.listenbrainz_enabled = FALSE` (defensive: bad token shouldn't keep + retrying or feed future rows). + +### 5.5 API endpoints (`internal/api/me.go`) + +``` +GET /api/me/listenbrainz + → 200 { enabled: boolean, token_set: boolean, last_scrobbled_at: string|null } + +PUT /api/me/listenbrainz + body: { token?: string, enabled?: boolean } + - token: any string sets the token. Empty string clears it AND forces enabled=false. + - enabled: requires a non-empty stored token. 400 if attempting enabled=true with no token. + → 200 { enabled, token_set, last_scrobbled_at } +``` + +The token is **write-only** — `GET` never returns the actual value, only +`token_set: bool`. `last_scrobbled_at` is computed via `MAX(scrobbled_at) +FROM play_events WHERE user_id = $1`. + +## 6. Frontend (minimal `/settings`) + +New page `web/src/routes/settings/+page.svelte` with a single "ListenBrainz" +section. Form: + +- **Token field**: password input with Save button when unset; masked + "••••••••• (set)" + Clear button when set. Save calls `PUT /api/me/listenbrainz + { token }`. Clear calls `PUT { token: "" }`. +- **Enabled checkbox**: "Send my plays to ListenBrainz". Disabled when no + token. Toggle calls `PUT { enabled: !current }`. +- **Last scrobbled at**: localized timestamp of `last_scrobbled_at` from the + GET response. +- **Disclaimer**: "Tokens are stored unencrypted in this server's database — + treat as sensitive." + +Shell nav (`web/src/lib/components/Shell.svelte`) gains `{ href: '/settings', +label: 'Settings' }` after `/playlists`. + +Helpers in `web/src/lib/api/client.ts` — confirm `apiPut` exists; add it if +not (mirrors the existing `apiGet` shape). + +## 7. Test plan + +### 7.1 Pure unit tests + +- **`threshold_test.go`**: boundary cases (exactly 240s, exactly 50%, just + under both, both true). +- **`worker_test.go`**: `backoffDelay(1) == 1m`, `backoffDelay(2) == 5m`, + `backoffDelay(3) == 30m`, `backoffDelay(4) == 2h`, `backoffDelay(5) == 6h`, + `backoffDelay(6)` returns `_, false`. +- **`listenbrainz/client_test.go`** (uses `httptest.Server`): + - 2xx → nil + - 401 → `ErrAuth` + - 400, 403 → `ErrPermanent` + - 500, 503 → `ErrTransient` + - 429 with `Retry-After: 60` → `*RetryAfterError{Wait: 60s}` + - Network failure mid-request → `ErrTransient` + - Body shape (JSON) and `Authorization: Token ` header asserted + +### 7.2 Integration (live test DB) + +- **`queue_test.go`**: + - Idempotent: `MaybeEnqueue` twice for same play_event = one row + - User with `listenbrainz_enabled=false` → no row + - User with empty token → no row + - Play_event below threshold → no row + - Play_event passing threshold → row inserted (status=pending, attempts=0, + next_attempt_at ≈ now()) + +- **`worker_integration_test.go`** (mocked LB via httptest): + - 1 pending row + 200 → row deleted, `play_events.scrobbled_at` populated + - 503 once → row remains, `attempts=1`, `next_attempt_at ≈ now() + 1m` + - 503 five times → row marked `failed`, `last_error` populated + - 401 → row marked `failed` immediately, `users.listenbrainz_enabled` set to + `FALSE` + - 429 with `Retry-After: 300` → row remains, `next_attempt_at ≈ now() + 5m`, + `attempts` NOT incremented + - Batch of 10 pending → single LB POST, 10 listens in payload + +- **`internal/api/me_test.go`** extensions: + - GET when no token → `{enabled:false, token_set:false, last_scrobbled_at:null}` + - PUT `{token: "abc"}` → DB updated, GET shows `token_set:true` + - PUT `{token: ""}` → token cleared, `enabled` forced to false + - PUT `{enabled: true}` while no token → 400 + - `last_scrobbled_at` populated from `MAX(play_events.scrobbled_at)` + +### 7.3 Frontend (`web/src/routes/settings/settings.test.ts`) + +- Token-not-set state: input + Save button render +- Token-set state: masked + Clear button render +- Save mutation: PUT body has the typed token +- Enabled checkbox: disabled when no token +- Last-scrobbled-at: localized timestamp renders when present +- Mocked `apiGet`/`apiPut`; no real network + +### 7.4 Coverage target + +`internal/scrobble/...` ≥ 80%. M3 combined coverage stays well above the 70% +floor with these additions. + +### 7.5 Manual verification post-merge + +1. Generate a token at https://listenbrainz.org/profile/ +2. /settings → paste token → Save → toggle enabled +3. Play a track for ≥240s OR finish it +4. Within 30s, check listenbrainz.org/ → listen appears +5. /settings → "Last scrobbled" timestamp populates + +## 8. Backwards compatibility + +- New migration; no changes to existing schema beyond two nullable columns + on `users` and the new `scrobble_queue` table. +- `/api/me/listenbrainz` is new; no existing endpoints change. +- `MaybeEnqueue` is a new hook in `playevents.Writer`; if all users have + `listenbrainz_enabled=false` (the default after migration), the hook is a + no-op and behavior is unchanged. +- The new `/settings` page is additive; the rest of the SPA is unchanged. +- `playevents.Writer.RecordPlayEnded` and `RecordSyntheticCompletedPlay` + signatures are unchanged. Only their bodies gain the enqueue call. + +## 9. Decisions ledger + +| # | Decision | Rationale | +|---|---|---| +| 1 | Per-user token (vs global) | Forward-compat to multi-user; cost is 2 nullable columns. | +| 2 | Plaintext token storage (vs encrypted) | Industry norm for self-hosted scrobble apps; key-management is a separate scope. | +| 3 | Threshold ≥240s OR ≥50% (separate from skip threshold) | Matches LB recommendation; skip threshold serves a different purpose. | +| 4 | Pull-based 30s timer (vs push) | Survives restarts/outages; failure handling natural; latency invisible at single-user scale. | +| 5 | Minimal `/settings` page in M4a (vs API-only) | User needs somewhere to put the token; scaffold for M6 to extend. | +| 6 | Patient backoff: 1m → 5m → 30m → 2h → 6h, 5 attempts (~9h window) | Survives overnight LB outages without unbounded queue growth. | +| 7 | Delete sent rows; keep failed rows | Canonical record in `play_events.scrobbled_at`; queue is a work list. | + +## 10. Sub-plan progression (M4) + +- **M4a (this) — outbound scrobble worker**. +- M4b — inbound similarity ingest (`track_similarity` table + LB similarity + fetcher; weekly cache). +- M4c — radio similarity-driven candidate pool + queue-refresh-at-80% + (closes M4).