From 86cb8e5cbf0f01cb10da1450f55f7e0cfbed3427 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 28 Apr 2026 19:46:23 -0400 Subject: [PATCH] docs(spec): add M4b ListenBrainz inbound similarity ingest design Second M4 sub-plan (Fable #346). Periodic worker pulls track-track and artist-artist similarity edges from LB's public /explore/* endpoints, filters to the local library, stores top-20 per source track in two new tables (track_similarity, artist_similarity). Hourly tick, batch=5, weekly re-fetch cap per row, passive retry via timer. No auth (public endpoints). Discovery within library handled by LB's collaborative- filtering response naturally surfacing unplayed library tracks; spec notes M4c will add a serendipity floor + lazy fetch + sparse-fallback. Co-Authored-By: Claude Opus 4.7 --- .../specs/2026-04-28-m4b-similarity-design.md | 338 ++++++++++++++++++ 1 file changed, 338 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-28-m4b-similarity-design.md diff --git a/docs/superpowers/specs/2026-04-28-m4b-similarity-design.md b/docs/superpowers/specs/2026-04-28-m4b-similarity-design.md new file mode 100644 index 00000000..3add5b22 --- /dev/null +++ b/docs/superpowers/specs/2026-04-28-m4b-similarity-design.md @@ -0,0 +1,338 @@ +# M4b — ListenBrainz inbound similarity ingest + +**Status:** Spec draft, 2026-04-28 +**Tracking:** Fable #346 +**Milestone:** M4 — ListenBrainz scrobble + similarity + radio +**Builds on:** M4a (outbound scrobble worker — shipped as PR #26) + +## 1. Goal + +A periodic background worker that pulls track-track and artist-artist similarity +edges from ListenBrainz's `/explore/similar-recordings/{mbid}` and +`/explore/similar-artists/{mbid}` endpoints and stores them in two new tables +(`track_similarity`, `artist_similarity`). Refreshes each row at most once per +7 days; bounded scope to "tracks the user has played" so cost stays +proportional to actual usage. + +When this slice ships, M4c can build candidate pools for radio from a +similarity graph rather than the user's whole library. + +## 2. Non-goals (explicit) + +- **Lazy fetch on radio request** — M4c. If a user clicks a never-played track + as seed and `track_similarity` is empty for it, M4c can synchronously call + `SimilarRecordings` then. +- **Score normalization to [0, 1]** — store raw LB scores + (`DOUBLE PRECISION`); M4c normalizes at query time if its scoring formula + needs it. +- **Symmetric edges** (storing both `(A, B)` and `(B, A)`) — store one-way as + LB returns. M4c queries `WHERE track_a_id = $seed`. +- **`musicbrainz_tag` and `user_cooccurrence` source values** — schema reserves + them via the `source` enum, but M4b only writes `'listenbrainz'`. +- **Suggested-additions / Lidarr integration** (LB-returned MBIDs not in the + library) — M5. Spec line 225 covers it. +- **Configurable LB algorithm parameter** — hardcoded for v1. +- **Per-user similarity overrides** — `track_similarity` has no `user_id`; + data is global per-instance. +- **Force-refresh HTTP endpoint** — operators can `UPDATE … SET fetched_at = + '1970-01-01' WHERE …` for break-glass. +- **Multi-instance worker safety** — single-process worker assumed (matches + M4a). +- **Frontend surface** — M4b is invisible until M4c uses the data. + +## 3. Architecture overview + +``` + ┌────────────────────────────────────┐ + │ similarity.Worker │ hourly tick + │ - SELECT distinct played tracks │ + │ with mbid where (no row OR │ + │ fetched_at < now() - 7d) │ + │ - LIMIT 5–10 per tick │ + └────────────┬───────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────┐ + │ listenbrainz.Client │ + │ (extended from M4a) │ + │ GET /1/explore/similar-recordings/ │ + │ {mbid} │ + │ GET /1/explore/similar-artists/ │ + │ {artist_mbid} │ + │ No auth — public endpoints │ + └────────────┬─────────────────────────┘ + │ + ▼ + For each LB response: + - Filter to MBIDs we have in our local library + - Take top 20 (sorted by LB score) + - UPSERT into track_similarity / artist_similarity +``` + +### 3.1 New Go package + +**`internal/similarity/`** — the worker: + +- `Worker` struct (pool, client, logger, tick, batch, topK) +- `NewWorker(pool, client, logger) *Worker` — production defaults: 1h tick, + batch=5, topK=20 +- `(w *Worker) Run(ctx)` — blocks until ctx cancelled +- `(w *Worker) tickOnce(ctx) error` — drains one batch of tracks AND one + batch of artists; injectable for tests + +### 3.2 Existing-code extensions + +- **`internal/scrobble/listenbrainz/client.go`** — gains two methods on the + existing `Client` (which already houses `SubmitListens` from M4a): + - `SimilarRecordings(ctx, mbid, limit) ([]SimilarRecording, error)` + - `SimilarArtists(ctx, mbid, limit) ([]SimilarArtist, error)` + - Both return the same typed errors as `SubmitListens` (`ErrTransient`, + `ErrPermanent`, `*RetryAfterError`). 401 is defensive only — these are + public endpoints. + - The package stays at its current path. A future cleanup could move it + to `internal/listenbrainz/`, but that's a non-blocking refactor. + +- **`cmd/minstrel/main.go`** — start the worker alongside the M4a scrobble + worker: + ```go + similarityWorker := similarity.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "similarity")) + go similarityWorker.Run(ctx) + ``` + +### 3.3 Failure handling + +**Passive retry via timer.** Unlike M4a's durable scrobble queue: +- A failed `SimilarRecordings` call does NOT update `fetched_at`. The next + hourly tick selects the row again (it still satisfies the "needs fetch" + predicate) and retries. +- 429 with `Retry-After`: the worker logs the value and **aborts the current + tick** without updating `fetched_at` on any in-flight rows. The next + hourly tick (typically far longer than any LB-suggested back-off) picks + the work back up. Avoids mid-tick sleeps that would block the goroutine. +- ErrPermanent (4xx): logged as a warning + skipped. Permanent errors on + similarity reads typically mean the MBID isn't in LB's graph — there's no + remediation, but `fetched_at` stays old so we'll just retry forever (cheap + no-op since LB returns 4xx fast). Acceptable; could mark "permanently + empty" in a future iteration if telemetry shows it matters. +- ErrTransient (5xx, network): logged + skipped, retry next tick. + +No `scrobble_queue`-equivalent table needed; the work list IS the played-tracks +set + the `fetched_at` watermark. + +## 4. Database schema + +New migration `0009_similarity.up.sql`: + +```sql +CREATE TABLE track_similarity ( + track_a_id uuid NOT NULL REFERENCES tracks(id) ON DELETE CASCADE, + track_b_id uuid NOT NULL REFERENCES tracks(id) ON DELETE CASCADE, + score DOUBLE PRECISION NOT NULL, + source TEXT NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')), + fetched_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (track_a_id, track_b_id, source), + CHECK (track_a_id <> track_b_id) +); + +CREATE INDEX track_similarity_a_score_idx + ON track_similarity (track_a_id, score DESC); + +CREATE TABLE artist_similarity ( + artist_a_id uuid NOT NULL REFERENCES artists(id) ON DELETE CASCADE, + artist_b_id uuid NOT NULL REFERENCES artists(id) ON DELETE CASCADE, + score DOUBLE PRECISION NOT NULL, + source TEXT NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')), + fetched_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (artist_a_id, artist_b_id, source), + CHECK (artist_a_id <> artist_b_id) +); + +CREATE INDEX artist_similarity_a_score_idx + ON artist_similarity (artist_a_id, score DESC); +``` + +Down migration drops both tables. + +**Notes:** +- Primary key includes `source` so the schema can hold multiple parallel + similarity sources (per spec line 119). +- `(track_a_id, score DESC)` index matches the M4c hot-path query: "for seed + T, give me top-N similar tracks descending by score." +- `CHECK (a <> b)` prevents self-edges. +- `ON DELETE CASCADE` from both endpoints so deleting a track cleans up edges + on either side. + +## 5. New sqlc queries + +`internal/db/queries/similarity.sql`: + +```sql +-- name: ListPlayedTracksNeedingSimilarity :many +SELECT DISTINCT t.id, t.mbid +FROM tracks t +JOIN play_events pe ON pe.track_id = t.id +WHERE t.mbid IS NOT NULL + AND NOT EXISTS ( + SELECT 1 FROM track_similarity ts + WHERE ts.track_a_id = t.id + AND ts.source = 'listenbrainz' + AND ts.fetched_at > now() - interval '7 days' + ) +ORDER BY t.id +LIMIT $1; + +-- name: ListPlayedArtistsNeedingSimilarity :many +SELECT DISTINCT ar.id, ar.mbid +FROM artists ar +JOIN tracks t ON t.artist_id = ar.id +JOIN play_events pe ON pe.track_id = t.id +WHERE ar.mbid IS NOT NULL + AND NOT EXISTS ( + SELECT 1 FROM artist_similarity asim + WHERE asim.artist_a_id = ar.id + AND asim.source = 'listenbrainz' + AND asim.fetched_at > now() - interval '7 days' + ) +ORDER BY ar.id +LIMIT $1; + +-- name: GetTracksByMBIDs :many +SELECT id, mbid FROM tracks WHERE mbid = ANY($1::text[]); + +-- name: GetArtistsByMBIDs :many +SELECT id, mbid FROM artists WHERE mbid = ANY($1::text[]); + +-- name: UpsertTrackSimilarity :exec +INSERT INTO track_similarity (track_a_id, track_b_id, score, source, fetched_at) +VALUES ($1, $2, $3, 'listenbrainz', now()) +ON CONFLICT (track_a_id, track_b_id, source) +DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at; + +-- name: UpsertArtistSimilarity :exec +INSERT INTO artist_similarity (artist_a_id, artist_b_id, score, source, fetched_at) +VALUES ($1, $2, $3, 'listenbrainz', now()) +ON CONFLICT (artist_a_id, artist_b_id, source) +DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at; +``` + +## 6. Worker algorithm + +`tickOnce(ctx)`: + +1. **Track pass:** + - `q.ListPlayedTracksNeedingSimilarity(batch=5)` → `[(track_id, mbid)…]` + - For each `(track_id, mbid)`: + - Call `c.SimilarRecordings(ctx, mbid, 100)` + - On 429 → log the `Retry-After` and **return from `tickOnce` early** + (don't update `fetched_at`; the next hourly tick will pick up the + work, which is virtually always longer than LB's `Retry-After`) + - On other error → log warn, skip (no `fetched_at` update; next tick + retries) + - On success: collect returned MBIDs → `q.GetTracksByMBIDs(returnedMBIDs)` + → take top 20 by score → for each `(local_id, score)` call + `q.UpsertTrackSimilarity(track_id, local_id, score)` +2. **Artist pass:** symmetric, using `ListPlayedArtistsNeedingSimilarity`, + `SimilarArtists`, `GetArtistsByMBIDs`, `UpsertArtistSimilarity`. + +**Constants:** +- Tick interval: 1 hour (production); injectable to ms-scale for tests. +- Batch size: 5 (production). 5 LB calls per pass × 2 passes = 10 LB calls/tick + → ~240/day, well under LB's documented rate limits (~100/5min unauth). +- Top-K: 20 per LB query. +- LB algorithm: hardcoded constant in the client (use LB's documented default + at implementation time). + +## 7. Test plan + +### 7.1 LB client unit tests (httptest) + +In `internal/scrobble/listenbrainz/client_test.go`, add 7 tests for each new +method: + +For `SimilarRecordings`: +- 200 + valid body → returns slice ordered by score +- 401 → `ErrAuth` (defensive — public endpoint shouldn't 401) +- 400 → `ErrPermanent` +- 503 → `ErrTransient` +- 429 with `Retry-After` → `*RetryAfterError` +- URL contains `algorithm=…` +- URL contains `limit=N` + +Same 7 tests for `SimilarArtists`. + +### 7.2 Worker integration tests (live DB + httptest) + +In `internal/similarity/worker_integration_test.go`: + +- `TickOnce_NoPlayedTracks_NoOp`: empty `play_events` → returns nil, no rows + in `track_similarity` +- `TickOnce_MapsLBResponseToLocalLibrary`: seed one played track with MBID; + LB returns 3 MBIDs (2 in library, 1 not); assert 2 rows inserted +- `TickOnce_TopKEnforced`: LB returns 50; assert + `count(*) WHERE track_a_id = $1` ≤ 20 +- `TickOnce_RespectsSevenDayCap`: row with `fetched_at = now()` → not + re-queried (the LB endpoint isn't called) +- `TickOnce_RefreshesStaleRow`: row with `fetched_at = now() - interval '8 + days'` → re-fetched, score updated, fetched_at bumps +- `TickOnce_429AbortsTick`: first response 429 with Retry-After → tickOnce + returns early; no `fetched_at` updates; subsequent tracks in the batch + are NOT processed (they're picked up on the next tick) +- `TickOnce_TransientErrorSkipsTrack`: 503 on one track → `fetched_at` + unchanged; other tracks in same batch process normally +- `TickOnce_FiltersInLibrary`: LB returns 5 MBIDs none of which are in the + library → 0 rows inserted (no error) +- `TickOnce_ArtistPassMirrors`: same coverage for the artist branch +- `TickOnce_NoMBIDOnTrack_Skipped`: track with `mbid IS NULL` → not selected + by `ListPlayedTracksNeedingSimilarity` + +### 7.3 Coverage target + +`internal/similarity` ≥ 75%. The worker has fewer error branches than M4a +because passive retry-via-timer eliminates the durable-queue state machine. + +The new `Similar*` methods in `internal/scrobble/listenbrainz` add ~14 tests +to that package; should keep its coverage ≥ 85%. + +### 7.4 Manual verification post-merge + +1. Restart server with M4b code. +2. After ≥1 hour, `psql -c "SELECT count(*) FROM track_similarity WHERE + source = 'listenbrainz'"` → non-zero (assuming user has any MBID-tagged + played tracks). +3. After ≥7 days, observe a `fetched_at` timestamp that's recent — + confirms re-fetch cadence. +4. If MBID coverage in the library is sparse, the table will be small. Not a + bug — M5 (Lidarr suggested-additions) is the eventual answer for + tracks-not-in-library; tagging coverage via Picard is a separate user-side + improvement. + +## 8. Backwards compatibility + +- New migration; no changes to existing schema. +- New package; no changes to consumer code (M4c will consume; M3's `Score()` + doesn't need the data until then). +- `MaybeEnqueue` and other M4a paths unchanged. +- Worker is new; production behavior identical to pre-M4b until the worker's + first tick fires (1 hour after restart). + +## 9. Decisions ledger + +| # | Decision | Rationale | +|---|---|---| +| 1 | Both `track_similarity` and `artist_similarity` in M4b | Symmetric pattern, M4c needs both, single-PR cost is small | +| 2 | Played-tracks-only input scope | Bounds work to user's actual interaction graph; LB's response naturally surfaces in-library tracks the user hasn't played, preserving discovery | +| 3 | Hourly tick, batch=5 | 240 LB calls/day fits well under LB rate limits; initial backfill in 24-48h | +| 4 | Top-K=20 per LB query, in-library only | Cuts long-tail noise; out-of-library tracks deferred to M5/Lidarr | +| 5 | Public endpoint, no auth | Similarity data is global to the instance; LB requires no token for `/explore/*` | +| 6 | Passive retry via timer (no durable queue) | Failure cost is "1 hour of staleness"; durable queue would be over-engineering vs. M4a's "lost scrobble" stakes | +| 7 | Hardcoded `algorithm` parameter | YAGNI; expose as YAML if telemetry warrants | + +## 10. Sub-plan progression (M4) + +- M4a (done) — outbound scrobble worker (PR #26). +- **M4b (this) — inbound LB similarity ingest.** +- M4c — radio similarity-driven candidate pool + queue refresh at 80% + (closes M4). M4c also picks up the discovery-mitigation work flagged in + brainstorm: serendipity floor (% random library picks), fallback to wider + pool when similarity-row count is sparse, lazy fetch on radio for + never-played seeds.