bvandeusen/minstrel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 38 Wiki Activity

docs(spec): add M4b ListenBrainz inbound similarity ingest design

Browse Source 
Second M4 sub-plan (Fable #346). Periodic worker pulls track-track and
artist-artist similarity edges from LB's public /explore/* endpoints,
filters to the local library, stores top-20 per source track in two new
tables (track_similarity, artist_similarity). Hourly tick, batch=5,
weekly re-fetch cap per row, passive retry via timer. No auth (public
endpoints). Discovery within library handled by LB's collaborative-
filtering response naturally surfacing unplayed library tracks; spec
notes M4c will add a serendipity floor + lazy fetch + sparse-fallback.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bryan Van Deusen
2026-04-28 19:46:23 -04:00
parent 13e12d97ae
commit 86cb8e5cbf
1 changed files with 338 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-04-28-m4b-similarity-design.md
+338
View File
@@ -0,0 +1,338 @@
# M4b — ListenBrainz inbound similarity ingest
**Status:** Spec draft, 2026-04-28
**Tracking:** Fable #346
**Milestone:** M4 — ListenBrainz scrobble + similarity + radio
**Builds on:** M4a (outbound scrobble worker — shipped as PR #26)
## 1. Goal
A periodic background worker that pulls track-track and artist-artist similarity
edges from ListenBrainz's `/explore/similar-recordings/{mbid}` and
`/explore/similar-artists/{mbid}` endpoints and stores them in two new tables
(`track_similarity`, `artist_similarity`). Refreshes each row at most once per
7 days; bounded scope to "tracks the user has played" so cost stays
proportional to actual usage.
When this slice ships, M4c can build candidate pools for radio from a
similarity graph rather than the user's whole library.
## 2. Non-goals (explicit)
- **Lazy fetch on radio request** — M4c. If a user clicks a never-played track
as seed and `track_similarity` is empty for it, M4c can synchronously call
`SimilarRecordings` then.
- **Score normalization to [0, 1]** — store raw LB scores
(`DOUBLE PRECISION`); M4c normalizes at query time if its scoring formula
needs it.
- **Symmetric edges** (storing both `(A, B)` and `(B, A)`) — store one-way as
LB returns. M4c queries `WHERE track_a_id = $seed`.
- **`musicbrainz_tag` and `user_cooccurrence` source values** — schema reserves
them via the `source` enum, but M4b only writes `'listenbrainz'`.
- **Suggested-additions / Lidarr integration** (LB-returned MBIDs not in the
library) — M5. Spec line 225 covers it.
- **Configurable LB algorithm parameter** — hardcoded for v1.
- **Per-user similarity overrides** — `track_similarity` has no `user_id`;
data is global per-instance.
- **Force-refresh HTTP endpoint** — operators can `UPDATE … SET fetched_at =
'1970-01-01' WHERE …` for break-glass.
- **Multi-instance worker safety** — single-process worker assumed (matches
M4a).
- **Frontend surface** — M4b is invisible until M4c uses the data.
## 3. Architecture overview
```
┌────────────────────────────────────┐
│ similarity.Worker │ hourly tick
│ - SELECT distinct played tracks │
│ with mbid where (no row OR │
│ fetched_at < now() - 7d) │
│ - LIMIT 5–10 per tick │
└────────────┬───────────────────────┘
│
▼
┌──────────────────────────────────────┐
│ listenbrainz.Client │
│ (extended from M4a) │
│ GET /1/explore/similar-recordings/ │
│ {mbid} │
│ GET /1/explore/similar-artists/ │
│ {artist_mbid} │
│ No auth — public endpoints │
└────────────┬─────────────────────────┘
│
▼
For each LB response:
- Filter to MBIDs we have in our local library
- Take top 20 (sorted by LB score)
- UPSERT into track_similarity / artist_similarity
```
### 3.1 New Go package
**`internal/similarity/`** — the worker:
- `Worker` struct (pool, client, logger, tick, batch, topK)
- `NewWorker(pool, client, logger) *Worker` — production defaults: 1h tick,
batch=5, topK=20
- `(w *Worker) Run(ctx)` — blocks until ctx cancelled
- `(w *Worker) tickOnce(ctx) error` — drains one batch of tracks AND one
batch of artists; injectable for tests
### 3.2 Existing-code extensions
- **`internal/scrobble/listenbrainz/client.go`** — gains two methods on the
existing `Client` (which already houses `SubmitListens` from M4a):
- `SimilarRecordings(ctx, mbid, limit) ([]SimilarRecording, error)`
- `SimilarArtists(ctx, mbid, limit) ([]SimilarArtist, error)`
- Both return the same typed errors as `SubmitListens` (`ErrTransient`,
`ErrPermanent`, `*RetryAfterError`). 401 is defensive only — these are
public endpoints.
- The package stays at its current path. A future cleanup could move it
to `internal/listenbrainz/`, but that's a non-blocking refactor.
- **`cmd/minstrel/main.go`** — start the worker alongside the M4a scrobble
worker:
```go
similarityWorker := similarity.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "similarity"))
go similarityWorker.Run(ctx)
```
### 3.3 Failure handling
**Passive retry via timer.** Unlike M4a's durable scrobble queue:
- A failed `SimilarRecordings` call does NOT update `fetched_at`. The next
hourly tick selects the row again (it still satisfies the "needs fetch"
predicate) and retries.
- 429 with `Retry-After`: the worker logs the value and **aborts the current
tick** without updating `fetched_at` on any in-flight rows. The next
hourly tick (typically far longer than any LB-suggested back-off) picks
the work back up. Avoids mid-tick sleeps that would block the goroutine.
- ErrPermanent (4xx): logged as a warning + skipped. Permanent errors on
similarity reads typically mean the MBID isn't in LB's graph — there's no
remediation, but `fetched_at` stays old so we'll just retry forever (cheap
no-op since LB returns 4xx fast). Acceptable; could mark "permanently
empty" in a future iteration if telemetry shows it matters.
- ErrTransient (5xx, network): logged + skipped, retry next tick.
No `scrobble_queue`-equivalent table needed; the work list IS the played-tracks
set + the `fetched_at` watermark.
## 4. Database schema
New migration `0009_similarity.up.sql`:
```sql
CREATE TABLE track_similarity (
track_a_id uuid NOT NULL REFERENCES tracks(id) ON DELETE CASCADE,
track_b_id uuid NOT NULL REFERENCES tracks(id) ON DELETE CASCADE,
score DOUBLE PRECISION NOT NULL,
source TEXT NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')),
fetched_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (track_a_id, track_b_id, source),
CHECK (track_a_id <> track_b_id)
);
CREATE INDEX track_similarity_a_score_idx
ON track_similarity (track_a_id, score DESC);
CREATE TABLE artist_similarity (
artist_a_id uuid NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
artist_b_id uuid NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
score DOUBLE PRECISION NOT NULL,
source TEXT NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')),
fetched_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (artist_a_id, artist_b_id, source),
CHECK (artist_a_id <> artist_b_id)
);
CREATE INDEX artist_similarity_a_score_idx
ON artist_similarity (artist_a_id, score DESC);
```
Down migration drops both tables.
**Notes:**
- Primary key includes `source` so the schema can hold multiple parallel
similarity sources (per spec line 119).
- `(track_a_id, score DESC)` index matches the M4c hot-path query: "for seed
T, give me top-N similar tracks descending by score."
- `CHECK (a <> b)` prevents self-edges.
- `ON DELETE CASCADE` from both endpoints so deleting a track cleans up edges
on either side.
## 5. New sqlc queries
`internal/db/queries/similarity.sql`:
```sql
-- name: ListPlayedTracksNeedingSimilarity :many
SELECT DISTINCT t.id, t.mbid
FROM tracks t
JOIN play_events pe ON pe.track_id = t.id
WHERE t.mbid IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM track_similarity ts
WHERE ts.track_a_id = t.id
AND ts.source = 'listenbrainz'
AND ts.fetched_at > now() - interval '7 days'
)
ORDER BY t.id
LIMIT $1;
-- name: ListPlayedArtistsNeedingSimilarity :many
SELECT DISTINCT ar.id, ar.mbid
FROM artists ar
JOIN tracks t ON t.artist_id = ar.id
JOIN play_events pe ON pe.track_id = t.id
WHERE ar.mbid IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM artist_similarity asim
WHERE asim.artist_a_id = ar.id
AND asim.source = 'listenbrainz'
AND asim.fetched_at > now() - interval '7 days'
)
ORDER BY ar.id
LIMIT $1;
-- name: GetTracksByMBIDs :many
SELECT id, mbid FROM tracks WHERE mbid = ANY($1::text[]);
-- name: GetArtistsByMBIDs :many
SELECT id, mbid FROM artists WHERE mbid = ANY($1::text[]);
-- name: UpsertTrackSimilarity :exec
INSERT INTO track_similarity (track_a_id, track_b_id, score, source, fetched_at)
VALUES ($1, $2, $3, 'listenbrainz', now())
ON CONFLICT (track_a_id, track_b_id, source)
DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at;
-- name: UpsertArtistSimilarity :exec
INSERT INTO artist_similarity (artist_a_id, artist_b_id, score, source, fetched_at)
VALUES ($1, $2, $3, 'listenbrainz', now())
ON CONFLICT (artist_a_id, artist_b_id, source)
DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at;
```
## 6. Worker algorithm
`tickOnce(ctx)`:
1. **Track pass:**
- `q.ListPlayedTracksNeedingSimilarity(batch=5)` → `[(track_id, mbid)…]`
- For each `(track_id, mbid)`:
- Call `c.SimilarRecordings(ctx, mbid, 100)`
- On 429 → log the `Retry-After` and **return from `tickOnce` early**
(don't update `fetched_at`; the next hourly tick will pick up the
work, which is virtually always longer than LB's `Retry-After`)
- On other error → log warn, skip (no `fetched_at` update; next tick
retries)
- On success: collect returned MBIDs → `q.GetTracksByMBIDs(returnedMBIDs)`
→ take top 20 by score → for each `(local_id, score)` call
`q.UpsertTrackSimilarity(track_id, local_id, score)`
2. **Artist pass:** symmetric, using `ListPlayedArtistsNeedingSimilarity`,
`SimilarArtists`, `GetArtistsByMBIDs`, `UpsertArtistSimilarity`.
**Constants:**
- Tick interval: 1 hour (production); injectable to ms-scale for tests.
- Batch size: 5 (production). 5 LB calls per pass × 2 passes = 10 LB calls/tick
→ ~240/day, well under LB's documented rate limits (~100/5min unauth).
- Top-K: 20 per LB query.
- LB algorithm: hardcoded constant in the client (use LB's documented default
at implementation time).
## 7. Test plan
### 7.1 LB client unit tests (httptest)
In `internal/scrobble/listenbrainz/client_test.go`, add 7 tests for each new
method:
For `SimilarRecordings`:
- 200 + valid body → returns slice ordered by score
- 401 → `ErrAuth` (defensive — public endpoint shouldn't 401)
- 400 → `ErrPermanent`
- 503 → `ErrTransient`
- 429 with `Retry-After` → `*RetryAfterError`
- URL contains `algorithm=…`
- URL contains `limit=N`
Same 7 tests for `SimilarArtists`.
### 7.2 Worker integration tests (live DB + httptest)
In `internal/similarity/worker_integration_test.go`:
- `TickOnce_NoPlayedTracks_NoOp`: empty `play_events` → returns nil, no rows
in `track_similarity`
- `TickOnce_MapsLBResponseToLocalLibrary`: seed one played track with MBID;
LB returns 3 MBIDs (2 in library, 1 not); assert 2 rows inserted
- `TickOnce_TopKEnforced`: LB returns 50; assert
`count(*) WHERE track_a_id = $1` ≤ 20
- `TickOnce_RespectsSevenDayCap`: row with `fetched_at = now()` → not
re-queried (the LB endpoint isn't called)
- `TickOnce_RefreshesStaleRow`: row with `fetched_at = now() - interval '8
days'` → re-fetched, score updated, fetched_at bumps
- `TickOnce_429AbortsTick`: first response 429 with Retry-After → tickOnce
returns early; no `fetched_at` updates; subsequent tracks in the batch
are NOT processed (they're picked up on the next tick)
- `TickOnce_TransientErrorSkipsTrack`: 503 on one track → `fetched_at`
unchanged; other tracks in same batch process normally
- `TickOnce_FiltersInLibrary`: LB returns 5 MBIDs none of which are in the
library → 0 rows inserted (no error)
- `TickOnce_ArtistPassMirrors`: same coverage for the artist branch
- `TickOnce_NoMBIDOnTrack_Skipped`: track with `mbid IS NULL` → not selected
by `ListPlayedTracksNeedingSimilarity`
### 7.3 Coverage target
`internal/similarity` ≥ 75%. The worker has fewer error branches than M4a
because passive retry-via-timer eliminates the durable-queue state machine.
The new `Similar*` methods in `internal/scrobble/listenbrainz` add ~14 tests
to that package; should keep its coverage ≥ 85%.
### 7.4 Manual verification post-merge
1. Restart server with M4b code.
2. After ≥1 hour, `psql -c "SELECT count(*) FROM track_similarity WHERE
source = 'listenbrainz'"` → non-zero (assuming user has any MBID-tagged
played tracks).
3. After ≥7 days, observe a `fetched_at` timestamp that's recent —
confirms re-fetch cadence.
4. If MBID coverage in the library is sparse, the table will be small. Not a
bug — M5 (Lidarr suggested-additions) is the eventual answer for
tracks-not-in-library; tagging coverage via Picard is a separate user-side
improvement.
## 8. Backwards compatibility
- New migration; no changes to existing schema.
- New package; no changes to consumer code (M4c will consume; M3's `Score()`
doesn't need the data until then).
- `MaybeEnqueue` and other M4a paths unchanged.
- Worker is new; production behavior identical to pre-M4b until the worker's
first tick fires (1 hour after restart).
## 9. Decisions ledger
| # | Decision | Rationale |
|---|---|---|
| 1 | Both `track_similarity` and `artist_similarity` in M4b | Symmetric pattern, M4c needs both, single-PR cost is small |
| 2 | Played-tracks-only input scope | Bounds work to user's actual interaction graph; LB's response naturally surfaces in-library tracks the user hasn't played, preserving discovery |
| 3 | Hourly tick, batch=5 | 240 LB calls/day fits well under LB rate limits; initial backfill in 24-48h |
| 4 | Top-K=20 per LB query, in-library only | Cuts long-tail noise; out-of-library tracks deferred to M5/Lidarr |
| 5 | Public endpoint, no auth | Similarity data is global to the instance; LB requires no token for `/explore/*` |
| 6 | Passive retry via timer (no durable queue) | Failure cost is "1 hour of staleness"; durable queue would be over-engineering vs. M4a's "lost scrobble" stakes |
| 7 | Hardcoded `algorithm` parameter | YAGNI; expose as YAML if telemetry warrants |
## 10. Sub-plan progression (M4)
- M4a (done) — outbound scrobble worker (PR #26).
- **M4b (this) — inbound LB similarity ingest.**
- M4c — radio similarity-driven candidate pool + queue refresh at 80%
(closes M4). M4c also picks up the discovery-mitigation work flagged in
brainstorm: serendipity floor (% random library picks), fallback to wider
pool when similarity-row count is sparse, lazy fetch on radio for
never-played seeds.