minstrel/docs/superpowers/specs/2026-04-28-m4b-similarity-design.md

M4b — ListenBrainz inbound similarity ingest

Status: Spec draft, 2026-04-28 Tracking: Fable #346 Milestone: M4 — ListenBrainz scrobble + similarity + radio Builds on: M4a (outbound scrobble worker — shipped as PR #26)

1. Goal

A periodic background worker that pulls track-track and artist-artist similarity edges from ListenBrainz's /explore/similar-recordings/{mbid} and /explore/similar-artists/{mbid} endpoints and stores them in two new tables (track_similarity, artist_similarity). Refreshes each row at most once per 7 days; bounded scope to "tracks the user has played" so cost stays proportional to actual usage.

When this slice ships, M4c can build candidate pools for radio from a similarity graph rather than the user's whole library.

2. Non-goals (explicit)

• Lazy fetch on radio request — M4c. If a user clicks a never-played track as seed and track_similarity is empty for it, M4c can synchronously call SimilarRecordings then.
• Score normalization to [0, 1] — store raw LB scores (DOUBLE PRECISION); M4c normalizes at query time if its scoring formula needs it.
• Symmetric edges (storing both (A, B) and (B, A)) — store one-way as LB returns. M4c queries WHERE track_a_id = $seed.
• musicbrainz_tag and user_cooccurrence source values — schema reserves them via the source enum, but M4b only writes 'listenbrainz'.
• Suggested-additions / Lidarr integration (LB-returned MBIDs not in the library) — M5. Spec line 225 covers it.
• Configurable LB algorithm parameter — hardcoded for v1.
• Per-user similarity overrides — track_similarity has no user_id; data is global per-instance.
• Force-refresh HTTP endpoint — operators can UPDATE … SET fetched_at = '1970-01-01' WHERE … for break-glass.
• Multi-instance worker safety — single-process worker assumed (matches M4a).
• Frontend surface — M4b is invisible until M4c uses the data.

3. Architecture overview

                       ┌────────────────────────────────────┐
                       │  similarity.Worker                  │  hourly tick
                       │  - SELECT distinct played tracks    │
                       │    with mbid where (no row OR       │
                       │    fetched_at < now() - 7d)         │
                       │  - LIMIT 5–10 per tick              │
                       └────────────┬───────────────────────┘
                                    │
                                    ▼
                       ┌──────────────────────────────────────┐
                       │  listenbrainz.Client                 │
                       │  (extended from M4a)                 │
                       │  GET /1/explore/similar-recordings/  │
                       │      {mbid}                          │
                       │  GET /1/explore/similar-artists/     │
                       │      {artist_mbid}                   │
                       │  No auth — public endpoints          │
                       └────────────┬─────────────────────────┘
                                    │
                                    ▼
            For each LB response:
              - Filter to MBIDs we have in our local library
              - Take top 20 (sorted by LB score)
              - UPSERT into track_similarity / artist_similarity

3.1 New Go package

internal/similarity/ — the worker:

• Worker struct (pool, client, logger, tick, batch, topK)
• NewWorker(pool, client, logger) *Worker — production defaults: 1h tick, batch=5, topK=20
• (w *Worker) Run(ctx) — blocks until ctx cancelled
• (w *Worker) tickOnce(ctx) error — drains one batch of tracks AND one batch of artists; injectable for tests

3.2 Existing-code extensions

• internal/scrobble/listenbrainz/client.go — gains two methods on the existing Client (which already houses SubmitListens from M4a):

  • SimilarRecordings(ctx, mbid, limit) ([]SimilarRecording, error)
  • SimilarArtists(ctx, mbid, limit) ([]SimilarArtist, error)
  • Both return the same typed errors as SubmitListens (ErrTransient, ErrPermanent, *RetryAfterError). 401 is defensive only — these are public endpoints.
  • The package stays at its current path. A future cleanup could move it to internal/listenbrainz/, but that's a non-blocking refactor.

• cmd/minstrel/main.go — start the worker alongside the M4a scrobble worker:

  similarityWorker := similarity.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "similarity"))
  go similarityWorker.Run(ctx)
  

3.3 Failure handling

Passive retry via timer. Unlike M4a's durable scrobble queue:

• A failed SimilarRecordings call does NOT update fetched_at. The next hourly tick selects the row again (it still satisfies the "needs fetch" predicate) and retries.
• 429 with Retry-After: the worker logs the value and aborts the current tick without updating fetched_at on any in-flight rows. The next hourly tick (typically far longer than any LB-suggested back-off) picks the work back up. Avoids mid-tick sleeps that would block the goroutine.
• ErrPermanent (4xx): logged as a warning + skipped. Permanent errors on similarity reads typically mean the MBID isn't in LB's graph — there's no remediation, but fetched_at stays old so we'll just retry forever (cheap no-op since LB returns 4xx fast). Acceptable; could mark "permanently empty" in a future iteration if telemetry shows it matters.
• ErrTransient (5xx, network): logged + skipped, retry next tick.

No scrobble_queue-equivalent table needed; the work list IS the played-tracks set + the fetched_at watermark.

4. Database schema

New migration 0009_similarity.up.sql:

CREATE TABLE track_similarity (
    track_a_id   uuid        NOT NULL REFERENCES tracks(id) ON DELETE CASCADE,
    track_b_id   uuid        NOT NULL REFERENCES tracks(id) ON DELETE CASCADE,
    score        DOUBLE PRECISION NOT NULL,
    source       TEXT        NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')),
    fetched_at   timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (track_a_id, track_b_id, source),
    CHECK (track_a_id <> track_b_id)
);

CREATE INDEX track_similarity_a_score_idx
    ON track_similarity (track_a_id, score DESC);

CREATE TABLE artist_similarity (
    artist_a_id  uuid        NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
    artist_b_id  uuid        NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
    score        DOUBLE PRECISION NOT NULL,
    source       TEXT        NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')),
    fetched_at   timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (artist_a_id, artist_b_id, source),
    CHECK (artist_a_id <> artist_b_id)
);

CREATE INDEX artist_similarity_a_score_idx
    ON artist_similarity (artist_a_id, score DESC);

Down migration drops both tables.

Notes:

• Primary key includes source so the schema can hold multiple parallel similarity sources (per spec line 119).
• (track_a_id, score DESC) index matches the M4c hot-path query: "for seed T, give me top-N similar tracks descending by score."
• CHECK (a <> b) prevents self-edges.
• ON DELETE CASCADE from both endpoints so deleting a track cleans up edges on either side.

5. New sqlc queries

internal/db/queries/similarity.sql:

-- name: ListPlayedTracksNeedingSimilarity :many
SELECT DISTINCT t.id, t.mbid
FROM tracks t
JOIN play_events pe ON pe.track_id = t.id
WHERE t.mbid IS NOT NULL
  AND NOT EXISTS (
      SELECT 1 FROM track_similarity ts
      WHERE ts.track_a_id = t.id
        AND ts.source = 'listenbrainz'
        AND ts.fetched_at > now() - interval '7 days'
  )
ORDER BY t.id
LIMIT $1;

-- name: ListPlayedArtistsNeedingSimilarity :many
SELECT DISTINCT ar.id, ar.mbid
FROM artists ar
JOIN tracks t ON t.artist_id = ar.id
JOIN play_events pe ON pe.track_id = t.id
WHERE ar.mbid IS NOT NULL
  AND NOT EXISTS (
      SELECT 1 FROM artist_similarity asim
      WHERE asim.artist_a_id = ar.id
        AND asim.source = 'listenbrainz'
        AND asim.fetched_at > now() - interval '7 days'
  )
ORDER BY ar.id
LIMIT $1;

-- name: GetTracksByMBIDs :many
SELECT id, mbid FROM tracks WHERE mbid = ANY($1::text[]);

-- name: GetArtistsByMBIDs :many
SELECT id, mbid FROM artists WHERE mbid = ANY($1::text[]);

-- name: UpsertTrackSimilarity :exec
INSERT INTO track_similarity (track_a_id, track_b_id, score, source, fetched_at)
VALUES ($1, $2, $3, 'listenbrainz', now())
ON CONFLICT (track_a_id, track_b_id, source)
DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at;

-- name: UpsertArtistSimilarity :exec
INSERT INTO artist_similarity (artist_a_id, artist_b_id, score, source, fetched_at)
VALUES ($1, $2, $3, 'listenbrainz', now())
ON CONFLICT (artist_a_id, artist_b_id, source)
DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at;

6. Worker algorithm

tickOnce(ctx):

1. Track pass:
   • q.ListPlayedTracksNeedingSimilarity(batch=5) → [(track_id, mbid)…]
   • For each (track_id, mbid):
     • Call c.SimilarRecordings(ctx, mbid, 100)
     • On 429 → log the Retry-After and return from tickOnce early (don't update fetched_at; the next hourly tick will pick up the work, which is virtually always longer than LB's Retry-After)
     • On other error → log warn, skip (no fetched_at update; next tick retries)
     • On success: collect returned MBIDs → q.GetTracksByMBIDs(returnedMBIDs) → take top 20 by score → for each (local_id, score) call q.UpsertTrackSimilarity(track_id, local_id, score)
2. Artist pass: symmetric, using ListPlayedArtistsNeedingSimilarity, SimilarArtists, GetArtistsByMBIDs, UpsertArtistSimilarity.

Constants:

• Tick interval: 1 hour (production); injectable to ms-scale for tests.
• Batch size: 5 (production). 5 LB calls per pass × 2 passes = 10 LB calls/tick → ~240/day, well under LB's documented rate limits (~100/5min unauth).
• Top-K: 20 per LB query.
• LB algorithm: hardcoded constant in the client (use LB's documented default at implementation time).

7. Test plan

7.1 LB client unit tests (httptest)

In internal/scrobble/listenbrainz/client_test.go, add 7 tests for each new method:

For SimilarRecordings:

• 200 + valid body → returns slice ordered by score
• 401 → ErrAuth (defensive — public endpoint shouldn't 401)
• 400 → ErrPermanent
• 503 → ErrTransient
• 429 with Retry-After → *RetryAfterError
• URL contains algorithm=…
• URL contains limit=N

Same 7 tests for SimilarArtists.

7.2 Worker integration tests (live DB + httptest)

In internal/similarity/worker_integration_test.go:

• TickOnce_NoPlayedTracks_NoOp: empty play_events → returns nil, no rows in track_similarity
• TickOnce_MapsLBResponseToLocalLibrary: seed one played track with MBID; LB returns 3 MBIDs (2 in library, 1 not); assert 2 rows inserted
• TickOnce_TopKEnforced: LB returns 50; assert count(*) WHERE track_a_id = $1 ≤ 20
• TickOnce_RespectsSevenDayCap: row with fetched_at = now() → not re-queried (the LB endpoint isn't called)
• TickOnce_RefreshesStaleRow: row with fetched_at = now() - interval '8 days' → re-fetched, score updated, fetched_at bumps
• TickOnce_429AbortsTick: first response 429 with Retry-After → tickOnce returns early; no fetched_at updates; subsequent tracks in the batch are NOT processed (they're picked up on the next tick)
• TickOnce_TransientErrorSkipsTrack: 503 on one track → fetched_at unchanged; other tracks in same batch process normally
• TickOnce_FiltersInLibrary: LB returns 5 MBIDs none of which are in the library → 0 rows inserted (no error)
• TickOnce_ArtistPassMirrors: same coverage for the artist branch
• TickOnce_NoMBIDOnTrack_Skipped: track with mbid IS NULL → not selected by ListPlayedTracksNeedingSimilarity

7.3 Coverage target

internal/similarity ≥ 75%. The worker has fewer error branches than M4a because passive retry-via-timer eliminates the durable-queue state machine.

The new Similar* methods in internal/scrobble/listenbrainz add ~14 tests to that package; should keep its coverage ≥ 85%.

7.4 Manual verification post-merge

1. Restart server with M4b code.
2. After ≥1 hour, psql -c "SELECT count(*) FROM track_similarity WHERE source = 'listenbrainz'" → non-zero (assuming user has any MBID-tagged played tracks).
3. After ≥7 days, observe a fetched_at timestamp that's recent — confirms re-fetch cadence.
4. If MBID coverage in the library is sparse, the table will be small. Not a bug — M5 (Lidarr suggested-additions) is the eventual answer for tracks-not-in-library; tagging coverage via Picard is a separate user-side improvement.

8. Backwards compatibility

• New migration; no changes to existing schema.
• New package; no changes to consumer code (M4c will consume; M3's Score() doesn't need the data until then).
• MaybeEnqueue and other M4a paths unchanged.
• Worker is new; production behavior identical to pre-M4b until the worker's first tick fires (1 hour after restart).

9. Decisions ledger

#	Decision	Rationale
1	Both track_similarity and artist_similarity in M4b	Symmetric pattern, M4c needs both, single-PR cost is small
2	Played-tracks-only input scope	Bounds work to user's actual interaction graph; LB's response naturally surfaces in-library tracks the user hasn't played, preserving discovery
3	Hourly tick, batch=5	240 LB calls/day fits well under LB rate limits; initial backfill in 24-48h
4	Top-K=20 per LB query, in-library only	Cuts long-tail noise; out-of-library tracks deferred to M5/Lidarr
5	Public endpoint, no auth	Similarity data is global to the instance; LB requires no token for /explore/*
6	Passive retry via timer (no durable queue)	Failure cost is "1 hour of staleness"; durable queue would be over-engineering vs. M4a's "lost scrobble" stakes
7	Hardcoded algorithm parameter	YAGNI; expose as YAML if telemetry warrants

10. Sub-plan progression (M4)

• M4a (done) — outbound scrobble worker (PR #26).
• M4b (this) — inbound LB similarity ingest.
• M4c — radio similarity-driven candidate pool + queue refresh at 80% (closes M4). M4c also picks up the discovery-mitigation work flagged in brainstorm: serendipity floor (% random library picks), fallback to wider pool when similarity-row count is sparse, lazy fetch on radio for never-played seeds.