minstrel/docs/superpowers/specs/2026-04-28-m4b-similarity-design.md

# M4b — ListenBrainz inbound similarity ingest

**Status:** Spec draft, 2026-04-28
**Tracking:** Fable #346
**Milestone:** M4 — ListenBrainz scrobble + similarity + radio
**Builds on:** M4a (outbound scrobble worker — shipped as PR #26)

## 1. Goal

A periodic background worker that pulls track-track and artist-artist similarity
edges from ListenBrainz's `/explore/similar-recordings/{mbid}` and
`/explore/similar-artists/{mbid}` endpoints and stores them in two new tables
(`track_similarity`, `artist_similarity`). Refreshes each row at most once per
7 days; bounded scope to "tracks the user has played" so cost stays
proportional to actual usage.

When this slice ships, M4c can build candidate pools for radio from a
similarity graph rather than the user's whole library.

## 2. Non-goals (explicit)

- **Lazy fetch on radio request** — M4c. If a user clicks a never-played track
  as seed and `track_similarity` is empty for it, M4c can synchronously call
  `SimilarRecordings` then.
- **Score normalization to [0, 1]** — store raw LB scores
  (`DOUBLE PRECISION`); M4c normalizes at query time if its scoring formula
  needs it.
- **Symmetric edges** (storing both `(A, B)` and `(B, A)`) — store one-way as
  LB returns. M4c queries `WHERE track_a_id = $seed`.
- **`musicbrainz_tag` and `user_cooccurrence` source values** — schema reserves
  them via the `source` enum, but M4b only writes `'listenbrainz'`.
- **Suggested-additions / Lidarr integration** (LB-returned MBIDs not in the
  library) — M5. Spec line 225 covers it.
- **Configurable LB algorithm parameter** — hardcoded for v1.
- **Per-user similarity overrides** — `track_similarity` has no `user_id`;
  data is global per-instance.
- **Force-refresh HTTP endpoint** — operators can `UPDATE … SET fetched_at =
  '1970-01-01' WHERE …` for break-glass.
- **Multi-instance worker safety** — single-process worker assumed (matches
  M4a).
- **Frontend surface** — M4b is invisible until M4c uses the data.

## 3. Architecture overview

```
                       ┌────────────────────────────────────┐
                       │  similarity.Worker                  │  hourly tick
                       │  - SELECT distinct played tracks    │
                       │    with mbid where (no row OR       │
                       │    fetched_at < now() - 7d)         │
                       │  - LIMIT 5–10 per tick              │
                       └────────────┬───────────────────────┘
                                    │
                                    ▼
                       ┌──────────────────────────────────────┐
                       │  listenbrainz.Client                 │
                       │  (extended from M4a)                 │
                       │  GET /1/explore/similar-recordings/  │
                       │      {mbid}                          │
                       │  GET /1/explore/similar-artists/     │
                       │      {artist_mbid}                   │
                       │  No auth — public endpoints          │
                       └────────────┬─────────────────────────┘
                                    │
                                    ▼
            For each LB response:
              - Filter to MBIDs we have in our local library
              - Take top 20 (sorted by LB score)
              - UPSERT into track_similarity / artist_similarity
```

### 3.1 New Go package

**`internal/similarity/`** — the worker:

- `Worker` struct (pool, client, logger, tick, batch, topK)
- `NewWorker(pool, client, logger) *Worker` — production defaults: 1h tick,
  batch=5, topK=20
- `(w *Worker) Run(ctx)` — blocks until ctx cancelled
- `(w *Worker) tickOnce(ctx) error` — drains one batch of tracks AND one
  batch of artists; injectable for tests

### 3.2 Existing-code extensions

- **`internal/scrobble/listenbrainz/client.go`** — gains two methods on the
  existing `Client` (which already houses `SubmitListens` from M4a):
  - `SimilarRecordings(ctx, mbid, limit) ([]SimilarRecording, error)`
  - `SimilarArtists(ctx, mbid, limit) ([]SimilarArtist, error)`
  - Both return the same typed errors as `SubmitListens` (`ErrTransient`,
    `ErrPermanent`, `*RetryAfterError`). 401 is defensive only — these are
    public endpoints.
  - The package stays at its current path. A future cleanup could move it
    to `internal/listenbrainz/`, but that's a non-blocking refactor.

- **`cmd/minstrel/main.go`** — start the worker alongside the M4a scrobble
  worker:
  ```go
  similarityWorker := similarity.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "similarity"))
  go similarityWorker.Run(ctx)
  ```

### 3.3 Failure handling

**Passive retry via timer.** Unlike M4a's durable scrobble queue:
- A failed `SimilarRecordings` call does NOT update `fetched_at`. The next
  hourly tick selects the row again (it still satisfies the "needs fetch"
  predicate) and retries.
- 429 with `Retry-After`: the worker logs the value and **aborts the current
  tick** without updating `fetched_at` on any in-flight rows. The next
  hourly tick (typically far longer than any LB-suggested back-off) picks
  the work back up. Avoids mid-tick sleeps that would block the goroutine.
- ErrPermanent (4xx): logged as a warning + skipped. Permanent errors on
  similarity reads typically mean the MBID isn't in LB's graph — there's no
  remediation, but `fetched_at` stays old so we'll just retry forever (cheap
  no-op since LB returns 4xx fast). Acceptable; could mark "permanently
  empty" in a future iteration if telemetry shows it matters.
- ErrTransient (5xx, network): logged + skipped, retry next tick.

No `scrobble_queue`-equivalent table needed; the work list IS the played-tracks
set + the `fetched_at` watermark.

## 4. Database schema

New migration `0009_similarity.up.sql`:

```sql
CREATE TABLE track_similarity (
    track_a_id   uuid        NOT NULL REFERENCES tracks(id) ON DELETE CASCADE,
    track_b_id   uuid        NOT NULL REFERENCES tracks(id) ON DELETE CASCADE,
    score        DOUBLE PRECISION NOT NULL,
    source       TEXT        NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')),
    fetched_at   timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (track_a_id, track_b_id, source),
    CHECK (track_a_id <> track_b_id)
);

CREATE INDEX track_similarity_a_score_idx
    ON track_similarity (track_a_id, score DESC);

CREATE TABLE artist_similarity (
    artist_a_id  uuid        NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
    artist_b_id  uuid        NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
    score        DOUBLE PRECISION NOT NULL,
    source       TEXT        NOT NULL CHECK (source IN ('listenbrainz', 'musicbrainz_tag', 'user_cooccurrence')),
    fetched_at   timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (artist_a_id, artist_b_id, source),
    CHECK (artist_a_id <> artist_b_id)
);

CREATE INDEX artist_similarity_a_score_idx
    ON artist_similarity (artist_a_id, score DESC);
```

Down migration drops both tables.

**Notes:**
- Primary key includes `source` so the schema can hold multiple parallel
  similarity sources (per spec line 119).
- `(track_a_id, score DESC)` index matches the M4c hot-path query: "for seed
  T, give me top-N similar tracks descending by score."
- `CHECK (a <> b)` prevents self-edges.
- `ON DELETE CASCADE` from both endpoints so deleting a track cleans up edges
  on either side.

## 5. New sqlc queries

`internal/db/queries/similarity.sql`:

```sql
-- name: ListPlayedTracksNeedingSimilarity :many
SELECT DISTINCT t.id, t.mbid
FROM tracks t
JOIN play_events pe ON pe.track_id = t.id
WHERE t.mbid IS NOT NULL
  AND NOT EXISTS (
      SELECT 1 FROM track_similarity ts
      WHERE ts.track_a_id = t.id
        AND ts.source = 'listenbrainz'
        AND ts.fetched_at > now() - interval '7 days'
  )
ORDER BY t.id
LIMIT $1;

-- name: ListPlayedArtistsNeedingSimilarity :many
SELECT DISTINCT ar.id, ar.mbid
FROM artists ar
JOIN tracks t ON t.artist_id = ar.id
JOIN play_events pe ON pe.track_id = t.id
WHERE ar.mbid IS NOT NULL
  AND NOT EXISTS (
      SELECT 1 FROM artist_similarity asim
      WHERE asim.artist_a_id = ar.id
        AND asim.source = 'listenbrainz'
        AND asim.fetched_at > now() - interval '7 days'
  )
ORDER BY ar.id
LIMIT $1;

-- name: GetTracksByMBIDs :many
SELECT id, mbid FROM tracks WHERE mbid = ANY($1::text[]);

-- name: GetArtistsByMBIDs :many
SELECT id, mbid FROM artists WHERE mbid = ANY($1::text[]);

-- name: UpsertTrackSimilarity :exec
INSERT INTO track_similarity (track_a_id, track_b_id, score, source, fetched_at)
VALUES ($1, $2, $3, 'listenbrainz', now())
ON CONFLICT (track_a_id, track_b_id, source)
DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at;

-- name: UpsertArtistSimilarity :exec
INSERT INTO artist_similarity (artist_a_id, artist_b_id, score, source, fetched_at)
VALUES ($1, $2, $3, 'listenbrainz', now())
ON CONFLICT (artist_a_id, artist_b_id, source)
DO UPDATE SET score = EXCLUDED.score, fetched_at = EXCLUDED.fetched_at;
```

## 6. Worker algorithm

`tickOnce(ctx)`:

1. **Track pass:**
   - `q.ListPlayedTracksNeedingSimilarity(batch=5)` → `[(track_id, mbid)…]`
   - For each `(track_id, mbid)`:
     - Call `c.SimilarRecordings(ctx, mbid, 100)`
     - On 429 → log the `Retry-After` and **return from `tickOnce` early**
       (don't update `fetched_at`; the next hourly tick will pick up the
       work, which is virtually always longer than LB's `Retry-After`)
     - On other error → log warn, skip (no `fetched_at` update; next tick
       retries)
     - On success: collect returned MBIDs → `q.GetTracksByMBIDs(returnedMBIDs)`
       → take top 20 by score → for each `(local_id, score)` call
       `q.UpsertTrackSimilarity(track_id, local_id, score)`
2. **Artist pass:** symmetric, using `ListPlayedArtistsNeedingSimilarity`,
   `SimilarArtists`, `GetArtistsByMBIDs`, `UpsertArtistSimilarity`.

**Constants:**
- Tick interval: 1 hour (production); injectable to ms-scale for tests.
- Batch size: 5 (production). 5 LB calls per pass × 2 passes = 10 LB calls/tick
  → ~240/day, well under LB's documented rate limits (~100/5min unauth).
- Top-K: 20 per LB query.
- LB algorithm: hardcoded constant in the client (use LB's documented default
  at implementation time).

## 7. Test plan

### 7.1 LB client unit tests (httptest)

In `internal/scrobble/listenbrainz/client_test.go`, add 7 tests for each new
method:

For `SimilarRecordings`:
- 200 + valid body → returns slice ordered by score
- 401 → `ErrAuth` (defensive — public endpoint shouldn't 401)
- 400 → `ErrPermanent`
- 503 → `ErrTransient`
- 429 with `Retry-After` → `*RetryAfterError`
- URL contains `algorithm=…`
- URL contains `limit=N`

Same 7 tests for `SimilarArtists`.

### 7.2 Worker integration tests (live DB + httptest)

In `internal/similarity/worker_integration_test.go`:

- `TickOnce_NoPlayedTracks_NoOp`: empty `play_events` → returns nil, no rows
  in `track_similarity`
- `TickOnce_MapsLBResponseToLocalLibrary`: seed one played track with MBID;
  LB returns 3 MBIDs (2 in library, 1 not); assert 2 rows inserted
- `TickOnce_TopKEnforced`: LB returns 50; assert
  `count(*) WHERE track_a_id = $1` ≤ 20
- `TickOnce_RespectsSevenDayCap`: row with `fetched_at = now()` → not
  re-queried (the LB endpoint isn't called)
- `TickOnce_RefreshesStaleRow`: row with `fetched_at = now() - interval '8
  days'` → re-fetched, score updated, fetched_at bumps
- `TickOnce_429AbortsTick`: first response 429 with Retry-After → tickOnce
  returns early; no `fetched_at` updates; subsequent tracks in the batch
  are NOT processed (they're picked up on the next tick)
- `TickOnce_TransientErrorSkipsTrack`: 503 on one track → `fetched_at`
  unchanged; other tracks in same batch process normally
- `TickOnce_FiltersInLibrary`: LB returns 5 MBIDs none of which are in the
  library → 0 rows inserted (no error)
- `TickOnce_ArtistPassMirrors`: same coverage for the artist branch
- `TickOnce_NoMBIDOnTrack_Skipped`: track with `mbid IS NULL` → not selected
  by `ListPlayedTracksNeedingSimilarity`

### 7.3 Coverage target

`internal/similarity` ≥ 75%. The worker has fewer error branches than M4a
because passive retry-via-timer eliminates the durable-queue state machine.

The new `Similar*` methods in `internal/scrobble/listenbrainz` add ~14 tests
to that package; should keep its coverage ≥ 85%.

### 7.4 Manual verification post-merge

1. Restart server with M4b code.
2. After ≥1 hour, `psql -c "SELECT count(*) FROM track_similarity WHERE
   source = 'listenbrainz'"` → non-zero (assuming user has any MBID-tagged
   played tracks).
3. After ≥7 days, observe a `fetched_at` timestamp that's recent —
   confirms re-fetch cadence.
4. If MBID coverage in the library is sparse, the table will be small. Not a
   bug — M5 (Lidarr suggested-additions) is the eventual answer for
   tracks-not-in-library; tagging coverage via Picard is a separate user-side
   improvement.

## 8. Backwards compatibility

- New migration; no changes to existing schema.
- New package; no changes to consumer code (M4c will consume; M3's `Score()`
  doesn't need the data until then).
- `MaybeEnqueue` and other M4a paths unchanged.
- Worker is new; production behavior identical to pre-M4b until the worker's
  first tick fires (1 hour after restart).

## 9. Decisions ledger

| # | Decision | Rationale |
|---|---|---|
| 1 | Both `track_similarity` and `artist_similarity` in M4b | Symmetric pattern, M4c needs both, single-PR cost is small |
| 2 | Played-tracks-only input scope | Bounds work to user's actual interaction graph; LB's response naturally surfaces in-library tracks the user hasn't played, preserving discovery |
| 3 | Hourly tick, batch=5 | 240 LB calls/day fits well under LB rate limits; initial backfill in 24-48h |
| 4 | Top-K=20 per LB query, in-library only | Cuts long-tail noise; out-of-library tracks deferred to M5/Lidarr |
| 5 | Public endpoint, no auth | Similarity data is global to the instance; LB requires no token for `/explore/*` |
| 6 | Passive retry via timer (no durable queue) | Failure cost is "1 hour of staleness"; durable queue would be over-engineering vs. M4a's "lost scrobble" stakes |
| 7 | Hardcoded `algorithm` parameter | YAGNI; expose as YAML if telemetry warrants |

## 10. Sub-plan progression (M4)

- M4a (done) — outbound scrobble worker (PR #26).
- **M4b (this) — inbound LB similarity ingest.**
- M4c — radio similarity-driven candidate pool + queue refresh at 80%
  (closes M4). M4c also picks up the discovery-mitigation work flagged in
  brainstorm: serendipity floor (% random library picks), fallback to wider
  pool when similarity-row count is sparse, lazy fetch on radio for
  never-played seeds.