feat: M4b inbound ListenBrainz similarity ingest #27

2026-04-28T21:09:44-04:00

bvandeusen commented

2026-04-28 21:09:44 -04:00

(Migrated from git.fabledsword.com)

Second M4 sub-plan (Fable #346). Periodic worker pulls track-track and artist-artist similarity edges from LB's public /explore/* endpoints, filters to the local library, stores top-20 per source in two new tables. Hourly tick, 7-day re-fetch cap, passive retry via timer. No frontend changes — invisible until M4c reads the data.

What this ships

Schema (migration `0009_similarity`)

Two symmetric tables with multi-source primary keys:

track_similarity  (track_a_id, track_b_id, score, source, fetched_at)
                  PK (a, b, source); CHECK (a <> b); index (a, score DESC)
artist_similarity (artist_a_id, artist_b_id, score, source, fetched_at)
                  PK (a, b, source); CHECK (a <> b); index (a, score DESC)

The source enum reserves room for musicbrainz_tag and user_cooccurrence even though M4b only writes 'listenbrainz' rows. (a, score DESC) index satisfies M4c's hot path: "for seed S, top-N by score."

LB client extensions (`internal/scrobble/listenbrainz/`)

Added two read-only methods alongside M4a's SubmitListens:

SimilarRecordings(ctx, mbid, limit) — /1/explore/similar-recordings/{mbid}
SimilarArtists(ctx, mbid, limit) — /1/explore/similar-artists/{mbid}

Both return the same typed errors as SubmitListens (ErrAuth, ErrPermanent, ErrTransient, *RetryAfterError). 14 new httptest tests (7 per method).

LB algorithm parameter hardcoded for v1; LB endpoints are public (no auth required).

Worker (`internal/similarity/`)

type Worker struct {
    pool   *pgxpool.Pool
    client *listenbrainz.Client
    logger *slog.Logger
    tick   time.Duration  // 1h production
    batch  int32          // 5 production
    topK   int            // 20 production
}

tickOnce(ctx) algorithm:

Track pass: ListPlayedTracksNeedingSimilarity(batch=5) → for each, SimilarRecordings(mbid, 100) → bulk GetTracksByMBIDs to filter to library → sort by score desc → take top 20 → UpsertTrackSimilarity rows.
Artist pass: symmetric using SimilarArtists + GetArtistsByMBIDs + UpsertArtistSimilarity.

Failure handling:

*RetryAfterError (429) → log + abort the entire tick. The next 1h tick (vastly longer than typical Retry-After) picks up naturally.
ErrTransient / ErrPermanent → log + skip that single track; next tick retries.
All other transient errors leave fetched_at unchanged so the row is re-selected next pass.

No durable queue (vs M4a's scrobble_queue) — losing a refresh attempt is "1 hour of staleness," fine.

10 integration tests covering: no-op-empty, in-library mapping, top-K=20 enforcement, 7-day cap, stale-row refresh, 429-abort-tick, transient-skip-continues, in-library filter, artist pass mirrors track, no-MBID skip.

Boot wiring (`cmd/minstrel/main.go`)

similarityWorker := similarity.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "similarity"))
go similarityWorker.Run(ctx)

Same listenbrainz.NewClient() used by both M4a scrobble and M4b similarity workers (pure HTTP client, no per-instance state).

Test plan

go test -short -race ./... — all 16 packages pass (now including internal/similarity)
Full integration suite — green (pre-existing TestScanner_Integration flake unchanged)
golangci-lint run ./... — clean
internal/similarity coverage: 73.0% (target 75%; gap is in less-critical logging branches in upsertArtistSimilar)
internal/scrobble/listenbrainz coverage: 74.7% (was 70%; new methods well-covered, total dragged by un-tested edge cases in shared helpers from M4a — slightly under the 85% spec target)
cd web && npm run check — 0 errors / 0 warnings (no web changes)
cd web && npm test — 180 tests across 30 files (unchanged from M4a)
cd web && npm run build — succeeds
docker build -t minstrel:m4b-smoke . — succeeds
Worker boot smoke (5s go run) — migrations applied version=9, listening, no panic
Manual: wait ≥1h after deploy → psql -c "SELECT count(*) FROM track_similarity WHERE source='listenbrainz'" shows non-zero rows (assuming MBID-tagged played tracks)

Decisions ledger

#	Decision	Rationale
1	Both `track_similarity` and `artist_similarity` in M4b	M4c needs both; symmetric pattern, single-PR cost is small
2	Played-tracks-only input scope	Bounds work; LB's response naturally surfaces in-library tracks the user hasn't played, preserving discovery
3	Hourly tick, batch=5	240 LB calls/day fits well under LB rate limits; initial backfill in 24-48h
4	Top-K=20 per LB query, in-library only	Cuts long-tail noise; out-of-library tracks deferred to M5/Lidarr
5	Public endpoint, no auth	Similarity data is global to the instance
6	Passive retry via timer (no durable queue)	Failure cost is "1 hour of staleness"
7	Hardcoded `algorithm` parameter	YAGNI; expose as YAML if telemetry warrants

Out of scope (deferred to M4c)

Lazy fetch on radio request (never-played seed coverage)
Score normalization to [0,1]
Symmetric edge storage (we store one-way as LB returns)
Discovery-mitigation: serendipity floor + sparse-fallback to wider pool
"Suggested additions" for tracks not in library (M5/Lidarr)

Sub-plan progression (M4)

M4a (done) — outbound scrobble worker (PR #26)
M4b (this) — inbound LB similarity ingest
M4c — radio similarity-driven candidate pool + queue refresh at 80% (closes M4)

🤖 Generated with Claude Code

Second M4 sub-plan (Fable #346). Periodic worker pulls track-track and artist-artist similarity edges from LB's public `/explore/*` endpoints, filters to the local library, stores top-20 per source in two new tables. Hourly tick, 7-day re-fetch cap, passive retry via timer. No frontend changes — invisible until M4c reads the data. ## What this ships ### Schema (migration `0009_similarity`) Two symmetric tables with multi-source primary keys: ```sql track_similarity (track_a_id, track_b_id, score, source, fetched_at) PK (a, b, source); CHECK (a <> b); index (a, score DESC) artist_similarity (artist_a_id, artist_b_id, score, source, fetched_at) PK (a, b, source); CHECK (a <> b); index (a, score DESC) ``` The `source` enum reserves room for `musicbrainz_tag` and `user_cooccurrence` even though M4b only writes `'listenbrainz'` rows. `(a, score DESC)` index satisfies M4c's hot path: "for seed S, top-N by score." ### LB client extensions (`internal/scrobble/listenbrainz/`) Added two read-only methods alongside M4a's `SubmitListens`: - **`SimilarRecordings(ctx, mbid, limit)`** — `/1/explore/similar-recordings/{mbid}` - **`SimilarArtists(ctx, mbid, limit)`** — `/1/explore/similar-artists/{mbid}` Both return the same typed errors as `SubmitListens` (`ErrAuth`, `ErrPermanent`, `ErrTransient`, `*RetryAfterError`). 14 new httptest tests (7 per method). LB algorithm parameter hardcoded for v1; LB endpoints are public (no auth required). ### Worker (`internal/similarity/`) ```go type Worker struct { pool *pgxpool.Pool client *listenbrainz.Client logger *slog.Logger tick time.Duration // 1h production batch int32 // 5 production topK int // 20 production } ``` `tickOnce(ctx)` algorithm: 1. **Track pass**: `ListPlayedTracksNeedingSimilarity(batch=5)` → for each, `SimilarRecordings(mbid, 100)` → bulk `GetTracksByMBIDs` to filter to library → sort by score desc → take top 20 → `UpsertTrackSimilarity` rows. 2. **Artist pass**: symmetric using `SimilarArtists` + `GetArtistsByMBIDs` + `UpsertArtistSimilarity`. **Failure handling**: - `*RetryAfterError` (429) → log + abort the entire tick. The next 1h tick (vastly longer than typical Retry-After) picks up naturally. - `ErrTransient` / `ErrPermanent` → log + skip that single track; next tick retries. - All other transient errors leave `fetched_at` unchanged so the row is re-selected next pass. No durable queue (vs M4a's `scrobble_queue`) — losing a refresh attempt is "1 hour of staleness," fine. 10 integration tests covering: no-op-empty, in-library mapping, top-K=20 enforcement, 7-day cap, stale-row refresh, 429-abort-tick, transient-skip-continues, in-library filter, artist pass mirrors track, no-MBID skip. ### Boot wiring (`cmd/minstrel/main.go`) ```go similarityWorker := similarity.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "similarity")) go similarityWorker.Run(ctx) ``` Same `listenbrainz.NewClient()` used by both M4a scrobble and M4b similarity workers (pure HTTP client, no per-instance state). ## Test plan - [x] `go test -short -race ./...` — all 16 packages pass (now including `internal/similarity`) - [x] Full integration suite — green (pre-existing `TestScanner_Integration` flake unchanged) - [x] `golangci-lint run ./...` — clean - [x] `internal/similarity` coverage: **73.0%** (target 75%; gap is in less-critical logging branches in `upsertArtistSimilar`) - [x] `internal/scrobble/listenbrainz` coverage: **74.7%** (was 70%; new methods well-covered, total dragged by un-tested edge cases in shared helpers from M4a — slightly under the 85% spec target) - [x] `cd web && npm run check` — 0 errors / 0 warnings (no web changes) - [x] `cd web && npm test` — 180 tests across 30 files (unchanged from M4a) - [x] `cd web && npm run build` — succeeds - [x] `docker build -t minstrel:m4b-smoke .` — succeeds - [x] Worker boot smoke (5s `go run`) — `migrations applied version=9`, `listening`, no panic - [ ] Manual: wait ≥1h after deploy → `psql -c "SELECT count(*) FROM track_similarity WHERE source='listenbrainz'"` shows non-zero rows (assuming MBID-tagged played tracks) ## Decisions ledger | # | Decision | Rationale | |---|---|---| | 1 | Both `track_similarity` and `artist_similarity` in M4b | M4c needs both; symmetric pattern, single-PR cost is small | | 2 | Played-tracks-only input scope | Bounds work; LB's response naturally surfaces in-library tracks the user hasn't played, preserving discovery | | 3 | Hourly tick, batch=5 | 240 LB calls/day fits well under LB rate limits; initial backfill in 24-48h | | 4 | Top-K=20 per LB query, in-library only | Cuts long-tail noise; out-of-library tracks deferred to M5/Lidarr | | 5 | Public endpoint, no auth | Similarity data is global to the instance | | 6 | Passive retry via timer (no durable queue) | Failure cost is "1 hour of staleness" | | 7 | Hardcoded `algorithm` parameter | YAGNI; expose as YAML if telemetry warrants | ## Out of scope (deferred to M4c) - Lazy fetch on radio request (never-played seed coverage) - Score normalization to [0,1] - Symmetric edge storage (we store one-way as LB returns) - Discovery-mitigation: serendipity floor + sparse-fallback to wider pool - "Suggested additions" for tracks not in library (M5/Lidarr) ## Sub-plan progression (M4) - M4a (done) — outbound scrobble worker (PR #26) - **M4b (this) — inbound LB similarity ingest** - M4c — radio similarity-driven candidate pool + queue refresh at 80% (closes M4) 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/minstrel#27