From 267c4ad80bbf522b3d3888c41393dffb69d0ee6c Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Tue, 28 Apr 2026 23:02:01 -0400 Subject: [PATCH] docs(spec): add M4c radio similarity-driven candidate pool design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third and final M4 sub-plan (Fable #347) — closes M4. Replaces M3's whole-library candidate pool with a 5-way SQL UNION (LB-similar tracks / tracks by similar artists / MB-tag overlap / likes-overlap / random fill). Adds a new SimilarityScore × SimilarityWeight term to M3's Score() formula. Web client auto-refreshes the queue when 80% consumed via a new ?exclude= query param. No lazy LB fetch (M4b's worker + random augmentation handle sparse data). Fallback to M3 LoadCandidates on any similarity-pool error. Co-Authored-By: Claude Opus 4.7 --- .../specs/2026-04-28-m4c-radio-design.md | 444 ++++++++++++++++++ 1 file changed, 444 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-28-m4c-radio-design.md diff --git a/docs/superpowers/specs/2026-04-28-m4c-radio-design.md b/docs/superpowers/specs/2026-04-28-m4c-radio-design.md new file mode 100644 index 00000000..320c54d3 --- /dev/null +++ b/docs/superpowers/specs/2026-04-28-m4c-radio-design.md @@ -0,0 +1,444 @@ +# M4c — Radio similarity-driven candidate pool + queue refresh at 80% (closes M4) + +**Status:** Spec draft, 2026-04-28 +**Tracking:** Fable #347 +**Milestone:** M4 — ListenBrainz scrobble + similarity + radio +**Builds on:** M4a (PR #26 — outbound scrobble), M4b (PR #27 — inbound similarity ingest) + +## 1. Goal + +Replace M3's "whole library minus seed minus recently-played" candidate pool +with a similarity-driven pool drawn from four sources (LB-similar tracks, +tracks by similar artists, MB-tag overlap, user's general likes overlapping +seed tags) plus a random-fill source that guarantees a minimum pool size. +Add a sixth scoring term (`SimilarityScore × SimilarityWeight`) so within-pool +ranking reflects per-track similarity strength. Web client auto-refreshes +the radio queue when 80% consumed. + +When this slice merges, the M4 milestone closes: the engine has all three v1 +components (scoring, session vectors, LB-derived similarity) wired through +both backend and frontend. + +## 2. Non-goals (explicit) + +- **Lazy LB fetch on radio click** — radio handler stays synchronous and + uses only data already in `track_similarity` / `artist_similarity`. + Sparse-data UX is handled by random-fill augmentation. +- **YAML-configurable per-source Ks** — hardcoded constants for v1. +- **Symmetric edge storage** — still one-way as M4b stored. +- **Per-source score weights as YAML** — tunable in code only. +- **Token-based queue-refresh continuation** — explicit `?exclude=...` + query param. +- **Pre-fetch similarity on track play** — M4b's hourly tick is the only + fetch trigger. +- **Suggested additions / Lidarr** — out-of-library LB matches discarded; + M5 territory. +- **Multi-seed / persistent radio stations** — every call is single-seed + and stateless. +- **Cross-user collaborative filtering source** — `user_cooccurrence` + schema slot reserved, not populated. +- **`SimilarityWeight` per-user override** — operator-only YAML for v1. + +## 3. Architecture overview + +``` +GET /api/radio?seed_track=&limit=N&exclude=t1,t2,... + ↓ +handleRadio + 1. Auth + parse params (existing M3) + 2. Get seed track + album + artist (existing) + 3. q.GetCurrentSessionVectorForUser() (existing — M3) + 4. recommendation.LoadCandidatesFromSimilarity(...) ← NEW + - 5-way SQL UNION: LB-similar / similar-artist tracks / + tag-overlap / likes-overlap / random-fill + - Excludes seed + ?exclude= list + recently-played + - Returns []Candidate with per-row SimilarityScore + - On error → fallback to M3's LoadCandidates (logged) + 5. recommendation.Shuffle(candidates, weights, ...) ← extended + - M3's Score() gains SimilarityScore × SimilarityWeight term + - Otherwise unchanged + 6. Resolve album/artist for picks (existing) + 7. Return RadioResponse{Tracks: [seed, pick1, ...]} + ↓ + Web client + 8. Player store watches currentIndex / queue.length ← NEW + - At ≥ 80% consumed, AND radioSeedId is set, AND no refresh + in-flight: GET /api/radio?seed_track=…&exclude= + - Append response.tracks (skipping index 0 = seed already in queue) + - radioSeedId cleared when user manually enqueues from elsewhere +``` + +## 4. Candidate pool SQL (`LoadRadioCandidatesV2`) + +5-way UNION; each branch produces `(track_id, similarity_score)`. After UNION, +the outer SELECT joins `tracks` + general_likes (for `is_liked`) + LATERAL +play_events aggregation (for `last_played_at` / `play_count` / `skip_count`), +GROUP BY track id, taking `max(similarity_score)` across sources. + +```sql +-- name: LoadRadioCandidatesV2 :many +WITH +seed_artist AS (SELECT artist_id FROM tracks WHERE id = $2), +seed_tags AS ( + SELECT trim(g) AS tag + FROM tracks t, + regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g + WHERE t.id = $2 AND trim(g) <> '' +), +exclude_set AS ( + SELECT unnest($5::uuid[]) AS id + UNION ALL + SELECT track_id FROM play_events + WHERE user_id = $1 AND started_at > now() - $4 * interval '1 hour' +), +lb_similar AS ( + SELECT ts.track_b_id AS id, ts.score AS sim_score + FROM track_similarity ts + WHERE ts.track_a_id = $2 + AND ts.source = 'listenbrainz' + AND ts.track_b_id NOT IN (SELECT id FROM exclude_set) + ORDER BY ts.score DESC + LIMIT $6 +), +similar_artists AS ( + SELECT t.id, asim.score * 0.5 AS sim_score + FROM artist_similarity asim + JOIN tracks t ON t.artist_id = asim.artist_b_id + JOIN seed_artist sa ON asim.artist_a_id = sa.artist_id + WHERE asim.source = 'listenbrainz' + AND t.id NOT IN (SELECT id FROM exclude_set) + ORDER BY asim.score DESC, random() + LIMIT $7 +), +tag_overlap AS ( + SELECT t.id, + (count(DISTINCT trim(g_raw))::float8 + / GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score + FROM tracks t, + regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_raw + WHERE trim(g_raw) IN (SELECT tag FROM seed_tags) + AND t.id NOT IN (SELECT id FROM exclude_set) + AND t.id <> $2 + GROUP BY t.id + HAVING count(DISTINCT trim(g_raw)) > 0 + ORDER BY sim_score DESC + LIMIT $8 +), +likes_overlap AS ( + SELECT gl.track_id AS id, 0.6::float8 AS sim_score + FROM general_likes gl + JOIN tracks t ON t.id = gl.track_id + WHERE gl.user_id = $1 + AND t.id NOT IN (SELECT id FROM exclude_set) + AND EXISTS ( + SELECT 1 FROM regexp_split_to_table(coalesce(t.genre, ''), '[;,]') g + WHERE trim(g) IN (SELECT tag FROM seed_tags) + ) + ORDER BY random() + LIMIT $9 +), +random_fill AS ( + SELECT t.id, 0.0::float8 AS sim_score + FROM tracks t + WHERE t.id NOT IN (SELECT id FROM exclude_set) + AND t.id <> $2 + AND t.id NOT IN ( + SELECT id FROM lb_similar + UNION SELECT id FROM similar_artists + UNION SELECT id FROM tag_overlap + UNION SELECT id FROM likes_overlap + ) + ORDER BY random() + LIMIT $10 +) +SELECT + sqlc.embed(t), + (l.user_id IS NOT NULL)::bool AS is_liked, + pe.last_played_at::timestamptz AS last_played_at, + pe.play_count, + pe.skip_count, + max(u.sim_score) AS similarity_score +FROM ( + SELECT * FROM lb_similar + UNION ALL SELECT * FROM similar_artists + UNION ALL SELECT * FROM tag_overlap + UNION ALL SELECT * FROM likes_overlap + UNION ALL SELECT * FROM random_fill +) u +JOIN tracks t ON t.id = u.id +LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id +LEFT JOIN LATERAL ( + SELECT max(started_at) AS last_played_at, + count(*) AS play_count, + count(*) FILTER (WHERE was_skipped) AS skip_count + FROM play_events + WHERE user_id = $1 AND track_id = t.id +) pe ON true +GROUP BY t.id, t.title, t.album_id, t.artist_id, t.duration_ms, t.file_path, + t.file_format, t.file_size, t.bitrate, t.track_number, t.disc_number, + t.mbid, t.genre, t.created_at, t.updated_at, + l.user_id, pe.last_played_at, pe.play_count, pe.skip_count; +``` + +### 4.1 Per-source K and score (defaults, hardcoded for v1) + +| Source | K | sim_score per row | +|---|---|---| +| `lb_similar` (track_similarity) | 30 | LB raw score (0–1) | +| `similar_artists` | 30 | `artist_similarity.score × 0.5` | +| `tag_overlap` | 20 | jaccard: `shared_tags / seed_tag_count` | +| `likes_overlap` | 20 | constant `0.6` | +| `random_fill` | 30 | `0.0` | + +Total ideal: 130 candidates pre-dedup; 60–100 after dedup. Random fill is +drawn AFTER the 4 similarity sources are exhausted (`NOT IN (lb_similar +UNION similar_artists UNION tag_overlap UNION likes_overlap)`), so it +strictly augments rather than overlaps. On very small libraries (<130 +tracks total), the pool is naturally smaller — there is no hard floor; +the design assumes typical libraries have hundreds of tracks. M3's `Score()` ++ `Shuffle()` happily ranks small pools. + +`max(sim_score)` on dedup so a track in multiple sources keeps its strongest +signal (LB's 0.85 beats tag's 0.4). + +## 5. Score() formula extension + +`internal/recommendation/score.go`: + +```go +type ScoringInputs struct { + IsGeneralLiked bool + LastPlayedAt *time.Time + PlayCount int + SkipCount int + ContextualMatchScore float64 // M3 + SimilarityScore float64 // NEW — max across the 4 similarity sources, in [0,1] +} + +type ScoringWeights struct { + BaseWeight float64 + LikeBoost float64 + RecencyWeight float64 + SkipPenalty float64 + JitterMagnitude float64 + ContextWeight float64 // M3 + SimilarityWeight float64 // NEW — default 2.0 +} +``` + +Updated formula: + +``` +score = base + + (is_general_liked ? LikeBoost : 0) + + recency_decay * RecencyWeight + - skip_ratio * SkipPenalty + + contextual_match_score * ContextWeight + + similarity_score * SimilarityWeight ← NEW + + jitter +``` + +`config.RecommendationConfig` gains `SimilarityWeight float64` (yaml +`similarity_weight`, default `2.0`). Same magnitude as `LikeBoost` and +`ContextWeight` — at perfect similarity (1.0), an LB-similar track gets a ++2.0 boost equivalent to an explicit general like. + +**Backwards compatible:** zero-value `ScoringInputs{}` and `ScoringWeights{}` +produce M3 behavior because both new fields are zero-defaulted. + +## 6. Go-side wiring + +### 6.1 `LoadCandidatesFromSimilarity` + +`internal/recommendation/candidates.go` gains a sibling to `LoadCandidates`: + +```go +type CandidateSourceLimits struct { + LBSimilar int // 30 + SimilarArtist int // 30 + TagOverlap int // 20 + LikesOverlap int // 20 + RandomFill int // 30 — drawn from tracks NOT already returned by the 4 similarity sources +} + +func DefaultCandidateSourceLimits() CandidateSourceLimits + +// LoadCandidatesFromSimilarity is M4c's primary candidate-pool loader. +// Returns []Candidate (same type as M3 LoadCandidates) so Shuffle() is +// unchanged. Caller falls back to LoadCandidates on error. +func LoadCandidatesFromSimilarity( + ctx context.Context, + q *dbq.Queries, + userID, seedID pgtype.UUID, + recentlyPlayedHours int, + currentVector SessionVector, + exclude []pgtype.UUID, + limits CandidateSourceLimits, +) ([]Candidate, error) +``` + +Body: +1. `q.LoadRadioCandidatesV2(...)` with the 10 params from §4 +2. Existing `loadContextualLikesByTrack(...)` for the contextual scoring inputs +3. Project rows → `[]Candidate` with `Inputs.SimilarityScore = row.SimilarityScore` + +`LoadCandidates` (M3 fallback) **stays in place**, still consumed by callers +that want whole-library scoring (unit tests, the radio handler's error path). + +### 6.2 Radio handler change + +`internal/api/radio.go`: + +```go +exclude := parseExcludeParam(r.URL.Query().Get("exclude")) // []pgtype.UUID +exclude = append(exclude, seedID) // always exclude the seed +limits := recommendation.DefaultCandidateSourceLimits() + +candidates, err := recommendation.LoadCandidatesFromSimilarity( + r.Context(), q, user.ID, seedID, + h.recCfg.RecentlyPlayedHours, currentVec, + exclude, limits, +) +if err != nil { + h.logger.Warn("api: radio: similarity-pool failed, falling back", + "err", err) + candidates, err = recommendation.LoadCandidates( + r.Context(), q, user.ID, seedID, + h.recCfg.RecentlyPlayedHours, currentVec, + ) + if err != nil { + h.logger.Error("api: radio: load candidates", "err", err) + writeErr(w, http.StatusInternalServerError, "server_error", + "candidate load failed") + return + } +} + +weights := recommendation.ScoringWeights{ + BaseWeight: h.recCfg.BaseWeight, + LikeBoost: h.recCfg.LikeBoost, + RecencyWeight: h.recCfg.RecencyWeight, + SkipPenalty: h.recCfg.SkipPenalty, + JitterMagnitude: h.recCfg.JitterMagnitude, + ContextWeight: h.recCfg.ContextWeight, + SimilarityWeight: h.recCfg.SimilarityWeight, // NEW +} +``` + +`parseExcludeParam(s string) []pgtype.UUID` — splits on `,`, parses each +UUID, silently drops malformed entries. Returns nil for empty input. + +## 7. Frontend (queue refresh at 80%) + +`web/src/lib/player/store.svelte.ts`: + +- New `radioSeedId` `$state` set inside `playRadio()`. +- New `$effect` watching `(player.currentIndex + 1) / player.queue.length`: + - Fires when `radioSeedId` is set, queue is non-empty, ratio ≥ 0.8, and + no refresh in-flight. + - Calls `/api/radio?seed_track=&exclude=t.id).join(',')>`. + - On success: appends `response.tracks.slice(1)` (drops the seed at index 0). + - On failure: logs + clears in-flight flag (next track-advance can retry). +- New `appendToQueue(tracks)` helper — pushes onto `player.queue` without + changing `currentIndex`. +- `radioSeedId = null` when user enqueues from non-radio paths (`playQueue`, + `enqueueTrack`, `enqueueTracks`) so the auto-refresh doesn't fire on + manually-built queues. + +`RadioResponse` JSON shape unchanged. `TrackRef[]` array works for both +initial calls and refresh calls. + +## 8. Test plan + +### 8.1 Backend pure tests + +`internal/recommendation/score_test.go` extensions: +- `TestScore_SimilarityScore_PerfectMatch_AddsWeightedTerm` — `1.0 × 2.0 = +2.0` over baseline +- `TestScore_SimilarityScore_HalfMatch` — `0.5 × 2.0 = +1.0` +- `TestScore_SimilarityScore_Zero_NoEffect` — random/serendipity tracks score same as M3 baseline + +### 8.2 Backend integration tests + +`internal/recommendation/candidates_v2_test.go` (new): +- `TestLoadCandidatesFromSimilarity_LBSimilarSourceContributes` +- `TestLoadCandidatesFromSimilarity_SimilarArtistTracksContribute` (artist score × 0.5 verified) +- `TestLoadCandidatesFromSimilarity_TagOverlapContributes` (jaccard score) +- `TestLoadCandidatesFromSimilarity_LikesOverlapContributes` (0.6 constant) +- `TestLoadCandidatesFromSimilarity_RandomFillToTargetSize` (empty similarity tables → pool ≥ 60) +- `TestLoadCandidatesFromSimilarity_ExcludeListRespected` +- `TestLoadCandidatesFromSimilarity_RecentlyPlayedExcluded` +- `TestLoadCandidatesFromSimilarity_DedupTakesMaxScore` (LB 0.85 beats tag 0.4) +- `TestLoadCandidatesFromSimilarity_SeedAlwaysExcluded` +- `TestLoadCandidatesFromSimilarity_EmptyLibrary_NoError` + +### 8.3 Backend HTTP tests + +`internal/api/radio_test.go` extensions: +- `TestHandleRadio_WithSimilarityPool_RanksLBSimilarHigher` (deterministic via fixed RNG) +- `TestHandleRadio_ExcludeParam_FiltersOut` +- `TestHandleRadio_ExcludeParam_MalformedSkipped` +- `TestHandleRadio_FallbackToM3OnSimilarityError` (inject fault) + +### 8.4 Frontend tests + +`web/src/lib/player/store.test.ts` extensions: +- `radio refresh fires at 80% queue consumption` (5-track queue at index 3) +- `radio refresh appends new tracks (excluding seed)` (queue 5 → 9 after 5-track response) +- `radio refresh does NOT double-fire` (in-flight guard) +- `radio refresh resets when user enqueues from non-radio source` +- `radio refresh below threshold` (5-track queue at index 2 → no refresh) + +### 8.5 Coverage targets + +- `internal/recommendation` post-M4c: ≥ 80% (currently 73%) +- `internal/api/radio.go`: ≥ 75% +- Web `player/store`: ≥ 80% on the new refresh logic + +### 8.6 Manual end-to-end gate (closes M4) + +After deploy + M4b worker has filled `track_similarity` for ≥10 played tracks: + +1. Click radio from a played track — queue should differ noticeably from M3 +2. Listen through ~80% of the queue — observe queue length increase as + auto-refresh fires +3. `psql -c "SELECT count(*) FROM track_similarity"` shows non-trivial data +4. Subjectively: radio quality should feel meaningfully better than M3's + baseline. **This is the closing gate for M4.** + +## 9. Decisions ledger + +| # | Decision | Rationale | +|---|---|---| +| 1 | 4-source pool composition + random fill | Cross-source diversity; sparse-fallback is automatic via random fill | +| 2 | Always augment with random to floor of 60 | Never returns empty radio; serendipity built in; degrades gracefully on sparse libraries | +| 3 | New `SimilarityScore × SimilarityWeight` term in M3 Score() | Wastes the LB scores otherwise; consistent with the M3 multi-input pattern | +| 4 | No lazy LB fetch | Keeps radio handler synchronous; M4b worker + augmentation cover the gap | +| 5 | `?exclude=...` query param for queue refresh | Stateless, simple, no new server state | +| 6 | Per-source K + score: hardcoded for v1 | YAGNI; expose later if telemetry warrants | +| 7 | Fallback to M3 LoadCandidates on similarity-pool error | Defense in depth for the central radio surface | + +## 10. Backwards compatibility + +- New SQL query alongside existing `LoadRadioCandidates`; M3 callers + unaffected. +- M3 `LoadCandidates` retained for the fallback path and direct test usage. +- `Score()` signature unchanged; new fields are zero-defaulted so existing + zero-value `ScoringInputs{}`/`ScoringWeights{}` constructions produce M3 + scores. +- `/api/radio` request shape extended (adds optional `?exclude=`); existing + callers that don't pass it work identically. +- `RadioResponse` shape unchanged. +- No schema migration — relies on M4b's `track_similarity` / + `artist_similarity` and existing M2 `general_likes` / M0-M1 `tracks.genre`. + +## 11. M4 closure + +After this slice merges, M4 is complete: +- M4a (PR #26) — outbound LB scrobble worker +- M4b (PR #27) — inbound LB similarity ingest +- M4c (this) — radio similarity-driven candidate pool + 80% queue refresh + +Unblocks M5 (Lidarr quarantine + suggested-additions for tracks not in +library). Out-of-library LB-returned tracks (currently filtered out by the +in-library JOIN) become the natural input for M5's "would you like to add +this?" workflow.