bvandeusen/minstrel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 38 Wiki Activity

docs(spec): add M4c radio similarity-driven candidate pool design

Browse Source 
Third and final M4 sub-plan (Fable #347) — closes M4. Replaces M3's
whole-library candidate pool with a 5-way SQL UNION (LB-similar tracks /
tracks by similar artists / MB-tag overlap / likes-overlap / random
fill). Adds a new SimilarityScore × SimilarityWeight term to M3's Score()
formula. Web client auto-refreshes the queue when 80% consumed via a new
?exclude= query param. No lazy LB fetch (M4b's worker + random
augmentation handle sparse data). Fallback to M3 LoadCandidates on any
similarity-pool error.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bryan Van Deusen
2026-04-28 23:02:01 -04:00
parent 358bfd9453
commit 267c4ad80b
1 changed files with 444 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-04-28-m4c-radio-design.md
+444
View File
@@ -0,0 +1,444 @@
# M4c — Radio similarity-driven candidate pool + queue refresh at 80% (closes M4)
**Status:** Spec draft, 2026-04-28
**Tracking:** Fable #347
**Milestone:** M4 — ListenBrainz scrobble + similarity + radio
**Builds on:** M4a (PR #26 — outbound scrobble), M4b (PR #27 — inbound similarity ingest)
## 1. Goal
Replace M3's "whole library minus seed minus recently-played" candidate pool
with a similarity-driven pool drawn from four sources (LB-similar tracks,
tracks by similar artists, MB-tag overlap, user's general likes overlapping
seed tags) plus a random-fill source that guarantees a minimum pool size.
Add a sixth scoring term (`SimilarityScore × SimilarityWeight`) so within-pool
ranking reflects per-track similarity strength. Web client auto-refreshes
the radio queue when 80% consumed.
When this slice merges, the M4 milestone closes: the engine has all three v1
components (scoring, session vectors, LB-derived similarity) wired through
both backend and frontend.
## 2. Non-goals (explicit)
- **Lazy LB fetch on radio click** — radio handler stays synchronous and
uses only data already in `track_similarity` / `artist_similarity`.
Sparse-data UX is handled by random-fill augmentation.
- **YAML-configurable per-source Ks** — hardcoded constants for v1.
- **Symmetric edge storage** — still one-way as M4b stored.
- **Per-source score weights as YAML** — tunable in code only.
- **Token-based queue-refresh continuation** — explicit `?exclude=...`
query param.
- **Pre-fetch similarity on track play** — M4b's hourly tick is the only
fetch trigger.
- **Suggested additions / Lidarr** — out-of-library LB matches discarded;
M5 territory.
- **Multi-seed / persistent radio stations** — every call is single-seed
and stateless.
- **Cross-user collaborative filtering source** — `user_cooccurrence`
schema slot reserved, not populated.
- **`SimilarityWeight` per-user override** — operator-only YAML for v1.
## 3. Architecture overview
```
GET /api/radio?seed_track=<uuid>&limit=N&exclude=t1,t2,...
↓
handleRadio
1. Auth + parse params (existing M3)
2. Get seed track + album + artist (existing)
3. q.GetCurrentSessionVectorForUser() (existing — M3)
4. recommendation.LoadCandidatesFromSimilarity(...) ← NEW
- 5-way SQL UNION: LB-similar / similar-artist tracks /
tag-overlap / likes-overlap / random-fill
- Excludes seed + ?exclude= list + recently-played
- Returns []Candidate with per-row SimilarityScore
- On error → fallback to M3's LoadCandidates (logged)
5. recommendation.Shuffle(candidates, weights, ...) ← extended
- M3's Score() gains SimilarityScore × SimilarityWeight term
- Otherwise unchanged
6. Resolve album/artist for picks (existing)
7. Return RadioResponse{Tracks: [seed, pick1, ...]}
↓
Web client
8. Player store watches currentIndex / queue.length ← NEW
- At ≥ 80% consumed, AND radioSeedId is set, AND no refresh
in-flight: GET /api/radio?seed_track=…&exclude=<queue>
- Append response.tracks (skipping index 0 = seed already in queue)
- radioSeedId cleared when user manually enqueues from elsewhere
```
## 4. Candidate pool SQL (`LoadRadioCandidatesV2`)
5-way UNION; each branch produces `(track_id, similarity_score)`. After UNION,
the outer SELECT joins `tracks` + general_likes (for `is_liked`) + LATERAL
play_events aggregation (for `last_played_at` / `play_count` / `skip_count`),
GROUP BY track id, taking `max(similarity_score)` across sources.
```sql
-- name: LoadRadioCandidatesV2 :many
WITH
seed_artist AS (SELECT artist_id FROM tracks WHERE id = $2),
seed_tags AS (
SELECT trim(g) AS tag
FROM tracks t,
regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g
WHERE t.id = $2 AND trim(g) <> ''
),
exclude_set AS (
SELECT unnest($5::uuid[]) AS id
UNION ALL
SELECT track_id FROM play_events
WHERE user_id = $1 AND started_at > now() - $4 * interval '1 hour'
),
lb_similar AS (
SELECT ts.track_b_id AS id, ts.score AS sim_score
FROM track_similarity ts
WHERE ts.track_a_id = $2
AND ts.source = 'listenbrainz'
AND ts.track_b_id NOT IN (SELECT id FROM exclude_set)
ORDER BY ts.score DESC
LIMIT $6
),
similar_artists AS (
SELECT t.id, asim.score * 0.5 AS sim_score
FROM artist_similarity asim
JOIN tracks t ON t.artist_id = asim.artist_b_id
JOIN seed_artist sa ON asim.artist_a_id = sa.artist_id
WHERE asim.source = 'listenbrainz'
AND t.id NOT IN (SELECT id FROM exclude_set)
ORDER BY asim.score DESC, random()
LIMIT $7
),
tag_overlap AS (
SELECT t.id,
(count(DISTINCT trim(g_raw))::float8
/ GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score
FROM tracks t,
regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_raw
WHERE trim(g_raw) IN (SELECT tag FROM seed_tags)
AND t.id NOT IN (SELECT id FROM exclude_set)
AND t.id <> $2
GROUP BY t.id
HAVING count(DISTINCT trim(g_raw)) > 0
ORDER BY sim_score DESC
LIMIT $8
),
likes_overlap AS (
SELECT gl.track_id AS id, 0.6::float8 AS sim_score
FROM general_likes gl
JOIN tracks t ON t.id = gl.track_id
WHERE gl.user_id = $1
AND t.id NOT IN (SELECT id FROM exclude_set)
AND EXISTS (
SELECT 1 FROM regexp_split_to_table(coalesce(t.genre, ''), '[;,]') g
WHERE trim(g) IN (SELECT tag FROM seed_tags)
)
ORDER BY random()
LIMIT $9
),
random_fill AS (
SELECT t.id, 0.0::float8 AS sim_score
FROM tracks t
WHERE t.id NOT IN (SELECT id FROM exclude_set)
AND t.id <> $2
AND t.id NOT IN (
SELECT id FROM lb_similar
UNION SELECT id FROM similar_artists
UNION SELECT id FROM tag_overlap
UNION SELECT id FROM likes_overlap
)
ORDER BY random()
LIMIT $10
)
SELECT
sqlc.embed(t),
(l.user_id IS NOT NULL)::bool AS is_liked,
pe.last_played_at::timestamptz AS last_played_at,
pe.play_count,
pe.skip_count,
max(u.sim_score) AS similarity_score
FROM (
SELECT * FROM lb_similar
UNION ALL SELECT * FROM similar_artists
UNION ALL SELECT * FROM tag_overlap
UNION ALL SELECT * FROM likes_overlap
UNION ALL SELECT * FROM random_fill
) u
JOIN tracks t ON t.id = u.id
LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id
LEFT JOIN LATERAL (
SELECT max(started_at) AS last_played_at,
count(*) AS play_count,
count(*) FILTER (WHERE was_skipped) AS skip_count
FROM play_events
WHERE user_id = $1 AND track_id = t.id
) pe ON true
GROUP BY t.id, t.title, t.album_id, t.artist_id, t.duration_ms, t.file_path,
t.file_format, t.file_size, t.bitrate, t.track_number, t.disc_number,
t.mbid, t.genre, t.created_at, t.updated_at,
l.user_id, pe.last_played_at, pe.play_count, pe.skip_count;
```
### 4.1 Per-source K and score (defaults, hardcoded for v1)
| Source | K | sim_score per row |
|---|---|---|
| `lb_similar` (track_similarity) | 30 | LB raw score (0–1) |
| `similar_artists` | 30 | `artist_similarity.score × 0.5` |
| `tag_overlap` | 20 | jaccard: `shared_tags / seed_tag_count` |
| `likes_overlap` | 20 | constant `0.6` |
| `random_fill` | 30 | `0.0` |
Total ideal: 130 candidates pre-dedup; 60–100 after dedup. Random fill is
drawn AFTER the 4 similarity sources are exhausted (`NOT IN (lb_similar
UNION similar_artists UNION tag_overlap UNION likes_overlap)`), so it
strictly augments rather than overlaps. On very small libraries (<130
tracks total), the pool is naturally smaller — there is no hard floor;
the design assumes typical libraries have hundreds of tracks. M3's `Score()`
+ `Shuffle()` happily ranks small pools.
`max(sim_score)` on dedup so a track in multiple sources keeps its strongest
signal (LB's 0.85 beats tag's 0.4).
## 5. Score() formula extension
`internal/recommendation/score.go`:
```go
type ScoringInputs struct {
IsGeneralLiked bool
LastPlayedAt *time.Time
PlayCount int
SkipCount int
ContextualMatchScore float64 // M3
SimilarityScore float64 // NEW — max across the 4 similarity sources, in [0,1]
}
type ScoringWeights struct {
BaseWeight float64
LikeBoost float64
RecencyWeight float64
SkipPenalty float64
JitterMagnitude float64
ContextWeight float64 // M3
SimilarityWeight float64 // NEW — default 2.0
}
```
Updated formula:
```
score = base
+ (is_general_liked ? LikeBoost : 0)
+ recency_decay * RecencyWeight
- skip_ratio * SkipPenalty
+ contextual_match_score * ContextWeight
+ similarity_score * SimilarityWeight ← NEW
+ jitter
```
`config.RecommendationConfig` gains `SimilarityWeight float64` (yaml
`similarity_weight`, default `2.0`). Same magnitude as `LikeBoost` and
`ContextWeight` — at perfect similarity (1.0), an LB-similar track gets a
+2.0 boost equivalent to an explicit general like.
**Backwards compatible:** zero-value `ScoringInputs{}` and `ScoringWeights{}`
produce M3 behavior because both new fields are zero-defaulted.
## 6. Go-side wiring
### 6.1 `LoadCandidatesFromSimilarity`
`internal/recommendation/candidates.go` gains a sibling to `LoadCandidates`:
```go
type CandidateSourceLimits struct {
LBSimilar int // 30
SimilarArtist int // 30
TagOverlap int // 20
LikesOverlap int // 20
RandomFill int // 30 — drawn from tracks NOT already returned by the 4 similarity sources
}
func DefaultCandidateSourceLimits() CandidateSourceLimits
// LoadCandidatesFromSimilarity is M4c's primary candidate-pool loader.
// Returns []Candidate (same type as M3 LoadCandidates) so Shuffle() is
// unchanged. Caller falls back to LoadCandidates on error.
func LoadCandidatesFromSimilarity(
ctx context.Context,
q *dbq.Queries,
userID, seedID pgtype.UUID,
recentlyPlayedHours int,
currentVector SessionVector,
exclude []pgtype.UUID,
limits CandidateSourceLimits,
) ([]Candidate, error)
```
Body:
1. `q.LoadRadioCandidatesV2(...)` with the 10 params from §4
2. Existing `loadContextualLikesByTrack(...)` for the contextual scoring inputs
3. Project rows → `[]Candidate` with `Inputs.SimilarityScore = row.SimilarityScore`
`LoadCandidates` (M3 fallback) **stays in place**, still consumed by callers
that want whole-library scoring (unit tests, the radio handler's error path).
### 6.2 Radio handler change
`internal/api/radio.go`:
```go
exclude := parseExcludeParam(r.URL.Query().Get("exclude")) // []pgtype.UUID
exclude = append(exclude, seedID) // always exclude the seed
limits := recommendation.DefaultCandidateSourceLimits()
candidates, err := recommendation.LoadCandidatesFromSimilarity(
r.Context(), q, user.ID, seedID,
h.recCfg.RecentlyPlayedHours, currentVec,
exclude, limits,
)
if err != nil {
h.logger.Warn("api: radio: similarity-pool failed, falling back",
"err", err)
candidates, err = recommendation.LoadCandidates(
r.Context(), q, user.ID, seedID,
h.recCfg.RecentlyPlayedHours, currentVec,
)
if err != nil {
h.logger.Error("api: radio: load candidates", "err", err)
writeErr(w, http.StatusInternalServerError, "server_error",
"candidate load failed")
return
}
}
weights := recommendation.ScoringWeights{
BaseWeight: h.recCfg.BaseWeight,
LikeBoost: h.recCfg.LikeBoost,
RecencyWeight: h.recCfg.RecencyWeight,
SkipPenalty: h.recCfg.SkipPenalty,
JitterMagnitude: h.recCfg.JitterMagnitude,
ContextWeight: h.recCfg.ContextWeight,
SimilarityWeight: h.recCfg.SimilarityWeight, // NEW
}
```
`parseExcludeParam(s string) []pgtype.UUID` — splits on `,`, parses each
UUID, silently drops malformed entries. Returns nil for empty input.
## 7. Frontend (queue refresh at 80%)
`web/src/lib/player/store.svelte.ts`:
- New `radioSeedId` `$state<string | null>` set inside `playRadio()`.
- New `$effect` watching `(player.currentIndex + 1) / player.queue.length`:
- Fires when `radioSeedId` is set, queue is non-empty, ratio ≥ 0.8, and
no refresh in-flight.
- Calls `/api/radio?seed_track=<radioSeedId>&exclude=<queue.map(t=>t.id).join(',')>`.
- On success: appends `response.tracks.slice(1)` (drops the seed at index 0).
- On failure: logs + clears in-flight flag (next track-advance can retry).
- New `appendToQueue(tracks)` helper — pushes onto `player.queue` without
changing `currentIndex`.
- `radioSeedId = null` when user enqueues from non-radio paths (`playQueue`,
`enqueueTrack`, `enqueueTracks`) so the auto-refresh doesn't fire on
manually-built queues.
`RadioResponse` JSON shape unchanged. `TrackRef[]` array works for both
initial calls and refresh calls.
## 8. Test plan
### 8.1 Backend pure tests
`internal/recommendation/score_test.go` extensions:
- `TestScore_SimilarityScore_PerfectMatch_AddsWeightedTerm` — `1.0 × 2.0 = +2.0` over baseline
- `TestScore_SimilarityScore_HalfMatch` — `0.5 × 2.0 = +1.0`
- `TestScore_SimilarityScore_Zero_NoEffect` — random/serendipity tracks score same as M3 baseline
### 8.2 Backend integration tests
`internal/recommendation/candidates_v2_test.go` (new):
- `TestLoadCandidatesFromSimilarity_LBSimilarSourceContributes`
- `TestLoadCandidatesFromSimilarity_SimilarArtistTracksContribute` (artist score × 0.5 verified)
- `TestLoadCandidatesFromSimilarity_TagOverlapContributes` (jaccard score)
- `TestLoadCandidatesFromSimilarity_LikesOverlapContributes` (0.6 constant)
- `TestLoadCandidatesFromSimilarity_RandomFillToTargetSize` (empty similarity tables → pool ≥ 60)
- `TestLoadCandidatesFromSimilarity_ExcludeListRespected`
- `TestLoadCandidatesFromSimilarity_RecentlyPlayedExcluded`
- `TestLoadCandidatesFromSimilarity_DedupTakesMaxScore` (LB 0.85 beats tag 0.4)
- `TestLoadCandidatesFromSimilarity_SeedAlwaysExcluded`
- `TestLoadCandidatesFromSimilarity_EmptyLibrary_NoError`
### 8.3 Backend HTTP tests
`internal/api/radio_test.go` extensions:
- `TestHandleRadio_WithSimilarityPool_RanksLBSimilarHigher` (deterministic via fixed RNG)
- `TestHandleRadio_ExcludeParam_FiltersOut`
- `TestHandleRadio_ExcludeParam_MalformedSkipped`
- `TestHandleRadio_FallbackToM3OnSimilarityError` (inject fault)
### 8.4 Frontend tests
`web/src/lib/player/store.test.ts` extensions:
- `radio refresh fires at 80% queue consumption` (5-track queue at index 3)
- `radio refresh appends new tracks (excluding seed)` (queue 5 → 9 after 5-track response)
- `radio refresh does NOT double-fire` (in-flight guard)
- `radio refresh resets when user enqueues from non-radio source`
- `radio refresh below threshold` (5-track queue at index 2 → no refresh)
### 8.5 Coverage targets
- `internal/recommendation` post-M4c: ≥ 80% (currently 73%)
- `internal/api/radio.go`: ≥ 75%
- Web `player/store`: ≥ 80% on the new refresh logic
### 8.6 Manual end-to-end gate (closes M4)
After deploy + M4b worker has filled `track_similarity` for ≥10 played tracks:
1. Click radio from a played track — queue should differ noticeably from M3
2. Listen through ~80% of the queue — observe queue length increase as
auto-refresh fires
3. `psql -c "SELECT count(*) FROM track_similarity"` shows non-trivial data
4. Subjectively: radio quality should feel meaningfully better than M3's
baseline. **This is the closing gate for M4.**
## 9. Decisions ledger
| # | Decision | Rationale |
|---|---|---|
| 1 | 4-source pool composition + random fill | Cross-source diversity; sparse-fallback is automatic via random fill |
| 2 | Always augment with random to floor of 60 | Never returns empty radio; serendipity built in; degrades gracefully on sparse libraries |
| 3 | New `SimilarityScore × SimilarityWeight` term in M3 Score() | Wastes the LB scores otherwise; consistent with the M3 multi-input pattern |
| 4 | No lazy LB fetch | Keeps radio handler synchronous; M4b worker + augmentation cover the gap |
| 5 | `?exclude=...` query param for queue refresh | Stateless, simple, no new server state |
| 6 | Per-source K + score: hardcoded for v1 | YAGNI; expose later if telemetry warrants |
| 7 | Fallback to M3 LoadCandidates on similarity-pool error | Defense in depth for the central radio surface |
## 10. Backwards compatibility
- New SQL query alongside existing `LoadRadioCandidates`; M3 callers
unaffected.
- M3 `LoadCandidates` retained for the fallback path and direct test usage.
- `Score()` signature unchanged; new fields are zero-defaulted so existing
zero-value `ScoringInputs{}`/`ScoringWeights{}` constructions produce M3
scores.
- `/api/radio` request shape extended (adds optional `?exclude=`); existing
callers that don't pass it work identically.
- `RadioResponse` shape unchanged.
- No schema migration — relies on M4b's `track_similarity` /
`artist_similarity` and existing M2 `general_likes` / M0-M1 `tracks.genre`.
## 11. M4 closure
After this slice merges, M4 is complete:
- M4a (PR #26) — outbound LB scrobble worker
- M4b (PR #27) — inbound LB similarity ingest
- M4c (this) — radio similarity-driven candidate pool + 80% queue refresh
Unblocks M5 (Lidarr quarantine + suggested-additions for tracks not in
library). Out-of-library LB-returned tracks (currently filtered out by the
in-library JOIN) become the natural input for M5's "would you like to add
this?" workflow.