minstrel/docs/superpowers/specs/2026-04-28-m4c-radio-design.md

M4c — Radio similarity-driven candidate pool + queue refresh at 80% (closes M4)

Status: Spec draft, 2026-04-28 Tracking: Fable #347 Milestone: M4 — ListenBrainz scrobble + similarity + radio Builds on: M4a (PR #26 — outbound scrobble), M4b (PR #27 — inbound similarity ingest)

1. Goal

Replace M3's "whole library minus seed minus recently-played" candidate pool with a similarity-driven pool drawn from four sources (LB-similar tracks, tracks by similar artists, MB-tag overlap, user's general likes overlapping seed tags) plus a random-fill source that guarantees a minimum pool size. Add a sixth scoring term (SimilarityScore × SimilarityWeight) so within-pool ranking reflects per-track similarity strength. Web client auto-refreshes the radio queue when 80% consumed.

When this slice merges, the M4 milestone closes: the engine has all three v1 components (scoring, session vectors, LB-derived similarity) wired through both backend and frontend.

2. Non-goals (explicit)

• Lazy LB fetch on radio click — radio handler stays synchronous and uses only data already in track_similarity / artist_similarity. Sparse-data UX is handled by random-fill augmentation.
• YAML-configurable per-source Ks — hardcoded constants for v1.
• Symmetric edge storage — still one-way as M4b stored.
• Per-source score weights as YAML — tunable in code only.
• Token-based queue-refresh continuation — explicit ?exclude=... query param.
• Pre-fetch similarity on track play — M4b's hourly tick is the only fetch trigger.
• Suggested additions / Lidarr — out-of-library LB matches discarded; M5 territory.
• Multi-seed / persistent radio stations — every call is single-seed and stateless.
• Cross-user collaborative filtering source — user_cooccurrence schema slot reserved, not populated.
• SimilarityWeight per-user override — operator-only YAML for v1.

3. Architecture overview

GET /api/radio?seed_track=<uuid>&limit=N&exclude=t1,t2,...
  ↓
handleRadio
  1. Auth + parse params (existing M3)
  2. Get seed track + album + artist (existing)
  3. q.GetCurrentSessionVectorForUser() (existing — M3)
  4. recommendation.LoadCandidatesFromSimilarity(...)            ← NEW
       - 5-way SQL UNION: LB-similar / similar-artist tracks /
         tag-overlap / likes-overlap / random-fill
       - Excludes seed + ?exclude= list + recently-played
       - Returns []Candidate with per-row SimilarityScore
       - On error → fallback to M3's LoadCandidates (logged)
  5. recommendation.Shuffle(candidates, weights, ...)            ← extended
       - M3's Score() gains SimilarityScore × SimilarityWeight term
       - Otherwise unchanged
  6. Resolve album/artist for picks (existing)
  7. Return RadioResponse{Tracks: [seed, pick1, ...]}
                                                         ↓
                                                Web client
  8. Player store watches currentIndex / queue.length              ← NEW
       - At ≥ 80% consumed, AND radioSeedId is set, AND no refresh
         in-flight: GET /api/radio?seed_track=…&exclude=<queue>
       - Append response.tracks (skipping index 0 = seed already in queue)
       - radioSeedId cleared when user manually enqueues from elsewhere

4. Candidate pool SQL (LoadRadioCandidatesV2)

5-way UNION; each branch produces (track_id, similarity_score). After UNION, the outer SELECT joins tracks + general_likes (for is_liked) + LATERAL play_events aggregation (for last_played_at / play_count / skip_count), GROUP BY track id, taking max(similarity_score) across sources.

-- name: LoadRadioCandidatesV2 :many
WITH
seed_artist AS (SELECT artist_id FROM tracks WHERE id = $2),
seed_tags AS (
    SELECT trim(g) AS tag
    FROM tracks t,
         regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g
    WHERE t.id = $2 AND trim(g) <> ''
),
exclude_set AS (
    SELECT unnest($5::uuid[]) AS id
    UNION ALL
    SELECT track_id FROM play_events
    WHERE user_id = $1 AND started_at > now() - $4 * interval '1 hour'
),
lb_similar AS (
    SELECT ts.track_b_id AS id, ts.score AS sim_score
    FROM track_similarity ts
    WHERE ts.track_a_id = $2
      AND ts.source = 'listenbrainz'
      AND ts.track_b_id NOT IN (SELECT id FROM exclude_set)
    ORDER BY ts.score DESC
    LIMIT $6
),
similar_artists AS (
    SELECT t.id, asim.score * 0.5 AS sim_score
    FROM artist_similarity asim
    JOIN tracks t ON t.artist_id = asim.artist_b_id
    JOIN seed_artist sa ON asim.artist_a_id = sa.artist_id
    WHERE asim.source = 'listenbrainz'
      AND t.id NOT IN (SELECT id FROM exclude_set)
    ORDER BY asim.score DESC, random()
    LIMIT $7
),
tag_overlap AS (
    SELECT t.id,
           (count(DISTINCT trim(g_raw))::float8
            / GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score
    FROM tracks t,
         regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_raw
    WHERE trim(g_raw) IN (SELECT tag FROM seed_tags)
      AND t.id NOT IN (SELECT id FROM exclude_set)
      AND t.id <> $2
    GROUP BY t.id
    HAVING count(DISTINCT trim(g_raw)) > 0
    ORDER BY sim_score DESC
    LIMIT $8
),
likes_overlap AS (
    SELECT gl.track_id AS id, 0.6::float8 AS sim_score
    FROM general_likes gl
    JOIN tracks t ON t.id = gl.track_id
    WHERE gl.user_id = $1
      AND t.id NOT IN (SELECT id FROM exclude_set)
      AND EXISTS (
          SELECT 1 FROM regexp_split_to_table(coalesce(t.genre, ''), '[;,]') g
          WHERE trim(g) IN (SELECT tag FROM seed_tags)
      )
    ORDER BY random()
    LIMIT $9
),
random_fill AS (
    SELECT t.id, 0.0::float8 AS sim_score
    FROM tracks t
    WHERE t.id NOT IN (SELECT id FROM exclude_set)
      AND t.id <> $2
      AND t.id NOT IN (
          SELECT id FROM lb_similar
          UNION SELECT id FROM similar_artists
          UNION SELECT id FROM tag_overlap
          UNION SELECT id FROM likes_overlap
      )
    ORDER BY random()
    LIMIT $10
)
SELECT
    sqlc.embed(t),
    (l.user_id IS NOT NULL)::bool AS is_liked,
    pe.last_played_at::timestamptz AS last_played_at,
    pe.play_count,
    pe.skip_count,
    max(u.sim_score) AS similarity_score
FROM (
    SELECT * FROM lb_similar
    UNION ALL SELECT * FROM similar_artists
    UNION ALL SELECT * FROM tag_overlap
    UNION ALL SELECT * FROM likes_overlap
    UNION ALL SELECT * FROM random_fill
) u
JOIN tracks t ON t.id = u.id
LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id
LEFT JOIN LATERAL (
    SELECT max(started_at) AS last_played_at,
           count(*) AS play_count,
           count(*) FILTER (WHERE was_skipped) AS skip_count
    FROM play_events
    WHERE user_id = $1 AND track_id = t.id
) pe ON true
GROUP BY t.id, t.title, t.album_id, t.artist_id, t.duration_ms, t.file_path,
         t.file_format, t.file_size, t.bitrate, t.track_number, t.disc_number,
         t.mbid, t.genre, t.created_at, t.updated_at,
         l.user_id, pe.last_played_at, pe.play_count, pe.skip_count;

4.1 Per-source K and score (defaults, hardcoded for v1)

Source	K	sim_score per row
lb_similar (track_similarity)	30	LB raw score (0–1)
similar_artists	30	artist_similarity.score × 0.5
tag_overlap	20	jaccard: shared_tags / seed_tag_count
likes_overlap	20	constant 0.6
random_fill	30	0.0

Total ideal: 130 candidates pre-dedup; 60–100 after dedup. Random fill is drawn AFTER the 4 similarity sources are exhausted (NOT IN (lb_similar UNION similar_artists UNION tag_overlap UNION likes_overlap)), so it strictly augments rather than overlaps. On very small libraries (<130 tracks total), the pool is naturally smaller — there is no hard floor; the design assumes typical libraries have hundreds of tracks. M3's Score()

• Shuffle() happily ranks small pools.

max(sim_score) on dedup so a track in multiple sources keeps its strongest signal (LB's 0.85 beats tag's 0.4).

5. Score() formula extension

internal/recommendation/score.go:

type ScoringInputs struct {
    IsGeneralLiked       bool
    LastPlayedAt         *time.Time
    PlayCount            int
    SkipCount            int
    ContextualMatchScore float64  // M3
    SimilarityScore      float64  // NEW — max across the 4 similarity sources, in [0,1]
}

type ScoringWeights struct {
    BaseWeight       float64
    LikeBoost        float64
    RecencyWeight    float64
    SkipPenalty      float64
    JitterMagnitude  float64
    ContextWeight    float64  // M3
    SimilarityWeight float64  // NEW — default 2.0
}

Updated formula:

score = base
      + (is_general_liked ? LikeBoost : 0)
      + recency_decay * RecencyWeight
      - skip_ratio * SkipPenalty
      + contextual_match_score * ContextWeight
      + similarity_score * SimilarityWeight   ← NEW
      + jitter

config.RecommendationConfig gains SimilarityWeight float64 (yaml similarity_weight, default 2.0). Same magnitude as LikeBoost and ContextWeight — at perfect similarity (1.0), an LB-similar track gets a +2.0 boost equivalent to an explicit general like.

Backwards compatible: zero-value ScoringInputs{} and ScoringWeights{} produce M3 behavior because both new fields are zero-defaulted.

6. Go-side wiring

6.1 LoadCandidatesFromSimilarity

internal/recommendation/candidates.go gains a sibling to LoadCandidates:

type CandidateSourceLimits struct {
    LBSimilar      int  // 30
    SimilarArtist  int  // 30
    TagOverlap     int  // 20
    LikesOverlap   int  // 20
    RandomFill     int  // 30 — drawn from tracks NOT already returned by the 4 similarity sources
}

func DefaultCandidateSourceLimits() CandidateSourceLimits

// LoadCandidatesFromSimilarity is M4c's primary candidate-pool loader.
// Returns []Candidate (same type as M3 LoadCandidates) so Shuffle() is
// unchanged. Caller falls back to LoadCandidates on error.
func LoadCandidatesFromSimilarity(
    ctx context.Context,
    q *dbq.Queries,
    userID, seedID pgtype.UUID,
    recentlyPlayedHours int,
    currentVector SessionVector,
    exclude []pgtype.UUID,
    limits CandidateSourceLimits,
) ([]Candidate, error)

Body:

1. q.LoadRadioCandidatesV2(...) with the 10 params from §4
2. Existing loadContextualLikesByTrack(...) for the contextual scoring inputs
3. Project rows → []Candidate with Inputs.SimilarityScore = row.SimilarityScore

LoadCandidates (M3 fallback) stays in place, still consumed by callers that want whole-library scoring (unit tests, the radio handler's error path).

6.2 Radio handler change

internal/api/radio.go:

exclude := parseExcludeParam(r.URL.Query().Get("exclude"))  // []pgtype.UUID
exclude = append(exclude, seedID)                            // always exclude the seed
limits := recommendation.DefaultCandidateSourceLimits()

candidates, err := recommendation.LoadCandidatesFromSimilarity(
    r.Context(), q, user.ID, seedID,
    h.recCfg.RecentlyPlayedHours, currentVec,
    exclude, limits,
)
if err != nil {
    h.logger.Warn("api: radio: similarity-pool failed, falling back",
        "err", err)
    candidates, err = recommendation.LoadCandidates(
        r.Context(), q, user.ID, seedID,
        h.recCfg.RecentlyPlayedHours, currentVec,
    )
    if err != nil {
        h.logger.Error("api: radio: load candidates", "err", err)
        writeErr(w, http.StatusInternalServerError, "server_error",
            "candidate load failed")
        return
    }
}

weights := recommendation.ScoringWeights{
    BaseWeight:       h.recCfg.BaseWeight,
    LikeBoost:        h.recCfg.LikeBoost,
    RecencyWeight:    h.recCfg.RecencyWeight,
    SkipPenalty:      h.recCfg.SkipPenalty,
    JitterMagnitude:  h.recCfg.JitterMagnitude,
    ContextWeight:    h.recCfg.ContextWeight,
    SimilarityWeight: h.recCfg.SimilarityWeight,  // NEW
}

parseExcludeParam(s string) []pgtype.UUID — splits on ,, parses each UUID, silently drops malformed entries. Returns nil for empty input.

7. Frontend (queue refresh at 80%)

web/src/lib/player/store.svelte.ts:

• New radioSeedId $state<string | null> set inside playRadio().
• New $effect watching (player.currentIndex + 1) / player.queue.length:
  • Fires when radioSeedId is set, queue is non-empty, ratio ≥ 0.8, and no refresh in-flight.
  • Calls /api/radio?seed_track=<radioSeedId>&exclude=<queue.map(t=>t.id).join(',')>.
  • On success: appends response.tracks.slice(1) (drops the seed at index 0).
  • On failure: logs + clears in-flight flag (next track-advance can retry).
• New appendToQueue(tracks) helper — pushes onto player.queue without changing currentIndex.
• radioSeedId = null when user enqueues from non-radio paths (playQueue, enqueueTrack, enqueueTracks) so the auto-refresh doesn't fire on manually-built queues.

RadioResponse JSON shape unchanged. TrackRef[] array works for both initial calls and refresh calls.

8. Test plan

8.1 Backend pure tests

internal/recommendation/score_test.go extensions:

• TestScore_SimilarityScore_PerfectMatch_AddsWeightedTerm — 1.0 × 2.0 = +2.0 over baseline
• TestScore_SimilarityScore_HalfMatch — 0.5 × 2.0 = +1.0
• TestScore_SimilarityScore_Zero_NoEffect — random/serendipity tracks score same as M3 baseline

8.2 Backend integration tests

internal/recommendation/candidates_v2_test.go (new):

• TestLoadCandidatesFromSimilarity_LBSimilarSourceContributes
• TestLoadCandidatesFromSimilarity_SimilarArtistTracksContribute (artist score × 0.5 verified)
• TestLoadCandidatesFromSimilarity_TagOverlapContributes (jaccard score)
• TestLoadCandidatesFromSimilarity_LikesOverlapContributes (0.6 constant)
• TestLoadCandidatesFromSimilarity_RandomFillToTargetSize (empty similarity tables → pool ≥ 60)
• TestLoadCandidatesFromSimilarity_ExcludeListRespected
• TestLoadCandidatesFromSimilarity_RecentlyPlayedExcluded
• TestLoadCandidatesFromSimilarity_DedupTakesMaxScore (LB 0.85 beats tag 0.4)
• TestLoadCandidatesFromSimilarity_SeedAlwaysExcluded
• TestLoadCandidatesFromSimilarity_EmptyLibrary_NoError

8.3 Backend HTTP tests

internal/api/radio_test.go extensions:

• TestHandleRadio_WithSimilarityPool_RanksLBSimilarHigher (deterministic via fixed RNG)
• TestHandleRadio_ExcludeParam_FiltersOut
• TestHandleRadio_ExcludeParam_MalformedSkipped
• TestHandleRadio_FallbackToM3OnSimilarityError (inject fault)

8.4 Frontend tests

web/src/lib/player/store.test.ts extensions:

• radio refresh fires at 80% queue consumption (5-track queue at index 3)
• radio refresh appends new tracks (excluding seed) (queue 5 → 9 after 5-track response)
• radio refresh does NOT double-fire (in-flight guard)
• radio refresh resets when user enqueues from non-radio source
• radio refresh below threshold (5-track queue at index 2 → no refresh)

8.5 Coverage targets

• internal/recommendation post-M4c: ≥ 80% (currently 73%)
• internal/api/radio.go: ≥ 75%
• Web player/store: ≥ 80% on the new refresh logic

8.6 Manual end-to-end gate (closes M4)

After deploy + M4b worker has filled track_similarity for ≥10 played tracks:

1. Click radio from a played track — queue should differ noticeably from M3
2. Listen through ~80% of the queue — observe queue length increase as auto-refresh fires
3. psql -c "SELECT count(*) FROM track_similarity" shows non-trivial data
4. Subjectively: radio quality should feel meaningfully better than M3's baseline. This is the closing gate for M4.

9. Decisions ledger

#	Decision	Rationale
1	4-source pool composition + random fill	Cross-source diversity; sparse-fallback is automatic via random fill
2	Always augment with random to floor of 60	Never returns empty radio; serendipity built in; degrades gracefully on sparse libraries
3	New SimilarityScore × SimilarityWeight term in M3 Score()	Wastes the LB scores otherwise; consistent with the M3 multi-input pattern
4	No lazy LB fetch	Keeps radio handler synchronous; M4b worker + augmentation cover the gap
5	?exclude=... query param for queue refresh	Stateless, simple, no new server state
6	Per-source K + score: hardcoded for v1	YAGNI; expose later if telemetry warrants
7	Fallback to M3 LoadCandidates on similarity-pool error	Defense in depth for the central radio surface

10. Backwards compatibility

• New SQL query alongside existing LoadRadioCandidates; M3 callers unaffected.
• M3 LoadCandidates retained for the fallback path and direct test usage.
• Score() signature unchanged; new fields are zero-defaulted so existing zero-value ScoringInputs{}/ScoringWeights{} constructions produce M3 scores.
• /api/radio request shape extended (adds optional ?exclude=); existing callers that don't pass it work identically.
• RadioResponse shape unchanged.
• No schema migration — relies on M4b's track_similarity / artist_similarity and existing M2 general_likes / M0-M1 tracks.genre.

11. M4 closure

After this slice merges, M4 is complete:

• M4a (PR #26) — outbound LB scrobble worker
• M4b (PR #27) — inbound LB similarity ingest
• M4c (this) — radio similarity-driven candidate pool + 80% queue refresh

Unblocks M5 (Lidarr quarantine + suggested-additions for tracks not in library). Out-of-library LB-returned tracks (currently filtered out by the in-library JOIN) become the natural input for M5's "would you like to add this?" workflow.