minstrel/docs/superpowers/specs/2026-04-28-m4c-radio-design.md

# M4c — Radio similarity-driven candidate pool + queue refresh at 80% (closes M4)

**Status:** Spec draft, 2026-04-28
**Tracking:** Fable #347
**Milestone:** M4 — ListenBrainz scrobble + similarity + radio
**Builds on:** M4a (PR #26 — outbound scrobble), M4b (PR #27 — inbound similarity ingest)

## 1. Goal

Replace M3's "whole library minus seed minus recently-played" candidate pool
with a similarity-driven pool drawn from four sources (LB-similar tracks,
tracks by similar artists, MB-tag overlap, user's general likes overlapping
seed tags) plus a random-fill source that guarantees a minimum pool size.
Add a sixth scoring term (`SimilarityScore × SimilarityWeight`) so within-pool
ranking reflects per-track similarity strength. Web client auto-refreshes
the radio queue when 80% consumed.

When this slice merges, the M4 milestone closes: the engine has all three v1
components (scoring, session vectors, LB-derived similarity) wired through
both backend and frontend.

## 2. Non-goals (explicit)

- **Lazy LB fetch on radio click** — radio handler stays synchronous and
  uses only data already in `track_similarity` / `artist_similarity`.
  Sparse-data UX is handled by random-fill augmentation.
- **YAML-configurable per-source Ks** — hardcoded constants for v1.
- **Symmetric edge storage** — still one-way as M4b stored.
- **Per-source score weights as YAML** — tunable in code only.
- **Token-based queue-refresh continuation** — explicit `?exclude=...`
  query param.
- **Pre-fetch similarity on track play** — M4b's hourly tick is the only
  fetch trigger.
- **Suggested additions / Lidarr** — out-of-library LB matches discarded;
  M5 territory.
- **Multi-seed / persistent radio stations** — every call is single-seed
  and stateless.
- **Cross-user collaborative filtering source** — `user_cooccurrence`
  schema slot reserved, not populated.
- **`SimilarityWeight` per-user override** — operator-only YAML for v1.

## 3. Architecture overview

```
GET /api/radio?seed_track=<uuid>&limit=N&exclude=t1,t2,...
  ↓
handleRadio
  1. Auth + parse params (existing M3)
  2. Get seed track + album + artist (existing)
  3. q.GetCurrentSessionVectorForUser() (existing — M3)
  4. recommendation.LoadCandidatesFromSimilarity(...)            ← NEW
       - 5-way SQL UNION: LB-similar / similar-artist tracks /
         tag-overlap / likes-overlap / random-fill
       - Excludes seed + ?exclude= list + recently-played
       - Returns []Candidate with per-row SimilarityScore
       - On error → fallback to M3's LoadCandidates (logged)
  5. recommendation.Shuffle(candidates, weights, ...)            ← extended
       - M3's Score() gains SimilarityScore × SimilarityWeight term
       - Otherwise unchanged
  6. Resolve album/artist for picks (existing)
  7. Return RadioResponse{Tracks: [seed, pick1, ...]}
                                                         ↓
                                                Web client
  8. Player store watches currentIndex / queue.length              ← NEW
       - At ≥ 80% consumed, AND radioSeedId is set, AND no refresh
         in-flight: GET /api/radio?seed_track=…&exclude=<queue>
       - Append response.tracks (skipping index 0 = seed already in queue)
       - radioSeedId cleared when user manually enqueues from elsewhere
```

## 4. Candidate pool SQL (`LoadRadioCandidatesV2`)

5-way UNION; each branch produces `(track_id, similarity_score)`. After UNION,
the outer SELECT joins `tracks` + general_likes (for `is_liked`) + LATERAL
play_events aggregation (for `last_played_at` / `play_count` / `skip_count`),
GROUP BY track id, taking `max(similarity_score)` across sources.

```sql
-- name: LoadRadioCandidatesV2 :many
WITH
seed_artist AS (SELECT artist_id FROM tracks WHERE id = $2),
seed_tags AS (
    SELECT trim(g) AS tag
    FROM tracks t,
         regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g
    WHERE t.id = $2 AND trim(g) <> ''
),
exclude_set AS (
    SELECT unnest($5::uuid[]) AS id
    UNION ALL
    SELECT track_id FROM play_events
    WHERE user_id = $1 AND started_at > now() - $4 * interval '1 hour'
),
lb_similar AS (
    SELECT ts.track_b_id AS id, ts.score AS sim_score
    FROM track_similarity ts
    WHERE ts.track_a_id = $2
      AND ts.source = 'listenbrainz'
      AND ts.track_b_id NOT IN (SELECT id FROM exclude_set)
    ORDER BY ts.score DESC
    LIMIT $6
),
similar_artists AS (
    SELECT t.id, asim.score * 0.5 AS sim_score
    FROM artist_similarity asim
    JOIN tracks t ON t.artist_id = asim.artist_b_id
    JOIN seed_artist sa ON asim.artist_a_id = sa.artist_id
    WHERE asim.source = 'listenbrainz'
      AND t.id NOT IN (SELECT id FROM exclude_set)
    ORDER BY asim.score DESC, random()
    LIMIT $7
),
tag_overlap AS (
    SELECT t.id,
           (count(DISTINCT trim(g_raw))::float8
            / GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score
    FROM tracks t,
         regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_raw
    WHERE trim(g_raw) IN (SELECT tag FROM seed_tags)
      AND t.id NOT IN (SELECT id FROM exclude_set)
      AND t.id <> $2
    GROUP BY t.id
    HAVING count(DISTINCT trim(g_raw)) > 0
    ORDER BY sim_score DESC
    LIMIT $8
),
likes_overlap AS (
    SELECT gl.track_id AS id, 0.6::float8 AS sim_score
    FROM general_likes gl
    JOIN tracks t ON t.id = gl.track_id
    WHERE gl.user_id = $1
      AND t.id NOT IN (SELECT id FROM exclude_set)
      AND EXISTS (
          SELECT 1 FROM regexp_split_to_table(coalesce(t.genre, ''), '[;,]') g
          WHERE trim(g) IN (SELECT tag FROM seed_tags)
      )
    ORDER BY random()
    LIMIT $9
),
random_fill AS (
    SELECT t.id, 0.0::float8 AS sim_score
    FROM tracks t
    WHERE t.id NOT IN (SELECT id FROM exclude_set)
      AND t.id <> $2
      AND t.id NOT IN (
          SELECT id FROM lb_similar
          UNION SELECT id FROM similar_artists
          UNION SELECT id FROM tag_overlap
          UNION SELECT id FROM likes_overlap
      )
    ORDER BY random()
    LIMIT $10
)
SELECT
    sqlc.embed(t),
    (l.user_id IS NOT NULL)::bool AS is_liked,
    pe.last_played_at::timestamptz AS last_played_at,
    pe.play_count,
    pe.skip_count,
    max(u.sim_score) AS similarity_score
FROM (
    SELECT * FROM lb_similar
    UNION ALL SELECT * FROM similar_artists
    UNION ALL SELECT * FROM tag_overlap
    UNION ALL SELECT * FROM likes_overlap
    UNION ALL SELECT * FROM random_fill
) u
JOIN tracks t ON t.id = u.id
LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id
LEFT JOIN LATERAL (
    SELECT max(started_at) AS last_played_at,
           count(*) AS play_count,
           count(*) FILTER (WHERE was_skipped) AS skip_count
    FROM play_events
    WHERE user_id = $1 AND track_id = t.id
) pe ON true
GROUP BY t.id, t.title, t.album_id, t.artist_id, t.duration_ms, t.file_path,
         t.file_format, t.file_size, t.bitrate, t.track_number, t.disc_number,
         t.mbid, t.genre, t.created_at, t.updated_at,
         l.user_id, pe.last_played_at, pe.play_count, pe.skip_count;
```

### 4.1 Per-source K and score (defaults, hardcoded for v1)

| Source | K | sim_score per row |
|---|---|---|
| `lb_similar` (track_similarity) | 30 | LB raw score (0–1) |
| `similar_artists` | 30 | `artist_similarity.score × 0.5` |
| `tag_overlap` | 20 | jaccard: `shared_tags / seed_tag_count` |
| `likes_overlap` | 20 | constant `0.6` |
| `random_fill` | 30 | `0.0` |

Total ideal: 130 candidates pre-dedup; 60–100 after dedup. Random fill is
drawn AFTER the 4 similarity sources are exhausted (`NOT IN (lb_similar
UNION similar_artists UNION tag_overlap UNION likes_overlap)`), so it
strictly augments rather than overlaps. On very small libraries (<130
tracks total), the pool is naturally smaller — there is no hard floor;
the design assumes typical libraries have hundreds of tracks. M3's `Score()`
+ `Shuffle()` happily ranks small pools.

`max(sim_score)` on dedup so a track in multiple sources keeps its strongest
signal (LB's 0.85 beats tag's 0.4).

## 5. Score() formula extension

`internal/recommendation/score.go`:

```go
type ScoringInputs struct {
    IsGeneralLiked       bool
    LastPlayedAt         *time.Time
    PlayCount            int
    SkipCount            int
    ContextualMatchScore float64  // M3
    SimilarityScore      float64  // NEW — max across the 4 similarity sources, in [0,1]
}

type ScoringWeights struct {
    BaseWeight       float64
    LikeBoost        float64
    RecencyWeight    float64
    SkipPenalty      float64
    JitterMagnitude  float64
    ContextWeight    float64  // M3
    SimilarityWeight float64  // NEW — default 2.0
}
```

Updated formula:

```
score = base
      + (is_general_liked ? LikeBoost : 0)
      + recency_decay * RecencyWeight
      - skip_ratio * SkipPenalty
      + contextual_match_score * ContextWeight
      + similarity_score * SimilarityWeight   ← NEW
      + jitter
```

`config.RecommendationConfig` gains `SimilarityWeight float64` (yaml
`similarity_weight`, default `2.0`). Same magnitude as `LikeBoost` and
`ContextWeight` — at perfect similarity (1.0), an LB-similar track gets a
+2.0 boost equivalent to an explicit general like.

**Backwards compatible:** zero-value `ScoringInputs{}` and `ScoringWeights{}`
produce M3 behavior because both new fields are zero-defaulted.

## 6. Go-side wiring

### 6.1 `LoadCandidatesFromSimilarity`

`internal/recommendation/candidates.go` gains a sibling to `LoadCandidates`:

```go
type CandidateSourceLimits struct {
    LBSimilar      int  // 30
    SimilarArtist  int  // 30
    TagOverlap     int  // 20
    LikesOverlap   int  // 20
    RandomFill     int  // 30 — drawn from tracks NOT already returned by the 4 similarity sources
}

func DefaultCandidateSourceLimits() CandidateSourceLimits

// LoadCandidatesFromSimilarity is M4c's primary candidate-pool loader.
// Returns []Candidate (same type as M3 LoadCandidates) so Shuffle() is
// unchanged. Caller falls back to LoadCandidates on error.
func LoadCandidatesFromSimilarity(
    ctx context.Context,
    q *dbq.Queries,
    userID, seedID pgtype.UUID,
    recentlyPlayedHours int,
    currentVector SessionVector,
    exclude []pgtype.UUID,
    limits CandidateSourceLimits,
) ([]Candidate, error)
```

Body:
1. `q.LoadRadioCandidatesV2(...)` with the 10 params from §4
2. Existing `loadContextualLikesByTrack(...)` for the contextual scoring inputs
3. Project rows → `[]Candidate` with `Inputs.SimilarityScore = row.SimilarityScore`

`LoadCandidates` (M3 fallback) **stays in place**, still consumed by callers
that want whole-library scoring (unit tests, the radio handler's error path).

### 6.2 Radio handler change

`internal/api/radio.go`:

```go
exclude := parseExcludeParam(r.URL.Query().Get("exclude"))  // []pgtype.UUID
exclude = append(exclude, seedID)                            // always exclude the seed
limits := recommendation.DefaultCandidateSourceLimits()

candidates, err := recommendation.LoadCandidatesFromSimilarity(
    r.Context(), q, user.ID, seedID,
    h.recCfg.RecentlyPlayedHours, currentVec,
    exclude, limits,
)
if err != nil {
    h.logger.Warn("api: radio: similarity-pool failed, falling back",
        "err", err)
    candidates, err = recommendation.LoadCandidates(
        r.Context(), q, user.ID, seedID,
        h.recCfg.RecentlyPlayedHours, currentVec,
    )
    if err != nil {
        h.logger.Error("api: radio: load candidates", "err", err)
        writeErr(w, http.StatusInternalServerError, "server_error",
            "candidate load failed")
        return
    }
}

weights := recommendation.ScoringWeights{
    BaseWeight:       h.recCfg.BaseWeight,
    LikeBoost:        h.recCfg.LikeBoost,
    RecencyWeight:    h.recCfg.RecencyWeight,
    SkipPenalty:      h.recCfg.SkipPenalty,
    JitterMagnitude:  h.recCfg.JitterMagnitude,
    ContextWeight:    h.recCfg.ContextWeight,
    SimilarityWeight: h.recCfg.SimilarityWeight,  // NEW
}
```

`parseExcludeParam(s string) []pgtype.UUID` — splits on `,`, parses each
UUID, silently drops malformed entries. Returns nil for empty input.

## 7. Frontend (queue refresh at 80%)

`web/src/lib/player/store.svelte.ts`:

- New `radioSeedId` `$state<string | null>` set inside `playRadio()`.
- New `$effect` watching `(player.currentIndex + 1) / player.queue.length`:
  - Fires when `radioSeedId` is set, queue is non-empty, ratio ≥ 0.8, and
    no refresh in-flight.
  - Calls `/api/radio?seed_track=<radioSeedId>&exclude=<queue.map(t=>t.id).join(',')>`.
  - On success: appends `response.tracks.slice(1)` (drops the seed at index 0).
  - On failure: logs + clears in-flight flag (next track-advance can retry).
- New `appendToQueue(tracks)` helper — pushes onto `player.queue` without
  changing `currentIndex`.
- `radioSeedId = null` when user enqueues from non-radio paths (`playQueue`,
  `enqueueTrack`, `enqueueTracks`) so the auto-refresh doesn't fire on
  manually-built queues.

`RadioResponse` JSON shape unchanged. `TrackRef[]` array works for both
initial calls and refresh calls.

## 8. Test plan

### 8.1 Backend pure tests

`internal/recommendation/score_test.go` extensions:
- `TestScore_SimilarityScore_PerfectMatch_AddsWeightedTerm` — `1.0 × 2.0 = +2.0` over baseline
- `TestScore_SimilarityScore_HalfMatch` — `0.5 × 2.0 = +1.0`
- `TestScore_SimilarityScore_Zero_NoEffect` — random/serendipity tracks score same as M3 baseline

### 8.2 Backend integration tests

`internal/recommendation/candidates_v2_test.go` (new):
- `TestLoadCandidatesFromSimilarity_LBSimilarSourceContributes`
- `TestLoadCandidatesFromSimilarity_SimilarArtistTracksContribute` (artist score × 0.5 verified)
- `TestLoadCandidatesFromSimilarity_TagOverlapContributes` (jaccard score)
- `TestLoadCandidatesFromSimilarity_LikesOverlapContributes` (0.6 constant)
- `TestLoadCandidatesFromSimilarity_RandomFillToTargetSize` (empty similarity tables → pool ≥ 60)
- `TestLoadCandidatesFromSimilarity_ExcludeListRespected`
- `TestLoadCandidatesFromSimilarity_RecentlyPlayedExcluded`
- `TestLoadCandidatesFromSimilarity_DedupTakesMaxScore` (LB 0.85 beats tag 0.4)
- `TestLoadCandidatesFromSimilarity_SeedAlwaysExcluded`
- `TestLoadCandidatesFromSimilarity_EmptyLibrary_NoError`

### 8.3 Backend HTTP tests

`internal/api/radio_test.go` extensions:
- `TestHandleRadio_WithSimilarityPool_RanksLBSimilarHigher` (deterministic via fixed RNG)
- `TestHandleRadio_ExcludeParam_FiltersOut`
- `TestHandleRadio_ExcludeParam_MalformedSkipped`
- `TestHandleRadio_FallbackToM3OnSimilarityError` (inject fault)

### 8.4 Frontend tests

`web/src/lib/player/store.test.ts` extensions:
- `radio refresh fires at 80% queue consumption` (5-track queue at index 3)
- `radio refresh appends new tracks (excluding seed)` (queue 5 → 9 after 5-track response)
- `radio refresh does NOT double-fire` (in-flight guard)
- `radio refresh resets when user enqueues from non-radio source`
- `radio refresh below threshold` (5-track queue at index 2 → no refresh)

### 8.5 Coverage targets

- `internal/recommendation` post-M4c: ≥ 80% (currently 73%)
- `internal/api/radio.go`: ≥ 75%
- Web `player/store`: ≥ 80% on the new refresh logic

### 8.6 Manual end-to-end gate (closes M4)

After deploy + M4b worker has filled `track_similarity` for ≥10 played tracks:

1. Click radio from a played track — queue should differ noticeably from M3
2. Listen through ~80% of the queue — observe queue length increase as
   auto-refresh fires
3. `psql -c "SELECT count(*) FROM track_similarity"` shows non-trivial data
4. Subjectively: radio quality should feel meaningfully better than M3's
   baseline. **This is the closing gate for M4.**

## 9. Decisions ledger

| # | Decision | Rationale |
|---|---|---|
| 1 | 4-source pool composition + random fill | Cross-source diversity; sparse-fallback is automatic via random fill |
| 2 | Always augment with random to floor of 60 | Never returns empty radio; serendipity built in; degrades gracefully on sparse libraries |
| 3 | New `SimilarityScore × SimilarityWeight` term in M3 Score() | Wastes the LB scores otherwise; consistent with the M3 multi-input pattern |
| 4 | No lazy LB fetch | Keeps radio handler synchronous; M4b worker + augmentation cover the gap |
| 5 | `?exclude=...` query param for queue refresh | Stateless, simple, no new server state |
| 6 | Per-source K + score: hardcoded for v1 | YAGNI; expose later if telemetry warrants |
| 7 | Fallback to M3 LoadCandidates on similarity-pool error | Defense in depth for the central radio surface |

## 10. Backwards compatibility

- New SQL query alongside existing `LoadRadioCandidates`; M3 callers
  unaffected.
- M3 `LoadCandidates` retained for the fallback path and direct test usage.
- `Score()` signature unchanged; new fields are zero-defaulted so existing
  zero-value `ScoringInputs{}`/`ScoringWeights{}` constructions produce M3
  scores.
- `/api/radio` request shape extended (adds optional `?exclude=`); existing
  callers that don't pass it work identically.
- `RadioResponse` shape unchanged.
- No schema migration — relies on M4b's `track_similarity` /
  `artist_similarity` and existing M2 `general_likes` / M0-M1 `tracks.genre`.

## 11. M4 closure

After this slice merges, M4 is complete:
- M4a (PR #26) — outbound LB scrobble worker
- M4b (PR #27) — inbound LB similarity ingest
- M4c (this) — radio similarity-driven candidate pool + 80% queue refresh

Unblocks M5 (Lidarr quarantine + suggested-additions for tracks not in
library). Out-of-library LB-returned tracks (currently filtered out by the
in-library JOIN) become the natural input for M5's "would you like to add
this?" workflow.