minstrel/docs/superpowers/specs/2026-04-27-m3-shuffle-design.md

# M3 Weighted Shuffle v1 — Design Spec

**Status:** approved 2026-04-27
**Slice:** M3 sub-plan #1 of 3 (recommendation engine v1 + contextual likes). Spec §13 step 7. Step 8 (session vectors + contextual_match_score) is the next sub-plan; this slice ships the scoring foundation without the contextual term.
**Fable task:** #340.

## Goal

Replace the M2 stub `/api/radio` (currently returns just the seed track) with a real weighted-shuffle implementation that scores user library tracks against a per-track-stats formula and returns a top-N queue ordered by score. After this slice, clicking a track in the SPA's search/track-row "play radio" surface produces a meaningful 50-track queue instead of a one-track stub.

## Non-goals

- Session vectors / `contextual_match_score` (sub-plan #2 / Fable #341 covers the write path; sub-plan #3 / Fable #342 folds the score into this function).
- Real "radio" candidate pools from ListenBrainz similarity / similar artists / similar tags (M4).
- Album-seed or artist-seed radio (`?seed_album=`, `?seed_artist=`). v1 is track-seeded only; the Fable task covers that scope explicitly.
- Periodic radio refresh (the spec's "regenerate at 80% consumed"). Player-side concern; lands later.
- Persistent radio queues. Each `/api/radio` call returns a fresh selection — stateless on the server.
- Performance optimization for very large libraries (>50k tracks). At v1 scale the wide-pool scan is fine; M3.5 / M4 can add sampling if telemetry shows it matters.
- `Subsonic` star-radio compatibility. Subsonic's `getRandomSongs` is a different shape; we don't try to map weighted shuffle into it for v1.

## Architecture

A new `internal/recommendation` package owns:

- A pure scoring function `Score(inputs, weights, now, rng) → float64` with no DB dependency. The `rng` parameter is injectable so tests pin jitter to deterministic values.
- A candidate loader `LoadCandidates(ctx, q, userID, seedID, recentlyPlayedHours) → []Candidate` that runs ONE SQL query joining `tracks`, `general_likes`, and aggregated `play_events` to produce the per-track stats the scoring function needs. Excludes the seed and any track played within the recently-played window.
- A `Shuffle(candidates, weights, now, rng, limit)` orchestrator that scores each candidate, sorts descending, truncates to `limit`. Pure.

The `/api/radio` handler becomes a thin shim: validate the seed → call `LoadCandidates` → call `Shuffle` → prepend the seed to the result → project to `[]TrackRef` → return JSON.

**Why split into three layers:**
- `score.go` is the math; trivially unit-testable, no infra.
- `candidates.go` is the data access; integration-tested against a live DB.
- `shuffle.go` is the composition; pure-function tests against fake candidate sets.
- The HTTP handler is the I/O glue; lives in `api/radio.go` like the existing pattern.

This separation also sets up sub-plan #3 (contextual scoring) cleanly — we'll add `LoadContextualSimilarity` next to `LoadCandidates`, extend `ScoringInputs` with `ContextualMatchScore`, and the rest of the chain doesn't change.

## Schema

No migration. All inputs are computed on demand from existing tables:

- `general_likes` for `is_general_liked`.
- `play_events` for `last_played_at`, `play_count`, `skip_count`. Aggregations done in the candidate-loader query.
- `tracks` provides the candidate set itself.

The recently-played-in-the-last-hour filter is applied at candidate-load time (a single `WHERE NOT EXISTS (... AND started_at > now() - interval '1 hour')` clause). Hard suppression — these tracks aren't even scored.

## Scoring math

```go
type ScoringInputs struct {
    IsGeneralLiked bool
    LastPlayedAt   *time.Time  // nil = never played
    PlayCount      int           // total play_events
    SkipCount      int           // play_events with was_skipped=true
}

type ScoringWeights struct {
    BaseWeight      float64  // default 1.0
    LikeBoost       float64  // default 2.0
    RecencyWeight   float64  // default 1.0
    SkipPenalty     float64  // default 1.0
    JitterMagnitude float64  // default 0.1
}

func Score(in ScoringInputs, w ScoringWeights, now time.Time, rng func() float64) float64 {
    s := w.BaseWeight
    if in.IsGeneralLiked {
        s += w.LikeBoost
    }
    s += recencyDecay(in.LastPlayedAt, now) * w.RecencyWeight
    s -= skipRatio(in.PlayCount, in.SkipCount) * w.SkipPenalty
    s += (rng()*2 - 1) * w.JitterMagnitude
    return s
}
```

**`recencyDecay(lastPlayed *time.Time, now time.Time) float64`** — returns `[0, 1]`:
- Never played (`lastPlayed == nil`) → `1.0`. Cold-start tracks compete favorably with stale ones; otherwise the recommendation engine never surfaces music the user hasn't tried.
- Played within last hour: doesn't reach this function (filtered earlier at the candidate-pool stage).
- Otherwise: `min(age_days / 30.0, 1.0)`. Linear ramp; tracks ≥ 30 days stale hit max recency boost.

**`skipRatio(plays, skips int) float64`** — returns `[0, 1]`:
- `plays == 0` → `0.0`. Don't penalize never-played tracks.
- Otherwise → `float64(skips) / float64(plays)`.

**Random jitter** — `(rng()*2 - 1) * JitterMagnitude` produces values in `[-magnitude, +magnitude]`. The `rng` parameter is `func() float64` (matches `math/rand.Float64`); tests inject a fixed-value version for deterministic ordering.

**Score range under defaults:**
- Min (unliked, recent, all-skips): `1.0 + 0 + 0 - 1.0 - 0.1 = -0.1`
- Max (liked, ≥30d stale, never skipped): `1.0 + 2.0 + 1.0 - 0 + 0.1 = 4.1`
- Liked-track structural advantage (`LikeBoost = 2.0`) dominates the jitter band (`±0.1`), so liked tracks always rank above identical unliked tracks.

Configurable via YAML / env. Operators can crank `LikeBoost` up if their library is dominated by stuff they don't actually like, or `JitterMagnitude` up if they want more variety.

## API contracts

**Request:**

```
GET /api/radio?seed_track=<uuid>&limit=<int>
```

- `seed_track` (required) — UUID of the track that seeds the radio.
- `limit` (optional, default 50, max 200) — total number of tracks to return *including* the seed.

**Response (shape unchanged from M2 stub, only contents grew):**

```ts
type RadioResponse = { tracks: TrackRef[] };
```

- `tracks[0]` is always the seed track (so `playQueue(resp.tracks, 0)` plays the seed).
- `tracks[1..]` are the top-scored candidates from the user's library, descending. Length up to `limit - 1`.
- Cold-start (empty library beyond the seed): returns `{ tracks: [<seed>] }`.

**Errors (unchanged from M2 stub):**
- `400 bad_request` — missing `seed_track`, malformed UUID, or `limit < 1`.
- `404 not_found` — `seed_track` doesn't exist.
- `500 server_error` — DB issue.

**No web client changes.** The existing `playRadio(seedTrackId)` already calls this endpoint and feeds `resp.tracks` into `playQueue(tracks, 0)`. After this slice the queue is fuller; the call shape is identical.

**No Subsonic changes.** Track-seeded radio isn't part of Subsonic's surface in our v1 scope.

## Components & files

### New server files

| Path | Responsibility |
|---|---|
| `internal/recommendation/score.go` | Pure scoring function + `recencyDecay` + `skipRatio` helpers. No DB. |
| `internal/recommendation/score_test.go` | Boundary cases: every term, cold-start, jitter determinism, score ranges. |
| `internal/recommendation/candidates.go` | `LoadCandidates(...)` — single SQL query returning `[]Candidate` (`Track + ScoringInputs`). |
| `internal/recommendation/candidates_test.go` | Live-DB tests: seed exclusion, recently-played exclusion, stat-join correctness, cross-user isolation. |
| `internal/recommendation/shuffle.go` | `Shuffle(candidates, weights, now, rng, limit) []Candidate` — composes Score + sort + truncate. Pure. |
| `internal/recommendation/shuffle_test.go` | Pure-function tests: liked-rank-higher, high-skip-rejected, jitter-doesn't-reorder-structural-winners, limit. |
| `internal/db/queries/recommendation.sql` | sqlc query: `LoadRadioCandidates` — SELECT tracks WITH LEFT JOIN general_likes, LEFT JOIN aggregated play_events stats. WHERE clause excludes seed_id and last-hour plays. |
| `internal/db/dbq/recommendation.sql.go` | Generated bindings. |

### Modified server files

| Path | Change |
|---|---|
| `internal/api/radio.go` | Replace stub. New flow: validate `seed_track` and `limit`, call `recommendation.LoadCandidates`, call `recommendation.Shuffle`, prepend seed, project to `[]TrackRef`, return JSON. |
| `internal/api/radio_test.go` | Replace stub tests with: cold-start, typical (seed + scored picks), 404 on unknown seed, 400 on bad seed/limit, cross-user isolation. |
| `internal/config/config.go` + `config.example.yaml` | Add `RecommendationConfig` struct: `BaseWeight` (1.0), `LikeBoost` (2.0), `RecencyWeight` (1.0), `SkipPenalty` (1.0), `JitterMagnitude` (0.1), `RecentlyPlayedHours` (1), `RadioSize` (50), `RadioSizeMax` (200). YAML key `recommendation:`. |
| `internal/api/api.go` | `handlers` struct gains `recCfg config.RecommendationConfig`. `Mount` signature gains the config arg; constructs the handler with it; the radio handler builds `ScoringWeights` from it per request. |
| `internal/server/server.go` + `cmd/minstrel/main.go` | Pass `cfg.Recommendation` through `Mount`. |

### No web changes

The existing `playRadio(seedTrackId)` already consumes `RadioResponse`. The web slice for this PR is empty — server-only.

## Data flow

1. SPA calls `playRadio(seedTrackId)` (existing player-store action). It POSTs `GET /api/radio?seed_track=<id>` (no explicit `limit`, server defaults to 50).
2. `handleRadio` validates auth + seed UUID. Looks up the track to confirm it exists (404 otherwise).
3. Reads `limit` from query (default `cfg.Recommendation.RadioSize`, clamped to `cfg.Recommendation.RadioSizeMax`).
4. Calls `recommendation.LoadCandidates(ctx, q, userID, seedID, cfg.Recommendation.RecentlyPlayedHours)`. The SQL query joins:
   - `tracks t` — base set
   - `LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id` → `is_liked`
   - `LEFT JOIN LATERAL (SELECT max(started_at) FROM play_events WHERE user_id = $1 AND track_id = t.id) pe_last → last_played_at`
   - `LEFT JOIN LATERAL (SELECT count(*), count(*) FILTER (WHERE was_skipped) FROM play_events WHERE user_id = $1 AND track_id = t.id) pe_stats → play_count, skip_count`
   - `WHERE t.id <> seed_id AND NOT EXISTS (SELECT 1 FROM play_events WHERE user_id = $1 AND track_id = t.id AND started_at > now() - interval '$2 hours')`
5. Iterates the candidates, calls `Score(...)` per track, sorts descending, truncates to `limit - 1`.
6. Prepends the seed track, projects each `Candidate.Track` to `TrackRef` (existing `trackRefFrom` helper), writes `RadioResponse{ Tracks: ... }`.
7. SPA receives 50 tracks, plays from index 0 (the seed).

**Stateless** — every call recomputes from scratch. No persisted radio queue, no cursor tracking, no "you already heard this" memory beyond the recently-played-hours window.

## Testing

### Server (`go test`)

**`internal/recommendation/score_test.go`** (pure unit tests):
- Base case (never played, not liked, no skip data) with deterministic RNG `() => 0.5` → score = `BaseWeight + RecencyWeight + 0` exactly.
- Liked boost: same inputs but `IsGeneralLiked=true` → score increases by `LikeBoost`.
- Recency ramp: `lastPlayedAt = now - 15d` → `recencyDecay = 0.5`. `lastPlayedAt = now - 60d` → `1.0` (capped).
- Skip ratio: `PlayCount=4, SkipCount=2` → ratio `0.5`, score loses `0.5 * SkipPenalty`.
- Cold-start skip: `PlayCount=0, SkipCount=0` → ratio `0.0`, no penalty.
- Jitter bounds: 1000 `Score(...)` calls with `math/rand.Float64`, every result within `[mid - JitterMagnitude, mid + JitterMagnitude]` where `mid` is the deterministic-RNG score.
- Determinism: same `(inputs, weights, now, rng)` returns the same score (regression guard for hidden global state).

**`internal/recommendation/shuffle_test.go`** (pure unit tests):
- Liked-vs-not: two otherwise-identical candidates → liked one ranks higher.
- High-skip rejected: candidate with `skipRatio=1.0` ranks last among otherwise-identical candidates.
- Limit truncates: 100 candidates, `limit=10` → result has 10.
- Jitter doesn't reorder structural winners: 100 random RNG seeds → liked track ALWAYS ranks above an equivalent unliked track.
- Empty input → empty output.

**`internal/recommendation/candidates_test.go`** (live DB):
- Seed exclusion: 5 tracks in library, ask for radio with one as seed → returns the other 4.
- Recently-played exclusion: seed a `play_events` row with `started_at = now - 30 minutes` for track A → radio excludes A.
- Stat join: track with one play + one skip → `play_count=1, skip_count=1`. Liked track → `is_liked=true`. Never-played → `last_played_at IS NULL`.
- Cross-user: Alice's plays / likes do NOT appear in Bob's stats.

**`internal/api/radio_test.go`** (HTTP integration, replacing existing stub tests):
- Cold-start: seed only in library → response is `{ tracks: [seed] }`.
- Typical: seed + 5 other tracks → response is `[seed, 5 ranked]`. Seed always at index 0.
- 404 on unknown seed.
- 400 on missing/malformed seed.
- 400 on `limit=0` or `limit < 0`.
- `limit > RadioSizeMax` clamped (returns at most `RadioSizeMax`, no error).
- Cross-user isolation: Alice has many plays + likes → Bob's radio uses Bob's clean stats.

**Coverage:** `go test -coverprofile=cover.out ./internal/recommendation/...` → ≥ 70% per the M3 milestone description. The pure-function tests should hit ~100% on `score.go` + `shuffle.go`; `candidates.go` reaches similar coverage via the integration tests.

### End-to-end manual

Final task in the implementation plan:

1. Sign in. Play 3 tracks all the way through (build `play_count` history).
2. Skip a 4th track within 10 seconds (creates a high `skip_ratio`).
3. Like 2 tracks via the heart button.
4. Click radio on a 5th track.
5. Inspect the response in dev tools network tab — 50 tracks, seed first.
6. The 2 liked tracks appear early in the list (within first ~20).
7. The just-skipped track appears late or absent.
8. The 3 just-played tracks are absent (recently-played exclusion).
9. Trigger radio again with a different seed; ordering varies (jitter), liked tracks still rank prominently.

## Risks & mitigations

- **Wide-pool scan performance.** For a 50k-track library: 50k rows × scoring overhead × sort ≈ a few hundred ms in Go. Below the user-perceptible threshold for a click-to-play. If telemetry later shows >1s P95 latency, M3.5 / M4 can add sampling (random-N candidate pre-filter) or a partial index. The candidate-pool function is the swap point; the scoring function and HTTP handler don't change.
- **Recently-played window edge effects.** If the user plays 50 tracks in an hour, all of them get excluded from the next radio call. Library < 100 tracks could leave the candidate pool too thin to fill `RadioSize`. Mitigation: handler returns whatever it can fit (length might be < `RadioSize`); SPA's `playQueue` works on any non-empty array. Document as acceptable degradation; users with tiny libraries get short radios.
- **Skip-ratio gaming.** A user who skips a track once (during initial library discovery) gets a `skipRatio=1.0` for that track. With `SkipPenalty=1.0` that's a `-1.0` hit — could permanently demote the track even if they like it. Mitigation: weights are tunable; user can lower `SkipPenalty`. Future: smoothing (e.g. `(skips + α) / (plays + β)`) instead of raw ratio. Out of scope for v1.
- **Cold-start RNG dominance.** A brand-new user with zero plays / likes gets every track scored at `BaseWeight + RecencyWeight + jitter`. Top-N is essentially random. That's fine — it matches user expectation ("I haven't told you anything about me, give me random music"). The recency-decay max-for-never-played avoids amplifying new tracks just because they're new.
- **Score formula evolution.** Adding `contextual_match_score` in sub-plan #3 means changing `ScoringInputs` and the `Score` function signature. Mitigation: the function is internal-package only; sub-plan #3 changes both ends of the call atomically. No external consumers.