docs(spec): add M3 weighted shuffle v1 design
Replaces the M2 stub /api/radio with real weighted-shuffle scoring: LikeBoost + recency_decay - skip_ratio penalty + jitter, hard suppression of last-hour plays. Wide candidate pool (whole library) for v1; M4 adds similarity-based pool refinement. Pure scoring function in internal/recommendation; live-DB candidate loader; HTTP handler is a thin shim. No web changes — existing playRadio action already calls /api/radio. Sub-plan #1 of 3 in M3 (Fable #340). Session vectors and contextual match score land in the next two sub-plans.
This commit is contained in:
@@ -0,0 +1,230 @@
|
||||
# M3 Weighted Shuffle v1 — Design Spec
|
||||
|
||||
**Status:** approved 2026-04-27
|
||||
**Slice:** M3 sub-plan #1 of 3 (recommendation engine v1 + contextual likes). Spec §13 step 7. Step 8 (session vectors + contextual_match_score) is the next sub-plan; this slice ships the scoring foundation without the contextual term.
|
||||
**Fable task:** #340.
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the M2 stub `/api/radio` (currently returns just the seed track) with a real weighted-shuffle implementation that scores user library tracks against a per-track-stats formula and returns a top-N queue ordered by score. After this slice, clicking a track in the SPA's search/track-row "play radio" surface produces a meaningful 50-track queue instead of a one-track stub.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Session vectors / `contextual_match_score` (sub-plan #2 / Fable #341 covers the write path; sub-plan #3 / Fable #342 folds the score into this function).
|
||||
- Real "radio" candidate pools from ListenBrainz similarity / similar artists / similar tags (M4).
|
||||
- Album-seed or artist-seed radio (`?seed_album=`, `?seed_artist=`). v1 is track-seeded only; the Fable task covers that scope explicitly.
|
||||
- Periodic radio refresh (the spec's "regenerate at 80% consumed"). Player-side concern; lands later.
|
||||
- Persistent radio queues. Each `/api/radio` call returns a fresh selection — stateless on the server.
|
||||
- Performance optimization for very large libraries (>50k tracks). At v1 scale the wide-pool scan is fine; M3.5 / M4 can add sampling if telemetry shows it matters.
|
||||
- `Subsonic` star-radio compatibility. Subsonic's `getRandomSongs` is a different shape; we don't try to map weighted shuffle into it for v1.
|
||||
|
||||
## Architecture
|
||||
|
||||
A new `internal/recommendation` package owns:
|
||||
|
||||
- A pure scoring function `Score(inputs, weights, now, rng) → float64` with no DB dependency. The `rng` parameter is injectable so tests pin jitter to deterministic values.
|
||||
- A candidate loader `LoadCandidates(ctx, q, userID, seedID, recentlyPlayedHours) → []Candidate` that runs ONE SQL query joining `tracks`, `general_likes`, and aggregated `play_events` to produce the per-track stats the scoring function needs. Excludes the seed and any track played within the recently-played window.
|
||||
- A `Shuffle(candidates, weights, now, rng, limit)` orchestrator that scores each candidate, sorts descending, truncates to `limit`. Pure.
|
||||
|
||||
The `/api/radio` handler becomes a thin shim: validate the seed → call `LoadCandidates` → call `Shuffle` → prepend the seed to the result → project to `[]TrackRef` → return JSON.
|
||||
|
||||
**Why split into three layers:**
|
||||
- `score.go` is the math; trivially unit-testable, no infra.
|
||||
- `candidates.go` is the data access; integration-tested against a live DB.
|
||||
- `shuffle.go` is the composition; pure-function tests against fake candidate sets.
|
||||
- The HTTP handler is the I/O glue; lives in `api/radio.go` like the existing pattern.
|
||||
|
||||
This separation also sets up sub-plan #3 (contextual scoring) cleanly — we'll add `LoadContextualSimilarity` next to `LoadCandidates`, extend `ScoringInputs` with `ContextualMatchScore`, and the rest of the chain doesn't change.
|
||||
|
||||
## Schema
|
||||
|
||||
No migration. All inputs are computed on demand from existing tables:
|
||||
|
||||
- `general_likes` for `is_general_liked`.
|
||||
- `play_events` for `last_played_at`, `play_count`, `skip_count`. Aggregations done in the candidate-loader query.
|
||||
- `tracks` provides the candidate set itself.
|
||||
|
||||
The recently-played-in-the-last-hour filter is applied at candidate-load time (a single `WHERE NOT EXISTS (... AND started_at > now() - interval '1 hour')` clause). Hard suppression — these tracks aren't even scored.
|
||||
|
||||
## Scoring math
|
||||
|
||||
```go
|
||||
type ScoringInputs struct {
|
||||
IsGeneralLiked bool
|
||||
LastPlayedAt *time.Time // nil = never played
|
||||
PlayCount int // total play_events
|
||||
SkipCount int // play_events with was_skipped=true
|
||||
}
|
||||
|
||||
type ScoringWeights struct {
|
||||
BaseWeight float64 // default 1.0
|
||||
LikeBoost float64 // default 2.0
|
||||
RecencyWeight float64 // default 1.0
|
||||
SkipPenalty float64 // default 1.0
|
||||
JitterMagnitude float64 // default 0.1
|
||||
}
|
||||
|
||||
func Score(in ScoringInputs, w ScoringWeights, now time.Time, rng func() float64) float64 {
|
||||
s := w.BaseWeight
|
||||
if in.IsGeneralLiked {
|
||||
s += w.LikeBoost
|
||||
}
|
||||
s += recencyDecay(in.LastPlayedAt, now) * w.RecencyWeight
|
||||
s -= skipRatio(in.PlayCount, in.SkipCount) * w.SkipPenalty
|
||||
s += (rng()*2 - 1) * w.JitterMagnitude
|
||||
return s
|
||||
}
|
||||
```
|
||||
|
||||
**`recencyDecay(lastPlayed *time.Time, now time.Time) float64`** — returns `[0, 1]`:
|
||||
- Never played (`lastPlayed == nil`) → `1.0`. Cold-start tracks compete favorably with stale ones; otherwise the recommendation engine never surfaces music the user hasn't tried.
|
||||
- Played within last hour: doesn't reach this function (filtered earlier at the candidate-pool stage).
|
||||
- Otherwise: `min(age_days / 30.0, 1.0)`. Linear ramp; tracks ≥ 30 days stale hit max recency boost.
|
||||
|
||||
**`skipRatio(plays, skips int) float64`** — returns `[0, 1]`:
|
||||
- `plays == 0` → `0.0`. Don't penalize never-played tracks.
|
||||
- Otherwise → `float64(skips) / float64(plays)`.
|
||||
|
||||
**Random jitter** — `(rng()*2 - 1) * JitterMagnitude` produces values in `[-magnitude, +magnitude]`. The `rng` parameter is `func() float64` (matches `math/rand.Float64`); tests inject a fixed-value version for deterministic ordering.
|
||||
|
||||
**Score range under defaults:**
|
||||
- Min (unliked, recent, all-skips): `1.0 + 0 + 0 - 1.0 - 0.1 = -0.1`
|
||||
- Max (liked, ≥30d stale, never skipped): `1.0 + 2.0 + 1.0 - 0 + 0.1 = 4.1`
|
||||
- Liked-track structural advantage (`LikeBoost = 2.0`) dominates the jitter band (`±0.1`), so liked tracks always rank above identical unliked tracks.
|
||||
|
||||
Configurable via YAML / env. Operators can crank `LikeBoost` up if their library is dominated by stuff they don't actually like, or `JitterMagnitude` up if they want more variety.
|
||||
|
||||
## API contracts
|
||||
|
||||
**Request:**
|
||||
|
||||
```
|
||||
GET /api/radio?seed_track=<uuid>&limit=<int>
|
||||
```
|
||||
|
||||
- `seed_track` (required) — UUID of the track that seeds the radio.
|
||||
- `limit` (optional, default 50, max 200) — total number of tracks to return *including* the seed.
|
||||
|
||||
**Response (shape unchanged from M2 stub, only contents grew):**
|
||||
|
||||
```ts
|
||||
type RadioResponse = { tracks: TrackRef[] };
|
||||
```
|
||||
|
||||
- `tracks[0]` is always the seed track (so `playQueue(resp.tracks, 0)` plays the seed).
|
||||
- `tracks[1..]` are the top-scored candidates from the user's library, descending. Length up to `limit - 1`.
|
||||
- Cold-start (empty library beyond the seed): returns `{ tracks: [<seed>] }`.
|
||||
|
||||
**Errors (unchanged from M2 stub):**
|
||||
- `400 bad_request` — missing `seed_track`, malformed UUID, or `limit < 1`.
|
||||
- `404 not_found` — `seed_track` doesn't exist.
|
||||
- `500 server_error` — DB issue.
|
||||
|
||||
**No web client changes.** The existing `playRadio(seedTrackId)` already calls this endpoint and feeds `resp.tracks` into `playQueue(tracks, 0)`. After this slice the queue is fuller; the call shape is identical.
|
||||
|
||||
**No Subsonic changes.** Track-seeded radio isn't part of Subsonic's surface in our v1 scope.
|
||||
|
||||
## Components & files
|
||||
|
||||
### New server files
|
||||
|
||||
| Path | Responsibility |
|
||||
|---|---|
|
||||
| `internal/recommendation/score.go` | Pure scoring function + `recencyDecay` + `skipRatio` helpers. No DB. |
|
||||
| `internal/recommendation/score_test.go` | Boundary cases: every term, cold-start, jitter determinism, score ranges. |
|
||||
| `internal/recommendation/candidates.go` | `LoadCandidates(...)` — single SQL query returning `[]Candidate` (`Track + ScoringInputs`). |
|
||||
| `internal/recommendation/candidates_test.go` | Live-DB tests: seed exclusion, recently-played exclusion, stat-join correctness, cross-user isolation. |
|
||||
| `internal/recommendation/shuffle.go` | `Shuffle(candidates, weights, now, rng, limit) []Candidate` — composes Score + sort + truncate. Pure. |
|
||||
| `internal/recommendation/shuffle_test.go` | Pure-function tests: liked-rank-higher, high-skip-rejected, jitter-doesn't-reorder-structural-winners, limit. |
|
||||
| `internal/db/queries/recommendation.sql` | sqlc query: `LoadRadioCandidates` — SELECT tracks WITH LEFT JOIN general_likes, LEFT JOIN aggregated play_events stats. WHERE clause excludes seed_id and last-hour plays. |
|
||||
| `internal/db/dbq/recommendation.sql.go` | Generated bindings. |
|
||||
|
||||
### Modified server files
|
||||
|
||||
| Path | Change |
|
||||
|---|---|
|
||||
| `internal/api/radio.go` | Replace stub. New flow: validate `seed_track` and `limit`, call `recommendation.LoadCandidates`, call `recommendation.Shuffle`, prepend seed, project to `[]TrackRef`, return JSON. |
|
||||
| `internal/api/radio_test.go` | Replace stub tests with: cold-start, typical (seed + scored picks), 404 on unknown seed, 400 on bad seed/limit, cross-user isolation. |
|
||||
| `internal/config/config.go` + `config.example.yaml` | Add `RecommendationConfig` struct: `BaseWeight` (1.0), `LikeBoost` (2.0), `RecencyWeight` (1.0), `SkipPenalty` (1.0), `JitterMagnitude` (0.1), `RecentlyPlayedHours` (1), `RadioSize` (50), `RadioSizeMax` (200). YAML key `recommendation:`. |
|
||||
| `internal/api/api.go` | `handlers` struct gains `recCfg config.RecommendationConfig`. `Mount` signature gains the config arg; constructs the handler with it; the radio handler builds `ScoringWeights` from it per request. |
|
||||
| `internal/server/server.go` + `cmd/minstrel/main.go` | Pass `cfg.Recommendation` through `Mount`. |
|
||||
|
||||
### No web changes
|
||||
|
||||
The existing `playRadio(seedTrackId)` already consumes `RadioResponse`. The web slice for this PR is empty — server-only.
|
||||
|
||||
## Data flow
|
||||
|
||||
1. SPA calls `playRadio(seedTrackId)` (existing player-store action). It POSTs `GET /api/radio?seed_track=<id>` (no explicit `limit`, server defaults to 50).
|
||||
2. `handleRadio` validates auth + seed UUID. Looks up the track to confirm it exists (404 otherwise).
|
||||
3. Reads `limit` from query (default `cfg.Recommendation.RadioSize`, clamped to `cfg.Recommendation.RadioSizeMax`).
|
||||
4. Calls `recommendation.LoadCandidates(ctx, q, userID, seedID, cfg.Recommendation.RecentlyPlayedHours)`. The SQL query joins:
|
||||
- `tracks t` — base set
|
||||
- `LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id` → `is_liked`
|
||||
- `LEFT JOIN LATERAL (SELECT max(started_at) FROM play_events WHERE user_id = $1 AND track_id = t.id) pe_last → last_played_at`
|
||||
- `LEFT JOIN LATERAL (SELECT count(*), count(*) FILTER (WHERE was_skipped) FROM play_events WHERE user_id = $1 AND track_id = t.id) pe_stats → play_count, skip_count`
|
||||
- `WHERE t.id <> seed_id AND NOT EXISTS (SELECT 1 FROM play_events WHERE user_id = $1 AND track_id = t.id AND started_at > now() - interval '$2 hours')`
|
||||
5. Iterates the candidates, calls `Score(...)` per track, sorts descending, truncates to `limit - 1`.
|
||||
6. Prepends the seed track, projects each `Candidate.Track` to `TrackRef` (existing `trackRefFrom` helper), writes `RadioResponse{ Tracks: ... }`.
|
||||
7. SPA receives 50 tracks, plays from index 0 (the seed).
|
||||
|
||||
**Stateless** — every call recomputes from scratch. No persisted radio queue, no cursor tracking, no "you already heard this" memory beyond the recently-played-hours window.
|
||||
|
||||
## Testing
|
||||
|
||||
### Server (`go test`)
|
||||
|
||||
**`internal/recommendation/score_test.go`** (pure unit tests):
|
||||
- Base case (never played, not liked, no skip data) with deterministic RNG `() => 0.5` → score = `BaseWeight + RecencyWeight + 0` exactly.
|
||||
- Liked boost: same inputs but `IsGeneralLiked=true` → score increases by `LikeBoost`.
|
||||
- Recency ramp: `lastPlayedAt = now - 15d` → `recencyDecay = 0.5`. `lastPlayedAt = now - 60d` → `1.0` (capped).
|
||||
- Skip ratio: `PlayCount=4, SkipCount=2` → ratio `0.5`, score loses `0.5 * SkipPenalty`.
|
||||
- Cold-start skip: `PlayCount=0, SkipCount=0` → ratio `0.0`, no penalty.
|
||||
- Jitter bounds: 1000 `Score(...)` calls with `math/rand.Float64`, every result within `[mid - JitterMagnitude, mid + JitterMagnitude]` where `mid` is the deterministic-RNG score.
|
||||
- Determinism: same `(inputs, weights, now, rng)` returns the same score (regression guard for hidden global state).
|
||||
|
||||
**`internal/recommendation/shuffle_test.go`** (pure unit tests):
|
||||
- Liked-vs-not: two otherwise-identical candidates → liked one ranks higher.
|
||||
- High-skip rejected: candidate with `skipRatio=1.0` ranks last among otherwise-identical candidates.
|
||||
- Limit truncates: 100 candidates, `limit=10` → result has 10.
|
||||
- Jitter doesn't reorder structural winners: 100 random RNG seeds → liked track ALWAYS ranks above an equivalent unliked track.
|
||||
- Empty input → empty output.
|
||||
|
||||
**`internal/recommendation/candidates_test.go`** (live DB):
|
||||
- Seed exclusion: 5 tracks in library, ask for radio with one as seed → returns the other 4.
|
||||
- Recently-played exclusion: seed a `play_events` row with `started_at = now - 30 minutes` for track A → radio excludes A.
|
||||
- Stat join: track with one play + one skip → `play_count=1, skip_count=1`. Liked track → `is_liked=true`. Never-played → `last_played_at IS NULL`.
|
||||
- Cross-user: Alice's plays / likes do NOT appear in Bob's stats.
|
||||
|
||||
**`internal/api/radio_test.go`** (HTTP integration, replacing existing stub tests):
|
||||
- Cold-start: seed only in library → response is `{ tracks: [seed] }`.
|
||||
- Typical: seed + 5 other tracks → response is `[seed, 5 ranked]`. Seed always at index 0.
|
||||
- 404 on unknown seed.
|
||||
- 400 on missing/malformed seed.
|
||||
- 400 on `limit=0` or `limit < 0`.
|
||||
- `limit > RadioSizeMax` clamped (returns at most `RadioSizeMax`, no error).
|
||||
- Cross-user isolation: Alice has many plays + likes → Bob's radio uses Bob's clean stats.
|
||||
|
||||
**Coverage:** `go test -coverprofile=cover.out ./internal/recommendation/...` → ≥ 70% per the M3 milestone description. The pure-function tests should hit ~100% on `score.go` + `shuffle.go`; `candidates.go` reaches similar coverage via the integration tests.
|
||||
|
||||
### End-to-end manual
|
||||
|
||||
Final task in the implementation plan:
|
||||
|
||||
1. Sign in. Play 3 tracks all the way through (build `play_count` history).
|
||||
2. Skip a 4th track within 10 seconds (creates a high `skip_ratio`).
|
||||
3. Like 2 tracks via the heart button.
|
||||
4. Click radio on a 5th track.
|
||||
5. Inspect the response in dev tools network tab — 50 tracks, seed first.
|
||||
6. The 2 liked tracks appear early in the list (within first ~20).
|
||||
7. The just-skipped track appears late or absent.
|
||||
8. The 3 just-played tracks are absent (recently-played exclusion).
|
||||
9. Trigger radio again with a different seed; ordering varies (jitter), liked tracks still rank prominently.
|
||||
|
||||
## Risks & mitigations
|
||||
|
||||
- **Wide-pool scan performance.** For a 50k-track library: 50k rows × scoring overhead × sort ≈ a few hundred ms in Go. Below the user-perceptible threshold for a click-to-play. If telemetry later shows >1s P95 latency, M3.5 / M4 can add sampling (random-N candidate pre-filter) or a partial index. The candidate-pool function is the swap point; the scoring function and HTTP handler don't change.
|
||||
- **Recently-played window edge effects.** If the user plays 50 tracks in an hour, all of them get excluded from the next radio call. Library < 100 tracks could leave the candidate pool too thin to fill `RadioSize`. Mitigation: handler returns whatever it can fit (length might be < `RadioSize`); SPA's `playQueue` works on any non-empty array. Document as acceptable degradation; users with tiny libraries get short radios.
|
||||
- **Skip-ratio gaming.** A user who skips a track once (during initial library discovery) gets a `skipRatio=1.0` for that track. With `SkipPenalty=1.0` that's a `-1.0` hit — could permanently demote the track even if they like it. Mitigation: weights are tunable; user can lower `SkipPenalty`. Future: smoothing (e.g. `(skips + α) / (plays + β)`) instead of raw ratio. Out of scope for v1.
|
||||
- **Cold-start RNG dominance.** A brand-new user with zero plays / likes gets every track scored at `BaseWeight + RecencyWeight + jitter`. Top-N is essentially random. That's fine — it matches user expectation ("I haven't told you anything about me, give me random music"). The recency-decay max-for-never-played avoids amplifying new tracks just because they're new.
|
||||
- **Score formula evolution.** Adding `contextual_match_score` in sub-plan #3 means changing `ScoringInputs` and the `Score` function signature. Mitigation: the function is internal-package only; sub-plan #3 changes both ends of the call atomically. No external consumers.
|
||||
Reference in New Issue
Block a user