The verify half of the tune→verify loop (#1251), on the same admin
Tuning page as the knobs:
- RecommendationWeeklyTrends: weekly per-source outcomes aggregated
across all users (the knobs are global, so judging a turn needs
global outcomes — rows carry rates only, no track/user identity),
with a taste-hit count per bucket: plays whose track's artist has a
positive weight in the player's current taste profile. That's the
"cheap recompute" reading — retroactive over the whole window, at
the cost of profile drift.
- GET /api/admin/recommendation-trends?weeks=N (default 12, cap 52):
per-family weekly series (skip rate, sample-weighted completion,
taste-hit rate) plus the tuning-audit markers inside the window.
- Web: sparkline table under the tuning cards — skip rate per week on
a shared axis with dashed ticks at knob turns, latest-week columns,
window taste-hit rate, low-volume rows dimmed as anecdote, and a
plain-text list of the window's tuning changes.
Also fixes the revive unused-parameter lint on the tuning GET handler
that failed CI run 1903 on the previous commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsF3cNoKrqCYsU78cXC8U6
Milestone #127 step 2 (#1249). For You deliberately blends two
populations — a head of top-scored taste picks and a tail sampled from
deeper ranking (the freshness injection) — but the metrics judged it as
one blob, so its skip rate couldn't distinguish "the taste engine is
missing" from "the freshness tax is too high". That number decides the
exploration share before we tune it.
- Migration 0038: nullable pick_kind ('taste'|'fresh') on both
playlist_tracks (stamped at snapshot build) and play_events (frozen at
play-ingestion — the snapshot rebuilds daily, so attribution cannot be
reconstructed at read time).
- Builder: pickHeadAndTail marks head=taste / tail=fresh; the small-pool
fallback is all taste (top-N-by-score IS the taste mechanism). Other
variants persist NULL.
- Ingestion: for_you plays (live + offline replay) look the track up in
the user's current snapshot; not found → unattributed, never guessed.
- Metrics: For You's row gains a breakdown (taste / fresh / earlier
unattributed plays), parent row stays the sum; web card renders the
sub-rows indented with the same baseline deltas + low-data dimming.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsF3cNoKrqCYsU78cXC8U6
The recommendation metrics table was observable but not actionable: raw
source strings (album:<uuid> one-offs) drowned the stable surfaces, and
manual plays were excluded so skip rates had no control group.
- SQL: include NULL-source rows (the baseline) and carry completion_n
so family merges can weight avg_completion correctly.
- Handler buckets raw sources into stable families (radio:<uuid> →
Radio, album:/artist: → direct plays, etc.) grouped by surface
intent: go-to / discovery / direct — each band judged against its
job, since discovery mixes are expected to skip hotter. Families
under 20 plays are flagged low-confidence, not hidden.
- Settings card renders the baseline row and per-surface deltas in
percentage points vs baseline (worse-than-baseline deltas in danger
color), intent hint copy per group, low-data rows dimmed.
- Pure-unit test for the bucketing/merge; DB test updated to the new
contract (baseline included, radio:<uuid> collapse).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsF3cNoKrqCYsU78cXC8U6
Per-source play outcomes so the operator can see whether each recommendation
surface is landing and tune the now-operator-tunable taste weights.
Server:
- query RecommendationSourceMetricsForUser: groups the user's play_events by
source (system-playlist surface), reporting plays / skips / avg completion
over a window; NULL-source (library/radio) plays excluded.
- GET /api/me/recommendation-metrics?days=30 (default 30, capped 365) →
{window_days, sources:[{source, plays, skips, skip_rate, avg_completion}]}.
- handler test: 401 unauth; per-source aggregation + NULL-source exclusion +
skip_rate / avg_completion math.
Web:
- lib/api/metrics.ts: query + friendly source labels.
- settings page gains a "Recommendation metrics" card (table of surface / plays
/ skip rate / avg completion), with loading/error/empty states.
- settings tests mock the new query (manual subscribe-store, hoisting-safe).
Note: You-might-like plays aren't source-tagged (it's a Home row, not a system
playlist), so this covers For-You / Discover / the mixes. Tagging YML plays
would be a client follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bvandeusen