bvandeusen
481f906059
The metrics card had one volume threshold doing two jobs. recMetricsLowVolume = 20 is a DISPLAY floor — below that a skip rate is anecdote — but the card then presented deltas as though it were also a DECISION floor. Those differ by an order of magnitude: detecting the ~13pp differences that matter needs ~133 plays per arm for 80% power at a=0.05. So Discover's taste-matched (59 plays) and random-unheard (70) both rendered as full-confidence rows with a bold delta beside them, and that comparison sits at p ~ 0.06. The card said "signal"; the arithmetic said "maybe". It produced a recommendation the data didn't support, and any reader with the same numbers would have made the same call. Deltas now carry a 95% margin of error and a `distinguishable` flag, computed server-side so both clients read the same arithmetic instead of each re-deriving it. Skip rate is a two-proportion difference; completion is Welch, which needs a variance — hence completion_sqsum in the query. It is the sum of squares rather than stddev_samp on purpose: raw source rows are merged into surface families in Go, and sums of squares combine across groups exactly whereas standard deviations cannot. recMetricsLowVolume is untouched. "Too thin to show" and "too thin to act on" are different questions. Web renders an indistinguishable delta as dimmed and prefixed "≈", with the range on hover and a legend explaining the glyph. Colour is withheld unless the delta clears its margin — colouring noise red is what made the old card misleading. Breakdown rows go through the same path; those are the thinnest samples on screen and where the old card misled most. Also fixes the admin trends view, which had the same problem worse: its "Latest skip"/"Latest completion" columns are one WEEK while the adjacent Plays column is the whole window. I misread exactly that and briefly concluded Deep cuts was the worst surface, from ~17 plays in a single week — over 180 days it is one of the best. Headers now name their period and the skip cell carries that week's play count. #2524: resolveArtist now recognises a duplicate-MBID unique violation as the expected condition it is, matching resolveAlbum. Two rows mapping to one MusicBrainz artist is a merge candidate, not a fault; without the branch it logged a generic warning plus a Postgres ERROR line on every scan, which teaches an operator to ignore database errors.
67 lines
3.3 KiB
SQL
67 lines
3.3 KiB
SQL
-- Recommendation observability (#796 phase 4). Per-source play outcomes so the
|
||
-- operator can see whether each recommendation surface is landing and tune the
|
||
-- taste weights. Source is stamped on play_events when a play is launched from
|
||
-- a recommendation surface; NULL means the user picked the track manually —
|
||
-- those rows are INCLUDED here as the baseline control group the surfaces are
|
||
-- judged against (milestone #127: delta-vs-baseline is what makes the numbers
|
||
-- actionable). Raw source strings are bucketed into stable surface families in
|
||
-- the Go handler; completion_n is carried so family merges can weight
|
||
-- avg_completion correctly.
|
||
|
||
-- name: RecommendationWeeklyTrends :many
|
||
-- Weekly per-source outcome series for the tuning lab's trend view
|
||
-- (#1251). Aggregated across ALL users: the tuning knobs are global,
|
||
-- so judging a knob turn needs global outcomes — rows carry rates
|
||
-- only, no track or user identity. NULL-source (manual) rows are
|
||
-- included as the baseline family.
|
||
--
|
||
-- taste_hits counts plays whose track's artist has a positive weight
|
||
-- in that user's CURRENT taste profile — the "cheap recompute" option:
|
||
-- retroactive over the whole window, at the cost of drift (the profile
|
||
-- is today's, the play may be weeks old). Good enough to read whether
|
||
-- a surface is feeding taste-fitting tracks.
|
||
-- $1 window in weeks.
|
||
SELECT
|
||
date_trunc('week', pe.started_at)::date AS week_start,
|
||
pe.source,
|
||
count(*)::bigint AS plays,
|
||
count(*) FILTER (WHERE pe.was_skipped)::bigint AS skips,
|
||
count(pe.completion_ratio)::bigint AS completion_n,
|
||
COALESCE(avg(pe.completion_ratio), 0)::float8 AS avg_completion,
|
||
count(*) FILTER (WHERE tpa.artist_id IS NOT NULL)::bigint AS taste_hits
|
||
FROM play_events pe
|
||
JOIN tracks t ON t.id = pe.track_id
|
||
LEFT JOIN taste_profile_artists tpa
|
||
ON tpa.user_id = pe.user_id
|
||
AND tpa.artist_id = t.artist_id
|
||
AND tpa.weight > 0
|
||
WHERE pe.started_at > now() - (sqlc.arg(weeks)::int * INTERVAL '1 week')
|
||
GROUP BY 1, 2
|
||
ORDER BY 1, 2;
|
||
|
||
-- name: RecommendationSourceMetricsForUser :many
|
||
-- $1 user_id, $2 window_days. plays/skips are counts; avg_completion is the
|
||
-- mean completion ratio over the completion_n plays that recorded one.
|
||
-- pick_kind splits For You plays into taste/fresh/unattributed (#1249);
|
||
-- it is NULL for every other source, so those still group to one row.
|
||
--
|
||
-- completion_sqsum carries the sum of SQUARED completion ratios so the Go
|
||
-- handler can compute a variance — needed for the margin of error on a
|
||
-- completion delta (#2495). It is the sum rather than `stddev_samp` on purpose:
|
||
-- raw source rows get merged into surface families in Go, and sums of squares
|
||
-- add across groups exactly, whereas standard deviations cannot be combined
|
||
-- without them. Variance = (sqsum - sum²/n) / (n-1), with sum = avg × n.
|
||
SELECT
|
||
pe.source,
|
||
pe.pick_kind,
|
||
count(*)::bigint AS plays,
|
||
count(*) FILTER (WHERE pe.was_skipped)::bigint AS skips,
|
||
count(pe.completion_ratio)::bigint AS completion_n,
|
||
COALESCE(avg(pe.completion_ratio), 0)::float8 AS avg_completion,
|
||
COALESCE(sum(pe.completion_ratio * pe.completion_ratio), 0)::float8 AS completion_sqsum
|
||
FROM play_events pe
|
||
WHERE pe.user_id = $1
|
||
AND pe.started_at > now() - ($2::float8 * INTERVAL '1 day')
|
||
GROUP BY pe.source, pe.pick_kind
|
||
ORDER BY plays DESC;
|