Recommendation metrics: publish the margin of error on every delta #122

Merged

bvandeusen merged 4 commits from dev into main 2026-08-06 21:50:42 -04:00

Conversation 0 Commits 4 Files Changed 11

+538 -50

4 Commits

Author	SHA1	Message	Date
bvandeusen	304de88c50	test(tuning): assert headers by exact accessible name — #2495 ... Third attempt at the same assertion, so I stopped guessing and got vitest running locally instead: the web lane uses the same ci-go image, so `docker run ... -w /src/web ci-go:1.26 npx vitest run` works and turns a 5-minute CI round trip into 7 seconds. /^Skip/ matched the "Skip rate by week" sparkline column as well as "Skip (last wk)", just as /Plays/ had matched the caption. Exact names say what the assertion means and cannot drift onto a neighbour. Verified locally before pushing: svelte-check 0 errors, 110 files / 786 tests pass.	2026-08-06 21:27:22 -04:00
bvandeusen	96abb48086	test(tuning): query the window/last-week headers as column headers — #2495 ... getByText(/Plays/) matched my own new caption as well as the header, since the caption explains which columns cover the window. Query by columnheader role instead, which is what the assertion actually means. Also reordered the caption: prepending the clarification turned it into a run-on that opened mid-explanation before saying what the chart was.	2026-08-06 21:20:18 -04:00
bvandeusen	a094d5f8b0	test(metrics): target deltas by test id, not by glyph — #2495 ... Two CI failures, both in my own new tests, both informative. settings: queryByText(/≈/) matched the LEGEND explaining the glyph rather than a delta, so the "no delta" case failed on the explanation being present. Delta spans now carry data-testid so a test can name what it means instead of pattern-matching prose that sits next to it. tuning: getByText("40%") found two elements. testing-library matches an element and its OWN direct text nodes, so the skip cell still matches "40%" despite the trailing play-count span — and discover late-week completion is also 40%. Genuinely ambiguous now; assert the count.	2026-08-06 21:14:53 -04:00
bvandeusen	481f906059	feat(metrics): publish margin of error on every delta — #2495, #2524 ... The metrics card had one volume threshold doing two jobs. recMetricsLowVolume = 20 is a DISPLAY floor — below that a skip rate is anecdote — but the card then presented deltas as though it were also a DECISION floor. Those differ by an order of magnitude: detecting the ~13pp differences that matter needs ~133 plays per arm for 80% power at a=0.05. So Discover's taste-matched (59 plays) and random-unheard (70) both rendered as full-confidence rows with a bold delta beside them, and that comparison sits at p ~ 0.06. The card said "signal"; the arithmetic said "maybe". It produced a recommendation the data didn't support, and any reader with the same numbers would have made the same call. Deltas now carry a 95% margin of error and a `distinguishable` flag, computed server-side so both clients read the same arithmetic instead of each re-deriving it. Skip rate is a two-proportion difference; completion is Welch, which needs a variance — hence completion_sqsum in the query. It is the sum of squares rather than stddev_samp on purpose: raw source rows are merged into surface families in Go, and sums of squares combine across groups exactly whereas standard deviations cannot. recMetricsLowVolume is untouched. "Too thin to show" and "too thin to act on" are different questions. Web renders an indistinguishable delta as dimmed and prefixed "≈", with the range on hover and a legend explaining the glyph. Colour is withheld unless the delta clears its margin — colouring noise red is what made the old card misleading. Breakdown rows go through the same path; those are the thinnest samples on screen and where the old card misled most. Also fixes the admin trends view, which had the same problem worse: its "Latest skip"/"Latest completion" columns are one WEEK while the adjacent Plays column is the whole window. I misread exactly that and briefly concluded Deep cuts was the worst surface, from ~17 plays in a single week — over 180 days it is one of the best. Headers now name their period and the skip cell carries that week's play count. #2524: resolveArtist now recognises a duplicate-MBID unique violation as the expected condition it is, matching resolveAlbum. Two rows mapping to one MusicBrainz artist is a merge candidate, not a fault; without the branch it logged a generic warning plus a Postgres ERROR line on every scan, which teaches an operator to ignore database errors.	2026-08-06 21:08:58 -04:00