bvandeusen
481f906059
The metrics card had one volume threshold doing two jobs. recMetricsLowVolume = 20 is a DISPLAY floor — below that a skip rate is anecdote — but the card then presented deltas as though it were also a DECISION floor. Those differ by an order of magnitude: detecting the ~13pp differences that matter needs ~133 plays per arm for 80% power at a=0.05. So Discover's taste-matched (59 plays) and random-unheard (70) both rendered as full-confidence rows with a bold delta beside them, and that comparison sits at p ~ 0.06. The card said "signal"; the arithmetic said "maybe". It produced a recommendation the data didn't support, and any reader with the same numbers would have made the same call. Deltas now carry a 95% margin of error and a `distinguishable` flag, computed server-side so both clients read the same arithmetic instead of each re-deriving it. Skip rate is a two-proportion difference; completion is Welch, which needs a variance — hence completion_sqsum in the query. It is the sum of squares rather than stddev_samp on purpose: raw source rows are merged into surface families in Go, and sums of squares combine across groups exactly whereas standard deviations cannot. recMetricsLowVolume is untouched. "Too thin to show" and "too thin to act on" are different questions. Web renders an indistinguishable delta as dimmed and prefixed "≈", with the range on hover and a legend explaining the glyph. Colour is withheld unless the delta clears its margin — colouring noise red is what made the old card misleading. Breakdown rows go through the same path; those are the thinnest samples on screen and where the old card misled most. Also fixes the admin trends view, which had the same problem worse: its "Latest skip"/"Latest completion" columns are one WEEK while the adjacent Plays column is the whole window. I misread exactly that and briefly concluded Deep cuts was the worst surface, from ~17 plays in a single week — over 180 days it is one of the best. Headers now name their period and the skip cell carries that week's play count. #2524: resolveArtist now recognises a duplicate-MBID unique violation as the expected condition it is, matching resolveAlbum. Two rows mapping to one MusicBrainz artist is a merge candidate, not a fault; without the branch it logged a generic warning plus a Postgres ERROR line on every scan, which teaches an operator to ignore database errors.
118 lines
4.9 KiB
Go
118 lines
4.9 KiB
Go
package api
|
||
|
||
import "math"
|
||
|
||
// Uncertainty on the deltas the recommendation-metrics card shows (#2495).
|
||
//
|
||
// Why this exists: the card had exactly one volume threshold,
|
||
// recMetricsLowVolume = 20, and it was doing two jobs. Twenty plays is enough to
|
||
// be worth DISPLAYING — below that a skip rate is anecdote — but it is nowhere
|
||
// near enough to ACT on. Detecting the ~13pp differences that actually matter
|
||
// needs roughly 133 plays per arm for 80% power at α=0.05.
|
||
//
|
||
// So Discover's taste-matched (59 plays) and random-unheard (70) both rendered as
|
||
// full-confidence rows with a bold delta beside them, and that comparison sits at
|
||
// p ≈ 0.06. The card said "signal"; the arithmetic said "maybe". It led directly
|
||
// to a recommendation the data didn't support, and any reader with the same
|
||
// numbers would have made the same call.
|
||
//
|
||
// The fix is to publish the margin of error next to the delta and flag when the
|
||
// delta is smaller than it — i.e. not distinguishable from zero. Computed here,
|
||
// server-side, so both clients agree rather than each re-deriving it.
|
||
//
|
||
// recMetricsLowVolume stays exactly as it was. This is a second, independent
|
||
// signal, not a replacement: "too thin to show" and "too thin to act on" are
|
||
// different questions and deserve different answers.
|
||
|
||
// deltaZ is the two-sided 95% normal critical value. Normal rather than
|
||
// Student's t: at the sample sizes where a delta is worth acting on (n in the
|
||
// hundreds) the difference is immaterial, and a household dashboard does not
|
||
// need a t-table.
|
||
const deltaZ = 1.96
|
||
|
||
// metricDelta is a difference from the baseline, with its uncertainty.
|
||
//
|
||
// Both figures are in PERCENTAGE POINTS, matching how the card reads them out —
|
||
// a skip rate of 0.153 against a baseline of 0.270 is "-11.7", not "-0.117".
|
||
type metricDelta struct {
|
||
// DeltaPP is surface minus baseline. Negative skip is better; negative
|
||
// completion is worse. The client owns that colouring.
|
||
DeltaPP float64 `json:"delta_pp"`
|
||
// MarginPP is the 95% margin of error on DeltaPP. Read the delta as
|
||
// DeltaPP ± MarginPP.
|
||
MarginPP float64 `json:"margin_pp"`
|
||
// Distinguishable reports |DeltaPP| >= MarginPP: the interval excludes
|
||
// zero, so the difference is worth reading as a difference. When false the
|
||
// number may be pure noise no matter how large it looks.
|
||
Distinguishable bool `json:"distinguishable"`
|
||
}
|
||
|
||
// proportionDelta compares two rates (skips/plays) as a two-proportion
|
||
// difference. Returns nil when either sample is empty, or when either rate is
|
||
// degenerate (0 or 1) — a rate with no observed variation has an SE of 0 on its
|
||
// side, which would report a spuriously narrow margin rather than an honest one.
|
||
func proportionDelta(rate1 float64, n1 int64, rate2 float64, n2 int64) *metricDelta {
|
||
if n1 <= 0 || n2 <= 0 {
|
||
return nil
|
||
}
|
||
v1 := rate1 * (1 - rate1) / float64(n1)
|
||
v2 := rate2 * (1 - rate2) / float64(n2)
|
||
se := math.Sqrt(v1 + v2)
|
||
if se <= 0 {
|
||
// Both rates are 0 or both are 1. The delta is exactly zero and the
|
||
// margin is meaningless; reporting nothing is more honest than
|
||
// reporting certainty.
|
||
return nil
|
||
}
|
||
return newDelta((rate1-rate2)*100, deltaZ*se*100)
|
||
}
|
||
|
||
// meanDelta compares two means (average completion ratio) using Welch's
|
||
// standard error, which does not assume equal variances between the two groups.
|
||
//
|
||
// Note the margins here are wider than intuition suggests, and that is correct:
|
||
// completion is strongly bimodal — a play is either abandoned early (≈0.05) or
|
||
// finished (≈1.0), with little in between — so its standard deviation is large
|
||
// (~0.4) even though the mean looks stable.
|
||
func meanDelta(mean1 float64, variance1 float64, n1 int64, mean2 float64, variance2 float64, n2 int64) *metricDelta {
|
||
// Two observations minimum per side: a sample variance needs n-1 > 0.
|
||
if n1 < 2 || n2 < 2 {
|
||
return nil
|
||
}
|
||
se := math.Sqrt(variance1/float64(n1) + variance2/float64(n2))
|
||
if se <= 0 || math.IsNaN(se) || math.IsInf(se, 0) {
|
||
return nil
|
||
}
|
||
return newDelta((mean1-mean2)*100, deltaZ*se*100)
|
||
}
|
||
|
||
func newDelta(deltaPP, marginPP float64) *metricDelta {
|
||
return &metricDelta{
|
||
DeltaPP: deltaPP,
|
||
MarginPP: marginPP,
|
||
// >= rather than >: a delta exactly equal to its margin sits on the
|
||
// boundary, and calling the boundary "distinguishable" is the
|
||
// conventional reading of a 95% interval that just excludes zero.
|
||
Distinguishable: math.Abs(deltaPP) >= marginPP,
|
||
}
|
||
}
|
||
|
||
// sampleVariance recovers the sample variance from the aggregates the SQL
|
||
// returns. sum is mean×n rather than a selected column, which keeps the query to
|
||
// one extra expression.
|
||
//
|
||
// The subtraction can go very slightly negative through floating-point
|
||
// cancellation when every observation is identical, so the result is clamped —
|
||
// a negative variance would produce NaN downstream.
|
||
func sampleVariance(sum, sqSum float64, n int64) float64 {
|
||
if n < 2 {
|
||
return 0
|
||
}
|
||
nf := float64(n)
|
||
v := (sqSum - (sum * sum / nf)) / (nf - 1)
|
||
if v < 0 {
|
||
return 0
|
||
}
|
||
return v
|
||
}
|