The metrics card had one volume threshold doing two jobs. recMetricsLowVolume = 20 is a DISPLAY floor — below that a skip rate is anecdote — but the card then presented deltas as though it were also a DECISION floor. Those differ by an order of magnitude: detecting the ~13pp differences that matter needs ~133 plays per arm for 80% power at a=0.05. So Discover's taste-matched (59 plays) and random-unheard (70) both rendered as full-confidence rows with a bold delta beside them, and that comparison sits at p ~ 0.06. The card said "signal"; the arithmetic said "maybe". It produced a recommendation the data didn't support, and any reader with the same numbers would have made the same call. Deltas now carry a 95% margin of error and a `distinguishable` flag, computed server-side so both clients read the same arithmetic instead of each re-deriving it. Skip rate is a two-proportion difference; completion is Welch, which needs a variance — hence completion_sqsum in the query. It is the sum of squares rather than stddev_samp on purpose: raw source rows are merged into surface families in Go, and sums of squares combine across groups exactly whereas standard deviations cannot. recMetricsLowVolume is untouched. "Too thin to show" and "too thin to act on" are different questions. Web renders an indistinguishable delta as dimmed and prefixed "≈", with the range on hover and a legend explaining the glyph. Colour is withheld unless the delta clears its margin — colouring noise red is what made the old card misleading. Breakdown rows go through the same path; those are the thinnest samples on screen and where the old card misled most. Also fixes the admin trends view, which had the same problem worse: its "Latest skip"/"Latest completion" columns are one WEEK while the adjacent Plays column is the whole window. I misread exactly that and briefly concluded Deep cuts was the worst surface, from ~17 plays in a single week — over 180 days it is one of the best. Headers now name their period and the skip cell carries that week's play count. #2524: resolveArtist now recognises a duplicate-MBID unique violation as the expected condition it is, matching resolveAlbum. Two rows mapping to one MusicBrainz artist is a merge candidate, not a fault; without the branch it logged a generic warning plus a Postgres ERROR line on every scan, which teaches an operator to ignore database errors.
This commit is contained in:
@@ -4,6 +4,20 @@ import { api } from './client';
|
||||
// Mirrors internal/api/me_recommendation_metrics.go: raw play sources are
|
||||
// bucketed server-side into stable surface families, grouped by intent, and
|
||||
// anchored by the manual-plays baseline (milestone #127).
|
||||
// A difference from the baseline, with its uncertainty (#2495). Both figures
|
||||
// are already in percentage points — the server does the arithmetic so both
|
||||
// clients read the same numbers.
|
||||
//
|
||||
// `distinguishable: false` means |delta_pp| < margin_pp: the delta cannot be
|
||||
// told apart from zero, however large it looks. That distinction is the whole
|
||||
// point of this type — `low_confidence` answers "is this worth showing?", which
|
||||
// is a much lower bar than "is this worth acting on?".
|
||||
export type MetricDelta = {
|
||||
delta_pp: number;
|
||||
margin_pp: number;
|
||||
distinguishable: boolean;
|
||||
};
|
||||
|
||||
export type SurfaceMetric = {
|
||||
key: string;
|
||||
label: string;
|
||||
@@ -12,6 +26,10 @@ export type SurfaceMetric = {
|
||||
skip_rate: number;
|
||||
avg_completion: number;
|
||||
low_confidence: boolean;
|
||||
// Absent on the baseline row itself, and whenever the samples are too thin
|
||||
// for a margin to mean anything.
|
||||
skip_delta?: MetricDelta;
|
||||
completion_delta?: MetricDelta;
|
||||
// Present when the surface's builder stamps pick-kind provenance and
|
||||
// the window holds attributed plays (#1249, generalized #1270): For
|
||||
// You's taste/fresh split, Discover's candidate buckets, the tiered
|
||||
|
||||
@@ -216,9 +216,17 @@
|
||||
return plays > 0 ? hits / plays : 0;
|
||||
}
|
||||
|
||||
function latest(s: TrendSeries): { skip: number; completion: number } {
|
||||
// The "latest" columns are ONE WEEK while the Plays column is the whole
|
||||
// window, which is a trap: a 40% skip rate off 17 plays sat next to a
|
||||
// four-figure Plays total and read as a solid signal. It isn't — I misread
|
||||
// exactly this and briefly concluded Deep cuts was the worst surface, when
|
||||
// over 180 days it's one of the best (#2495). So the week's own play count
|
||||
// comes back with the rates and is rendered beside them.
|
||||
function latest(s: TrendSeries): { skip: number; completion: number; plays: number } {
|
||||
const last = s.points[s.points.length - 1];
|
||||
return last ? { skip: last.skip_rate, completion: last.avg_completion } : { skip: 0, completion: 0 };
|
||||
return last
|
||||
? { skip: last.skip_rate, completion: last.avg_completion, plays: last.plays }
|
||||
: { skip: 0, completion: 0, plays: 0 };
|
||||
}
|
||||
|
||||
function pct(v: number): string {
|
||||
@@ -420,6 +428,10 @@
|
||||
<div>
|
||||
<h2 class="text-lg font-semibold">Weekly trends</h2>
|
||||
<p class="text-xs text-text-secondary">
|
||||
<span class="font-medium">Skip and completion are the most recent week only</span> —
|
||||
the number after the skip rate is that week's play count, so a rate off a
|
||||
handful of plays reads as what it is. Plays and Taste hit cover the whole
|
||||
window.
|
||||
Skip rate per surface over the last {trends?.weeks ?? 12} weeks (lower is better; all
|
||||
users aggregated, rates only). Dashed ticks mark tuning changes. Taste hit is the share
|
||||
of plays whose artist fits the current taste profile.
|
||||
@@ -439,10 +451,10 @@
|
||||
<tr class="text-left text-text-secondary">
|
||||
<th class="py-1 font-medium">Surface</th>
|
||||
<th class="py-1 font-medium">Skip rate by week</th>
|
||||
<th class="py-1 text-right font-medium">Plays</th>
|
||||
<th class="py-1 text-right font-medium">Latest skip</th>
|
||||
<th class="py-1 text-right font-medium">Latest completion</th>
|
||||
<th class="py-1 text-right font-medium">Taste hit</th>
|
||||
<th class="py-1 text-right font-medium">Plays<span class="font-normal text-xs"> (window)</span></th>
|
||||
<th class="py-1 text-right font-medium">Skip<span class="font-normal text-xs"> (last wk)</span></th>
|
||||
<th class="py-1 text-right font-medium">Completion<span class="font-normal text-xs"> (last wk)</span></th>
|
||||
<th class="py-1 text-right font-medium">Taste hit<span class="font-normal text-xs"> (window)</span></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
@@ -493,7 +505,10 @@
|
||||
</svg>
|
||||
</td>
|
||||
<td class="py-1.5 text-right tabular-nums">{s.plays}</td>
|
||||
<td class="py-1.5 text-right tabular-nums">{pct(latest(s).skip)}</td>
|
||||
<td class="py-1.5 text-right tabular-nums">
|
||||
{pct(latest(s).skip)}
|
||||
<span class="text-xs text-text-secondary">/{latest(s).plays}</span>
|
||||
</td>
|
||||
<td class="py-1.5 text-right tabular-nums">{pct(latest(s).completion)}</td>
|
||||
<td class="py-1.5 text-right tabular-nums">{pct(windowTasteHitRate(s))}</td>
|
||||
</tr>
|
||||
|
||||
@@ -195,9 +195,20 @@ describe('Admin tuning page', () => {
|
||||
await waitFor(() => expect(screen.getByText('Weekly trends')).toBeInTheDocument());
|
||||
expect(screen.getByTestId('sparkline-radio')).toBeInTheDocument();
|
||||
expect(screen.getByTestId('sparkline-discover')).toBeInTheDocument();
|
||||
// Latest skip rate column for radio = 40% (also discover's latest
|
||||
// completion, hence getAllBy).
|
||||
expect(screen.getAllByText('40%').length).toBeGreaterThan(0);
|
||||
// The skip column is the LAST WEEK's rate, now carrying that week's play
|
||||
// count so a rate off a handful of plays reads as what it is (#2495).
|
||||
// Radio's latest week: 40% skip over 15 plays; Discover's: 60% over 5.
|
||||
expect(screen.getByText('/15')).toBeInTheDocument();
|
||||
expect(screen.getByText('/5')).toBeInTheDocument();
|
||||
// Completion columns are unchanged and still bare percentages — radio 70%,
|
||||
// discover 40%.
|
||||
expect(screen.getByText('70%')).toBeInTheDocument();
|
||||
expect(screen.getByText('40%')).toBeInTheDocument();
|
||||
// The window/last-week distinction has to be visible in the headers, or the
|
||||
// Plays total reads as the denominator of the skip rate. That misreading is
|
||||
// what #2495 was filed over.
|
||||
expect(screen.getByText(/Plays/)).toHaveTextContent('(window)');
|
||||
expect(screen.getByText(/^Skip/)).toHaveTextContent('(last wk)');
|
||||
// The knob turn is listed under the chart AND tooltipped on each
|
||||
// sparkline's marker tick, hence getAllBy.
|
||||
expect(
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
createRecommendationMetricsQuery,
|
||||
type RecommendationMetrics,
|
||||
type SurfaceIntent,
|
||||
type SurfaceMetric
|
||||
type MetricDelta
|
||||
} from '$lib/api/metrics';
|
||||
import { theme, setTheme, type ThemePreference } from '$lib/stores/theme.svelte';
|
||||
import { player, setCrossfade } from '$lib/player/store.svelte';
|
||||
@@ -45,20 +45,39 @@
|
||||
return `${(v * 100).toFixed(0)}%`;
|
||||
}
|
||||
|
||||
// Delta in percentage points vs the baseline, signed ("+12" / "−5").
|
||||
function deltaPts(value: number, baseline: number): string {
|
||||
const pts = Math.round((value - baseline) * 100);
|
||||
return pts > 0 ? `+${pts}` : `${pts}`;
|
||||
// Deltas come from the server with their margin of error (#2495). The client
|
||||
// no longer subtracts rates itself: the margin needs the sample sizes and
|
||||
// variances, and having both clients re-derive it invites them to disagree.
|
||||
//
|
||||
// A delta that isn't distinguishable from zero is prefixed "≈" and dimmed.
|
||||
// That is the point of this whole change — the card used to render a −12 on
|
||||
// 59 plays exactly as boldly as a −6 on 400, and the first of those is noise.
|
||||
function deltaText(d: MetricDelta | undefined): string {
|
||||
if (!d) return '';
|
||||
const pts = Math.round(d.delta_pp);
|
||||
const signed = pts > 0 ? `+${pts}` : `${pts}`;
|
||||
return d.distinguishable ? signed : `≈${signed}`;
|
||||
}
|
||||
|
||||
// A surface's skip delta is "worse" when it skips more than the
|
||||
// baseline; completion delta is "worse" when it completes less.
|
||||
function skipDeltaClass(m: SurfaceMetric, baseline: SurfaceMetric): string {
|
||||
return m.skip_rate > baseline.skip_rate ? 'text-danger' : 'text-text-secondary';
|
||||
function deltaTitle(d: MetricDelta | undefined): string | undefined {
|
||||
if (!d) return undefined;
|
||||
const range = `${d.delta_pp.toFixed(1)} ± ${d.margin_pp.toFixed(1)} points vs baseline`;
|
||||
return d.distinguishable
|
||||
? `${range} (95% confidence)`
|
||||
: `${range} — not distinguishable from zero at 95% confidence, so read this as no measured difference.`;
|
||||
}
|
||||
|
||||
function completionDeltaClass(m: SurfaceMetric, baseline: SurfaceMetric): string {
|
||||
return m.avg_completion < baseline.avg_completion ? 'text-danger' : 'text-text-secondary';
|
||||
// A skip delta is "worse" above the baseline; a completion delta is "worse"
|
||||
// below it. Neither gets a colour unless it's distinguishable — colouring
|
||||
// noise red is what made the old card misleading.
|
||||
function skipDeltaClass(d: MetricDelta | undefined): string {
|
||||
if (!d?.distinguishable) return 'text-text-secondary opacity-60';
|
||||
return d.delta_pp > 0 ? 'text-danger' : 'text-text-secondary';
|
||||
}
|
||||
|
||||
function completionDeltaClass(d: MetricDelta | undefined): string {
|
||||
if (!d?.distinguishable) return 'text-text-secondary opacity-60';
|
||||
return d.delta_pp < 0 ? 'text-danger' : 'text-text-secondary';
|
||||
}
|
||||
|
||||
// Pick-kind breakdowns are collapsed by default (#1270): with every
|
||||
@@ -384,18 +403,16 @@
|
||||
<td class="py-1 text-right tabular-nums">{m.plays}</td>
|
||||
<td class="py-1 text-right tabular-nums">
|
||||
{pct(m.skip_rate)}
|
||||
{#if baseline}
|
||||
<span class="ml-1 text-xs {skipDeltaClass(m, baseline)}">
|
||||
{deltaPts(m.skip_rate, baseline.skip_rate)}
|
||||
</span>
|
||||
{#if m.skip_delta}
|
||||
<span class="ml-1 text-xs {skipDeltaClass(m.skip_delta)}"
|
||||
title={deltaTitle(m.skip_delta)}>{deltaText(m.skip_delta)}</span>
|
||||
{/if}
|
||||
</td>
|
||||
<td class="py-1 text-right tabular-nums">
|
||||
{pct(m.avg_completion)}
|
||||
{#if baseline}
|
||||
<span class="ml-1 text-xs {completionDeltaClass(m, baseline)}">
|
||||
{deltaPts(m.avg_completion, baseline.avg_completion)}
|
||||
</span>
|
||||
{#if m.completion_delta}
|
||||
<span class="ml-1 text-xs {completionDeltaClass(m.completion_delta)}"
|
||||
title={deltaTitle(m.completion_delta)}>{deltaText(m.completion_delta)}</span>
|
||||
{/if}
|
||||
</td>
|
||||
</tr>
|
||||
@@ -420,18 +437,16 @@
|
||||
<td class="py-1 text-right text-xs tabular-nums">{b.plays}</td>
|
||||
<td class="py-1 text-right text-xs tabular-nums">
|
||||
{pct(b.skip_rate)}
|
||||
{#if baseline}
|
||||
<span class="ml-1 {skipDeltaClass(b, baseline)}">
|
||||
{deltaPts(b.skip_rate, baseline.skip_rate)}
|
||||
</span>
|
||||
{#if b.skip_delta}
|
||||
<span class="ml-1 {skipDeltaClass(b.skip_delta)}"
|
||||
title={deltaTitle(b.skip_delta)}>{deltaText(b.skip_delta)}</span>
|
||||
{/if}
|
||||
</td>
|
||||
<td class="py-1 text-right text-xs tabular-nums">
|
||||
{pct(b.avg_completion)}
|
||||
{#if baseline}
|
||||
<span class="ml-1 {completionDeltaClass(b, baseline)}">
|
||||
{deltaPts(b.avg_completion, baseline.avg_completion)}
|
||||
</span>
|
||||
{#if b.completion_delta}
|
||||
<span class="ml-1 {completionDeltaClass(b.completion_delta)}"
|
||||
title={deltaTitle(b.completion_delta)}>{deltaText(b.completion_delta)}</span>
|
||||
{/if}
|
||||
</td>
|
||||
</tr>
|
||||
@@ -442,6 +457,12 @@
|
||||
</table>
|
||||
</div>
|
||||
{/each}
|
||||
<p class="text-xs text-text-secondary">
|
||||
Deltas compare each surface with your manual plays. A delta marked
|
||||
<span class="opacity-60">≈</span> is smaller than its own margin of error at this
|
||||
sample size — it can't be told apart from no difference, however big it looks.
|
||||
Hover any delta for its range.
|
||||
</p>
|
||||
{:else}
|
||||
<p class="text-sm text-text-secondary">
|
||||
No plays recorded yet. Play something from For You, Discover, or a mix.
|
||||
|
||||
@@ -241,6 +241,71 @@ describe('Settings page — Recommendation metrics card', () => {
|
||||
expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
// #2495: the card used to render a delta computed client-side with no notion
|
||||
// of uncertainty, so a -12 on 59 plays looked exactly as solid as a -6 on 400.
|
||||
// Deltas now arrive from the server with a margin, and an indistinguishable
|
||||
// one is marked with "≈" and dimmed rather than coloured.
|
||||
test('a delta smaller than its margin is marked as indistinguishable', async () => {
|
||||
setupPage();
|
||||
metricsMock.data = {
|
||||
window_days: 30,
|
||||
baseline: metric('manual', 'Manual library plays', { plays: 400, skip_rate: 0.27 }),
|
||||
groups: [
|
||||
{
|
||||
intent: 'discovery',
|
||||
label: 'Discovery mixes',
|
||||
surfaces: [
|
||||
metric('discover', 'Discover', {
|
||||
plays: 59,
|
||||
skip_rate: 0.153,
|
||||
// 13.3pp gap, but the margin at n=59 is wider than the gap.
|
||||
skip_delta: { delta_pp: -13.3, margin_pp: 14.5, distinguishable: false },
|
||||
completion_delta: { delta_pp: 28.0, margin_pp: 12.1, distinguishable: true }
|
||||
})
|
||||
]
|
||||
}
|
||||
]
|
||||
};
|
||||
render(SettingsPage);
|
||||
await waitFor(() => expect(screen.getByText('Discover')).toBeInTheDocument());
|
||||
|
||||
// The indistinguishable skip delta is prefixed and explained on hover.
|
||||
const skip = screen.getByText('≈-13');
|
||||
expect(skip).toBeInTheDocument();
|
||||
expect(skip).toHaveAttribute('title', expect.stringContaining('not distinguishable from zero'));
|
||||
// It must NOT be coloured as a real regression/improvement.
|
||||
expect(skip.className).toContain('opacity-60');
|
||||
|
||||
// The completion delta clears its margin, so it renders plainly.
|
||||
const completion = screen.getByText('+28');
|
||||
expect(completion).toBeInTheDocument();
|
||||
expect(completion.className).not.toContain('opacity-60');
|
||||
|
||||
// And the legend explains the glyph rather than leaving it a mystery.
|
||||
expect(screen.getByText(/smaller than its own margin of error/i)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
// A delta is omitted entirely when the samples are too thin for a margin to
|
||||
// mean anything — the server decides that, and the cell must simply show the
|
||||
// rate rather than a bare "0".
|
||||
test('a surface with no delta shows its rate and nothing else', async () => {
|
||||
setupPage();
|
||||
metricsMock.data = {
|
||||
window_days: 30,
|
||||
baseline: metric('manual', 'Manual library plays', { plays: 400 }),
|
||||
groups: [
|
||||
{
|
||||
intent: 'go_to',
|
||||
label: 'Go-to surfaces',
|
||||
surfaces: [metric('radio', 'Radio', { plays: 1, skip_rate: 0 })]
|
||||
}
|
||||
]
|
||||
};
|
||||
render(SettingsPage);
|
||||
await waitFor(() => expect(screen.getByText('Radio')).toBeInTheDocument());
|
||||
expect(screen.queryByText(/≈/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
test('surfaces without a breakdown render no toggle and no sub-rows', async () => {
|
||||
setupPage();
|
||||
metricsMock.data = {
|
||||
|
||||
Reference in New Issue
Block a user