Compare commits

..
Author SHA1 Message Date
bvandeusen d5aa081157 Merge pull request 'Recommendation metrics: publish the margin of error on every delta' (#122) from dev into main
test-web / test (push) Successful in 52s
test-go / test (push) Successful in 1m10s
test-go / integration (push) Successful in 4m53s
release / Build signed APK (tag releases only) (push) Successful in 3m57s
release / Build + push container image (push) Successful in 1m37s
2026-08-06 21:50:41 -04:00
bvandeusen 304de88c50 test(tuning): assert headers by exact accessible name — #2495
test-web / test (push) Successful in 34s
Third attempt at the same assertion, so I stopped guessing and got vitest
running locally instead: the web lane uses the same ci-go image, so
`docker run ... -w /src/web ci-go:1.26 npx vitest run` works and turns a
5-minute CI round trip into 7 seconds.

/^Skip/ matched the "Skip rate by week" sparkline column as well as
"Skip (last wk)", just as /Plays/ had matched the caption. Exact names
say what the assertion means and cannot drift onto a neighbour.

Verified locally before pushing: svelte-check 0 errors, 110 files /
786 tests pass.
2026-08-06 21:27:22 -04:00
bvandeusen 96abb48086 test(tuning): query the window/last-week headers as column headers — #2495
test-web / test (push) Failing after 33s
getByText(/Plays/) matched my own new caption as well as the header, since
the caption explains which columns cover the window. Query by columnheader
role instead, which is what the assertion actually means.

Also reordered the caption: prepending the clarification turned it into a
run-on that opened mid-explanation before saying what the chart was.
2026-08-06 21:20:18 -04:00
bvandeusen a094d5f8b0 test(metrics): target deltas by test id, not by glyph — #2495
test-web / test (push) Failing after 35s
Two CI failures, both in my own new tests, both informative.

settings: queryByText(/≈/) matched the LEGEND explaining the glyph rather
than a delta, so the "no delta" case failed on the explanation being
present. Delta spans now carry data-testid so a test can name what it
means instead of pattern-matching prose that sits next to it.

tuning: getByText("40%") found two elements. testing-library matches an
element and its OWN direct text nodes, so the skip cell still matches
"40%" despite the trailing play-count span — and discover late-week
completion is also 40%. Genuinely ambiguous now; assert the count.
2026-08-06 21:14:53 -04:00
bvandeusen 481f906059 feat(metrics): publish margin of error on every delta — #2495, #2524
test-web / test (push) Failing after 43s
test-go / test (push) Successful in 1m0s
test-go / integration (push) Successful in 4m58s
The metrics card had one volume threshold doing two jobs.
recMetricsLowVolume = 20 is a DISPLAY floor — below that a skip rate is
anecdote — but the card then presented deltas as though it were also a
DECISION floor. Those differ by an order of magnitude: detecting the ~13pp
differences that matter needs ~133 plays per arm for 80% power at a=0.05.

So Discover's taste-matched (59 plays) and random-unheard (70) both rendered
as full-confidence rows with a bold delta beside them, and that comparison
sits at p ~ 0.06. The card said "signal"; the arithmetic said "maybe". It
produced a recommendation the data didn't support, and any reader with the
same numbers would have made the same call.

Deltas now carry a 95% margin of error and a `distinguishable` flag, computed
server-side so both clients read the same arithmetic instead of each
re-deriving it. Skip rate is a two-proportion difference; completion is
Welch, which needs a variance — hence completion_sqsum in the query. It is
the sum of squares rather than stddev_samp on purpose: raw source rows are
merged into surface families in Go, and sums of squares combine across groups
exactly whereas standard deviations cannot.

recMetricsLowVolume is untouched. "Too thin to show" and "too thin to act on"
are different questions.

Web renders an indistinguishable delta as dimmed and prefixed "≈", with the
range on hover and a legend explaining the glyph. Colour is withheld unless
the delta clears its margin — colouring noise red is what made the old card
misleading. Breakdown rows go through the same path; those are the thinnest
samples on screen and where the old card misled most.

Also fixes the admin trends view, which had the same problem worse: its
"Latest skip"/"Latest completion" columns are one WEEK while the adjacent
Plays column is the whole window. I misread exactly that and briefly
concluded Deep cuts was the worst surface, from ~17 plays in a single week —
over 180 days it is one of the best. Headers now name their period and the
skip cell carries that week's play count.

#2524: resolveArtist now recognises a duplicate-MBID unique violation as the
expected condition it is, matching resolveAlbum. Two rows mapping to one
MusicBrainz artist is a merge candidate, not a fault; without the branch it
logged a generic warning plus a Postgres ERROR line on every scan, which
teaches an operator to ignore database errors.
2026-08-06 21:08:58 -04:00
bvandeusen a99f855e98 Merge pull request 'Missing files: detect them, stop offering them, and follow them when they move' (#121) from dev into main
test-go / test (push) Successful in 56s
test-go / integration (push) Successful in 5m0s
release / Build signed APK (tag releases only) (push) Successful in 4m14s
release / Build + push container image (push) Successful in 15s
2026-08-06 20:40:39 -04:00
bvandeusen 24d330424f feat(library): adopt moved files instead of forking their history — #2528
test-go / test (push) Successful in 52s
test-go / integration (push) Successful in 5m0s
Track identity was file_path, so a file that came back renamed or in a
different directory looked like a deletion plus an unrelated new track:
the old row kept the like and every play_event while a fresh zero-history
row appeared, and nothing connected them. A liked song read as unliked, its
play count reset, and Rediscover could offer it as a discovery — silently.
Renumbering an album was enough, which is what happened to the operator's
copy of Minutes to Midnight.

Adoption re-points the existing row's file_path at the new location and
clears its missing mark. The normal UpsertTrack then conflicts on file_path
and updates THAT row, so the track id survives and likes, plays and
playlist memberships travel with it — and clients see an update rather than
a delete-and-create, so no cache churn either.

Matching is MBID first (identifies the recording, so it survives a
re-encode), then file_size + duration_ms for untagged files. Both
fingerprint components must be non-zero: duration_ms is 0 when ffprobe
failed, and matching 0 against 0 would pair up unrelated broken files.
Only rows already marked missing are eligible — a row whose file is present
elsewhere is a duplicate, not a move, and re-pointing it would corrupt the
copy that still exists. An ambiguous match inserts fresh rather than
adopting one arbitrarily: a fork is recoverable later, a wrong merge isn't.

Scan is now three phases, and the order is the point. Adoption can only
claim a row that is ALREADY marked missing, but reconcile previously ran
after processing — so a rename performed while the server was down surfaced
the deletion and the addition in the same scan, the new path inserted first,
and the fork became permanent. Enumeration is therefore separated from
processing so reconcile can run between them: walk (paths only, no tag
reads or probes) -> reconcile -> process in walk order.

Consequence worth knowing: when reconcile refuses (an absent root, or a
reorganisation exceeding the 25% mark cap) adoption cannot fire and renamed
files fork as before. That's the pre-#2528 behaviour rather than a new
failure, and the warning now names it.

The old outer walk-error branch was unreachable — the callback always
returned nil, so WalkDir never surfaced an error — and verifyRootsPresent is
the real protection, so enumerate counts walk errors instead of pretending
to abort on them.
2026-08-06 15:56:07 -04:00
bvandeusen f6d1cf24f0 feat(library): detect missing files and stop offering them — #2523
test-go / test (push) Successful in 1m0s
test-go / integration (push) Successful in 5m10s
Nothing in Minstrel ever noticed a deleted file. The walk only visits
paths that exist, so a row whose file was gone was never scanned, never
errored, never counted — permanently invisible. classifyEvent ignores
fsnotify removals by design, and the safety-net scan is the same walk, so
it covers additions only. Rows accumulated forever.

Found on the operator's library: a completed scan reported
skipped=24185 errored=0 while the MBID backfill (which opens files by DB
path rather than walking) logged ~40 "no such file or directory" across
three reorganised albums. Those rows also kept their pre-#2499 welded
genre, which is how this surfaced — the version-stamped tag re-read can
only reach files the walk visits.

The harm is not cosmetic. tracks is the candidate universe for
recommendation.sql / discover.sql / system_mixes.sql and nothing filtered
on file existence, so a mix could spend a slot on a track that cannot
stream.

Marks rather than deletes. A missing file is a claim about the filesystem
and the filesystem lies transiently — an unmounted volume, a network
blip, a container that started before its media mount attached. Every
sweep in internal/gc resolves a truth INSIDE the database and is safe to
run blind; this one is not, so no deletion happens here. Three guards
refuse to act on ambiguous evidence: every scan root must resolve to a
non-empty directory, the walk must have seen at least one file, and one
reconcile may newly mark at most 25% of the library. Clearing a mark is
never the dangerous direction, so it runs unconditionally — otherwise a
library that tripped the cap could never recover once the mount returned.

Only a full Scan reconciles. The walk's set of seen paths is the
evidence, and ScanFiles has no basis for concluding anything about files
it did not look at.

Excludes marked tracks from all 13 track-emitting queries (radio x2,
system mixes x5, discover x4, most-played x2), the 6 play-history seed
picks, and the genre browse axis. Deliberately NOT filtered: the shared
ListPlaylistTracks read path, because it also serves user-curated
playlists where hiding a track the user added would be wrong — system
playlists shed orphans on their next daily rebuild instead. History and
the taste profile also keep them: those record the past, and a track you
played 200 times still says something about your taste.

Reconcile tallies land in scan_runs so a disappearance is visible rather
than discovered when a mix comes up short.
2026-08-06 14:34:53 -04:00
35 changed files with 2126 additions and 128 deletions
+46 -3
View File
@@ -39,6 +39,14 @@ type surfaceMetric struct {
SkipRate float64 `json:"skip_rate"` // skips / plays, [0,1] SkipRate float64 `json:"skip_rate"` // skips / plays, [0,1]
AvgCompletion float64 `json:"avg_completion"` // mean completion ratio, [0,1] AvgCompletion float64 `json:"avg_completion"` // mean completion ratio, [0,1]
LowConfidence bool `json:"low_confidence"` // plays < recMetricsLowVolume LowConfidence bool `json:"low_confidence"` // plays < recMetricsLowVolume
// SkipDelta / CompletionDelta are this row's difference from the manual
// baseline WITH its margin of error (#2495). nil on the baseline row
// itself, and whenever the samples are too thin for a margin to mean
// anything. Computed server-side so both clients read the same arithmetic
// instead of each re-deriving it — and so `low_confidence` is no longer
// mistaken for a decision threshold, which it never was.
SkipDelta *metricDelta `json:"skip_delta,omitempty"`
CompletionDelta *metricDelta `json:"completion_delta,omitempty"`
// Breakdown splits the family into the pick-kind populations its // Breakdown splits the family into the pick-kind populations its
// builder stamped (#1249, generalized #1270): For You's taste/fresh, // builder stamped (#1249, generalized #1270): For You's taste/fresh,
// Discover's buckets, tier1-3 for tiered mixes — plus earlier plays // Discover's buckets, tier1-3 for tiered mixes — plus earlier plays
@@ -119,6 +127,10 @@ type familyAccum struct {
// completionSum is avg*count re-expanded, so merging N raw rows // completionSum is avg*count re-expanded, so merging N raw rows
// reduces to a single weighted division at the end. // reduces to a single weighted division at the end.
completionSum float64 completionSum float64
// completionSqSum is the sum of squared completion ratios, which is what
// makes the variance mergeable across raw source rows (#2495). Standard
// deviations cannot be combined; sums of squares add exactly.
completionSqSum float64
} }
func (a *familyAccum) add(row dbq.RecommendationSourceMetricsForUserRow) { func (a *familyAccum) add(row dbq.RecommendationSourceMetricsForUserRow) {
@@ -126,6 +138,7 @@ func (a *familyAccum) add(row dbq.RecommendationSourceMetricsForUserRow) {
a.skips += row.Skips a.skips += row.Skips
a.completionN += row.CompletionN a.completionN += row.CompletionN
a.completionSum += row.AvgCompletion * float64(row.CompletionN) a.completionSum += row.AvgCompletion * float64(row.CompletionN)
a.completionSqSum += row.CompletionSqsum
} }
func (a *familyAccum) metric() surfaceMetric { func (a *familyAccum) metric() surfaceMetric {
@@ -145,6 +158,33 @@ func (a *familyAccum) metric() surfaceMetric {
return m return m
} }
// completionVariance is the sample variance of this family's completion ratios.
func (a *familyAccum) completionVariance() float64 {
return sampleVariance(a.completionSum, a.completionSqSum, a.completionN)
}
// applyDeltas attaches baseline-relative deltas + margins to a metric.
// Split out so every row — parent surfaces and breakdown rows alike — goes
// through the identical arithmetic; a breakdown arm is exactly where the old
// card was most misleading, because those are the thinnest samples on screen.
func applyDeltas(m *surfaceMetric, acc *familyAccum, baseline *familyAccum) {
if baseline == nil || baseline.plays == 0 {
return
}
m.SkipDelta = proportionDelta(
m.SkipRate, acc.plays,
float64(baseline.skips)/float64(baseline.plays), baseline.plays,
)
baseMean := 0.0
if baseline.completionN > 0 {
baseMean = baseline.completionSum / float64(baseline.completionN)
}
m.CompletionDelta = meanDelta(
m.AvgCompletion, acc.completionVariance(), acc.completionN,
baseMean, baseline.completionVariance(), baseline.completionN,
)
}
// handleGetRecommendationMetrics implements GET /api/me/recommendation-metrics. // handleGetRecommendationMetrics implements GET /api/me/recommendation-metrics.
// Bucketed per-surface-family outcomes for the caller over the last `days` // Bucketed per-surface-family outcomes for the caller over the last `days`
// (default 30, capped at 365), grouped by surface intent and anchored by the // (default 30, capped at 365), grouped by surface intent and anchored by the
@@ -214,7 +254,7 @@ func pickKindFamily(parent recFamily, kind string) recFamily {
// Breakdown rows. Attached only when at least one attributed play // Breakdown rows. Attached only when at least one attributed play
// exists — an all-unattributed breakdown would just repeat the parent // exists — an all-unattributed breakdown would just repeat the parent
// row, and families that never stamp (radio, direct plays) stay flat. // row, and families that never stamp (radio, direct plays) stay flat.
func pickKindBreakdown(picks map[string]*familyAccum) []surfaceMetric { func pickKindBreakdown(picks map[string]*familyAccum, baseline *familyAccum) []surfaceMetric {
attributed := int64(0) attributed := int64(0)
for kind, acc := range picks { for kind, acc := range picks {
if kind != "" { if kind != "" {
@@ -227,7 +267,9 @@ func pickKindBreakdown(picks map[string]*familyAccum) []surfaceMetric {
out := make([]surfaceMetric, 0, len(picks)) out := make([]surfaceMetric, 0, len(picks))
for _, kind := range pickKindOrder { for _, kind := range pickKindOrder {
if acc, ok := picks[kind]; ok && acc.plays > 0 { if acc, ok := picks[kind]; ok && acc.plays > 0 {
out = append(out, acc.metric()) m := acc.metric()
applyDeltas(&m, acc, baseline)
out = append(out, m)
} }
} }
return out return out
@@ -287,7 +329,8 @@ func bucketMetricsResponse(
for _, acc := range families { for _, acc := range families {
if acc.fam.intent == g.intent { if acc.fam.intent == g.intent {
m := acc.metric() m := acc.metric()
m.Breakdown = pickKindBreakdown(picks[acc.fam.key]) applyDeltas(&m, acc, baseline)
m.Breakdown = pickKindBreakdown(picks[acc.fam.key], baseline)
group.Surfaces = append(group.Surfaces, m) group.Surfaces = append(group.Surfaces, m)
} }
} }
+117
View File
@@ -0,0 +1,117 @@
package api
import "math"
// Uncertainty on the deltas the recommendation-metrics card shows (#2495).
//
// Why this exists: the card had exactly one volume threshold,
// recMetricsLowVolume = 20, and it was doing two jobs. Twenty plays is enough to
// be worth DISPLAYING — below that a skip rate is anecdote — but it is nowhere
// near enough to ACT on. Detecting the ~13pp differences that actually matter
// needs roughly 133 plays per arm for 80% power at α=0.05.
//
// So Discover's taste-matched (59 plays) and random-unheard (70) both rendered as
// full-confidence rows with a bold delta beside them, and that comparison sits at
// p ≈ 0.06. The card said "signal"; the arithmetic said "maybe". It led directly
// to a recommendation the data didn't support, and any reader with the same
// numbers would have made the same call.
//
// The fix is to publish the margin of error next to the delta and flag when the
// delta is smaller than it — i.e. not distinguishable from zero. Computed here,
// server-side, so both clients agree rather than each re-deriving it.
//
// recMetricsLowVolume stays exactly as it was. This is a second, independent
// signal, not a replacement: "too thin to show" and "too thin to act on" are
// different questions and deserve different answers.
// deltaZ is the two-sided 95% normal critical value. Normal rather than
// Student's t: at the sample sizes where a delta is worth acting on (n in the
// hundreds) the difference is immaterial, and a household dashboard does not
// need a t-table.
const deltaZ = 1.96
// metricDelta is a difference from the baseline, with its uncertainty.
//
// Both figures are in PERCENTAGE POINTS, matching how the card reads them out —
// a skip rate of 0.153 against a baseline of 0.270 is "-11.7", not "-0.117".
type metricDelta struct {
// DeltaPP is surface minus baseline. Negative skip is better; negative
// completion is worse. The client owns that colouring.
DeltaPP float64 `json:"delta_pp"`
// MarginPP is the 95% margin of error on DeltaPP. Read the delta as
// DeltaPP ± MarginPP.
MarginPP float64 `json:"margin_pp"`
// Distinguishable reports |DeltaPP| >= MarginPP: the interval excludes
// zero, so the difference is worth reading as a difference. When false the
// number may be pure noise no matter how large it looks.
Distinguishable bool `json:"distinguishable"`
}
// proportionDelta compares two rates (skips/plays) as a two-proportion
// difference. Returns nil when either sample is empty, or when either rate is
// degenerate (0 or 1) — a rate with no observed variation has an SE of 0 on its
// side, which would report a spuriously narrow margin rather than an honest one.
func proportionDelta(rate1 float64, n1 int64, rate2 float64, n2 int64) *metricDelta {
if n1 <= 0 || n2 <= 0 {
return nil
}
v1 := rate1 * (1 - rate1) / float64(n1)
v2 := rate2 * (1 - rate2) / float64(n2)
se := math.Sqrt(v1 + v2)
if se <= 0 {
// Both rates are 0 or both are 1. The delta is exactly zero and the
// margin is meaningless; reporting nothing is more honest than
// reporting certainty.
return nil
}
return newDelta((rate1-rate2)*100, deltaZ*se*100)
}
// meanDelta compares two means (average completion ratio) using Welch's
// standard error, which does not assume equal variances between the two groups.
//
// Note the margins here are wider than intuition suggests, and that is correct:
// completion is strongly bimodal — a play is either abandoned early (≈0.05) or
// finished (≈1.0), with little in between — so its standard deviation is large
// (~0.4) even though the mean looks stable.
func meanDelta(mean1 float64, variance1 float64, n1 int64, mean2 float64, variance2 float64, n2 int64) *metricDelta {
// Two observations minimum per side: a sample variance needs n-1 > 0.
if n1 < 2 || n2 < 2 {
return nil
}
se := math.Sqrt(variance1/float64(n1) + variance2/float64(n2))
if se <= 0 || math.IsNaN(se) || math.IsInf(se, 0) {
return nil
}
return newDelta((mean1-mean2)*100, deltaZ*se*100)
}
func newDelta(deltaPP, marginPP float64) *metricDelta {
return &metricDelta{
DeltaPP: deltaPP,
MarginPP: marginPP,
// >= rather than >: a delta exactly equal to its margin sits on the
// boundary, and calling the boundary "distinguishable" is the
// conventional reading of a 95% interval that just excludes zero.
Distinguishable: math.Abs(deltaPP) >= marginPP,
}
}
// sampleVariance recovers the sample variance from the aggregates the SQL
// returns. sum is mean×n rather than a selected column, which keeps the query to
// one extra expression.
//
// The subtraction can go very slightly negative through floating-point
// cancellation when every observation is identical, so the result is clamped —
// a negative variance would produce NaN downstream.
func sampleVariance(sum, sqSum float64, n int64) float64 {
if n < 2 {
return 0
}
nf := float64(n)
v := (sqSum - (sum * sum / nf)) / (nf - 1)
if v < 0 {
return 0
}
return v
}
+152
View File
@@ -0,0 +1,152 @@
package api
import (
"math"
"testing"
)
func TestProportionDelta_ReproducesTheDiscoverCase(t *testing.T) {
// The comparison that motivated #2495: Discover taste-matched (59 plays,
// 15.3% skip) vs random-unheard (70 plays, 28.6%). A 13.3pp gap that the old
// card rendered as a confident coloured number, sitting at p ≈ 0.06.
d := proportionDelta(0.153, 59, 0.286, 70)
if d == nil {
t.Fatal("expected a delta for two real samples")
}
if math.Abs(d.DeltaPP-(-13.3)) > 0.1 {
t.Errorf("DeltaPP = %.2f, want ≈ -13.3", d.DeltaPP)
}
// This is the assertion the whole task exists for: at these sample sizes the
// margin swallows the difference.
if d.Distinguishable {
t.Errorf("13.3pp on n=59/70 reported as distinguishable (margin %.2f) — "+
"this is exactly the false confidence #2495 set out to remove", d.MarginPP)
}
if d.MarginPP <= 13.3 {
t.Errorf("MarginPP = %.2f, expected it to exceed the 13.3pp delta", d.MarginPP)
}
}
// Same effect size, ~10x the volume: now it is real. Proves the flag tracks
// sample size rather than just the size of the gap.
func TestProportionDelta_SameGapBecomesDistinguishableWithVolume(t *testing.T) {
d := proportionDelta(0.153, 600, 0.286, 700)
if d == nil {
t.Fatal("expected a delta")
}
if !d.Distinguishable {
t.Errorf("13.3pp on n=600/700 should be distinguishable (margin %.2f)", d.MarginPP)
}
}
func TestProportionDelta_SignAndDirection(t *testing.T) {
// Surface skips MORE than baseline -> positive delta (worse for skip rate).
worse := proportionDelta(0.40, 500, 0.25, 500)
if worse == nil || worse.DeltaPP <= 0 {
t.Fatalf("expected a positive delta, got %+v", worse)
}
better := proportionDelta(0.10, 500, 0.25, 500)
if better == nil || better.DeltaPP >= 0 {
t.Fatalf("expected a negative delta, got %+v", better)
}
}
func TestProportionDelta_EmptySamples(t *testing.T) {
if d := proportionDelta(0.2, 0, 0.3, 100); d != nil {
t.Errorf("n1=0 produced a delta: %+v", d)
}
if d := proportionDelta(0.2, 100, 0.3, 0); d != nil {
t.Errorf("n2=0 produced a delta: %+v", d)
}
}
// Two degenerate rates have zero standard error, which would report a margin of
// 0 and therefore "distinguishable" for a delta of exactly 0. Reporting nothing
// is the honest answer.
func TestProportionDelta_DegenerateRates(t *testing.T) {
if d := proportionDelta(0, 50, 0, 50); d != nil {
t.Errorf("both rates 0 produced a delta: %+v", d)
}
if d := proportionDelta(1, 50, 1, 50); d != nil {
t.Errorf("both rates 1 produced a delta: %+v", d)
}
// One degenerate side is still informative — the other side carries variance.
if d := proportionDelta(0, 200, 0.3, 200); d == nil {
t.Error("one degenerate rate should still yield a delta")
}
}
func TestMeanDelta(t *testing.T) {
// Completion is bimodal, so ~0.16 variance (sd ≈ 0.4) is realistic.
const v = 0.16
thin := meanDelta(0.82, v, 59, 0.54, v, 70)
if thin == nil {
t.Fatal("expected a delta")
}
if math.Abs(thin.DeltaPP-28.0) > 0.1 {
t.Errorf("DeltaPP = %.2f, want ≈ 28.0", thin.DeltaPP)
}
// 28pp is large enough to survive even a wide margin at this n.
if !thin.Distinguishable {
t.Errorf("28pp on n=59/70 with sd 0.4 should be distinguishable (margin %.2f)", thin.MarginPP)
}
// A small completion gap at the same volume should not be.
small := meanDelta(0.56, v, 59, 0.54, v, 70)
if small == nil {
t.Fatal("expected a delta")
}
if small.Distinguishable {
t.Errorf("2pp on n=59/70 reported as distinguishable (margin %.2f)", small.MarginPP)
}
}
// A sample variance needs at least two observations per side.
func TestMeanDelta_NeedsTwoObservations(t *testing.T) {
if d := meanDelta(0.8, 0.1, 1, 0.5, 0.1, 100); d != nil {
t.Errorf("n1=1 produced a delta: %+v", d)
}
if d := meanDelta(0.8, 0.1, 100, 0.5, 0.1, 1); d != nil {
t.Errorf("n2=1 produced a delta: %+v", d)
}
}
func TestMeanDelta_ZeroVarianceBothSides(t *testing.T) {
if d := meanDelta(0.8, 0, 50, 0.5, 0, 50); d != nil {
t.Errorf("zero variance on both sides produced a delta: %+v", d)
}
}
func TestSampleVariance(t *testing.T) {
// Observations 0, 1: mean 0.5, sample variance 0.5.
if got := sampleVariance(1.0, 1.0, 2); math.Abs(got-0.5) > 1e-9 {
t.Errorf("sampleVariance = %v, want 0.5", got)
}
// Identical observations -> zero variance, and must not go negative through
// floating-point cancellation.
if got := sampleVariance(4.0, 4.0, 4); got != 0 {
t.Errorf("identical observations gave variance %v, want 0", got)
}
if got := sampleVariance(0, 0, 1); got != 0 {
t.Errorf("n=1 gave variance %v, want 0", got)
}
}
// Clamping matters: a negative variance would become NaN in the square root and
// propagate into the JSON as a null-ish number.
func TestSampleVariance_NeverNegative(t *testing.T) {
// sqSum slightly below sum²/n, as cancellation can produce.
if got := sampleVariance(10.0, 24.999999999, 4); got < 0 {
t.Errorf("variance went negative: %v", got)
}
}
func TestNewDelta_BoundaryCountsAsDistinguishable(t *testing.T) {
d := newDelta(5.0, 5.0)
if !d.Distinguishable {
t.Error("a delta exactly equal to its margin should count as distinguishable")
}
d = newDelta(4.999, 5.0)
if d.Distinguishable {
t.Error("a delta just inside its margin should not count as distinguishable")
}
}
+17
View File
@@ -18,6 +18,7 @@ WHERE EXISTS (
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.album_id = albums.id WHERE tracks.album_id = albums.id
AND tracks.missing_since IS NULL
AND trim(g.genre) = trim($1::text) AND trim(g.genre) = trim($1::text)
) )
` `
@@ -97,6 +98,7 @@ WHERE EXISTS (
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.album_id = albums.id WHERE tracks.album_id = albums.id
AND tracks.missing_since IS NULL
AND trim(g.genre) = trim($1::text) AND trim(g.genre) = trim($1::text)
) )
ORDER BY albums.sort_title, albums.id ORDER BY albums.sort_title, albums.id
@@ -219,6 +221,7 @@ SELECT DISTINCT trim(g.genre) AS genre
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.album_id = $1 AND trim(g.genre) <> '' WHERE tracks.album_id = $1 AND trim(g.genre) <> ''
AND tracks.missing_since IS NULL
ORDER BY trim(g.genre) ORDER BY trim(g.genre)
` `
@@ -251,6 +254,7 @@ SELECT DISTINCT trim(g.genre) AS genre
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.artist_id = $1 AND trim(g.genre) <> '' WHERE tracks.artist_id = $1 AND trim(g.genre) <> ''
AND tracks.missing_since IS NULL
ORDER BY trim(g.genre) ORDER BY trim(g.genre)
` `
@@ -278,10 +282,12 @@ func (q *Queries) ListGenresForArtist(ctx context.Context, artistID pgtype.UUID)
} }
const listGenresWithCount = `-- name: ListGenresWithCount :many const listGenresWithCount = `-- name: ListGenresWithCount :many
SELECT trim(g.genre) AS genre, COUNT(DISTINCT tracks.id)::bigint AS track_count SELECT trim(g.genre) AS genre, COUNT(DISTINCT tracks.id)::bigint AS track_count
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE trim(g.genre) <> '' WHERE trim(g.genre) <> ''
AND tracks.missing_since IS NULL
GROUP BY trim(g.genre) GROUP BY trim(g.genre)
ORDER BY track_count DESC, trim(g.genre) ORDER BY track_count DESC, trim(g.genre)
` `
@@ -291,6 +297,17 @@ type ListGenresWithCountRow struct {
TrackCount int64 TrackCount int64
} }
// Every query in this file filters `tracks.missing_since IS NULL` (#2523).
// A row whose file has vanished keeps its genre forever — the scanner walks the
// filesystem, so it never revisits a path that no longer exists — which is how
// pre-#2499 welded genres survived a full re-scan and kept showing in the index.
// Browsing is a way of finding something to play, so a track that cannot play
// should not shape it.
//
// Year queries below join albums only and are deliberately left alone: an album
// is still a real release even if some of its tracks are gone. An album whose
// EVERY track is missing will linger on the year axis; that's a narrower case,
// tracked with the rest of the cleanup work.
// Genre browse index (#367). // Genre browse index (#367).
// //
// Genres live inline on tracks.genre as a delimited string, so this splits on // Genres live inline on tracks.genre as a delimited string, so this splits on
+8 -4
View File
@@ -15,7 +15,8 @@ const listCrossUserLikedTracksForDiscover = `-- name: ListCrossUserLikedTracksFo
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM general_likes gl FROM general_likes gl
JOIN tracks t ON t.id = gl.track_id JOIN tracks t ON t.id = gl.track_id
WHERE gl.user_id != $1 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND gl.user_id != $1
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
@@ -95,7 +96,8 @@ dormant_artists AS (
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN dormant_artists da ON da.id = t.artist_id JOIN dormant_artists da ON da.id = t.artist_id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.track_id = t.id AND pe.track_id = t.id
@@ -159,7 +161,8 @@ func (q *Queries) ListDormantArtistTracksForDiscover(ctx context.Context, arg Li
const listRandomUnheardTracksForDiscover = `-- name: ListRandomUnheardTracksForDiscover :many const listRandomUnheardTracksForDiscover = `-- name: ListRandomUnheardTracksForDiscover :many
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.track_id = t.id AND pe.track_id = t.id
@@ -217,7 +220,8 @@ SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true
JOIN taste_profile_tags nt ON nt.user_id = $1 AND trim(g_split.g) = nt.tag JOIN taste_profile_tags nt ON nt.user_id = $1 AND trim(g_split.g) = nt.tag
WHERE nt.weight > 0 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND nt.weight > 0
AND trim(g_split.g) <> '' AND trim(g_split.g) <> ''
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
+2 -1
View File
@@ -261,7 +261,7 @@ func (q *Queries) InsertSkipEvent(ctx context.Context, arg InsertSkipEventParams
} }
const listRecentSessionTracks = `-- name: ListRecentSessionTracks :many const listRecentSessionTracks = `-- name: ListRecentSessionTracks :many
SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version FROM tracks t SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since FROM tracks t
JOIN play_events pe ON pe.track_id = t.id JOIN play_events pe ON pe.track_id = t.id
WHERE pe.session_id = $1 WHERE pe.session_id = $1
AND pe.started_at < $2 AND pe.started_at < $2
@@ -306,6 +306,7 @@ func (q *Queries) ListRecentSessionTracks(ctx context.Context, arg ListRecentSes
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
+2 -1
View File
@@ -14,7 +14,7 @@ import (
const listUserHistory = `-- name: ListUserHistory :many const listUserHistory = `-- name: ListUserHistory :many
SELECT pe.id AS event_id, SELECT pe.id AS event_id,
pe.started_at, pe.started_at,
t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since,
albums.title AS album_title, albums.title AS album_title,
artists.name AS artist_name artists.name AS artist_name
FROM play_events pe FROM play_events pe
@@ -80,6 +80,7 @@ func (q *Queries) ListUserHistory(ctx context.Context, arg ListUserHistoryParams
&i.Track.TagSource, &i.Track.TagSource,
&i.Track.TagSourcesVersion, &i.Track.TagSourcesVersion,
&i.Track.TagReadVersion, &i.Track.TagReadVersion,
&i.Track.MissingSince,
&i.AlbumTitle, &i.AlbumTitle,
&i.ArtistName, &i.ArtistName,
); err != nil { ); err != nil {
+2 -1
View File
@@ -259,7 +259,7 @@ func (q *Queries) ListLikedTrackIDs(ctx context.Context, userID pgtype.UUID) ([]
} }
const listLikedTrackRows = `-- name: ListLikedTrackRows :many const listLikedTrackRows = `-- name: ListLikedTrackRows :many
SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version FROM tracks t SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since FROM tracks t
JOIN general_likes l ON l.track_id = t.id JOIN general_likes l ON l.track_id = t.id
WHERE l.user_id = $1 WHERE l.user_id = $1
ORDER BY l.liked_at DESC ORDER BY l.liked_at DESC
@@ -300,6 +300,7 @@ func (q *Queries) ListLikedTrackRows(ctx context.Context, arg ListLikedTrackRows
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
+1
View File
@@ -643,6 +643,7 @@ type Track struct {
TagSource *string TagSource *string
TagSourcesVersion int32 TagSourcesVersion int32
TagReadVersion int16 TagReadVersion int16
MissingSince pgtype.Timestamptz
} }
type TrackSimilarity struct { type TrackSimilarity struct {
+13 -6
View File
@@ -208,7 +208,7 @@ WITH plays AS (
WHERE user_id = $2 AND was_skipped = false WHERE user_id = $2 AND was_skipped = false
GROUP BY track_id GROUP BY track_id
) )
SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since,
albums.title AS album_title, albums.title AS album_title,
artists.name AS artist_name artists.name AS artist_name
FROM plays p FROM plays p
@@ -216,6 +216,7 @@ JOIN tracks t ON t.id = p.track_id
JOIN albums ON albums.id = t.album_id JOIN albums ON albums.id = t.album_id
JOIN artists ON artists.id = t.artist_id JOIN artists ON artists.id = t.artist_id
WHERE t.artist_id = $1 WHERE t.artist_id = $1
AND t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = $2 AND q.track_id = t.id WHERE q.user_id = $2 AND q.track_id = t.id
@@ -268,6 +269,7 @@ func (q *Queries) ListMostPlayedTracksForArtist(ctx context.Context, arg ListMos
&i.Track.TagSource, &i.Track.TagSource,
&i.Track.TagSourcesVersion, &i.Track.TagSourcesVersion,
&i.Track.TagReadVersion, &i.Track.TagReadVersion,
&i.Track.MissingSince,
&i.AlbumTitle, &i.AlbumTitle,
&i.ArtistName, &i.ArtistName,
); err != nil { ); err != nil {
@@ -288,14 +290,15 @@ WITH plays AS (
WHERE user_id = $1 AND was_skipped = false WHERE user_id = $1 AND was_skipped = false
GROUP BY track_id GROUP BY track_id
) )
SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since,
albums.title AS album_title, albums.title AS album_title,
artists.name AS artist_name artists.name AS artist_name
FROM plays p FROM plays p
JOIN tracks t ON t.id = p.track_id JOIN tracks t ON t.id = p.track_id
JOIN albums ON albums.id = t.album_id JOIN albums ON albums.id = t.album_id
JOIN artists ON artists.id = t.artist_id JOIN artists ON artists.id = t.artist_id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = $1 AND q.track_id = t.id WHERE q.user_id = $1 AND q.track_id = t.id
) )
@@ -350,6 +353,7 @@ func (q *Queries) ListMostPlayedTracksForUser(ctx context.Context, arg ListMostP
&i.Track.TagSource, &i.Track.TagSource,
&i.Track.TagSourcesVersion, &i.Track.TagSourcesVersion,
&i.Track.TagReadVersion, &i.Track.TagReadVersion,
&i.Track.MissingSince,
&i.AlbumTitle, &i.AlbumTitle,
&i.ArtistName, &i.ArtistName,
); err != nil { ); err != nil {
@@ -687,7 +691,7 @@ func (q *Queries) ListRediscoverArtistsForUser(ctx context.Context, arg ListRedi
const loadRadioCandidates = `-- name: LoadRadioCandidates :many const loadRadioCandidates = `-- name: LoadRadioCandidates :many
SELECT SELECT
t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since,
(l.user_id IS NOT NULL)::bool AS is_liked, (l.user_id IS NOT NULL)::bool AS is_liked,
pe.last_played_at::timestamptz AS last_played_at, pe.last_played_at::timestamptz AS last_played_at,
pe.play_count, pe.play_count,
@@ -705,6 +709,7 @@ LEFT JOIN LATERAL (
WHERE user_id = $1 AND track_id = t.id WHERE user_id = $1 AND track_id = t.id
) pe ON true ) pe ON true
WHERE t.id <> $2 WHERE t.id <> $2
AND t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM play_events SELECT 1 FROM play_events
WHERE user_id = $1 AND track_id = t.id WHERE user_id = $1 AND track_id = t.id
@@ -766,6 +771,7 @@ func (q *Queries) LoadRadioCandidates(ctx context.Context, arg LoadRadioCandidat
&i.Track.TagSource, &i.Track.TagSource,
&i.Track.TagSourcesVersion, &i.Track.TagSourcesVersion,
&i.Track.TagReadVersion, &i.Track.TagReadVersion,
&i.Track.MissingSince,
&i.IsLiked, &i.IsLiked,
&i.LastPlayedAt, &i.LastPlayedAt,
&i.PlayCount, &i.PlayCount,
@@ -898,7 +904,7 @@ random_fill AS (
LIMIT $9 LIMIT $9
) )
SELECT SELECT
t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since,
(l.user_id IS NOT NULL)::bool AS is_liked, (l.user_id IS NOT NULL)::bool AS is_liked,
pe.last_played_at::timestamptz AS last_played_at, pe.last_played_at::timestamptz AS last_played_at,
pe.play_count, pe.play_count,
@@ -914,7 +920,7 @@ FROM (
UNION ALL SELECT track_id, sim_score FROM coplay_artists UNION ALL SELECT track_id, sim_score FROM coplay_artists
UNION ALL SELECT track_id, sim_score FROM random_fill UNION ALL SELECT track_id, sim_score FROM random_fill
) u ) u
JOIN tracks t ON t.id = u.track_id JOIN tracks t ON t.id = u.track_id AND t.missing_since IS NULL -- #2523: never offer a file that is gone
JOIN albums al ON al.id = t.album_id JOIN albums al ON al.id = t.album_id
LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
@@ -1008,6 +1014,7 @@ func (q *Queries) LoadRadioCandidatesV2(ctx context.Context, arg LoadRadioCandid
&i.Track.TagSource, &i.Track.TagSource,
&i.Track.TagSourcesVersion, &i.Track.TagSourcesVersion,
&i.Track.TagReadVersion, &i.Track.TagReadVersion,
&i.Track.MissingSince,
&i.IsLiked, &i.IsLiked,
&i.LastPlayedAt, &i.LastPlayedAt,
&i.PlayCount, &i.PlayCount,
+17 -7
View File
@@ -18,7 +18,8 @@ SELECT
count(*)::bigint AS plays, count(*)::bigint AS plays,
count(*) FILTER (WHERE pe.was_skipped)::bigint AS skips, count(*) FILTER (WHERE pe.was_skipped)::bigint AS skips,
count(pe.completion_ratio)::bigint AS completion_n, count(pe.completion_ratio)::bigint AS completion_n,
COALESCE(avg(pe.completion_ratio), 0)::float8 AS avg_completion COALESCE(avg(pe.completion_ratio), 0)::float8 AS avg_completion,
COALESCE(sum(pe.completion_ratio * pe.completion_ratio), 0)::float8 AS completion_sqsum
FROM play_events pe FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - ($2::float8 * INTERVAL '1 day') AND pe.started_at > now() - ($2::float8 * INTERVAL '1 day')
@@ -32,18 +33,26 @@ type RecommendationSourceMetricsForUserParams struct {
} }
type RecommendationSourceMetricsForUserRow struct { type RecommendationSourceMetricsForUserRow struct {
Source *string Source *string
PickKind *string PickKind *string
Plays int64 Plays int64
Skips int64 Skips int64
CompletionN int64 CompletionN int64
AvgCompletion float64 AvgCompletion float64
CompletionSqsum float64
} }
// $1 user_id, $2 window_days. plays/skips are counts; avg_completion is the // $1 user_id, $2 window_days. plays/skips are counts; avg_completion is the
// mean completion ratio over the completion_n plays that recorded one. // mean completion ratio over the completion_n plays that recorded one.
// pick_kind splits For You plays into taste/fresh/unattributed (#1249); // pick_kind splits For You plays into taste/fresh/unattributed (#1249);
// it is NULL for every other source, so those still group to one row. // it is NULL for every other source, so those still group to one row.
//
// completion_sqsum carries the sum of SQUARED completion ratios so the Go
// handler can compute a variance — needed for the margin of error on a
// completion delta (#2495). It is the sum rather than `stddev_samp` on purpose:
// raw source rows get merged into surface families in Go, and sums of squares
// add across groups exactly, whereas standard deviations cannot be combined
// without them. Variance = (sqsum - sum²/n) / (n-1), with sum = avg × n.
func (q *Queries) RecommendationSourceMetricsForUser(ctx context.Context, arg RecommendationSourceMetricsForUserParams) ([]RecommendationSourceMetricsForUserRow, error) { func (q *Queries) RecommendationSourceMetricsForUser(ctx context.Context, arg RecommendationSourceMetricsForUserParams) ([]RecommendationSourceMetricsForUserRow, error) {
rows, err := q.db.Query(ctx, recommendationSourceMetricsForUser, arg.UserID, arg.Column2) rows, err := q.db.Query(ctx, recommendationSourceMetricsForUser, arg.UserID, arg.Column2)
if err != nil { if err != nil {
@@ -60,6 +69,7 @@ func (q *Queries) RecommendationSourceMetricsForUser(ctx context.Context, arg Re
&i.Skips, &i.Skips,
&i.CompletionN, &i.CompletionN,
&i.AvgCompletion, &i.AvgCompletion,
&i.CompletionSqsum,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
+10 -5
View File
@@ -39,7 +39,8 @@ SELECT t.id, t.album_id, t.artist_id
JOIN affinity_artists aa ON aa.artist_id = t.artist_id JOIN affinity_artists aa ON aa.artist_id = t.artist_id
LEFT JOIN play_counts pc ON pc.track_id = t.id LEFT JOIN play_counts pc ON pc.track_id = t.id
LEFT JOIN skip_counts sc ON sc.track_id = t.id LEFT JOIN skip_counts sc ON sc.track_id = t.id
WHERE COALESCE(pc.c, 0) <= 2 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND COALESCE(pc.c, 0) <= 2
AND COALESCE(sc.c, 0) < 2 AND COALESCE(sc.c, 0) < 2
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
@@ -124,7 +125,8 @@ SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
FROM tracks t FROM tracks t
JOIN albums al ON al.id = t.album_id JOIN albums al ON al.id = t.album_id
JOIN albums_tiered alt ON alt.album_id = al.id JOIN albums_tiered alt ON alt.album_id = al.id
WHERE alt.tier IS NOT NULL WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND alt.tier IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id) AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id)
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
@@ -225,7 +227,8 @@ albums_tiered AS (
SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
FROM tracks t FROM tracks t
JOIN albums_tiered alt ON alt.album_id = t.album_id JOIN albums_tiered alt ON alt.album_id = t.album_id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = $1 AND q.track_id = t.id WHERE q.user_id = $1 AND q.track_id = t.id
) )
@@ -303,7 +306,8 @@ WITH windowed AS (
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN windowed w ON w.track_id = t.id JOIN windowed w ON w.track_id = t.id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = $1 AND q.track_id = t.id WHERE q.user_id = $1 AND q.track_id = t.id
) )
@@ -367,7 +371,8 @@ WITH stats AS (
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN stats s ON s.track_id = t.id JOIN stats s ON s.track_id = t.id
WHERE s.c >= 3 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND s.c >= 3
AND s.last_at <= now() - interval '30 days' AND s.last_at <= now() - interval '30 days'
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
+14 -6
View File
@@ -189,6 +189,7 @@ func (q *Queries) GetSystemPlaylistRun(ctx context.Context, userID pgtype.UUID)
const listActiveUsersForSystemPlaylists = `-- name: ListActiveUsersForSystemPlaylists :many const listActiveUsersForSystemPlaylists = `-- name: ListActiveUsersForSystemPlaylists :many
SELECT u.id FROM users u SELECT u.id FROM users u
WHERE EXISTS ( WHERE EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
@@ -197,6 +198,13 @@ SELECT u.id FROM users u
) )
` `
// Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a
// seed or For-You candidate has to be something that can actually play. Note
// this only affects newly GENERATED playlists — already-stored system
// playlists keep their rows until the next daily rebuild, which is why the
// shared ListPlaylistTracks read path is deliberately left unfiltered (it
// also serves user-curated playlists, where hiding a track the user added
// themselves would be wrong).
// M7 #352 slice 2: system-generated playlist queries. // M7 #352 slice 2: system-generated playlist queries.
// Active = had a play in the last 7 days. The cron iterates this list. // Active = had a play in the last 7 days. The cron iterates this list.
func (q *Queries) ListActiveUsersForSystemPlaylists(ctx context.Context) ([]pgtype.UUID, error) { func (q *Queries) ListActiveUsersForSystemPlaylists(ctx context.Context) ([]pgtype.UUID, error) {
@@ -298,7 +306,7 @@ recent7 AS (
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count, COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
0 AS tier 0 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '7 days' AND pe.started_at > now() - INTERVAL '7 days'
AND t.artist_id IS NOT NULL AND t.artist_id IS NOT NULL
@@ -309,7 +317,7 @@ recent30 AS (
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count, COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
1 AS tier 1 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '30 days' AND pe.started_at > now() - INTERVAL '30 days'
AND t.artist_id IS NOT NULL AND t.artist_id IS NOT NULL
@@ -320,7 +328,7 @@ alltime AS (
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count, COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
2 AS tier 2 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND t.artist_id IS NOT NULL AND t.artist_id IS NOT NULL
GROUP BY t.artist_id GROUP BY t.artist_id
@@ -432,7 +440,7 @@ const pickTopPlayedTrackForArtistByUser = `-- name: PickTopPlayedTrackForArtistB
SELECT COALESCE( SELECT COALESCE(
(SELECT t.id (SELECT t.id
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND t.artist_id = $2 AND t.artist_id = $2
AND pe.started_at > now() - INTERVAL '7 days' AND pe.started_at > now() - INTERVAL '7 days'
@@ -472,7 +480,7 @@ const pickTopPlayedTracksForUser = `-- name: PickTopPlayedTracksForUser :many
WITH recent AS ( WITH recent AS (
SELECT t.id, COUNT(*) AS c, 0 AS tier SELECT t.id, COUNT(*) AS c, 0 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '30 days' AND pe.started_at > now() - INTERVAL '30 days'
AND pe.was_skipped = false AND pe.was_skipped = false
@@ -481,7 +489,7 @@ WITH recent AS (
alltime AS ( alltime AS (
SELECT t.id, COUNT(*) AS c, 1 AS tier SELECT t.id, COUNT(*) AS c, 1 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.was_skipped = false AND pe.was_skipped = false
GROUP BY t.id GROUP BY t.id
+207 -8
View File
@@ -11,6 +11,52 @@ import (
"github.com/jackc/pgx/v5/pgtype" "github.com/jackc/pgx/v5/pgtype"
) )
const adoptTrackPath = `-- name: AdoptTrackPath :execrows
UPDATE tracks
SET file_path = $1,
missing_since = NULL
WHERE id = $2
AND missing_since IS NOT NULL
`
type AdoptTrackPathParams struct {
FilePath string
ID pgtype.UUID
}
// Re-points a missing row at the path its file turned up on, and clears the
// mark. The caller's normal UpsertTrack then conflicts on file_path and updates
// THIS row in place, so the track id survives and its likes, play history and
// playlist memberships come with it.
//
// `missing_since IS NOT NULL` again, this time as a race guard: two files can't
// both adopt the same row, and :execrows reports 0 to whichever loses.
func (q *Queries) AdoptTrackPath(ctx context.Context, arg AdoptTrackPathParams) (int64, error) {
result, err := q.db.Exec(ctx, adoptTrackPath, arg.FilePath, arg.ID)
if err != nil {
return 0, err
}
return result.RowsAffected(), nil
}
const clearTracksMissing = `-- name: ClearTracksMissing :execrows
UPDATE tracks
SET missing_since = NULL
WHERE id = ANY($1::uuid[])
AND missing_since IS NOT NULL
`
// Clears the mark on rows whose file is back. Runs independently of the mtime
// skip check, so a file that reappears unchanged is un-marked even though the
// scanner skips re-reading its tags.
func (q *Queries) ClearTracksMissing(ctx context.Context, ids []pgtype.UUID) (int64, error) {
result, err := q.db.Exec(ctx, clearTracksMissing, ids)
if err != nil {
return 0, err
}
return result.RowsAffected(), nil
}
const countTracksByAlbum = `-- name: CountTracksByAlbum :one const countTracksByAlbum = `-- name: CountTracksByAlbum :one
SELECT count(*) FROM tracks WHERE album_id = $1 SELECT count(*) FROM tracks WHERE album_id = $1
` `
@@ -89,8 +135,95 @@ func (q *Queries) DeleteTrack(ctx context.Context, id pgtype.UUID) (DeleteTrackR
return i, err return i, err
} }
const findMissingTrackByFingerprint = `-- name: FindMissingTrackByFingerprint :many
SELECT id, file_path FROM tracks
WHERE missing_since IS NOT NULL
AND file_size = $1
AND duration_ms = $2
LIMIT 2
`
type FindMissingTrackByFingerprintParams struct {
FileSize int64
DurationMs int32
}
type FindMissingTrackByFingerprintRow struct {
ID pgtype.UUID
FilePath string
}
// Move detection fallback for files with no MBID (#2528). Exact byte size AND
// exact decoded duration is a strong pair: a plain move or rename preserves
// both, while a re-encode changes at least one — and a re-encode genuinely is a
// different file, so failing to match there is correct rather than a gap.
//
// Same missing-only constraint and same LIMIT 2 rationale as the MBID variant.
func (q *Queries) FindMissingTrackByFingerprint(ctx context.Context, arg FindMissingTrackByFingerprintParams) ([]FindMissingTrackByFingerprintRow, error) {
rows, err := q.db.Query(ctx, findMissingTrackByFingerprint, arg.FileSize, arg.DurationMs)
if err != nil {
return nil, err
}
defer rows.Close()
var items []FindMissingTrackByFingerprintRow
for rows.Next() {
var i FindMissingTrackByFingerprintRow
if err := rows.Scan(&i.ID, &i.FilePath); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
const findMissingTrackByMbid = `-- name: FindMissingTrackByMbid :many
SELECT id, file_path FROM tracks
WHERE missing_since IS NOT NULL
AND mbid IS NOT NULL
AND mbid = $1::text
LIMIT 2
`
type FindMissingTrackByMbidRow struct {
ID pgtype.UUID
FilePath string
}
// Move detection, strongest signal (#2528). A file that turned up at a new path
// carrying a recording MBID we already have on a MISSING row is that recording,
// moved — not a new track.
//
// `missing_since IS NOT NULL` is the safety constraint, not an optimisation: a
// row whose file is present elsewhere on disk is a DUPLICATE, and re-pointing
// its file_path would corrupt the copy that still exists.
//
// LIMIT 2 because the caller only needs to know "exactly one" vs "more than
// one" — an ambiguous match must not be adopted arbitrarily.
func (q *Queries) FindMissingTrackByMbid(ctx context.Context, mbid string) ([]FindMissingTrackByMbidRow, error) {
rows, err := q.db.Query(ctx, findMissingTrackByMbid, mbid)
if err != nil {
return nil, err
}
defer rows.Close()
var items []FindMissingTrackByMbidRow
for rows.Next() {
var i FindMissingTrackByMbidRow
if err := rows.Scan(&i.ID, &i.FilePath); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
const getTrackByID = `-- name: GetTrackByID :one const getTrackByID = `-- name: GetTrackByID :one
SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version FROM tracks WHERE id = $1 SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since FROM tracks WHERE id = $1
` `
func (q *Queries) GetTrackByID(ctx context.Context, id pgtype.UUID) (Track, error) { func (q *Queries) GetTrackByID(ctx context.Context, id pgtype.UUID) (Track, error) {
@@ -115,12 +248,13 @@ func (q *Queries) GetTrackByID(ctx context.Context, id pgtype.UUID) (Track, erro
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
) )
return i, err return i, err
} }
const getTrackByPath = `-- name: GetTrackByPath :one const getTrackByPath = `-- name: GetTrackByPath :one
SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version FROM tracks WHERE file_path = $1 SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since FROM tracks WHERE file_path = $1
` `
func (q *Queries) GetTrackByPath(ctx context.Context, filePath string) (Track, error) { func (q *Queries) GetTrackByPath(ctx context.Context, filePath string) (Track, error) {
@@ -145,12 +279,13 @@ func (q *Queries) GetTrackByPath(ctx context.Context, filePath string) (Track, e
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
) )
return i, err return i, err
} }
const getTracksByIDs = `-- name: GetTracksByIDs :many const getTracksByIDs = `-- name: GetTracksByIDs :many
SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version FROM tracks WHERE id = ANY($1::uuid[]) SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since FROM tracks WHERE id = ANY($1::uuid[])
` `
// Batched lookup used by /api/library/sync to hydrate upsert payloads // Batched lookup used by /api/library/sync to hydrate upsert payloads
@@ -183,6 +318,7 @@ func (q *Queries) GetTracksByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
@@ -195,7 +331,7 @@ func (q *Queries) GetTracksByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([
} }
const listArtistTracksForUser = `-- name: ListArtistTracksForUser :many const listArtistTracksForUser = `-- name: ListArtistTracksForUser :many
SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since,
albums.title AS album_title, albums.title AS album_title,
artists.name AS artist_name artists.name AS artist_name
FROM tracks t FROM tracks t
@@ -254,6 +390,7 @@ func (q *Queries) ListArtistTracksForUser(ctx context.Context, arg ListArtistTra
&i.Track.TagSource, &i.Track.TagSource,
&i.Track.TagSourcesVersion, &i.Track.TagSourcesVersion,
&i.Track.TagReadVersion, &i.Track.TagReadVersion,
&i.Track.MissingSince,
&i.AlbumTitle, &i.AlbumTitle,
&i.ArtistName, &i.ArtistName,
); err != nil { ); err != nil {
@@ -268,7 +405,7 @@ func (q *Queries) ListArtistTracksForUser(ctx context.Context, arg ListArtistTra
} }
const listRandomTracksForUser = `-- name: ListRandomTracksForUser :many const listRandomTracksForUser = `-- name: ListRandomTracksForUser :many
SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since,
albums.title AS album_title, albums.title AS album_title,
artists.name AS artist_name artists.name AS artist_name
FROM tracks t FROM tracks t
@@ -324,6 +461,7 @@ func (q *Queries) ListRandomTracksForUser(ctx context.Context, arg ListRandomTra
&i.Track.TagSource, &i.Track.TagSource,
&i.Track.TagSourcesVersion, &i.Track.TagSourcesVersion,
&i.Track.TagReadVersion, &i.Track.TagReadVersion,
&i.Track.MissingSince,
&i.AlbumTitle, &i.AlbumTitle,
&i.ArtistName, &i.ArtistName,
); err != nil { ); err != nil {
@@ -337,8 +475,43 @@ func (q *Queries) ListRandomTracksForUser(ctx context.Context, arg ListRandomTra
return items, nil return items, nil
} }
const listTrackPathsForReconcile = `-- name: ListTrackPathsForReconcile :many
SELECT id, file_path, missing_since FROM tracks
`
type ListTrackPathsForReconcileRow struct {
ID pgtype.UUID
FilePath string
MissingSince pgtype.Timestamptz
}
// Every row's path + current missing mark, for the scanner's reconcile pass
// (#2523). Deliberately unfiltered and unpaged: reconcile has to compare the
// WHOLE table against what the walk saw, and a filtered subset would let rows
// outside it drift forever. Three narrow columns keep it cheap even on a
// library of a few hundred thousand tracks.
func (q *Queries) ListTrackPathsForReconcile(ctx context.Context) ([]ListTrackPathsForReconcileRow, error) {
rows, err := q.db.Query(ctx, listTrackPathsForReconcile)
if err != nil {
return nil, err
}
defer rows.Close()
var items []ListTrackPathsForReconcileRow
for rows.Next() {
var i ListTrackPathsForReconcileRow
if err := rows.Scan(&i.ID, &i.FilePath, &i.MissingSince); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
const listTracksByAlbum = `-- name: ListTracksByAlbum :many const listTracksByAlbum = `-- name: ListTracksByAlbum :many
SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version FROM tracks SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since FROM tracks
WHERE album_id = $1 WHERE album_id = $1
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
@@ -383,6 +556,7 @@ func (q *Queries) ListTracksByAlbum(ctx context.Context, arg ListTracksByAlbumPa
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
@@ -429,8 +603,31 @@ func (q *Queries) ListTracksMissingMbidWithPath(ctx context.Context, limit int32
return items, nil return items, nil
} }
const markTracksMissing = `-- name: MarkTracksMissing :execrows
UPDATE tracks
SET missing_since = now()
WHERE id = ANY($1::uuid[])
AND missing_since IS NULL
`
// Marks rows whose file the walk did not see. `missing_since IS NULL` in the
// predicate makes this idempotent: a row already marked keeps its ORIGINAL
// timestamp, so "how long has it been gone" survives repeated scans. Losing
// that would make any age-based cleanup policy meaningless.
//
// updated_at is deliberately NOT touched. It tracks content changes and gates
// the scanner's mtime skip; moving it here would make a returning file look
// newer than its own mtime and stop its tags being re-read.
func (q *Queries) MarkTracksMissing(ctx context.Context, ids []pgtype.UUID) (int64, error) {
result, err := q.db.Exec(ctx, markTracksMissing, ids)
if err != nil {
return 0, err
}
return result.RowsAffected(), nil
}
const searchTracks = `-- name: SearchTracks :many const searchTracks = `-- name: SearchTracks :many
SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version FROM tracks SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since FROM tracks
WHERE title ILIKE '%' || $1::text || '%' WHERE title ILIKE '%' || $1::text || '%'
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
@@ -482,6 +679,7 @@ func (q *Queries) SearchTracks(ctx context.Context, arg SearchTracksParams) ([]T
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
@@ -533,7 +731,7 @@ ON CONFLICT (file_path) DO UPDATE SET
-- next scan can short-circuit them again (#2499). -- next scan can short-circuit them again (#2499).
tag_read_version = EXCLUDED.tag_read_version, tag_read_version = EXCLUDED.tag_read_version,
updated_at = now() updated_at = now()
RETURNING id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version RETURNING id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since
` `
type UpsertTrackParams struct { type UpsertTrackParams struct {
@@ -589,6 +787,7 @@ func (q *Queries) UpsertTrack(ctx context.Context, arg UpsertTrackParams) (Track
&i.TagSource, &i.TagSource,
&i.TagSourcesVersion, &i.TagSourcesVersion,
&i.TagReadVersion, &i.TagReadVersion,
&i.MissingSince,
) )
return i, err return i, err
} }
@@ -0,0 +1,4 @@
DROP INDEX IF EXISTS tracks_missing_since_idx;
ALTER TABLE tracks
DROP COLUMN missing_since;
@@ -0,0 +1,25 @@
-- Marks a track whose file the scanner could no longer find (#2523).
--
-- NULL means present. A timestamp means the file was absent as of that scan,
-- and is the point from which "how long has this been gone" is measured — which
-- is what a later cleanup pass needs in order to require a grace period rather
-- than deleting on a single missed stat.
--
-- Deliberately a nullable timestamp rather than a boolean: "missing" is not a
-- state we want to act on immediately, and the age is the only thing that makes
-- an automated deletion safe to reason about.
--
-- No default and no backfill. Existing rows start NULL (present) and the next
-- full scan sets the mark where it belongs — a migration cannot check the
-- filesystem, and guessing here would mark the whole library on a server whose
-- media volume happens to be detached at upgrade time.
ALTER TABLE tracks
ADD COLUMN missing_since timestamptz;
-- Partial index: the only query that filters on this column positively is the
-- admin "what's missing" list, which is a small set. Playback and browse
-- queries filter `missing_since IS NULL`, which matches nearly every row and is
-- better served by a sequential scan than an index lookup.
CREATE INDEX tracks_missing_since_idx
ON tracks (missing_since)
WHERE missing_since IS NOT NULL;
+17
View File
@@ -1,3 +1,15 @@
-- Every query in this file filters `tracks.missing_since IS NULL` (#2523).
-- A row whose file has vanished keeps its genre forever — the scanner walks the
-- filesystem, so it never revisits a path that no longer exists — which is how
-- pre-#2499 welded genres survived a full re-scan and kept showing in the index.
-- Browsing is a way of finding something to play, so a track that cannot play
-- should not shape it.
--
-- Year queries below join albums only and are deliberately left alone: an album
-- is still a real release even if some of its tracks are gone. An album whose
-- EVERY track is missing will linger on the year axis; that's a narrower case,
-- tracked with the rest of the cleanup work.
-- name: ListGenresWithCount :many -- name: ListGenresWithCount :many
-- Genre browse index (#367). -- Genre browse index (#367).
-- --
@@ -22,6 +34,7 @@ SELECT trim(g.genre) AS genre, COUNT(DISTINCT tracks.id)::bigint AS track_count
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE trim(g.genre) <> '' WHERE trim(g.genre) <> ''
AND tracks.missing_since IS NULL
GROUP BY trim(g.genre) GROUP BY trim(g.genre)
-- Ordered by the expression, not the output alias: `ORDER BY genre` is -- Ordered by the expression, not the output alias: `ORDER BY genre` is
-- ambiguous between the alias and tracks.genre, and sqlc rejects it. -- ambiguous between the alias and tracks.genre, and sqlc rejects it.
@@ -41,6 +54,7 @@ WHERE EXISTS (
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.album_id = albums.id WHERE tracks.album_id = albums.id
AND tracks.missing_since IS NULL
AND trim(g.genre) = trim(sqlc.arg(genre)::text) AND trim(g.genre) = trim(sqlc.arg(genre)::text)
) )
ORDER BY albums.sort_title, albums.id ORDER BY albums.sort_title, albums.id
@@ -56,6 +70,7 @@ WHERE EXISTS (
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.album_id = albums.id WHERE tracks.album_id = albums.id
AND tracks.missing_since IS NULL
AND trim(g.genre) = trim(sqlc.arg(genre)::text) AND trim(g.genre) = trim(sqlc.arg(genre)::text)
); );
@@ -96,6 +111,7 @@ SELECT DISTINCT trim(g.genre) AS genre
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.album_id = $1 AND trim(g.genre) <> '' WHERE tracks.album_id = $1 AND trim(g.genre) <> ''
AND tracks.missing_since IS NULL
ORDER BY trim(g.genre); ORDER BY trim(g.genre);
-- name: ListGenresForArtist :many -- name: ListGenresForArtist :many
@@ -106,4 +122,5 @@ SELECT DISTINCT trim(g.genre) AS genre
FROM tracks FROM tracks
JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
WHERE tracks.artist_id = $1 AND trim(g.genre) <> '' WHERE tracks.artist_id = $1 AND trim(g.genre) <> ''
AND tracks.missing_since IS NULL
ORDER BY trim(g.genre); ORDER BY trim(g.genre);
+8 -4
View File
@@ -29,7 +29,8 @@ dormant_artists AS (
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN dormant_artists da ON da.id = t.artist_id JOIN dormant_artists da ON da.id = t.artist_id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.track_id = t.id AND pe.track_id = t.id
@@ -60,7 +61,8 @@ SELECT t.id, t.album_id, t.artist_id
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM general_likes gl FROM general_likes gl
JOIN tracks t ON t.id = gl.track_id JOIN tracks t ON t.id = gl.track_id
WHERE gl.user_id != $1 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND gl.user_id != $1
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
@@ -86,7 +88,8 @@ SELECT t.id, t.album_id, t.artist_id
-- $1 = user_id, $2 = date string for md5 ordering. -- $1 = user_id, $2 = date string for md5 ordering.
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.track_id = t.id AND pe.track_id = t.id
@@ -117,7 +120,8 @@ SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true
JOIN taste_profile_tags nt ON nt.user_id = $1 AND trim(g_split.g) = nt.tag JOIN taste_profile_tags nt ON nt.user_id = $1 AND trim(g_split.g) = nt.tag
WHERE nt.weight > 0 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND nt.weight > 0
AND trim(g_split.g) <> '' AND trim(g_split.g) <> ''
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM play_events pe SELECT 1 FROM play_events pe
+5 -2
View File
@@ -24,6 +24,7 @@ LEFT JOIN LATERAL (
WHERE user_id = $1 AND track_id = t.id WHERE user_id = $1 AND track_id = t.id
) pe ON true ) pe ON true
WHERE t.id <> $2 WHERE t.id <> $2
AND t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM play_events SELECT 1 FROM play_events
WHERE user_id = $1 AND track_id = t.id WHERE user_id = $1 AND track_id = t.id
@@ -177,7 +178,7 @@ FROM (
UNION ALL SELECT track_id, sim_score FROM coplay_artists UNION ALL SELECT track_id, sim_score FROM coplay_artists
UNION ALL SELECT track_id, sim_score FROM random_fill UNION ALL SELECT track_id, sim_score FROM random_fill
) u ) u
JOIN tracks t ON t.id = u.track_id JOIN tracks t ON t.id = u.track_id AND t.missing_since IS NULL -- #2523: never offer a file that is gone
JOIN albums al ON al.id = t.album_id JOIN albums al ON al.id = t.album_id
LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id
LEFT JOIN LATERAL ( LEFT JOIN LATERAL (
@@ -382,7 +383,8 @@ FROM plays p
JOIN tracks t ON t.id = p.track_id JOIN tracks t ON t.id = p.track_id
JOIN albums ON albums.id = t.album_id JOIN albums ON albums.id = t.album_id
JOIN artists ON artists.id = t.artist_id JOIN artists ON artists.id = t.artist_id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = $1 AND q.track_id = t.id WHERE q.user_id = $1 AND q.track_id = t.id
) )
@@ -408,6 +410,7 @@ JOIN tracks t ON t.id = p.track_id
JOIN albums ON albums.id = t.album_id JOIN albums ON albums.id = t.album_id
JOIN artists ON artists.id = t.artist_id JOIN artists ON artists.id = t.artist_id
WHERE t.artist_id = sqlc.arg(artist_id) WHERE t.artist_id = sqlc.arg(artist_id)
AND t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = sqlc.arg(user_id) AND q.track_id = t.id WHERE q.user_id = sqlc.arg(user_id) AND q.track_id = t.id
@@ -44,13 +44,21 @@ ORDER BY 1, 2;
-- mean completion ratio over the completion_n plays that recorded one. -- mean completion ratio over the completion_n plays that recorded one.
-- pick_kind splits For You plays into taste/fresh/unattributed (#1249); -- pick_kind splits For You plays into taste/fresh/unattributed (#1249);
-- it is NULL for every other source, so those still group to one row. -- it is NULL for every other source, so those still group to one row.
--
-- completion_sqsum carries the sum of SQUARED completion ratios so the Go
-- handler can compute a variance — needed for the margin of error on a
-- completion delta (#2495). It is the sum rather than `stddev_samp` on purpose:
-- raw source rows get merged into surface families in Go, and sums of squares
-- add across groups exactly, whereas standard deviations cannot be combined
-- without them. Variance = (sqsum - sum²/n) / (n-1), with sum = avg × n.
SELECT SELECT
pe.source, pe.source,
pe.pick_kind, pe.pick_kind,
count(*)::bigint AS plays, count(*)::bigint AS plays,
count(*) FILTER (WHERE pe.was_skipped)::bigint AS skips, count(*) FILTER (WHERE pe.was_skipped)::bigint AS skips,
count(pe.completion_ratio)::bigint AS completion_n, count(pe.completion_ratio)::bigint AS completion_n,
COALESCE(avg(pe.completion_ratio), 0)::float8 AS avg_completion COALESCE(avg(pe.completion_ratio), 0)::float8 AS avg_completion,
COALESCE(sum(pe.completion_ratio * pe.completion_ratio), 0)::float8 AS completion_sqsum
FROM play_events pe FROM play_events pe
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - ($2::float8 * INTERVAL '1 day') AND pe.started_at > now() - ($2::float8 * INTERVAL '1 day')
+10 -5
View File
@@ -40,7 +40,8 @@ SELECT t.id, t.album_id, t.artist_id
JOIN affinity_artists aa ON aa.artist_id = t.artist_id JOIN affinity_artists aa ON aa.artist_id = t.artist_id
LEFT JOIN play_counts pc ON pc.track_id = t.id LEFT JOIN play_counts pc ON pc.track_id = t.id
LEFT JOIN skip_counts sc ON sc.track_id = t.id LEFT JOIN skip_counts sc ON sc.track_id = t.id
WHERE COALESCE(pc.c, 0) <= 2 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND COALESCE(pc.c, 0) <= 2
AND COALESCE(sc.c, 0) < 2 AND COALESCE(sc.c, 0) < 2
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
@@ -70,7 +71,8 @@ WITH stats AS (
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN stats s ON s.track_id = t.id JOIN stats s ON s.track_id = t.id
WHERE s.c >= 3 WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND s.c >= 3
AND s.last_at <= now() - interval '30 days' AND s.last_at <= now() - interval '30 days'
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
@@ -149,7 +151,8 @@ albums_tiered AS (
SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
FROM tracks t FROM tracks t
JOIN albums_tiered alt ON alt.album_id = t.album_id JOIN albums_tiered alt ON alt.album_id = t.album_id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = $1 AND q.track_id = t.id WHERE q.user_id = $1 AND q.track_id = t.id
) )
@@ -187,7 +190,8 @@ WITH windowed AS (
SELECT t.id, t.album_id, t.artist_id SELECT t.id, t.album_id, t.artist_id
FROM tracks t FROM tracks t
JOIN windowed w ON w.track_id = t.id JOIN windowed w ON w.track_id = t.id
WHERE NOT EXISTS ( WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
WHERE q.user_id = $1 AND q.track_id = t.id WHERE q.user_id = $1 AND q.track_id = t.id
) )
@@ -240,7 +244,8 @@ SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
FROM tracks t FROM tracks t
JOIN albums al ON al.id = t.album_id JOIN albums al ON al.id = t.album_id
JOIN albums_tiered alt ON alt.album_id = al.id JOIN albums_tiered alt ON alt.album_id = al.id
WHERE alt.tier IS NOT NULL WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
AND alt.tier IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id) AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id)
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM lidarr_quarantine q SELECT 1 FROM lidarr_quarantine q
+14 -6
View File
@@ -1,3 +1,11 @@
-- Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a
-- seed or For-You candidate has to be something that can actually play. Note
-- this only affects newly GENERATED playlists — already-stored system
-- playlists keep their rows until the next daily rebuild, which is why the
-- shared ListPlaylistTracks read path is deliberately left unfiltered (it
-- also serves user-curated playlists, where hiding a track the user added
-- themselves would be wrong).
-- M7 #352 slice 2: system-generated playlist queries. -- M7 #352 slice 2: system-generated playlist queries.
-- name: ListActiveUsersForSystemPlaylists :many -- name: ListActiveUsersForSystemPlaylists :many
@@ -72,7 +80,7 @@ recent7 AS (
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count, COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
0 AS tier 0 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '7 days' AND pe.started_at > now() - INTERVAL '7 days'
AND t.artist_id IS NOT NULL AND t.artist_id IS NOT NULL
@@ -83,7 +91,7 @@ recent30 AS (
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count, COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
1 AS tier 1 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '30 days' AND pe.started_at > now() - INTERVAL '30 days'
AND t.artist_id IS NOT NULL AND t.artist_id IS NOT NULL
@@ -94,7 +102,7 @@ alltime AS (
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count, COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
2 AS tier 2 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND t.artist_id IS NOT NULL AND t.artist_id IS NOT NULL
GROUP BY t.artist_id GROUP BY t.artist_id
@@ -139,7 +147,7 @@ SELECT c.artist_id,
WITH recent AS ( WITH recent AS (
SELECT t.id, COUNT(*) AS c, 0 AS tier SELECT t.id, COUNT(*) AS c, 0 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '30 days' AND pe.started_at > now() - INTERVAL '30 days'
AND pe.was_skipped = false AND pe.was_skipped = false
@@ -148,7 +156,7 @@ WITH recent AS (
alltime AS ( alltime AS (
SELECT t.id, COUNT(*) AS c, 1 AS tier SELECT t.id, COUNT(*) AS c, 1 AS tier
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND pe.was_skipped = false AND pe.was_skipped = false
GROUP BY t.id GROUP BY t.id
@@ -181,7 +189,7 @@ SELECT id
SELECT COALESCE( SELECT COALESCE(
(SELECT t.id (SELECT t.id
FROM play_events pe FROM play_events pe
JOIN tracks t ON t.id = pe.track_id JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1 WHERE pe.user_id = $1
AND t.artist_id = $2 AND t.artist_id = $2
AND pe.started_at > now() - INTERVAL '7 days' AND pe.started_at > now() - INTERVAL '7 days'
+75
View File
@@ -137,3 +137,78 @@ RETURNING id, album_id, artist_id, file_path, mbid;
-- Batched lookup used by /api/library/sync to hydrate upsert payloads -- Batched lookup used by /api/library/sync to hydrate upsert payloads
-- (#357). Mirror of GetArtistsByIDs. -- (#357). Mirror of GetArtistsByIDs.
SELECT * FROM tracks WHERE id = ANY($1::uuid[]); SELECT * FROM tracks WHERE id = ANY($1::uuid[]);
-- name: FindMissingTrackByMbid :many
-- Move detection, strongest signal (#2528). A file that turned up at a new path
-- carrying a recording MBID we already have on a MISSING row is that recording,
-- moved — not a new track.
--
-- `missing_since IS NOT NULL` is the safety constraint, not an optimisation: a
-- row whose file is present elsewhere on disk is a DUPLICATE, and re-pointing
-- its file_path would corrupt the copy that still exists.
--
-- LIMIT 2 because the caller only needs to know "exactly one" vs "more than
-- one" — an ambiguous match must not be adopted arbitrarily.
SELECT id, file_path FROM tracks
WHERE missing_since IS NOT NULL
AND mbid IS NOT NULL
AND mbid = sqlc.arg(mbid)::text
LIMIT 2;
-- name: FindMissingTrackByFingerprint :many
-- Move detection fallback for files with no MBID (#2528). Exact byte size AND
-- exact decoded duration is a strong pair: a plain move or rename preserves
-- both, while a re-encode changes at least one — and a re-encode genuinely is a
-- different file, so failing to match there is correct rather than a gap.
--
-- Same missing-only constraint and same LIMIT 2 rationale as the MBID variant.
SELECT id, file_path FROM tracks
WHERE missing_since IS NOT NULL
AND file_size = sqlc.arg(file_size)
AND duration_ms = sqlc.arg(duration_ms)
LIMIT 2;
-- name: AdoptTrackPath :execrows
-- Re-points a missing row at the path its file turned up on, and clears the
-- mark. The caller's normal UpsertTrack then conflicts on file_path and updates
-- THIS row in place, so the track id survives and its likes, play history and
-- playlist memberships come with it.
--
-- `missing_since IS NOT NULL` again, this time as a race guard: two files can't
-- both adopt the same row, and :execrows reports 0 to whichever loses.
UPDATE tracks
SET file_path = sqlc.arg(file_path),
missing_since = NULL
WHERE id = sqlc.arg(id)
AND missing_since IS NOT NULL;
-- name: ListTrackPathsForReconcile :many
-- Every row's path + current missing mark, for the scanner's reconcile pass
-- (#2523). Deliberately unfiltered and unpaged: reconcile has to compare the
-- WHOLE table against what the walk saw, and a filtered subset would let rows
-- outside it drift forever. Three narrow columns keep it cheap even on a
-- library of a few hundred thousand tracks.
SELECT id, file_path, missing_since FROM tracks;
-- name: MarkTracksMissing :execrows
-- Marks rows whose file the walk did not see. `missing_since IS NULL` in the
-- predicate makes this idempotent: a row already marked keeps its ORIGINAL
-- timestamp, so "how long has it been gone" survives repeated scans. Losing
-- that would make any age-based cleanup policy meaningless.
--
-- updated_at is deliberately NOT touched. It tracks content changes and gates
-- the scanner's mtime skip; moving it here would make a returning file look
-- newer than its own mtime and stop its tags being re-read.
UPDATE tracks
SET missing_since = now()
WHERE id = ANY(sqlc.arg(ids)::uuid[])
AND missing_since IS NULL;
-- name: ClearTracksMissing :execrows
-- Clears the mark on rows whose file is back. Runs independently of the mtime
-- skip check, so a file that reappears unchanged is un-marked even though the
-- scanner skips re-reading its tags.
UPDATE tracks
SET missing_since = NULL
WHERE id = ANY(sqlc.arg(ids)::uuid[])
AND missing_since IS NOT NULL;
+151
View File
@@ -0,0 +1,151 @@
package library
import (
"context"
"github.com/jackc/pgx/v5/pgtype"
"git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq"
syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync"
)
// Move detection (#2528).
//
// Track identity is file_path: UpsertTrack conflicts on it, and the reconcile
// pass in reconcile.go clears a missing mark when the walk sees that same path
// again. So a file that comes back exactly where it was restores cleanly, but a
// file that comes back RENAMED or in a different directory looked, to the
// scanner, like a deletion plus an unrelated new track:
//
// - the old row stayed marked missing, holding the like and every play_event
// - a fresh row appeared with no history
// - nothing connected them
//
// A liked song read as unliked after a retag, its play count reset to zero, and
// Rediscover could offer it as a discovery. All silently. Renumbering an album
// was enough to do it — which is exactly what happened on the operator's copy of
// Minutes to Midnight.
//
// The fix adopts the existing row rather than inserting: re-point its file_path
// at the new location and clear the mark. The caller's normal UpsertTrack then
// conflicts on file_path and updates THAT row, so the track id survives and
// likes, plays and playlist memberships travel with it. Clients see an update
// rather than a delete-and-create, so no cache churn either.
//
// Only rows already marked missing are eligible. A row whose file is present
// elsewhere is a duplicate, not a move, and re-pointing it would corrupt the
// copy that still exists. That constraint is what makes this safe, and the
// marking added in #2523 is what makes it expressible.
// trackAdopter is the slice of dbq.Queries move detection needs, narrowed so the
// match/ambiguity logic can be tested against a fake.
type trackAdopter interface {
FindMissingTrackByMbid(ctx context.Context, mbid string) ([]dbq.FindMissingTrackByMbidRow, error)
FindMissingTrackByFingerprint(ctx context.Context, arg dbq.FindMissingTrackByFingerprintParams) ([]dbq.FindMissingTrackByFingerprintRow, error)
AdoptTrackPath(ctx context.Context, arg dbq.AdoptTrackPathParams) (int64, error)
}
// adoptMovedTrack looks for a missing row that is the same recording as the file
// at newPath and re-points it there. Reports whether a row was adopted.
//
// Never returns an error: failing to detect a move is a missed optimisation, not
// a broken scan. The caller carries on and inserts a fresh row, which is the
// pre-#2528 behaviour.
func (s *Scanner) adoptMovedTrack(
ctx context.Context, q trackAdopter, newPath string,
fileSize int64, durationMs int32, recordingMBID string,
) bool {
// MBID first. It identifies the recording rather than the bytes, so it
// survives a re-encode that the fingerprint cannot.
if recordingMBID != "" {
rows, err := q.FindMissingTrackByMbid(ctx, recordingMBID)
if err != nil {
s.logger.Warn("library scan: move lookup by mbid failed",
"path", newPath, "err", err)
} else if c, ok := s.uniqueMatch(rowsFromMbid(rows), newPath, "mbid"); ok {
return s.adopt(ctx, q, c, newPath, "mbid")
}
}
// Fingerprint fallback for untagged files. Both components must be real:
// duration_ms is 0 when ffprobe failed, and matching 0 against 0 would pair
// up unrelated broken files.
if fileSize > 0 && durationMs > 0 {
rows, err := q.FindMissingTrackByFingerprint(ctx, dbq.FindMissingTrackByFingerprintParams{
FileSize: fileSize,
DurationMs: durationMs,
})
if err != nil {
s.logger.Warn("library scan: move lookup by fingerprint failed",
"path", newPath, "err", err)
} else if c, ok := s.uniqueMatch(rowsFromFingerprint(rows), newPath, "fingerprint"); ok {
return s.adopt(ctx, q, c, newPath, "fingerprint")
}
}
return false
}
// candidate is the shared shape of both lookups, so uniqueMatch is written once.
type candidate struct {
id pgtype.UUID
filePath string
}
func rowsFromMbid(rows []dbq.FindMissingTrackByMbidRow) []candidate {
out := make([]candidate, 0, len(rows))
for _, r := range rows {
out = append(out, candidate{id: r.ID, filePath: r.FilePath})
}
return out
}
func rowsFromFingerprint(rows []dbq.FindMissingTrackByFingerprintRow) []candidate {
out := make([]candidate, 0, len(rows))
for _, r := range rows {
out = append(out, candidate{id: r.ID, filePath: r.FilePath})
}
return out
}
// uniqueMatch requires exactly one candidate. Adopting an arbitrary row out of
// several would attach this file's future history to a coin flip, which is worse
// than starting a fresh row — a fork is recoverable later, a wrong merge isn't.
// Libraries with genuine duplicates hit this, so it's logged rather than silent.
func (s *Scanner) uniqueMatch(
cands []candidate, newPath, via string,
) (candidate, bool) {
switch len(cands) {
case 0:
return candidate{}, false
case 1:
return cands[0], true
default:
s.logger.Info("library scan: ambiguous move match, inserting a new track instead",
"path", newPath, "via", via, "candidates", len(cands))
return candidate{}, false
}
}
func (s *Scanner) adopt(
ctx context.Context, q trackAdopter, c candidate, newPath, via string,
) bool {
n, err := q.AdoptTrackPath(ctx, dbq.AdoptTrackPathParams{ID: c.id, FilePath: newPath})
if err != nil {
// A unique violation on file_path means something else claimed this path
// first. Fall through to a normal insert rather than failing the file.
s.logger.Warn("library scan: adopting moved track failed",
"path", newPath, "via", via, "err", err)
return false
}
if n == 0 {
// Lost the race: another file adopted this row between lookup and
// update, so its mark was already cleared.
return false
}
// Logged with both paths: this is the operator's only window onto a
// reorganisation being understood as a move rather than a new track.
s.logger.Info("library scan: track moved, history preserved",
"from", c.filePath, "to", newPath, "via", via, "track_id", syncpkg.FormatUUID(c.id))
return true
}
+295
View File
@@ -0,0 +1,295 @@
package library
import (
"context"
"errors"
"testing"
"github.com/jackc/pgx/v5/pgtype"
"git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq"
)
type fakeAdopter struct {
byMbid []dbq.FindMissingTrackByMbidRow
byFingerprint []dbq.FindMissingTrackByFingerprintRow
mbidErr error
fingerprintErr error
adoptErr error
adoptRows int64
mbidQueried []string
fingerprintQueried []dbq.FindMissingTrackByFingerprintParams
adopted []dbq.AdoptTrackPathParams
}
func (f *fakeAdopter) FindMissingTrackByMbid(_ context.Context, mbid string) ([]dbq.FindMissingTrackByMbidRow, error) {
f.mbidQueried = append(f.mbidQueried, mbid)
return f.byMbid, f.mbidErr
}
func (f *fakeAdopter) FindMissingTrackByFingerprint(
_ context.Context, arg dbq.FindMissingTrackByFingerprintParams,
) ([]dbq.FindMissingTrackByFingerprintRow, error) {
f.fingerprintQueried = append(f.fingerprintQueried, arg)
return f.byFingerprint, f.fingerprintErr
}
func (f *fakeAdopter) AdoptTrackPath(_ context.Context, arg dbq.AdoptTrackPathParams) (int64, error) {
f.adopted = append(f.adopted, arg)
if f.adoptErr != nil {
return 0, f.adoptErr
}
return f.adoptRows, nil
}
// The narrowed interface must not drift from the real queries.
var _ trackAdopter = (*dbq.Queries)(nil)
func mbidRow(n byte, path string) dbq.FindMissingTrackByMbidRow {
return dbq.FindMissingTrackByMbidRow{ID: testUUID(n), FilePath: path}
}
func fpRow(n byte, path string) dbq.FindMissingTrackByFingerprintRow {
return dbq.FindMissingTrackByFingerprintRow{ID: testUUID(n), FilePath: path}
}
const (
oldPath = "/music/Linkin Park/Minutes to Midnight/02 - Bleed It Out.mp3"
newPath = "/music/Linkin Park/Minutes to Midnight/04 - Bleed It Out.mp3"
)
func TestAdoptMovedTrack_MatchesByMbid(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{byMbid: []dbq.FindMissingTrackByMbidRow{mbidRow(7, oldPath)}, adoptRows: 1}
if !s.adoptMovedTrack(context.Background(), q, newPath, 5_000_000, 200_000, "rec-mbid") {
t.Fatal("expected the moved track to be adopted")
}
if len(q.adopted) != 1 {
t.Fatalf("adopted %d rows, want 1", len(q.adopted))
}
if q.adopted[0].ID != testUUID(7) {
t.Errorf("adopted the wrong row: %v", q.adopted[0].ID)
}
if q.adopted[0].FilePath != newPath {
t.Errorf("adopted FilePath = %q, want %q", q.adopted[0].FilePath, newPath)
}
// MBID matched, so the weaker signal should not have been consulted.
if len(q.fingerprintQueried) != 0 {
t.Errorf("queried the fingerprint despite an MBID match")
}
}
func TestAdoptMovedTrack_FallsBackToFingerprint(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{byFingerprint: []dbq.FindMissingTrackByFingerprintRow{fpRow(3, oldPath)}, adoptRows: 1}
// No MBID: an untagged file, which is exactly what the fallback is for.
if !s.adoptMovedTrack(context.Background(), q, newPath, 4_200_000, 187_000, "") {
t.Fatal("expected adoption via fingerprint")
}
if len(q.mbidQueried) != 0 {
t.Errorf("queried by MBID with no MBID available")
}
if len(q.fingerprintQueried) != 1 {
t.Fatalf("fingerprint queried %d times, want 1", len(q.fingerprintQueried))
}
got := q.fingerprintQueried[0]
if got.FileSize != 4_200_000 || got.DurationMs != 187_000 {
t.Errorf("fingerprint = %+v, want size 4200000 duration 187000", got)
}
if len(q.adopted) != 1 || q.adopted[0].ID != testUUID(3) {
t.Errorf("adopted = %+v, want row 3", q.adopted)
}
}
// Two missing rows carrying the same recording MBID means real duplicates.
// Adopting one arbitrarily would attach this file's future history to a coin
// flip, so it must insert fresh instead.
func TestAdoptMovedTrack_RefusesAmbiguousMbidMatch(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{byMbid: []dbq.FindMissingTrackByMbidRow{
mbidRow(1, "/music/a.mp3"),
mbidRow(2, "/music/b.mp3"),
}, adoptRows: 1}
if s.adoptMovedTrack(context.Background(), q, newPath, 0, 0, "rec-mbid") {
t.Fatal("expected refusal on an ambiguous MBID match")
}
if len(q.adopted) != 0 {
t.Errorf("adopted despite ambiguity: %+v", q.adopted)
}
}
// An ambiguous MBID may still be resolvable by the fingerprint, which is a
// narrower signal — so falling through is allowed to succeed.
func TestAdoptMovedTrack_AmbiguousMbidFallsThroughToFingerprint(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{
byMbid: []dbq.FindMissingTrackByMbidRow{
mbidRow(1, "/music/a.mp3"),
mbidRow(2, "/music/b.mp3"),
},
byFingerprint: []dbq.FindMissingTrackByFingerprintRow{fpRow(2, "/music/b.mp3")},
adoptRows: 1,
}
if !s.adoptMovedTrack(context.Background(), q, newPath, 1000, 2000, "rec-mbid") {
t.Fatal("expected the fingerprint to disambiguate")
}
if len(q.adopted) != 1 || q.adopted[0].ID != testUUID(2) {
t.Errorf("adopted = %+v, want row 2", q.adopted)
}
}
func TestAdoptMovedTrack_RefusesAmbiguousFingerprintMatch(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{byFingerprint: []dbq.FindMissingTrackByFingerprintRow{
fpRow(1, "/music/a.mp3"),
fpRow(2, "/music/b.mp3"),
}, adoptRows: 1}
if s.adoptMovedTrack(context.Background(), q, newPath, 1000, 2000, "") {
t.Fatal("expected refusal on an ambiguous fingerprint match")
}
if len(q.adopted) != 0 {
t.Errorf("adopted despite ambiguity: %+v", q.adopted)
}
}
// duration_ms is 0 when ffprobe failed. Matching 0 against 0 would pair up
// unrelated broken files, so the fingerprint must not be attempted.
func TestAdoptMovedTrack_SkipsFingerprintWithoutRealValues(t *testing.T) {
tests := []struct {
name string
size int64
duration int32
}{
{"no duration", 1000, 0},
{"no size", 0, 2000},
{"neither", 0, 0},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{
byFingerprint: []dbq.FindMissingTrackByFingerprintRow{fpRow(1, oldPath)},
adoptRows: 1,
}
if s.adoptMovedTrack(context.Background(), q, newPath, tc.size, tc.duration, "") {
t.Error("adopted on an unusable fingerprint")
}
if len(q.fingerprintQueried) != 0 {
t.Error("queried the fingerprint with unusable values")
}
})
}
}
func TestAdoptMovedTrack_NoCandidates(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{adoptRows: 1}
if s.adoptMovedTrack(context.Background(), q, newPath, 1000, 2000, "rec-mbid") {
t.Fatal("expected no adoption when nothing matches")
}
if len(q.adopted) != 0 {
t.Errorf("adopted with no candidates: %+v", q.adopted)
}
}
// The row's mark was cleared between lookup and update — another file adopted it
// first. AdoptTrackPath's `missing_since IS NOT NULL` predicate reports 0 rows.
func TestAdoptMovedTrack_LostRaceReportsNotAdopted(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{
byMbid: []dbq.FindMissingTrackByMbidRow{mbidRow(5, oldPath)},
adoptRows: 0,
}
if s.adoptMovedTrack(context.Background(), q, newPath, 1000, 2000, "rec-mbid") {
t.Fatal("expected not-adopted when the update matched no rows")
}
}
// Failing to detect a move must never fail the file: the caller falls back to
// inserting a fresh row, which is the pre-#2528 behaviour.
func TestAdoptMovedTrack_ToleratesQueryErrors(t *testing.T) {
sentinel := errors.New("db down")
tests := []struct {
name string
q *fakeAdopter
}{
{"mbid lookup fails", &fakeAdopter{mbidErr: sentinel}},
{"fingerprint lookup fails", &fakeAdopter{fingerprintErr: sentinel}},
{"adopt fails", &fakeAdopter{
byMbid: []dbq.FindMissingTrackByMbidRow{mbidRow(1, oldPath)},
adoptErr: sentinel,
}},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
s := testScanner(t)
if s.adoptMovedTrack(context.Background(), tc.q, newPath, 1000, 2000, "rec-mbid") {
t.Error("reported adoption despite a query error")
}
})
}
}
// A failed MBID lookup must not stop the fingerprint from being tried.
func TestAdoptMovedTrack_MbidErrorStillTriesFingerprint(t *testing.T) {
s := testScanner(t)
q := &fakeAdopter{
mbidErr: errors.New("db hiccup"),
byFingerprint: []dbq.FindMissingTrackByFingerprintRow{fpRow(9, oldPath)},
adoptRows: 1,
}
if !s.adoptMovedTrack(context.Background(), q, newPath, 1000, 2000, "rec-mbid") {
t.Fatal("expected the fingerprint to be tried after an MBID lookup error")
}
if len(q.adopted) != 1 || q.adopted[0].ID != testUUID(9) {
t.Errorf("adopted = %+v, want row 9", q.adopted)
}
}
func TestUniqueMatch(t *testing.T) {
s := testScanner(t)
if _, ok := s.uniqueMatch(nil, newPath, "mbid"); ok {
t.Error("empty candidate set matched")
}
c, ok := s.uniqueMatch([]candidate{{id: testUUID(4), filePath: oldPath}}, newPath, "mbid")
if !ok {
t.Fatal("single candidate did not match")
}
if c.id != testUUID(4) || c.filePath != oldPath {
t.Errorf("candidate = %+v, want id 4 at %q", c, oldPath)
}
if _, ok := s.uniqueMatch([]candidate{
{id: testUUID(1)}, {id: testUUID(2)},
}, newPath, "mbid"); ok {
t.Error("multiple candidates matched")
}
}
func TestRowConverters(t *testing.T) {
got := rowsFromMbid([]dbq.FindMissingTrackByMbidRow{mbidRow(1, "/a"), mbidRow(2, "/b")})
if len(got) != 2 || got[0].id != testUUID(1) || got[1].filePath != "/b" {
t.Errorf("rowsFromMbid = %+v", got)
}
got = rowsFromFingerprint([]dbq.FindMissingTrackByFingerprintRow{fpRow(3, "/c")})
if len(got) != 1 || got[0].id != testUUID(3) || got[0].filePath != "/c" {
t.Errorf("rowsFromFingerprint = %+v", got)
}
}
// pgtype.UUID zero value must not be mistaken for a real id.
func TestUniqueMatch_ZeroUUIDNotValid(t *testing.T) {
var zero pgtype.UUID
if zero.Valid {
t.Fatal("zero pgtype.UUID should not be Valid")
}
}
+155
View File
@@ -0,0 +1,155 @@
package library
import (
"context"
"errors"
"fmt"
"os"
"github.com/jackc/pgx/v5/pgtype"
"git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq"
)
// trackReconciler is the slice of dbq.Queries reconcileMissing needs. Narrowed
// to an interface so the guard logic — which is the part that can do damage —
// is unit-testable against a fake without a database.
type trackReconciler interface {
ListTrackPathsForReconcile(ctx context.Context) ([]dbq.ListTrackPathsForReconcileRow, error)
MarkTracksMissing(ctx context.Context, ids []pgtype.UUID) (int64, error)
ClearTracksMissing(ctx context.Context, ids []pgtype.UUID) (int64, error)
}
// Reconcile marks tracks whose files have disappeared (#2523).
//
// Why this exists: nothing in Minstrel used to notice a deleted file. The walk
// only visits paths that exist, so a row whose file is gone was never scanned,
// never errored, never counted — permanently invisible. The watcher ignores
// removals by design (see classifyEvent), and the safety-net scan is the same
// walk, so it covers additions only. Rows accumulated forever, kept being
// offered to recommendations, and failed at playback.
//
// Why it MARKS rather than deletes: a missing file is a claim about the
// filesystem, and the filesystem lies transiently — an unmounted volume, a
// network-storage blip, a container that started before its media mount
// attached. Every other sweep in this codebase (internal/gc) resolves a truth
// *inside* the database and is safe to run blind. This one isn't, so the
// destructive step is deliberately not here. Marking is reversible: the next
// good scan clears it.
// missingMarkMaxFraction caps how much of the library one reconcile may newly
// mark missing. A partially-attached mount is the failure this defends against:
// the roots resolve, the walk succeeds, and it legitimately sees only part of
// the library — evidence indistinguishable from a mass deletion.
//
// A quarter is deliberately conservative. A genuine bulk deletion trips it and
// gets logged rather than applied, which needs a second scan (or operator
// action) to take effect. That's the right trade: the cost of over-refusing is
// a stale row and a log line, and the cost of over-marking is a chunk of the
// library silently vanishing from every mix.
const missingMarkMaxFraction = 0.25
// reconcileMissing diffs the paths the walk saw against every row in the table.
// Rows not seen get marked; rows seen that carry a mark get cleared.
//
// seen must come from a COMPLETE walk of every configured root. Callers with a
// partial view must not call this.
func (s *Scanner) reconcileMissing(
ctx context.Context, q trackReconciler, seen map[string]struct{}, stats *Stats,
) error {
if err := s.verifyRootsPresent(); err != nil {
return err
}
// Roots resolved but the walk found nothing. Either the library is genuinely
// empty — in which case there is nothing to reconcile — or the mount is
// hollow. Both mean: don't act.
if len(seen) == 0 {
return errors.New("walk saw no audio files; refusing to reconcile")
}
rows, err := q.ListTrackPathsForReconcile(ctx)
if err != nil {
return fmt.Errorf("list track paths: %w", err)
}
if len(rows) == 0 {
return nil
}
var toMark, toClear []pgtype.UUID
for _, row := range rows {
_, present := seen[row.FilePath]
switch {
case !present && !row.MissingSince.Valid:
toMark = append(toMark, row.ID)
case present && row.MissingSince.Valid:
toClear = append(toClear, row.ID)
}
}
// Clear before marking, and unconditionally. Restoring a file is never the
// dangerous direction, so it must not be blocked by the guard below —
// otherwise a library that tripped the cap once could never recover its
// marks even after the mount came back.
if len(toClear) > 0 {
n, err := q.ClearTracksMissing(ctx, toClear)
if err != nil {
return fmt.Errorf("clear missing marks: %w", err)
}
stats.Restored = int(n)
s.logger.Info("library scan: files returned", "count", n)
}
if len(toMark) == 0 {
return nil
}
if fraction := float64(len(toMark)) / float64(len(rows)); fraction > missingMarkMaxFraction {
return fmt.Errorf(
"refusing to mark %d of %d tracks missing (%.0f%% > %.0f%% cap): "+
"this looks like an unavailable mount rather than a deletion",
len(toMark), len(rows), fraction*100, missingMarkMaxFraction*100,
)
}
n, err := q.MarkTracksMissing(ctx, toMark)
if err != nil {
return fmt.Errorf("mark tracks missing: %w", err)
}
stats.Missing = int(n)
// Warn, not Info: every one of these is a library entry the operator
// probably didn't intend to lose, and the only place it surfaces today is
// this line.
s.logger.Warn("library scan: tracks marked missing (files not found)",
"count", n, "library_total", len(rows))
return nil
}
// verifyRootsPresent is the first and most important guard. If a configured root
// doesn't resolve to a readable directory, the walk beneath it found nothing and
// every row under it would look deleted. An unmounted media volume is the
// obvious case, and it is common enough — a container restart racing its volume
// mount does exactly this.
func (s *Scanner) verifyRootsPresent() error {
if len(s.paths) == 0 {
return errors.New("no scan roots configured")
}
for _, root := range s.paths {
info, err := os.Stat(root)
if err != nil {
return fmt.Errorf("scan root %q unavailable: %w", root, err)
}
if !info.IsDir() {
return fmt.Errorf("scan root %q is not a directory", root)
}
entries, err := os.ReadDir(root)
if err != nil {
return fmt.Errorf("scan root %q unreadable: %w", root, err)
}
// An empty root is the signature of a mount point with nothing mounted
// on it. `os.Stat` succeeds on the bare directory, so this is the only
// cheap way to tell the two apart.
if len(entries) == 0 {
return fmt.Errorf("scan root %q is empty; refusing to reconcile", root)
}
}
return nil
}
+326
View File
@@ -0,0 +1,326 @@
package library
import (
"context"
"errors"
"fmt"
"io"
"log/slog"
"os"
"path/filepath"
"testing"
"github.com/jackc/pgx/v5/pgtype"
"git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq"
)
// fakeReconciler records what reconcileMissing decided to do, so the guards can
// be tested without a database. The guards are the whole point of this pass —
// they are what stands between an unmounted volume and the library disappearing
// from every mix — so they get tested directly rather than via integration.
type fakeReconciler struct {
rows []dbq.ListTrackPathsForReconcileRow
marked []pgtype.UUID
cleared []pgtype.UUID
listErr error
markErr error
clearErr error
}
func (f *fakeReconciler) ListTrackPathsForReconcile(context.Context) ([]dbq.ListTrackPathsForReconcileRow, error) {
return f.rows, f.listErr
}
func (f *fakeReconciler) MarkTracksMissing(_ context.Context, ids []pgtype.UUID) (int64, error) {
if f.markErr != nil {
return 0, f.markErr
}
f.marked = append(f.marked, ids...)
return int64(len(ids)), nil
}
func (f *fakeReconciler) ClearTracksMissing(_ context.Context, ids []pgtype.UUID) (int64, error) {
if f.clearErr != nil {
return 0, f.clearErr
}
f.cleared = append(f.cleared, ids...)
return int64(len(ids)), nil
}
// Compile-time proof the real queries still satisfy what reconcile needs — the
// interface exists to narrow dbq.Queries, not to diverge from it.
var _ trackReconciler = (*dbq.Queries)(nil)
func testUUID(n byte) pgtype.UUID {
var u pgtype.UUID
u.Bytes[15] = n
u.Valid = true
return u
}
func markedAt() pgtype.Timestamptz {
return pgtype.Timestamptz{Valid: true}
}
func row(n byte, path string, missing bool) dbq.ListTrackPathsForReconcileRow {
r := dbq.ListTrackPathsForReconcileRow{ID: testUUID(n), FilePath: path}
if missing {
r.MissingSince = markedAt()
}
return r
}
// populatedRoot returns a directory containing one file, so verifyRootsPresent
// treats it as a real, mounted library root.
func populatedRoot(t *testing.T) string {
t.Helper()
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "a.mp3"), []byte("x"), 0o600); err != nil {
t.Fatal(err)
}
return dir
}
func testScanner(t *testing.T, roots ...string) *Scanner {
t.Helper()
return &Scanner{
logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
paths: roots,
}
}
func TestReconcileMissing_MarksRowsTheWalkDidNotSee(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
// 10 rows with 2 absent — 20%, deliberately under missingMarkMaxFraction so
// this exercises marking rather than the cap. (An earlier version of this
// test used 2-of-4 and was really testing the guard by accident.)
rows := make([]dbq.ListTrackPathsForReconcileRow, 0, 10)
seen := map[string]struct{}{}
for i := 0; i < 10; i++ {
p := fmt.Sprintf("/music/track-%02d.mp3", i)
rows = append(rows, row(byte(i), p, false))
if i >= 2 {
seen[p] = struct{}{}
}
}
q := &fakeReconciler{rows: rows}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, seen, &stats); err != nil {
t.Fatalf("reconcile: %v", err)
}
if len(q.marked) != 2 {
t.Fatalf("marked %d rows, want 2", len(q.marked))
}
if q.marked[0] != testUUID(0) || q.marked[1] != testUUID(1) {
t.Errorf("marked the wrong rows: %v", q.marked)
}
if stats.Missing != 2 {
t.Errorf("stats.Missing = %d, want 2", stats.Missing)
}
if len(q.cleared) != 0 {
t.Errorf("cleared %d rows, want 0", len(q.cleared))
}
}
func TestReconcileMissing_ClearsRowsWhoseFileReturned(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
q := &fakeReconciler{rows: []dbq.ListTrackPathsForReconcileRow{
row(1, "/music/back.mp3", true),
row(2, "/music/still-here.mp3", false),
}}
seen := map[string]struct{}{
"/music/back.mp3": {},
"/music/still-here.mp3": {},
}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, seen, &stats); err != nil {
t.Fatalf("reconcile: %v", err)
}
if len(q.cleared) != 1 || q.cleared[0] != testUUID(1) {
t.Fatalf("cleared = %v, want just row 1", q.cleared)
}
if stats.Restored != 1 {
t.Errorf("stats.Restored = %d, want 1", stats.Restored)
}
if len(q.marked) != 0 {
t.Errorf("marked %d rows, want 0", len(q.marked))
}
}
// An already-marked row must not be re-marked: the timestamp is the "how long
// has this been gone" clock that any future cleanup policy depends on.
func TestReconcileMissing_DoesNotRemarkAlreadyMissingRows(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
q := &fakeReconciler{rows: []dbq.ListTrackPathsForReconcileRow{
row(1, "/music/long-gone.mp3", true),
row(2, "/music/present.mp3", false),
}}
seen := map[string]struct{}{"/music/present.mp3": {}}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, seen, &stats); err != nil {
t.Fatalf("reconcile: %v", err)
}
if len(q.marked) != 0 {
t.Errorf("re-marked an already-missing row: %v", q.marked)
}
if len(q.cleared) != 0 {
t.Errorf("cleared = %v, want none", q.cleared)
}
}
// The guard that matters most. A half-attached mount makes the walk succeed
// while seeing only part of the library — evidence indistinguishable from a mass
// deletion, so reconcile must refuse rather than guess.
func TestReconcileMissing_RefusesWhenTooMuchWouldBeMarked(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
rows := make([]dbq.ListTrackPathsForReconcileRow, 0, 100)
seen := map[string]struct{}{}
for i := 0; i < 100; i++ {
p := fmt.Sprintf("/music/track-%03d.mp3", i)
rows = append(rows, row(byte(i), p, false))
// Only 60 of 100 present -> 40% would be marked, over the 25% cap.
if i < 60 {
seen[p] = struct{}{}
}
}
q := &fakeReconciler{rows: rows}
var stats Stats
err := s.reconcileMissing(context.Background(), q, seen, &stats)
if err == nil {
t.Fatal("expected reconcile to refuse, got nil error")
}
if len(q.marked) != 0 {
t.Errorf("marked %d rows despite refusing", len(q.marked))
}
if stats.Missing != 0 {
t.Errorf("stats.Missing = %d, want 0", stats.Missing)
}
}
// Restoring is never the dangerous direction, so it must survive the cap —
// otherwise a library that tripped the cap once could never clear its marks
// even after the volume came back.
func TestReconcileMissing_ClearsEvenWhenMarkCapTrips(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
rows := []dbq.ListTrackPathsForReconcileRow{row(1, "/music/back.mp3", true)}
seen := map[string]struct{}{"/music/back.mp3": {}}
// Add enough absent rows to blow the cap.
for i := 2; i < 10; i++ {
rows = append(rows, row(byte(i), fmt.Sprintf("/music/absent-%02d.mp3", i), false))
}
q := &fakeReconciler{rows: rows}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, seen, &stats); err == nil {
t.Fatal("expected the mark cap to trip")
}
if len(q.cleared) != 1 {
t.Errorf("cleared %d rows, want 1 — restores must not be blocked by the cap", len(q.cleared))
}
if stats.Restored != 1 {
t.Errorf("stats.Restored = %d, want 1", stats.Restored)
}
}
func TestReconcileMissing_RefusesOnEmptyWalk(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
q := &fakeReconciler{rows: []dbq.ListTrackPathsForReconcileRow{
row(1, "/music/a.mp3", false),
}}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, map[string]struct{}{}, &stats); err == nil {
t.Fatal("expected refusal when the walk saw no files")
}
if len(q.marked) != 0 {
t.Errorf("marked rows on an empty walk: %v", q.marked)
}
}
// The unmounted-volume case: the configured root doesn't exist at all.
func TestReconcileMissing_RefusesWhenRootMissing(t *testing.T) {
s := testScanner(t, filepath.Join(t.TempDir(), "not-mounted"))
q := &fakeReconciler{rows: []dbq.ListTrackPathsForReconcileRow{
row(1, "/music/a.mp3", false),
}}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, map[string]struct{}{"/x": {}}, &stats); err == nil {
t.Fatal("expected refusal when a scan root is absent")
}
if len(q.marked) != 0 {
t.Errorf("marked rows with an absent root: %v", q.marked)
}
}
// A mount point that exists but has nothing mounted on it: os.Stat succeeds on
// the bare directory, which is why emptiness is checked separately.
func TestReconcileMissing_RefusesWhenRootEmpty(t *testing.T) {
s := testScanner(t, t.TempDir())
q := &fakeReconciler{rows: []dbq.ListTrackPathsForReconcileRow{
row(1, "/music/a.mp3", false),
}}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, map[string]struct{}{"/x": {}}, &stats); err == nil {
t.Fatal("expected refusal when a scan root is empty")
}
}
// Several roots, one detached. Marking must not proceed on partial evidence just
// because the other roots looked fine.
func TestReconcileMissing_RefusesWhenAnyRootMissing(t *testing.T) {
good := populatedRoot(t)
s := testScanner(t, good, filepath.Join(t.TempDir(), "detached"))
q := &fakeReconciler{rows: []dbq.ListTrackPathsForReconcileRow{
row(1, "/music/a.mp3", false),
}}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, map[string]struct{}{"/x": {}}, &stats); err == nil {
t.Fatal("expected refusal when one of several roots is absent")
}
}
func TestReconcileMissing_NoRowsIsNotAnError(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
q := &fakeReconciler{}
var stats Stats
if err := s.reconcileMissing(context.Background(), q, map[string]struct{}{"/x": {}}, &stats); err != nil {
t.Fatalf("empty library should reconcile cleanly, got %v", err)
}
}
func TestReconcileMissing_PropagatesListError(t *testing.T) {
root := populatedRoot(t)
s := testScanner(t, root)
sentinel := errors.New("boom")
q := &fakeReconciler{listErr: sentinel}
var stats Stats
err := s.reconcileMissing(context.Background(), q, map[string]struct{}{"/x": {}}, &stats)
if !errors.Is(err, sentinel) {
t.Fatalf("err = %v, want it to wrap %v", err, sentinel)
}
}
func TestVerifyRootsPresent_NoRootsConfigured(t *testing.T) {
s := testScanner(t)
if err := s.verifyRootsPresent(); err == nil {
t.Fatal("expected an error with no scan roots configured")
}
}
+134 -31
View File
@@ -60,6 +60,11 @@ type Stats struct {
Updated int `json:"updated"` Updated int `json:"updated"`
Skipped int `json:"skipped"` Skipped int `json:"skipped"`
Errored int `json:"errored"` Errored int `json:"errored"`
// Missing / Restored come from the reconcile pass, not the walk (#2523):
// rows whose file the walk didn't find, and rows whose file came back.
// Only a full Scan sets these — see reconcileMissing.
Missing int `json:"missing"`
Restored int `json:"restored"`
} }
type Scanner struct { type Scanner struct {
@@ -76,6 +81,12 @@ func New(pool *pgxpool.Pool, logger *slog.Logger, paths []string) *Scanner {
// newer than the existing row's updated_at. Walk errors and per-file errors // newer than the existing row's updated_at. Walk errors and per-file errors
// are logged + counted; the scan keeps going. // are logged + counted; the scan keeps going.
// //
// It then reconciles: rows whose file the walk never saw get marked missing,
// and rows whose file has come back get un-marked (#2523). Only a FULL scan may
// do this — the walk's set of seen paths is the evidence, and a partial
// (watcher-driven) scan has no basis for concluding anything about files it
// didn't look at. That's why ScanFiles does not reconcile.
//
// progressCb (may be nil) receives the current Stats snapshot after each // progressCb (may be nil) receives the current Stats snapshot after each
// processed file. Used by the orchestrator to drive partial-tally writes. // processed file. Used by the orchestrator to drive partial-tally writes.
func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, error) { func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, error) {
@@ -83,35 +94,55 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
q := dbq.New(s.pool) q := dbq.New(s.pool)
start := time.Now() start := time.Now()
for _, root := range s.paths { // PHASE 1 — enumerate. Collect every audio path without touching tags or
if err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { // ffprobe. Cheap: WalkDir already stats each entry, so this adds a directory
if ctx.Err() != nil { // traversal and nothing else.
return fs.SkipAll //
} // The order matters and is the whole reason enumeration is separate.
if err != nil { // Reconcile has to mark disappeared rows BEFORE any file is processed,
s.logger.Warn("library scan walk error", "path", path, "err", err) // because move detection (#2528) can only adopt a row that is already marked
stats.Errored++ // missing. A rename performed while the server was down surfaces the deletion
if progressCb != nil { // and the addition in the SAME scan — so if reconcile ran at the end, the new
progressCb(stats) // path would insert a fresh row first and the fork would be permanent.
} paths, walkErrs := s.enumerate(ctx, progressCb, &stats)
return nil stats.Errored += walkErrs
} if err := ctx.Err(); err != nil {
if d.IsDir() { return stats, err
return nil }
}
if !audioExtensions[strings.ToLower(filepath.Ext(path))] { // PHASE 2 — reconcile. Only ever on a COMPLETE enumeration: a cancelled walk
return nil // has a partial view and would mark everything it hadn't reached.
} seen := make(map[string]struct{}, len(paths))
if _, _, err := s.scanFile(ctx, q, path, &stats); err != nil { for _, p := range paths {
s.logger.Warn("library scan file error", "path", path, "err", err) seen[p] = struct{}{}
stats.Errored++ }
} if err := s.reconcileMissing(ctx, q, seen, &stats); err != nil {
if progressCb != nil { // Not fatal. The guards deliberately refuse to act on ambiguous
progressCb(stats) // evidence, and that refusal arrives here as an error.
} //
return nil // The consequence is named explicitly because it is not obvious: move
}); err != nil { // detection (#2528) can only adopt a row that is already marked missing,
return stats, fmt.Errorf("library: walk %q: %w", root, err) // so a refused reconcile also means renamed files insert fresh rows and
// fork their history. That's the pre-#2528 behaviour rather than a new
// failure, but it's worth knowing which scan it happened on. It bites
// hardest when a large fraction of a small library is reorganised at
// once, which trips the mark cap.
s.logger.Warn("library scan: reconcile skipped — moved files will fork rather than adopt",
"err", err)
}
// PHASE 3 — process, in walk order so logs and cover-art batching stay
// grouped by directory rather than following map iteration order.
for _, path := range paths {
if ctx.Err() != nil {
break
}
if _, _, err := s.scanFile(ctx, q, path, &stats); err != nil {
s.logger.Warn("library scan file error", "path", path, "err", err)
stats.Errored++
}
if progressCb != nil {
progressCb(stats)
} }
} }
@@ -121,6 +152,8 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
"updated", stats.Updated, "updated", stats.Updated,
"skipped", stats.Skipped, "skipped", stats.Skipped,
"errored", stats.Errored, "errored", stats.Errored,
"missing", stats.Missing,
"restored", stats.Restored,
"duration_ms", time.Since(start).Milliseconds(), "duration_ms", time.Since(start).Milliseconds(),
) )
if err := ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
@@ -129,6 +162,46 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
return stats, nil return stats, nil
} }
// enumerate walks every configured root and returns the audio paths found, in
// walk order, plus a count of walk errors.
//
// A path is recorded even if it will later fail to parse: an unreadable file is a
// broken file, not a missing one, and letting reconcile mark it missing would
// hide it from the operator behind the wrong explanation.
func (s *Scanner) enumerate(
ctx context.Context, progressCb func(Stats), stats *Stats,
) ([]string, int) {
paths := make([]string, 0, 8192)
errs := 0
for _, root := range s.paths {
// WalkDir's own error return is folded into the per-entry handler below,
// so a bad root is counted rather than aborting the whole scan — one
// unreadable root shouldn't discard the others' results.
_ = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if ctx.Err() != nil {
return fs.SkipAll
}
if err != nil {
s.logger.Warn("library scan walk error", "path", path, "err", err)
errs++
if progressCb != nil {
progressCb(*stats)
}
return nil
}
if d.IsDir() {
return nil
}
if !audioExtensions[strings.ToLower(filepath.Ext(path))] {
return nil
}
paths = append(paths, path)
return nil
})
}
return paths, errs
}
// scanFile upserts a single audio file. Returns the album ID the track // scanFile upserts a single audio file. Returns the album ID the track
// belongs to and whether the file was added/updated (false = skipped as // belongs to and whether the file was added/updated (false = skipped as
// unchanged), so watcher-driven callers can enrich just the changed albums. // unchanged), so watcher-driven callers can enrich just the changed albums.
@@ -218,6 +291,23 @@ func (s *Scanner) scanFile(
durationMs = probed durationMs = probed
} }
// A path we've never seen might not be a new track — it might be one that
// moved or was renamed (#2528). Adopting re-points the existing row at this
// path and clears its missing mark, so the UpsertTrack below conflicts on
// file_path and updates THAT row: same track id, likes and play history
// intact. Without this, renumbering an album forks every track on it.
//
// Runs here rather than earlier because the fingerprint needs the probed
// duration, and only for genuinely unknown paths — a known path is already
// the row we're going to update.
if !knownTrack {
if s.adoptMovedTrack(ctx, q, path, info.Size(), durationMs, recordingMBID) {
// Count it as an update: the row existed, and reporting it as Added
// would overstate library growth on every reorganisation.
knownTrack = true
}
}
params := dbq.UpsertTrackParams{ params := dbq.UpsertTrackParams{
Title: trackTitle, Title: trackTitle,
AlbumID: album.ID, AlbumID: album.ID,
@@ -324,8 +414,21 @@ func (s *Scanner) resolveArtist(ctx context.Context, q *dbq.Queries, name, mbid
ID: existing.ID, ID: existing.ID,
Mbid: &m, Mbid: &m,
}); uerr != nil { }); uerr != nil {
s.logger.Warn("library scan: heal artist mbid failed", if isUniqueViolation(uerr) {
"artist_id", existing.ID, "err", uerr) // Another artist row already owns this MBID — two rows that
// should be merged (usually two spellings of one name).
// Expected, not a fault: leave NULL and let the operator
// merge. Mirrors resolveAlbum, which has always handled it
// this way — without this branch the identical benign
// condition logged a generic warning plus a Postgres ERROR
// line on every scan, which teaches an operator to ignore
// database errors (#2524).
s.logger.Info("library scan: duplicate artist mbid (canonical row already owns it)",
"artist_id", existing.ID, "artist", name, "mbid", mbid)
} else {
s.logger.Warn("library scan: heal artist mbid failed",
"artist_id", existing.ID, "err", uerr)
}
} else { } else {
existing.Mbid = &m existing.Mbid = &m
} }
+103
View File
@@ -205,3 +205,106 @@ func writeTestMP3(t *testing.T, path string, frames map[string]string) {
t.Fatal(err) t.Fatal(err)
} }
} }
// TestScanner_AdoptsMovedFile_Integration is the #2528 proof: a renamed file
// must keep its existing tracks row — same id, so likes, play history and
// playlist memberships travel with it — rather than forking into a marked ghost
// plus a fresh zero-history row.
//
// Uses the MBID path. The synthetic MP3s here carry no real audio, so ffprobe
// yields duration 0 and the size+duration fingerprint is deliberately unusable —
// which is why the recording MBID is the signal under test.
//
// Eight tracks with one rename keeps the marked fraction at 12.5%, under
// missingMarkMaxFraction. That is load-bearing: if the rename exceeded the cap,
// reconcile would refuse to mark, adoption could not fire, and the file would
// fork. See the "reconcile skipped" warning in Scan.
func TestScanner_AdoptsMovedFile_Integration(t *testing.T) {
if testing.Short() {
t.Skip("skipping scanner integration in -short mode")
}
dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL")
if dsn == "" {
t.Skip("MINSTREL_TEST_DATABASE_URL not set")
}
ctx := context.Background()
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
if err := db.Migrate(dsn, logger); err != nil {
t.Fatalf("migrate: %v", err)
}
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
t.Fatalf("pool: %v", err)
}
t.Cleanup(pool.Close)
if _, err := pool.Exec(ctx, "TRUNCATE tracks, albums, artists RESTART IDENTITY CASCADE"); err != nil {
t.Fatalf("truncate: %v", err)
}
root := t.TempDir()
const movedMBID = "11111111-2222-3333-4444-555555555555"
movedFrom := filepath.Join(root, "artistM/albumM/04 - Bleed It Out.mp3")
writeTestMP3(t, movedFrom, map[string]string{
"TIT2": "Bleed It Out", "TPE1": "Artist M", "TALB": "Album M", "TRCK": "4",
// dhowden surfaces TXXX as a Comm whose Description is the Picard tag
// name; "MusicBrainz Track Id" is mbz.Recording.
"TXXX": "MusicBrainz Track Id\x00" + movedMBID,
})
// Filler so one rename stays under the mark cap.
for i := 1; i <= 7; i++ {
writeTestMP3(t, filepath.Join(root, "artistM/albumM/filler", string(rune('a'+i))+".mp3"),
map[string]string{
"TIT2": "Filler " + string(rune('0'+i)), "TPE1": "Artist M", "TALB": "Album M",
})
}
scanner := New(pool, logger, []string{root})
if _, err := scanner.Scan(ctx, nil); err != nil {
t.Fatalf("first scan: %v", err)
}
q := dbq.New(pool)
before, err := q.GetTrackByPath(ctx, movedFrom)
if err != nil {
t.Fatalf("track not indexed on first scan: %v", err)
}
if before.Mbid == nil || *before.Mbid != movedMBID {
t.Fatalf("recording mbid not stored: %v", before.Mbid)
}
// Renumber the file, exactly as a tag editor would.
movedTo := filepath.Join(root, "artistM/albumM/02 - Bleed It Out.mp3")
if err := os.Rename(movedFrom, movedTo); err != nil {
t.Fatalf("rename: %v", err)
}
if _, err := scanner.Scan(ctx, nil); err != nil {
t.Fatalf("second scan: %v", err)
}
after, err := q.GetTrackByPath(ctx, movedTo)
if err != nil {
t.Fatalf("track not found at its new path: %v", err)
}
if after.ID != before.ID {
t.Errorf("track id changed on rename: %v -> %v (history would be stranded)",
before.ID, after.ID)
}
if after.MissingSince.Valid {
t.Errorf("adopted row is still marked missing: %v", after.MissingSince)
}
// The old path must be gone entirely — not lingering as a marked ghost.
if _, err := q.GetTrackByPath(ctx, movedFrom); err == nil {
t.Error("old path still has a tracks row; the track forked instead of moving")
}
var total int
if err := pool.QueryRow(ctx, "SELECT count(*) FROM tracks").Scan(&total); err != nil {
t.Fatalf("count: %v", err)
}
if total != 8 {
t.Errorf("tracks = %d, want 8 — a rename must not add a row", total)
}
}
+5
View File
@@ -24,6 +24,11 @@ type LibraryStageTallies struct {
Updated int `json:"updated"` Updated int `json:"updated"`
Skipped int `json:"skipped"` Skipped int `json:"skipped"`
Errored int `json:"errored"` Errored int `json:"errored"`
// Reconcile results (#2523). Surfaced in the scan record because a track
// disappearing from the library is something the operator should be able to
// see happened, rather than discovering it when a mix comes up short.
Missing int `json:"missing"`
Restored int `json:"restored"`
} }
// MBIDBackfillStageTallies wires BackfillMBIDsResult into the scan_runs jsonb column. // MBIDBackfillStageTallies wires BackfillMBIDsResult into the scan_runs jsonb column.
+18
View File
@@ -4,6 +4,20 @@ import { api } from './client';
// Mirrors internal/api/me_recommendation_metrics.go: raw play sources are // Mirrors internal/api/me_recommendation_metrics.go: raw play sources are
// bucketed server-side into stable surface families, grouped by intent, and // bucketed server-side into stable surface families, grouped by intent, and
// anchored by the manual-plays baseline (milestone #127). // anchored by the manual-plays baseline (milestone #127).
// A difference from the baseline, with its uncertainty (#2495). Both figures
// are already in percentage points — the server does the arithmetic so both
// clients read the same numbers.
//
// `distinguishable: false` means |delta_pp| < margin_pp: the delta cannot be
// told apart from zero, however large it looks. That distinction is the whole
// point of this type — `low_confidence` answers "is this worth showing?", which
// is a much lower bar than "is this worth acting on?".
export type MetricDelta = {
delta_pp: number;
margin_pp: number;
distinguishable: boolean;
};
export type SurfaceMetric = { export type SurfaceMetric = {
key: string; key: string;
label: string; label: string;
@@ -12,6 +26,10 @@ export type SurfaceMetric = {
skip_rate: number; skip_rate: number;
avg_completion: number; avg_completion: number;
low_confidence: boolean; low_confidence: boolean;
// Absent on the baseline row itself, and whenever the samples are too thin
// for a margin to mean anything.
skip_delta?: MetricDelta;
completion_delta?: MetricDelta;
// Present when the surface's builder stamps pick-kind provenance and // Present when the surface's builder stamps pick-kind provenance and
// the window holds attributed plays (#1249, generalized #1270): For // the window holds attributed plays (#1249, generalized #1270): For
// You's taste/fresh split, Discover's candidate buckets, the tiered // You's taste/fresh split, Discover's candidate buckets, the tiered
+21 -7
View File
@@ -216,9 +216,17 @@
return plays > 0 ? hits / plays : 0; return plays > 0 ? hits / plays : 0;
} }
function latest(s: TrendSeries): { skip: number; completion: number } { // The "latest" columns are ONE WEEK while the Plays column is the whole
// window, which is a trap: a 40% skip rate off 17 plays sat next to a
// four-figure Plays total and read as a solid signal. It isn't — I misread
// exactly this and briefly concluded Deep cuts was the worst surface, when
// over 180 days it's one of the best (#2495). So the week's own play count
// comes back with the rates and is rendered beside them.
function latest(s: TrendSeries): { skip: number; completion: number; plays: number } {
const last = s.points[s.points.length - 1]; const last = s.points[s.points.length - 1];
return last ? { skip: last.skip_rate, completion: last.avg_completion } : { skip: 0, completion: 0 }; return last
? { skip: last.skip_rate, completion: last.avg_completion, plays: last.plays }
: { skip: 0, completion: 0, plays: 0 };
} }
function pct(v: number): string { function pct(v: number): string {
@@ -423,6 +431,9 @@
Skip rate per surface over the last {trends?.weeks ?? 12} weeks (lower is better; all Skip rate per surface over the last {trends?.weeks ?? 12} weeks (lower is better; all
users aggregated, rates only). Dashed ticks mark tuning changes. Taste hit is the share users aggregated, rates only). Dashed ticks mark tuning changes. Taste hit is the share
of plays whose artist fits the current taste profile. of plays whose artist fits the current taste profile.
<span class="font-medium">The skip and completion columns show the most recent week
alone</span>, not the whole window — the figure after the skip rate is that week's
play count, so a rate drawn from a handful of listens reads as what it is.
</p> </p>
</div> </div>
{#if trendsFailed} {#if trendsFailed}
@@ -439,10 +450,10 @@
<tr class="text-left text-text-secondary"> <tr class="text-left text-text-secondary">
<th class="py-1 font-medium">Surface</th> <th class="py-1 font-medium">Surface</th>
<th class="py-1 font-medium">Skip rate by week</th> <th class="py-1 font-medium">Skip rate by week</th>
<th class="py-1 text-right font-medium">Plays</th> <th class="py-1 text-right font-medium">Plays<span class="font-normal text-xs"> (window)</span></th>
<th class="py-1 text-right font-medium">Latest skip</th> <th class="py-1 text-right font-medium">Skip<span class="font-normal text-xs"> (last wk)</span></th>
<th class="py-1 text-right font-medium">Latest completion</th> <th class="py-1 text-right font-medium">Completion<span class="font-normal text-xs"> (last wk)</span></th>
<th class="py-1 text-right font-medium">Taste hit</th> <th class="py-1 text-right font-medium">Taste hit<span class="font-normal text-xs"> (window)</span></th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
@@ -493,7 +504,10 @@
</svg> </svg>
</td> </td>
<td class="py-1.5 text-right tabular-nums">{s.plays}</td> <td class="py-1.5 text-right tabular-nums">{s.plays}</td>
<td class="py-1.5 text-right tabular-nums">{pct(latest(s).skip)}</td> <td class="py-1.5 text-right tabular-nums">
{pct(latest(s).skip)}
<span class="text-xs text-text-secondary">/{latest(s).plays}</span>
</td>
<td class="py-1.5 text-right tabular-nums">{pct(latest(s).completion)}</td> <td class="py-1.5 text-right tabular-nums">{pct(latest(s).completion)}</td>
<td class="py-1.5 text-right tabular-nums">{pct(windowTasteHitRate(s))}</td> <td class="py-1.5 text-right tabular-nums">{pct(windowTasteHitRate(s))}</td>
</tr> </tr>
+23 -3
View File
@@ -195,9 +195,29 @@ describe('Admin tuning page', () => {
await waitFor(() => expect(screen.getByText('Weekly trends')).toBeInTheDocument()); await waitFor(() => expect(screen.getByText('Weekly trends')).toBeInTheDocument());
expect(screen.getByTestId('sparkline-radio')).toBeInTheDocument(); expect(screen.getByTestId('sparkline-radio')).toBeInTheDocument();
expect(screen.getByTestId('sparkline-discover')).toBeInTheDocument(); expect(screen.getByTestId('sparkline-discover')).toBeInTheDocument();
// Latest skip rate column for radio = 40% (also discover's latest // The skip column is the LAST WEEK's rate, now carrying that week's play
// completion, hence getAllBy). // count so a rate off a handful of plays reads as what it is (#2495).
expect(screen.getAllByText('40%').length).toBeGreaterThan(0); // Radio's latest week: 40% skip over 15 plays; Discover's: 60% over 5.
expect(screen.getByText('/15')).toBeInTheDocument();
expect(screen.getByText('/5')).toBeInTheDocument();
// Completion columns are unchanged and still bare percentages — radio 70%.
expect(screen.getByText('70%')).toBeInTheDocument();
// '40%' is now genuinely ambiguous: radio's latest SKIP rate and discover's
// latest COMPLETION are both 40%. testing-library matches an element's own
// direct text nodes, so the skip cell still matches despite its trailing
// play-count span. Assert the count rather than pretending it's unique.
expect(screen.getAllByText('40%')).toHaveLength(2);
// The window/last-week distinction has to be visible in the headers, or the
// Plays total reads as the denominator of the skip rate. That misreading is
// what #2495 was filed over.
//
// Queried as column headers rather than by text: the caption below also
// mentions "Plays", and matching on the word finds the prose too.
// Exact accessible names: /^Skip/ also matches the "Skip rate by week"
// sparkline column, and /Plays/ matched the caption prose before that.
expect(screen.getByRole('columnheader', { name: 'Plays (window)' })).toBeInTheDocument();
expect(screen.getByRole('columnheader', { name: 'Skip (last wk)' })).toBeInTheDocument();
expect(screen.getByRole('columnheader', { name: 'Completion (last wk)' })).toBeInTheDocument();
// The knob turn is listed under the chart AND tooltipped on each // The knob turn is listed under the chart AND tooltipped on each
// sparkline's marker tick, hence getAllBy. // sparkline's marker tick, hence getAllBy.
expect( expect(
+52 -27
View File
@@ -12,7 +12,7 @@
createRecommendationMetricsQuery, createRecommendationMetricsQuery,
type RecommendationMetrics, type RecommendationMetrics,
type SurfaceIntent, type SurfaceIntent,
type SurfaceMetric type MetricDelta
} from '$lib/api/metrics'; } from '$lib/api/metrics';
import { theme, setTheme, type ThemePreference } from '$lib/stores/theme.svelte'; import { theme, setTheme, type ThemePreference } from '$lib/stores/theme.svelte';
import { player, setCrossfade } from '$lib/player/store.svelte'; import { player, setCrossfade } from '$lib/player/store.svelte';
@@ -45,20 +45,39 @@
return `${(v * 100).toFixed(0)}%`; return `${(v * 100).toFixed(0)}%`;
} }
// Delta in percentage points vs the baseline, signed ("+12" / "5"). // Deltas come from the server with their margin of error (#2495). The client
function deltaPts(value: number, baseline: number): string { // no longer subtracts rates itself: the margin needs the sample sizes and
const pts = Math.round((value - baseline) * 100); // variances, and having both clients re-derive it invites them to disagree.
return pts > 0 ? `+${pts}` : `${pts}`; //
// A delta that isn't distinguishable from zero is prefixed "≈" and dimmed.
// That is the point of this whole change — the card used to render a 12 on
// 59 plays exactly as boldly as a 6 on 400, and the first of those is noise.
function deltaText(d: MetricDelta | undefined): string {
if (!d) return '';
const pts = Math.round(d.delta_pp);
const signed = pts > 0 ? `+${pts}` : `${pts}`;
return d.distinguishable ? signed : `≈${signed}`;
} }
// A surface's skip delta is "worse" when it skips more than the function deltaTitle(d: MetricDelta | undefined): string | undefined {
// baseline; completion delta is "worse" when it completes less. if (!d) return undefined;
function skipDeltaClass(m: SurfaceMetric, baseline: SurfaceMetric): string { const range = `${d.delta_pp.toFixed(1)} ± ${d.margin_pp.toFixed(1)} points vs baseline`;
return m.skip_rate > baseline.skip_rate ? 'text-danger' : 'text-text-secondary'; return d.distinguishable
? `${range} (95% confidence)`
: `${range} — not distinguishable from zero at 95% confidence, so read this as no measured difference.`;
} }
function completionDeltaClass(m: SurfaceMetric, baseline: SurfaceMetric): string { // A skip delta is "worse" above the baseline; a completion delta is "worse"
return m.avg_completion < baseline.avg_completion ? 'text-danger' : 'text-text-secondary'; // below it. Neither gets a colour unless it's distinguishable — colouring
// noise red is what made the old card misleading.
function skipDeltaClass(d: MetricDelta | undefined): string {
if (!d?.distinguishable) return 'text-text-secondary opacity-60';
return d.delta_pp > 0 ? 'text-danger' : 'text-text-secondary';
}
function completionDeltaClass(d: MetricDelta | undefined): string {
if (!d?.distinguishable) return 'text-text-secondary opacity-60';
return d.delta_pp < 0 ? 'text-danger' : 'text-text-secondary';
} }
// Pick-kind breakdowns are collapsed by default (#1270): with every // Pick-kind breakdowns are collapsed by default (#1270): with every
@@ -384,18 +403,18 @@
<td class="py-1 text-right tabular-nums">{m.plays}</td> <td class="py-1 text-right tabular-nums">{m.plays}</td>
<td class="py-1 text-right tabular-nums"> <td class="py-1 text-right tabular-nums">
{pct(m.skip_rate)} {pct(m.skip_rate)}
{#if baseline} {#if m.skip_delta}
<span class="ml-1 text-xs {skipDeltaClass(m, baseline)}"> <span class="ml-1 text-xs {skipDeltaClass(m.skip_delta)}"
{deltaPts(m.skip_rate, baseline.skip_rate)} data-testid="skip-delta-{m.key}"
</span> title={deltaTitle(m.skip_delta)}>{deltaText(m.skip_delta)}</span>
{/if} {/if}
</td> </td>
<td class="py-1 text-right tabular-nums"> <td class="py-1 text-right tabular-nums">
{pct(m.avg_completion)} {pct(m.avg_completion)}
{#if baseline} {#if m.completion_delta}
<span class="ml-1 text-xs {completionDeltaClass(m, baseline)}"> <span class="ml-1 text-xs {completionDeltaClass(m.completion_delta)}"
{deltaPts(m.avg_completion, baseline.avg_completion)} data-testid="completion-delta-{m.key}"
</span> title={deltaTitle(m.completion_delta)}>{deltaText(m.completion_delta)}</span>
{/if} {/if}
</td> </td>
</tr> </tr>
@@ -420,18 +439,18 @@
<td class="py-1 text-right text-xs tabular-nums">{b.plays}</td> <td class="py-1 text-right text-xs tabular-nums">{b.plays}</td>
<td class="py-1 text-right text-xs tabular-nums"> <td class="py-1 text-right text-xs tabular-nums">
{pct(b.skip_rate)} {pct(b.skip_rate)}
{#if baseline} {#if b.skip_delta}
<span class="ml-1 {skipDeltaClass(b, baseline)}"> <span class="ml-1 {skipDeltaClass(b.skip_delta)}"
{deltaPts(b.skip_rate, baseline.skip_rate)} data-testid="skip-delta-{b.key}"
</span> title={deltaTitle(b.skip_delta)}>{deltaText(b.skip_delta)}</span>
{/if} {/if}
</td> </td>
<td class="py-1 text-right text-xs tabular-nums"> <td class="py-1 text-right text-xs tabular-nums">
{pct(b.avg_completion)} {pct(b.avg_completion)}
{#if baseline} {#if b.completion_delta}
<span class="ml-1 {completionDeltaClass(b, baseline)}"> <span class="ml-1 {completionDeltaClass(b.completion_delta)}"
{deltaPts(b.avg_completion, baseline.avg_completion)} data-testid="completion-delta-{b.key}"
</span> title={deltaTitle(b.completion_delta)}>{deltaText(b.completion_delta)}</span>
{/if} {/if}
</td> </td>
</tr> </tr>
@@ -442,6 +461,12 @@
</table> </table>
</div> </div>
{/each} {/each}
<p class="text-xs text-text-secondary">
Deltas compare each surface with your manual plays. A delta marked
<span class="opacity-60"></span> is smaller than its own margin of error at this
sample size — it can't be told apart from no difference, however big it looks.
Hover any delta for its range.
</p>
{:else} {:else}
<p class="text-sm text-text-secondary"> <p class="text-sm text-text-secondary">
No plays recorded yet. Play something from For You, Discover, or a mix. No plays recorded yet. Play something from For You, Discover, or a mix.
+68
View File
@@ -241,6 +241,74 @@ describe('Settings page — Recommendation metrics card', () => {
expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument(); expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument();
}); });
// #2495: the card used to render a delta computed client-side with no notion
// of uncertainty, so a -12 on 59 plays looked exactly as solid as a -6 on 400.
// Deltas now arrive from the server with a margin, and an indistinguishable
// one is marked with "≈" and dimmed rather than coloured.
test('a delta smaller than its margin is marked as indistinguishable', async () => {
setupPage();
metricsMock.data = {
window_days: 30,
baseline: metric('manual', 'Manual library plays', { plays: 400, skip_rate: 0.27 }),
groups: [
{
intent: 'discovery',
label: 'Discovery mixes',
surfaces: [
metric('discover', 'Discover', {
plays: 59,
skip_rate: 0.153,
// 13.3pp gap, but the margin at n=59 is wider than the gap.
skip_delta: { delta_pp: -13.3, margin_pp: 14.5, distinguishable: false },
completion_delta: { delta_pp: 28.0, margin_pp: 12.1, distinguishable: true }
})
]
}
]
};
render(SettingsPage);
await waitFor(() => expect(screen.getByText('Discover')).toBeInTheDocument());
// Targeted by test id rather than text: the legend below the table also
// contains a "≈", so matching on the glyph finds the explanation instead of
// the delta. (It did, on the first attempt at this test.)
const skip = screen.getByTestId('skip-delta-discover');
expect(skip).toHaveTextContent('≈-13');
expect(skip).toHaveAttribute('title', expect.stringContaining('not distinguishable from zero'));
// It must NOT be coloured as a real regression/improvement.
expect(skip.className).toContain('opacity-60');
// The completion delta clears its margin, so it renders plainly.
const completion = screen.getByTestId('completion-delta-discover');
expect(completion).toHaveTextContent('+28');
expect(completion.className).not.toContain('opacity-60');
// And the legend explains the glyph rather than leaving it a mystery.
expect(screen.getByText(/smaller than its own margin of error/i)).toBeInTheDocument();
});
// A delta is omitted entirely when the samples are too thin for a margin to
// mean anything — the server decides that, and the cell must simply show the
// rate rather than a bare "0".
test('a surface with no delta shows its rate and nothing else', async () => {
setupPage();
metricsMock.data = {
window_days: 30,
baseline: metric('manual', 'Manual library plays', { plays: 400 }),
groups: [
{
intent: 'go_to',
label: 'Go-to surfaces',
surfaces: [metric('radio', 'Radio', { plays: 1, skip_rate: 0 })]
}
]
};
render(SettingsPage);
await waitFor(() => expect(screen.getByText('Radio')).toBeInTheDocument());
expect(screen.queryByTestId('skip-delta-radio')).not.toBeInTheDocument();
expect(screen.queryByTestId('completion-delta-radio')).not.toBeInTheDocument();
});
test('surfaces without a breakdown render no toggle and no sub-rows', async () => { test('surfaces without a breakdown render no toggle and no sub-rows', async () => {
setupPage(); setupPage();
metricsMock.data = { metricsMock.data = {