The metrics card had one volume threshold doing two jobs.
recMetricsLowVolume = 20 is a DISPLAY floor — below that a skip rate is
anecdote — but the card then presented deltas as though it were also a
DECISION floor. Those differ by an order of magnitude: detecting the ~13pp
differences that matter needs ~133 plays per arm for 80% power at a=0.05.
So Discover's taste-matched (59 plays) and random-unheard (70) both rendered
as full-confidence rows with a bold delta beside them, and that comparison
sits at p ~ 0.06. The card said "signal"; the arithmetic said "maybe". It
produced a recommendation the data didn't support, and any reader with the
same numbers would have made the same call.
Deltas now carry a 95% margin of error and a `distinguishable` flag, computed
server-side so both clients read the same arithmetic instead of each
re-deriving it. Skip rate is a two-proportion difference; completion is
Welch, which needs a variance — hence completion_sqsum in the query. It is
the sum of squares rather than stddev_samp on purpose: raw source rows are
merged into surface families in Go, and sums of squares combine across groups
exactly whereas standard deviations cannot.
recMetricsLowVolume is untouched. "Too thin to show" and "too thin to act on"
are different questions.
Web renders an indistinguishable delta as dimmed and prefixed "≈", with the
range on hover and a legend explaining the glyph. Colour is withheld unless
the delta clears its margin — colouring noise red is what made the old card
misleading. Breakdown rows go through the same path; those are the thinnest
samples on screen and where the old card misled most.
Also fixes the admin trends view, which had the same problem worse: its
"Latest skip"/"Latest completion" columns are one WEEK while the adjacent
Plays column is the whole window. I misread exactly that and briefly
concluded Deep cuts was the worst surface, from ~17 plays in a single week —
over 180 days it is one of the best. Headers now name their period and the
skip cell carries that week's play count.
#2524: resolveArtist now recognises a duplicate-MBID unique violation as the
expected condition it is, matching resolveAlbum. Two rows mapping to one
MusicBrainz artist is a merge candidate, not a fault; without the branch it
logged a generic warning plus a Postgres ERROR line on every scan, which
teaches an operator to ignore database errors.
Track identity was file_path, so a file that came back renamed or in a
different directory looked like a deletion plus an unrelated new track:
the old row kept the like and every play_event while a fresh zero-history
row appeared, and nothing connected them. A liked song read as unliked, its
play count reset, and Rediscover could offer it as a discovery — silently.
Renumbering an album was enough, which is what happened to the operator's
copy of Minutes to Midnight.
Adoption re-points the existing row's file_path at the new location and
clears its missing mark. The normal UpsertTrack then conflicts on file_path
and updates THAT row, so the track id survives and likes, plays and
playlist memberships travel with it — and clients see an update rather than
a delete-and-create, so no cache churn either.
Matching is MBID first (identifies the recording, so it survives a
re-encode), then file_size + duration_ms for untagged files. Both
fingerprint components must be non-zero: duration_ms is 0 when ffprobe
failed, and matching 0 against 0 would pair up unrelated broken files.
Only rows already marked missing are eligible — a row whose file is present
elsewhere is a duplicate, not a move, and re-pointing it would corrupt the
copy that still exists. An ambiguous match inserts fresh rather than
adopting one arbitrarily: a fork is recoverable later, a wrong merge isn't.
Scan is now three phases, and the order is the point. Adoption can only
claim a row that is ALREADY marked missing, but reconcile previously ran
after processing — so a rename performed while the server was down surfaced
the deletion and the addition in the same scan, the new path inserted first,
and the fork became permanent. Enumeration is therefore separated from
processing so reconcile can run between them: walk (paths only, no tag
reads or probes) -> reconcile -> process in walk order.
Consequence worth knowing: when reconcile refuses (an absent root, or a
reorganisation exceeding the 25% mark cap) adoption cannot fire and renamed
files fork as before. That's the pre-#2528 behaviour rather than a new
failure, and the warning now names it.
The old outer walk-error branch was unreachable — the callback always
returned nil, so WalkDir never surfaced an error — and verifyRootsPresent is
the real protection, so enumerate counts walk errors instead of pretending
to abort on them.
Nothing in Minstrel ever noticed a deleted file. The walk only visits
paths that exist, so a row whose file was gone was never scanned, never
errored, never counted — permanently invisible. classifyEvent ignores
fsnotify removals by design, and the safety-net scan is the same walk, so
it covers additions only. Rows accumulated forever.
Found on the operator's library: a completed scan reported
skipped=24185 errored=0 while the MBID backfill (which opens files by DB
path rather than walking) logged ~40 "no such file or directory" across
three reorganised albums. Those rows also kept their pre-#2499 welded
genre, which is how this surfaced — the version-stamped tag re-read can
only reach files the walk visits.
The harm is not cosmetic. tracks is the candidate universe for
recommendation.sql / discover.sql / system_mixes.sql and nothing filtered
on file existence, so a mix could spend a slot on a track that cannot
stream.
Marks rather than deletes. A missing file is a claim about the filesystem
and the filesystem lies transiently — an unmounted volume, a network
blip, a container that started before its media mount attached. Every
sweep in internal/gc resolves a truth INSIDE the database and is safe to
run blind; this one is not, so no deletion happens here. Three guards
refuse to act on ambiguous evidence: every scan root must resolve to a
non-empty directory, the walk must have seen at least one file, and one
reconcile may newly mark at most 25% of the library. Clearing a mark is
never the dangerous direction, so it runs unconditionally — otherwise a
library that tripped the cap could never recover once the mount returned.
Only a full Scan reconciles. The walk's set of seen paths is the
evidence, and ScanFiles has no basis for concluding anything about files
it did not look at.
Excludes marked tracks from all 13 track-emitting queries (radio x2,
system mixes x5, discover x4, most-played x2), the 6 play-history seed
picks, and the genre browse axis. Deliberately NOT filtered: the shared
ListPlaylistTracks read path, because it also serves user-curated
playlists where hiding a track the user added would be wrong — system
playlists shed orphans on their next daily rebuild instead. History and
the taste profile also keep them: those record the past, and a track you
played 200 times still says something about your taste.
Reconcile tallies land in scan_runs so a disappearance is visible rather
than discovered when a mix comes up short.
dhowden/tag's readTFrame splits ID3v2 null-separated multi-value text
frames and rejoins them with the EMPTY string, so a file tagged
"Alternative Rock" + "Rock" was stored as "Alternative RockRock". It also
leaves bare numeric ID3v1 references unresolved, which is why the
library showed genres like "4017" and "526617".
This corrupted more than the browse axis added in #367: taste_profile.sql
reads tracks.genre directly, so the welded tokens were entering the taste
profile's tag vocabulary, and recommendation.sql/discover.sql were
comparing them as single opaque tags. Genre counts were wrong everywhere.
ffprobe is not a fix — ffmpeg's read_ttag calls decode_str once with no
loop, keeping only the first value. Truncating multi-genre tags would
blunt the similarity signal genre mainly feeds. So the TCON frame is now
parsed directly (ID3v2.2/2.3/2.4, all four text encodings, per-frame and
tag-level unsynchronisation, numeric and parenthesised ID3v1 references);
everything else still comes from dhowden/tag. Values are stored
";"-delimited, which the read side already splits on, so no query changes.
Existing rows are repaired without an operator-run rebuild: migration
0054 adds tracks.tag_read_version DEFAULT 0, below the scanner's current
tagReadVersion, so the next scan re-reads tags it would otherwise skip on
mtime. Such a re-read reuses the stored duration instead of re-running
ffprobe, keeping a repair pass tag-read-bound rather than one fork+exec
per file. Bumping the constant is how a future extraction fix reaches an
existing library.
Only ID3v2 is in scope — dhowden welds nowhere else. The Vorbis/MP4
repeated-field question is #2500, unproven and deliberately not built.
Add Scanner.ScanFiles (watcher-driven targeted scan returning changed album
IDs) and a recursive fsnotify Watcher that debounces filesystem events and
enriches just the affected albums inline. Pure classifyEvent/drainPending
seams unit-tested; ScanFiles covered in the scanner integration test.
Six findings from the 2026-06-02 multi-system drift audit (Scribe
parent task #552):
- **#553 (web)** Tailwind class fix: web admin playback-errors Delete
confirm button was using `bg-action-danger`, an undefined token —
swap to `bg-action-destructive` to match every other destructive
button. Restored the Oxblood signal that distinguishes Delete from
Cancel.
- **#555 (web)** Type the `source` field on `play_started` in the
EventRequest discriminated union. Server's eventRequest accepts it;
web's TS type was missing the slot, so a "drop extra properties"
refactor could silently strip the source tag and break system-
playlist rotation attribution.
- **#556 + #557 (server)** Coverage rollup whitelist was pinned to
('sidecar','embedded','mbcaa','theaudiodb'); migration 0020 added
'deezer' and 'lastfm' as valid cover_art_source values but those
never got wired in, so albums with art from those providers
silently counted as MISSING in the admin Coverage dashboard. The
rollup test was seeding only the pre-0020 sources, masking the
gap in CI. Extend the query to include deezer + lastfm; seed the
test with one row per valid source (regression-guards future
additions).
- **#558 (web)** Auth gate was blocking /forgot-password and
/reset-password/<token> — both are entered without a session by
definition, so the email-link reset flow was bouncing signed-out
users to /login. Add /forgot-password to the public set and a
/reset-password/ prefix matcher. New tests assert both routes
reach their pages without redirect.
- **#571 (server)** Library scanner was indexing only .mp3/.m4a/.flac
/.ogg while the stream handler (media.go) had been extended to
serve .opus, .aac, and .wav. A user with .opus files in their
library never saw them in artist/album listings because the
scanner skipped indexing — silent data loss. Aligned the scanner
to match the media handler.
Scribe statuses updated to in_progress; flipping to done after the
push since these are mechanical and verified directly against the
cited file:lines.
tracks.mbid was 100% NULL: the scanner only extracted album + artist
MBIDs, never the recording MBID. The ListenBrainz similarity worker is
gated on tracks.mbid IS NOT NULL, so track_similarity could never
populate — starving For-You/radio's strongest candidate source
(lb_similar) on every library.
- mbids.go: add extractRecordingMBID (mbz.Recording / Picard
musicbrainz_recordingid). Separate fn so extractMBIDs' signature +
unit tests stay untouched.
- scanner.go: persist recording MBID via UpsertTrack (heals on the
file_path conflict, so re-scans backfill for free).
- BackfillTrackMBIDs: one-shot pass mirroring BackfillMBIDs, wired as
scan Stage 2b (idempotent via SetTrackMbidIfNull, gated by
BackfillCap, log-only progress).
- migration 0029: tracks_mbid_unique (0002, written when the column
was always NULL) wrongly assumes one MBID == one track. A recording
appears on multiple releases, so rows legitimately share a recording
MBID. Replace with a non-unique partial index. Zero-risk: column is
100% NULL at migration time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires sync.LogChange into the library mutation sites so /api/library/sync
reflects upserts and deletes.
Architectural pivot: LogChange's signature is now (ctx, dbq.DBTX, ...) so
it works with both *pgxpool.Pool and pgx.Tx. The scanner doesn't run
mutations in explicit transactions, so it pool-binds; delete.go matches.
Tx-bound callers (likes/playlists in subsequent commits) keep atomicity.
Also: sync.FormatUUID centralizes the pgtype.UUID → canonical string
conversion that both the scanner and the sync handler need; library_sync.go
now uses it instead of a local copy.
Best-effort logging on scanner failures (Warn, don't fail the scan): a
LogChange error after a successful upsert is rare and self-healing — the
next scan that touches the entity re-emits the change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each of the 4 stage workers (library walk, MBID backfill, cover
enrich, artist art enrich) gains a progressCb parameter that
receives a snapshot of the current running state per item processed.
The orchestrator stores the snapshot in a local var that the
publisher's marshal closure reads at flush time.
scanrun.go's 4 stage blocks are rewritten to construct one publisher
per stage, pass a Tick-firing closure to the worker as progressCb,
and call Flush() after the worker returns (routing any persist error
to captureErr — preserves today's diagnostic behavior).
The previous post-stage UpdateScanRun* calls are removed; the
publisher's Flush handles the final write. Cadence: every 500 items
OR every 1s, whichever fires first.
Outside-orchestrator callers (scanner_test.go, enricher_test.go)
pass nil for progressCb — no-op behavior preserved for tests that
don't need progress observability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Forward-fix from running the prior two commits in production. Backfill
on a real library exposed two further issues:
(1) gofmt -s flagged the column-aligned stubMeta methods in
mbids_test.go. Switched to the gofmt-canonical single-space form.
(2) Many albums failed to heal with SQLSTATE 23505 (unique constraint
albums_mbid_unique). Cause: the operator's library has duplicate
album rows in the DB — same MusicBrainz release, split into
multiple rows by the pre-MBID scanner because of subtle title or
artist disagreements between files. When backfill now correctly
extracts the MBID, two rows want to claim the same one and the
partial unique index rejects the second.
The conflict is correct DB behavior — we shouldn't have two rows
with the same MBID — but it's not a write failure to alarm the
operator about. Detect 23505 specifically:
- downgrade the log line from Warn to Info
- track these in a new BackfillMBIDsResult.Duplicates counter
(separate from Skipped, which retains its "no MBID in tag"
meaning)
- leave the duplicate row's mbid NULL; merging duplicates is a
separate (future) operator workflow
Same handling threaded through the scanner heal path so a regular
rescan doesn't generate the noise either.
JSON tally + admin UI gain a "Duplicates" line so the operator
can see how many duplicate rows their library carries.
Follow-up scope (separate task): a duplicate-album merge UX that
reparents the duplicate's tracks to the canonical row and deletes
the orphaned album row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The hand-rolled extractor in internal/library/mbids.go looked up keys that
dhowden/tag never produces, for every supported audio format:
Vorbis (FLAC/OGG): dhowden lowercases keys at parse time (vorbis.go:59),
we looked up MUSICBRAINZ_ALBUMID. Always missed.
ID3v2 (MP3): dhowden stores TXXX frames under bare TXXX/TXXX_N keys
with *tag.Comm values; the Picard tag name is on the value's
Description field, not part of the key. We looked up
TXXX:MusicBrainz Album Id. Always missed.
MP4 (M4A): dhowden stores freeform iTunes atoms under their bare
sub-name. We looked up ----:com.apple.iTunes:MusicBrainz Album Id.
Always missed.
Net effect: every album in the m7-380 boot backfill reported
processed=N, healed=0, skipped=N. The m7-379 enricher gate then
short-circuited NULL on every album, so MBCAA was never queried.
Symptom: "all albums skipped" during cover enrichment.
The pre-existing unit tests passed because they hand-built Raw() maps
with the intended-but-incorrect keys, never round-tripping through
dhowden's parser.
Replace the hand-rolled extractor with dhowden's own mbz sub-package,
which knows the per-format Raw() conventions and uses lowercase
canonical keys (mbz.Album, mbz.Artist). Update both call sites to pass
the full tag.Metadata instead of just Raw(). Rewrite the test with a
stubMeta implementing tag.Metadata, asserting against the actual
per-format Raw() shape, plus a regression guard that pins "uppercase
Vorbis keys must not match" so the v0 bug can't sneak back.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scrubbing/seeking in clients was a no-op because every track shipped
with duration_ms=0. Shell out to ffprobe (already in the image) per
file during scan and record the parsed duration. ffprobe failures are
warned + recorded as 0 so a single bad file doesn't sink the scan.
Tightened the incremental skip to also require duration_ms > 0 so
existing libraries get backfilled on the next rescan instead of
needing a wipe.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tag-supplied years were being passed straight to Postgres' date column
without validation. Files with corrupt or 5+ digit years (seen in the
wild on a couple of dozen albums) tripped SQLSTATE 22008 and the entire
album upsert failed, dropping every track on those albums from the
library.
Validate the year is within 1..9999 before constructing the date. If
it's outside that window, log a warning naming the album and year, and
insert the album with no release_date — a soft field shouldn't take
down the whole row.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bvandeusen
