docs: library comments say what the scanner and resolver do, not how they came to be (#6277)
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped

Rewrites the internal/library comment blocks to rule 32. Task, milestone,
decision and rule ids are gone; the incidents behind them are restated as
the constraint they left (why a file is deleted before its row, why the
missing mark is a mark, why duplicates block on duration). Counts and
album names from one library are dropped from production comments; test
comments keep the names their fixtures use. The tagReadVersion log keeps
each version's before and after, without the measurements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-23 12:27:34 -04:00
co-authored by Claude Opus 5.5
parent 08a2c06b85
commit 7e6228c13f
66 changed files with 578 additions and 564 deletions
+61 -65
View File
@@ -28,8 +28,7 @@ import (
// `internal/api/media.go` MIME detection — the stream handler must be
// able to serve every extension the scanner indexes, and there is no
// point in adding extensions to the stream handler that the scanner
// will silently skip. Drift #571 caught the divergence after .opus,
// .aac, and .wav were added to media.go but not here.
// will silently skip.
var audioExtensions = map[string]bool{
".mp3": true,
".m4a": true,
@@ -43,7 +42,7 @@ var audioExtensions = map[string]bool{
// tagReadVersion is the version of this package's tag-extraction logic. Rows
// whose tracks.tag_read_version is lower get their tags re-read on the next
// scan even when the file itself hasn't changed, so a fix reaches an existing
// library without the operator rebuilding it (migration 0054).
// library without anyone rebuilding it (migration 0054).
//
// Bump this whenever a change to tag extraction should reach already-indexed
// files, and say why below.
@@ -52,35 +51,33 @@ var audioExtensions = map[string]bool{
// dhowden/tag welds null-separated multi-values into one token
// ("Alternative Rock" + "Rock" -> "Alternative RockRock"), which corrupted
// the genre browse axis and polluted the taste profile's tag vocabulary,
// and left bare ID3v1 numeric references unresolved (#2499).
// 2: acronym and apostrophe casing repaired on genre values (#2468) — "Edm" ->
// and left bare ID3v1 numeric references unresolved.
// 2: acronym and apostrophe casing repaired on genre values — "Edm" ->
// "EDM", "Children'S Music" -> "Children's Music". Bumped rather than left
// to new files only because taste_profile.sql reads tracks.genre directly,
// so a half-repaired library would carry both spellings as separate tags.
// 3: the MusicBrainz release-group id is read into albums.release_group_mbid
// (#5241). Wants name albums by release group, not by the release id
// 3: the MusicBrainz release-group id is read into albums.release_group_mbid.
// Wants name albums by release group, not by the release id
// albums.mbid holds, so re-acquisition and request completion need it for
// the albums already in the library, not only for files added from now on.
// 4: every value of a repeated genre field is read for FLAC, Ogg Vorbis, Opus
// and MP4 (#2500). dhowden/tag kept only the last, and 3,658 of the
// operator's 4,092 FLACs declare more than one, so most of their genres
// were never stored.
// 5: the recording id is read from FLAC, Ogg Vorbis and Opus (#5637). Picard
// and MP4. dhowden/tag kept only the last, and most FLACs that carry
// genres declare more than one.
// 5: the recording id is read from FLAC, Ogg Vorbis and Opus. Picard
// calls it MUSICBRAINZ_TRACKID there and dhowden/tag files that under the
// release-track name, so every Vorbis-tagged track came in without its
// tag id and waited on an AcoustID lookup. A re-read stores the tag id,
// over a looked-up one (D4).
// 6: a Vorbis TRACKNUMBER or DISCNUMBER written as "3/20" is read (#6171).
// over a looked-up one.
// 6: a Vorbis TRACKNUMBER or DISCNUMBER written as "3/20" is read.
// dhowden/tag reads it as 0, and Minstrel's own importer wrote it that
// way, so all 245 FLAC tracks it imported came in with no track or disc
// number and their albums lost their order.
// way, so the tracks it imported came in with no track or disc number.
// 7: an album of the same name in another release group gets its own row
// (resolveAlbum). Weezer's six self-titled albums were one 118-track row,
// so their merges, wants and upgrades all counted as one album; a re-read
// files each existing track under its own album.
// (resolveAlbum). Self-titled albums by one artist shared a row, so their
// merges, wants and upgrades all counted as one album; a re-read files
// each existing track under its own album.
// 8: a file tagged with no release group joins the same-named album that
// already holds a track from its folder. Under 7 it took the oldest row,
// which left three of Weezer's 1994 bonus tracks on the 2016 album.
// which could file a bonus track on a later album of the same name.
const tagReadVersion int16 = 8
type Stats struct {
@@ -89,31 +86,31 @@ type Stats struct {
Updated int `json:"updated"`
Skipped int `json:"skipped"`
Errored int `json:"errored"`
// Missing / Restored come from the reconcile pass, not the walk (#2523):
// Missing / Restored come from the reconcile pass, not the walk:
// rows whose file the walk didn't find, and rows whose file came back.
// Only a full Scan sets these — see reconcileMissing.
Missing int `json:"missing"`
Restored int `json:"restored"`
// marked are the tracks this scan's reconcile marked missing, so the scan
// can tell admins which of them are still gone once it has adopted moves
// and folded replacements (M542).
// and folded replacements.
marked []pgtype.UUID
}
type Scanner struct {
pool *pgxpool.Pool
logger *slog.Logger
// paths are the library's folders, changed by SetRoots (#6129). Read them
// paths are the library's folders, changed by SetRoots. Read them
// through Roots.
paths []string
rootsMu sync.RWMutex
rootsChanged chan struct{}
// fingerprint derives a file's acoustic identity (M400). A field so an
// fingerprint derives a file's acoustic identity. A field so an
// integration test can substitute a deterministic one: CI has no real audio
// to fingerprint, and what the test pins is WHEN the scan fingerprints, not
// what the tools print. Call it through fingerprintFile.
fingerprint func(ctx context.Context, path string, opts fingerprintOptions) fingerprintResult
// settings is the operator's fingerprinting policy (#3913), shared with the
// settings is the fingerprinting policy, shared with the
// workers and the admin API. Nil fingerprints with the defaults.
settings *FingerprintSettingsService
}
@@ -128,7 +125,7 @@ func New(pool *pgxpool.Pool, logger *slog.Logger, paths []string, settings *Fing
// are logged + counted; the scan keeps going.
//
// It then reconciles: rows whose file the walk never saw get marked missing,
// and rows whose file has come back get un-marked (#2523). Only a FULL scan may
// and rows whose file has come back get un-marked. Only a FULL scan may
// do this — the walk's set of seen paths is the evidence, and a partial
// (watcher-driven) scan has no basis for concluding anything about files it
// didn't look at. That's why ScanFiles does not reconcile.
@@ -141,23 +138,24 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
start := time.Now()
// PHASE 1 — enumerate. Collect every audio path without touching tags or
// ffprobe. Cheap: WalkDir already stats each entry, so this adds a directory
// traversal and nothing else.
// ffprobe. Cheap: WalkDir already stats each entry, so this adds a
// directory traversal and nothing else.
//
// The order matters and is the whole reason enumeration is separate.
// Reconcile has to mark disappeared rows BEFORE any file is processed,
// because move detection (#2528) can only adopt a row that is already marked
// missing. A rename performed while the server was down surfaces the deletion
// and the addition in the SAME scan — so if reconcile ran at the end, the new
// path would insert a fresh row first and the fork would be permanent.
// because move detection can only adopt a row that is already marked
// missing. A rename performed while the server was down surfaces the
// deletion and the addition in the SAME scan — so if reconcile ran at the
// end, the new path would insert a fresh row first and the fork would be
// permanent.
paths, walkErrs := s.enumerate(ctx, progressCb, &stats)
stats.Errored += walkErrs
if err := ctx.Err(); err != nil {
return stats, err
}
// PHASE 2 — reconcile. Only ever on a COMPLETE enumeration: a cancelled walk
// has a partial view and would mark everything it hadn't reached.
// PHASE 2 — reconcile. Only ever on a COMPLETE enumeration: a cancelled
// walk has a partial view and would mark everything it hadn't reached.
seen := make(map[string]struct{}, len(paths))
for _, p := range paths {
seen[p] = struct{}{}
@@ -167,10 +165,9 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
// evidence, and that refusal arrives here as an error.
//
// The consequence is named explicitly because it is not obvious: move
// detection (#2528) can only adopt a row that is already marked missing,
// so a refused reconcile also means renamed files insert fresh rows and
// fork their history. That's the pre-#2528 behaviour rather than a new
// failure, but it's worth knowing which scan it happened on. It bites
// detection can only adopt a row that is already marked missing, so a
// refused reconcile also means renamed files insert fresh rows and fork
// their history. It's worth knowing which scan it happened on. It bites
// hardest when a large fraction of a small library is reorganised at
// once, which trips the mark cap.
s.logger.Warn("library scan: reconcile skipped — moved files will fork rather than adopt",
@@ -207,7 +204,7 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
}
// PHASE 4 — fold missing rows into replacements this scan brought in or
// that were already here (M485). After processing, so both this scan's
// that were already here. After processing, so both this scan's
// marks and its arrivals are in. Not fatal: an unfolded pair is only a row
// left in the missing list, and the next scan tries again.
if _, err := PairMissingTracks(ctx, s.pool, s.logger); err != nil && ctx.Err() == nil {
@@ -222,18 +219,18 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
// enumerate walks every configured root and returns the audio paths found, in
// walk order, plus a count of walk errors.
//
// A path is recorded even if it will later fail to parse: an unreadable file is a
// broken file, not a missing one, and letting reconcile mark it missing would
// hide it from the operator behind the wrong explanation.
// A path is recorded even if it will later fail to parse: an unreadable file is
// a broken file, not a missing one, and letting reconcile mark it missing would
// hide it from an admin behind the wrong explanation.
func (s *Scanner) enumerate(
ctx context.Context, progressCb func(Stats), stats *Stats,
) ([]string, int) {
paths := make([]string, 0, 8192)
errs := 0
for _, root := range s.Roots() {
// WalkDir's own error return is folded into the per-entry handler below,
// so a bad root is counted rather than aborting the whole scan — one
// unreadable root shouldn't discard the others' results.
// WalkDir's own error return is folded into the per-entry handler
// below, so a bad root is counted rather than aborting the whole scan —
// one unreadable root shouldn't discard the others' results.
_ = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if ctx.Err() != nil {
return fs.SkipAll
@@ -280,11 +277,10 @@ func (s *Scanner) scanFile(
}
// Incremental skip: only when the file hasn't changed AND we already have a
// real duration AND the row's tag-derived columns were written by the
// current extraction logic. The duration clause lets older scans that
// recorded duration_ms=0 (before ffprobe was wired) get backfilled without
// forcing the operator to wipe the library; the tag-version clause does the
// same job for tag-extraction fixes (#2499). Once both are current,
// subsequent scans short-circuit as before.
// current extraction logic. The duration clause backfills rows that
// recorded duration_ms=0 without anyone wiping the library; the
// tag-version clause does the same job for tag-extraction fixes. Once both
// are current, subsequent scans short-circuit.
unchanged := knownTrack && !existing.UpdatedAt.Time.Before(mtime)
if unchanged && existing.DurationMs > 0 && existing.TagReadVersion >= tagReadVersion {
stats.Skipped++
@@ -354,15 +350,15 @@ func (s *Scanner) scanFile(
// above — a tagReadVersion bump must stay bound by tag reads, not become a
// decode of the whole library.
//
// An unchanged file with NO fingerprint yet is the backfill's job (#3908),
// An unchanged file with NO fingerprint yet is the backfill's job,
// deliberately not the scan's. Folding it into the skip check would make
// the first scan after an upgrade re-decode every track and push a sync
// change to every client for each one.
//
// Computed before move adoption so adoption can match on the audio hash
// (#3914); stored after the upsert, once the row id is known.
// Computed before move adoption so adoption can match on the audio hash;
// stored after the upsert, once the row id is known.
//
// With fingerprinting switched off (#3913) the scan decodes nothing, but it
// With fingerprinting switched off the scan decodes nothing, but it
// still takes the stream hash: a demux rather than a decode, and the only
// thing that recognises a moved file with no MBID.
var fp fingerprintResult
@@ -376,13 +372,13 @@ func (s *Scanner) scanFile(
}
// A path we've never seen might not be a new track — it might be one that
// moved or was renamed (#2528). Adopting re-points the existing row at this
// moved or was renamed. Adopting re-points the existing row at this
// path and clears its missing mark, so the UpsertTrack below conflicts on
// file_path and updates THAT row: same track id, likes and play history
// intact. Without this, renumbering an album forks every track on it.
//
// Runs after fingerprinting because, for a file with no MBID, adoption matches
// on its audio hash (#3914), and only for genuinely unknown paths — a known
// Runs after fingerprinting because, for a file with no MBID, adoption
// matches on its audio hash, and only for genuinely unknown paths — a known
// path is already the row we're going to update.
if !knownTrack {
if s.adoptMovedTrack(ctx, q, path, fp.streamSHA256, recordingMBID) {
@@ -444,7 +440,8 @@ func (s *Scanner) scanFile(
storeFingerprint(ctx, q, s.logger, track.ID, path, fp, fpCfg.ChromaprintLengthSec)
} else if err := q.DeleteTrackFingerprint(ctx, track.ID); err != nil {
// Off, nothing is stored — but a row describing the previous bytes
// must not outlive them, or the sweep would compare audio that is gone.
// must not outlive them, or the sweep would compare audio that is
// gone.
s.logger.Warn("fingerprint: clearing stale fingerprint failed", "path", path, "err", err)
}
// Loudness is measured by its own worker, never inline: it decodes the
@@ -454,7 +451,7 @@ func (s *Scanner) scanFile(
if err := q.DeleteTrackLoudness(ctx, track.ID); err != nil {
s.logger.Warn("loudness: clearing stale measurement failed", "path", path, "err", err)
}
// Likewise an AcoustID lookup (M401): the worker looks the new bytes
// Likewise an AcoustID lookup: the worker looks the new bytes
// up again. A looked-up MBID stays meanwhile and is taken back only if
// the new lookup does not match it.
if err := q.DeleteAcoustIDLookup(ctx, track.ID); err != nil {
@@ -507,7 +504,7 @@ func (s *Scanner) ScanFiles(ctx context.Context, paths []string) ([]pgtype.UUID,
"errored", stats.Errored,
)
// A file the watcher just brought in may be the replacement for a row
// an earlier scan marked missing (M485).
// an earlier scan marked missing.
if _, err := PairMissingTracks(ctx, s.pool, s.logger); err != nil && ctx.Err() == nil {
s.logger.Warn("library watch: missing-pair pass failed", "err", err)
}
@@ -528,12 +525,11 @@ func (s *Scanner) resolveArtist(ctx context.Context, q *dbq.Queries, name, mbid
if isUniqueViolation(uerr) {
// Another artist row already owns this MBID — two rows that
// should be merged (usually two spellings of one name).
// Expected, not a fault: leave NULL and let the operator
// merge. Mirrors resolveAlbum, which has always handled it
// this way — without this branch the identical benign
// condition logged a generic warning plus a Postgres ERROR
// line on every scan, which teaches an operator to ignore
// database errors (#2524).
// Expected, not a fault: leave NULL and let an admin
// merge. Mirrors resolveAlbum. Logged at Info, because a
// warning plus a Postgres ERROR line on every scan for a
// benign condition teaches an admin to ignore database
// errors.
s.logger.Info("library scan: duplicate artist mbid (canonical row already owns it)",
"artist_id", existing.ID, "artist", name, "mbid", mbid)
} else {