docs: library comments say what the scanner and resolver do, not how they came to be (#6277)
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped
Rewrites the internal/library comment blocks to rule 32. Task, milestone, decision and rule ids are gone; the incidents behind them are restated as the constraint they left (why a file is deleted before its row, why the missing mark is a mark, why duplicates block on duration). Counts and album names from one library are dropped from production comments; test comments keep the names their fixtures use. The tagReadVersion log keeps each version's before and after, without the measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+61
-65
@@ -28,8 +28,7 @@ import (
|
||||
// `internal/api/media.go` MIME detection — the stream handler must be
|
||||
// able to serve every extension the scanner indexes, and there is no
|
||||
// point in adding extensions to the stream handler that the scanner
|
||||
// will silently skip. Drift #571 caught the divergence after .opus,
|
||||
// .aac, and .wav were added to media.go but not here.
|
||||
// will silently skip.
|
||||
var audioExtensions = map[string]bool{
|
||||
".mp3": true,
|
||||
".m4a": true,
|
||||
@@ -43,7 +42,7 @@ var audioExtensions = map[string]bool{
|
||||
// tagReadVersion is the version of this package's tag-extraction logic. Rows
|
||||
// whose tracks.tag_read_version is lower get their tags re-read on the next
|
||||
// scan even when the file itself hasn't changed, so a fix reaches an existing
|
||||
// library without the operator rebuilding it (migration 0054).
|
||||
// library without anyone rebuilding it (migration 0054).
|
||||
//
|
||||
// Bump this whenever a change to tag extraction should reach already-indexed
|
||||
// files, and say why below.
|
||||
@@ -52,35 +51,33 @@ var audioExtensions = map[string]bool{
|
||||
// dhowden/tag welds null-separated multi-values into one token
|
||||
// ("Alternative Rock" + "Rock" -> "Alternative RockRock"), which corrupted
|
||||
// the genre browse axis and polluted the taste profile's tag vocabulary,
|
||||
// and left bare ID3v1 numeric references unresolved (#2499).
|
||||
// 2: acronym and apostrophe casing repaired on genre values (#2468) — "Edm" ->
|
||||
// and left bare ID3v1 numeric references unresolved.
|
||||
// 2: acronym and apostrophe casing repaired on genre values — "Edm" ->
|
||||
// "EDM", "Children'S Music" -> "Children's Music". Bumped rather than left
|
||||
// to new files only because taste_profile.sql reads tracks.genre directly,
|
||||
// so a half-repaired library would carry both spellings as separate tags.
|
||||
// 3: the MusicBrainz release-group id is read into albums.release_group_mbid
|
||||
// (#5241). Wants name albums by release group, not by the release id
|
||||
// 3: the MusicBrainz release-group id is read into albums.release_group_mbid.
|
||||
// Wants name albums by release group, not by the release id
|
||||
// albums.mbid holds, so re-acquisition and request completion need it for
|
||||
// the albums already in the library, not only for files added from now on.
|
||||
// 4: every value of a repeated genre field is read for FLAC, Ogg Vorbis, Opus
|
||||
// and MP4 (#2500). dhowden/tag kept only the last, and 3,658 of the
|
||||
// operator's 4,092 FLACs declare more than one, so most of their genres
|
||||
// were never stored.
|
||||
// 5: the recording id is read from FLAC, Ogg Vorbis and Opus (#5637). Picard
|
||||
// and MP4. dhowden/tag kept only the last, and most FLACs that carry
|
||||
// genres declare more than one.
|
||||
// 5: the recording id is read from FLAC, Ogg Vorbis and Opus. Picard
|
||||
// calls it MUSICBRAINZ_TRACKID there and dhowden/tag files that under the
|
||||
// release-track name, so every Vorbis-tagged track came in without its
|
||||
// tag id and waited on an AcoustID lookup. A re-read stores the tag id,
|
||||
// over a looked-up one (D4).
|
||||
// 6: a Vorbis TRACKNUMBER or DISCNUMBER written as "3/20" is read (#6171).
|
||||
// over a looked-up one.
|
||||
// 6: a Vorbis TRACKNUMBER or DISCNUMBER written as "3/20" is read.
|
||||
// dhowden/tag reads it as 0, and Minstrel's own importer wrote it that
|
||||
// way, so all 245 FLAC tracks it imported came in with no track or disc
|
||||
// number and their albums lost their order.
|
||||
// way, so the tracks it imported came in with no track or disc number.
|
||||
// 7: an album of the same name in another release group gets its own row
|
||||
// (resolveAlbum). Weezer's six self-titled albums were one 118-track row,
|
||||
// so their merges, wants and upgrades all counted as one album; a re-read
|
||||
// files each existing track under its own album.
|
||||
// (resolveAlbum). Self-titled albums by one artist shared a row, so their
|
||||
// merges, wants and upgrades all counted as one album; a re-read files
|
||||
// each existing track under its own album.
|
||||
// 8: a file tagged with no release group joins the same-named album that
|
||||
// already holds a track from its folder. Under 7 it took the oldest row,
|
||||
// which left three of Weezer's 1994 bonus tracks on the 2016 album.
|
||||
// which could file a bonus track on a later album of the same name.
|
||||
const tagReadVersion int16 = 8
|
||||
|
||||
type Stats struct {
|
||||
@@ -89,31 +86,31 @@ type Stats struct {
|
||||
Updated int `json:"updated"`
|
||||
Skipped int `json:"skipped"`
|
||||
Errored int `json:"errored"`
|
||||
// Missing / Restored come from the reconcile pass, not the walk (#2523):
|
||||
// Missing / Restored come from the reconcile pass, not the walk:
|
||||
// rows whose file the walk didn't find, and rows whose file came back.
|
||||
// Only a full Scan sets these — see reconcileMissing.
|
||||
Missing int `json:"missing"`
|
||||
Restored int `json:"restored"`
|
||||
// marked are the tracks this scan's reconcile marked missing, so the scan
|
||||
// can tell admins which of them are still gone once it has adopted moves
|
||||
// and folded replacements (M542).
|
||||
// and folded replacements.
|
||||
marked []pgtype.UUID
|
||||
}
|
||||
|
||||
type Scanner struct {
|
||||
pool *pgxpool.Pool
|
||||
logger *slog.Logger
|
||||
// paths are the library's folders, changed by SetRoots (#6129). Read them
|
||||
// paths are the library's folders, changed by SetRoots. Read them
|
||||
// through Roots.
|
||||
paths []string
|
||||
rootsMu sync.RWMutex
|
||||
rootsChanged chan struct{}
|
||||
// fingerprint derives a file's acoustic identity (M400). A field so an
|
||||
// fingerprint derives a file's acoustic identity. A field so an
|
||||
// integration test can substitute a deterministic one: CI has no real audio
|
||||
// to fingerprint, and what the test pins is WHEN the scan fingerprints, not
|
||||
// what the tools print. Call it through fingerprintFile.
|
||||
fingerprint func(ctx context.Context, path string, opts fingerprintOptions) fingerprintResult
|
||||
// settings is the operator's fingerprinting policy (#3913), shared with the
|
||||
// settings is the fingerprinting policy, shared with the
|
||||
// workers and the admin API. Nil fingerprints with the defaults.
|
||||
settings *FingerprintSettingsService
|
||||
}
|
||||
@@ -128,7 +125,7 @@ func New(pool *pgxpool.Pool, logger *slog.Logger, paths []string, settings *Fing
|
||||
// are logged + counted; the scan keeps going.
|
||||
//
|
||||
// It then reconciles: rows whose file the walk never saw get marked missing,
|
||||
// and rows whose file has come back get un-marked (#2523). Only a FULL scan may
|
||||
// and rows whose file has come back get un-marked. Only a FULL scan may
|
||||
// do this — the walk's set of seen paths is the evidence, and a partial
|
||||
// (watcher-driven) scan has no basis for concluding anything about files it
|
||||
// didn't look at. That's why ScanFiles does not reconcile.
|
||||
@@ -141,23 +138,24 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
|
||||
start := time.Now()
|
||||
|
||||
// PHASE 1 — enumerate. Collect every audio path without touching tags or
|
||||
// ffprobe. Cheap: WalkDir already stats each entry, so this adds a directory
|
||||
// traversal and nothing else.
|
||||
// ffprobe. Cheap: WalkDir already stats each entry, so this adds a
|
||||
// directory traversal and nothing else.
|
||||
//
|
||||
// The order matters and is the whole reason enumeration is separate.
|
||||
// Reconcile has to mark disappeared rows BEFORE any file is processed,
|
||||
// because move detection (#2528) can only adopt a row that is already marked
|
||||
// missing. A rename performed while the server was down surfaces the deletion
|
||||
// and the addition in the SAME scan — so if reconcile ran at the end, the new
|
||||
// path would insert a fresh row first and the fork would be permanent.
|
||||
// because move detection can only adopt a row that is already marked
|
||||
// missing. A rename performed while the server was down surfaces the
|
||||
// deletion and the addition in the SAME scan — so if reconcile ran at the
|
||||
// end, the new path would insert a fresh row first and the fork would be
|
||||
// permanent.
|
||||
paths, walkErrs := s.enumerate(ctx, progressCb, &stats)
|
||||
stats.Errored += walkErrs
|
||||
if err := ctx.Err(); err != nil {
|
||||
return stats, err
|
||||
}
|
||||
|
||||
// PHASE 2 — reconcile. Only ever on a COMPLETE enumeration: a cancelled walk
|
||||
// has a partial view and would mark everything it hadn't reached.
|
||||
// PHASE 2 — reconcile. Only ever on a COMPLETE enumeration: a cancelled
|
||||
// walk has a partial view and would mark everything it hadn't reached.
|
||||
seen := make(map[string]struct{}, len(paths))
|
||||
for _, p := range paths {
|
||||
seen[p] = struct{}{}
|
||||
@@ -167,10 +165,9 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
|
||||
// evidence, and that refusal arrives here as an error.
|
||||
//
|
||||
// The consequence is named explicitly because it is not obvious: move
|
||||
// detection (#2528) can only adopt a row that is already marked missing,
|
||||
// so a refused reconcile also means renamed files insert fresh rows and
|
||||
// fork their history. That's the pre-#2528 behaviour rather than a new
|
||||
// failure, but it's worth knowing which scan it happened on. It bites
|
||||
// detection can only adopt a row that is already marked missing, so a
|
||||
// refused reconcile also means renamed files insert fresh rows and fork
|
||||
// their history. It's worth knowing which scan it happened on. It bites
|
||||
// hardest when a large fraction of a small library is reorganised at
|
||||
// once, which trips the mark cap.
|
||||
s.logger.Warn("library scan: reconcile skipped — moved files will fork rather than adopt",
|
||||
@@ -207,7 +204,7 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
|
||||
}
|
||||
|
||||
// PHASE 4 — fold missing rows into replacements this scan brought in or
|
||||
// that were already here (M485). After processing, so both this scan's
|
||||
// that were already here. After processing, so both this scan's
|
||||
// marks and its arrivals are in. Not fatal: an unfolded pair is only a row
|
||||
// left in the missing list, and the next scan tries again.
|
||||
if _, err := PairMissingTracks(ctx, s.pool, s.logger); err != nil && ctx.Err() == nil {
|
||||
@@ -222,18 +219,18 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro
|
||||
// enumerate walks every configured root and returns the audio paths found, in
|
||||
// walk order, plus a count of walk errors.
|
||||
//
|
||||
// A path is recorded even if it will later fail to parse: an unreadable file is a
|
||||
// broken file, not a missing one, and letting reconcile mark it missing would
|
||||
// hide it from the operator behind the wrong explanation.
|
||||
// A path is recorded even if it will later fail to parse: an unreadable file is
|
||||
// a broken file, not a missing one, and letting reconcile mark it missing would
|
||||
// hide it from an admin behind the wrong explanation.
|
||||
func (s *Scanner) enumerate(
|
||||
ctx context.Context, progressCb func(Stats), stats *Stats,
|
||||
) ([]string, int) {
|
||||
paths := make([]string, 0, 8192)
|
||||
errs := 0
|
||||
for _, root := range s.Roots() {
|
||||
// WalkDir's own error return is folded into the per-entry handler below,
|
||||
// so a bad root is counted rather than aborting the whole scan — one
|
||||
// unreadable root shouldn't discard the others' results.
|
||||
// WalkDir's own error return is folded into the per-entry handler
|
||||
// below, so a bad root is counted rather than aborting the whole scan —
|
||||
// one unreadable root shouldn't discard the others' results.
|
||||
_ = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
|
||||
if ctx.Err() != nil {
|
||||
return fs.SkipAll
|
||||
@@ -280,11 +277,10 @@ func (s *Scanner) scanFile(
|
||||
}
|
||||
// Incremental skip: only when the file hasn't changed AND we already have a
|
||||
// real duration AND the row's tag-derived columns were written by the
|
||||
// current extraction logic. The duration clause lets older scans that
|
||||
// recorded duration_ms=0 (before ffprobe was wired) get backfilled without
|
||||
// forcing the operator to wipe the library; the tag-version clause does the
|
||||
// same job for tag-extraction fixes (#2499). Once both are current,
|
||||
// subsequent scans short-circuit as before.
|
||||
// current extraction logic. The duration clause backfills rows that
|
||||
// recorded duration_ms=0 without anyone wiping the library; the
|
||||
// tag-version clause does the same job for tag-extraction fixes. Once both
|
||||
// are current, subsequent scans short-circuit.
|
||||
unchanged := knownTrack && !existing.UpdatedAt.Time.Before(mtime)
|
||||
if unchanged && existing.DurationMs > 0 && existing.TagReadVersion >= tagReadVersion {
|
||||
stats.Skipped++
|
||||
@@ -354,15 +350,15 @@ func (s *Scanner) scanFile(
|
||||
// above — a tagReadVersion bump must stay bound by tag reads, not become a
|
||||
// decode of the whole library.
|
||||
//
|
||||
// An unchanged file with NO fingerprint yet is the backfill's job (#3908),
|
||||
// An unchanged file with NO fingerprint yet is the backfill's job,
|
||||
// deliberately not the scan's. Folding it into the skip check would make
|
||||
// the first scan after an upgrade re-decode every track and push a sync
|
||||
// change to every client for each one.
|
||||
//
|
||||
// Computed before move adoption so adoption can match on the audio hash
|
||||
// (#3914); stored after the upsert, once the row id is known.
|
||||
// Computed before move adoption so adoption can match on the audio hash;
|
||||
// stored after the upsert, once the row id is known.
|
||||
//
|
||||
// With fingerprinting switched off (#3913) the scan decodes nothing, but it
|
||||
// With fingerprinting switched off the scan decodes nothing, but it
|
||||
// still takes the stream hash: a demux rather than a decode, and the only
|
||||
// thing that recognises a moved file with no MBID.
|
||||
var fp fingerprintResult
|
||||
@@ -376,13 +372,13 @@ func (s *Scanner) scanFile(
|
||||
}
|
||||
|
||||
// A path we've never seen might not be a new track — it might be one that
|
||||
// moved or was renamed (#2528). Adopting re-points the existing row at this
|
||||
// moved or was renamed. Adopting re-points the existing row at this
|
||||
// path and clears its missing mark, so the UpsertTrack below conflicts on
|
||||
// file_path and updates THAT row: same track id, likes and play history
|
||||
// intact. Without this, renumbering an album forks every track on it.
|
||||
//
|
||||
// Runs after fingerprinting because, for a file with no MBID, adoption matches
|
||||
// on its audio hash (#3914), and only for genuinely unknown paths — a known
|
||||
// Runs after fingerprinting because, for a file with no MBID, adoption
|
||||
// matches on its audio hash, and only for genuinely unknown paths — a known
|
||||
// path is already the row we're going to update.
|
||||
if !knownTrack {
|
||||
if s.adoptMovedTrack(ctx, q, path, fp.streamSHA256, recordingMBID) {
|
||||
@@ -444,7 +440,8 @@ func (s *Scanner) scanFile(
|
||||
storeFingerprint(ctx, q, s.logger, track.ID, path, fp, fpCfg.ChromaprintLengthSec)
|
||||
} else if err := q.DeleteTrackFingerprint(ctx, track.ID); err != nil {
|
||||
// Off, nothing is stored — but a row describing the previous bytes
|
||||
// must not outlive them, or the sweep would compare audio that is gone.
|
||||
// must not outlive them, or the sweep would compare audio that is
|
||||
// gone.
|
||||
s.logger.Warn("fingerprint: clearing stale fingerprint failed", "path", path, "err", err)
|
||||
}
|
||||
// Loudness is measured by its own worker, never inline: it decodes the
|
||||
@@ -454,7 +451,7 @@ func (s *Scanner) scanFile(
|
||||
if err := q.DeleteTrackLoudness(ctx, track.ID); err != nil {
|
||||
s.logger.Warn("loudness: clearing stale measurement failed", "path", path, "err", err)
|
||||
}
|
||||
// Likewise an AcoustID lookup (M401): the worker looks the new bytes
|
||||
// Likewise an AcoustID lookup: the worker looks the new bytes
|
||||
// up again. A looked-up MBID stays meanwhile and is taken back only if
|
||||
// the new lookup does not match it.
|
||||
if err := q.DeleteAcoustIDLookup(ctx, track.ID); err != nil {
|
||||
@@ -507,7 +504,7 @@ func (s *Scanner) ScanFiles(ctx context.Context, paths []string) ([]pgtype.UUID,
|
||||
"errored", stats.Errored,
|
||||
)
|
||||
// A file the watcher just brought in may be the replacement for a row
|
||||
// an earlier scan marked missing (M485).
|
||||
// an earlier scan marked missing.
|
||||
if _, err := PairMissingTracks(ctx, s.pool, s.logger); err != nil && ctx.Err() == nil {
|
||||
s.logger.Warn("library watch: missing-pair pass failed", "err", err)
|
||||
}
|
||||
@@ -528,12 +525,11 @@ func (s *Scanner) resolveArtist(ctx context.Context, q *dbq.Queries, name, mbid
|
||||
if isUniqueViolation(uerr) {
|
||||
// Another artist row already owns this MBID — two rows that
|
||||
// should be merged (usually two spellings of one name).
|
||||
// Expected, not a fault: leave NULL and let the operator
|
||||
// merge. Mirrors resolveAlbum, which has always handled it
|
||||
// this way — without this branch the identical benign
|
||||
// condition logged a generic warning plus a Postgres ERROR
|
||||
// line on every scan, which teaches an operator to ignore
|
||||
// database errors (#2524).
|
||||
// Expected, not a fault: leave NULL and let an admin
|
||||
// merge. Mirrors resolveAlbum. Logged at Info, because a
|
||||
// warning plus a Postgres ERROR line on every scan for a
|
||||
// benign condition teaches an admin to ignore database
|
||||
// errors.
|
||||
s.logger.Info("library scan: duplicate artist mbid (canonical row already owns it)",
|
||||
"artist_id", existing.ID, "artist", name, "mbid", mbid)
|
||||
} else {
|
||||
|
||||
Reference in New Issue
Block a user