feat(library): backfill fingerprints for the existing library — M400 #3908
test-web / test (push) Failing after 50s
test-go / test (push) Successful in 1m7s
test-go / integration (push) Successful in 3m27s
release / Build + push container image (push) Canceled after 0s
release / Verify release artifacts (tag releases only) (push) Canceled after 0s
release / Build signed APK (releases and dev) (push) Canceled after 4m23s

The scan fingerprints only bytes it has not seen, so everything imported
before fingerprinting existed, and any row derived by an older
fingerprintVersion, needs a pass of its own.

That pass is a background worker, not a stage in RunScan. RunScan runs
at boot and then every 12h, and an in-flight scan older than an hour is
reaped and a second started beside it. A stage would have to stop inside
the hour: a few hundred decodes a run, so about a month for a 50k-track
library. It would also hold the run in flight and answer manual
rescans with 409 while it worked.

FingerprintBackfillWorker runs once at start, then hourly. Nothing a
pass does (error or panic) can stop the next tick. A pass walks tracks
with no fingerprint or a stale version, skipping missing tracks,
keyset-paged on id. The cursor is what lets a pass end: an inconclusive
attempt writes no row, so a file that keeps timing out would otherwise
be re-listed and retried forever. Two decodes at a time, deliberately:
they compete with transcoding for CPU and with streaming for the mount.

storeFingerprint is now one package function shared by the scan and
the worker, and reports whether the attempt was fingerprinted,
rejected, inconclusive or failed to store.

Progress is a live gauge on the Admin scan card, served by
GET /api/admin/library/fingerprints: fingerprinted / rejected / pending
of total, with missing tracks excluded so it can reach the end.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
This commit is contained in:
2026-09-11 14:56:18 -04:00
co-authored by Claude Opus 5
parent 71d4335584
commit b8855b480f
13 changed files with 633 additions and 16 deletions
+33 -15
View File
@@ -7,6 +7,7 @@ import (
"encoding/hex"
"errors"
"fmt"
"log/slog"
"os/exec"
"strconv"
"strings"
@@ -148,38 +149,55 @@ func (s *Scanner) fingerprintFile(ctx context.Context, path string) fingerprintR
return s.fingerprint(ctx, path)
}
// storeFingerprint records one attempt for a track whose bytes are new or have
// changed. It never fails the scan: a missing fingerprint only keeps a track
// out of duplicate detection, which is not worth dropping the track over.
func (s *Scanner) storeFingerprint(
ctx context.Context, q *dbq.Queries, trackID pgtype.UUID, path string, fp fingerprintResult,
) {
// fingerprintOutcome is what storeFingerprint did with one attempt.
type fingerprintOutcome int
const (
outcomeFingerprinted fingerprintOutcome = iota // both halves stored
outcomeRejected // stored with a NULL half: a verdict
outcomeInconclusive // nothing stored; worth trying again
outcomeStoreFailed // the write itself failed
)
// storeFingerprint records one attempt, for the scan (new or changed bytes) and
// the backfill (#3908) alike, so there is one rule for what gets written. It
// never fails its caller: a missing fingerprint only keeps a track out of
// duplicate detection, which is not worth dropping a scan or a pass over.
func storeFingerprint(
ctx context.Context, q *dbq.Queries, logger *slog.Logger,
trackID pgtype.UUID, path string, fp fingerprintResult,
) fingerprintOutcome {
if fp.hashErr != nil {
s.logger.Warn("library scan: audio stream hash failed", "path", path, "err", fp.hashErr)
logger.Warn("fingerprint: audio stream hash failed", "path", path, "err", fp.hashErr)
}
if fp.printErr != nil {
s.logger.Warn("library scan: chromaprint failed", "path", path, "err", fp.printErr)
logger.Warn("fingerprint: chromaprint failed", "path", path, "err", fp.printErr)
}
if fp.inconclusive() {
// Any row this track holds describes its PREVIOUS bytes. Drop it and
// leave the track to the backfill, rather than stamping a failure that
// says nothing about this file.
// Any row this track holds describes bytes we could not confirm — the
// previous bytes for the scan, an older derivation for the backfill.
// Drop it rather than stamp a failure that says nothing about the file.
if err := q.DeleteTrackFingerprint(ctx, trackID); err != nil {
s.logger.Warn("library scan: clearing stale fingerprint failed", "path", path, "err", err)
logger.Warn("fingerprint: clearing stale fingerprint failed", "path", path, "err", err)
}
return
return outcomeInconclusive
}
// A NULL half here is a verdict — the tool ran and rejected this file — and
// is stamped at the current version so the backfill does not retry it on
// every boot. It is retried when the file changes.
// every pass. It is retried when the file changes.
if err := q.UpsertTrackFingerprint(ctx, dbq.UpsertTrackFingerprintParams{
TrackID: trackID,
AudioStreamSha256: fp.streamSHA256,
Chromaprint: fp.chromaprint,
FingerprintVersion: fingerprintVersion,
}); err != nil {
s.logger.Warn("library scan: storing fingerprint failed", "path", path, "err", err)
logger.Warn("fingerprint: storing fingerprint failed", "path", path, "err", err)
return outcomeStoreFailed
}
if fp.hashErr != nil || fp.printErr != nil {
return outcomeRejected
}
return outcomeFingerprinted
}
// computeAudioStreamSHA256 returns the SHA-256 of the file's encoded audio.