Backfill fingerprints for the existing library (M400 #3908) #133

Merged
bvandeusen merged 2 commits from dev into main 2026-09-11 15:34:19 -04:00
Owner

Two commits from dev. CI is green on the head 21c698a6:

  • Web: run 6514
  • Signed APK + :dev image: run 6513
  • Go (short + integration): run 6511 on b8855b48. No Go file changed after it.

What this ships — b8855b48

PR #132 fingerprinted only new or changed files, so every track already in the library has no fingerprint yet. This adds the pass that fills them in.

A background worker, not a scan stage. A library scan runs at boot and then every 12h, and an in-flight scan older than an hour is reaped and replaced. A stage would have to stop inside the hour, which means about a month to cover a 50k-track library, and manual rescans would return 409 while it ran.

How a pass works:

  • The worker runs once at startup, then hourly.
  • It picks up tracks with no fingerprint, or one from an older fingerprint version. Tracks whose files are missing are skipped.
  • It pages by track id, so a file that keeps timing out is tried once per pass rather than retried in a loop.
  • It decodes two files at a time, deliberately, because fpcalc and the stream hash compete with playback transcoding for CPU and with streaming for the disk.
  • A panic or error in one pass can't stop the next tick.

Progress: a "Fingerprints" line on the Admin scan card shows fingerprinted / pending / unreadable of total. It's served by GET /api/admin/library/fingerprints, and missing tracks are excluded so the count can reach the end.

Operational effect: after deploying, expect sustained fpcalc/ffmpeg work until the library has caught up. Roughly: tracks × ~1.5s ÷ 2.

21c698a6

A test fix: the Admin page test's mock of $lib/api/admin was missing the new query, so every test that renders the page threw.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH

Two commits from `dev`. CI is green on the head `21c698a6`: - **Web:** run 6514 - **Signed APK + `:dev` image:** run 6513 - **Go (short + integration):** run 6511 on `b8855b48`. No Go file changed after it. ## What this ships — `b8855b48` PR #132 fingerprinted only **new or changed** files, so every track already in the library has no fingerprint yet. This adds the pass that fills them in. **A background worker, not a scan stage.** A library scan runs at boot and then every 12h, and an in-flight scan older than an hour is reaped and replaced. A stage would have to stop inside the hour, which means about a month to cover a 50k-track library, and manual rescans would return 409 while it ran. **How a pass works:** - The worker runs once at startup, then hourly. - It picks up tracks with no fingerprint, or one from an older fingerprint version. Tracks whose files are missing are skipped. - It pages by track id, so a file that keeps timing out is tried once per pass rather than retried in a loop. - It decodes **two files at a time**, deliberately, because fpcalc and the stream hash compete with playback transcoding for CPU and with streaming for the disk. - A panic or error in one pass can't stop the next tick. **Progress:** a "Fingerprints" line on the Admin scan card shows fingerprinted / pending / unreadable of total. It's served by `GET /api/admin/library/fingerprints`, and missing tracks are excluded so the count can reach the end. **Operational effect:** after deploying, expect sustained fpcalc/ffmpeg work until the library has caught up. Roughly: tracks × ~1.5s ÷ 2. ## `21c698a6` A test fix: the Admin page test's mock of `$lib/api/admin` was missing the new query, so every test that renders the page threw. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
bvandeusen added 2 commits 2026-09-11 15:34:12 -04:00
feat(library): backfill fingerprints for the existing library — M400 #3908
test-web / test (push) Failing after 50s
test-go / test (push) Successful in 1m7s
test-go / integration (push) Successful in 3m27s
release / Build + push container image (push) Canceled after 0s
release / Verify release artifacts (tag releases only) (push) Canceled after 0s
release / Build signed APK (releases and dev) (push) Canceled after 4m23s
b8855b480f
The scan fingerprints only bytes it has not seen, so everything imported
before fingerprinting existed, and any row derived by an older
fingerprintVersion, needs a pass of its own.

That pass is a background worker, not a stage in RunScan. RunScan runs
at boot and then every 12h, and an in-flight scan older than an hour is
reaped and a second started beside it. A stage would have to stop inside
the hour: a few hundred decodes a run, so about a month for a 50k-track
library. It would also hold the run in flight and answer manual
rescans with 409 while it worked.

FingerprintBackfillWorker runs once at start, then hourly. Nothing a
pass does (error or panic) can stop the next tick. A pass walks tracks
with no fingerprint or a stale version, skipping missing tracks,
keyset-paged on id. The cursor is what lets a pass end: an inconclusive
attempt writes no row, so a file that keeps timing out would otherwise
be re-listed and retried forever. Two decodes at a time, deliberately:
they compete with transcoding for CPU and with streaming for the mount.

storeFingerprint is now one package function shared by the scan and
the worker, and reports whether the attempt was fingerprinted,
rejected, inconclusive or failed to store.

Progress is a live gauge on the Admin scan card, served by
GET /api/admin/library/fingerprints: fingerprinted / rejected / pending
of total, with missing tracks excluded so it can reach the end.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
test(web): give the admin page mock the fingerprint coverage query
test-web / test (push) Successful in 35s
release / Build signed APK (releases and dev) (push) Successful in 4m25s
release / Build + push container image (push) Successful in 34s
release / Verify release artifacts (tag releases only) (push) Skipped
21c698a616
b8855b48 made the admin overview page create a fingerprint coverage
query, but admin.test.ts mocks $lib/api/admin with an explicit factory
that only returned the cover coverage query. Every test that renders
the page threw on the missing export (run 6512, 11 failures). The mock
now returns it in the same empty-store shape, so the gauge stays hidden
in these tests the way the cover gauge does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
bvandeusen merged commit 18618bd135 into main 2026-09-11 15:34:19 -04:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: bvandeusen/minstrel#133