M400: acoustic duplicate detection, history-preserving merge, and fingerprinting settings #134

Merged

bvandeusen merged 8 commits from dev into main 2026-09-11 20:56:56 -04:00

Conversation 0 Commits 8 Files Changed 68

+6250 -314

8 Commits

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	516413f4ca	fix(admin): re-acquisition settings take effect without a restart, and say why a save was refused (#3936, #3937) ... #3936: Router() built a reacquisition.SettingsService of its own, so a save from the admin card refreshed that instance's cache while the sweeper in main.go kept serving what it loaded at boot. The card showed the new policy, the feature ran the old one, and only a restart reconciled them. main.go now hands its instance to the server (srv.ReacqSettings), as it already did for RecSettings, TagSettings and FingerprintSettings, and Router() constructs one only when that field is nil. The regression test saves through the router and reads the sweeper's instance. #3937: the card's catch tested `e instanceof Error`, but api.put throws a plain {code, message, status} object, so every reason the server gave was discarded in favour of "Couldn't save settings." It now uses errMessage, which appends the server's message for invalid_setting. Its test rejected with an Error no code path produces, so it passed throughout; it now rejects with what the client actually throws. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 20:15:35 -04:00
bvandeusenandClaude Opus 5	37d4906033	test(api): pass fingerprint settings to Mount in the route-registration test (M400 #3913) ... The unprefixed Mount call in library_test.go was missed when #3913 added the parameter, failing go vet. Also pins the fingerprint coverage, fingerprint settings and duplicates routes as admin-gated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 17:58:05 -04:00
bvandeusenandClaude Opus 5	077ae61235	feat(admin): fingerprinting settings — on/off, length, match threshold, concurrency, sweep interval (M400 #3913) ... Rule 25: the fingerprinting knobs move out of source into a DB-backed singleton (migration 0061), edited from a card on the Duplicates page and shared live with the scanner, the backfill and the duplicate sweep through one service instance, so a save needs no restart. The length is the knob that can silently break the library: prints taken at two lengths never match. Each track_fingerprints row now records the length it was taken at, and every reader filters on the current one — the backfill treats another length as stale, the gauge counts it pending, the sweep never streams it. Equivalent to a version bump, except that setting the length back makes rows not yet redone current again. The card warns before a length change re-fingerprints the library. Off stops every decode: the scan takes only the stream hash (a demux, and what recognises a moved file) and stores nothing, dropping a changed file's stale row; the backfill idles. A save also makes a sweep due, since a new threshold or length changes what the same prints group into, and the sweep interval gains slack so an hourly interval on an hourly tick doesn't skip every other tick. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 17:53:55 -04:00
bvandeusenandClaude Opus 5	c8bf9dc929	refactor(library): move detection matches on the audio hash, not size and duration (M400 #3914) ... A file that comes back renamed or moved keeps its track row, and with it its likes and play history, by being matched to the missing row it replaces (#2528). Untagged files were matched on (file_size, duration_ms), which was never a fingerprint. It could pair two unrelated files that happened to share a byte count and a duration, and it missed a file retagged in place, whose size changes. The only defence was requiring a unique match and otherwise giving up. Now there is a real identity. FindMissingTrackByAudioHash matches a missing track by the SHA-256 of its encoded audio (track_fingerprints, #3906). That survives a rename, a move and a retag, and only an identical recording can match it. adoptMovedTrack takes the new file's hash, which the scan already computes before adoption. The size and duration query and fallback are removed outright, with no second path (rule 22). Unchanged: - MBID first: it identifies the recording and survives a re-encode that even the hash does not - a unique match is still required - an absent hash is never looked up, so unhashable files cannot pair with each other The test fake answers the hash lookup only for the hash it holds, so the tests can tell adoption by identity apart from adoption by coincidence. That includes the case the old pair got wrong: different audio of equal size and duration is not adopted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 17:31:42 -04:00
bvandeusenandClaude Opus 5	11ef044ef6	feat(library): merge duplicates without losing history (M400 #3911) ... Merge keeps one copy of a duplicate group and removes the rest. Every table that references tracks does so ON DELETE CASCADE, so deleting a duplicate's row outright would silently destroy its likes, plays, playlist entries and tags. The merge moves all of that onto the kept copy first, then deletes the empty row. In one transaction, holding a lock on the group: - repoints play_events, skip_events, contextual_likes, playback_errors, lidarr_requests.matched_track_id and playlist_tracks. The last is keyed by position, so every entry stays where it was. - merges general_likes one per user, dated to the earlier like - takes the union of track_tags, keeping the kept copy's own weight on a shared tag - rewrites track_similarity onto the kept copy, dropping edges that would point a track at itself and keeping the kept copy's existing edge on a collision - lets the kept copy take a recording MBID only the removed copy had - deletes the removed copies' rows, tidies emptied albums and artists, marks the group merged - logs sync changes: track deletes, and like and playlist-track delete/upsert pairs The removed copies' files are deleted first, before any row changes, through the same helper as DeleteTrackFile (now shared, along with the album tidy-up). A merge that left the file behind would be undone by the next scan re-importing it. An unwritable library answers 409 library_not_writable and nothing changes. tracks.Service.MergeDuplicates wraps it with the opt-in Lidarr unmonitor from RemoveTrack, skipped when the removed copy is a second file of the kept copy's own album track: unmonitoring that would stop Lidarr managing the kept file. It writes a duplicate_merge audit row after commit, per the audit package's best-effort contract, naming both paths. POST /api/admin/library/duplicates/{id}/merge takes an optional survivor_track_id (the report's proposal otherwise) and unmonitor. On the report page: - each copy gets a Keep choice, defaulting to the proposed one - Merge needs a second click, on a button that says how many files it removes, with the consequence stated beside an opt-in Lidarr checkbox Integration tests cover: - every piece of history landing on the kept copy exactly: likes deduped at the earlier time, plays and skips counted, playlist position unchanged, tags unioned, similarity rewritten with no duplicate or self-edge, MBID inherited - the removed file gone, and a second merge refused - an unwritable file leaving likes, plays, row and group untouched - a survivor outside the group refused Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 17:25:06 -04:00
bvandeusenandClaude Opus 5	ff493a8c7d	feat(admin): the duplicates report — review proposed duplicate groups (M400 #3912) ... A new admin tab, Duplicates, beside Missing files: the proposals from the duplicate sweep, with a Sweep now trigger and a Not duplicates dismissal. Nothing on it merges or deletes; the merge is #3911. Each group shows: - whether it is identical audio or the same recording, with a match percentage from the weakest link between members - every copy's format, size, duration, path, and the likes and plays it carries (every user's; this is admin-only, and it is what decides which copy to keep) - the copy proposed to keep, and the rule that chose it The survivor rule is library.ProposeSurvivor, a pure function the merge will reuse: lossless over lossy, then the larger file, then the copy in the library longest, then lowest id. Bitrate is not in it because the scanner never fills tracks.bitrate, and for one recording at one duration a larger file is the higher bitrate. m4a is not counted as lossless: it may be AAC. The reason names the rule that separated first place from second, not every rule the winner passed. An empty report has three causes, and the page says which: still fingerprinting, the sweep has never run, or it ran and found nothing. The sweep's state and the backfill's progress come back with the groups for that reason. Groups left with fewer than two members since the sweep are not shown. GET /api/admin/library/duplicates, POST .../sweep (202, or 409 sweep_in_progress), POST .../{id}/dismiss (404 duplicate_group_not_pending when already resolved). Migration 0060 indexes play_events by track_id. Its only indexes led with user_id, so each copy's play count, and the merge's repointing of play history, would scan the whole table. Web only, like Missing files: Android has no library-health admin screens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 17:11:11 -04:00
bvandeusenandClaude Opus 5	6379b6c31d	feat(library): the duplicate sweep — propose duplicate groups from fingerprints (M400 #3910) ... Reads fingerprints, runs them through the matcher, and records proposals in duplicate_groups (migration 0059). Nothing is merged or deleted: a group is a proposal for the admin report (#3912). Streaming. The whole library's fingerprints are hundreds of megabytes, but tracks are only compared within 3s of each other in duration. So candidates stream in (duration_ms, id) order, keyset-paged on a new tracks(duration_ms, id) index. The grouper holds only the tracks within 3s of the oldest one not yet settled. A seed is settled once a track arrives beyond its window, which gives the same result as grouping the whole sorted list. groupDuplicates is rebuilt on the same streamGrouper, so there is one grouping rule and the #3909 tests still cover it. Each fingerprint's alignment index and variety check are computed once instead of for every pair. Exact duplicates are grouped library-wide in SQL. The first member the stream meets stands in for the whole group in the acoustic pass. An exact group caught in an oversize acoustic cluster is still proposed: the acoustic evidence is discarded, identical bytes are not. Re-sweeping: - a group is identified by its sorted member ids, so finding it again refreshes the row in place - a proposal whose members all sat in one dismissed group is not proposed again (a subset repeats the verdict; a superset is new evidence) - a pending proposal no sweep has found again is retired, but only after a complete sweep, and only if an earlier sweep last confirmed it, so two overlapping sweeps cannot delete each other's findings - dismissals are kept DuplicateSweepWorker checks hourly and sweeps only when a fingerprint was written after the last sweep started. TryStartDuplicateSweep guards against two sweeps at once and reaps one stuck in flight for 2h. The sweep row is closed on a detached context with a deadline, so a sweep cancelled at shutdown still records that it ended. The integration test pages one row at a time and checks: - an acoustic pair and an exact pair are found - a track with no fingerprint, a missing track and a near-duration unrelated song are left out - a dismissed group is suppressed while the pending one refreshes without duplicating - a proposal that stops holding is retired and the dismissal survives Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 17:00:07 -04:00
bvandeusenandClaude Opus 5	c06af48cd6	feat(library): the duplicate matcher — a pure comparison over fingerprints (M400 #3909) ... Decides whether tracks are proposed as one recording. No database, no files, so every rule is falsifiable in a unit test. Two tiers: - exact: equal audio_stream_sha256 (identical encoded audio bytes). No threshold and no false positives. - acoustic: chromaprint fingerprints that agree once aligned. Two fingerprints can start at slightly different points in the audio (padding trimmed differently), so offsets within ±120 items (~15s) are voted on using items that share their high 14 bits. Bit-error rate is then measured over the overlap at the winning offset. The approach and both constants follow AcoustID's pg_acoustid; it was reimplemented from that description and no code was copied. No verdict below ~10s of overlap, or for low-information fingerprints (silence, a sustained tone). Two such tracks agree without being one recording. Grouping uses complete linkage: a track joins a group only if it matches every member. Otherwise A close to B and B close to C would merge A and C, which are not close, and it means any member can be the survivor. Other rules: - durations must be within 3s - acoustic groups are capped at 8, and larger clusters are reported and discarded as a likely shared jingle - an exact group absorbed into an acoustic one takes the acoustic tier - output does not depend on input order The acoustic threshold is 0.15 bit-error rate: deliberately conservative, since the operator's concern is different recordings of one song being merged, and an instrumental shares its vocal's harmony. It is unmeasured, and needs calibrating against real pairs once the backfill has populated fingerprints (#3913 exposes it). Nothing calls this yet; the sweep (#3910) does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH	2026-09-11 16:48:06 -04:00