bvandeusenandClaude Opus 5
6379b6c31d
test-go / test (push) Successful in 1m0s
test-go / integration (push) Successful in 3m17s
release / Build signed APK (releases and dev) (push) Successful in 4m51s
release / Build + push container image (push) Successful in 14s
release / Verify release artifacts (tag releases only) (push) Skipped
Reads fingerprints, runs them through the matcher, and records proposals in duplicate_groups (migration 0059). Nothing is merged or deleted: a group is a proposal for the admin report (#3912). Streaming. The whole library's fingerprints are hundreds of megabytes, but tracks are only compared within 3s of each other in duration. So candidates stream in (duration_ms, id) order, keyset-paged on a new tracks(duration_ms, id) index. The grouper holds only the tracks within 3s of the oldest one not yet settled. A seed is settled once a track arrives beyond its window, which gives the same result as grouping the whole sorted list. groupDuplicates is rebuilt on the same streamGrouper, so there is one grouping rule and the #3909 tests still cover it. Each fingerprint's alignment index and variety check are computed once instead of for every pair. Exact duplicates are grouped library-wide in SQL. The first member the stream meets stands in for the whole group in the acoustic pass. An exact group caught in an oversize acoustic cluster is still proposed: the acoustic evidence is discarded, identical bytes are not. Re-sweeping: - a group is identified by its sorted member ids, so finding it again refreshes the row in place - a proposal whose members all sat in one dismissed group is not proposed again (a subset repeats the verdict; a superset is new evidence) - a pending proposal no sweep has found again is retired, but only after a complete sweep, and only if an earlier sweep last confirmed it, so two overlapping sweeps cannot delete each other's findings - dismissals are kept DuplicateSweepWorker checks hourly and sweeps only when a fingerprint was written after the last sweep started. TryStartDuplicateSweep guards against two sweeps at once and reaps one stuck in flight for 2h. The sweep row is closed on a detached context with a deadline, so a sweep cancelled at shutdown still records that it ended. The integration test pages one row at a time and checks: - an acoustic pair and an exact pair are found - a track with no fingerprint, a missing track and a near-duration unrelated song are left out - a dismissed group is suppressed while the pending one refreshes without duplicating - a proposal that stops holding is retired and the dismissal survives Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
55 lines
2.7 KiB
SQL
55 lines
2.7 KiB
SQL
-- 0059_duplicate_groups.up.sql — proposed duplicates and the sweeps that find
|
|
-- them (Scribe milestone #400: #3910).
|
|
--
|
|
-- The sweep compares fingerprints (track_fingerprints, 0058) and proposes groups
|
|
-- of tracks that hold one recording. Nothing here merges anything: a group is a
|
|
-- proposal the operator reviews, and the merge (#3911) is a separate act.
|
|
|
|
-- One row per sweep. Lets the report tell "the sweep has never run" apart from
|
|
-- "it ran and found nothing", and gives the in-flight guard something to check,
|
|
-- the same way scan_runs does for the library scan.
|
|
CREATE TABLE duplicate_sweeps (
|
|
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
started_at timestamptz NOT NULL DEFAULT now(),
|
|
finished_at timestamptz,
|
|
candidates integer,
|
|
groups_found integer,
|
|
oversize_clusters integer,
|
|
error_message text
|
|
);
|
|
CREATE INDEX duplicate_sweeps_started_at_idx ON duplicate_sweeps (started_at DESC);
|
|
|
|
CREATE TABLE duplicate_groups (
|
|
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
-- The group's identity: its member track ids, sorted and joined. A sweep
|
|
-- that finds the same tracks again updates this row rather than proposing
|
|
-- them twice, and a dismissal stays attached to the set it was made about.
|
|
member_key text NOT NULL UNIQUE,
|
|
-- Rule 36: a new value for either CHECK swaps the constraint in the same
|
|
-- migration.
|
|
tier text NOT NULL CHECK (tier IN ('exact', 'acoustic')),
|
|
-- Largest disagreement between any two members; NULL for exact groups,
|
|
-- which have no score.
|
|
worst_bit_error_rate real,
|
|
status text NOT NULL DEFAULT 'pending'
|
|
CHECK (status IN ('pending', 'dismissed', 'merged')),
|
|
detected_at timestamptz NOT NULL DEFAULT now(),
|
|
last_seen_sweep_id uuid REFERENCES duplicate_sweeps (id) ON DELETE SET NULL,
|
|
resolved_at timestamptz
|
|
);
|
|
CREATE INDEX duplicate_groups_status_idx ON duplicate_groups (status);
|
|
|
|
CREATE TABLE duplicate_group_members (
|
|
group_id uuid NOT NULL REFERENCES duplicate_groups (id) ON DELETE CASCADE,
|
|
-- CASCADE is right here: a track that genuinely leaves the library has no
|
|
-- place in a proposal about its duplicates.
|
|
track_id uuid NOT NULL REFERENCES tracks (id) ON DELETE CASCADE,
|
|
PRIMARY KEY (group_id, track_id)
|
|
);
|
|
CREATE INDEX duplicate_group_members_track_idx ON duplicate_group_members (track_id);
|
|
|
|
-- The sweep streams candidates in (duration_ms, id) order, keyset-paged, so it
|
|
-- only ever holds a few seconds' worth of durations in memory. Without this each
|
|
-- page would sort the whole library again.
|
|
CREATE INDEX tracks_duration_id_idx ON tracks (duration_ms, id);
|