test-go / test (push) Failing after 44s
test-web / test (push) Successful in 49s
test-go / integration (push) Failing after 2m42s
release / Build + push container image (push) Canceled after 0s
release / Verify release artifacts (tag releases only) (push) Canceled after 0s
release / Build signed APK (releases and dev) (push) Canceled after 4m8s
Rule 25: the fingerprinting knobs move out of source into a DB-backed singleton (migration 0061), edited from a card on the Duplicates page and shared live with the scanner, the backfill and the duplicate sweep through one service instance, so a save needs no restart. The length is the knob that can silently break the library: prints taken at two lengths never match. Each track_fingerprints row now records the length it was taken at, and every reader filters on the current one — the backfill treats another length as stale, the gauge counts it pending, the sweep never streams it. Equivalent to a version bump, except that setting the length back makes rows not yet redone current again. The card warns before a length change re-fingerprints the library. Off stops every decode: the scan takes only the stream hash (a demux, and what recognises a moved file) and stores nothing, dropping a changed file's stale row; the backfill idles. A save also makes a sweep due, since a new threshold or length changes what the same prints group into, and the sweep interval gains slack so an hourly interval on an hourly tick doesn't skip every other tick. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
157 lines
6.6 KiB
SQL
157 lines
6.6 KiB
SQL
-- name: StartDuplicateSweep :one
|
|
INSERT INTO duplicate_sweeps DEFAULT VALUES RETURNING id, started_at;
|
|
|
|
-- name: FinishDuplicateSweep :exec
|
|
UPDATE duplicate_sweeps
|
|
SET finished_at = now(),
|
|
candidates = sqlc.arg(candidates),
|
|
groups_found = sqlc.arg(groups_found),
|
|
oversize_clusters = sqlc.arg(oversize_clusters),
|
|
error_message = NULLIF(sqlc.arg(error_message)::text, '')
|
|
WHERE id = sqlc.arg(id);
|
|
|
|
-- name: GetInFlightDuplicateSweep :one
|
|
-- The guard against two sweeps at once: "in flight" is finished_at IS NULL.
|
|
SELECT id, started_at
|
|
FROM duplicate_sweeps
|
|
WHERE finished_at IS NULL
|
|
ORDER BY started_at DESC
|
|
LIMIT 1;
|
|
|
|
-- name: GetLatestDuplicateSweep :one
|
|
SELECT id, started_at, finished_at, candidates, groups_found, oversize_clusters, error_message
|
|
FROM duplicate_sweeps
|
|
ORDER BY started_at DESC
|
|
LIMIT 1;
|
|
|
|
-- name: GetLatestFingerprintComputedAt :one
|
|
-- Whether a sweep has anything new to look at: fingerprints written since the
|
|
-- last sweep started.
|
|
SELECT max(computed_at)::timestamptz AS latest FROM track_fingerprints;
|
|
|
|
-- name: ListExactDuplicateHashes :many
|
|
-- The exact tier, library-wide in one pass: identical encoded audio shared by
|
|
-- more than one present track.
|
|
SELECT f.audio_stream_sha256,
|
|
array_agg(t.id ORDER BY t.id)::uuid[] AS track_ids
|
|
FROM track_fingerprints f
|
|
JOIN tracks t ON t.id = f.track_id
|
|
WHERE t.missing_since IS NULL
|
|
AND f.fingerprint_version >= sqlc.arg(current_version)
|
|
AND f.audio_stream_sha256 IS NOT NULL
|
|
GROUP BY f.audio_stream_sha256
|
|
HAVING count(*) > 1;
|
|
|
|
-- name: ListDuplicateCandidates :many
|
|
-- The acoustic tier's input, one page at a time in (duration_ms, id) order so the
|
|
-- sweep holds only a sliding window of durations. Tracks without a chromaprint
|
|
-- cannot be compared acoustically and are left out; any exact duplicates among
|
|
-- them come from ListExactDuplicateHashes.
|
|
SELECT t.id, t.duration_ms, f.audio_stream_sha256, f.chromaprint
|
|
FROM tracks t
|
|
JOIN track_fingerprints f ON f.track_id = t.id
|
|
WHERE t.missing_since IS NULL
|
|
AND f.fingerprint_version >= sqlc.arg(current_version)
|
|
AND f.chromaprint IS NOT NULL
|
|
-- Only chromaprints taken at the current length: prints at two lengths are not
|
|
-- comparable, and after a length change the backfill is still re-deriving the
|
|
-- rest (#3913).
|
|
AND f.chromaprint_length_sec = sqlc.arg(chromaprint_length_sec)
|
|
AND (t.duration_ms, t.id) > (sqlc.arg(after_duration_ms)::integer, sqlc.arg(after_id)::uuid)
|
|
ORDER BY t.duration_ms, t.id
|
|
LIMIT sqlc.arg(page_limit);
|
|
|
|
-- name: ListDismissedDuplicateMemberSets :many
|
|
-- What the operator has already said are not duplicates. A new proposal whose
|
|
-- every member sat together in one of these is not proposed again.
|
|
SELECT g.id, array_agg(m.track_id ORDER BY m.track_id)::uuid[] AS track_ids
|
|
FROM duplicate_groups g
|
|
JOIN duplicate_group_members m ON m.group_id = g.id
|
|
WHERE g.status = 'dismissed'
|
|
GROUP BY g.id;
|
|
|
|
-- name: UpsertDuplicateGroup :one
|
|
-- Proposes a group, or refreshes one already pending. A group already dismissed
|
|
-- or merged is left exactly as it is: the WHERE on the update makes the conflict
|
|
-- a no-op, and the caller sees no row.
|
|
INSERT INTO duplicate_groups (member_key, tier, worst_bit_error_rate, last_seen_sweep_id)
|
|
VALUES (sqlc.arg(member_key), sqlc.arg(tier), sqlc.narg(worst_bit_error_rate), sqlc.arg(sweep_id))
|
|
ON CONFLICT (member_key) DO UPDATE
|
|
SET tier = EXCLUDED.tier,
|
|
worst_bit_error_rate = EXCLUDED.worst_bit_error_rate,
|
|
last_seen_sweep_id = EXCLUDED.last_seen_sweep_id
|
|
WHERE duplicate_groups.status = 'pending'
|
|
RETURNING id;
|
|
|
|
-- name: AddDuplicateGroupMember :exec
|
|
INSERT INTO duplicate_group_members (group_id, track_id)
|
|
VALUES (sqlc.arg(group_id), sqlc.arg(track_id))
|
|
ON CONFLICT DO NOTHING;
|
|
|
|
-- name: DeleteStalePendingDuplicateGroups :execrows
|
|
-- A pending proposal this sweep did not find again no longer describes the
|
|
-- library: a member was re-fingerprinted, merged away or went missing. Dismissed
|
|
-- groups are kept regardless — they are the memory of a decision.
|
|
--
|
|
-- Only proposals last confirmed by an EARLIER sweep go. Should two sweeps ever
|
|
-- overlap (a manual trigger racing the worker), neither may delete what the other
|
|
-- has just found.
|
|
DELETE FROM duplicate_groups g
|
|
WHERE g.status = 'pending'
|
|
AND g.last_seen_sweep_id IS DISTINCT FROM sqlc.arg(sweep_id)
|
|
AND (g.last_seen_sweep_id IS NULL
|
|
OR (SELECT s.started_at FROM duplicate_sweeps s WHERE s.id = g.last_seen_sweep_id)
|
|
< (SELECT s.started_at FROM duplicate_sweeps s WHERE s.id = sqlc.arg(sweep_id)));
|
|
|
|
-- name: CountPendingDuplicateGroups :one
|
|
-- Proposals awaiting review. A group left with one member — its other tracks
|
|
-- deleted since the sweep — is no proposal at all and is not counted; the next
|
|
-- sweep retires it.
|
|
SELECT count(*)::bigint
|
|
FROM duplicate_groups g
|
|
WHERE g.status = 'pending'
|
|
AND (SELECT count(*) FROM duplicate_group_members m WHERE m.group_id = g.id) >= 2;
|
|
|
|
-- name: ListPendingDuplicateGroupMembers :many
|
|
-- One page of proposals, newest first, flattened to one row per member so the
|
|
-- handler folds them without a query per group. What each copy carries — likes
|
|
-- and plays from every user — is here because it is what the operator weighs
|
|
-- when deciding which copy to keep.
|
|
WITH page AS (
|
|
SELECT g.id, g.tier, g.worst_bit_error_rate, g.detected_at
|
|
FROM duplicate_groups g
|
|
WHERE g.status = 'pending'
|
|
AND (SELECT count(*) FROM duplicate_group_members m WHERE m.group_id = g.id) >= 2
|
|
ORDER BY g.detected_at DESC, g.id
|
|
LIMIT sqlc.arg(page_limit) OFFSET sqlc.arg(page_offset)
|
|
)
|
|
SELECT p.id AS group_id,
|
|
p.tier,
|
|
p.worst_bit_error_rate,
|
|
p.detected_at,
|
|
t.id AS track_id,
|
|
t.title,
|
|
artists.name AS artist_name,
|
|
albums.id AS album_id,
|
|
albums.title AS album_title,
|
|
t.file_path,
|
|
t.file_format,
|
|
t.file_size,
|
|
t.duration_ms,
|
|
t.added_at,
|
|
(SELECT count(*) FROM general_likes l WHERE l.track_id = t.id)::bigint AS like_count,
|
|
(SELECT count(*) FROM play_events e WHERE e.track_id = t.id)::bigint AS play_count
|
|
FROM page p
|
|
JOIN duplicate_group_members m ON m.group_id = p.id
|
|
JOIN tracks t ON t.id = m.track_id
|
|
JOIN albums ON albums.id = t.album_id
|
|
JOIN artists ON artists.id = t.artist_id
|
|
ORDER BY p.detected_at DESC, p.id, t.id;
|
|
|
|
-- name: DismissDuplicateGroup :execrows
|
|
-- "These are not duplicates." Only a pending group can be dismissed; zero rows
|
|
-- means it was already resolved or no longer exists.
|
|
UPDATE duplicate_groups
|
|
SET status = 'dismissed', resolved_at = now()
|
|
WHERE id = sqlc.arg(id) AND status = 'pending';
|