docs: library comments say what the scanner and resolver do, not how they came to be (#6277)
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped

Rewrites the internal/library comment blocks to rule 32. Task, milestone,
decision and rule ids are gone; the incidents behind them are restated as
the constraint they left (why a file is deleted before its row, why the
missing mark is a mark, why duplicates block on duration). Counts and
album names from one library are dropped from production comments; test
comments keep the names their fixtures use. The tagReadVersion log keeps
each version's before and after, without the measurements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-23 12:27:34 -04:00
co-authored by Claude Opus 5.5
parent 08a2c06b85
commit 7e6228c13f
66 changed files with 578 additions and 564 deletions
+32 -29
View File
@@ -17,37 +17,38 @@ import (
syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync"
)
// Duplicate sweep (M400 #3910).
// Duplicate sweep.
//
// Reads fingerprints, runs them through the matcher and records what it proposes
// in duplicate_groups. It never merges or deletes anything the operator has not
// asked for: a group is a proposal, reviewed in the admin report (#3912).
// Reads fingerprints, runs them through the matcher and records what it
// proposes in duplicate_groups. It never merges or deletes anything itself: a
// group is a proposal, for the resolver and the admin report.
//
// Blocked on duration, deliberately not on title: the #3885 pair are titled
// "WWW" and "WWW (instrumental)", so a title block would have missed the case
// that started the milestone. Candidates stream in (duration_ms, id) order and
// the grouper holds only a few seconds of durations at a time.
// Blocked on duration, deliberately not on title: one file filed as "WWW" and
// "WWW (instrumental)" would escape a title block. Candidates stream in
// (duration_ms, id) order and the grouper holds only a few seconds of durations
// at a time.
// duplicateCandidatePage is how many candidates one query returns. Each row
// carries a ~4 KB fingerprint, so a page is about 2 MB.
const duplicateCandidatePage = 500
// duplicateSweepTick is how often the worker checks whether a sweep is due. The
// operator's sweep interval (#3913) is the least time between sweeps; the tick
// only bounds how late past it one starts. With nothing due, a tick is two cheap
// aggregate queries.
// sweep interval setting is the least time between sweeps; the tick only bounds
// how late past it one starts. With nothing due, a tick is two cheap aggregate
// queries.
const duplicateSweepTick = time.Hour
// sweepIntervalSlack absorbs the moment between a tick and the sweep it starts
// stamping started_at. Without it a one-hour interval checked on a one-hour tick
// would find the last sweep a moment under an hour old, and skip every other tick.
// stamping started_at. Without it a one-hour interval checked on a one-hour
// tick would find the last sweep a moment under an hour old, and skip every
// other tick.
const sweepIntervalSlack = 5 * time.Minute
// staleDuplicateSweepThreshold is the age past which an in-flight sweep is
// assumed dead — a crash mid-sweep leaves finished_at NULL for ever — and another
// may start. Twice the library scan's threshold, because a sweep compares
// fingerprints across the whole library and can legitimately run long on a big
// one.
// assumed dead — a crash mid-sweep leaves finished_at NULL for ever — and
// another may start. Twice the library scan's threshold, because a sweep
// compares fingerprints across the whole library and can legitimately run long
// on a big one.
const staleDuplicateSweepThreshold = 2 * time.Hour
// duplicateSweepFinishTimeout bounds recording that a sweep ended. It runs on a
@@ -60,7 +61,7 @@ type DuplicateSweepResult struct {
Candidates int // tracks with a chromaprint that were streamed
Groups int // groups the matcher found
Proposed int // written as pending, new or refreshed
Suppressed int // not proposed: already dismissed or resolved by the operator
Suppressed int // not proposed: already dismissed or resolved
Retired int // pending proposals this sweep did not find again, removed
Oversize int // acoustic clusters too large to propose
}
@@ -128,15 +129,17 @@ func sweepDuplicates(
exactSeen := make([]bool, len(exactRows))
// Acoustic tier, streamed in duration order. The first member of an exact
// group the stream meets stands in for the whole group; the rest are skipped.
// Only prints at the current length are streamed: a print at another length
// cannot be compared, and is waiting on the backfill to be re-derived.
// group the stream meets stands in for the whole group; the rest are
// skipped. Only prints at the current length are streamed: a print at
// another length cannot be compared, and is waiting on the backfill to be
// re-derived.
grouper := newStreamGrouper(cfg.AcousticMaxBitErrorRate)
params := dbq.ListDuplicateCandidatesParams{
CurrentVersion: fingerprintVersion,
ChromaprintLengthSec: cfg.ChromaprintLengthSec,
// Durations are never negative, and the all-zero uuid sorts first: every
// row is after this cursor. Valid must be true, or "> NULL" matches nothing.
// Durations are never negative, and the all-zero uuid sorts first:
// every row is after this cursor. Valid must be true, or "> NULL"
// matches nothing.
AfterDurationMs: -1,
AfterID: pgtype.UUID{Valid: true},
PageLimit: pageSize,
@@ -242,7 +245,7 @@ func sweepDuplicates(
}
// coveredByDismissal reports whether every member of a proposal sat together in
// one group the operator dismissed. A subset counts: dismissing {A, B, C} said
// one group an admin dismissed. A subset counts: dismissing {A, B, C} said
// none of them are copies of each other, so proposing {A, B} again would be
// asking the same question twice. A superset does not count: a new copy joining
// is new evidence, and worth asking about.
@@ -305,7 +308,7 @@ func TryStartDuplicateSweep(
}
// DuplicateSweepWorker sweeps whenever its input has changed, at most once per
// the operator's sweep interval.
// sweep interval.
type DuplicateSweepWorker struct {
pool *pgxpool.Pool
logger *slog.Logger
@@ -336,7 +339,7 @@ func (w *DuplicateSweepWorker) Run(ctx context.Context) {
}
}
// tickOnce contains one check so nothing it does can stop the next tick (rule 157).
// tickOnce contains one check so nothing it does can stop the next tick.
func (w *DuplicateSweepWorker) tickOnce(ctx context.Context) {
defer func() {
if r := recover(); r != nil {
@@ -377,12 +380,12 @@ func duplicateSweepDue(ctx context.Context, q *dbq.Queries, cfg FingerprintSetti
}
// sweepIsDue reports whether a sweep should start: something it reads has
// changed since the last sweep started, and the operator's interval has passed.
// changed since the last sweep started, and the sweep interval has passed.
//
// Two things can change its answer. Fingerprints are its input, so any written
// after the last sweep began count; while the backfill runs that is true every
// tick, which is what the interval is for. And a settings save counts, because a
// new threshold or length changes what the same fingerprints group into.
// tick, which is what the interval is for. And a settings save counts, because
// a new threshold or length changes what the same fingerprints group into.
func sweepIsDue(latestFingerprint, lastSweepStart pgtype.Timestamptz, cfg FingerprintSettings, now time.Time) bool {
if !latestFingerprint.Valid {
return false // nothing fingerprinted yet