docs: library comments say what the scanner and resolver do, not how they came to be (#6277)
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped
Rewrites the internal/library comment blocks to rule 32. Task, milestone, decision and rule ids are gone; the incidents behind them are restated as the constraint they left (why a file is deleted before its row, why the missing mark is a mark, why duplicates block on duration). Counts and album names from one library are dropped from production comments; test comments keep the names their fixtures use. The tagReadVersion log keeps each version's before and after, without the measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -17,37 +17,38 @@ import (
|
||||
syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync"
|
||||
)
|
||||
|
||||
// Duplicate sweep (M400 #3910).
|
||||
// Duplicate sweep.
|
||||
//
|
||||
// Reads fingerprints, runs them through the matcher and records what it proposes
|
||||
// in duplicate_groups. It never merges or deletes anything the operator has not
|
||||
// asked for: a group is a proposal, reviewed in the admin report (#3912).
|
||||
// Reads fingerprints, runs them through the matcher and records what it
|
||||
// proposes in duplicate_groups. It never merges or deletes anything itself: a
|
||||
// group is a proposal, for the resolver and the admin report.
|
||||
//
|
||||
// Blocked on duration, deliberately not on title: the #3885 pair are titled
|
||||
// "WWW" and "WWW (instrumental)", so a title block would have missed the case
|
||||
// that started the milestone. Candidates stream in (duration_ms, id) order and
|
||||
// the grouper holds only a few seconds of durations at a time.
|
||||
// Blocked on duration, deliberately not on title: one file filed as "WWW" and
|
||||
// "WWW (instrumental)" would escape a title block. Candidates stream in
|
||||
// (duration_ms, id) order and the grouper holds only a few seconds of durations
|
||||
// at a time.
|
||||
|
||||
// duplicateCandidatePage is how many candidates one query returns. Each row
|
||||
// carries a ~4 KB fingerprint, so a page is about 2 MB.
|
||||
const duplicateCandidatePage = 500
|
||||
|
||||
// duplicateSweepTick is how often the worker checks whether a sweep is due. The
|
||||
// operator's sweep interval (#3913) is the least time between sweeps; the tick
|
||||
// only bounds how late past it one starts. With nothing due, a tick is two cheap
|
||||
// aggregate queries.
|
||||
// sweep interval setting is the least time between sweeps; the tick only bounds
|
||||
// how late past it one starts. With nothing due, a tick is two cheap aggregate
|
||||
// queries.
|
||||
const duplicateSweepTick = time.Hour
|
||||
|
||||
// sweepIntervalSlack absorbs the moment between a tick and the sweep it starts
|
||||
// stamping started_at. Without it a one-hour interval checked on a one-hour tick
|
||||
// would find the last sweep a moment under an hour old, and skip every other tick.
|
||||
// stamping started_at. Without it a one-hour interval checked on a one-hour
|
||||
// tick would find the last sweep a moment under an hour old, and skip every
|
||||
// other tick.
|
||||
const sweepIntervalSlack = 5 * time.Minute
|
||||
|
||||
// staleDuplicateSweepThreshold is the age past which an in-flight sweep is
|
||||
// assumed dead — a crash mid-sweep leaves finished_at NULL for ever — and another
|
||||
// may start. Twice the library scan's threshold, because a sweep compares
|
||||
// fingerprints across the whole library and can legitimately run long on a big
|
||||
// one.
|
||||
// assumed dead — a crash mid-sweep leaves finished_at NULL for ever — and
|
||||
// another may start. Twice the library scan's threshold, because a sweep
|
||||
// compares fingerprints across the whole library and can legitimately run long
|
||||
// on a big one.
|
||||
const staleDuplicateSweepThreshold = 2 * time.Hour
|
||||
|
||||
// duplicateSweepFinishTimeout bounds recording that a sweep ended. It runs on a
|
||||
@@ -60,7 +61,7 @@ type DuplicateSweepResult struct {
|
||||
Candidates int // tracks with a chromaprint that were streamed
|
||||
Groups int // groups the matcher found
|
||||
Proposed int // written as pending, new or refreshed
|
||||
Suppressed int // not proposed: already dismissed or resolved by the operator
|
||||
Suppressed int // not proposed: already dismissed or resolved
|
||||
Retired int // pending proposals this sweep did not find again, removed
|
||||
Oversize int // acoustic clusters too large to propose
|
||||
}
|
||||
@@ -128,15 +129,17 @@ func sweepDuplicates(
|
||||
exactSeen := make([]bool, len(exactRows))
|
||||
|
||||
// Acoustic tier, streamed in duration order. The first member of an exact
|
||||
// group the stream meets stands in for the whole group; the rest are skipped.
|
||||
// Only prints at the current length are streamed: a print at another length
|
||||
// cannot be compared, and is waiting on the backfill to be re-derived.
|
||||
// group the stream meets stands in for the whole group; the rest are
|
||||
// skipped. Only prints at the current length are streamed: a print at
|
||||
// another length cannot be compared, and is waiting on the backfill to be
|
||||
// re-derived.
|
||||
grouper := newStreamGrouper(cfg.AcousticMaxBitErrorRate)
|
||||
params := dbq.ListDuplicateCandidatesParams{
|
||||
CurrentVersion: fingerprintVersion,
|
||||
ChromaprintLengthSec: cfg.ChromaprintLengthSec,
|
||||
// Durations are never negative, and the all-zero uuid sorts first: every
|
||||
// row is after this cursor. Valid must be true, or "> NULL" matches nothing.
|
||||
// Durations are never negative, and the all-zero uuid sorts first:
|
||||
// every row is after this cursor. Valid must be true, or "> NULL"
|
||||
// matches nothing.
|
||||
AfterDurationMs: -1,
|
||||
AfterID: pgtype.UUID{Valid: true},
|
||||
PageLimit: pageSize,
|
||||
@@ -242,7 +245,7 @@ func sweepDuplicates(
|
||||
}
|
||||
|
||||
// coveredByDismissal reports whether every member of a proposal sat together in
|
||||
// one group the operator dismissed. A subset counts: dismissing {A, B, C} said
|
||||
// one group an admin dismissed. A subset counts: dismissing {A, B, C} said
|
||||
// none of them are copies of each other, so proposing {A, B} again would be
|
||||
// asking the same question twice. A superset does not count: a new copy joining
|
||||
// is new evidence, and worth asking about.
|
||||
@@ -305,7 +308,7 @@ func TryStartDuplicateSweep(
|
||||
}
|
||||
|
||||
// DuplicateSweepWorker sweeps whenever its input has changed, at most once per
|
||||
// the operator's sweep interval.
|
||||
// sweep interval.
|
||||
type DuplicateSweepWorker struct {
|
||||
pool *pgxpool.Pool
|
||||
logger *slog.Logger
|
||||
@@ -336,7 +339,7 @@ func (w *DuplicateSweepWorker) Run(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// tickOnce contains one check so nothing it does can stop the next tick (rule 157).
|
||||
// tickOnce contains one check so nothing it does can stop the next tick.
|
||||
func (w *DuplicateSweepWorker) tickOnce(ctx context.Context) {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
@@ -377,12 +380,12 @@ func duplicateSweepDue(ctx context.Context, q *dbq.Queries, cfg FingerprintSetti
|
||||
}
|
||||
|
||||
// sweepIsDue reports whether a sweep should start: something it reads has
|
||||
// changed since the last sweep started, and the operator's interval has passed.
|
||||
// changed since the last sweep started, and the sweep interval has passed.
|
||||
//
|
||||
// Two things can change its answer. Fingerprints are its input, so any written
|
||||
// after the last sweep began count; while the backfill runs that is true every
|
||||
// tick, which is what the interval is for. And a settings save counts, because a
|
||||
// new threshold or length changes what the same fingerprints group into.
|
||||
// tick, which is what the interval is for. And a settings save counts, because
|
||||
// a new threshold or length changes what the same fingerprints group into.
|
||||
func sweepIsDue(latestFingerprint, lastSweepStart pgtype.Timestamptz, cfg FingerprintSettings, now time.Time) bool {
|
||||
if !latestFingerprint.Valid {
|
||||
return false // nothing fingerprinted yet
|
||||
|
||||
Reference in New Issue
Block a user