docs: library comments say what the scanner and resolver do, not how they came to be (#6277)
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped
release / govulncheck (push) Successful in 20s
release / web (push) Successful in 1m20s
release / go (push) Successful in 1m37s
release / integration (push) Successful in 5m15s
release / android (push) Successful in 5m26s
release / Build signed APK (releases and dev) (push) Successful in 5m45s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 16s
release / Verify release artifacts (tag releases only) (push) Skipped
Rewrites the internal/library comment blocks to rule 32. Task, milestone, decision and rule ids are gone; the incidents behind them are restated as the constraint they left (why a file is deleted before its row, why the missing mark is a mark, why duplicates block on duration). Counts and album names from one library are dropped from production comments; test comments keep the names their fixtures use. The tagReadVersion log keeps each version's before and after, without the measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -6,16 +6,16 @@ import (
|
||||
"sort"
|
||||
)
|
||||
|
||||
// Duplicate matching (M400 #3909).
|
||||
// Duplicate matching.
|
||||
//
|
||||
// Pure functions over fingerprints: no database, no files. This is the part that
|
||||
// decides whether two tracks in the operator's library are proposed as one
|
||||
// recording, so every rule in it has to be falsifiable in a unit test.
|
||||
// Pure functions over fingerprints: no database, no files. This is the part
|
||||
// that decides whether two tracks in the library are proposed as one recording,
|
||||
// so every rule in it has to be falsifiable in a unit test.
|
||||
//
|
||||
// Two tiers, answering different questions:
|
||||
//
|
||||
// exact equal audio_stream_sha256 — the same encoded audio bytes. No score,
|
||||
// no threshold, no false positives (the #3885 pair).
|
||||
// no threshold, no false positives.
|
||||
// acoustic chromaprint fingerprints that agree closely once aligned — the
|
||||
// same recording at another bitrate or in another codec.
|
||||
//
|
||||
@@ -52,23 +52,22 @@ const minDistinctFraction = 0.3
|
||||
// fingerprints may show and still be proposed as one recording. Unrelated audio
|
||||
// sits near 0.5; the same recording re-encoded lands well under 0.1.
|
||||
//
|
||||
// Deliberately conservative. The operator's stated worry is the opposite of a
|
||||
// missed duplicate: "the same song can appear in different albums, usually it's
|
||||
// a different recording", and an instrumental shares its vocal version's
|
||||
// harmony, which chroma features capture. A false merge is the failure that
|
||||
// matters, and the report is reviewed anyway. This is an unmeasured default:
|
||||
// calibrate it against real pairs once the backfill (#3908) has populated the
|
||||
// library, then expose it in Settings (#3913).
|
||||
// Deliberately conservative. A false merge is the failure that matters, worse
|
||||
// than a missed duplicate: the same song on different albums is usually a
|
||||
// different recording, and an instrumental shares its vocal version's harmony,
|
||||
// which chroma features capture. An admin can change it in Settings
|
||||
// (FingerprintSettings).
|
||||
const defaultAcousticMaxBitErrorRate = 0.15
|
||||
|
||||
// durationToleranceMs is how far apart two tracks' durations may be and still be
|
||||
// compared. Encoders pad and trim a little; different edits differ by more.
|
||||
// durationToleranceMs is how far apart two tracks' durations may be and still
|
||||
// be compared. Encoders pad and trim a little; different edits differ by more.
|
||||
const durationToleranceMs = 3000
|
||||
|
||||
// maxAcousticGroupSize caps an acoustic group. A cluster bigger than this is far
|
||||
// more likely a shared jingle, a skit or a low-information pattern than eight
|
||||
// copies of one recording, and proposing it would bury the real duplicates.
|
||||
// Exact-tier groups are not capped: identical bytes are identical however many.
|
||||
// maxAcousticGroupSize caps an acoustic group. A cluster bigger than this is
|
||||
// far more likely a shared jingle, a skit or a low-information pattern than
|
||||
// eight copies of one recording, and proposing it would bury the real
|
||||
// duplicates. Exact-tier groups are not capped: identical bytes are identical
|
||||
// however many.
|
||||
const maxAcousticGroupSize = 8
|
||||
|
||||
// acousticScore is the result of comparing two fingerprints.
|
||||
@@ -84,8 +83,8 @@ type acousticScore struct {
|
||||
|
||||
// preparedPrint is a fingerprint with the parts every comparison needs worked
|
||||
// out once. The sweep compares each track with every other track within a few
|
||||
// seconds of its duration, so rebuilding the alignment index for each pair would
|
||||
// dominate its cost.
|
||||
// seconds of its duration, so rebuilding the alignment index for each pair
|
||||
// would dominate its cost.
|
||||
type preparedPrint struct {
|
||||
items []int32
|
||||
index map[uint32][]int
|
||||
@@ -208,13 +207,13 @@ type duplicateGroup struct {
|
||||
type groupingResult struct {
|
||||
Groups []duplicateGroup
|
||||
// OversizeClusters counts acoustic clusters discarded for exceeding
|
||||
// maxAcousticGroupSize. Reported rather than silent: a sudden rise means the
|
||||
// cap or the information floor needs attention.
|
||||
// maxAcousticGroupSize. Reported rather than silent: a sudden rise means
|
||||
// the cap or the information floor needs attention.
|
||||
OversizeClusters int
|
||||
}
|
||||
|
||||
// groupUnit is one thing the acoustic pass compares: a single track, or an exact
|
||||
// group standing in for all its byte-identical copies.
|
||||
// groupUnit is one thing the acoustic pass compares: a single track, or an
|
||||
// exact group standing in for all its byte-identical copies.
|
||||
type groupUnit struct {
|
||||
ids []string // every member, sorted
|
||||
durationMs int32
|
||||
@@ -224,11 +223,11 @@ type groupUnit struct {
|
||||
assigned bool
|
||||
}
|
||||
|
||||
// streamGrouper is the acoustic pass over units arriving in (durationMs, sortKey)
|
||||
// order. It holds only the units within durationToleranceMs of the oldest one
|
||||
// not yet settled, so memory is bounded by the densest few seconds of the
|
||||
// library rather than by its size — the whole library's fingerprints would be
|
||||
// hundreds of megabytes.
|
||||
// streamGrouper is the acoustic pass over units arriving in (durationMs,
|
||||
// sortKey) order. It holds only the units within durationToleranceMs of the
|
||||
// oldest one not yet settled, so memory is bounded by the densest few seconds
|
||||
// of the library rather than by its size — the whole library's fingerprints
|
||||
// would be hundreds of megabytes.
|
||||
//
|
||||
// A seed can be settled as soon as a unit arrives beyond its window: everything
|
||||
// it could group with has already arrived, and no later seed can reach back to
|
||||
@@ -237,9 +236,9 @@ type groupUnit struct {
|
||||
//
|
||||
// Grouping is COMPLETE-LINKAGE: a unit joins a group only if it matches every
|
||||
// unit already in it, within the duration tolerance and the bit-error limit.
|
||||
// Single-linkage would let a chain of near-misses — A close to B, B close to C —
|
||||
// drag A and C, which are not close, into one proposed merge. Complete linkage
|
||||
// also means any member can be chosen as the survivor (#3911).
|
||||
// Single-linkage would let a chain of near-misses — A close to B, B close to C
|
||||
// — drag A and C, which are not close, into one proposed merge. Complete
|
||||
// linkage also means any member can be chosen as the survivor.
|
||||
//
|
||||
// When an acoustic group absorbs an exact group, the result is tier acoustic: a
|
||||
// group is only as certain as its weakest link.
|
||||
@@ -320,7 +319,8 @@ func (g *streamGrouper) settleOldest() {
|
||||
for _, member := range group {
|
||||
member.assigned = true
|
||||
// The acoustic evidence is untrustworthy; identical bytes are not.
|
||||
// An exact group caught inside an oversize cluster is still proposed.
|
||||
// An exact group caught inside an oversize cluster is still
|
||||
// proposed.
|
||||
if member.exact {
|
||||
g.res.Groups = append(g.res.Groups, duplicateGroup{Tier: tierExact, Members: member.ids})
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user