feat: duplicates resolve themselves where Lidarr says it is safe (M498)
release / govulncheck (push) Successful in 45s
release / web (push) Successful in 1m23s
release / go (push) Successful in 1m39s
release / integration (push) Successful in 4m25s
release / android (push) Successful in 6m17s
release / Build signed APK (releases and dev) (push) Successful in 5m57s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 1m54s
release / Verify release artifacts (tag releases only) (push) Skipped
release / govulncheck (push) Successful in 45s
release / web (push) Successful in 1m23s
release / go (push) Successful in 1m39s
release / integration (push) Successful in 4m25s
release / android (push) Successful in 6m17s
release / Build signed APK (releases and dev) (push) Successful in 5m57s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 1m54s
release / Verify release artifacts (tag releases only) (push) Skipped
The duplicate sweep proposed 4,197 groups and every one waited for the operator. Most are safe to settle, and Lidarr defines what safe means: it maps one file to each track of the release it monitors and downloads any mapped file that disappears. Deleting a mapped copy opens exactly the hole the operator saw Lidarr fill. Classify (#5435) - Migration 0075: duplicate_groups.class (same_release, cross_release, mismatch, review), resolve_note, resolved_automatically; duplicate_group_members.lidarr_state (tracked, unmapped); fingerprint_settings.auto_resolve; notification kind duplicates_resolved with both kind CHECKs swapped (rule 36). - library.ClassifyDuplicateGroup, with MatchTitleKey dropping featuring credits, remaster notes and video-rip markers, and keeping live, demo, remix and instrumental. The rip markers move from api to library. Choose the copy to keep (#5436) - ProposeSurvivor ranks the copy Lidarr maps first, then tag fit (a clash-free track number, no rip marker in the name, an MBID), then the quality rules. File size picked the wrong Humanz copy in 6 of 21 groups. Act (#5437) - An hourly resolver pass reads Lidarr's unmapped files, matched by the last three path components, and records each copy's state. - Same album, with at most one copy mapped: merged into the mapped copy. The merge is guarded, so a mapped copy can never be removed (MergeDuplicateGroupGuarded, ErrCopyTrackedByLidarr). - Same album, every copy mapped: the monitored release lists the song twice (Humanz's 14x12" box set). The pass moves Lidarr to the release that lists each song once and best covers what is on disk. It never picks one covering less, and is capped at 10 albums per pass. - Fixed point (lesson #4183): the chosen release no longer repeats. - The album is left alone for 24h while Lidarr rescans, so "every copy unmapped" mid-rescan is never read as licence to merge. - Both actions are audited with no actor and summarised to admins. The operator can switch them off in the Fingerprinting card (rule 25). - Manual merges use the same guard: 409 copy_tracked_by_lidarr, or 503 lidarr_unavailable when Lidarr cannot say. Web - Duplicates gets tabs: Needs review, Across releases, Resolved automatically. Each loads as you scroll (rule 172), replacing the pager. - Each copy says whether Lidarr uses it. - The resolver's note shows on each group. - The merge confirm blocks, before sending, a merge that would remove the copy Lidarr uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,143 @@
|
||||
package library
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// Duplicate classification (M498 #5435).
|
||||
//
|
||||
// A duplicate group is one recording found more than once. What should happen
|
||||
// to it depends on where the copies sit, and the measured library (milestone
|
||||
// 498's body) splits into four shapes:
|
||||
//
|
||||
// - same_release: every copy is on one album. An album holds a song once, so
|
||||
// the extra copies are the ones to fold away — when Lidarr does not need
|
||||
// them (the resolver checks).
|
||||
// - cross_release: the copies are on different albums under one title. A
|
||||
// single and the album it came from, a deluxe edition and the standard one:
|
||||
// each copy fulfils its own release in Lidarr, so none is deleted.
|
||||
// - mismatch: byte-identical audio on different albums under different
|
||||
// titles. One file carries another song's tags — a wrong-file import.
|
||||
// - review: anything else. The operator decides.
|
||||
|
||||
// DuplicateClass is a group's shape. The values are duplicate_groups.class.
|
||||
type DuplicateClass string
|
||||
|
||||
const (
|
||||
ClassSameRelease DuplicateClass = "same_release"
|
||||
ClassCrossRelease DuplicateClass = "cross_release"
|
||||
ClassMismatch DuplicateClass = "mismatch"
|
||||
ClassReview DuplicateClass = "review"
|
||||
)
|
||||
|
||||
// ClassifyMember is what classifying needs to know about one copy.
|
||||
type ClassifyMember struct {
|
||||
AlbumID string
|
||||
Title string
|
||||
ArtistName string
|
||||
}
|
||||
|
||||
// ClassifyDuplicateGroup decides a group's shape from its tier ("exact" or
|
||||
// "acoustic") and its copies.
|
||||
//
|
||||
// Titles are compared by MatchTitleKey, which drops what a release or an upload
|
||||
// adds to a title (featuring credits, remaster notes, video markers) and keeps
|
||||
// what names a different recording (live, demo, remix, instrumental). So one
|
||||
// album holding "WWW" and "WWW (instrumental)" as an acoustic match is review,
|
||||
// not a copy to fold away: the #3885 case, where they were in fact different
|
||||
// recordings mislabelled. Byte-identical copies on one album are the same file
|
||||
// twice whatever their tags say.
|
||||
func ClassifyDuplicateGroup(tier string, members []ClassifyMember) DuplicateClass {
|
||||
if len(members) < 2 {
|
||||
return ClassReview
|
||||
}
|
||||
oneAlbum, oneTitle := true, true
|
||||
key := MatchTitleKey(members[0].Title, members[0].ArtistName)
|
||||
for _, m := range members[1:] {
|
||||
if m.AlbumID != members[0].AlbumID {
|
||||
oneAlbum = false
|
||||
}
|
||||
if MatchTitleKey(m.Title, m.ArtistName) != key {
|
||||
oneTitle = false
|
||||
}
|
||||
}
|
||||
exact := tier == string(tierExact)
|
||||
switch {
|
||||
case oneAlbum && (oneTitle || exact):
|
||||
return ClassSameRelease
|
||||
case oneAlbum:
|
||||
return ClassReview
|
||||
case oneTitle:
|
||||
return ClassCrossRelease
|
||||
case exact:
|
||||
return ClassMismatch
|
||||
default:
|
||||
return ClassReview
|
||||
}
|
||||
}
|
||||
|
||||
var (
|
||||
// A bracketed credit: "(feat. Panda Bear)", "[ft. X]", "(featuring Y)".
|
||||
featBracket = regexp.MustCompile(`(?i)[(\[][^)\]]*\b(feat\.?|ft\.?|featuring)\s[^)\]]*[)\]]`)
|
||||
// A credit trailing the title without brackets: "Doin' It Right ft. Panda Bear".
|
||||
featTrailing = regexp.MustCompile(`(?i)\s(feat\.?|ft\.?|featuring)\s.*$`)
|
||||
// "(2011 Remaster)", "[Remastered]", "- Remastered 2009".
|
||||
remasterBracket = regexp.MustCompile(`(?i)[(\[][^)\]]*remaster[^)\]]*[)\]]`)
|
||||
remasterSuffix = regexp.MustCompile(`(?i)\s-\s[^-]*remaster.*$`)
|
||||
// Any bracketed segment, to test each against the video-rip markers.
|
||||
bracketed = regexp.MustCompile(`[(\[][^)\]]*[)\]]`)
|
||||
)
|
||||
|
||||
// MatchTitleKey reduces a title to what identifies the song, for deciding
|
||||
// whether two copies carry the same one.
|
||||
//
|
||||
// Dropped: a leading "Artist - " (a video upload's title), bracketed featuring
|
||||
// credits and remaster notes, and bracketed segments carrying a video-rip marker
|
||||
// ("(Official Video)", "[HD]"). Kept: every other bracketed word, so "(live)",
|
||||
// "(demo)", "(acoustic)", "(remix)" and "(instrumental)" still tell recordings
|
||||
// apart. Then case, "&" against "and", and punctuation and spacing are ignored.
|
||||
func MatchTitleKey(title, artist string) string {
|
||||
t := strings.TrimSpace(title)
|
||||
if a := strings.TrimSpace(artist); a != "" {
|
||||
if prefix := a + " - "; len(t) > len(prefix) && strings.EqualFold(t[:len(prefix)], prefix) {
|
||||
t = t[len(prefix):]
|
||||
}
|
||||
}
|
||||
t = featBracket.ReplaceAllString(t, " ")
|
||||
t = remasterBracket.ReplaceAllString(t, " ")
|
||||
t = bracketed.ReplaceAllStringFunc(t, func(seg string) string {
|
||||
for _, re := range suspectSourceRegexps {
|
||||
if re.MatchString(seg) {
|
||||
return " "
|
||||
}
|
||||
}
|
||||
return seg
|
||||
})
|
||||
t = remasterSuffix.ReplaceAllString(t, "")
|
||||
t = featTrailing.ReplaceAllString(t, "")
|
||||
if key := alnumKey(t); key != "" {
|
||||
return key
|
||||
}
|
||||
return alnumKey(title)
|
||||
}
|
||||
|
||||
// ReleaseTitleKey is the lighter comparison for asking whether one release
|
||||
// lists a song twice: case, "&" and punctuation only. Every word stays, so a
|
||||
// box set's "Song" and "Song (demo)" are two songs, as the release means them.
|
||||
func ReleaseTitleKey(title string) string {
|
||||
return alnumKey(title)
|
||||
}
|
||||
|
||||
// alnumKey lowercases, reads "&" as "and", and keeps only letters and digits.
|
||||
func alnumKey(s string) string {
|
||||
s = strings.ReplaceAll(strings.ToLower(s), "&", " and ")
|
||||
var b strings.Builder
|
||||
for _, r := range s {
|
||||
if unicode.IsLetter(r) || unicode.IsDigit(r) {
|
||||
b.WriteRune(r)
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
Reference in New Issue
Block a user