feat: duplicates resolve themselves where Lidarr says it is safe (M498)
release / govulncheck (push) Successful in 45s
release / web (push) Successful in 1m23s
release / go (push) Successful in 1m39s
release / integration (push) Successful in 4m25s
release / android (push) Successful in 6m17s
release / Build signed APK (releases and dev) (push) Successful in 5m57s
release / Attach APK to the Release (tag releases only) (push) Skipped
release / Build + push container image (push) Successful in 1m54s
release / Verify release artifacts (tag releases only) (push) Skipped

The duplicate sweep proposed 4,197 groups and every one waited for the
operator. Most are safe to settle, and Lidarr defines what safe means: it
maps one file to each track of the release it monitors and downloads any
mapped file that disappears. Deleting a mapped copy opens exactly the hole
the operator saw Lidarr fill.

Classify (#5435)
- Migration 0075: duplicate_groups.class (same_release, cross_release,
  mismatch, review), resolve_note, resolved_automatically;
  duplicate_group_members.lidarr_state (tracked, unmapped);
  fingerprint_settings.auto_resolve; notification kind
  duplicates_resolved with both kind CHECKs swapped (rule 36).
- library.ClassifyDuplicateGroup, with MatchTitleKey dropping featuring
  credits, remaster notes and video-rip markers, and keeping live, demo,
  remix and instrumental. The rip markers move from api to library.

Choose the copy to keep (#5436)
- ProposeSurvivor ranks the copy Lidarr maps first, then tag fit (a
  clash-free track number, no rip marker in the name, an MBID), then the
  quality rules. File size picked the wrong Humanz copy in 6 of 21 groups.

Act (#5437)
- An hourly resolver pass reads Lidarr's unmapped files, matched by the
  last three path components, and records each copy's state.
- Same album, with at most one copy mapped: merged into the mapped copy.
  The merge is guarded, so a mapped copy can never be removed
  (MergeDuplicateGroupGuarded, ErrCopyTrackedByLidarr).
- Same album, every copy mapped: the monitored release lists the song
  twice (Humanz's 14x12" box set). The pass moves Lidarr to the release
  that lists each song once and best covers what is on disk. It never
  picks one covering less, and is capped at 10 albums per pass.
  - Fixed point (lesson #4183): the chosen release no longer repeats.
  - The album is left alone for 24h while Lidarr rescans, so "every copy
    unmapped" mid-rescan is never read as licence to merge.
- Both actions are audited with no actor and summarised to admins. The
  operator can switch them off in the Fingerprinting card (rule 25).
- Manual merges use the same guard: 409 copy_tracked_by_lidarr, or 503
  lidarr_unavailable when Lidarr cannot say.

Web
- Duplicates gets tabs: Needs review, Across releases, Resolved
  automatically. Each loads as you scroll (rule 172), replacing the
  pager.
- Each copy says whether Lidarr uses it.
- The resolver's note shows on each group.
- The merge confirm blocks, before sending, a merge that would remove
  the copy Lidarr uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 21:49:37 -04:00
co-authored by Claude Opus 5.5
parent 01e2294471
commit 4ecff52f19
48 changed files with 3068 additions and 254 deletions
+29 -4
View File
@@ -1,5 +1,13 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { dismissDuplicateGroup, listDuplicates, mergeDuplicateGroup, runDuplicateSweep } from './admin';
import {
dismissDuplicateGroup,
duplicatesNextOffset,
listDuplicates,
listResolvedDuplicates,
mergeDuplicateGroup,
runDuplicateSweep
} from './admin';
import type { AdminDuplicatesResponse } from './types';
vi.mock('./client', () => ({
api: { get: vi.fn(), post: vi.fn() }
@@ -10,10 +18,27 @@ import { api } from './client';
describe('admin duplicates API', () => {
beforeEach(() => vi.clearAllMocks());
it('listDuplicates GETs the paged report', async () => {
it('listDuplicates GETs one tab of the paged report', async () => {
(api.get as unknown as ReturnType<typeof vi.fn>).mockResolvedValueOnce({ groups: [] });
await listDuplicates(25, 25);
expect(api.get).toHaveBeenCalledWith('/api/admin/library/duplicates?limit=25&offset=25');
await listDuplicates('cross_release', 25);
expect(api.get).toHaveBeenCalledWith(
'/api/admin/library/duplicates?view=cross_release&limit=25&offset=25'
);
});
it('listResolvedDuplicates GETs what the resolver did', async () => {
(api.get as unknown as ReturnType<typeof vi.fn>).mockResolvedValueOnce({ items: [] });
await listResolvedDuplicates(50);
expect(api.get).toHaveBeenCalledWith('/api/admin/library/duplicates/resolved?limit=25&offset=50');
});
// The next offset counts groups, and stops at the total or on an empty page.
it('duplicatesNextOffset pages by groups', () => {
const page = (offset: number, n: number, total: number) =>
({ offset, total, groups: Array.from({ length: n }, () => ({})) }) as unknown as AdminDuplicatesResponse;
expect(duplicatesNextOffset(page(0, 25, 60))).toBe(25);
expect(duplicatesNextOffset(page(50, 10, 60))).toBeUndefined();
expect(duplicatesNextOffset(page(25, 0, 60))).toBeUndefined();
});
it('runDuplicateSweep POSTs the trigger', async () => {
+2 -1
View File
@@ -49,7 +49,8 @@ describe('admin fingerprint coverage API', () => {
chromaprint_length_sec: 120,
acoustic_max_bit_error_rate: 0.15,
backfill_concurrency: 2,
sweep_interval_hours: 1
sweep_interval_hours: 1,
auto_resolve: true
};
(api.get as unknown as ReturnType<typeof vi.fn>).mockResolvedValueOnce(settings);
(api.put as unknown as ReturnType<typeof vi.fn>).mockResolvedValueOnce(settings);
+46 -10
View File
@@ -6,7 +6,9 @@ import type {
AdminMissingResponse,
AdminSuspectResponse,
AdminDuplicatesResponse,
DuplicatesView,
MergeDuplicateResult,
ResolvedDuplicatesResponse,
AdminPlaybackError,
AdminQuarantineRow,
LidarrConfig,
@@ -353,6 +355,9 @@ export type FingerprintSettings = {
acoustic_max_bit_error_rate: number;
backfill_concurrency: number;
sweep_interval_hours: number;
// Lets the duplicate resolver act on its own (M498): merge copies Lidarr
// doesn't need and change a Lidarr release that lists songs twice.
auto_resolve: boolean;
};
export async function getFingerprintSettings(): Promise<FingerprintSettings> {
@@ -829,24 +834,55 @@ export async function updatePublicUrl(publicUrl: string): Promise<NetworkSetting
// Duplicates report (#3912) -------------------------------------------------
export const DUPLICATES_PAGE_SIZE = 25;
export async function listDuplicates(
view: DuplicatesView = 'review',
offset: number = 0,
limit: number = 25
limit: number = DUPLICATES_PAGE_SIZE
): Promise<AdminDuplicatesResponse> {
return api.get<AdminDuplicatesResponse>(
`/api/admin/library/duplicates?limit=${limit}&offset=${offset}`
`/api/admin/library/duplicates?view=${view}&limit=${limit}&offset=${offset}`
);
}
// Takes a plain offset, like createMissingFilesQuery, so a $derived caller
// re-creates the query on paging. Polls while a sweep might be running: the
// page is where the operator waits for one to finish.
export function createDuplicatesQuery(offset: number = 0, limit: number = 25) {
return createQuery({
queryKey: qk.adminDuplicates(offset),
queryFn: () => listDuplicates(offset, limit),
export function duplicatesNextOffset(last: AdminDuplicatesResponse): number | undefined {
const loaded = last.offset + last.groups.length;
return last.groups.length === 0 || loaded >= last.total ? undefined : loaded;
}
// Loads as the page scrolls (rule 172), one tab at a time. Polls so a sweep
// or resolver pass finishing shows up while the operator is on the page.
export function createDuplicatesQuery(view: DuplicatesView = 'review') {
return createInfiniteQuery({
queryKey: qk.adminDuplicates(view),
queryFn: ({ pageParam }) => listDuplicates(view, pageParam as number),
initialPageParam: 0,
getNextPageParam: duplicatesNextOffset,
staleTime: 30_000,
refetchInterval: 15_000
refetchInterval: 30_000
});
}
export async function listResolvedDuplicates(offset: number = 0): Promise<ResolvedDuplicatesResponse> {
return api.get<ResolvedDuplicatesResponse>(
`/api/admin/library/duplicates/resolved?limit=${DUPLICATES_PAGE_SIZE}&offset=${offset}`
);
}
export function resolvedDuplicatesNextOffset(last: ResolvedDuplicatesResponse): number | undefined {
const loaded = last.offset + last.items.length;
return last.items.length === 0 || loaded >= last.total ? undefined : loaded;
}
// What the resolver did by itself (M498), newest first.
export function createResolvedDuplicatesQuery() {
return createInfiniteQuery({
queryKey: qk.adminResolvedDuplicates(),
queryFn: ({ pageParam }) => listResolvedDuplicates(pageParam as number),
initialPageParam: 0,
getNextPageParam: resolvedDuplicatesNextOffset,
staleTime: 60_000
});
}
+1
View File
@@ -122,5 +122,6 @@ export const NOTIFICATION_KIND_LABELS: Record<string, string> = {
scan_failed: 'Library scan failed',
tracks_missing: 'Tracks gone missing',
duplicates_found: 'Duplicates to review',
duplicates_resolved: 'Duplicates resolved',
playback_errors: 'Playback errors'
};
+3 -2
View File
@@ -56,8 +56,9 @@ export const qk = {
['adminDiagnostics', f] as const,
adminMissingFiles: (offset?: number) =>
['adminMissingFiles', { offset: offset ?? 0 }] as const,
adminDuplicates: (offset?: number) =>
['adminDuplicates', { offset: offset ?? 0 }] as const,
adminDuplicates: (view: string = 'review') =>
['adminDuplicates', { view }] as const,
adminResolvedDuplicates: () => ['adminResolvedDuplicates'] as const,
adminSuspectSources: () => ['adminSuspectSources'] as const,
adminDiagnosticDevices: (userId?: string) =>
['adminDiagnosticDevices', { userId: userId ?? 'all' }] as const,
+44
View File
@@ -471,8 +471,17 @@ export type AdminDuplicateMember = {
added_at: string;
like_count: number;
play_count: number;
// Lidarr's view of the file as of the resolver's last pass (M498): tracked
// means Lidarr maps it to a track, so removing it would make Lidarr download
// it again. null when Lidarr wasn't asked.
lidarr_state: 'tracked' | 'unmapped' | null;
disc_number: number | null;
track_number: number | null;
};
// The resolver's verdict on a group (M498). null until its first pass.
export type DuplicateClass = 'same_release' | 'cross_release' | 'mismatch' | 'review';
// exact: identical encoded audio. acoustic: the same recording, differently
// encoded; worst_bit_error_rate is the weakest link between any two members.
export type AdminDuplicateGroup = {
@@ -482,6 +491,9 @@ export type AdminDuplicateGroup = {
detected_at: string;
survivor_track_id: string;
survivor_reason: string;
class: DuplicateClass | null;
// Why the resolver left the group for the operator, when it had something to say.
resolve_note: string | null;
members: AdminDuplicateMember[];
};
@@ -513,8 +525,40 @@ export type AdminDuplicatesResponse = {
pending: number;
enabled: boolean;
};
view: DuplicatesView;
// Each tab's size, for its pill.
counts: { review: number; cross_release: number; resolved: number };
total: number;
limit: number;
offset: number;
groups: AdminDuplicateGroup[];
};
// The report's group tabs. 'resolved' is the third tab, served separately.
export type DuplicatesView = 'review' | 'cross_release';
// One thing the resolver did on its own (M498), from the audit log.
export type ResolvedDuplicateActivity =
| {
id: string;
action: 'duplicate_merge';
created_at: string;
details: {
survivor_path?: string;
survivor_reason?: string;
removed?: { track_id: string; file_path: string }[];
};
}
| {
id: string;
action: 'lidarr_release_change';
created_at: string;
details: { album_title?: string; artist_name?: string; from_release?: string; to_release?: string };
};
export type ResolvedDuplicatesResponse = {
total: number;
limit: number;
offset: number;
items: ResolvedDuplicateActivity[];
};