Files
minstrel/internal/dbtest/reset.go
T
bvandeusenandClaude Opus 5 6379b6c31d
test-go / test (push) Successful in 1m0s
test-go / integration (push) Successful in 3m17s
release / Build signed APK (releases and dev) (push) Successful in 4m51s
release / Build + push container image (push) Successful in 14s
release / Verify release artifacts (tag releases only) (push) Skipped
feat(library): the duplicate sweep — propose duplicate groups from fingerprints (M400 #3910)
Reads fingerprints, runs them through the matcher, and records
proposals in duplicate_groups (migration 0059). Nothing is merged or
deleted: a group is a proposal for the admin report (#3912).

Streaming. The whole library's fingerprints are hundreds of megabytes,
but tracks are only compared within 3s of each other in duration. So
candidates stream in (duration_ms, id) order, keyset-paged on a new
tracks(duration_ms, id) index. The grouper holds only the tracks within
3s of the oldest one not yet settled. A seed is settled once a track
arrives beyond its window, which gives the same result as grouping the
whole sorted list. groupDuplicates is rebuilt on the same streamGrouper,
so there is one grouping rule and the #3909 tests still cover it. Each
fingerprint's alignment index and variety check are computed once
instead of for every pair.

Exact duplicates are grouped library-wide in SQL. The first member the
stream meets stands in for the whole group in the acoustic pass. An
exact group caught in an oversize acoustic cluster is still proposed:
the acoustic evidence is discarded, identical bytes are not.

Re-sweeping:
- a group is identified by its sorted member ids, so finding it again
  refreshes the row in place
- a proposal whose members all sat in one dismissed group is not
  proposed again (a subset repeats the verdict; a superset is new
  evidence)
- a pending proposal no sweep has found again is retired, but only
  after a complete sweep, and only if an earlier sweep last confirmed
  it, so two overlapping sweeps cannot delete each other's findings
- dismissals are kept

DuplicateSweepWorker checks hourly and sweeps only when a fingerprint
was written after the last sweep started. TryStartDuplicateSweep guards
against two sweeps at once and reaps one stuck in flight for 2h. The
sweep row is closed on a detached context with a deadline, so a sweep
cancelled at shutdown still records that it ended.

The integration test pages one row at a time and checks:
- an acoustic pair and an exact pair are found
- a track with no fingerprint, a missing track and a near-duration
  unrelated song are left out
- a dismissed group is suppressed while the pending one refreshes
  without duplicating
- a proposal that stops holding is retired and the dismissal survives

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
2026-09-11 17:00:07 -04:00

134 lines
5.4 KiB
Go

// Package dbtest provides shared helpers for integration tests that need
// a clean Postgres state without disturbing the operator's admin user.
//
// Background: when integration tests run against the same Postgres
// instance an operator uses for local dev (a common dev-laptop setup),
// a TRUNCATE that includes the users table wipes the operator's admin
// login between every test run. To avoid that, tests should:
//
// 1. Call ResetDB instead of issuing TRUNCATE statements directly. It
// truncates every data table EXCEPT users, then deletes only those
// users whose username starts with TestUserPrefix.
// 2. Always create test user rows with a username that begins with
// TestUserPrefix; otherwise those rows survive across runs and
// leak into other tests.
//
// One test legitimately needs to wipe the entire users table:
// internal/auth/bootstrap_test.go, which exercises first-time admin
// bootstrap. That file does its own TRUNCATE and does not use this
// helper.
package dbtest
import (
"context"
"strings"
"testing"
"github.com/jackc/pgx/v5/pgxpool"
)
// TestUserPrefix is the required username prefix for any user row
// created by an integration test. ResetDB removes every user matching
// this prefix and leaves all others intact.
const TestUserPrefix = "test-"
// dataTables is the union of every non-users table touched by any
// integration test in the repo. Truncating the union is harmless for
// callers that only care about a subset; truncating tables that don't
// exist would error, but every name here corresponds to a migration
// that has shipped.
var dataTables = []string{
"artist_similarity",
"track_similarity",
"artist_similarity_unmatched", // M5c
"scrobble_queue",
"contextual_likes",
"general_likes_albums",
"general_likes_artists",
"general_likes",
"play_events",
"skip_events",
"play_sessions",
"sessions",
"lidarr_quarantine_actions",
"lidarr_quarantine",
// #2374. Keyed by (user_id, candidate_mbid) with no FK to artists —
// candidates are out-of-library — so the CASCADE from artists/users
// does NOT reach it for a leftover row whose user survived. Truncate
// explicitly or a stale snooze silently hides a candidate from the
// next test's suggestion assertions.
"suggestion_snoozes",
// #2376. Same reasoning: keyed by candidate MBID with no FK anywhere,
// so nothing cascades to them. A leftover tag row would make a
// candidate look enriched to the next test, and a leftover state row
// would make it look already-settled and thus ineligible.
"candidate_artist_tags",
"candidate_artist_tag_state",
"playlist_tracks",
"playlists",
"library_changes", // M7 #357 — must reset to keep cursor isolated per test
// SettingsService.reconcile() idempotently re-UpsertProviderSettings
// for every registered provider at boot, so truncating this is the
// correct per-test reset (clears test-modified enabled/api_key rows).
// cover_art_sources_meta is NOT truncated — boot only READS it
// (never recreates the singleton, seeded once by 0018); ResetDB
// resets its counter via UPDATE below instead.
"cover_art_provider_settings",
// Same reasoning for the tag-enrichment settings (#1490): truncate the
// per-provider rows (reconcile re-seeds registered ones), and reset the
// tag_sources_meta counter via UPDATE below rather than truncating it.
"tag_provider_settings",
// recsettings.New reconciles shipped defaults on every construction
// (#1250), so truncating gives each test pristine tuning values.
"recommendation_weight_profiles",
"taste_tuning",
// #2377. Same reasoning as taste_tuning above: recsettings.New re-seeds
// shipped defaults on every construction, so truncating gives each test
// pristine Discover knobs rather than whatever a previous test tuned.
"discover_tuning",
"recommendation_tuning_audit",
"duplicate_group_members", // M400
"duplicate_groups",
"duplicate_sweeps",
"track_fingerprints", // M400
"tracks",
"albums",
"artists",
}
// ResetDB clears all data tables and removes any user whose username
// begins with TestUserPrefix. Users without that prefix (notably the
// operator's admin row) are left alone. Calls t.Fatalf on error.
func ResetDB(t *testing.T, pool *pgxpool.Pool) {
t.Helper()
ctx := context.Background()
stmt := "TRUNCATE " + strings.Join(dataTables, ", ") + " RESTART IDENTITY CASCADE"
if _, err := pool.Exec(ctx, stmt); err != nil {
t.Fatalf("dbtest.ResetDB truncate: %v", err)
}
if _, err := pool.Exec(ctx,
"DELETE FROM users WHERE username LIKE $1",
TestUserPrefix+"%",
); err != nil {
t.Fatalf("dbtest.ResetDB delete test users: %v", err)
}
// Reset the monotonic cover-art source-version counter to its
// post-migration seeded value. Truncating the row would break
// SettingsService boot, which reads (never recreates) this
// singleton; an UPDATE keeps the row while clearing cross-test
// version accumulation (CurrentVersion=4 want 1, key-only-bump).
if _, err := pool.Exec(ctx,
"UPDATE cover_art_sources_meta SET current_version = 1",
); err != nil {
t.Fatalf("dbtest.ResetDB reset cover-art version: %v", err)
}
// Same for the tag-sources counter (#1490) — clears cross-test version
// accumulation that would spuriously report version_bumped on a
// key-only change.
if _, err := pool.Exec(ctx,
"UPDATE tag_sources_meta SET current_version = 1, last_registered_providers_hash = ''",
); err != nil {
t.Fatalf("dbtest.ResetDB reset tag-sources version: %v", err)
}
}