Rule 25: the fingerprinting knobs move out of source into a DB-backed
singleton (migration 0061), edited from a card on the Duplicates page and
shared live with the scanner, the backfill and the duplicate sweep through
one service instance, so a save needs no restart.
The length is the knob that can silently break the library: prints taken
at two lengths never match. Each track_fingerprints row now records the
length it was taken at, and every reader filters on the current one — the
backfill treats another length as stale, the gauge counts it pending, the
sweep never streams it. Equivalent to a version bump, except that setting
the length back makes rows not yet redone current again. The card warns
before a length change re-fingerprints the library.
Off stops every decode: the scan takes only the stream hash (a demux, and
what recognises a moved file) and stores nothing, dropping a changed file's
stale row; the backfill idles. A save also makes a sweep due, since a new
threshold or length changes what the same prints group into, and the sweep
interval gains slack so an hourly interval on an hourly tick doesn't skip
every other tick.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
Reads fingerprints, runs them through the matcher, and records
proposals in duplicate_groups (migration 0059). Nothing is merged or
deleted: a group is a proposal for the admin report (#3912).
Streaming. The whole library's fingerprints are hundreds of megabytes,
but tracks are only compared within 3s of each other in duration. So
candidates stream in (duration_ms, id) order, keyset-paged on a new
tracks(duration_ms, id) index. The grouper holds only the tracks within
3s of the oldest one not yet settled. A seed is settled once a track
arrives beyond its window, which gives the same result as grouping the
whole sorted list. groupDuplicates is rebuilt on the same streamGrouper,
so there is one grouping rule and the #3909 tests still cover it. Each
fingerprint's alignment index and variety check are computed once
instead of for every pair.
Exact duplicates are grouped library-wide in SQL. The first member the
stream meets stands in for the whole group in the acoustic pass. An
exact group caught in an oversize acoustic cluster is still proposed:
the acoustic evidence is discarded, identical bytes are not.
Re-sweeping:
- a group is identified by its sorted member ids, so finding it again
refreshes the row in place
- a proposal whose members all sat in one dismissed group is not
proposed again (a subset repeats the verdict; a superset is new
evidence)
- a pending proposal no sweep has found again is retired, but only
after a complete sweep, and only if an earlier sweep last confirmed
it, so two overlapping sweeps cannot delete each other's findings
- dismissals are kept
DuplicateSweepWorker checks hourly and sweeps only when a fingerprint
was written after the last sweep started. TryStartDuplicateSweep guards
against two sweeps at once and reaps one stuck in flight for 2h. The
sweep row is closed on a detached context with a deadline, so a sweep
cancelled at shutdown still records that it ended.
The integration test pages one row at a time and checks:
- an acoustic pair and an exact pair are found
- a track with no fingerprint, a missing track and a near-duration
unrelated song are left out
- a dismissed group is suppressed while the pending one refreshes
without duplicating
- a proposal that stops holding is retired and the dismissal survives
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
Two identities per track, because they answer different questions:
- audio_stream_sha256: SHA-256 of the ENCODED audio packets
(ffmpeg -map 0:a -c:a copy -f hash). Equal means identical audio
whatever the tags say. Measured against the #3885 pair: the two WWW
files hash identically here and differently as whole files. Packets
rather than decoded samples, so an ffmpeg upgrade cannot silently
change every stored hash, and nothing is decoded.
- chromaprint: fpcalc -raw -signed. The same recording at another
bitrate or codec, for the acoustic tier.
fpcalc ships in the image (libchromaprint-tools); shelled out because
CGO_ENABLED=0 rules out bindings.
Stored in a track_fingerprints table rather than on tracks: eight
queries read tracks with SELECT *, including album pages, search and
the Subsonic surface, and a ~4 KB array there would be de-TOASTed on
every one of them.
The scan fingerprints only bytes it has not seen (a new path, or mtime
past the row's). A tag-repair pass leaves fingerprints alone, and
unchanged files with no fingerprint are the backfill's job (#3908).
Folding that into the skip check would re-decode the whole library on
the first scan after upgrade and push a sync change per track.
A failure that says nothing about the file (timeout, cancelled scan,
tool not installed) is never stored, and on changed bytes it removes
the old row. A tool that rejects the file stores NULL at the current
version, so the backfill does not retry it every boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQ31KQpYbStyK5y58UmPLH
The payoff slice. Until now a candidate's only claim on a slot was "some
artist you play is adjacent to it in a similarity graph" — a fact that says
nothing about whether the music sounds like anything you like. Now the
candidate's own folksonomy tags (cached by slice 5) are compared against
the user's taste-profile tags, so the deck ranks on taste and can say WHY.
The blend is MULTIPLICATIVE — score × (1 + weight × overlap) — and that
choice carries the whole safety argument:
- An untagged candidate has overlap 0, so its score is EXACTLY unchanged.
Tag coverage is permanently partial (#2376); it must cost a candidate
nothing, not sink it (rule #131).
- Nothing can leapfrog on tags alone. An additive term with a large
weight would let a near-zero-similarity artist outrank a strong match
for sharing one popular tag, which reads as noise.
- Weight 0 restores pure similarity order bit-for-bit, so the operator's
knob has a real off position.
overlap = Σ(shared) candWeight × normalizedTasteWeight ÷ Σ(all) candWeight.
Normalizing the taste side by the user's strongest tag makes the score
comparable across users (taste weights accumulate with listening, so a
heavy listener's raw numbers dwarf a new user's while meaning the same
thing). Dividing by the candidate's own mass makes it comparable across
candidates, so a densely-tagged artist can't win on tag count alone.
Applied to the whole over-fetched pool BEFORE selectSuggestions, so the
rotation and diversity rules operate on blended scores — boosting only the
twelve already chosen by similarity would leave the re-ranking undone.
A query failure is returned, NOT degraded past. Graceful degradation is
for expected absence (no taste profile, no cached tags) and both are
handled explicitly as empty inputs; swallowing a real error would hide a
broken DB behind a subtly worse ranking that nothing reports.
Migration 0051 adds a FOURTH tuning scope rather than columns on
taste_tuning, because snooze_days lives here too and a snooze must never
be read as taste signal (#2374) — filing it under 'taste' would put it one
careless join from the leak that design forbids. Expanding
recommendation_tuning_audit's CHECK is in the same migration per rule #36,
and a test asserts the audit row lands, which is what would catch its
absence.
snooze_days moves out of a Go constant onto the tuning card (rule #25),
closing the deferral from #2374.
Tag-overlap tests use deliberately SKEWED fixtures: an evenly-matching pool
cannot exercise a re-ranking, since every candidate gets the same
multiplier and the order is unchanged whether the blend works or not.
Admin UI + client attribution follow in this batch — rule #27.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migration 0050 adds candidate_artist_tags + candidate_artist_tag_state:
folksonomy tags for artists NOT in the library, which track_tags cannot
hold because it's FK'd to tracks(id) and a Discover candidate has no local
row. Slice 6 ranks against these; this slice only fills the cache.
The reuse the task claimed is real and verified: MusicBrainz's
fetchEntityTags(ctx, "artist", mbid, scale) already existed for the #1519
recording→artist fallback, so FetchArtistTags is a thin wrapper. Two
subtleties it does NOT inherit:
- Weight scale is 1.0, not artistTagWeightFactor (0.6). That discount
exists because FetchTrackTags uses artist tags as a *proxy* for a
track's; here the artist IS the subject. Applying it would make these
weights incomparable with track_tags — exactly the comparison slice 6
depends on. Pinned by a test.
- fetchEntityTags reports existing-but-untagged as (empty, nil) so the
track path can fall through. There's no next level here, so empty
becomes the terminal ErrNotFound; otherwise the enricher would settle
a candidate as "enriched" with zero tags.
ArtistTagProvider is the split TrackTagProvider's own doc comment
anticipated ("e.g. artist-level tags"). Last.fm gains artist.getTopTags,
which returns the same toptags envelope, so the response type and
normalizer are reused unchanged.
Rather than write the merge-and-classify loop twice, extracted it from
EnrichTrack into runChain(). The ErrNotFound-vs-transient split is the
load-bearing part — those lead to opposite persistence decisions — so it
now has direct unit tests it never had while inlined.
Bookkeeping is a separate table, not columns, because the "providers had
nothing" outcome must be recordable for a candidate with zero tag rows,
and there is no per-candidate row to hang columns off (
artist_similarity_unmatched holds many rows per candidate). Absence of a
state row means "never processed", so a transient failure writes nothing
and stays eligible.
Two capacity realities are designed for, not papered over:
- The pool is O(library artists x neighbours) and MusicBrainz allows
~1 req/s, so it can never drain in one pass. The eligibility query
returns candidates in descending summed-similarity order, so the ones
that can actually reach a deck are enriched first.
- candidateBatch (50) is smaller than the track batch (200): tracks are
finite and drain to completion, candidates are effectively unbounded
and would otherwise starve the track arm forever.
GC sweeps both tables — the similarity feed churns, and a candidate that
joins the library has its tags in track_tags now. Tags swept before state
so a mid-sweep crash leaves a valid state, not a re-fetch loop.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migration 0049 adds suggestion_snoozes(user_id, candidate_mbid,
candidate_name, snoozed_until), and SuggestArtistsForUser excludes rows
whose snooze hasn't expired.
This is NOT a dislike. Rule #101 forbids a "Not for me" / thumbs-down
UI; a snooze is the approved shape instead because it records no verdict
on the music, expires on its own (~90d), and never reaches the taste
profile. It's acquisition triage — "not right now" — so the filter sits
at the candidate stage rather than in the score, where it would become a
ranking signal by the back door.
Per-user throughout (rule #47): one household member parking a candidate
leaves everyone else's deck untouched.
candidate_name is denormalized because suggestions are out-of-library by
definition — there is no artists row to resolve a display name from, and
the un-snooze list has to show something. That list is why GET
/discover/snoozes exists at all: a parked candidate is by definition
absent from the deck, so without it the DELETE would be unreachable.
Also fixes a hole in the codegen check from #2380: `git diff` ignores
untracked paths, so a brand-new generated file would have passed it
silently. `git add -N` first. This commit is the first to add one.
Endpoints:
POST /api/discover/suggestions/{mbid}/snooze (body: name, days)
DELETE /api/discover/suggestions/{mbid}/snooze
GET /api/discover/snoozes
UI lands in slice 4 (#2375) before any of this merges — rule #27.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tag_provider_settings to the truncation list and reset
tag_sources_meta.current_version to 1 in ResetDB, mirroring the cover-art
settings reset. Without it the migration-seeded musicbrainz/lastfm rows
leaked across tests and desynced the enabled-set signature, so a key-only
PATCH spuriously reported version_bumped=true
(TestAdminUpdateTagSource_KeyOnlyDoesNotBump).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recommendation scoring knobs move out of YAML (radio profile) and
out of the systemMixWeights hard-code (daily_mix profile) into
DB-backed settings with live effect (#1250) — the defaults-discovery
lab per decision #1247: the operator turns knobs to find good values,
which then get baked back into shipped defaults; end users and other
operators should never need the card.
- Migration 0040: recommendation_weight_profiles (radio / daily_mix,
8 weight columns), taste_tuning singleton (engagement half-life +
completion-curve points), recommendation_tuning_audit (one row per
change with a {field, old, new} diff — the trend view's markers,
#1251).
- internal/recsettings: boot reconcile seeds shipped defaults without
clobbering tuned rows (coverart SettingsService pattern), validates
patches (bounds, curve ordering), writes audit rows, and pushes
daily_mix weights + taste config into package playlists. No-op
patches write no audit row.
- playlists gains SetSystemMixWeights / SetTasteConfig swap points
under a RWMutex — no signature threading through the producers; the
scheduler's taste rebuild reads the pushed config.
- Radio reads its weight profile from the service per request; the 8
weight fields leave config.RecommendationConfig (YAML keeps only
RecentlyPlayedHours / RadioSize / RadioSizeMax).
- Admin API: GET/PATCH/reset under /api/admin/recommendation-tuning,
echoing current + shipped values.
- Web: new admin Tuning tab — two weight profiles side by side, taste
card, per-scope save (changed fields only) + reset, deviation dots
against shipped defaults.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsF3cNoKrqCYsU78cXC8U6
A redo: the prior commit truncated cover_art_sources_meta, but
SettingsService.reconcile() only READS that singleton (seeded once by
migration 0018) and never recreates it → ~45 coverart/api tests hard-
failed "get current sources version: no rows". Correct reset: keep
cover_art_provider_settings in the truncate set (boot idempotently
re-UpsertProviderSettings), drop cover_art_sources_meta from it, and
instead `UPDATE cover_art_sources_meta SET current_version = 1` so the
row survives while cross-test version accumulation is cleared.
B residual (exposed once the play_events FK fix let these run):
- seedQuarantine used reason 'test-hide', invalid for the
lidarr_quarantine_reason enum (bad_rip/wrong_file/wrong_tags/
duplicate/other) → use 'other'.
- TestBuildSystemPlaylists_SufficientActivity predated #411/#352: the
variant switch errored on the 5 new seedless mixes and capped
track_count at 25. Accept deep_cuts/rediscover/new_for_you/
on_this_day/first_listens as seedless; raise the cap to 100.
Test/test-harness only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Latent failures exposed now that integration tests run in CI (#339).
All test/test-harness only — no production code changes.
A (dbtest.ResetDB): also truncate cover_art_provider_settings +
cover_art_sources_meta. The monotonic source-version counter (seeded 1
by 0018) accumulated across internal/coverart tests → CurrentVersion=4
want 1, key-only-bump assertions, enricher source skips. SettingsService
re-seeds both at boot, so a truncated start is the correct fresh state.
B (playlists system_test.seedPlayEvent): inserted play_events with a
random session_id → play_events_session_id_fkey violation (7 tests).
Create the parent play_sessions row in the same statement (CTE).
C (similarity worker_integration_test.newTestWorker): built the client
with only BaseURL. Post-4fca0e6 similarity hits the Labs API via
LabsBaseURL, so an unset LabsBaseURL fell through to the real labs.api
and the stub never ran → 0 similarity rows. Set LabsBaseURL to the stub.
F (library scanner_test): NOT a flake — deterministic. Synthetic
ID3-only MP3s have no decodable duration; the scanner intentionally
won't skip duration_ms=0 rows (retries duration backfill), so every
re-scan reported Updated not Skipped. Seed duration_ms>0 before the
second scan so the mtime-based incremental-skip path under test is
actually exercised. Production scanner behavior unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New package internal/sync. LogChange writes a library_changes row inside
the supplied tx. Encode helpers produce stable composite ids for like_*
and playlist_track entries. Subsequent commits wire LogChange into the
scanner / likes / playlists services.
Also: dbtest.dataTables now includes library_changes so test isolation
holds across runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Create / Get / List / Update / Delete with the visibility model from
the spec: private by default, owner-only mutations, public read for
non-owners. Update uses sqlc's CASE-WHEN-flag pattern for PATCH-style
partial updates without writing N variants.
Delete cleans up the cached cover file from disk best-effort. Track
operations (Append/Remove/Reorder) and the collage generator land
in subsequent tasks.
Also adds playlists + playlist_tracks to dbtest.ResetDB's truncate
list so integration tests in this package start from a clean state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- dbtest.ResetDB.dataTables now truncates lidarr_quarantine + lidarr_quarantine_actions
alongside the other M2-M4 data tables. Without this, M5b service tests would
inherit residual state across runs.
- DeleteTrackFile godoc spells out the post-file/pre-DB failure window
reconciles via the next library scan; the function is retry-safe by design,
not atomic.
Integration tests previously TRUNCATEd the users table at setup,
which wiped the operator's admin login every time the suite ran
against a Postgres instance shared with their dev environment.
This commit:
- Adds internal/dbtest/reset.go exposing ResetDB(t, pool) and
TestUserPrefix. ResetDB truncates every data table EXCEPT users,
then deletes only users whose username starts with TestUserPrefix.
- Migrates 9 test files (subsonic/scrobble, subsonic/star,
recommendation/candidates, scrobble/queue, similarity/worker,
playevents/writer, playsessions/service, api/auth, api/me) to
call ResetDB instead of issuing TRUNCATE-with-users.
- Renames hardcoded test usernames to use TestUserPrefix so ResetDB
cleans them between runs. seedUser in api/auth_test now prefixes
internally; existing call sites pass bare names.
- Leaves auth/bootstrap_test.go alone — it legitimately tests
first-time admin bootstrap and must wipe users.
Verified locally: full integration suite with -p 1 runs cleanly
under the existing MINSTREL_TEST_DATABASE_URL setup, and the admin
row in the shared dev DB survives the run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bvandeusenandClaude Opus 5