Nothing in Minstrel ever noticed a deleted file. The walk only visits
paths that exist, so a row whose file was gone was never scanned, never
errored, never counted — permanently invisible. classifyEvent ignores
fsnotify removals by design, and the safety-net scan is the same walk, so
it covers additions only. Rows accumulated forever.
Found on the operator's library: a completed scan reported
skipped=24185 errored=0 while the MBID backfill (which opens files by DB
path rather than walking) logged ~40 "no such file or directory" across
three reorganised albums. Those rows also kept their pre-#2499 welded
genre, which is how this surfaced — the version-stamped tag re-read can
only reach files the walk visits.
The harm is not cosmetic. tracks is the candidate universe for
recommendation.sql / discover.sql / system_mixes.sql and nothing filtered
on file existence, so a mix could spend a slot on a track that cannot
stream.
Marks rather than deletes. A missing file is a claim about the filesystem
and the filesystem lies transiently — an unmounted volume, a network
blip, a container that started before its media mount attached. Every
sweep in internal/gc resolves a truth INSIDE the database and is safe to
run blind; this one is not, so no deletion happens here. Three guards
refuse to act on ambiguous evidence: every scan root must resolve to a
non-empty directory, the walk must have seen at least one file, and one
reconcile may newly mark at most 25% of the library. Clearing a mark is
never the dangerous direction, so it runs unconditionally — otherwise a
library that tripped the cap could never recover once the mount returned.
Only a full Scan reconciles. The walk's set of seen paths is the
evidence, and ScanFiles has no basis for concluding anything about files
it did not look at.
Excludes marked tracks from all 13 track-emitting queries (radio x2,
system mixes x5, discover x4, most-played x2), the 6 play-history seed
picks, and the genre browse axis. Deliberately NOT filtered: the shared
ListPlaylistTracks read path, because it also serves user-curated
playlists where hiding a track the user added would be wrong — system
playlists shed orphans on their next daily rebuild instead. History and
the taste profile also keep them: those record the past, and a track you
played 200 times still says something about your taste.
Reconcile tallies land in scan_runs so a disappearance is visible rather
than discovered when a mix comes up short.
Both mixes move from a single hard eligibility rule to the tiered
ladder, with their tier stamped onto playlist_tracks.pick_kind via the
#1270 provenance pipeline.
New for you (#1267) — consume on play, degrade by stepping back:
- "Consumed" = any track attempted >=30s; played albums leave the mix
at the next build instead of crowding it until the calendar window
expires.
- Tier 1: unconsumed albums added <30d by direct-affinity artists.
Tier 2: unconsumed affinity albums from the wider 30-90d window —
added while you weren't looking. Tier 3: any unconsumed album added
<90d, newest first.
First listens (#1268) — track-level "attempted" threshold:
- A 2-second accidental brush no longer disqualifies a whole album;
"attempted" is duration_played_ms >= 30000 per track.
- Tier 1: albums with zero attempted tracks. Tier 2: barely-attempted
albums (<=25% of tracks reached 30s), minus the attempted tracks
themselves. The artist-affinity ordering signal also moves to the
>=30s definition so skip-only contact doesn't read as trust.
Producer plumbing: fetch adapters map the tier column onto pick kinds,
finishMix propagates PickKind into the persisted candidates, and
rotateForDay now rotates within contiguous same-pick-kind blocks so
daily rotation can't hoist tier-3 filler above tier-1's exact fits
(untiered pools are one block — original behavior).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsF3cNoKrqCYsU78cXC8U6
Three discovery-mix defects from the intent audit (Scribe note #1254),
all sharing the same root pattern — skips treated as non-events:
- Deep Cuts (#1257): eligibility counted only unskipped plays, so a
track skipped twice with zero completed listens read as "barely
heard" and kept being re-offered. Tracks with >=2 skips no longer
qualify; a single accidental skip doesn't banish.
- Rediscover (#1258): a skip on a rediscover-sourced play — the user
explicitly declining the resurfacing invitation — changed nothing,
so declined tracks re-qualified the next day forever. Such tracks
now sit out 90 days.
- On This Day (#1256): day-of-year distance used plain ABS, so
Dec 28 vs Jan 3 read as 359 days apart and the window silently
gutted itself for ~3 weeks around every New Year. Now circular
(LEAST(d, 365-d)), anchored on the build-date parameter instead of
now() so it's testable and consistent with the mix's daily
determinism.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsF3cNoKrqCYsU78cXC8U6
Confirmed against prod: exactly one track (17 plays, cold since May 21)
met the c>=5 + 30d-cold bar, and three process defects turned that into
a 1-track playlist instead of the locked placeholder.
- ListRediscoverTracks: collapse the two-tier UNION into one blended
pool. The old shallow-tier gate (WHERE NOT EXISTS deep) was
all-or-nothing — one 6-month row suppressed the entire 30-day tier —
and deep was a strict subset of shallow anyway. Eligibility drops to
>=3 non-skip plays (on a weeks-old history the >=5-play tracks are
precisely the ones still in rotation); ordering prefers >=6mo cold,
then >=5 plays, then raw count.
- Minimum viable mix floor for all five discovery mixes: below
minLen (15; 5 for the album-coherent NewForYou/FirstListens) the
variant is withheld so Home renders the 'listen more to unlock'
placeholder instead of a mix that reads as built-wrong.
- /api/events: clamp client-supplied 'at' to [user.created_at,
now+5m]. Unbounded client clocks could write arbitrarily old plays
and poison the 6-month ordering (prod data verified clean — no
scrub needed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TsF3cNoKrqCYsU78cXC8U6
For-You silently vanished after ~7 days of not listening (seed query
required a non-skip play in the last 7 days) and capped at ~40 on
self-hosted libraries with no ListenBrainz similarity data.
- PickTopPlayedTracksForUser: tiered seed — last 30d top plays, else
all-time top plays, else liked tracks. For-You only disappears now
if the account has zero plays AND zero likes.
- produceForYou uses deeper candidate source limits (raised random/
tag/similar K) so it reaches ~100 even with empty lb_similar /
similar_artists; richer when LB enrichment is present.
- Rediscover: tiered — 6-month-dormant, else ≥5-play 30-day-dormant.
- On This Day: floor 60→30 days, window ±7→±10 doy; still skips
cleanly (no rows → no playlist) on insufficient history.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each is one candidate query + one registry entry; zero client work
(R2 made tiles/refresh/shuffle generic, all are singleton kinds so
web's server `refreshable` flag and Flutter's derived getter both
light up automatically).
- deep_cuts (#419): <=2-play tracks from liked / heavily-played
artists; diversity-capped.
- rediscover (#420): >=5-play tracks not heard in 6 months, by
historical affection.
- new_for_you (#421): tracks from albums added <=30d whose artist
the user likes/plays; album-coherent (no cap).
- on_this_day (#422): tracks played within ±7 day-of-year in prior
windows (>60d ago), weighted by play count.
- first_listens (#423): never-played albums, tiered liked-artist →
played-artist → rest; album-coherent.
system_mixes.go producers mirror the Discover model (SQL gives the
ranking; finishMix caps+truncates to 100 to match For-You/Discover
shuffle depth; album-coherent mixes skip the cap). Builder query
failure is non-fatal (logged, yields no playlist) like Discover.
Existing system_test existence checks are unaffected.
Closes the #411 system-playlists-v2 umbrella's new-types thread.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusen
