feat(library): detect missing files and stop offering them — #2523
Nothing in Minstrel ever noticed a deleted file. The walk only visits paths that exist, so a row whose file was gone was never scanned, never errored, never counted — permanently invisible. classifyEvent ignores fsnotify removals by design, and the safety-net scan is the same walk, so it covers additions only. Rows accumulated forever. Found on the operator's library: a completed scan reported skipped=24185 errored=0 while the MBID backfill (which opens files by DB path rather than walking) logged ~40 "no such file or directory" across three reorganised albums. Those rows also kept their pre-#2499 welded genre, which is how this surfaced — the version-stamped tag re-read can only reach files the walk visits. The harm is not cosmetic. tracks is the candidate universe for recommendation.sql / discover.sql / system_mixes.sql and nothing filtered on file existence, so a mix could spend a slot on a track that cannot stream. Marks rather than deletes. A missing file is a claim about the filesystem and the filesystem lies transiently — an unmounted volume, a network blip, a container that started before its media mount attached. Every sweep in internal/gc resolves a truth INSIDE the database and is safe to run blind; this one is not, so no deletion happens here. Three guards refuse to act on ambiguous evidence: every scan root must resolve to a non-empty directory, the walk must have seen at least one file, and one reconcile may newly mark at most 25% of the library. Clearing a mark is never the dangerous direction, so it runs unconditionally — otherwise a library that tripped the cap could never recover once the mount returned. Only a full Scan reconciles. The walk's set of seen paths is the evidence, and ScanFiles has no basis for concluding anything about files it did not look at. Excludes marked tracks from all 13 track-emitting queries (radio x2, system mixes x5, discover x4, most-played x2), the 6 play-history seed picks, and the genre browse axis. Deliberately NOT filtered: the shared ListPlaylistTracks read path, because it also serves user-curated playlists where hiding a track the user added would be wrong — system playlists shed orphans on their next daily rebuild instead. History and the taste profile also keep them: those record the past, and a track you played 200 times still says something about your taste. Reconcile tallies land in scan_runs so a disappearance is visible rather than discovered when a mix comes up short.
This commit is contained in:
@@ -189,6 +189,7 @@ func (q *Queries) GetSystemPlaylistRun(ctx context.Context, userID pgtype.UUID)
|
||||
|
||||
const listActiveUsersForSystemPlaylists = `-- name: ListActiveUsersForSystemPlaylists :many
|
||||
|
||||
|
||||
SELECT u.id FROM users u
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM play_events pe
|
||||
@@ -197,6 +198,13 @@ SELECT u.id FROM users u
|
||||
)
|
||||
`
|
||||
|
||||
// Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a
|
||||
// seed or For-You candidate has to be something that can actually play. Note
|
||||
// this only affects newly GENERATED playlists — already-stored system
|
||||
// playlists keep their rows until the next daily rebuild, which is why the
|
||||
// shared ListPlaylistTracks read path is deliberately left unfiltered (it
|
||||
// also serves user-curated playlists, where hiding a track the user added
|
||||
// themselves would be wrong).
|
||||
// M7 #352 slice 2: system-generated playlist queries.
|
||||
// Active = had a play in the last 7 days. The cron iterates this list.
|
||||
func (q *Queries) ListActiveUsersForSystemPlaylists(ctx context.Context) ([]pgtype.UUID, error) {
|
||||
@@ -298,7 +306,7 @@ recent7 AS (
|
||||
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
|
||||
0 AS tier
|
||||
FROM play_events pe
|
||||
JOIN tracks t ON t.id = pe.track_id
|
||||
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
||||
WHERE pe.user_id = $1
|
||||
AND pe.started_at > now() - INTERVAL '7 days'
|
||||
AND t.artist_id IS NOT NULL
|
||||
@@ -309,7 +317,7 @@ recent30 AS (
|
||||
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
|
||||
1 AS tier
|
||||
FROM play_events pe
|
||||
JOIN tracks t ON t.id = pe.track_id
|
||||
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
||||
WHERE pe.user_id = $1
|
||||
AND pe.started_at > now() - INTERVAL '30 days'
|
||||
AND t.artist_id IS NOT NULL
|
||||
@@ -320,7 +328,7 @@ alltime AS (
|
||||
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
|
||||
2 AS tier
|
||||
FROM play_events pe
|
||||
JOIN tracks t ON t.id = pe.track_id
|
||||
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
||||
WHERE pe.user_id = $1
|
||||
AND t.artist_id IS NOT NULL
|
||||
GROUP BY t.artist_id
|
||||
@@ -432,7 +440,7 @@ const pickTopPlayedTrackForArtistByUser = `-- name: PickTopPlayedTrackForArtistB
|
||||
SELECT COALESCE(
|
||||
(SELECT t.id
|
||||
FROM play_events pe
|
||||
JOIN tracks t ON t.id = pe.track_id
|
||||
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
||||
WHERE pe.user_id = $1
|
||||
AND t.artist_id = $2
|
||||
AND pe.started_at > now() - INTERVAL '7 days'
|
||||
@@ -472,7 +480,7 @@ const pickTopPlayedTracksForUser = `-- name: PickTopPlayedTracksForUser :many
|
||||
WITH recent AS (
|
||||
SELECT t.id, COUNT(*) AS c, 0 AS tier
|
||||
FROM play_events pe
|
||||
JOIN tracks t ON t.id = pe.track_id
|
||||
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
||||
WHERE pe.user_id = $1
|
||||
AND pe.started_at > now() - INTERVAL '30 days'
|
||||
AND pe.was_skipped = false
|
||||
@@ -481,7 +489,7 @@ WITH recent AS (
|
||||
alltime AS (
|
||||
SELECT t.id, COUNT(*) AS c, 1 AS tier
|
||||
FROM play_events pe
|
||||
JOIN tracks t ON t.id = pe.track_id
|
||||
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
||||
WHERE pe.user_id = $1
|
||||
AND pe.was_skipped = false
|
||||
GROUP BY t.id
|
||||
|
||||
Reference in New Issue
Block a user