feat(library): detect missing files and stop offering them — #2523

Nothing in Minstrel ever noticed a deleted file. The walk only visits
paths that exist, so a row whose file was gone was never scanned, never
errored, never counted — permanently invisible. classifyEvent ignores
fsnotify removals by design, and the safety-net scan is the same walk, so
it covers additions only. Rows accumulated forever.

Found on the operator's library: a completed scan reported
skipped=24185 errored=0 while the MBID backfill (which opens files by DB
path rather than walking) logged ~40 "no such file or directory" across
three reorganised albums. Those rows also kept their pre-#2499 welded
genre, which is how this surfaced — the version-stamped tag re-read can
only reach files the walk visits.

The harm is not cosmetic. tracks is the candidate universe for
recommendation.sql / discover.sql / system_mixes.sql and nothing filtered
on file existence, so a mix could spend a slot on a track that cannot
stream.

Marks rather than deletes. A missing file is a claim about the filesystem
and the filesystem lies transiently — an unmounted volume, a network
blip, a container that started before its media mount attached. Every
sweep in internal/gc resolves a truth INSIDE the database and is safe to
run blind; this one is not, so no deletion happens here. Three guards
refuse to act on ambiguous evidence: every scan root must resolve to a
non-empty directory, the walk must have seen at least one file, and one
reconcile may newly mark at most 25% of the library. Clearing a mark is
never the dangerous direction, so it runs unconditionally — otherwise a
library that tripped the cap could never recover once the mount returned.

Only a full Scan reconciles. The walk's set of seen paths is the
evidence, and ScanFiles has no basis for concluding anything about files
it did not look at.

Excludes marked tracks from all 13 track-emitting queries (radio x2,
system mixes x5, discover x4, most-played x2), the 6 play-history seed
picks, and the genre browse axis. Deliberately NOT filtered: the shared
ListPlaylistTracks read path, because it also serves user-curated
playlists where hiding a track the user added would be wrong — system
playlists shed orphans on their next daily rebuild instead. History and
the taste profile also keep them: those record the past, and a track you
played 200 times still says something about your taste.

Reconcile tallies land in scan_runs so a disappearance is visible rather
than discovered when a mix comes up short.

internal/db/queries/browse.sql

@@ -1,3 +1,15 @@
+-- Every query in this file filters `tracks.missing_since IS NULL` (#2523).
+-- A row whose file has vanished keeps its genre forever — the scanner walks the
+-- filesystem, so it never revisits a path that no longer exists — which is how
+-- pre-#2499 welded genres survived a full re-scan and kept showing in the index.
+-- Browsing is a way of finding something to play, so a track that cannot play
+-- should not shape it.
+--
+-- Year queries below join albums only and are deliberately left alone: an album
+-- is still a real release even if some of its tracks are gone. An album whose
+-- EVERY track is missing will linger on the year axis; that's a narrower case,
+-- tracked with the rest of the cleanup work.
+
 -- name: ListGenresWithCount :many
 -- Genre browse index (#367).
 --
@@ -22,6 +34,7 @@ SELECT trim(g.genre) AS genre, COUNT(DISTINCT tracks.id)::bigint AS track_count
 FROM tracks
 JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
 WHERE trim(g.genre) <> ''
+  AND tracks.missing_since IS NULL
 GROUP BY trim(g.genre)
 -- Ordered by the expression, not the output alias: `ORDER BY genre` is
 -- ambiguous between the alias and tracks.genre, and sqlc rejects it.
@@ -41,6 +54,7 @@ WHERE EXISTS (
     FROM tracks
     JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
     WHERE tracks.album_id = albums.id
+      AND tracks.missing_since IS NULL
       AND trim(g.genre) = trim(sqlc.arg(genre)::text)
 )
 ORDER BY albums.sort_title, albums.id
@@ -56,6 +70,7 @@ WHERE EXISTS (
     FROM tracks
     JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
     WHERE tracks.album_id = albums.id
+      AND tracks.missing_since IS NULL
       AND trim(g.genre) = trim(sqlc.arg(genre)::text)
 );
 
@@ -96,6 +111,7 @@ SELECT DISTINCT trim(g.genre) AS genre
 FROM tracks
 JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
 WHERE tracks.album_id = $1 AND trim(g.genre) <> ''
+  AND tracks.missing_since IS NULL
 ORDER BY trim(g.genre);
 
 -- name: ListGenresForArtist :many
@@ -106,4 +122,5 @@ SELECT DISTINCT trim(g.genre) AS genre
 FROM tracks
 JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true
 WHERE tracks.artist_id = $1 AND trim(g.genre) <> ''
+  AND tracks.missing_since IS NULL
 ORDER BY trim(g.genre);

internal/db/queries/discover.sql

@@ -29,7 +29,8 @@ dormant_artists AS (
 SELECT t.id, t.album_id, t.artist_id
   FROM tracks t
   JOIN dormant_artists da ON da.id = t.artist_id
- WHERE NOT EXISTS (
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND NOT EXISTS (
        SELECT 1 FROM play_events pe
         WHERE pe.user_id = $1
           AND pe.track_id = t.id
@@ -60,7 +61,8 @@ SELECT t.id, t.album_id, t.artist_id
 SELECT t.id, t.album_id, t.artist_id
   FROM general_likes gl
   JOIN tracks t ON t.id = gl.track_id
- WHERE gl.user_id != $1
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND gl.user_id != $1
    AND NOT EXISTS (
        SELECT 1 FROM play_events pe
         WHERE pe.user_id = $1
@@ -86,7 +88,8 @@ SELECT t.id, t.album_id, t.artist_id
 -- $1 = user_id, $2 = date string for md5 ordering.
 SELECT t.id, t.album_id, t.artist_id
   FROM tracks t
- WHERE NOT EXISTS (
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND NOT EXISTS (
        SELECT 1 FROM play_events pe
         WHERE pe.user_id = $1
           AND pe.track_id = t.id
@@ -117,7 +120,8 @@ SELECT t.id, t.album_id, t.artist_id
   FROM tracks t
   JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true
   JOIN taste_profile_tags nt ON nt.user_id = $1 AND trim(g_split.g) = nt.tag
- WHERE nt.weight > 0
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND nt.weight > 0
    AND trim(g_split.g) <> ''
    AND NOT EXISTS (
        SELECT 1 FROM play_events pe

internal/db/queries/recommendation.sql

@@ -24,6 +24,7 @@ LEFT JOIN LATERAL (
     WHERE user_id = $1 AND track_id = t.id
 ) pe ON true
 WHERE t.id <> $2
+  AND t.missing_since IS NULL   -- #2523: never offer a file that is gone
   AND NOT EXISTS (
     SELECT 1 FROM play_events
     WHERE user_id = $1 AND track_id = t.id
@@ -177,7 +178,7 @@ FROM (
     UNION ALL SELECT track_id, sim_score FROM coplay_artists
     UNION ALL SELECT track_id, sim_score FROM random_fill
 ) u
-JOIN tracks t ON t.id = u.track_id
+JOIN tracks t ON t.id = u.track_id AND t.missing_since IS NULL   -- #2523: never offer a file that is gone
 JOIN albums al ON al.id = t.album_id
 LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id
 LEFT JOIN LATERAL (
@@ -382,7 +383,8 @@ FROM plays p
 JOIN tracks t   ON t.id = p.track_id
 JOIN albums     ON albums.id  = t.album_id
 JOIN artists    ON artists.id = t.artist_id
-WHERE NOT EXISTS (
+WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+  AND NOT EXISTS (
     SELECT 1 FROM lidarr_quarantine q
     WHERE q.user_id = $1 AND q.track_id = t.id
 )
@@ -408,6 +410,7 @@ JOIN tracks t   ON t.id = p.track_id
 JOIN albums     ON albums.id  = t.album_id
 JOIN artists    ON artists.id = t.artist_id
 WHERE t.artist_id = sqlc.arg(artist_id)
+  AND t.missing_since IS NULL   -- #2523: never offer a file that is gone
   AND NOT EXISTS (
       SELECT 1 FROM lidarr_quarantine q
       WHERE q.user_id = sqlc.arg(user_id) AND q.track_id = t.id

internal/db/queries/system_mixes.sql

@@ -40,7 +40,8 @@ SELECT t.id, t.album_id, t.artist_id
   JOIN affinity_artists aa ON aa.artist_id = t.artist_id
   LEFT JOIN play_counts pc ON pc.track_id = t.id
   LEFT JOIN skip_counts sc ON sc.track_id = t.id
- WHERE COALESCE(pc.c, 0) <= 2
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND COALESCE(pc.c, 0) <= 2
    AND COALESCE(sc.c, 0) < 2
    AND NOT EXISTS (
        SELECT 1 FROM lidarr_quarantine q
@@ -70,7 +71,8 @@ WITH stats AS (
 SELECT t.id, t.album_id, t.artist_id
   FROM tracks t
   JOIN stats s ON s.track_id = t.id
- WHERE s.c >= 3
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND s.c >= 3
    AND s.last_at <= now() - interval '30 days'
    AND NOT EXISTS (
        SELECT 1 FROM lidarr_quarantine q
@@ -149,7 +151,8 @@ albums_tiered AS (
 SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
   FROM tracks t
   JOIN albums_tiered alt ON alt.album_id = t.album_id
- WHERE NOT EXISTS (
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND NOT EXISTS (
        SELECT 1 FROM lidarr_quarantine q
         WHERE q.user_id = $1 AND q.track_id = t.id
    )
@@ -187,7 +190,8 @@ WITH windowed AS (
 SELECT t.id, t.album_id, t.artist_id
   FROM tracks t
   JOIN windowed w ON w.track_id = t.id
- WHERE NOT EXISTS (
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND NOT EXISTS (
        SELECT 1 FROM lidarr_quarantine q
         WHERE q.user_id = $1 AND q.track_id = t.id
    )
@@ -240,7 +244,8 @@ SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
   FROM tracks t
   JOIN albums al ON al.id = t.album_id
   JOIN albums_tiered alt ON alt.album_id = al.id
- WHERE alt.tier IS NOT NULL
+ WHERE t.missing_since IS NULL   -- #2523: never offer a file that is gone
+   AND alt.tier IS NOT NULL
    AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id)
    AND NOT EXISTS (
        SELECT 1 FROM lidarr_quarantine q

internal/db/queries/system_playlists.sql

@@ -1,3 +1,11 @@
+-- Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a
+-- seed or For-You candidate has to be something that can actually play. Note
+-- this only affects newly GENERATED playlists — already-stored system
+-- playlists keep their rows until the next daily rebuild, which is why the
+-- shared ListPlaylistTracks read path is deliberately left unfiltered (it
+-- also serves user-curated playlists, where hiding a track the user added
+-- themselves would be wrong).
+
 -- M7 #352 slice 2: system-generated playlist queries.
 
 -- name: ListActiveUsersForSystemPlaylists :many
@@ -72,7 +80,7 @@ recent7 AS (
            COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
            0 AS tier
       FROM play_events pe
-      JOIN tracks t ON t.id = pe.track_id
+      JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
      WHERE pe.user_id = $1
        AND pe.started_at > now() - INTERVAL '7 days'
        AND t.artist_id IS NOT NULL
@@ -83,7 +91,7 @@ recent30 AS (
            COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
            1 AS tier
       FROM play_events pe
-      JOIN tracks t ON t.id = pe.track_id
+      JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
      WHERE pe.user_id = $1
        AND pe.started_at > now() - INTERVAL '30 days'
        AND t.artist_id IS NOT NULL
@@ -94,7 +102,7 @@ alltime AS (
            COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
            2 AS tier
       FROM play_events pe
-      JOIN tracks t ON t.id = pe.track_id
+      JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
      WHERE pe.user_id = $1
        AND t.artist_id IS NOT NULL
      GROUP BY t.artist_id
@@ -139,7 +147,7 @@ SELECT c.artist_id,
 WITH recent AS (
     SELECT t.id, COUNT(*) AS c, 0 AS tier
       FROM play_events pe
-      JOIN tracks t ON t.id = pe.track_id
+      JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
      WHERE pe.user_id = $1
        AND pe.started_at > now() - INTERVAL '30 days'
        AND pe.was_skipped = false
@@ -148,7 +156,7 @@ WITH recent AS (
 alltime AS (
     SELECT t.id, COUNT(*) AS c, 1 AS tier
       FROM play_events pe
-      JOIN tracks t ON t.id = pe.track_id
+      JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
      WHERE pe.user_id = $1
        AND pe.was_skipped = false
      GROUP BY t.id
@@ -181,7 +189,7 @@ SELECT id
 SELECT COALESCE(
     (SELECT t.id
        FROM play_events pe
-       JOIN tracks t ON t.id = pe.track_id
+       JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
       WHERE pe.user_id = $1
         AND t.artist_id = $2
         AND pe.started_at > now() - INTERVAL '7 days'

internal/db/queries/tracks.sql

@@ -137,3 +137,34 @@ RETURNING id, album_id, artist_id, file_path, mbid;
 -- Batched lookup used by /api/library/sync to hydrate upsert payloads
 -- (#357). Mirror of GetArtistsByIDs.
 SELECT * FROM tracks WHERE id = ANY($1::uuid[]);
+
+-- name: ListTrackPathsForReconcile :many
+-- Every row's path + current missing mark, for the scanner's reconcile pass
+-- (#2523). Deliberately unfiltered and unpaged: reconcile has to compare the
+-- WHOLE table against what the walk saw, and a filtered subset would let rows
+-- outside it drift forever. Three narrow columns keep it cheap even on a
+-- library of a few hundred thousand tracks.
+SELECT id, file_path, missing_since FROM tracks;
+
+-- name: MarkTracksMissing :execrows
+-- Marks rows whose file the walk did not see. `missing_since IS NULL` in the
+-- predicate makes this idempotent: a row already marked keeps its ORIGINAL
+-- timestamp, so "how long has it been gone" survives repeated scans. Losing
+-- that would make any age-based cleanup policy meaningless.
+--
+-- updated_at is deliberately NOT touched. It tracks content changes and gates
+-- the scanner's mtime skip; moving it here would make a returning file look
+-- newer than its own mtime and stop its tags being re-read.
+UPDATE tracks
+   SET missing_since = now()
+ WHERE id = ANY(sqlc.arg(ids)::uuid[])
+   AND missing_since IS NULL;
+
+-- name: ClearTracksMissing :execrows
+-- Clears the mark on rows whose file is back. Runs independently of the mtime
+-- skip check, so a file that reappears unchanged is un-marked even though the
+-- scanner skips re-reading its tags.
+UPDATE tracks
+   SET missing_since = NULL
+ WHERE id = ANY(sqlc.arg(ids)::uuid[])
+   AND missing_since IS NOT NULL;