Nothing in Minstrel ever noticed a deleted file. The walk only visits paths that exist, so a row whose file was gone was never scanned, never errored, never counted — permanently invisible. classifyEvent ignores fsnotify removals by design, and the safety-net scan is the same walk, so it covers additions only. Rows accumulated forever. Found on the operator's library: a completed scan reported skipped=24185 errored=0 while the MBID backfill (which opens files by DB path rather than walking) logged ~40 "no such file or directory" across three reorganised albums. Those rows also kept their pre-#2499 welded genre, which is how this surfaced — the version-stamped tag re-read can only reach files the walk visits. The harm is not cosmetic. tracks is the candidate universe for recommendation.sql / discover.sql / system_mixes.sql and nothing filtered on file existence, so a mix could spend a slot on a track that cannot stream. Marks rather than deletes. A missing file is a claim about the filesystem and the filesystem lies transiently — an unmounted volume, a network blip, a container that started before its media mount attached. Every sweep in internal/gc resolves a truth INSIDE the database and is safe to run blind; this one is not, so no deletion happens here. Three guards refuse to act on ambiguous evidence: every scan root must resolve to a non-empty directory, the walk must have seen at least one file, and one reconcile may newly mark at most 25% of the library. Clearing a mark is never the dangerous direction, so it runs unconditionally — otherwise a library that tripped the cap could never recover once the mount returned. Only a full Scan reconciles. The walk's set of seen paths is the evidence, and ScanFiles has no basis for concluding anything about files it did not look at. Excludes marked tracks from all 13 track-emitting queries (radio x2, system mixes x5, discover x4, most-played x2), the 6 play-history seed picks, and the genre browse axis. Deliberately NOT filtered: the shared ListPlaylistTracks read path, because it also serves user-curated playlists where hiding a track the user added would be wrong — system playlists shed orphans on their next daily rebuild instead. History and the taste profile also keep them: those record the past, and a track you played 200 times still says something about your taste. Reconcile tallies land in scan_runs so a disappearance is visible rather than discovered when a mix comes up short.
266 lines
11 KiB
SQL
266 lines
11 KiB
SQL
-- Candidate queries for the discovery system-playlist mixes
|
|
-- (#419-423). Each returns (id, album_id, artist_id) rows; the Go
|
|
-- producer caps + truncates and converts to rankedCandidate. Daily-
|
|
-- deterministic md5(id||dateStr) ordering (where used) keeps a mix
|
|
-- stable within a day while rotating across days; shuffle-on-play
|
|
-- gives per-play variety on top.
|
|
|
|
-- name: ListDeepCutsTracks :many
|
|
-- #419 Deep Cuts: low-play tracks (<=2 plays) from artists the user
|
|
-- has liked OR played heavily (>=5 non-skip plays across the artist).
|
|
-- Tracks the user has skipped twice or more don't qualify (#1257):
|
|
-- eligibility used to count only unskipped plays, so the most actively
|
|
-- rejected tracks read as "barely heard" and kept being re-offered.
|
|
-- Threshold 2 so a single accidental skip doesn't banish a track;
|
|
-- passive-signal only (no dislike UI, rule #101).
|
|
-- $1 user_id, $2 date string.
|
|
WITH affinity_artists AS (
|
|
SELECT artist_id FROM general_likes_artists WHERE user_id = $1
|
|
UNION
|
|
SELECT t.artist_id
|
|
FROM play_events pe JOIN tracks t ON t.id = pe.track_id
|
|
WHERE pe.user_id = $1 AND pe.was_skipped = false
|
|
GROUP BY t.artist_id
|
|
HAVING COUNT(*) >= 5
|
|
),
|
|
play_counts AS (
|
|
SELECT track_id, COUNT(*) AS c
|
|
FROM play_events
|
|
WHERE user_id = $1 AND was_skipped = false
|
|
GROUP BY track_id
|
|
),
|
|
skip_counts AS (
|
|
SELECT track_id, COUNT(*) AS c
|
|
FROM play_events
|
|
WHERE user_id = $1 AND was_skipped = true
|
|
GROUP BY track_id
|
|
)
|
|
SELECT t.id, t.album_id, t.artist_id
|
|
FROM tracks t
|
|
JOIN affinity_artists aa ON aa.artist_id = t.artist_id
|
|
LEFT JOIN play_counts pc ON pc.track_id = t.id
|
|
LEFT JOIN skip_counts sc ON sc.track_id = t.id
|
|
WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
|
|
AND COALESCE(pc.c, 0) <= 2
|
|
AND COALESCE(sc.c, 0) < 2
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM lidarr_quarantine q
|
|
WHERE q.user_id = $1 AND q.track_id = t.id
|
|
)
|
|
ORDER BY md5(t.id::text || $2::text)
|
|
LIMIT 200;
|
|
|
|
-- name: ListRediscoverTracks :many
|
|
-- #420 Rediscover: tracks the user played a lot but has drifted away
|
|
-- from. One blended pool (issue #1246: the old two-tier UNION was
|
|
-- all-or-nothing — a single ">=6 months cold" row suppressed the whole
|
|
-- ">=30 days" tier, which is how a one-song playlist shipped; the 6mo
|
|
-- tier was also a strict subset of the 30d tier). Eligibility is >=3
|
|
-- non-skip plays and >=30 days cold; the bar sits at 3 rather than 5
|
|
-- so a weeks-old library still fields a pool — on young histories the
|
|
-- >=5-play tracks are precisely the ones still in rotation. Ordering
|
|
-- prefers true rediscoveries (>=6 months cold), then strong affection
|
|
-- (>=5 plays), then raw play count; the producer's minimum floor
|
|
-- decides whether the pool is big enough to ship at all. $1 user_id.
|
|
WITH stats AS (
|
|
SELECT pe.track_id, COUNT(*) AS c, MAX(pe.started_at) AS last_at
|
|
FROM play_events pe
|
|
WHERE pe.user_id = $1 AND pe.was_skipped = false
|
|
GROUP BY pe.track_id
|
|
)
|
|
SELECT t.id, t.album_id, t.artist_id
|
|
FROM tracks t
|
|
JOIN stats s ON s.track_id = t.id
|
|
WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
|
|
AND s.c >= 3
|
|
AND s.last_at <= now() - interval '30 days'
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM lidarr_quarantine q
|
|
WHERE q.user_id = $1 AND q.track_id = t.id
|
|
)
|
|
-- Declined-resurfacing cooldown (#1258): a skip on a
|
|
-- rediscover-sourced play is the user explicitly saying "I've moved
|
|
-- on" to the exact invitation this mix extends — such tracks sit
|
|
-- out ~90 days instead of re-qualifying the next day. Passive
|
|
-- signal only (rule #101).
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM play_events rp
|
|
WHERE rp.user_id = $1
|
|
AND rp.track_id = t.id
|
|
AND rp.source = 'rediscover'
|
|
AND rp.was_skipped = true
|
|
AND rp.started_at > now() - interval '90 days'
|
|
)
|
|
ORDER BY (s.last_at <= now() - interval '6 months') DESC,
|
|
(s.c >= 5) DESC,
|
|
s.c DESC, t.id
|
|
LIMIT 200;
|
|
|
|
-- name: ListNewForYouTracks :many
|
|
-- #421 New for you, tiered per project rule #131 (#1267). The old
|
|
-- single hard rule (added <30d AND direct affinity) had two failure
|
|
-- modes with one root: no notion of consumption — the album you
|
|
-- devoured in week one crowded the mix for three more weeks, and
|
|
-- after a quiet month nothing qualified at all.
|
|
--
|
|
-- "Consumed" = any track attempted for >=30s (the First Listens
|
|
-- threshold, #1268): once you've meaningfully engaged, the album is
|
|
-- no longer news. Applies to every tier — played albums leave the
|
|
-- mix the next build.
|
|
--
|
|
-- tier 1 the exact desire: unconsumed albums added <30d by
|
|
-- direct-affinity artists (liked, or >=3 attempted plays)
|
|
-- tier 2 step back a little: unconsumed affinity albums from the
|
|
-- wider 30-90d window — added while you weren't looking
|
|
-- tier 3 step back more: any unconsumed album added <90d
|
|
-- regardless of affinity, newest first
|
|
--
|
|
-- Album-coherent within tiers (newest album first, then disc/track);
|
|
-- the producer fills top-down and rotates within tier blocks. The
|
|
-- tier lands on playlist_tracks.pick_kind so metrics can price what
|
|
-- each step-back trades away. $1 user_id.
|
|
WITH attempted AS (
|
|
SELECT DISTINCT pe.track_id
|
|
FROM play_events pe
|
|
WHERE pe.user_id = $1 AND pe.duration_played_ms >= 30000
|
|
),
|
|
consumed_albums AS (
|
|
SELECT DISTINCT t.album_id
|
|
FROM attempted a JOIN tracks t ON t.id = a.track_id
|
|
),
|
|
affinity_artists AS (
|
|
SELECT gla.artist_id FROM general_likes_artists gla WHERE gla.user_id = $1
|
|
UNION
|
|
SELECT t.artist_id
|
|
FROM attempted a JOIN tracks t ON t.id = a.track_id
|
|
GROUP BY t.artist_id
|
|
HAVING COUNT(*) >= 3
|
|
),
|
|
albums_tiered AS (
|
|
SELECT al.id AS album_id, al.created_at,
|
|
CASE
|
|
WHEN al.created_at >= now() - interval '30 days'
|
|
AND al.artist_id IN (SELECT artist_id FROM affinity_artists) THEN 1
|
|
WHEN al.artist_id IN (SELECT artist_id FROM affinity_artists) THEN 2
|
|
ELSE 3
|
|
END AS tier
|
|
FROM albums al
|
|
WHERE al.created_at >= now() - interval '90 days'
|
|
AND al.id NOT IN (SELECT album_id FROM consumed_albums)
|
|
)
|
|
SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
|
|
FROM tracks t
|
|
JOIN albums_tiered alt ON alt.album_id = t.album_id
|
|
WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM lidarr_quarantine q
|
|
WHERE q.user_id = $1 AND q.track_id = t.id
|
|
)
|
|
ORDER BY alt.tier, alt.created_at DESC, t.album_id,
|
|
t.disc_number NULLS FIRST, t.track_number NULLS FIRST
|
|
LIMIT 300;
|
|
|
|
-- name: ListOnThisDayTracks :many
|
|
-- #422 On This Day: tracks the user played around this calendar date
|
|
-- (±10 day-of-year) in the past, excluding the very recent (>30 days
|
|
-- ago) so it's nostalgic, not just "last week". Weighted by how much
|
|
-- they were played in those windows. Floor relaxed from 60→30 days
|
|
-- and window ±7→±10 so it surfaces on a months-old library instead
|
|
-- of needing a full year of history; still skips cleanly (no rows →
|
|
-- no playlist) when there's no qualifying history yet.
|
|
-- Day-of-year distance is circular (#1256): plain ABS made Dec 28 vs
|
|
-- Jan 3 read as 359 days apart, silently gutting the window for ~3
|
|
-- weeks around every New Year — precisely when holiday nostalgia is
|
|
-- the point. LEAST(d, 365-d) wraps the boundary; leap-year drift of
|
|
-- ±1 is absorbed by the ±10 window. The build date ($2, already here
|
|
-- for the md5 rotation) anchors "today" instead of now() so the
|
|
-- window is testable and consistent with the mix's daily determinism.
|
|
-- $1 user_id, $2 date string.
|
|
WITH windowed AS (
|
|
SELECT track_id, COUNT(*) AS c
|
|
FROM play_events
|
|
WHERE user_id = $1 AND was_skipped = false
|
|
AND started_at < now() - interval '30 days'
|
|
AND LEAST(
|
|
ABS(EXTRACT(DOY FROM started_at) - EXTRACT(DOY FROM $2::date)),
|
|
365 - ABS(EXTRACT(DOY FROM started_at) - EXTRACT(DOY FROM $2::date))
|
|
) <= 10
|
|
GROUP BY track_id
|
|
)
|
|
SELECT t.id, t.album_id, t.artist_id
|
|
FROM tracks t
|
|
JOIN windowed w ON w.track_id = t.id
|
|
WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM lidarr_quarantine q
|
|
WHERE q.user_id = $1 AND q.track_id = t.id
|
|
)
|
|
ORDER BY w.c DESC, md5(t.id::text || $2::text)
|
|
LIMIT 200;
|
|
|
|
-- name: ListFirstListensTracks :many
|
|
-- #423 First Listens: songs the user has never heard or even
|
|
-- attempted. "Attempted" is track-level with a >=30s listen threshold
|
|
-- (#1268, operator decision 2026-07-03) — the old version disqualified
|
|
-- a whole album on ANY play_event, so a 2-second accidental brush
|
|
-- banished it permanently.
|
|
--
|
|
-- tier 1 the exact desire: albums with ZERO attempted tracks
|
|
-- tier 2 step back: barely-attempted albums (<=25% of tracks
|
|
-- reached 30s) — brushed, never explored. The already-
|
|
-- attempted tracks themselves are excluded; they're not
|
|
-- first listens.
|
|
--
|
|
-- Within each tier, affinity-ordered as before: liked artist, then
|
|
-- attempted-played artist (>=30s — skip-only contact isn't trust),
|
|
-- then the rest; album-coherent. The tier lands on
|
|
-- playlist_tracks.pick_kind via the producer. $1 user_id.
|
|
WITH attempted AS (
|
|
SELECT DISTINCT pe.track_id
|
|
FROM play_events pe
|
|
WHERE pe.user_id = $1 AND pe.duration_played_ms >= 30000
|
|
),
|
|
album_attempts AS (
|
|
SELECT t.album_id,
|
|
COUNT(a.track_id) AS attempted_count,
|
|
COUNT(*) AS track_count
|
|
FROM tracks t
|
|
LEFT JOIN attempted a ON a.track_id = t.id
|
|
GROUP BY t.album_id
|
|
),
|
|
attempted_artists AS (
|
|
SELECT DISTINCT t.artist_id
|
|
FROM attempted a JOIN tracks t ON t.id = a.track_id
|
|
),
|
|
albums_tiered AS (
|
|
SELECT aa.album_id,
|
|
CASE
|
|
WHEN aa.attempted_count = 0 THEN 1
|
|
WHEN aa.attempted_count::float / GREATEST(aa.track_count, 1) <= 0.25 THEN 2
|
|
END AS tier
|
|
FROM album_attempts aa
|
|
)
|
|
SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier
|
|
FROM tracks t
|
|
JOIN albums al ON al.id = t.album_id
|
|
JOIN albums_tiered alt ON alt.album_id = al.id
|
|
WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone
|
|
AND alt.tier IS NOT NULL
|
|
AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id)
|
|
AND NOT EXISTS (
|
|
SELECT 1 FROM lidarr_quarantine q
|
|
WHERE q.user_id = $1 AND q.track_id = t.id
|
|
)
|
|
ORDER BY
|
|
alt.tier,
|
|
(CASE
|
|
WHEN EXISTS (
|
|
SELECT 1 FROM general_likes_artists gla
|
|
WHERE gla.user_id = $1 AND gla.artist_id = al.artist_id
|
|
) THEN 0
|
|
WHEN al.artist_id IN (SELECT artist_id FROM attempted_artists) THEN 1
|
|
ELSE 2
|
|
END),
|
|
al.id, t.disc_number NULLS FIRST, t.track_number NULLS FIRST
|
|
LIMIT 300;
|