Files
minstrel/internal/db/queries/system_playlists.sql
T
bvandeusen f6d1cf24f0
test-go / test (push) Successful in 1m0s
test-go / integration (push) Successful in 5m10s
feat(library): detect missing files and stop offering them — #2523
Nothing in Minstrel ever noticed a deleted file. The walk only visits
paths that exist, so a row whose file was gone was never scanned, never
errored, never counted — permanently invisible. classifyEvent ignores
fsnotify removals by design, and the safety-net scan is the same walk, so
it covers additions only. Rows accumulated forever.

Found on the operator's library: a completed scan reported
skipped=24185 errored=0 while the MBID backfill (which opens files by DB
path rather than walking) logged ~40 "no such file or directory" across
three reorganised albums. Those rows also kept their pre-#2499 welded
genre, which is how this surfaced — the version-stamped tag re-read can
only reach files the walk visits.

The harm is not cosmetic. tracks is the candidate universe for
recommendation.sql / discover.sql / system_mixes.sql and nothing filtered
on file existence, so a mix could spend a slot on a track that cannot
stream.

Marks rather than deletes. A missing file is a claim about the filesystem
and the filesystem lies transiently — an unmounted volume, a network
blip, a container that started before its media mount attached. Every
sweep in internal/gc resolves a truth INSIDE the database and is safe to
run blind; this one is not, so no deletion happens here. Three guards
refuse to act on ambiguous evidence: every scan root must resolve to a
non-empty directory, the walk must have seen at least one file, and one
reconcile may newly mark at most 25% of the library. Clearing a mark is
never the dangerous direction, so it runs unconditionally — otherwise a
library that tripped the cap could never recover once the mount returned.

Only a full Scan reconciles. The walk's set of seen paths is the
evidence, and ScanFiles has no basis for concluding anything about files
it did not look at.

Excludes marked tracks from all 13 track-emitting queries (radio x2,
system mixes x5, discover x4, most-played x2), the 6 play-history seed
picks, and the genre browse axis. Deliberately NOT filtered: the shared
ListPlaylistTracks read path, because it also serves user-curated
playlists where hiding a track the user added would be wrong — system
playlists shed orphans on their next daily rebuild instead. History and
the taste profile also keep them: those record the past, and a track you
played 200 times still says something about your taste.

Reconcile tallies land in scan_runs so a disappearance is visible rather
than discovered when a mix comes up short.
2026-08-06 14:34:53 -04:00

272 lines
10 KiB
SQL

-- Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a
-- seed or For-You candidate has to be something that can actually play. Note
-- this only affects newly GENERATED playlists — already-stored system
-- playlists keep their rows until the next daily rebuild, which is why the
-- shared ListPlaylistTracks read path is deliberately left unfiltered (it
-- also serves user-curated playlists, where hiding a track the user added
-- themselves would be wrong).
-- M7 #352 slice 2: system-generated playlist queries.
-- name: ListActiveUsersForSystemPlaylists :many
-- Active = had a play in the last 7 days. The cron iterates this list.
SELECT u.id FROM users u
WHERE EXISTS (
SELECT 1 FROM play_events pe
WHERE pe.user_id = u.id
AND pe.started_at > now() - INTERVAL '7 days'
);
-- name: GetSystemPlaylistRun :one
SELECT user_id, last_run_at, last_run_date, in_flight, last_error
FROM system_playlist_runs
WHERE user_id = $1;
-- name: TryClaimSystemPlaylistRun :one
-- Atomic test-and-set on in_flight. Returns the row if we won the claim,
-- or pgx.ErrNoRows if someone else is already building.
INSERT INTO system_playlist_runs (user_id, in_flight)
VALUES ($1, true)
ON CONFLICT (user_id) DO UPDATE
SET in_flight = true
WHERE system_playlist_runs.in_flight = false
RETURNING user_id, last_run_at, last_run_date, in_flight, last_error;
-- name: FinishSystemPlaylistRun :exec
-- Mark a successful build complete.
UPDATE system_playlist_runs
SET last_run_at = $2,
last_run_date = $3,
in_flight = false,
last_error = NULL
WHERE user_id = $1;
-- name: FailSystemPlaylistRun :exec
-- Mark a failed build complete (in_flight=false, error captured).
UPDATE system_playlist_runs
SET in_flight = false,
last_error = $2
WHERE user_id = $1;
-- name: ClearStaleSystemPlaylistInFlight :exec
-- Cron startup recovery: clear any in_flight=true rows from a previously
-- crashed process. Safe because at startup, by definition, nothing is
-- still building.
UPDATE system_playlist_runs SET in_flight = false WHERE in_flight = true;
-- name: PickSeedArtists :many
-- Top-12 most-engaged distinct artist candidates, tiered so the
-- "Songs like X" mixes never silently vanish (#1255): the old hard
-- 7-day window emptied the seed pool after a quiet week, and the
-- daily atomic-replace build then deleted every existing mix until
-- the user played something again. Same fallback shape as
-- PickTopPlayedTracksForUser (For You's seeds):
-- tier 0 engagement in the last 7 days
-- tier 1 (only if tier 0 empty) last 30 days
-- tier 2 (only if tiers 0+1 empty) all-time
-- tier 3 (only if 0-2 empty) liked artists with no play history
-- All returned rows share one tier; produceSeedMixes maps it onto the
-- rule-#131 pick-kind ladder and stamps the built tracks, so metrics
-- can compare mixes seeded from fresh vs stale engagement.
-- Score = unskipped-play count + 5 if user has liked the artist. The
-- Go-side picker (pickSeedArtistsForDay) shuffles the 12
-- daily-deterministically and takes songsLikeSeedCount (6) so the mix
-- set both rotates day-to-day and fills the dedicated Home row (#1491).
WITH liked AS (
SELECT gla.artist_id FROM general_likes_artists gla WHERE gla.user_id = $1
),
recent7 AS (
SELECT t.artist_id,
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
0 AS tier
FROM play_events pe
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '7 days'
AND t.artist_id IS NOT NULL
GROUP BY t.artist_id
),
recent30 AS (
SELECT t.artist_id,
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
1 AS tier
FROM play_events pe
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '30 days'
AND t.artist_id IS NOT NULL
GROUP BY t.artist_id
),
alltime AS (
SELECT t.artist_id,
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
2 AS tier
FROM play_events pe
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1
AND t.artist_id IS NOT NULL
GROUP BY t.artist_id
),
likedonly AS (
SELECT l.artist_id, 0::bigint AS play_count, 3 AS tier
FROM liked l
),
chosen AS (
SELECT artist_id, play_count, tier FROM recent7
UNION ALL
SELECT artist_id, play_count, tier FROM recent30
WHERE NOT EXISTS (SELECT 1 FROM recent7)
UNION ALL
SELECT artist_id, play_count, tier FROM alltime
WHERE NOT EXISTS (SELECT 1 FROM recent7)
AND NOT EXISTS (SELECT 1 FROM recent30)
UNION ALL
SELECT artist_id, play_count, tier FROM likedonly
WHERE NOT EXISTS (SELECT 1 FROM recent7)
AND NOT EXISTS (SELECT 1 FROM recent30)
AND NOT EXISTS (SELECT 1 FROM alltime)
)
SELECT c.artist_id,
(c.play_count + CASE WHEN l.artist_id IS NOT NULL THEN 5 ELSE 0 END)::bigint AS score,
c.tier::int AS tier
FROM chosen c
LEFT JOIN liked l ON l.artist_id = c.artist_id
ORDER BY score DESC, c.artist_id
LIMIT 12;
-- name: PickTopPlayedTracksForUser :many
-- For-You candidate seeds, tiered so For-You never silently vanishes:
-- tier 0 top non-skip plays in the last 30 days
-- tier 1 (only if tier 0 empty) all-time top non-skip plays
-- tier 2 (only if tiers 0+1 empty) liked tracks
-- Returns up to 5 ids; tie-break by track_id for determinism. The
-- Go-side picker (pickDailySeeds) draws the day's seeds from these.
-- Widened from a hard 7-day window, which made For-You disappear
-- after a week of not listening and never recover on a self-hosted
-- library with sparse history.
WITH recent AS (
SELECT t.id, COUNT(*) AS c, 0 AS tier
FROM play_events pe
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1
AND pe.started_at > now() - INTERVAL '30 days'
AND pe.was_skipped = false
GROUP BY t.id
),
alltime AS (
SELECT t.id, COUNT(*) AS c, 1 AS tier
FROM play_events pe
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1
AND pe.was_skipped = false
GROUP BY t.id
),
liked AS (
SELECT gl.track_id AS id, 0::bigint AS c, 2 AS tier
FROM general_likes gl
WHERE gl.user_id = $1
),
chosen AS (
SELECT id, c, tier FROM recent
UNION ALL
SELECT id, c, tier FROM alltime
WHERE NOT EXISTS (SELECT 1 FROM recent)
UNION ALL
SELECT id, c, tier FROM liked
WHERE NOT EXISTS (SELECT 1 FROM recent)
AND NOT EXISTS (SELECT 1 FROM alltime)
)
SELECT id
FROM chosen
ORDER BY tier, c DESC, id
LIMIT 5;
-- name: PickTopPlayedTrackForArtistByUser :one
-- "Songs like X" seed selection. Returns the user's most-played non-skipped
-- track by artist X in the last 7 days. Falls back to the artist's most-
-- recent album's first track if no plays exist.
-- Cast to uuid so sqlc infers pgtype.UUID rather than interface{}.
SELECT COALESCE(
(SELECT t.id
FROM play_events pe
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
WHERE pe.user_id = $1
AND t.artist_id = $2
AND pe.started_at > now() - INTERVAL '7 days'
AND pe.was_skipped = false
GROUP BY t.id
ORDER BY COUNT(*) DESC, t.id
LIMIT 1),
(SELECT t.id
FROM tracks t
JOIN albums a ON a.id = t.album_id
WHERE t.artist_id = $2
ORDER BY a.release_date DESC NULLS LAST,
t.disc_number NULLS LAST,
t.track_number NULLS LAST,
t.id
LIMIT 1)
)::uuid AS id;
-- name: PickTopAlbumCoverForArtistByUser :one
-- "Songs like X" cover. Most-played album by user; tie-break by most-recent
-- release. NULL if the artist has no albums or all covers are NULL.
-- Note: albums uses cover_art_path (not cover_path).
SELECT a.cover_art_path
FROM albums a
LEFT JOIN tracks t ON t.album_id = a.id
LEFT JOIN play_events pe
ON pe.track_id = t.id
AND pe.user_id = $1
AND pe.was_skipped = false
WHERE a.artist_id = $2
AND a.cover_art_path IS NOT NULL
GROUP BY a.id, a.release_date, a.cover_art_path
ORDER BY COUNT(pe.id) DESC, a.release_date DESC NULLS LAST, a.id
LIMIT 1;
-- name: DeleteSystemPlaylistsForUser :exec
-- Atomic-replace step. CASCADE on playlist_tracks.playlist_id handles
-- the join rows.
DELETE FROM playlists WHERE user_id = $1 AND kind = 'system';
-- name: ListPlaylistsByUserAndKind :many
-- Used in integration tests; production filter applied in Go
-- (api.handleListPlaylists fetches via Service.List and filters by Kind in
-- memory so it can also surface other users' public playlists alongside
-- the caller's own kind-filtered ones).
-- Sorted by updated_at DESC.
SELECT id, user_id, name, description, is_public, kind, system_variant,
seed_artist_id, cover_path, track_count, duration_sec,
created_at, updated_at
FROM playlists
WHERE user_id = $1
AND ($2::text = 'all' OR kind = $2)
ORDER BY updated_at DESC;
-- name: CreateSystemPlaylist :one
-- Inserts a system-generated playlist. Used by BuildSystemPlaylists.
-- For 'for_you' variant, pass seed_artist_id as NULL (zero-value pgtype.UUID
-- with Valid=false).
INSERT INTO playlists (
user_id, name, description, is_public,
kind, system_variant, seed_artist_id, cover_path
) VALUES ($1, $2, '', false, 'system', $3, $4, $5)
RETURNING *;
-- name: GetSystemPlaylistByVariantForUser :one
-- Returns the user's system playlist for the given variant
-- ('for_you' / 'songs_like_artist' / 'discover'), or pgx.ErrNoRows
-- when the user hasn't built that variant yet. For
-- 'songs_like_artist' (which can have multiple rows per user, one
-- per seed), this returns whichever LIMIT 1 picks — callers that
-- need all instances should use a different query.
SELECT p.id, p.user_id, p.name, p.kind, p.system_variant,
p.seed_artist_id, p.cover_path, p.track_count
FROM playlists p
WHERE p.user_id = $1
AND p.kind = 'system'
AND p.system_variant = $2
ORDER BY p.created_at DESC
LIMIT 1;