Nothing in Minstrel ever noticed a deleted file. The walk only visits paths that exist, so a row whose file was gone was never scanned, never errored, never counted — permanently invisible. classifyEvent ignores fsnotify removals by design, and the safety-net scan is the same walk, so it covers additions only. Rows accumulated forever. Found on the operator's library: a completed scan reported skipped=24185 errored=0 while the MBID backfill (which opens files by DB path rather than walking) logged ~40 "no such file or directory" across three reorganised albums. Those rows also kept their pre-#2499 welded genre, which is how this surfaced — the version-stamped tag re-read can only reach files the walk visits. The harm is not cosmetic. tracks is the candidate universe for recommendation.sql / discover.sql / system_mixes.sql and nothing filtered on file existence, so a mix could spend a slot on a track that cannot stream. Marks rather than deletes. A missing file is a claim about the filesystem and the filesystem lies transiently — an unmounted volume, a network blip, a container that started before its media mount attached. Every sweep in internal/gc resolves a truth INSIDE the database and is safe to run blind; this one is not, so no deletion happens here. Three guards refuse to act on ambiguous evidence: every scan root must resolve to a non-empty directory, the walk must have seen at least one file, and one reconcile may newly mark at most 25% of the library. Clearing a mark is never the dangerous direction, so it runs unconditionally — otherwise a library that tripped the cap could never recover once the mount returned. Only a full Scan reconciles. The walk's set of seen paths is the evidence, and ScanFiles has no basis for concluding anything about files it did not look at. Excludes marked tracks from all 13 track-emitting queries (radio x2, system mixes x5, discover x4, most-played x2), the 6 play-history seed picks, and the genre browse axis. Deliberately NOT filtered: the shared ListPlaylistTracks read path, because it also serves user-curated playlists where hiding a track the user added would be wrong — system playlists shed orphans on their next daily rebuild instead. History and the taste profile also keep them: those record the past, and a track you played 200 times still says something about your taste. Reconcile tallies land in scan_runs so a disappearance is visible rather than discovered when a mix comes up short.
272 lines
10 KiB
SQL
272 lines
10 KiB
SQL
-- Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a
|
|
-- seed or For-You candidate has to be something that can actually play. Note
|
|
-- this only affects newly GENERATED playlists — already-stored system
|
|
-- playlists keep their rows until the next daily rebuild, which is why the
|
|
-- shared ListPlaylistTracks read path is deliberately left unfiltered (it
|
|
-- also serves user-curated playlists, where hiding a track the user added
|
|
-- themselves would be wrong).
|
|
|
|
-- M7 #352 slice 2: system-generated playlist queries.
|
|
|
|
-- name: ListActiveUsersForSystemPlaylists :many
|
|
-- Active = had a play in the last 7 days. The cron iterates this list.
|
|
SELECT u.id FROM users u
|
|
WHERE EXISTS (
|
|
SELECT 1 FROM play_events pe
|
|
WHERE pe.user_id = u.id
|
|
AND pe.started_at > now() - INTERVAL '7 days'
|
|
);
|
|
|
|
-- name: GetSystemPlaylistRun :one
|
|
SELECT user_id, last_run_at, last_run_date, in_flight, last_error
|
|
FROM system_playlist_runs
|
|
WHERE user_id = $1;
|
|
|
|
-- name: TryClaimSystemPlaylistRun :one
|
|
-- Atomic test-and-set on in_flight. Returns the row if we won the claim,
|
|
-- or pgx.ErrNoRows if someone else is already building.
|
|
INSERT INTO system_playlist_runs (user_id, in_flight)
|
|
VALUES ($1, true)
|
|
ON CONFLICT (user_id) DO UPDATE
|
|
SET in_flight = true
|
|
WHERE system_playlist_runs.in_flight = false
|
|
RETURNING user_id, last_run_at, last_run_date, in_flight, last_error;
|
|
|
|
-- name: FinishSystemPlaylistRun :exec
|
|
-- Mark a successful build complete.
|
|
UPDATE system_playlist_runs
|
|
SET last_run_at = $2,
|
|
last_run_date = $3,
|
|
in_flight = false,
|
|
last_error = NULL
|
|
WHERE user_id = $1;
|
|
|
|
-- name: FailSystemPlaylistRun :exec
|
|
-- Mark a failed build complete (in_flight=false, error captured).
|
|
UPDATE system_playlist_runs
|
|
SET in_flight = false,
|
|
last_error = $2
|
|
WHERE user_id = $1;
|
|
|
|
-- name: ClearStaleSystemPlaylistInFlight :exec
|
|
-- Cron startup recovery: clear any in_flight=true rows from a previously
|
|
-- crashed process. Safe because at startup, by definition, nothing is
|
|
-- still building.
|
|
UPDATE system_playlist_runs SET in_flight = false WHERE in_flight = true;
|
|
|
|
-- name: PickSeedArtists :many
|
|
-- Top-12 most-engaged distinct artist candidates, tiered so the
|
|
-- "Songs like X" mixes never silently vanish (#1255): the old hard
|
|
-- 7-day window emptied the seed pool after a quiet week, and the
|
|
-- daily atomic-replace build then deleted every existing mix until
|
|
-- the user played something again. Same fallback shape as
|
|
-- PickTopPlayedTracksForUser (For You's seeds):
|
|
-- tier 0 engagement in the last 7 days
|
|
-- tier 1 (only if tier 0 empty) last 30 days
|
|
-- tier 2 (only if tiers 0+1 empty) all-time
|
|
-- tier 3 (only if 0-2 empty) liked artists with no play history
|
|
-- All returned rows share one tier; produceSeedMixes maps it onto the
|
|
-- rule-#131 pick-kind ladder and stamps the built tracks, so metrics
|
|
-- can compare mixes seeded from fresh vs stale engagement.
|
|
-- Score = unskipped-play count + 5 if user has liked the artist. The
|
|
-- Go-side picker (pickSeedArtistsForDay) shuffles the 12
|
|
-- daily-deterministically and takes songsLikeSeedCount (6) so the mix
|
|
-- set both rotates day-to-day and fills the dedicated Home row (#1491).
|
|
WITH liked AS (
|
|
SELECT gla.artist_id FROM general_likes_artists gla WHERE gla.user_id = $1
|
|
),
|
|
recent7 AS (
|
|
SELECT t.artist_id,
|
|
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
|
|
0 AS tier
|
|
FROM play_events pe
|
|
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
|
WHERE pe.user_id = $1
|
|
AND pe.started_at > now() - INTERVAL '7 days'
|
|
AND t.artist_id IS NOT NULL
|
|
GROUP BY t.artist_id
|
|
),
|
|
recent30 AS (
|
|
SELECT t.artist_id,
|
|
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
|
|
1 AS tier
|
|
FROM play_events pe
|
|
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
|
WHERE pe.user_id = $1
|
|
AND pe.started_at > now() - INTERVAL '30 days'
|
|
AND t.artist_id IS NOT NULL
|
|
GROUP BY t.artist_id
|
|
),
|
|
alltime AS (
|
|
SELECT t.artist_id,
|
|
COUNT(*) FILTER (WHERE pe.was_skipped = false) AS play_count,
|
|
2 AS tier
|
|
FROM play_events pe
|
|
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
|
WHERE pe.user_id = $1
|
|
AND t.artist_id IS NOT NULL
|
|
GROUP BY t.artist_id
|
|
),
|
|
likedonly AS (
|
|
SELECT l.artist_id, 0::bigint AS play_count, 3 AS tier
|
|
FROM liked l
|
|
),
|
|
chosen AS (
|
|
SELECT artist_id, play_count, tier FROM recent7
|
|
UNION ALL
|
|
SELECT artist_id, play_count, tier FROM recent30
|
|
WHERE NOT EXISTS (SELECT 1 FROM recent7)
|
|
UNION ALL
|
|
SELECT artist_id, play_count, tier FROM alltime
|
|
WHERE NOT EXISTS (SELECT 1 FROM recent7)
|
|
AND NOT EXISTS (SELECT 1 FROM recent30)
|
|
UNION ALL
|
|
SELECT artist_id, play_count, tier FROM likedonly
|
|
WHERE NOT EXISTS (SELECT 1 FROM recent7)
|
|
AND NOT EXISTS (SELECT 1 FROM recent30)
|
|
AND NOT EXISTS (SELECT 1 FROM alltime)
|
|
)
|
|
SELECT c.artist_id,
|
|
(c.play_count + CASE WHEN l.artist_id IS NOT NULL THEN 5 ELSE 0 END)::bigint AS score,
|
|
c.tier::int AS tier
|
|
FROM chosen c
|
|
LEFT JOIN liked l ON l.artist_id = c.artist_id
|
|
ORDER BY score DESC, c.artist_id
|
|
LIMIT 12;
|
|
|
|
-- name: PickTopPlayedTracksForUser :many
|
|
-- For-You candidate seeds, tiered so For-You never silently vanishes:
|
|
-- tier 0 top non-skip plays in the last 30 days
|
|
-- tier 1 (only if tier 0 empty) all-time top non-skip plays
|
|
-- tier 2 (only if tiers 0+1 empty) liked tracks
|
|
-- Returns up to 5 ids; tie-break by track_id for determinism. The
|
|
-- Go-side picker (pickDailySeeds) draws the day's seeds from these.
|
|
-- Widened from a hard 7-day window, which made For-You disappear
|
|
-- after a week of not listening and never recover on a self-hosted
|
|
-- library with sparse history.
|
|
WITH recent AS (
|
|
SELECT t.id, COUNT(*) AS c, 0 AS tier
|
|
FROM play_events pe
|
|
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
|
WHERE pe.user_id = $1
|
|
AND pe.started_at > now() - INTERVAL '30 days'
|
|
AND pe.was_skipped = false
|
|
GROUP BY t.id
|
|
),
|
|
alltime AS (
|
|
SELECT t.id, COUNT(*) AS c, 1 AS tier
|
|
FROM play_events pe
|
|
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
|
WHERE pe.user_id = $1
|
|
AND pe.was_skipped = false
|
|
GROUP BY t.id
|
|
),
|
|
liked AS (
|
|
SELECT gl.track_id AS id, 0::bigint AS c, 2 AS tier
|
|
FROM general_likes gl
|
|
WHERE gl.user_id = $1
|
|
),
|
|
chosen AS (
|
|
SELECT id, c, tier FROM recent
|
|
UNION ALL
|
|
SELECT id, c, tier FROM alltime
|
|
WHERE NOT EXISTS (SELECT 1 FROM recent)
|
|
UNION ALL
|
|
SELECT id, c, tier FROM liked
|
|
WHERE NOT EXISTS (SELECT 1 FROM recent)
|
|
AND NOT EXISTS (SELECT 1 FROM alltime)
|
|
)
|
|
SELECT id
|
|
FROM chosen
|
|
ORDER BY tier, c DESC, id
|
|
LIMIT 5;
|
|
|
|
-- name: PickTopPlayedTrackForArtistByUser :one
|
|
-- "Songs like X" seed selection. Returns the user's most-played non-skipped
|
|
-- track by artist X in the last 7 days. Falls back to the artist's most-
|
|
-- recent album's first track if no plays exist.
|
|
-- Cast to uuid so sqlc infers pgtype.UUID rather than interface{}.
|
|
SELECT COALESCE(
|
|
(SELECT t.id
|
|
FROM play_events pe
|
|
JOIN tracks t ON t.id = pe.track_id AND t.missing_since IS NULL
|
|
WHERE pe.user_id = $1
|
|
AND t.artist_id = $2
|
|
AND pe.started_at > now() - INTERVAL '7 days'
|
|
AND pe.was_skipped = false
|
|
GROUP BY t.id
|
|
ORDER BY COUNT(*) DESC, t.id
|
|
LIMIT 1),
|
|
(SELECT t.id
|
|
FROM tracks t
|
|
JOIN albums a ON a.id = t.album_id
|
|
WHERE t.artist_id = $2
|
|
ORDER BY a.release_date DESC NULLS LAST,
|
|
t.disc_number NULLS LAST,
|
|
t.track_number NULLS LAST,
|
|
t.id
|
|
LIMIT 1)
|
|
)::uuid AS id;
|
|
|
|
-- name: PickTopAlbumCoverForArtistByUser :one
|
|
-- "Songs like X" cover. Most-played album by user; tie-break by most-recent
|
|
-- release. NULL if the artist has no albums or all covers are NULL.
|
|
-- Note: albums uses cover_art_path (not cover_path).
|
|
SELECT a.cover_art_path
|
|
FROM albums a
|
|
LEFT JOIN tracks t ON t.album_id = a.id
|
|
LEFT JOIN play_events pe
|
|
ON pe.track_id = t.id
|
|
AND pe.user_id = $1
|
|
AND pe.was_skipped = false
|
|
WHERE a.artist_id = $2
|
|
AND a.cover_art_path IS NOT NULL
|
|
GROUP BY a.id, a.release_date, a.cover_art_path
|
|
ORDER BY COUNT(pe.id) DESC, a.release_date DESC NULLS LAST, a.id
|
|
LIMIT 1;
|
|
|
|
-- name: DeleteSystemPlaylistsForUser :exec
|
|
-- Atomic-replace step. CASCADE on playlist_tracks.playlist_id handles
|
|
-- the join rows.
|
|
DELETE FROM playlists WHERE user_id = $1 AND kind = 'system';
|
|
|
|
-- name: ListPlaylistsByUserAndKind :many
|
|
-- Used in integration tests; production filter applied in Go
|
|
-- (api.handleListPlaylists fetches via Service.List and filters by Kind in
|
|
-- memory so it can also surface other users' public playlists alongside
|
|
-- the caller's own kind-filtered ones).
|
|
-- Sorted by updated_at DESC.
|
|
SELECT id, user_id, name, description, is_public, kind, system_variant,
|
|
seed_artist_id, cover_path, track_count, duration_sec,
|
|
created_at, updated_at
|
|
FROM playlists
|
|
WHERE user_id = $1
|
|
AND ($2::text = 'all' OR kind = $2)
|
|
ORDER BY updated_at DESC;
|
|
|
|
-- name: CreateSystemPlaylist :one
|
|
-- Inserts a system-generated playlist. Used by BuildSystemPlaylists.
|
|
-- For 'for_you' variant, pass seed_artist_id as NULL (zero-value pgtype.UUID
|
|
-- with Valid=false).
|
|
INSERT INTO playlists (
|
|
user_id, name, description, is_public,
|
|
kind, system_variant, seed_artist_id, cover_path
|
|
) VALUES ($1, $2, '', false, 'system', $3, $4, $5)
|
|
RETURNING *;
|
|
|
|
-- name: GetSystemPlaylistByVariantForUser :one
|
|
-- Returns the user's system playlist for the given variant
|
|
-- ('for_you' / 'songs_like_artist' / 'discover'), or pgx.ErrNoRows
|
|
-- when the user hasn't built that variant yet. For
|
|
-- 'songs_like_artist' (which can have multiple rows per user, one
|
|
-- per seed), this returns whichever LIMIT 1 picks — callers that
|
|
-- need all instances should use a different query.
|
|
SELECT p.id, p.user_id, p.name, p.kind, p.system_variant,
|
|
p.seed_artist_id, p.cover_path, p.track_count
|
|
FROM playlists p
|
|
WHERE p.user_id = $1
|
|
AND p.kind = 'system'
|
|
AND p.system_variant = $2
|
|
ORDER BY p.created_at DESC
|
|
LIMIT 1;
|