bvandeusen/minstrel
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 52 Wiki Activity

docs: wanted, dbtest, recommendation and playlists comments say what the code does (#6277)

Browse Source 
The same rewrite as d191a282, for four packages whose comments carried the
most history: tuning stories, quotes from the operator, single-install
measurements in library_scale.go, and 'used to' accounts of bugs. The
constraint each story protected stays as a present-tense reason (why tier 3
is reduced not removed, why the randomised arms take an order seed, why the
seed artist is eligible in Songs-like). Comments naming system_cron.go and
config.RecommendationConfig, neither of which holds what they claimed, now
name the scheduler and internal/recsettings.

Checked: non-comment tokens identical to HEAD; gofmt clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
bvandeusen
2026-10-23 12:19:15 -04:00
co-authored by Claude Opus 5.5
parent d191a28217
commit fcf5b7e5a8
48 changed files with 391 additions and 501 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/playlists/system.go
+66 -82
View File
@@ -1,5 +1,5 @@
// Package playlists' system.go implements the system-generated mix
// builder (M7 #352 slice 2). The cron loop and lazy fallback both call
// builder. The daily scheduler and the on-demand refresh both call
// BuildSystemPlaylists; the helpers in this file (pickSeedArtists,
// pickRepresentativeTrack, tieBreakHash) are the pure parts.
package playlists
@@ -52,15 +52,10 @@ func pickSeedArtistsFromRows(rows []seedArtistRow) []pgtype.UUID {
// daily-deterministic ordering. Same inputs always produce the same
// output; different inputs almost always differ.
//
// Switched from FNV-1a (May 2026) because two date strings differing
// only in the last character (e.g. "2026-05-07" vs "2026-05-08") gave
// hash values whose RELATIVE ORDERING across our small candidate pools
// was identical — the dateStr-driven divergence concentrated in low
// bits that lost out to high-bit ordering during sort. SHA-256
// truncated to 8 bytes has full avalanche, so any single-byte input
// change roughly half-flips the output bits and reorders meaningfully.
// The hash isn't security-load-bearing; we just want strong avalanche
// for tiny input deltas.
// SHA-256 truncated to 8 bytes, not FNV-1a: two date strings differing only
// in the last character must reorder the pool, and FNV's divergence for such
// inputs sits in low bits that lose to high-bit ordering during the sort.
// SHA-256 has full avalanche. The hash isn't security-load-bearing.
func tieBreakHash(trackID pgtype.UUID, dateStr string) uint64 {
h := sha256.New()
if trackID.Valid {
@@ -111,12 +106,9 @@ func pickDailySeeds(pool []pgtype.UUID, userID pgtype.UUID, dateStr string, n in
}
// songsLikeSeedCount is how many "Songs like {artist}" mixes to build
// each day (#1491). Bumped from 3 when Songs-like was promoted to its
// own dedicated Home row — the best-performing surface (low skip / high
// completion) was buried as 3 tiles in the shared Playlists carousel, so
// the wider row now shows a wider spread. Drawn from PickSeedArtists'
// deeper top-12 pool via a daily-deterministic shuffle so the set still
// rotates day to day rather than pinning the same six artists.
// each day, enough to fill Songs-like's own Home row. Drawn from
// PickSeedArtists' deeper top-12 pool via a daily-deterministic shuffle so
// the set still rotates day to day rather than pinning the same six artists.
const songsLikeSeedCount = 6
// pickSeedArtistsForDay picks up to songsLikeSeedCount seed artists for
@@ -127,10 +119,10 @@ func pickSeedArtistsForDay(pool []pgtype.UUID, userID pgtype.UUID, dateStr strin
}
// rankedCandidate is a (track_id, score) pair used during in-memory
// sorting before insert into playlist_tracks. T5 fills these from
// sorting before insert into playlist_tracks, filled from
// recommendation.Candidate scores. PickKind is the track's provenance
// within its mix — For-You's head/tail split (#1249) and Discover's
// buckets (#1270); empty persists as NULL (variant doesn't stamp yet).
// within its mix — For-You's head/tail split and Discover's buckets;
// empty persists as NULL (a variant that doesn't stamp).
type rankedCandidate struct {
TrackID pgtype.UUID
Score float64
@@ -140,8 +132,8 @@ type rankedCandidate struct {
// Pick kinds, persisted on playlist_tracks.pick_kind so plays can
// attribute skips to the population that sourced the track — For You's
// "taste engine missing" vs "freshness tax", Discover's three buckets
// (#1270). Values mirror the CHECK in migration 0041; taste_unheard is
// Discover's taste-targeted-novelty arm (#1488).
// Values mirror the CHECK in migration 0041; taste_unheard is Discover's
// taste-targeted-novelty arm.
const (
pickKindTaste = "taste"
pickKindFresh = "fresh"
@@ -155,7 +147,7 @@ const (
)
// pickKindForSeedTier maps PickSeedArtists' seed-pool fallback tier
// onto the rule-#131 pick-kind ladder: fresh 7-day engagement pins the
// onto the tiered pick-kind ladder: fresh 7-day engagement pins the
// mix's exact desire (tier1), the 30-day window steps back a little
// (tier2), and all-time / liked-only seeds are the far fallback
// (tier3). Stamped mix-wide — seed staleness is a property of the
@@ -172,7 +164,7 @@ func pickKindForSeedTier(tier int32) string {
}
// pickKindForMixTier maps a tiered mix query's 1-based tier column
// (numbered to match rule #131's ladder directly) onto the pick-kind
// (numbered to match the tier ladder directly) onto the pick-kind
// vocabulary. Distinct from pickKindForSeedTier, whose 0-based tiers
// count fallback steps of the seed pool rather than eligibility rungs.
func pickKindForMixTier(tier int32) string {
@@ -194,12 +186,11 @@ const systemMixLength = 25
// scores within a day, but near-tied candidates reshuffle across days
// so the playlist doesn't feel frozen.
//
// DB-tunable since #1250: the recsettings service pushes the current
// daily_mix profile via SetSystemMixWeights at boot and on every admin
// change (coverart Configure() pattern — no signature threading, live
// effect without restart). The literal here is only the pre-push
// value; shipped defaults live in recsettings.ShippedDailyMixWeights,
// which must stay in sync with it.
// DB-tunable: the recsettings service pushes the current daily_mix profile via
// SetSystemMixWeights at boot and on every admin change (coverart Configure()
// pattern — no signature threading, live effect without restart). The literal
// here is only the pre-push value; shipped defaults live in
// recsettings.ShippedDailyMixWeights, which must stay in sync with it.
var (
systemTuningMu sync.RWMutex
systemMixWeights = recommendation.ScoringWeights{
@@ -210,16 +201,16 @@ var (
JitterMagnitude: 0.1,
ContextWeight: 0.5,
SimilarityWeight: 1.5,
// Taste profile (#796 phase 2): the daily mixes are the primary
// Taste profile: the daily mixes are the primary
// taste-driven surface, so they lean on it. TasteMatchScore is in
// [-1,+1], so 1.5 makes a strong taste fit comparable to a like boost
// while passive avoidance (negative) gently demotes.
TasteWeight: 1.5,
// Time-of-day/weekday context affinity (#1531), in [-1,+1]. Starts
// uniform with radio pending trend data.
// Time-of-day/weekday context affinity, in [-1,+1]. The same as
// radio's until trend data says otherwise.
ContextTimeWeight: 1.0,
}
// Songs-like's own profile (#3881). Pre-push literal only; shipped
// Songs-like's own profile. Pre-push literal only; shipped
// defaults live in recsettings.ShippedSongsLikeWeights and must stay in
// sync with it, exactly as systemMixWeights does above.
//
@@ -239,7 +230,7 @@ var (
}
systemTasteConfig = taste.DefaultConfig()
// Sizes the candidate pool to the library (#3880). Cached because the
// Sizes the candidate pool to the library. Cached because the
// count is a full table scan and the daily build runs it once per user;
// within the TTL every user in a build shares one count.
systemLibrarySize = recommendation.NewLibrarySize(nil)
@@ -253,13 +244,13 @@ func SetSystemMixWeights(w recommendation.ScoringWeights) {
systemMixWeights = w
}
// SetSongsLikeWeights installs the songs_like scoring profile (#3881).
// SetSongsLikeWeights installs the songs_like scoring profile.
// Same push model as SetSystemMixWeights — recsettings calls it on boot and
// after every knob turn, so a tuning change takes effect on the next daily
// build with no restart.
//
// Separate from systemMixWeights because Songs-like and For-You want opposite
// things: For-You roams, Songs-like must not. Sharing one profile is what made
// things: For-You roams, Songs-like must not. Sharing one profile makes
// "Songs like X" wander.
func SetSongsLikeWeights(w recommendation.ScoringWeights) {
systemTuningMu.Lock()
@@ -268,7 +259,7 @@ func SetSongsLikeWeights(w recommendation.ScoringWeights) {
}
// SetTasteConfig installs the taste-profile build configuration
// (half-life + engagement curve, #1250). Same push model as
// (half-life + engagement curve). Same push model as
// SetSystemMixWeights.
func SetTasteConfig(c taste.Config) {
systemTuningMu.Lock()
@@ -276,13 +267,11 @@ func SetTasteConfig(c taste.Config) {
systemTasteConfig = c
}
// dailyOrderSeed is the value the randomised candidate arms order by (#3889).
// dailyOrderSeed is the value the randomised candidate arms order by.
//
// Per (user, day) so a same-day rebuild draws the SAME set — which is what
// TestBuildSystemPlaylists_DailyNonceDeterminism asserts and what those arms
// only ever achieved by accident before, when their limits happened to exceed
// the eligible rows. It changes on the day boundary, so the mixes still move
// daily.
// Per (user, day) so a same-day rebuild draws the SAME set, which is what
// TestBuildSystemPlaylists_DailyNonceDeterminism asserts. It changes on the
// day boundary, so the mixes still move daily.
func dailyOrderSeed(userID pgtype.UUID, dateStr string) string {
return uuidStringPL(userID) + ":" + dateStr
}
@@ -326,7 +315,7 @@ const (
)
// forYouSeedCount is how many of the user's top-5 tracks seed each
// day's build (#1269). A single rotating seed made the mix bipolar
// day's build. A single rotating seed would make the mix bipolar
// day-to-day — swinging whole neighborhoods as the rotation moved
// between dissimilar anchors; blending 3 spans neighborhoods within
// one day and smooths the swing across days.
@@ -334,7 +323,7 @@ const forYouSeedCount = 3
// headQuotas fixes each seed's share of the For-You head so one
// neighborhood can't monopolize it even when its candidates out-score
// the others (#1269): 50/30/20 for three seeds, 60/40 for two, all of
// the others: 50/30/20 for three seeds, 60/40 for two, all of
// it for one. Rounding remainder goes to the primary seed; a thin
// seed's unfilled quota spills best-first in pickQuotaHead.
func headQuotas(numSeeds, headN int) []int {
@@ -401,7 +390,7 @@ func pickQuotaHead(capped []recommendation.Candidate, seedOf map[pgtype.UUID]int
return head
}
// forYouTailHalfLifeRanks tunes the fresh tail's rank bias (#1269): a
// forYouTailHalfLifeRanks tunes the fresh tail's rank bias: a
// candidate's sampling weight halves every 50 ranks, so the 101st-best
// candidate is far likelier than the 401st-best instead of equal —
// freshness keeps its "you'll probably enjoy this" half.
@@ -450,8 +439,8 @@ func pickWeightedTail(tailPool []recommendation.Candidate, dateStr string, tailN
// because this sort IS the selection: the caller caps and truncates in the
// order this returns, so whatever profile ranks here decides which tracks
// reach the playlist. Scoring with daily_mix here and re-scoring with
// songs_like afterwards would have let the new profile relabel tracks it had
// no part in choosing — inert where it matters (#3881).
// songs_like afterwards would let the profile relabel tracks it had no part
// in choosing — inert where it matters.
func scoreAndSortCandidates(cands []recommendation.Candidate, userID pgtype.UUID, dateStr string, now time.Time, weights recommendation.ScoringWeights) []recommendation.Candidate {
rng := rand.New(rand.NewSource(int64(userIDHash(userID, dateStr))))
type scored struct {
@@ -603,9 +592,9 @@ func systemForYouSourceLimits() recommendation.CandidateSourceLimits {
RandomFill: 150,
// For-You / You-might-like are the taste-driven surfaces, so pull a
// deep slice of the user's top taste-profile artists into the pool
// (#796 phase 2b). Empty for cold-start users (no profile yet).
// Empty for cold-start users (no profile yet).
TasteOverlap: 80,
// Household co-play (#1533): a deeper collaborative slice for the
// Household co-play: a deeper collaborative slice for the
// taste surfaces. Empty on single-user servers.
UserCoplay: 40,
}
@@ -613,7 +602,7 @@ func systemForYouSourceLimits() recommendation.CandidateSourceLimits {
// produceForYou: blend candidate pools from up to forYouSeedCount of
// the user's top-5 played tracks (rotating daily via pickDailySeeds),
// then head+tail composition with per-seed head quotas (#1269). The
// then head+tail composition with per-seed head quotas. The
// base seed query failing is fatal; a per-seed candidate-load failure
// is logged and that seed just contributes nothing.
func produceForYou(
@@ -667,7 +656,7 @@ func produceForYou(
// produceSeedMixes: up to songsLikeSeedCount "Songs like {artist}"
// mixes. Seed artists rotate daily-deterministically; the seed query falls back
// through widening engagement windows (#1255) and every returned row
// through widening engagement windows and every returned row
// shares the winning tier, stamped onto the built tracks as their
// pick_kind. The base seed-artist query failing is fatal; per-artist
// failures are logged + skipped.
@@ -714,12 +703,12 @@ func produceSeedMixes(
continue
}
zeroVec := recommendation.SessionVector{Seed: true}
// Songs-like's own pool shape, not the default (#3881). Same total
// size; the composition shifts toward arms that actually measure
// distance from the seed. The default gave ~29% of candidates a
// sim_score of literally 0.
// Songs-like's own pool shape, not the default. Same total size; the
// composition shifts toward arms that actually measure distance from
// the seed, where the default gives ~29% of candidates a sim_score of
// 0.
//
// Then scaled to the library (#3880): a bigger collection should put
// Then scaled to the library: a bigger collection should put
// more genuinely-similar candidates in reach, not the same ~170
// regardless. The weights still rank sim_score-0 arms last, so the
// growth reaches coherence rather than working against it.
@@ -737,16 +726,11 @@ func produceSeedMixes(
}
// The seed artist's own songs are ELIGIBLE here, deliberately.
//
// This used to filter them out — "Songs like X excludes X's own
// songs" — which reads as obviously right and is not. The seed is a
// TRACK, and the tracks most likely to sound like it are usually the
// rest of that artist's catalogue; excluding them threw away the
// nearest neighbours of the very thing the mix is built around, and
// then reached further out to replace them. On a surface whose whole
// job is staying in one neighbourhood, that is backwards.
//
// Operator, 2026-09-10: "it should also be able to include music from
// the same artist."
// "Songs like X excludes X's own songs" reads as obviously right and
// is not. The seed is a TRACK, and the tracks most likely to sound
// like it are usually the rest of that artist's catalogue; excluding
// them would throw away the nearest neighbours of the very thing the
// mix is built around, and reach further out to replace them.
//
// Domination is bounded by the cap rather than by exclusion, which is
// the distinction that makes this safe: capCandidatesByAlbumAndArtist
@@ -791,12 +775,12 @@ func produceDiscover(
return []builtPlaylist{{Name: "Discover", Variant: "discover", Tracks: tracks}}, nil
}
// BuildSystemPlaylists builds the user's daily system mixes (one For-You +
// up to songsLikeSeedCount Songs-like-{seed} mixes). Atomic-replace inside one tx;
// concurrency-guarded via system_playlist_runs.in_flight; deterministic
// within a day via tieBreakHash(track_id, now.UTC().Format("2006-01-02")).
// BuildSystemPlaylists builds the user's daily system mixes (one For-You + up
// to songsLikeSeedCount Songs-like-{seed} mixes). Atomic-replace inside one tx;
// concurrency-guarded via system_playlist_runs.in_flight; deterministic within
// a day via tieBreakHash(track_id, now.UTC().Format("2006-01-02")).
//
// Callable from both the cron loop (system_cron.go) and the lazy fallback.
// Called by the daily scheduler and by the on-demand refresh endpoints.
// Returns nil when the user has no library activity to build from. Errors
// are captured in system_playlist_runs.last_error before returning.
func BuildSystemPlaylists(ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, userID pgtype.UUID, now time.Time, dataDir string) error {
@@ -929,13 +913,13 @@ func BuildSystemPlaylists(ctx context.Context, pool *pgxpool.Pool, logger *slog.
// truncates to n. Used by Songs-like-X (and as the fallback inside
// pickHeadAndTail for small pools).
func pickTopN(cands []recommendation.Candidate, userID pgtype.UUID, dateStr string, now time.Time, n int) []rankedCandidate {
// songs_like, not daily_mix (#3881). produceSeedMixes is this function's
// only caller, so the switch moves exactly one surface — For-You ranks
// through pickHeadAndTail and keeps the broader daily_mix profile.
// songs_like, not daily_mix. produceSeedMixes is this function's only
// caller, so this moves exactly one surface — For-You ranks through
// pickHeadAndTail and keeps the broader daily_mix profile.
//
// The SAME profile does the selection sort and the final score. Passing
// one and using the other is the subtle version of this bug: the playlist
// would still be chosen by daily_mix and merely wear songs_like numbers.
// one and using the other would leave the playlist chosen by daily_mix
// and merely wearing songs_like numbers.
weights := currentSongsLikeWeights()
sorted := scoreAndSortCandidates(cands, userID, dateStr, now, weights)
capped := capCandidatesByAlbumAndArtist(sorted)
@@ -955,7 +939,7 @@ func pickTopN(cands []recommendation.Candidate, userID pgtype.UUID, dateStr stri
// pickHeadAndTail picks headN taste anchors from the score-sorted pool
// (under per-seed quotas when seedOf/numSeeds describe a multi-seed
// blend, #1269) plus tailN freshness picks sampled rank-weighted from
// blend) plus tailN freshness picks sampled rank-weighted from
// positions 2*headN onward. Caps applied before the head/tail split.
// Used by For-You only.
//
@@ -1022,7 +1006,7 @@ func pickHeadAndTail(
// Combine: head first (score-sorted under quotas), then the fresh
// sample. "First similar, then surprise" reads naturally in
// playback. Head entries are the taste picks; tail entries are the
// freshness injection (#1249) — the split the metrics page
// freshness injection — the split the metrics page
// attributes skips to.
combined := make([]recommendation.Candidate, 0, len(head)+len(tail))
combined = append(combined, head...)
@@ -1044,7 +1028,7 @@ func pickHeadAndTail(
}
// insertSystemPlaylist inserts a kind='system' playlists row plus its
// playlist_tracks via AppendPlaylistTrack (slice 1's existing query, which
// playlist_tracks via AppendPlaylistTrack (which
// does the snapshot lookup from tracks/albums/artists). Then recomputes
// rollups. Returns the new playlist's ID so the caller can drive
// post-commit collage generation.
@@ -1101,8 +1085,8 @@ func insertSystemPlaylist(ctx context.Context, qtx *dbq.Queries, userID pgtype.U
}
// oneCopyPerSong keeps the first of a song's copies on different releases
// (M498 #5438): a mix holds the song once, in its best-ranked place. A video
// rip held back from the mixes (#5439) is left out, and a clean copy of the
// a mix holds the song once, in its best-ranked place. A video rip held back
// from the mixes is left out, and a clean copy of the
// same song can take its place.
func oneCopyPerSong(ctx context.Context, qtx *dbq.Queries, tracks []rankedCandidate) ([]rankedCandidate, error) {
ids := make([]pgtype.UUID, len(tracks))