fix(server/playlists): redistributeSlots double-counting + tieBreakHash avalanche
Two real algorithm bugs in F-T1's For-You composition + Discover
allocator. Both surfaced as failing unit tests under go test -race.
1. redistributeSlots was re-redistributing a bucket's full deficit
on every pass instead of just the residual. The loop computed
`deficit = b.want - final[i]` each iteration, but final[i] for
a deficit bucket never increases (its supply is exhausted), so
pass N saw the same deficit as pass N-1 and kept shoveling it
to peers. For [want:40 avail:100, want:30 avail:0, want:30 avail:100],
four passes pushed cross-user's deficit into dormant+random four
times each, hitting the 100-slot clamp at the end and producing
[50, 0, 50] instead of the spec'd [55, 0, 45].
Fix: track per-source `redistributed[i]` and subtract it from the
deficit each pass. Multi-pass behavior still works for the case
where a peer's supply runs out mid-distribution.
2. tieBreakHash used FNV-1a 64-bit with trackID + dateStr appended.
For dateStrs differing only in the last character ("2026-05-07"
vs "2026-05-08"), the FNV state diverged only in low bits at the
final byte; multiplication by FNV_prime propagates upward but the
relative ordering of 60 small candidate UUIDs (which differ only
in their last byte) ended up identical across the two dates. The
For-You head/tail test asserted that the tail's first 5 should
change across days; it didn't.
Fix: switch to SHA-256 truncated to 8 bytes. SHA-256 has full
avalanche, so any single-bit input change roughly half-flips the
output bits and meaningfully reorders.
The hash isn't security-load-bearing; we just need strong avalanche
for tiny dateStr deltas. Determinism (same inputs → same output) is
preserved.
This commit is contained in:
@@ -150,17 +150,24 @@ type bucketRequest struct {
|
||||
// 1. Initial pass: allocate min(want, available) to each bucket;
|
||||
// compute supply = available - allocated.
|
||||
// 2. For each deficit bucket (allocated < want), split the deficit
|
||||
// equally across peers that still have supply > 0.
|
||||
// 3. Repeat until no further movement (deficit redistribution
|
||||
// can create new deficits if the recipient was already at its
|
||||
// `want`; we cap each bucket at its `available` after every pass).
|
||||
// equally across peers that still have supply > 0. Track how
|
||||
// much of that deficit each pass actually placed (some peers may
|
||||
// not have enough supply for their full share).
|
||||
// 3. Repeat until no movement happens — handles the case where a
|
||||
// peer's supply ran out mid-distribution and the residual needs
|
||||
// to roll to other peers in a later pass.
|
||||
//
|
||||
// For 3 buckets, two redistribution passes are sufficient in practice;
|
||||
// we cap at 4 passes defensively.
|
||||
// Bug history (May 2026): an earlier version recomputed
|
||||
// `deficit = b.want - final[i]` every pass without subtracting
|
||||
// previously-redistributed amounts, so a bucket with permanently
|
||||
// 0 final and want=30 kept handing out 30 to peers each pass. The
|
||||
// `redistributed` array below tracks per-source absorbed deficit so
|
||||
// subsequent passes only move the residual.
|
||||
func redistributeSlots(buckets []bucketRequest) []int {
|
||||
n := len(buckets)
|
||||
final := make([]int, n)
|
||||
supply := make([]int, n)
|
||||
redistributed := make([]int, n)
|
||||
for i, b := range buckets {
|
||||
final[i] = b.want
|
||||
if final[i] > b.available {
|
||||
@@ -172,11 +179,12 @@ func redistributeSlots(buckets []bucketRequest) []int {
|
||||
for pass := 0; pass < 4; pass++ {
|
||||
anyMoved := false
|
||||
for i, b := range buckets {
|
||||
deficit := b.want - final[i]
|
||||
// Remaining deficit = original gap minus what we've already
|
||||
// pushed out to peers in earlier passes.
|
||||
deficit := b.want - final[i] - redistributed[i]
|
||||
if deficit <= 0 {
|
||||
continue
|
||||
}
|
||||
// Find peers with supply.
|
||||
peers := make([]int, 0, n-1)
|
||||
for j := 0; j < n; j++ {
|
||||
if j == i {
|
||||
@@ -202,6 +210,7 @@ func redistributeSlots(buckets []bucketRequest) []int {
|
||||
if take > 0 {
|
||||
final[j] += take
|
||||
supply[j] -= take
|
||||
redistributed[i] += take
|
||||
anyMoved = true
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,9 +6,10 @@ package playlists
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"hash/fnv"
|
||||
"log/slog"
|
||||
"sort"
|
||||
"time"
|
||||
@@ -42,19 +43,27 @@ func pickSeedArtistsFromRows(rows []seedArtistRow) []pgtype.UUID {
|
||||
}
|
||||
|
||||
// tieBreakHash returns a deterministic 64-bit hash of (track_id, date_str).
|
||||
// Used to break score ties in mix candidate ranking. Same inputs always
|
||||
// produce the same output; different inputs almost always differ.
|
||||
// Used to break score ties and to drive the For-You tail sample's
|
||||
// daily-deterministic ordering. Same inputs always produce the same
|
||||
// output; different inputs almost always differ.
|
||||
//
|
||||
// Uses FNV-1a 64-bit (stdlib hash/fnv) — fast, deterministic, no extra
|
||||
// dependencies. The exact hash isn't load-bearing; any deterministic
|
||||
// 64-bit hash works.
|
||||
// Switched from FNV-1a (May 2026) because two date strings differing
|
||||
// only in the last character (e.g. "2026-05-07" vs "2026-05-08") gave
|
||||
// hash values whose RELATIVE ORDERING across our small candidate pools
|
||||
// was identical — the dateStr-driven divergence concentrated in low
|
||||
// bits that lost out to high-bit ordering during sort. SHA-256
|
||||
// truncated to 8 bytes has full avalanche, so any single-byte input
|
||||
// change roughly half-flips the output bits and reorders meaningfully.
|
||||
// The hash isn't security-load-bearing; we just want strong avalanche
|
||||
// for tiny input deltas.
|
||||
func tieBreakHash(trackID pgtype.UUID, dateStr string) uint64 {
|
||||
h := fnv.New64a()
|
||||
h := sha256.New()
|
||||
if trackID.Valid {
|
||||
_, _ = h.Write(trackID.Bytes[:])
|
||||
}
|
||||
_, _ = h.Write([]byte(dateStr))
|
||||
return h.Sum64()
|
||||
sum := h.Sum(nil)
|
||||
return binary.BigEndian.Uint64(sum[:8])
|
||||
}
|
||||
|
||||
// rankedCandidate is a (track_id, score) pair used during in-memory
|
||||
|
||||
Reference in New Issue
Block a user