The worker kept only the similar recordings already in the library, at most
20 of ListenBrainz's 50, and judged freshness by the edges it had written.
Two failures followed, both measured on the operator's library (#3879):
- A seed whose answer matched nothing wrote nothing, so it was never fresh.
With the queue ordered by id, 25 such seeds held its head and were
re-asked every hour; 17 of 2,466 played seeds had any edges.
- A recording that reached the library after its seed was fetched (a
Lidarr import, an MBID from the AcoustID lookup) was never linked until
a refetch, which for the stuck seeds never came.
Now every answer is cached whole in listenbrainz_similar_recordings and
every answer, an empty one or a permanent 4xx included, is recorded in
track_similarity_fetches. The queue reads the fetch record: never-fetched
first, then the oldest, refreshed after 30 days. The listenbrainz edges are
derived in SQL from the cache, one present track per recording and no cap,
for the seed just fetched and for every seed once per tick, so new arrivals
link within the hour without asking ListenBrainz again.
Artists get the same queue fix via artist_similarity_fetches; their answer
was already kept in artist_similarity and artist_similarity_unmatched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
users.api_token held each user's apiKey in plaintext and was looked up by
equality, so a leaked row or backup handed out working keys. Migration
0063 replaces it with api_token_hash (sha256, hex), computed in place
from the existing keys so every Subsonic client keeps working.
The key can no longer be read back: GET /api/me/api-token is gone, and
POST returns the new key once. Settings shows it right after Regenerate
with a copy button and a "won't be shown again" note.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Latent failures exposed now that integration tests run in CI (#339).
All test/test-harness only — no production code changes.
A (dbtest.ResetDB): also truncate cover_art_provider_settings +
cover_art_sources_meta. The monotonic source-version counter (seeded 1
by 0018) accumulated across internal/coverart tests → CurrentVersion=4
want 1, key-only-bump assertions, enricher source skips. SettingsService
re-seeds both at boot, so a truncated start is the correct fresh state.
B (playlists system_test.seedPlayEvent): inserted play_events with a
random session_id → play_events_session_id_fkey violation (7 tests).
Create the parent play_sessions row in the same statement (CTE).
C (similarity worker_integration_test.newTestWorker): built the client
with only BaseURL. Post-4fca0e6 similarity hits the Labs API via
LabsBaseURL, so an unset LabsBaseURL fell through to the real labs.api
and the stub never ran → 0 similarity rows. Set LabsBaseURL to the stub.
F (library scanner_test): NOT a flake — deterministic. Synthetic
ID3-only MP3s have no decodable duration; the scanner intentionally
won't skip duration_ms=0 rows (retries duration backfill), so every
re-scan reported Updated not Skipped. Seed duration_ms>0 before the
second scan so the mtime-based incremental-skip path under test is
actually exercised. Production scanner behavior unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of zero LB recommendations (every similar-recordings AND
similar-artists call returned HTTP 404, worker logs):
- Wrong host/path: client called
api.listenbrainz.org/1/explore/similar-{recordings,artists}/{mbid}.
/explore/... is a WEBSITE route, not an API endpoint — it 308s then
404s. Similarity datasets live on the separate Labs API.
- Invalid algorithm: the hardcoded
session_…_session_30_…_limit_100_filter_True_… is not a permitted
Labs enum member (400s) regardless of host.
Verified against the live Labs API:
GET labs.api.listenbrainz.org/similar-recordings/json
?recording_mbids=<mbid>&algorithm=<algo>
GET labs.api.listenbrainz.org/similar-artists/json
?artist_mbids=<mbid>&algorithm=<algo>
algorithm=session_based_days_9000_session_300_contribution_5_threshold_15_limit_50_skip_30
→ 200 for both. Response field names (recording_mbid/artist_mbid/
name/score) already match the existing structs — parsing unchanged.
- Add defaultLabsBaseURL + Client.LabsBaseURL (separate from the main
BaseURL; scrobble submission still uses api.listenbrainz.org).
- Drop the count/limit query param — result size is encoded in the
algorithm name (limit_50); caller still applies its own top-K.
- Tests: newTestClient sets LabsBaseURL; the two *_LimitParamSet tests
become *_MbidParamSet (assert the Labs path + mbid query param).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follows ca1bc5a, which raised the worker batch default. The defaults
test pinned the old value; align it with the intended new default.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- similarity.Worker batch 5→25 (tracks AND artists per 1h tick). At
5/h a freshly-scrobbled library took days to build a usable
similarity pool; 25/h converges in hours, still well under
ListenBrainz rate limits (429 aborts the tick).
- scanrun Stage 2b track backfill now runs unbounded (-1) instead of
reusing the album backfill's 5000 staged cap. It's a one-time
whole-library heal with no progress UI; a cap just left tracks.mbid
partially NULL (3634/18056 after one pass) until several future
scans caught up, re-reading untagged files each time. One uncapped
pass converges; later scans only re-read the remaining NULL rows.
Album backfill keeps its 5000 cap (it has staged scan_runs UX).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
upsertArtistSimilar keeps the existing matched path (top-K rows into
artist_similarity) and adds a parallel unmatched-persist loop with the
same top-K cap. Empty-name rows are skipped — we can't render a
suggestion card without a name. Logs unmatched-side errors at WARN
without aborting the tick (mirrors the matched-path policy).
Integration tests previously TRUNCATEd the users table at setup,
which wiped the operator's admin login every time the suite ran
against a Postgres instance shared with their dev environment.
This commit:
- Adds internal/dbtest/reset.go exposing ResetDB(t, pool) and
TestUserPrefix. ResetDB truncates every data table EXCEPT users,
then deletes only users whose username starts with TestUserPrefix.
- Migrates 9 test files (subsonic/scrobble, subsonic/star,
recommendation/candidates, scrobble/queue, similarity/worker,
playevents/writer, playsessions/service, api/auth, api/me) to
call ResetDB instead of issuing TRUNCATE-with-users.
- Renames hardcoded test usernames to use TestUserPrefix so ResetDB
cleans them between runs. seedUser in api/auth_test now prefixes
internally; existing call sites pass bare names.
- Leaves auth/bootstrap_test.go alone — it legitimately tests
first-time admin bootstrap and must wipe users.
Verified locally: full integration suite with -p 1 runs cleanly
under the existing MINSTREL_TEST_DATABASE_URL setup, and the admin
row in the shared dev DB survives the run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace stub with full tickOnce: drains played tracks/artists via
ListPlayedTracksNeedingSimilarity / ListPlayedArtistsNeedingSimilarity,
calls LB SimilarRecordings/SimilarArtists, filters to local library via
bulk MBID lookup, enforces top-K=20 by score, and upserts similarity
rows. 429 aborts the entire tick; other errors log-and-skip. Ten
integration tests covering no-op, library filtering, top-K cap, 7-day
freshness cap, stale refresh, 429 abort, transient skip, and artist
pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bvandeusenandClaude Opus 5.5