diff --git a/.dockerignore b/.dockerignore index 0ab4ab54..be1135a4 100644 --- a/.dockerignore +++ b/.dockerignore @@ -9,12 +9,8 @@ web/build # The Android client — built by its own job, never from this context. The APK # reaches the image through client/, downloaded as a CI artifact, so nothing # here reads android/ sources. -# -# This block named `flutter_client/` until 2026-09-10 and lost its PATTERN when -# that tree was deleted, leaving a comment describing an exclusion that was no -# longer happening. android/ never took its place, so 4.1 MB of Gradle project -# has been entering the context and busting the `COPY . .` layer on every -# Android-only change. +# Excluded so the Gradle project neither bloats the context nor busts the +# `COPY . .` layer on every Android-only change. android/ # Local `make build` output — an 18 MB binary the image never uses, since the diff --git a/.env.example b/.env.example index 9745b25c..845fbb74 100644 --- a/.env.example +++ b/.env.example @@ -1,12 +1,11 @@ -# Minstrel environment variable overrides. -# Copy to .env for docker compose, or export before `go run ./cmd/minstrel`. -# Values shown are sensible defaults for the local docker-compose postgres. +# Environment for running the server outside Docker (`go run ./cmd/minstrel`). +# The values match the Postgres in docker-compose.yml. -SMARTMUSIC_SERVER_ADDRESS=:4533 -SMARTMUSIC_DATABASE_URL=postgres://minstrel:minstrel@localhost:5432/minstrel?sslmode=disable -SMARTMUSIC_LOG_LEVEL=info -SMARTMUSIC_LOG_FORMAT=text +MINSTREL_SERVER_ADDRESS=:4533 +MINSTREL_DATABASE_URL=postgres://minstrel:minstrel@localhost:5432/minstrel?sslmode=disable +MINSTREL_LOG_LEVEL=info +MINSTREL_LOG_FORMAT=text -# Set this when running `go test ./...` without -short to exercise the -# migration test against a running postgres. -# MINSTREL_TEST_DATABASE_URL=postgres://minstrel:minstrel@localhost:5432/minstrel?sslmode=disable +# Runs the database tests in `go test ./...` (without -short). Use a separate +# database: the tests truncate every table. +# MINSTREL_TEST_DATABASE_URL=postgres://minstrel:minstrel@localhost:5432/minstrel_test?sslmode=disable diff --git a/.gitea/workflows/release.yml b/.gitea/workflows/release.yml index a8c3a1f4..06bf08da 100644 --- a/.gitea/workflows/release.yml +++ b/.gitea/workflows/release.yml @@ -7,45 +7,39 @@ name: release # push tag vYYYY.MM.DD.HHMM → :latest (fresh APK bundled) # workflow_dispatch → manual trigger (same rules based on the ref) # -# That is the whole tag map, and it is family rule 145 + 147 as written. +# That is the whole tag map. # # : on main is the ROLLBACK UNIT — every production commit addressable # without a release ceremony. It is minted only on main, where rollback is -# actually worth having: merges are gated (rule 2) so they number in the dozens -# per year, while on dev they would be one per push, forever, for a channel -# whose entire contract is that it moves. +# actually worth having: merges to main are deliberate, so they number in the +# dozens per year, while on dev they would be one per push, forever, for a +# channel whose entire contract is that it moves. # -# There are NO : image tags. This repo published :vYYYY.MM.DD.HHMM -# until 2026-09-10 and it was the inverse of the rule on both counts — minting -# a version tag nobody pinned while the rollback unit the rule names did not -# exist here at all. Git and the build's own self-reported version answer -# "which build is this"; a third name for the same thing is upkeep for a model -# we do not run. Operator, 2026-09-10: "only things like the APK need that kind -# of versioning for their update process." +# There are NO : image tags. Git and the build's own self-reported +# version answer "which build is this"; a third name for the same thing is +# upkeep for a model we do not run. Only the APK needs version numbers, for +# its update process. # -# There is no :main either. :latest tracks main's tip with no gate between them -# (rule 147), so a second name for the same image sends readers looking for a +# There is no :main either. :latest tracks main's tip with no gate between +# them, so a second name for the same image sends readers looking for a # distinction that does not exist. # -# The dev channel exists so testing a build does not require shipping one. -# Before it, the only way to get an APK onto a phone was to cut a release, -# which made `main` the staging area by default. `:dev` carries its own -# freshly-built APK, signed with the SAME key as release builds — a different -# key cannot install over the stable app, so anyone crossing channels would -# have to uninstall and lose their data. +# The dev channel exists so testing a build does not require shipping one: +# without it the only way to get an APK onto a phone is to cut a release, +# which makes `main` the staging area. `:dev` carries its own freshly-built +# APK, signed with the SAME key as release builds — a different key cannot +# install over the stable app, so anyone crossing channels would have to +# uninstall and lose their data. # # :dev is published ALONE, with no per-commit tag. A rolling channel is # rolling by definition; a commit-addressable image for it would be a # rollback target nobody ever pulls, kept forever. Recovery on dev is to fix # forward. # -# Note what this repo does NOT need: a cross-repo dispatch to refresh the -# channel when its bundled APK is rebuilt. That mechanism exists elsewhere in -# the family because the app and the server live in separate repos. Minstrel -# is a monorepo — one push builds the APK and the image in the same run from -# the same commit, so the channel cannot go stale against its own artifact. -# The requirement is satisfied structurally; copying the mechanism would add -# a moving part to fix a problem that does not exist here. +# No cross-repo dispatch is needed to refresh a channel when its bundled APK +# is rebuilt: Minstrel is a monorepo, so one push builds the APK and the +# image in the same run from the same commit, and the channel cannot go +# stale against its own artifact. # # Release model: the tag IS the artifact's version name with a `v` in front. # `v2026.09.10.1432` and `2026.09.10.1432` are the same string, derived from @@ -55,15 +49,9 @@ name: release # # TAGS ARE IMMUTABLE. Never move, retarget or delete a published tag. A # same-day second release is not a collision — HHMM makes every tag unique -# by construction, so the answer is simply another tag. -# -# This block used to say the opposite: that the per-day tag was -# "intentionally mutable" and that a same-day re-cut should -# `git push -f origin vYYYY.MM.DD`. That instruction is what the family -# rulebook now forbids outright, and it has incidents behind it — moving a -# same-day tag forward once took a published release down with it. Anyone -# installing from a tag is holding something the tag no longer points at, -# which is a worse failure than an extra row in the tag list. +# by construction, so the answer is simply another tag. Anyone installing +# from a moved tag would hold something the tag no longer points at, which +# is a worse failure than an extra row in the tag list. # # :latest is updated by every main push AND every tag push, so it always # reflects the newest blessed image. @@ -83,15 +71,14 @@ name: release # recomputed ones — so no rebuild is needed, just a rebundle. Tag builds # keep bundling their own freshly-built APK. # -# THE GATE (rule 177, M462 #4984). Every verifying lane lives in this file — +# THE GATE. Every verifying lane lives in this file — # Go (vet, lint, short tests), the Postgres integration suite, the web app # (npm audit, svelte-check, vitest), Android (ktlint, detekt, unit tests) and # govulncheck — and every job that publishes something names each of them in # `needs:` and requires `success` from each, by name. Nothing publishes on red. # -# They used to be three separate workflows (test-go, test-web, android) on the -# same push trigger as this one. Separate workflows cannot see each other's -# verdict, so :dev meant "it built", never "it passed": a red test run and a +# One workflow, because separate workflows cannot see each other's verdict: +# :dev would mean "it built", never "it passed", and a red test run and a # fresh :dev could carry the same timestamp. One graph is the only place the # edge can be written. # @@ -125,10 +112,9 @@ on: type: boolean default: false -# A rapid re-push to main should supersede the in-flight build — the -# operator explicitly wants the later commit to win. Tags no longer enter -# into this: they are immutable and unique, so no tag build can ever be -# superseded by another run on the same ref. +# A rapid re-push to main supersedes the in-flight build: the later commit +# wins. Tags are immutable and unique, so no tag build can be superseded by +# another run on the same ref. concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true @@ -173,14 +159,14 @@ jobs: # Full `go test -race` against an ephemeral Postgres. # # DB wiring follows the act_runner shared-daemon pattern: the runner's Docker - # daemon also runs the operator's dev compose stack, so service containers - # get NO published ports (collision) and no service-name DNS. We discover the + # daemon also runs a dev compose stack, so service containers get NO + # published ports (collision) and no service-name DNS. We discover the # service container through the mounted docker socket and reach it by bridge # IP. The exactly-one assertion is a hard guard — pointing tests at the dev - # Postgres would truncate it (the disaster Fable #339 exists to prevent). + # Postgres would truncate it. # - # The key stays `integration` with no `name:` (rule 80): act_runner derives - # the service container's name from the job's display name. + # The key stays `integration` with no `name:`: act_runner derives the + # service container's name from the job's display name. # # `web/build/` has a committed placeholder index.html so go:embed succeeds # without the SPA being built first. @@ -195,7 +181,7 @@ jobs: POSTGRES_USER: minstrel POSTGRES_PASSWORD: minstrel POSTGRES_DB: minstrel_test - # No `ports:` — the runner shares the operator's dev compose + # No `ports:` — the runner shares the dev compose # Docker daemon; publishing a fixed host port collides. steps: @@ -209,13 +195,13 @@ jobs: # mounted docker socket. act_runner attaches the job # container and its service container(s) to a shared per-job # network, so scope discovery to a postgres that sits on a - # network THIS job container is also on. The old - # `--filter name=integration` matched EVERY concurrent - # integration run's postgres (a dev push + the main-merge run - # overlap → 2 candidates → false "expected exactly 1" abort). - # The operator's dev compose `minstrel-postgres-*` is never on - # this job's network; skip it explicitly as belt-and-suspenders - # (a wrong target would truncate real data). + # network THIS job container is also on. A name filter alone + # would match EVERY concurrent integration run's postgres (a dev + # push and a main-merge run overlapping give 2 candidates and a + # false "expected exactly 1" abort). The dev compose + # `minstrel-postgres-*` is never on this job's network; skip it + # explicitly as belt-and-suspenders (a wrong target would + # truncate real data). SELF=$(cat /etc/hostname) SELF_NETS=$(docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' "$SELF") test -n "$SELF_NETS" @@ -236,11 +222,11 @@ jobs: export MINSTREL_TEST_DATABASE_URL="postgres://minstrel:minstrel@${PG_IP}:5432/minstrel_test?sslmode=disable" # Wait for Postgres to accept connections. Asked of the service - # container itself: the run: shell is dash (rule 81), where the - # old `/dev/tcp` probe never connects and the loop silently - # burned its full two minutes on every run. + # container itself: the run: shell is dash, where a `/dev/tcp` + # probe never connects and the loop would silently burn its full + # two minutes. # - # Over TCP (-h 127.0.0.1), not the unix socket (#5426). On first + # Over TCP (-h 127.0.0.1), not the unix socket. On first # boot the image's entrypoint runs initdb under a temporary server # that listens on the socket only, then stops it and starts the # real one. A socket probe can catch the temporary server, and the @@ -267,7 +253,7 @@ jobs: # synchronous_commit is user-context, so pg_reload_conf() picks # all three up with no restart. # Non-fatal: a failure degrades to "slower", never red CI. But - # slower is ~15x on internal/api (#5426), so retry a few times and + # slower is ~15x on internal/api, so retry a few times and # print what took effect. relaxed="" for i in 1 2 3 4 5; do @@ -291,10 +277,10 @@ jobs: # every integration package TRUNCATEs the one shared test DB; # concurrent package binaries → TRUNCATE deadlocks. Serialize # package execution (the documented local invocation too). - # -timeout 20m: internal/api alone takes ~6.5 min under -race on - # an idle runner, and a dev and a main run sharing the runner - # pushed it past go test's default 10m (run 8368, 600.016s, the - # running test 2s old — nothing hung). + # -timeout 20m: internal/api alone takes several minutes under + # -race on an idle runner, and a dev and a main run sharing the + # runner can push it past go test's default 10m with nothing + # hung. MINSTREL_DATABASE_URL="$MINSTREL_TEST_DATABASE_URL" go run ./cmd/minstrel migrate go test -p 1 -race -timeout 20m ./... @@ -314,11 +300,8 @@ jobs: - name: Install deps run: npm ci - # The whole tree, build and test tooling included. Until #5021 this - # audited only what ships to browsers (`--omit=dev`), because vite, - # vitest, tailwind and kit carried advisories that needed major - # upgrades. Those upgrades landed and the full tree audits clean, so - # the tooling that builds the shipped bundle is held to the same bar. + # The whole tree, build and test tooling included: the tooling that + # builds the shipped bundle is held to the same bar as the bundle. - name: npm audit (all dependencies) run: npm audit --audit-level=moderate @@ -329,8 +312,8 @@ jobs: run: npm test android: - # Using flutter-ci runner label because it's the only proven-working - # label with docker that can pull our container.image. + # The flutter-ci runner label is the only proven-working label with + # docker that can pull our container.image. runs-on: flutter-ci container: image: git.fabledsword.com/bvandeusen/ci-android:36 @@ -376,11 +359,10 @@ jobs: - name: Unit tests run: ./gradlew testDebugUnitTest - # No debug APK is built or uploaded here. Main used to upload a - # debug-signed app-debug.apk: a build signed by a key regenerated in - # every container, which no install can update (family idea #5103, - # practice 2). Phones get builds from android-release, signed with - # the one release key, on dev and on tags. + # No debug APK is built or uploaded here: a debug build is signed by a + # key regenerated in every container, which no install can update. + # Phones get builds from android-release, signed with the one release + # key, on dev and on tags. # Known vulnerabilities in the Go code and the standard library it is built # with. Runs in the SAME image the Dockerfile's builder stage uses, so the @@ -388,7 +370,7 @@ jobs: # the ci-go image carries its own Go and would be checking a different # toolchain. Keep this image and the Dockerfile's builder in step. # - # govulncheck is fetched at CI time, unpinned (rule 154): the vulnerability + # govulncheck is fetched at CI time, unpinned: the vulnerability # database and the tool that reads it should both be current. govulncheck: runs-on: go-ci @@ -397,7 +379,7 @@ jobs: steps: # Plain git, not actions/checkout: that action runs on node, which the - # golang image does not carry. This step is dash (rule 81). + # golang image does not carry. This step is dash. - name: Checkout env: TOKEN: ${{ github.token }} @@ -420,7 +402,7 @@ jobs: android-release: name: Build signed APK (releases and dev) # Also builds on `dev`, which is what makes a test channel possible at - # all. Without it the only way to get a build onto a phone was to cut a + # all: without it the only way to get a build onto a phone is to cut a # release, which quietly turns `main` into the staging area. if: startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/dev' runs-on: flutter-ci @@ -453,7 +435,7 @@ jobs: - name: Checkout uses: actions/checkout@v4 with: - # Full history. The version name now reads only the tip commit's + # Full history. The version name reads only the tip commit's # timestamp, so a shallow clone would technically serve — but this # job derives a value that ships to devices, and a shallow checkout # changes what git-derived values resolve to WITHOUT failing. The @@ -470,7 +452,7 @@ jobs: # The derivation lives in ci/version.sh, not here, so it can be # executed by a test on every push. Anything inline in this file is # unverifiable until a release is already running. - out="$(ci/version.sh HEAD)" + out="$(ci/version.sh --apk HEAD)" printf '%s\n' "${out}" >> "$GITHUB_OUTPUT" # The channel is a property of the LANE, not of the commit, which is @@ -487,9 +469,9 @@ jobs: # Checked BEFORE the expensive work, not after it. "Attach APK to gitea # Release" below resolves the release by tag and fails if it is absent — - # but that is the final step, so a tag pushed without a release built an - # APK for several minutes first and only then discovered it had nowhere to - # put it. Same check, seconds in instead of minutes. + # but that is the final step, so a tag pushed without a release would + # build an APK for several minutes first and only then find it has + # nowhere to put it. Same check, seconds in instead of minutes. # # Releases are normally created through the API (which creates the tag and # the release together, so this passes). A bare `git push origin vX` is the @@ -549,16 +531,15 @@ jobs: # secret) reaches no installed phone. The certificate's digest is # pinned below; it is public, not a secret. Gradle signs with the # release key or leaves the APK unsigned, and an unsigned build fails - # here too, as there is no app-release.apk to verify (family idea - # #5103, practice 3). apksigner, not keytool: keytool prints nothing - # for a v2-only APK. + # here too, as there is no app-release.apk to verify. apksigner, not + # keytool: keytool prints nothing for a v2-only APK. # # Rotating the key on purpose means every install must be removed and # reinstalled; change the digest here in the same commit. - name: The APK carries the release key shell: bash env: - # CN=Minstrel, O=FabledSword. Read from run 8446 (#5116). + # CN=Minstrel, O=FabledSword. RELEASE_CERT_SHA256: 43d183307bc46b821789d90444a960b137f78f2166ff431efb2406d0fceaf612 run: | set -euo pipefail @@ -580,11 +561,10 @@ jobs: fi - name: Upload APK as workflow artifact - # Stock action (snippet #2271) — never @v3, which uploads something Gitea - # will never serve back. This is the producing half of a pair: - # image-release downloads `minstrel-apk` below. Any upload v4+ pairs with - # any download v4+ on this forge (every combination tested 2026-09-10, - # Scribe spike #3843), so the two pins need not move together. + # Stock action — never @v3, which uploads something Gitea will never + # serve back. This is the producing half of a pair: image-release + # downloads `minstrel-apk` below. Any upload v4+ pairs with any + # download v4+ on this forge, so the two pins need not move together. uses: actions/upload-artifact@v7 with: name: minstrel-apk @@ -687,7 +667,7 @@ jobs: image-release: name: Build + push container image - # Every lane must have SUCCEEDED, each named here (rule 177). Then the + # Every lane must have SUCCEEDED, each named here. Then the # APK: tag and dev pushes build one and it must have succeeded; main # pushes skip android-release and bundle the latest release's APK # instead, so for main alone a skipped android-release is expected. @@ -713,15 +693,13 @@ jobs: - name: Checkout uses: actions/checkout@v4 with: - # Full history, and rule 149 names this specifically: any job that - # DERIVES the version name needs it, because a shallow clone changes - # what git-derived values resolve to WITHOUT failing — a too-low - # value, silently, with every lane green. + # Full history: any job that DERIVES the version name needs it, + # because a shallow clone changes what git-derived values resolve to + # WITHOUT failing — a too-low value, silently, with every lane green. # - # This job was depth-1 while it took the version from GITHUB_REF. It - # now runs ci/version.sh itself, because with : image tags - # gone the server's self-reported version is the only thing that says - # which build an image is. + # This job runs ci/version.sh itself: with no : image tags, + # the server's self-reported version is the only thing that says which + # build an image is. fetch-depth: 0 - name: Detect buildable project @@ -743,21 +721,16 @@ jobs: set -euo pipefail # THE VERSION, and it is derived the same way on every ref — the - # branch decides the CHANNEL, never the version (family rule 149). + # branch decides the CHANNEL, never the version. # - # This used to be three different things: the literal string "main" - # on main, "dev" on dev, and the tag name on a tag. None of them - # ordered, and the first two were the same string forever — two dev - # images eight weeks apart were indistinguishable in the UI. That - # mattered little while :vYYYY.MM.DD.HHMM existed to identify a - # build; with version image tags gone, this IS how an operator tells - # which build a container is running. + # A per-ref string ("main", "dev", the tag name) would not order, and + # "dev" would be the same string forever — two dev images eight weeks + # apart indistinguishable in the UI. With no version image tags, this + # is how an admin tells which build a container is running. # # `sed -n s///p` rather than `grep`: it exits 0 when nothing matches, # so the empty check below is actually reachable. A grep here would - # kill the step at the assignment under the runner's pipefail — the - # exact bug that took down the first main build after the version - # rework. + # kill the step at the assignment under the runner's pipefail. VERSION="$(ci/version.sh HEAD | sed -n 's/^name=//p')" if [ -z "${VERSION}" ]; then echo "::error::could not derive a build version from ci/version.sh" @@ -770,13 +743,13 @@ jobs: # The tag build exists to produce the signed APK and attach it to # the release; the image it rebuilds is the SAME SOURCE as the main # build minutes earlier, differing only in which APK is baked in. - # Rule 145 is explicit about that case: when the same source is - # rebuilt with different contents, publish the moving channel tag - # and never a commit-addressable one. + # When the same source is rebuilt with different contents, + # publish the moving channel tag and never a commit-addressable + # one. # # :latest must move here rather than waiting for the next main # push, or the channel would carry the PREVIOUS release's APK - # indefinitely — a channel that cannot refresh itself (rule 146). + # indefinitely — a channel that cannot refresh itself. CHANNEL=stable echo "args=-t ${IMAGE}:latest" >> "$GITHUB_OUTPUT" echo "::notice::Release build ${VERSION}: refreshing :latest around the new APK" @@ -790,10 +763,9 @@ jobs: echo "args=-t ${IMAGE}:dev" >> "$GITHUB_OUTPUT" echo "::notice::Dev-branch build ${VERSION}: :dev" else - # The production line: :latest tracks main's tip (rule 147) and - # : is the rollback unit (rule 145). Full 40-char SHA, matching - # the family's other repos, so a rollback target is addressable - # straight from the commit anyone is reading. + # The production line: :latest tracks main's tip and : is + # the rollback unit. Full 40-char SHA, so a rollback target is + # addressable straight from the commit anyone is reading. CHANNEL=stable echo "args=-t ${IMAGE}:latest -t ${IMAGE}:${GITHUB_SHA}" >> "$GITHUB_OUTPUT" echo "::notice::Main-branch build ${VERSION}: :latest + :${GITHUB_SHA}" @@ -818,8 +790,8 @@ jobs: (startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/dev') # Consuming half of the pair: stock download-artifact, which works here for # the same reason as the upload (gitea/runner 3.x edits the GHES refusal - # out of the bundle; snippet #2271). v8 runs on node24, which every - # CI-runner image carries — the runner uses the image's own node. + # out of the bundle). v8 runs on node24, which every CI-runner image + # carries — the runner uses the image's own node. uses: actions/download-artifact@v8 with: name: minstrel-apk @@ -887,10 +859,10 @@ jobs: curl -fsSL -H "Authorization: token ${CI_TOKEN}" -o client/minstrel.apk "${APK_URL}" # Take the version the release RECORDED rather than recomputing it. - # This used to re-derive the name from the tagged commit, which meant - # the formula lived in two files that had to be kept in step, and it - # could only ever recover the name — the ordering key is build-time - # minutes and does not exist anywhere after that build ends. + # Re-deriving it from the tagged commit would keep the formula in two + # files that must stay in step, and could only ever recover the name — + # the ordering key is build-time minutes and exists nowhere after that + # build ends. SIDECAR_URL="$(printf '%s' "${REL_JSON}" | grep -oP '"browser_download_url":\s*"\K[^"]+' | grep -E '\.apk\.version$' | head -1)" || true if [ -n "${SIDECAR_URL}" ]; then curl -fsSL -H "Authorization: token ${CI_TOKEN}" -o client/minstrel.apk.version "${SIDECAR_URL}" @@ -924,18 +896,15 @@ jobs: # Verifies a tag release actually ended up complete, and names the specific # thing that's missing if not. # - # Added 2026-08-07 after v2026.08.07 was re-cut. The android-release job never - # started — no log was written at all — so all eight of its steps reported - # `failure` with none executed and image-release showed `skipped`. The run was - # red, but the *release page rendered fine*, and `main`'s own push build had - # already moved `:latest`, so the code was deployable and nothing looked - # obviously wrong. The release was simply missing its APK and its image, - # which is easy to skim past. + # A release can be incomplete while looking fine. If android-release never + # starts (a runner failing to launch), its steps all report `failure` with + # none executed and image-release shows `skipped`. The run is red, but the + # release page renders and main's own push build has already moved + # `:latest`, so a release missing its APK and its image is easy to skim + # past. # - # This job cannot prevent that (the cause was a runner failing to launch, not - # anything in this file). What it does is turn an incomplete release into an - # explicit, named error instead of eight mystery step failures — so the - # consequence is legible without having to infer it. + # This job cannot prevent that; it turns an incomplete release into an + # explicit, named error instead of a row of mystery step failures. # # `if: always()` is the whole point: it has to report precisely when the jobs # above did NOT succeed. @@ -977,22 +946,16 @@ jobs: echo "::notice::APK attached: ${APK}" # The other half. Checking only the APK would report success on a release - # whose image push failed — which is precisely the second thing that was - # missing when v2026.08.07 had to be re-cut. `always()` on this job means - # it runs even when image-release failed, so without this the guard would - # cheerfully verify an incomplete release. + # whose image push failed. `always()` on this job means it runs even when + # image-release failed, so without this the guard would cheerfully verify + # an incomplete release. # - # This asserted `:${TAG}` — the :vYYYY.MM.DD.HHMM image — until - # 2026-09-10. Version image tags are no longer published (rule 145), so - # that assertion would now fail every release for a tag nothing mints. - # The rollback target it was really protecting is the : image, which - # main's own build published for this same commit before the tag was cut. - # - # Checking it here earns its keep twice over: it still catches an image - # push that silently did not happen, and it additionally proves the - # ORDERING — a tag cut on a commit whose main build never completed has - # no rollback target, and that is worth failing on rather than - # discovering during an incident. + # The image checked is the : rollback target, which main's own build + # published for this same commit before the tag was cut (there are no + # version image tags). It catches an image push that silently did not + # happen, and it proves the ORDERING — a tag cut on a commit whose main + # build never completed has no rollback target, and that is worth failing + # on rather than discovering during an incident. - name: Rollback image must exist for the tagged commit shell: bash run: | diff --git a/.gitignore b/.gitignore index 11b676e2..97c979c9 100644 --- a/.gitignore +++ b/.gitignore @@ -12,13 +12,11 @@ # Test binary, built with `go test -c` *.test -# `make build` output. bin/minstrel was tracked until 2026-09-10 — an 18 MB -# binary committed by accident, last refreshed by a commit about web test -# mocks, and re-dirtied by every local build since. +# `make build` output — an 18 MB binary that is never committed. bin/ -# Bundled Android APK + version sidecar (#397). Populated by CI for -# tag releases; never committed. README in client/ explains the flow. +# Bundled Android APK + version sidecar. Populated by CI on every image +# build; never committed. README in client/ explains the flow. client/minstrel.apk client/minstrel.apk.version @@ -44,10 +42,9 @@ go.work.sum docs/superpowers/ # Per-machine Claude Code settings + remember-skill memory artifacts + -# project-level Claude/AI instruction files. All operator-scoped; the -# canonical project memory lives under ~/.claude/projects/ outside the -# repo. CLAUDE.md/AGENTS.md/GEMINI.md stay on the operator's machine -# only — they're tool-specific working notes, not project artifacts. +# project-level Claude/AI instruction files. All per-machine; +# CLAUDE.md/AGENTS.md/GEMINI.md are tool-specific working notes, not +# project artifacts. .claude/ .remember/ CLAUDE.md diff --git a/Dockerfile b/Dockerfile index 91903306..d69a99d8 100644 --- a/Dockerfile +++ b/Dockerfile @@ -19,25 +19,24 @@ ENV CGO_ENABLED=0 # (YYYY.MM.DD.HHMM) and the lane's channel; a local `docker build` falls back # to "dev"/"local". Both are surfaced at /healthz. # -# These are two values on purpose (family rule 149): the same commit built on -# dev and on main reports the same NAME and differs only in CHANNEL. Folding -# the channel into the version string is what the rule forbids — the version -# used to BE the channel word here ("main"/"dev"), which meant two dev images -# eight weeks apart were indistinguishable. +# These are two values on purpose: the same commit built on dev and on main +# reports the same NAME and differs only in CHANNEL. Folding the channel +# into the version string would make two dev images eight weeks apart +# indistinguishable. ARG MINSTREL_VERSION=dev ARG MINSTREL_CHANNEL=local RUN go build -trimpath \ -ldflags="-s -w \ - -X 'git.fabledsword.com/bvandeusen/minstrel/internal/server.ServerVersion=${MINSTREL_VERSION}' \ - -X 'git.fabledsword.com/bvandeusen/minstrel/internal/server.ServerChannel=${MINSTREL_CHANNEL}'" \ + -X 'git.fabledsword.com/bvandeusen/minstrel/internal/buildinfo.Version=${MINSTREL_VERSION}' \ + -X 'git.fabledsword.com/bvandeusen/minstrel/internal/buildinfo.Channel=${MINSTREL_CHANNEL}'" \ -o /out/minstrel ./cmd/minstrel FROM debian:bookworm-slim # ffmpeg: duration probes and the exact-tier audio hash (a SHA-256 of the # encoded audio packets, so no decode). libchromaprint-tools: fpcalc, the # acoustic fingerprint that tells the same recording at two bitrates apart -# from two different recordings (M400). Both are baked in at build time so a -# deployed instance never fetches either (rule 164); fpcalc is shelled out +# from two different recordings. Both are baked in at build time so a +# deployed instance never fetches either; fpcalc is shelled out # rather than bound because CGO_ENABLED=0 above rules out cgo. RUN apt-get update \ && apt-get install -y --no-install-recommends ca-certificates ffmpeg libchromaprint-tools \ @@ -57,10 +56,12 @@ COPY config.example.yaml /etc/smartmusic/config.yaml RUN mkdir -p /app/data /app/client && chown -R minstrel:minstrel /app WORKDIR /app -# In-app update channel (#397). client/ in the build context holds -# minstrel.apk + minstrel.apk.version (populated by release.yml on tag -# pushes; .gitkeep + README otherwise). Endpoints return 404 when the -# APK files aren't present, so non-tag images degrade gracefully. +# In-app update channel. client/ in the build context holds +# minstrel.apk + minstrel.apk.version, which release.yml fills on every +# build (dev and tag pushes bundle their own APK, main bundles the latest +# release's); a local build has only .gitkeep + README. Endpoints return +# 404 when the APK files aren't present, so such an image degrades +# gracefully. COPY --chown=minstrel:minstrel client/ /app/client/ USER minstrel diff --git a/Makefile b/Makefile index 7b5f63c0..be1fefb8 100644 --- a/Makefile +++ b/Makefile @@ -13,11 +13,8 @@ generate: generate-go: go run github.com/sqlc-dev/sqlc/cmd/sqlc@v$(SQLC_VERSION) generate -# Fail if the committed generated code no longer matches the .sql sources. -# -# Nothing verified this before, so internal/db/dbq could silently drift from -# internal/db/queries — a hand-edit, a half-applied regen, or a schema change -# without a regen would all pass CI while the typed layer lied about the SQL. +# Fail if the committed generated code no longer matches the .sql sources: +# a hand-edit, a half-applied regen, or a schema change without a regen. # # The diff is printed BEFORE the exit-code check on purpose: when this fails, # the log then contains sqlc's exact expected output, which is what you commit. @@ -36,8 +33,8 @@ test-short: go test -short -race ./... # Full suite incl. integration tests, against the dedicated minstrel_test -# DB so a run never truncates the dev `minstrel` DB (Fable #339). Ensures -# the test DB exists (idempotent — createdb errors if present, ignored). +# DB so a run never truncates the dev `minstrel` DB. Ensures the test DB +# exists (createdb fails harmlessly if it is already there). test-integration: docker compose up -d postgres -docker compose exec -T postgres createdb -U minstrel minstrel_test diff --git a/README.md b/README.md index 3437ba73..acb40640 100644 --- a/README.md +++ b/README.md @@ -1,60 +1,26 @@ -# Minstrel +

Minstrel logo

-A self-hosted music server that thinks for you. Smart shuffle, contextual likes, ListenBrainz-aware radio, and Lidarr automation — server-side, so every client (web, mobile, Subsonic third-party) gets the same intelligence. +

Minstrel

-> State and intelligence belong on the server, not the client. +

A self-hosted music server that picks what to play next.

+ +Minstrel serves your music library to the web, to its own Android app, and to any Subsonic client. Shuffle, radio, likes and play history are worked out on the server, so every device gets the same mixes and the same history. It can also find, download and file the albums you want, through a search service and a download client you run yourself. It is for anyone who keeps their own music collection and wants it to sound like more than a random shuffle. Minstrel home — your library at a glance -## Highlights - -- **OpenSubsonic-compatible.** Existing Subsonic clients (DSub, Symfonium, play:Sub, etc.) connect with no special configuration. -- **Server-side smart shuffle.** Track-similarity vectors, dual-like model (general + contextual), and session memory keep mixes coherent across devices. -- **ListenBrainz radio.** Session-aware "more like this" pulls from ListenBrainz similarity data, not a static genre tag. -- **Lidarr integration.** Triggered scans, request-driven album imports, and a quarantine flow when something doesn't fit — against a Lidarr instance *you* run and configure. Optional, and off until you supply a URL and API key. -- **Built-in web SPA.** Full-feature library, search, queue, playlists, and admin — no separate frontend container to deploy. -- **Native Android client, shipped with the server.** The signed APK is bundled into every image and attached to each [release](https://git.fabledsword.com/bvandeusen/minstrel/releases) — sideload it once, then the app self-updates straight from your own server (no app store, no separate download to track). - -## Scope and responsible use - -**Minstrel serves music you already have.** It is a library server: it indexes files on disk you point it at, and streams them to your own clients. It does not source, search for, or acquire content, and it has no opinion about where your files came from. - -Concretely, Minstrel ships **no** indexers, **no** trackers, **no** torrent / Usenet / NZB client, and **no** DRM circumvention of any kind. There is nothing to point at a content source because Minstrel has no such subsystem. - -The **Lidarr integration is optional and inert until you configure it.** You supply the URL and API key of a Lidarr instance you are already running; Minstrel then calls that instance's API to trigger scans, submit album requests, and reconcile imports. Minstrel neither bundles nor installs Lidarr, and configures no indexers on your behalf — Lidarr ships with none either, and any it uses are ones you added yourself. - -**What you put in your library, and what sources you configure in your own Lidarr, are your responsibility.** Copyright law applies to your collection the same way it applies to any other software that plays a file. Please respect it, and respect the terms of any service you connect. - -Minstrel is not affiliated with or endorsed by Lidarr, ListenBrainz, MusicBrainz, or Subsonic. - -## Quickstart +## Quick start ```yaml # compose.yaml services: minstrel: image: git.fabledsword.com/bvandeusen/minstrel:latest - # Reachable from your LAN at http://:4533. If this host faces the - # internet, bind it to 127.0.0.1 and put an HTTPS proxy in front instead: - # see docs/hosting.md. ports: ['4533:4533'] volumes: - # Your music library. Point ./music at wherever your audio files - # live. Writable, because Minstrel deletes a file when an admin asks - # it to (for example, quarantine's "Delete file"). It never moves, - # renames or retags anything. The container runs as uid 1000, so that - # user needs write access to the folders. Mount it :ro to forbid even - # deletes: those actions then refuse, say why, and delete nothing. - - ./music:/music - # Generated data: playlist cover collages, artist art, caches. - # The path must match MINSTREL_STORAGE_DATA_DIR, which the image - # sets to /app/data — keep this mount on /app/data or your cache - # won't survive a container recreate. - - minstrel-data:/app/data + - ./music:/music # your library; must be writable by uid 1000 + - minstrel-data:/app/data # art and caches environment: MINSTREL_DATABASE_URL: postgres://minstrel:minstrel@db:5432/minstrel?sslmode=disable - # Colon-separated library roots to scan; must match the container - # path of the music mount above (/music here). MINSTREL_LIBRARY_SCAN_PATHS: /music depends_on: [db] @@ -64,9 +30,7 @@ services: POSTGRES_USER: minstrel POSTGRES_PASSWORD: minstrel POSTGRES_DB: minstrel - # Postgres data dir — users, likes, play history, sessions, settings. - # The one volume you must never lose; back it up with pg_dump. - volumes: [pgdata:/var/lib/postgresql/data] + volumes: [pgdata:/var/lib/postgresql/data] # users, likes, history: back this up volumes: minstrel-data: @@ -75,105 +39,101 @@ volumes: ```bash docker compose up -d +docker compose logs minstrel | grep setup_token ``` -## First run +Open `http://:4533/register` and create the first account with that setup token. The first account is the administrator. -With the stack up, a handful of in-app steps get you to a working library. Use your own host in place of `localhost` if you're reaching the server over a LAN/VPN address. Plain `http://` is fine on a network you trust; a server reachable from the internet belongs behind HTTPS, which [docs/hosting.md](docs/hosting.md) walks through. +### First run -**1. Create your admin account.** Visit `http://localhost:4533/register`. The first account on a fresh instance becomes the administrator, and creating it asks for the **setup token** the server prints in its log (`docker compose logs minstrel | grep setup_token`), so nobody else can claim a newly exposed server first. Later users join through the same form or an invite token (step 5). - -Creating the first (admin) account on a fresh instance - -**2. Let the first library scan finish.** `scan_on_startup` is on by default, so Minstrel walks your mounted library on boot and imports artists, albums, and tracks — no button to press. Watch progress (and re-scan any time) on the **Admin** page (`/admin`); the scan runs in stages and is incremental, so later restarts only pick up what changed. +1. **Let the first scan finish.** Minstrel scans the library on startup. Watch it, or scan again, on the **Admin** page (`/admin`). Later scans only pick up what changed. +2. **Install the Android app.** In **Settings** (`/settings`), the *Install the Android app* card downloads the app this server ships. It updates itself from your server after that. +3. **Invite the household.** In **Admin › Users** (`/admin/users`), make an invite token or turn on open registration. Each person gets their own likes, history and recommendations. +4. **Optional: connect services.** Add a search service and a download client in **Admin › Acquisition** (`/admin/acquisition`). Each person adds their own ListenBrainz token in **Settings › Connections**. The Admin page, where the library scan runs and reports progress -**3. (Optional) Name the instance and wire up integrations.** In admin **Settings → Integrations** (`/admin/integrations`), add a ListenBrainz token (scrobbling + similarity radio) and/or a Lidarr URL + API key (the request flow). These live in the UI and apply without a restart; the display name can also be set via `MINSTREL_BRANDING_APP_NAME`. - -ListenBrainz and Lidarr integration cards in admin Settings - -**4. Install the Android app.** Open **Settings** (`/settings`) and use the *Install the Android app* card to download the APK that ships inside this server image, then sign in with the same account. From then on the app self-updates straight from your server. - -The "Install the Android app" download card in Settings - -**5. Invite the rest of the household.** From admin **Users** (`/admin/users`), generate an invite token (or enable open registration). Each person gets their own account, so likes, play history, and recommendations stay per-user. - -Generating an invite token in admin Users - -For the full configuration surface, see [`config.example.yaml`](./config.example.yaml). - -Hosting Minstrel on the internet: see [docs/hosting.md](docs/hosting.md). What Minstrel does to protect accounts, and why: [docs/security.md](docs/security.md). - ## Configuration -Most operators only need the env vars in the quickstart above. A few extras worth knowing: +The quick start sets everything that's required. These are the settings people most often change: -- `MINSTREL_BRANDING_APP_NAME` — rename the instance ("Family Jukebox", "Office Music"). Surfaces in the header, browser tab, and OG share previews. -- `MINSTREL_STORAGE_DATA_DIR` — where generated artefacts (playlist cover collages, artist art, caches) are written. The container image sets this to `/app/data`, which is why the quickstart mounts the `minstrel-data` volume there. -- `MINSTREL_LIBRARY_SCAN_PATHS` — colon-separated list of music library roots to scan. Supports multiple roots (`/music:/podcasts`). +| Variable | What it does | +|---|---| +| `MINSTREL_DATABASE_URL` | Postgres connection string. Required. | +| `MINSTREL_LIBRARY_SCAN_PATHS` | Library folders inside the container, colon-separated (`/music:/audiobooks`). They are used on first start only; after that, change them in **Admin › Library**. | +| `MINSTREL_BRANDING_APP_NAME` | The name shown in the header, the browser tab and link previews. | +| `MINSTREL_LOG_LEVEL` | `debug`, `info`, `warn` or `error`. | -ListenBrainz integration (per-user scrobble + similarity tokens) and Lidarr integration (URL + API key) are configured through the admin Settings UI rather than env vars or yaml — per Minstrel's "config in UI" rule, integration settings live where operators can edit them without restarting. +Search services, download clients, ListenBrainz, email and recommendation tuning are set in the web UI and take effect without a restart. The full file-based surface is in [`config.example.yaml`](./config.example.yaml); to use it, mount your copy at `/etc/smartmusic/config.yaml`. -Most operational keys have a `MINSTREL_
_` env override. Recommendation and events tuning are yaml-only. See [`config.example.yaml`](./config.example.yaml) for the authoritative surface. +## Setup pitfalls -## Updating +- **The library mount has to be writable by uid 1000.** Minstrel files the albums it downloads into the library, and it deletes a file when an admin asks it to. It never moves, renames or retags files that were already there. Mount the library `:ro` to forbid writes: those actions then refuse and say why. +- **`MINSTREL_LIBRARY_SCAN_PATHS` names the path inside the container** (`/music`), not the folder on your host. +- **On the internet, put it behind HTTPS.** Plain `http://` is fine on a network you trust. For a public server, bind the port to `127.0.0.1` and put a reverse proxy in front: see [docs/hosting.md](docs/hosting.md). -Image tags (`git.fabledsword.com/bvandeusen/minstrel:`): +## Updating and backups -- `:latest` — production. Tracks `main`'s tip and moves on every `main` push and every release. What most operators should run. -- `:` — the rollback unit. Every `main` push publishes one, so any production commit is addressable without a release ceremony. Immutable: a given SHA tag is never re-pushed. Pin one if you need a deployment that cannot change under you, and use it to roll back. -- `:dev` — the rolling test channel, rebuilt on every push to `dev` and carrying its own freshly-built Android APK. Run this to try something before it ships. It moves constantly, has no per-commit tag, and its only recovery path is forward — if a `:dev` image is broken, the fix is the next push, not a rollback. +```bash +docker compose pull && docker compose up -d +``` -That is the whole tag map. **There are no version-numbered image tags**, and no `:main`. Git and the build's own self-reported version answer "which build is this" — the Settings page shows it, and so does `/healthz`. Release *tags* in git are still `vYYYY.MM.DD.HHMM`; they name a changelog entry and the APK attached to it, not an image. +Database migrations run at startup. Image tags: -Rolling back to `:` pins the **server code** at that commit — not the server-and-app pair. The Android APK is baked in at image build time, so a SHA image carries whichever app was current when that commit was built, which may be older than what `:latest` bundles now. If both halves matter, check what the image bundles rather than trusting the tag's name. +- `:latest`: the stable release. Most people should run this. +- `:`: one fixed build. Pin one to stop updates, or to roll back. +- `:dev`: the test channel, rebuilt on every change. It may break, and the fix is the next build. -Every `:latest`, `:` and `:dev` bundles a signed Android APK, so the in-app update channel is always live. All are signed with the same key, so a phone can move between the stable and dev channels without uninstalling — point it at a `:dev` server and the in-app updater offers that channel's build. +Every image carries a signed Android app, all with the same key, so a phone can move between channels without reinstalling. Rolling back a SHA tag rolls back the app it carries too. -The app reports which channel it is on alongside its version, and decides whether an update is available using the build's ordering key rather than its displayed name — the same value Android installs by, so an offer it makes is one the platform will accept. +**Back up the Postgres volume** with `pg_dump`. It holds accounts, likes, play history, playlists and settings, and a migration can only be undone by restoring a dump. The `minstrel-data` volume holds artwork and caches that Minstrel rebuilds if lost. -Database migrations run automatically at startup; rollbacks require restoring a Postgres dump. +## Features -Releases up to 2026-09-10 also published a `:vYYYY.MM.DD[.HHMM]` image tag. Those images still exist and still work — they are simply not extended. +**Listening** +- Smart shuffle that weighs what you like, what you skip and what you've heard lately, instead of picking at random. +- Two kinds of like: a song you love, and a song that's right for what you're listening to now. +- Radio that starts from any song, using ListenBrainz similarity as well as your own history. +- Daily mixes and system playlists built from your listening. +- Playback on Sonos and other UPnP speakers from the Android app. +- Loudness levelling between tracks, so you never reach for the volume. -## Specs +**Your library** +- Browse by artist, album, genre and year, with search across everything. +- Duplicates, missing files and suspect copies are found and listed under **Collection** for an admin to fix. +- Missing album art and MusicBrainz ids are filled in automatically. -Authoritative scope lives under [`docs/`](./docs): +**Getting music** +- A wanted list: requests from the household, artists you follow, and albums whose files went missing. +- Minstrel searches the indexers you configure, hands the best release to your download client, checks what arrives against MusicBrainz, tags it, and files it. A better copy replaces a poor one. +- Off until you add a search service and a download client. -- [Server spec](./docs/smart-music-server-spec.md) — current implementation focus. -- [Client spec](./docs/smart-music-client-spec.md) — Flutter companion app. +**Clients and accounts** +- A full web app, built into the server. +- A native Android app that ships inside the server image, keeps playing what it has cached when the server is out of reach, and updates itself from your server. +- OpenSubsonic-compatible, so existing clients such as DSub, Symfonium and play:Sub connect as they are. +- Separate accounts for each person, with invites, and an inbox for what happened while you were away. + +## Scope and responsible use + +Minstrel indexes the files you point it at and streams them to your own clients. It ships no indexers, no trackers, no torrent, Usenet or NZB engine of its own, and no DRM circumvention. + +Getting music is optional and does nothing until you configure it. It searches only the search service (Prowlarr or a Torznab indexer) and the download client (qBittorrent) you add yourself. + +What you put in your library, and the sources you configure, are your responsibility. Copyright law applies to your collection as it does to any other software that plays a file; please respect it, and the terms of any service you connect. + +Minstrel is not affiliated with or endorsed by ListenBrainz, MusicBrainz, Prowlarr, qBittorrent or Subsonic. ## Development -Two concurrent dev processes: +- **Backend:** `docker compose up` in this repo runs Postgres and Minstrel on `:4533`. +- **Web:** `cd web && npm install && npm run dev` starts Vite on `:5173`. It proxies `/api/*` and `/rest/*` to `:4533`. +- **Tests:** `make test-short` runs the unit tests. `make test-integration` also runs the database tests, against a separate `minstrel_test` database so your dev data is never truncated. +- **Image:** `docker build -t minstrel .` builds the web app and embeds it in the server binary. +- **Branches:** work lands on `dev`; `main` changes through pull requests from `dev`. -1. **Backend:** `docker compose up` — Postgres + Minstrel on `:4533`. -2. **Frontend:** `cd web && npm install && npm run dev` — Vite dev server on `:5173` with HMR. The Vite server proxies `/api/*` and `/rest/*` to `:4533` so session cookies work. - -### Testing - -- Unit + race (no DB): `make test-short`. -- Full suite incl. integration tests: `make test-integration`. This runs - against a dedicated `minstrel_test` database so a test run never - truncates your dev `minstrel` data (admin user, library, likes). It - brings up the compose Postgres and creates the test DB if missing. -- CI runs both: a fast `go test -short -race` gate plus an integration - job with its own ephemeral Postgres (the `integration` lane in - `.gitea/workflows/release.yml`, which also gates every image publish). - -### Production build - -`docker build -t minstrel .` runs the SvelteKit build inside a `node` stage, copies the output into the `golang` stage, and `//go:embed`s it into the final binary. The container serves the SPA from `/` alongside the API surfaces; no separate static-file server is required. - -### Branches - -- Day-to-day work happens on `dev` (or feature branches merged into `dev`). -- `main` is **protected** — changes land via PR from `dev`. -- Releases are cut by tagging `v*` off `main`; the release workflow builds the signed APK, attaches it to the release, and refreshes `:latest` around it. - -Task and milestone tracking: Fable (`Minstrel` project, id 12). +How accounts and sessions are protected: [docs/security.md](docs/security.md). ## License -See [LICENSE](./LICENSE). +MIT. See [LICENSE](./LICENSE). diff --git a/android/app/build.gradle.kts b/android/app/build.gradle.kts index e185e97e..0b0cf2a7 100644 --- a/android/app/build.gradle.kts +++ b/android/app/build.gradle.kts @@ -1,9 +1,9 @@ plugins { alias(libs.plugins.android.application) // kotlin-android NOT applied: AGP 9 enables built-in Kotlin by default, - // and KSP 2.3.x supports it (PR #2674, merged Oct 2025). serialization - // + compose-compiler are language-level Kotlin compiler plugins and - // still need explicit application. + // and KSP 2.3.x supports it. serialization + compose-compiler are + // language-level Kotlin compiler plugins and still need explicit + // application. alias(libs.plugins.kotlin.serialization) alias(libs.plugins.compose.compiler) alias(libs.plugins.ksp) @@ -31,14 +31,9 @@ android { // // versionCode is minutes since 2020-01-01 at BUILD time. It is the // value the platform decides installs by, so it must be monotonic by - // construction. - // - // This comment used to say versionCode was a commit count and that it - // was "monotonic forever". It was neither — a commit count runs ahead - // on `dev`, so a dev build outranked the `main` release meant to - // replace it and Android refused the install as a downgrade. Worth - // knowing the claim was here, stated as a reassurance, while the bug - // it denied was live. + // construction. Not a commit count: that runs ahead on `dev`, so a dev + // build would outrank the `main` release meant to replace it and + // Android would refuse the install as a downgrade. val versionNameOverride = (project.findProperty("MINSTREL_VERSION_NAME") as String?)?.takeIf { it.isNotBlank() } val versionCodeOverride = @@ -73,9 +68,9 @@ android { "proguard-rules.pro", ) // Signed with the release key or not at all. Falling back to the - // debug key made a missing secret into a published APK that no - // install could ever update (family idea #5103, practice 2). An - // unsigned build installs nowhere, so the gap shows at once. + // debug key would turn a missing secret into a published APK that no + // install could ever update. An unsigned build installs nowhere, so + // the gap shows at once. signingConfig = if (System.getenv("ANDROID_KEYSTORE_PATH").isNullOrEmpty()) { null diff --git a/android/app/src/main/AndroidManifest.xml b/android/app/src/main/AndroidManifest.xml index a23ca32d..cd8f9563 100644 --- a/android/app/src/main/AndroidManifest.xml +++ b/android/app/src/main/AndroidManifest.xml @@ -8,7 +8,7 @@ - @@ -84,11 +84,8 @@ - + ( + keyOf = { it.key }, + lengthOf = { it.length }, + lruOrder = compareBy { it.touched }.thenBy { it.key }.thenBy { it.position }, + ).apply { + likedCapBytes = likedCap + recentCapBytes = recentCap + } + + /** Drop spans the way the evictor does, until both budgets fit. */ + private fun BucketLedger.drain(): List { + val dropped = mutableListOf() + while (true) { + val span = nextEviction() ?: return dropped + remove(span) + dropped += span + } + } + + @Test + fun `a liked track's bytes count against Liked only`() { + val l = ledger() + l.setLiked(setOf("liked")) + l.add(Span("liked", 0, 300, touched = 1)) + l.add(Span("other", 0, 200, touched = 2)) + + assertEquals(300, l.bytesIn(CacheBucket.LIKED)) + assertEquals(200, l.bytesIn(CacheBucket.RECENT)) + } + + @Test + fun `filling Recently played drops its oldest audio and never liked audio`() { + val l = ledger(likedCap = 1_000, recentCap = 250) + l.setLiked(setOf("liked")) + val liked = Span("liked", 0, 100, touched = 0) + val oldest = Span("a", 0, 100, touched = 1) + l.add(liked) + l.add(oldest) + l.add(Span("b", 0, 100, touched = 2)) + l.add(Span("c", 0, 100, touched = 3)) + + assertEquals(listOf(oldest), l.drain()) + assertEquals(200, l.bytesIn(CacheBucket.RECENT)) + assertEquals(100, l.bytesIn(CacheBucket.LIKED)) + } + + @Test + fun `filling Liked drops its least recently played liked audio`() { + val l = ledger(likedCap = 150, recentCap = 1_000) + l.setLiked(setOf("x", "y")) + val older = Span("x", 0, 100, touched = 1) + l.add(older) + l.add(Span("y", 0, 100, touched = 2)) + l.add(Span("recent", 0, 500, touched = 0)) + + assertEquals(listOf(older), l.drain()) + assertEquals(500, l.bytesIn(CacheBucket.RECENT)) + } + + @Test + fun `liking a cached track moves its bytes into Liked`() { + val l = ledger(likedCap = 1_000, recentCap = 150) + l.setLiked(emptySet()) + l.add(Span("t", 0, 100, touched = 1)) + l.add(Span("t", 100, 50, touched = 2)) + l.add(Span("u", 0, 100, touched = 3)) + assertEquals(250, l.bytesIn(CacheBucket.RECENT)) + + l.setLiked(setOf("t")) + + assertEquals(150, l.bytesIn(CacheBucket.LIKED)) + assertEquals(100, l.bytesIn(CacheBucket.RECENT)) + assertEquals(emptyList(), l.drain()) + } + + @Test + fun `unliking moves the bytes to Recently played, which then trims`() { + val l = ledger(likedCap = 1_000, recentCap = 150) + l.setLiked(setOf("t")) + val unliked = Span("t", 0, 100, touched = 1) + l.add(unliked) + l.add(Span("u", 0, 100, touched = 2)) + + l.setLiked(emptySet()) + + assertEquals(listOf(unliked), l.drain()) + assertEquals(0, l.bytesIn(CacheBucket.LIKED)) + } + + @Test + fun `shrinking a cap trims at once`() { + val l = ledger(recentCap = 1_000) + l.setLiked(emptySet()) + l.add(Span("a", 0, 100, touched = 1)) + l.add(Span("b", 0, 100, touched = 2)) + assertEquals(emptyList(), l.drain()) + + l.recentCapBytes = 100 + + assertEquals(listOf("a"), l.drain().map { it.key }) + } + + @Test + fun `a cap of 0 never evicts`() { + val l = ledger(likedCap = 0, recentCap = 0) + l.setLiked(setOf("liked")) + repeat(50) { l.add(Span("k$it", 0, 1_000_000, touched = it.toLong())) } + l.add(Span("liked", 0, 1_000_000, touched = 99)) + + assertNull(l.nextEviction()) + } + + @Test + fun `nothing is evicted before the liked set is known`() { + val l = ledger(likedCap = 100, recentCap = 100) + l.add(Span("maybe-liked", 0, 500, touched = 1)) + + assertNull(l.nextEviction()) + } + + @Test + fun `a write about to start makes room in its own bucket`() { + val l = ledger(likedCap = 1_000, recentCap = 200) + l.setLiked(setOf("liked")) + l.add(Span("a", 0, 150, touched = 1)) + + assertNull(l.nextEviction(pendingKey = "liked", pendingBytes = 100)) + assertEquals("a", l.nextEviction(pendingKey = "new", pendingBytes = 100)?.key) + } + + @Test + fun `Liked room is what the Liked cap has left`() { + val l = ledger(likedCap = 500) + l.setLiked(setOf("x")) + l.add(Span("x", 0, 200, touched = 1)) + + assertEquals(300, l.likedRoomBytes()) + l.likedCapBytes = 0 + assertEquals(Long.MAX_VALUE, l.likedRoomBytes()) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/cache/mutations/SupersededToggleIdsTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/cache/mutations/SupersededToggleIdsTest.kt index 3a223b45..c055e4fb 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/cache/mutations/SupersededToggleIdsTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/cache/mutations/SupersededToggleIdsTest.kt @@ -143,7 +143,7 @@ class SupersededToggleIdsTest { ) // There is one preference per user, so any two queued changes to it - // collapse, and only the newest is sent (#4998). + // collapse, and only the newest is sent. @Test fun `only the newest queued normalization change survives`() { val rows = listOf( diff --git a/android/app/src/test/java/com/fabledsword/minstrel/cache/sync/SyncControllerTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/cache/sync/SyncControllerTest.kt new file mode 100644 index 00000000..5cd31ff4 --- /dev/null +++ b/android/app/src/test/java/com/fabledsword/minstrel/cache/sync/SyncControllerTest.kt @@ -0,0 +1,112 @@ +package com.fabledsword.minstrel.cache.sync + +import com.fabledsword.minstrel.api.endpoints.SyncApi +import com.fabledsword.minstrel.auth.AuthStore +import com.fabledsword.minstrel.cache.db.dao.CachedAlbumDao +import com.fabledsword.minstrel.cache.db.dao.CachedArtistDao +import com.fabledsword.minstrel.cache.db.dao.CachedTrackDao +import com.fabledsword.minstrel.cache.db.dao.SyncMetadataDao +import com.fabledsword.minstrel.cache.db.entities.SyncMetadataEntity +import com.fabledsword.minstrel.connectivity.NetworkStatusController +import com.fabledsword.minstrel.connectivity.ServerHealth +import com.fabledsword.minstrel.models.wire.SyncDeletesWire +import com.fabledsword.minstrel.models.wire.SyncResponseWire +import io.mockk.coEvery +import io.mockk.coVerify +import io.mockk.every +import io.mockk.mockk +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.test.runTest +import okhttp3.ResponseBody.Companion.toResponseBody +import org.junit.jupiter.api.Test +import retrofit2.Response +import retrofit2.Retrofit +import kotlin.test.assertEquals +import kotlin.test.assertNull + +/** + * The server returns at most 1000 changes a call. One page per trigger + * would leave the cache behind the change log, so deletes from duplicate + * merges would never arrive; a pass pulls pages until the server answers + * 204. + */ +class SyncControllerTest { + private val api: SyncApi = mockk() + private val trackDao: CachedTrackDao = mockk(relaxed = true) + private val retrofit: Retrofit = mockk { + every { create(SyncApi::class.java) } returns api + } + private val authStore: AuthStore = mockk { + every { sessionCookie } returns MutableStateFlow(null) + } + private val networkStatus: NetworkStatusController = mockk { + // Offline, so construction starts no pass; each test runs one itself. + every { state } returns MutableStateFlow(ServerHealth.Offline) + } + + // The cursor round-trips through the DAO, so it is held for real. + private var stored: SyncMetadataEntity? = null + private val metadataDao: SyncMetadataDao = mockk { + coEvery { get() } answers { stored } + coEvery { upsert(any()) } answers { stored = firstArg() } + } + + private fun page(cursor: Long, deletedTracks: List = emptyList()) = + Response.success(SyncResponseWire(cursor = cursor, deletes = SyncDeletesWire(track = deletedTracks))) + + private fun noContent(): Response = Response.success(204, null as SyncResponseWire?) + + @Test + fun `pages are pulled until the server has nothing newer`() = runTest { + coEvery { api.sync(0) } returns page(1000) + coEvery { api.sync(1000) } returns page(2000, deletedTracks = listOf("merged-away")) + coEvery { api.sync(2000) } returns noContent() + + val controller = SyncController( + metadataDao, mockk(relaxed = true), mockk(relaxed = true), + trackDao, authStore, backgroundScope, networkStatus, retrofit, + ) + controller.syncSafe() + + assertNull(controller.lastSyncError.value) + + coVerify(exactly = 1) { api.sync(2000) } + coVerify { trackDao.deleteByIds(listOf("merged-away")) } + assertEquals(2000L, stored?.cursor) + } + + @Test + fun `a cursor that does not advance stops the pass`() = runTest { + stored = SyncMetadataEntity(cursor = 500) + coEvery { api.sync(500) } returns page(500) + + val controller = SyncController( + metadataDao, mockk(relaxed = true), mockk(relaxed = true), + trackDao, authStore, backgroundScope, networkStatus, retrofit, + ) + controller.syncSafe() + + assertNull(controller.lastSyncError.value) + + coVerify(exactly = 1) { api.sync(500) } + } + + @Test + fun `a stale cursor resets to zero and keeps pulling`() = runTest { + stored = SyncMetadataEntity(cursor = 7) + coEvery { api.sync(7) } returns Response.error(410, "".toResponseBody()) + coEvery { api.sync(0) } returns page(1000) + coEvery { api.sync(1000) } returns noContent() + + val controller = SyncController( + metadataDao, mockk(relaxed = true), mockk(relaxed = true), + trackDao, authStore, backgroundScope, networkStatus, retrofit, + ) + controller.syncSafe() + + assertNull(controller.lastSyncError.value) + + coVerify(exactly = 1) { api.sync(1000) } + assertEquals(1000L, stored?.cursor) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/connectivity/ReachabilityMachineTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/connectivity/ReachabilityMachineTest.kt index d1f7fb68..953fb028 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/connectivity/ReachabilityMachineTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/connectivity/ReachabilityMachineTest.kt @@ -54,16 +54,15 @@ class ReachabilityMachineTest { val m = machine() m.onLinkChange(up = true) m.onOpFailure(nowMs = 1_000) - // Spacing matters as of #1209: these must be far enough apart to be - // separate evidence rather than one event's worth of fallout. This - // test previously used 1_500 — 500ms — which is now deliberately - // treated as a burst and does NOT corroborate. + // These must be far enough apart to be separate evidence rather + // than one event's worth of fallout: 500ms apart is a burst and + // does NOT corroborate. m.onOpFailure(nowMs = 1_000 + CORROBORATION_MIN_SPACING_MS) m.onProbeFailure(nowMs = 1_000 + CORROBORATION_MIN_SPACING_MS + 500) assertEquals(ServerHealth.ServerDown, m.health()) } - // The #1209 mechanism: an OS network handoff fails every in-flight request + // An OS network handoff fails every in-flight request // at once. That must NOT reach ServerDown, because ServerDown makes // OfflineGatedDataSource refuse uncached tracks outright — the app would // decline to play music that plays fine, for a blip already over. @@ -108,8 +107,8 @@ class ReachabilityMachineTest { val m = machine() m.onLinkChange(up = true) m.onOpFailure(nowMs = 0) - // Spaced so this test exercises STALENESS, not the burst rule — with - // 1_000 it would have passed for the wrong reason after #1209. + // Spaced so this test exercises STALENESS, not the burst rule — + // closer failures would pass it for the wrong reason. m.onOpFailure(nowMs = CORROBORATION_MIN_SPACING_MS) // both op failures are now older than the corroboration window: m.onProbeFailure(nowMs = CORROBORATION_MIN_SPACING_MS + CORROBORATION_WINDOW_MS + 1) diff --git a/android/app/src/test/java/com/fabledsword/minstrel/diagnostics/TransportFlapDetectorTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/diagnostics/TransportFlapDetectorTest.kt index 490e4615..b066b1d2 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/diagnostics/TransportFlapDetectorTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/diagnostics/TransportFlapDetectorTest.kt @@ -10,7 +10,7 @@ import kotlin.test.assertNull * The rule deciding when renderer transport changes are worth recording as an * episode. Worth pinning because both failure directions are costly: too eager * and every track transition writes a summary that buries the real one, too - * shy and the operator's stutter goes unmeasured for another month. + * shy and a real stutter goes unmeasured. */ class TransportFlapDetectorTest { diff --git a/android/app/src/test/java/com/fabledsword/minstrel/metadata/MetadataProviderTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/metadata/MetadataProviderTest.kt index e9eddc06..8894b089 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/metadata/MetadataProviderTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/metadata/MetadataProviderTest.kt @@ -13,7 +13,10 @@ import kotlinx.coroutines.flow.flowOf import kotlinx.coroutines.launch import kotlinx.coroutines.test.UnconfinedTestDispatcher import kotlinx.coroutines.test.runTest +import okhttp3.ResponseBody.Companion.toResponseBody import org.junit.jupiter.api.Test +import retrofit2.HttpException +import retrofit2.Response import java.util.concurrent.atomic.AtomicInteger import kotlin.test.assertTrue @@ -36,6 +39,30 @@ class MetadataProviderTest { coVerify(exactly = 1) { library.refreshAlbumDetail("a1") } } + // A row the server no longer has (a duplicate merge) is dropped, or the + // freshness sweep asks for it every 30 minutes forever. + @Test + fun `a 404 drops the cached row`() = runTest { + coEvery { library.refreshTrack("t1") } throws httpError(404) + val provider = MetadataProvider(albumDao, artistDao, trackDao, library, this) + + provider.refreshTrack("t1").join() + + coVerify(exactly = 1) { trackDao.deleteByIds(listOf("t1")) } + } + + @Test + fun `any other failure keeps the cached row`() = runTest { + coEvery { library.refreshAlbumDetail("a1") } throws httpError(500) + val provider = MetadataProvider(albumDao, artistDao, trackDao, library, this) + + provider.refreshAlbum("a1").join() + + coVerify(exactly = 0) { albumDao.deleteByIds(any()) } + } + + private fun httpError(status: Int) = HttpException(Response.error(status, "".toResponseBody())) + @Test fun `at most four fetches run concurrently`() = runTest { val scope = this diff --git a/android/app/src/test/java/com/fabledsword/minstrel/models/ArtistSuggestionReasonTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/models/ArtistSuggestionReasonTest.kt index 0000c507..e002c3e6 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/models/ArtistSuggestionReasonTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/models/ArtistSuggestionReasonTest.kt @@ -4,7 +4,7 @@ import org.junit.jupiter.api.Test import kotlin.test.assertEquals /** - * The card's subtitle line (#2377). Wording is kept in lockstep with the web + * The card's subtitle line. Wording is kept in lockstep with the web * client's reasonText() in SuggestionFeed.svelte — these assertions are the * record of what that wording IS, so a change on one client without the other * shows up as a failure rather than as silent divergence between the two @@ -60,7 +60,7 @@ class ArtistSuggestionReasonTest { ) } - // The COMMON case: most candidates have no cached tags (#2376), so the card + // The COMMON case: most candidates have no cached tags, so the card // must fall back to seed attribution rather than going blank. @Test fun `no matched tags falls back to seed attribution`() { diff --git a/android/app/src/test/java/com/fabledsword/minstrel/models/SuggestionSnoozeRefTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/models/SuggestionSnoozeRefTest.kt index 2dc598ec..d777af99 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/models/SuggestionSnoozeRefTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/models/SuggestionSnoozeRefTest.kt @@ -5,7 +5,7 @@ import org.junit.jupiter.api.Test import kotlin.test.assertEquals /** - * `returnsIn` phrasing for the parked-suggestions list (#2375). The clock is + * `returnsIn` phrasing for the parked-suggestions list. The clock is * injected rather than frozen, so these assertions are stable. */ class SuggestionSnoozeRefTest { diff --git a/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationSettingsRepositoryTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationSettingsRepositoryTest.kt index bd034ae1..4ec205c1 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationSettingsRepositoryTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationSettingsRepositoryTest.kt @@ -24,7 +24,7 @@ import java.io.IOException import kotlin.test.assertEquals import kotlin.test.assertFalse -/** Settings follow snippet #5107: shown at once, never overwritten or reordered by an older change. */ +/** Settings are offline-first: shown at once, never overwritten or reordered by an older change. */ class NotificationSettingsRepositoryTest { private val json = Json { ignoreUnknownKeys = true } private val api: NotificationsApi = mockk(relaxed = true) diff --git a/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationsRepositoryTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationsRepositoryTest.kt index cc3e7343..1b8807ca 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationsRepositoryTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/notifications/data/NotificationsRepositoryTest.kt @@ -24,7 +24,7 @@ import kotlin.test.assertEquals import kotlin.test.assertNull /** - * Reads are offline-first (rule 100): the device marks the row at once, and a + * Reads are offline-first: the device marks the row at once, and a * read the server has not taken is queued, except when the server says the * notice is gone. */ diff --git a/android/app/src/test/java/com/fabledsword/minstrel/notifications/delivery/DeliveryPlanTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/notifications/delivery/DeliveryPlanTest.kt index e4e2c3a7..2cb85222 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/notifications/delivery/DeliveryPlanTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/notifications/delivery/DeliveryPlanTest.kt @@ -99,6 +99,16 @@ class DeliveryPlanTest { assertEquals(5, pile.count) assertEquals(ShadeChannel.YOUR_REQUESTS, pile.channel) assertEquals("5 new notifications", pileText(pile.count)) + assertEquals(listOf("t-n5", "t-n4", "t-n3", "t-n2", "t-n1"), pile.lines) + assertNull(pileMore(pile)) + } + + @Test + fun `a pile names its newest notices and counts the rest`() { + val many = (1..17).map { notice("n$it", it.toLong()) } + val pile = assertIs(announcementFor(many)) + assertEquals(listOf("t-n17", "t-n16", "t-n15", "t-n14", "t-n13"), pile.lines) + assertEquals("and 12 more", pileMore(pile)) } @Test diff --git a/android/app/src/test/java/com/fabledsword/minstrel/notifications/ui/NotificationUiTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/notifications/ui/NotificationUiTest.kt index 763e13ba..f988ac13 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/notifications/ui/NotificationUiTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/notifications/ui/NotificationUiTest.kt @@ -23,9 +23,11 @@ class NotificationUiTest { assertEquals(ArtistDetail("ar-1"), routeForLink("/artists/ar-1")) assertEquals(Requests, routeForLink("/requests")) assertEquals(AdminRequests, routeForLink("/admin/requests")) - assertEquals(AdminQuarantine, routeForLink("/admin/quarantine")) - // Admin pages the app has no screen for open the Admin landing. - assertEquals(Admin, routeForLink("/admin/duplicates")) + assertEquals(AdminQuarantine, routeForLink("/collection/reports")) + // Library health pages the app has no screen for open the Admin landing. + assertEquals(Admin, routeForLink("/collection/duplicates")) + assertEquals(Admin, routeForLink("/collection/duplicates?view=resolved")) + assertEquals(Admin, routeForLink("/collection/missing")) assertEquals(Admin, routeForLink("/admin")) assertNull(routeForLink("/somewhere-new")) } diff --git a/android/app/src/test/java/com/fabledsword/minstrel/notifications/ui/NotificationsViewModelTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/notifications/ui/NotificationsViewModelTest.kt new file mode 100644 index 00000000..afa33564 --- /dev/null +++ b/android/app/src/test/java/com/fabledsword/minstrel/notifications/ui/NotificationsViewModelTest.kt @@ -0,0 +1,81 @@ +package com.fabledsword.minstrel.notifications.ui + +import com.fabledsword.minstrel.cache.db.entities.CachedNotificationEntity +import com.fabledsword.minstrel.connectivity.NetworkStatusController +import com.fabledsword.minstrel.connectivity.ServerHealth +import com.fabledsword.minstrel.notifications.data.NotificationsRepository +import com.fabledsword.minstrel.testutil.MainDispatcherExtension +import io.mockk.coVerify +import io.mockk.every +import io.mockk.mockk +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.emptyFlow +import kotlinx.datetime.Instant +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.extension.ExtendWith +import kotlin.test.assertEquals + +/** Showing the inbox reads it: unread notices are marked read once, and keep looking new until the screen goes. */ +@ExtendWith(MainDispatcherExtension::class) +class NotificationsViewModelTest { + private val repository: NotificationsRepository = mockk(relaxed = true) { + every { notifications } returns emptyFlow() + } + private val network: NetworkStatusController = mockk { + every { state } returns MutableStateFlow(ServerHealth.Healthy) + } + // Lazy: built inside each test, after the extension has swapped in the test Main dispatcher. + private val viewModel by lazy { NotificationsViewModel(repository, network) } + + private fun row(id: String, read: Boolean) = CachedNotificationEntity( + id = id, + kind = "request_completed", + title = "t", + body = "b", + link = "/requests", + createdAt = Instant.parse("2026-10-23T12:00:00Z"), + readAt = if (read) Instant.parse("2026-10-23T12:05:00Z") else null, + ) + + @Test + fun `showing unread notices marks them read and keeps them looking new`() { + viewModel.seen(listOf(row("a", read = false), row("b", read = true))) + + coVerify(exactly = 1) { repository.markAllRead() } + assertEquals(setOf("a"), viewModel.fresh.value) + } + + @Test + fun `the same notices shown again are not marked twice`() { + viewModel.seen(listOf(row("a", read = false))) + // The local mark lands and the cache re-emits; a re-emit before it lands repeats the row unread. + viewModel.seen(listOf(row("a", read = false))) + viewModel.seen(listOf(row("a", read = true))) + + coVerify(exactly = 1) { repository.markAllRead() } + } + + @Test + fun `a notice arriving while the screen is up is marked too`() { + viewModel.seen(listOf(row("a", read = false))) + viewModel.seen(listOf(row("new", read = false), row("a", read = true))) + + coVerify(exactly = 2) { repository.markAllRead() } + assertEquals(setOf("a", "new"), viewModel.fresh.value) + } + + @Test + fun `nothing unread marks nothing`() { + viewModel.seen(listOf(row("a", read = true))) + + coVerify(exactly = 0) { repository.markAllRead() } + } + + @Test + fun `leaving the screen stops them looking new`() { + viewModel.seen(listOf(row("a", read = false))) + viewModel.left() + + assertEquals(emptySet(), viewModel.fresh.value) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/DropUnavailableTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/DropUnavailableTest.kt index fd332b5a..fa27cb36 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/player/DropUnavailableTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/DropUnavailableTest.kt @@ -6,7 +6,7 @@ import kotlin.test.assertEquals import kotlin.test.assertTrue /** - * The queue filter for #2704. The index arithmetic is the part worth pinning: + * The queue filter for server-missing tracks. The index arithmetic is the part worth pinning: * getting it wrong starts playback on the wrong track, which is a subtler and * more annoying bug than the one being fixed. */ diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/RemotePlayerStateTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/RemotePlayerStateTest.kt index b597187a..e444e0b7 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/player/RemotePlayerStateTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/RemotePlayerStateTest.kt @@ -74,7 +74,7 @@ class RemotePlayerStateTest { val state = RemotePlayerState() state.setPlayIntent(true) state.applyError(RuntimeException("connect timeout")) - // isPlaying is cleared by the error, but the operator's intent persists. + // isPlaying is cleared by the error, but the user's intent persists. assertFalse(state.isPlaying) assertTrue(state.lastPlayIntent) } diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/output/SingleTrackQueueTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/output/SingleTrackQueueTest.kt new file mode 100644 index 00000000..c690bb43 --- /dev/null +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/output/SingleTrackQueueTest.kt @@ -0,0 +1,65 @@ +package com.fabledsword.minstrel.player.output + +import com.fabledsword.minstrel.models.TrackRef +import com.fabledsword.minstrel.player.output.upnp.UpnpRoute +import okhttp3.HttpUrl.Companion.toHttpUrl +import org.junit.jupiter.api.Test +import kotlin.test.assertEquals +import kotlin.test.assertNull + +/** + * The pieces that let a renderer without a queue be driven from ours: which + * renderers get which mode, and mapping the URI it reports back to a queue + * position. + */ +class SingleTrackQueueTest { + + private fun t(id: String) = TrackRef(id = id, title = "", albumId = "", artistId = "") + + private fun route(manufacturer: String, zgt: Boolean = false) = UpnpRoute( + id = "uuid:x", + name = "Room", + manufacturer = manufacturer, + modelName = "", + avTransportControlUrl = "http://192.168.0.2:1400/AVTransport/Control".toHttpUrl(), + renderingControlUrl = null, + zoneGroupTopologyControlUrl = if (zgt) "http://192.168.0.2:1400/ZGT/Control".toHttpUrl() else null, + ) + + @Test + fun `a Sonos is driven through its native queue`() { + assertEquals(QueueMode.NATIVE_QUEUE, QueueMode.forRoute(route("Sonos, Inc.").isSonos)) + assertEquals(QueueMode.NATIVE_QUEUE, QueueMode.forRoute(route("", zgt = true).isSonos)) + } + + @Test + fun `any other renderer is driven one track at a time`() { + assertEquals(QueueMode.SINGLE_TRACK, QueueMode.forRoute(route("Yamaha Corporation").isSonos)) + assertEquals(QueueMode.SINGLE_TRACK, QueueMode.forRoute(route("").isSonos)) + } + + @Test + fun `the track id is read from a stream URL, leveled or not`() { + assertEquals("abc", streamUriTrackId("http://h:8080/api/tracks/abc/stream.flac?t=tok")) + assertEquals("abc", streamUriTrackId("http://h/api/tracks/abc/leveled.flac?t=tok")) + } + + @Test + fun `a URL that is not ours has no track id`() { + assertNull(streamUriTrackId("x-rincon-queue:RINCON_ABC#0")) + assertNull(streamUriTrackId("")) + assertNull(streamUriTrackId("http://h/api/tracks/")) + } + + @Test + fun `a track queued twice resolves to the copy nearest the cursor`() { + val queue = listOf(t("a"), t("b"), t("a"), t("c"), t("a")) + assertEquals(2, nearestIndexOf(queue, "a", preferNear = 3)) + assertEquals(0, nearestIndexOf(queue, "a", preferNear = 0)) + } + + @Test + fun `a track not in the queue has no index`() { + assertNull(nearestIndexOf(listOf(t("a")), "z", preferNear = 0)) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/output/SonosAlbumPlayTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/output/SonosAlbumPlayTest.kt index d5e28f69..6d4e67e7 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/player/output/SonosAlbumPlayTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/output/SonosAlbumPlayTest.kt @@ -6,7 +6,7 @@ import kotlin.test.assertEquals /** * The Sonos queue plays straight through, so a track's album play is judged - * from its neighbours in the list sent (M464 #5002). + * from its neighbours in the list sent. */ class SonosAlbumPlayTest { diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/output/TrackEndDetectorTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/output/TrackEndDetectorTest.kt new file mode 100644 index 00000000..a14b73f1 --- /dev/null +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/output/TrackEndDetectorTest.kt @@ -0,0 +1,68 @@ +package com.fabledsword.minstrel.player.output + +import com.fabledsword.minstrel.player.output.upnp.TransportState +import org.junit.jupiter.api.Test +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** + * A one-track renderer says STOPPED both when a track plays out and when it + * stops partway; only the first should start the next track. + */ +class TrackEndDetectorTest { + + private val d = TrackEndDetector() + + private fun poll(state: TransportState, positionMs: Long, durationMs: Long = 200_000L, known: Long = 0L) = + d.onPoll(state, positionMs, durationMs, known) + + @Test + fun `a stop just after the last playing reading near the end is an ending`() { + poll(TransportState.PLAYING, 198_000L) + assertTrue(poll(TransportState.STOPPED, 0L)) + } + + @Test + fun `a stop partway through is not an ending`() { + poll(TransportState.PLAYING, 90_000L) + assertFalse(poll(TransportState.STOPPED, 90_000L)) + } + + @Test + fun `the stop a renderer passes through while loading a track is not an ending`() { + // Short track, never seen playing yet: without the playing guard, + // position 0 would already be within the slack of the end. + assertFalse(poll(TransportState.STOPPED, 0L, durationMs = 3_000L)) + } + + @Test + fun `no media present after a track plays out is an ending`() { + poll(TransportState.PLAYING, 199_500L) + assertTrue(poll(TransportState.NO_MEDIA, 0L, durationMs = 0L)) + } + + @Test + fun `a pause near the end is not an ending`() { + poll(TransportState.PLAYING, 199_000L) + assertFalse(poll(TransportState.PAUSED, 199_000L)) + } + + @Test + fun `the library duration stands in for a renderer that reports none`() { + poll(TransportState.PLAYING, 178_000L, durationMs = 0L, known = 180_000L) + assertTrue(poll(TransportState.STOPPED, 0L, durationMs = 0L, known = 180_000L)) + } + + @Test + fun `with no duration from anywhere nothing is judged an ending`() { + poll(TransportState.PLAYING, 5_000L, durationMs = 0L) + assertFalse(poll(TransportState.STOPPED, 0L, durationMs = 0L)) + } + + @Test + fun `reset forgets the track that was playing`() { + poll(TransportState.PLAYING, 199_000L) + d.reset() + assertFalse(poll(TransportState.STOPPED, 0L)) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/AVTransportClientTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/AVTransportClientTest.kt index 48919b60..44fcd435 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/AVTransportClientTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/AVTransportClientTest.kt @@ -21,7 +21,7 @@ class AVTransportClientTest { server = MockWebServer() server.start() val controlUrl = server.url("/MediaRenderer/AVTransport/Control") - client = AVTransportClient(SoapClient(OkHttpClient()), controlUrl) + client = AVTransportClient(SoapClient(OkHttpClient()), controlUrl, sonos = true) } @AfterEach @@ -229,6 +229,45 @@ class AVTransportClientTest { assertEquals(0, client.getMediaInfo().nrTracks) } + @Test + fun `setNextAVTransportURI sends NextURI with its metadata`() = runTest { + server.enqueue(emptyResponse("SetNextAVTransportURI")) + client.setNextAVTransportURI( + uri = "http://example.com/api/tracks/t2/stream.flac", + mime = "audio/flac", + title = "Next", + ) + val body = server.takeRequest().body.readUtf8() + assertTrue(body.contains("SetNextAVTransportURI")) { "body: $body" } + assertTrue(body.contains("http://example.com/api/tracks/t2/stream.flac")) { + "body: $body" + } + assertTrue(body.contains("")) { "body: $body" } + } + + @Test + fun `a Sonos gets the Rincon metadata it requires`() = runTest { + server.enqueue(emptyResponse("SetAVTransportURI")) + client.setAVTransportURIWithMetadata("http://x/a.flac", "audio/flac", "Song") + val body = server.takeRequest().body.readUtf8() + assertTrue(body.contains("RINCON_AssociatedZPUDN")) { "body: $body" } + } + + @Test + fun `a generic renderer gets plain DIDL that declares the stream seekable`() = runTest { + val generic = AVTransportClient( + SoapClient(OkHttpClient()), + server.url("/MediaRenderer/AVTransport/Control"), + sonos = false, + ) + server.enqueue(emptyResponse("SetAVTransportURI")) + generic.setAVTransportURIWithMetadata("http://x/a.flac", "audio/flac", "Song") + val body = server.takeRequest().body.readUtf8() + assertFalse(body.contains("RINCON")) { "body: $body" } + assertFalse(body.contains("rinconnetworks")) { "body: $body" } + assertTrue(body.contains("audio/flac:DLNA.ORG_OP=01")) { "body: $body" } + } + private fun emptyResponse(action: String): MockResponse = MockResponse().setBody( """ diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/SoapClientTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/SoapClientTest.kt index ad052e72..d0decd5e 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/SoapClientTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/output/upnp/SoapClientTest.kt @@ -16,7 +16,7 @@ import kotlin.test.assertTrue * outgoing SOAP envelope + verify the SOAPACTION header shape, and * to drive the `` response path. * - * kxml2 is on the test classpath (Task 4), so + * kxml2 is on the test classpath, so * `XmlPullParserFactory.newInstance()` resolves on the JVM. */ class SoapClientTest { diff --git a/android/app/src/test/java/com/fabledsword/minstrel/player/ui/QueueDragMathTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/player/ui/QueueDragMathTest.kt index db08bd35..deeb9c10 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/player/ui/QueueDragMathTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/player/ui/QueueDragMathTest.kt @@ -4,7 +4,7 @@ import org.junit.jupiter.api.Test import kotlin.test.assertEquals /** - * The queue drag math (#2436). The delta cases mirror web's + * The queue drag math. The delta cases mirror web's * queue-row-math.test.ts one for one, so a change to either client's * rounding shows up as a disagreement here rather than as a drop that lands * one row off on one platform only. diff --git a/android/app/src/test/java/com/fabledsword/minstrel/playlists/data/PlaylistTrackAvailabilityTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/playlists/data/PlaylistTrackAvailabilityTest.kt index 318e7e05..ae19726f 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/playlists/data/PlaylistTrackAvailabilityTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/playlists/data/PlaylistTrackAvailabilityTest.kt @@ -9,7 +9,7 @@ import kotlin.test.assertTrue /** * A playlist row can outlive its audio two ways — the track was removed * from the library, or the track is still there but its file is missing - * (#2527). Both must be unplayable, and neither may be silently dropped + *. Both must be unplayable, and neither may be silently dropped * from the list the user curated. */ class PlaylistTrackAvailabilityTest { diff --git a/android/app/src/test/java/com/fabledsword/minstrel/settings/data/NormalizationRepositoryTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/settings/data/NormalizationRepositoryTest.kt index 6853887d..f47f0160 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/settings/data/NormalizationRepositoryTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/settings/data/NormalizationRepositoryTest.kt @@ -17,7 +17,7 @@ import retrofit2.Retrofit import java.io.IOException /** - * The normalization preference is written offline-first (rule 100): the + * The normalization preference is written offline-first: the * device shows the change at once, and a change the server has not seen * yet is never overwritten or reordered. */ diff --git a/android/app/src/test/java/com/fabledsword/minstrel/shared/UpdateVeilControllerTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/shared/UpdateVeilControllerTest.kt index 58c98cf6..733fc8e4 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/shared/UpdateVeilControllerTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/shared/UpdateVeilControllerTest.kt @@ -30,7 +30,7 @@ private const val DRAIN_MS = 3_000L /** * The veil's job is to go up only when content actually changes, and then to * stay up until the screen has stopped moving. Each test pins one of the ways - * the original fixed-delay implementation got that wrong (issue #2327). + * a fixed-delay veil gets that wrong. * * The controller is built on `backgroundScope` throughout: its consumer * loop runs forever, so hanging it off the test's own scope would stop @@ -40,8 +40,7 @@ private const val DRAIN_MS = 3_000L * `advanceTimeBy`: **`advanceUntilIdle()` is useless here.** It advances * only while *foreground* work remains, and everything this controller * does lives in `backgroundScope` — so it returns having run nothing, and - * assertions land on a session that never started (CI run 3163 failed all - * seven of these with "expected 3, actual 0" and friends). Drive the clock + * assertions land on a session that never started. Drive the clock * deliberately instead; don't "simplify" these back to advanceUntilIdle. */ @OptIn(ExperimentalCoroutinesApi::class) diff --git a/android/app/src/test/java/com/fabledsword/minstrel/theme/BundledFontsTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/theme/BundledFontsTest.kt index e061ba24..20ad44d3 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/theme/BundledFontsTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/theme/BundledFontsTest.kt @@ -9,11 +9,11 @@ import kotlin.test.assertTrue * Guards that the typefaces ship inside the APK instead of being fetched at * runtime. * - * Until 2026-09-09 these were resolved through the Play Services font - * provider. That needs a network the deployed app is not guaranteed, and a - * provider that devices without Play Services do not have at all. Both - * failures are silent — text just renders in the platform default, which - * reads as a styling regression rather than a missing dependency. + * Resolving them through the Play Services font provider needs a network + * the deployed app is not guaranteed, and a provider that devices without + * Play Services do not have at all. Both failures are silent — text just + * renders in the platform default, which reads as a styling regression + * rather than a missing dependency. * * Expectations are read out of Typography.kt itself rather than hardcoded, so * this cannot drift away from what the app actually declares: adding a face diff --git a/android/app/src/test/java/com/fabledsword/minstrel/update/data/AfterAskingTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/update/data/AfterAskingTest.kt new file mode 100644 index 00000000..24cc613e --- /dev/null +++ b/android/app/src/test/java/com/fabledsword/minstrel/update/data/AfterAskingTest.kt @@ -0,0 +1,28 @@ +package com.fabledsword.minstrel.update.data + +import org.junit.jupiter.api.Test +import kotlin.test.assertEquals + +/** Back from the "install unknown apps" page, an update that was asked for carries on. */ +class AfterAskingTest { + @Test + fun `allowed now, the update carries on`() { + assertEquals(AfterAsking.FETCH, afterAsking(asked = true, mayInstall = true, offered = true, busy = false)) + } + + @Test + fun `back without allowing it, the listener is told and the offer stays`() { + assertEquals(AfterAsking.REFUSED, afterAsking(asked = true, mayInstall = false, offered = true, busy = false)) + } + + @Test + fun `nothing asked, nothing happens`() { + assertEquals(AfterAsking.WAIT, afterAsking(asked = false, mayInstall = true, offered = true, busy = false)) + } + + @Test + fun `an offer not known yet, or an install under way, waits`() { + assertEquals(AfterAsking.WAIT, afterAsking(asked = true, mayInstall = true, offered = false, busy = false)) + assertEquals(AfterAsking.WAIT, afterAsking(asked = true, mayInstall = true, offered = true, busy = true)) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateRepositoryTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateRepositoryTest.kt new file mode 100644 index 00000000..1e295e79 --- /dev/null +++ b/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateRepositoryTest.kt @@ -0,0 +1,120 @@ +package com.fabledsword.minstrel.update.data + +import com.fabledsword.minstrel.models.wire.UpdateInfoWire +import com.fabledsword.minstrel.update.api.ClientVersionApi +import kotlinx.coroutines.test.runTest +import okhttp3.ResponseBody.Companion.toResponseBody +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import retrofit2.HttpException +import retrofit2.Response +import java.io.File +import java.io.IOException +import java.security.MessageDigest +import kotlin.test.assertEquals +import kotlin.test.assertIs +import kotlin.test.assertNotNull +import kotlin.test.assertNull +import kotlin.test.assertTrue + +/** + * The fetch worker and the listener's Update both download through here; a + * package only counts as ready once it is the whole build that was offered. + */ +class UpdateRepositoryTest { + @TempDir + lateinit var dir: File + + private val apkBytes = "the newer app".toByteArray() + private val sha = MessageDigest.getInstance("SHA-256").digest(apkBytes).joinToString("") { "%02x".format(it) } + + private fun wire(code: Long? = 200L, version: String = "2026.10.26.1200", sha256: String = sha) = UpdateInfoWire( + version = version, + code = code, + sizeBytes = apkBytes.size.toLong(), + sha256 = sha256, + ) + + private var downloads = 0 + + private fun repo( + version: () -> UpdateInfoWire = { wire() }, + body: () -> ByteArray = { apkBytes }, + offers: Boolean = true, + ) = UpdateRepository( + api = object : ClientVersionApi { + override suspend fun getVersion(): UpdateInfoWire = version() + }, + apk = ApkSource { + downloads += 1 + body().toResponseBody() + }, + dir = dir, + installedCode = 100L, + installedName = "2026.10.20.0900", + offers = offers, + ) + + @Test + fun `a newer build is offered`() = runTest { + assertIs(repo().check()) + } + + @Test + fun `the same app built again at a later minute is not offered`() = runTest { + assertEquals(Offer.Current, repo(version = { wire(version = "2026.10.20.0900") }).check()) + } + + @Test + fun `a server with no app to offer is not a problem`() = runTest { + val none = repo(version = { throw HttpException(Response.error(404, "".toResponseBody())) }) + assertEquals(Offer.Current, none.check()) + } + + @Test + fun `a server out of reach is a problem, not an all-clear`() = runTest { + assertIs(repo(version = { throw IOException("unreachable") }).check()) + } + + @Test + fun `a debug build is never offered one`() = runTest { + assertEquals(Offer.Current, repo(offers = false).check()) + } + + @Test + fun `a build already fetched is installed from disk without a second download`() = runTest { + val r = repo() + val info = (r.check() as Offer.Newer).info + assertIs(r.download(info) {}) + assertNotNull(r.ready(info)) + val again = r.download(info) {} + assertIs(again) + assertEquals(1, downloads) + } + + @Test + fun `a download cut off partway is not taken for the build`() = runTest { + val r = repo(body = { apkBytes.copyOf(5) }) + val info = (r.check() as Offer.Newer).info + assertIs(r.download(info) {}) + assertNull(r.ready(info)) + assertTrue(dir.listFiles().orEmpty().isEmpty(), "the part file is removed") + } + + @Test + fun `a download that does not match the digest is not taken for the build`() = runTest { + val r = repo(body = { "the other app".toByteArray() }) // same length, other bytes + val info = (r.check() as Offer.Newer).info + assertIs(r.download(info) {}) + assertNull(r.ready(info)) + } + + @Test + fun `tidy drops a package kept for a build no longer offered`() = runTest { + val r = repo() + val info = (r.check() as Offer.Newer).info + r.download(info) {} + r.tidy() + assertNull(r.ready(info)) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateToldTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateToldTest.kt new file mode 100644 index 00000000..226ce9ca --- /dev/null +++ b/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateToldTest.kt @@ -0,0 +1,53 @@ +package com.fabledsword.minstrel.update.data + +import org.junit.jupiter.api.Test +import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** Home tells of each build at most once a day, and says whether it is ready. */ +class UpdateToldTest { + private val day = SAY_AGAIN_AFTER_MS + private val t0 = 1_000_000_000L + + @Test + fun `a build not yet told of is told at once`() { + assertTrue(dueToSay("3523900", toldBuild = "3523847", toldAt = t0, now = t0 + 1)) + assertTrue(dueToSay("3523900", toldBuild = "", toldAt = 0L, now = t0)) + } + + @Test + fun `the same build is not told again within a day`() { + assertFalse(dueToSay("3523900", toldBuild = "3523900", toldAt = t0, now = t0 + day - 1)) + } + + @Test + fun `the same build is told again the day after`() { + assertTrue(dueToSay("3523900", toldBuild = "3523900", toldAt = t0, now = t0 + day)) + } + + @Test + fun `a clock set back tells rather than hiding the update`() { + assertTrue(dueToSay("3523900", toldBuild = "3523900", toldAt = t0, now = t0 - 1)) + } + + @Test + fun `a build is keyed by its code, or by its name from a server with none`() { + assertEquals("3523900", buildKey(3523900L, "2026.09.10.1432")) + assertEquals("2026.09.10.1432", buildKey(null, "2026.09.10.1432")) + } + + @Test + fun `the snackbar says whether the update is already here`() { + assertEquals( + "Version 2026.10.26.1200 of the app is on your server.", + toldWords("2026.10.26.1200", ready = false), + ) + assertEquals( + "Version 2026.10.26.1200 of the app is downloaded and ready.", + toldWords("2026.10.26.1200", ready = true), + ) + assertEquals("Update", toldAction(ready = false)) + assertEquals("Install", toldAction(ready = true)) + } +} diff --git a/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateVersioningTest.kt b/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateVersioningTest.kt index 130c8b6a..442222d2 100644 --- a/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateVersioningTest.kt +++ b/android/app/src/test/java/com/fabledsword/minstrel/update/data/UpdateVersioningTest.kt @@ -62,6 +62,21 @@ class UpdateVersioningTest { ) } + /** + * Every server image builds the app again at a later minute, so a CI + * re-run, or a change outside the shipped set, raises the code while the + * app is the same one. The name says so, and it is not offered. + */ + @Test + fun `the same app built again at a later minute is not offered`() { + assertFalse( + isUpdateAvailable( + serverCode = 3523900, serverName = "2026.09.10.1432", + installedCode = 3523847, installedName = "2026.09.10.1432", + ), + ) + } + @Test fun `falls back to the name when the server reports no ordering key`() { assertTrue( diff --git a/android/gradle/libs.versions.toml b/android/gradle/libs.versions.toml index a551209c..1f6a799d 100644 --- a/android/gradle/libs.versions.toml +++ b/android/gradle/libs.versions.toml @@ -5,7 +5,7 @@ # - Kotlin 2.3.x: AGP 9's built-in Kotlin path; the older kotlin-android # plugin can't cast AGP 9's ApplicationExtension to the removed # BaseExtension, and KSP 1.x isn't compatible with built-in Kotlin -# - KSP 2.3.x (PR #2674, merged Oct 2025): adds AGP 9 built-in Kotlin +# - KSP 2.3.x: adds AGP 9 built-in Kotlin # support, so we don't need the kotlin-android plugin at all agp = "9.0.1" kotlin = "2.3.21" diff --git a/ci-requirements.md b/ci-requirements.md index d933be14..d5418f6f 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -5,27 +5,20 @@ ## Runtime images -Minstrel's four workflows consume two CI images: +Minstrel's one workflow, `.gitea/workflows/release.yml`, consumes two CI images: ``` git.fabledsword.com/bvandeusen/ci-go:1.26 git.fabledsword.com/bvandeusen/ci-android:36 ``` -- `ci-go:1.26` — the `go`, `integration` and `web` lanes, `release-assets`, and the container build (`image-release`). All CI lives in one workflow, `.gitea/workflows/release.yml`, so every publishing job can depend on every lane (rule 177). +- `ci-go:1.26` — the `go`, `integration` and `web` lanes, `release-assets`, and the container build (`image-release`). All CI lives in that one workflow so every publishing job can depend on every lane. - `golang:1.26-bookworm` (Docker Hub) — the `govulncheck` lane. Deliberately the same image as the Dockerfile's builder stage rather than `ci-go`, so the standard library it checks is the one in the shipped binary. Keep the two in step. -- `ci-android:36` — native Kotlin/Compose client: ktlint + detekt + unit tests + debug APK (the `android` lane), and the signed release APK (`android-release`). +- `ci-android:36` — native Kotlin/Compose client: ktlint + detekt + unit tests (the `android` lane), and the signed release APK (`android-release`). -**`ci-flutter` is no longer consumed, and `ci-flutter` can now be retired.** -The M8 rewrite replaced the Flutter client with the native Android app and -`flutter.yml` was removed; `ci-android` took its place. `flutter_client/` -itself was deleted on 2026-08-16, which was the condition CI-Runner was -waiting on before dropping the image — nothing in this repo needs a Flutter -toolchain any more. - -Note this does **not** mean the `flutter-ci` runner *label* goes: the Android -jobs still schedule on it while pulling `ci-android:36`, per the label/image -split below. The label is a scheduling handle, not a toolchain assertion. +**No job uses `ci-flutter`**; nothing in this repo needs a Flutter toolchain. +The `flutter-ci` runner *label* is still used: the Android jobs schedule on it +while pulling `ci-android:36`, per the label/image split below. ## Image deps used @@ -43,8 +36,7 @@ split below. The label is a scheduling handle, not a toolchain assertion. error. The workflows also set `JAVA_TOOL_OPTIONS=--enable-native-access=ALL-UNNAMED` to silence Gradle's launcher-JVM restricted-method warning. - **Android SDK + cmdline-tools + build-tools 36.0.0** — APK assembly + signing. - No NDK: the native client has no C/C++ sources (this is why it isn't on - `ci-flutter`). + No NDK: the app has no C/C++ sources. - **ktlint + detekt** — `./gradlew ktlintCheck` and `./gradlew detekt` in the `android` lane. Image pins track `android/gradle/libs.versions.toml` so local and CI checks agree. @@ -61,32 +53,25 @@ None. - **Label/image split.** Workflows keep `runs-on: go-ci` / `runs-on: flutter-ci` as the scheduling label per the [`ci-runners.md`](https://…/FabledRulebook/ci-runners.md) "label = scheduling handle, image = `container.image`" pattern. The labels are intentional handles, not toolchain assertions — which is why the Android jobs still schedule on `flutter-ci` while pulling `ci-android:36`. Switch them to `android-ci` if that runner label is ever registered; nothing breaks either way. - **Integration-job docker-socket dependency.** The `integration` lane uses the runner's shared docker socket (`/var/run/docker.sock`) to bridge-IP-discover the per-job Postgres service container by name + network intersection — the dev compose's `minstrel-postgres-*` containers are explicitly skipped as belt-and-suspenders. Depends on `act_runner.valid_volumes` whitelisting the socket; if that ever stops auto-mounting, integration tests fail at the `docker inspect` step. - **Go toolchain pin.** `go.mod` is on `go 1.25.0` because `golang.org/x/crypto v0.51.0` declares 1.25 as its minimum. `ci-go:1.26` satisfies this with headroom. Future `x/crypto` bumps that move the Go floor should be paired with an image-tag bump in this file + the workflows. -- **In-app update channel — `needs:`, not polling.** `release.yml`'s `image-release` job declares `needs: [android-release]`, so on tag pushes the signed APK is guaranteed present before the image build starts — no polling window, no race. (The old cross-workflow polling against `flutter.yml` is gone with that workflow.) On non-tag `main` pushes `android-release` is skipped and `image-release` instead pulls the most recent release's APK and reconstructs its exact `versionName`, so `:latest` never ships without an update channel. It degrades to an empty `client/` — never a wrong version — if no release, asset, or tag commit-count can be resolved. -- **Cache server reachability.** The `web` lane does NOT use `cache: 'npm'` on `actions/setup-node` — the Gitea Actions cache server isn't reachable from this runner's container network and `setup-node` was burning ~4m41s on ETIMEDOUT before failing open. With the migration to `ci-go:1.26`, `setup-node` is removed entirely (Node is in the image). The cache concern reappears if a future change re-introduces a network-dependent action. +- **In-app update channel — `needs:`, not polling.** `release.yml`'s `image-release` job declares `needs: [android-release]`, so on dev and tag pushes the signed APK is guaranteed present before the image build starts — no polling window, no race. On `main` pushes `android-release` is skipped and `image-release` instead pulls the most recent release's APK and the version sidecar recorded beside it, so `:latest` never ships without an update channel. It degrades to an empty `client/` — never a wrong version — if no release or APK asset can be resolved. +- **Cache server reachability.** The `web` lane does NOT use `cache: 'npm'` on `actions/setup-node` — the Gitea Actions cache server isn't reachable from this runner's container network, so `setup-node` would spend minutes on ETIMEDOUT before failing open. `setup-node` is not used at all (Node is in the image). The cache concern reappears if a change introduces a network-dependent action. - **Artifacts — stock `actions/upload-artifact@v7` and `actions/download-artifact@v8`; never `@v3`.** ```yaml uses: actions/upload-artifact@v7 uses: actions/download-artifact@v8 ``` - Stock works on this forge since the runner moved to gitea/runner 3.x, which - edits the actions' client-side `isGhes()` refusal out of their bundles. Proven - on 2026-09-10 for upload v4–v7 and download v4–v8 (Scribe spike #3843). Until - then this repo pinned SHA mirrors of the Forgejo project's forks, because - upstream threw on the hostname before it opened a connection (Scribe 2255). + Stock works on this forge because gitea/runner 3.x edits the actions' + client-side `isGhes()` refusal out of their bundles; upload v4–v7 and + download v4–v8 are proven. `@v3` is still broken: it reports success, and Gitea serves artifacts back only through the v4 API (`content_encoding = application/zip`), so a v3 upload is - stored but invisible to every retrieval path. That is how 72 unreachable - artifacts accumulated on this repo (Scribe 2270). + stored but invisible to every retrieval path. - **Pairing no longer needs managing.** This entry used to pin upload v5 against - download v6 so both bundled `@actions/artifact` ^4.0.0, warning that a mismatch - across `release.yml`'s producer/consumer pair would list empty. Tested, and not - true on this instance: every download major v4–v8 read the artifacts of every - upload major v4–v7, by name and by pattern (CI-runner run 6312). The only real - protocol break is v3 → v4. node24 is no longer a concern either — every - CI-runner image carries Node 24 and the runner runs actions with the image's - `node`. + **The upload and download majors need not match.** Every download major + v4–v8 reads the artifacts of every upload major v4–v7, by name and by + pattern; the only protocol break is v3 → v4. Every CI-runner image carries + Node 24, and the runner runs actions with the image's `node`. Upload steps set `if-no-files-found: error` rather than the default `warn`, so an upload that matches nothing fails its own job instead of failing the diff --git a/ci/version.sh b/ci/version.sh index b12a13d0..fb7f0ae8 100755 --- a/ci/version.sh +++ b/ci/version.sh @@ -3,11 +3,17 @@ # Derives the three values a build is stamped with, and the tag that names it. # # name=YYYY.MM.DD.HHMM label for people, from the timestamp of the newest -# commit that CHANGED SOMETHING SHIPPED (see SHIPPED) +# commit that CHANGED SOMETHING SHIPPED (see SHIPPED, +# or SHIPPED_APK with --apk) # code= ordering key, minutes since 2020-01-01 at BUILD time # tag=v what a release of this commit must be called # -# Usage: ci/version.sh [] (default HEAD) +# Usage: ci/version.sh [--apk] [] (default HEAD) +# +# --apk names the Android app from android/ alone. The app compares this +# name with the one it is running, and a different name is what +# offers an update — so a name that moved for a server-only commit +# would offer every phone a rebuild of the app it already has. # # This exists as a script rather than inline workflow YAML for one reason: # release.yml only runs on `main` and on tags, so anything living inside it is @@ -30,6 +36,12 @@ set -euo pipefail readonly EPOCH_2020=1577836800 # 2020-01-01T00:00:00Z + +apk=false +if [ "${1:-}" = "--apk" ]; then + apk=true + shift +fi readonly REF="${1:-HEAD}" # Both clocks are overridable so a test can pin them. Nothing but tests should @@ -48,14 +60,17 @@ readonly REF="${1:-HEAD}" # wrongly included → version moves when nothing shipped. Cosmetic noise in # a string nobody sorts. # -# THIS REPO SHIPS TWO ARTIFACTS FROM ONE DERIVATION, and that is why the list -# is shorter than it looks like it should be. The server image ships cmd/, -# internal/, shared/, web/, config.example.yaml and client/; the APK ships -# android/. Neither ships the other's sources — but excluding android/ here -# would stop an Android-only commit from moving the APK's OWN version, which -# is the dangerous direction. So this is the union: exclude only what ships in -# NEITHER, and accept that an Android commit also nudges the server's reported -# version. Over-inclusion across the two, which is the harmless direction. +# THIS LIST NAMES THE SERVER IMAGE, and it is shorter than it looks like it +# should be. The image ships cmd/, internal/, shared/, web/, +# config.example.yaml and client/ — and client/ holds the APK, so android/ +# stays in: an Android-only commit changes what the image carries. Exclude +# only what ships in neither artifact; over-inclusion is the harmless +# direction here. +# +# The APK itself is named from SHIPPED_APK (below), where over-inclusion is +# NOT harmless: the app offers an update whenever the name differs from its +# own, so a name moved by a server-only commit offers every phone a rebuild +# of the app it already has. # # The family's other repos (roundtable / roundtable-android) each keep a # tighter list because they are separate repos with one artifact apiece. Do @@ -100,9 +115,24 @@ readonly SHIPPED=( ':!web/vitest.setup.ts' ) +# What the APK is built from: android/ and nothing else (the Gradle build +# reads no path outside it). Still a denylist within it, for the same reason +# as above — a new directory under android/ counts until named here. +readonly SHIPPED_APK=( + android + ':!android/app/src/test' # JVM unit tests + ':!android/*.md' +) + +if [ "${apk}" = true ]; then + shipped=("${SHIPPED_APK[@]}") +else + shipped=("${SHIPPED[@]}") +fi + commit_epoch="${MINSTREL_COMMIT_EPOCH:-}" if [ -z "${commit_epoch}" ]; then - commit_epoch="$(git log --format=%ct -1 "${REF}" -- "${SHIPPED[@]}")" + commit_epoch="$(git log --format=%ct -1 "${REF}" -- "${shipped[@]}")" # Loudly, on purpose. A silent fallback here is the landmine this whole # script exists to avoid: a plausible-looking version that is quietly wrong, # on a green run. Realistically this means a shallow clone (no commit in diff --git a/client/README.md b/client/README.md index 66fe8e9b..165b4db6 100644 --- a/client/README.md +++ b/client/README.md @@ -1,45 +1,58 @@ # Bundled client assets -Holds the Android APK (`minstrel.apk` + `minstrel.apk.version`) that the -server serves via `/api/client/version` and `/api/client/apk` for the -in-app update flow (#397). +Holds the Android APK (`minstrel.apk`) and its version sidecar +(`minstrel.apk.version`) that the server serves via `/api/client/version` +and `/api/client/apk` for the in-app update flow. ## Production -CI populates this directory in `release.yml` (no Flutter — the client is -native Android, built by the `android-release` job): +CI fills this directory in `release.yml` before every image build: -- **Tag release (`v*`):** the `android-release` job builds + signs the - APK, uploads it as a workflow artifact, and attaches it to the Gitea - release as `minstrel-.apk`. The `image-release` job - (`needs: android-release`) downloads that artifact into this directory - as `minstrel.apk` and writes the computed versionName to - `minstrel.apk.version`. -- **Main push (non-tag `:latest` / `:main`):** no APK is built, but the - image still moves `:latest`, so `image-release` pulls the most-recent - release's signed APK from the Gitea API and reconstructs its exact - versionName — keeping the in-app update channel alive on `:latest` - instead of stripping it until the next tag. +- **Dev push (`:dev`) and tag release (`v*`):** the `android-release` job + builds and signs the APK and uploads it as a workflow artifact; + `image-release` downloads it here as `minstrel.apk` and writes the + sidecar from that build's version outputs. A tag release also attaches + both files to the Gitea release. +- **Main push (`:latest` + `:`):** no APK is built, but the image + still moves `:latest`, so `image-release` downloads the most recent + release's APK and the sidecar recorded beside it. `:latest` keeps its + update channel between releases. Either way, `docker buildx build` bakes both files into the image via `COPY client/ /app/client/`. +## The sidecar + +`minstrel.apk.version` is JSON: + +```json +{"name": "2026.10.23.1200", "code": 3549060, "channel": "dev"} +``` + +- `name` is the version name the app shows. +- `code` is the ordering key Android installs by. `null` means unknown, + and clients then compare names. +- `channel` says which lane built it. + +A sidecar that is not valid JSON, or has no `name`, is an error rather +than a blank version. + ## Development -Empty directory works fine — the endpoints return 404 and the Android -update banner stays hidden ("no update channel" graceful degradation). +An empty directory works fine: the endpoints return 404 and the Android +update banner stays hidden. -To smoke-test the update flow locally, drop a real APK + a version -file into this directory: +To smoke-test the update flow locally, drop a real APK and a sidecar into +this directory: ```sh cp /path/to/app-release.apk client/minstrel.apk -echo "v0.2.0" > client/minstrel.apk.version +echo '{"name":"2099.01.01.0000","code":null,"channel":"dev"}' > client/minstrel.apk.version ``` ## Why a directory and not embed.FS? -The APK is 30-60 MB. Embedding bloats the Go binary and slows -`go build` for everyone, even when the APK isn't being changed. -File-on-disk also lets operators override at runtime via volume -mount on `/app/client/` if they want to ship their own build. +The APK is 30-60 MB. Embedding bloats the Go binary and slows `go build` +for everyone, even when the APK isn't being changed. File-on-disk also +lets an admin override it at runtime with a volume mount on +`/app/client/` to ship their own build. diff --git a/cmd/minstrel/admin.go b/cmd/minstrel/admin.go index c2dcd3eb..26103e30 100644 --- a/cmd/minstrel/admin.go +++ b/cmd/minstrel/admin.go @@ -55,14 +55,12 @@ func runAdmin(args []string) error { } } -// adminResetPassword resets a user's login password (password_hash). It -// recovers a locked-out operator when the bootstrap password was missed or -// the DB volume was recreated (Fable #321) without DB surgery. +// adminResetPassword resets a user's login password (password_hash), so an +// admin who is locked out can get back in without editing the database. // -// It deliberately leaves subsonic_password alone. That column is stored in -// plain text, and it used to receive the new login password here, so any -// account recovered this way had its login password readable in the database -// (#5026). The Subsonic password is generated separately in Settings. +// It leaves subsonic_password alone: that column is stored in plain text, so +// writing the login password there would make it readable in the database. +// The Subsonic password is generated separately in Settings. func adminResetPassword(args []string) error { fs := flag.NewFlagSet("admin reset-password", flag.ContinueOnError) configPath := fs.String("config", os.Getenv("MINSTREL_CONFIG"), "path to YAML config file") diff --git a/cmd/minstrel/main.go b/cmd/minstrel/main.go index 95df672f..7768650d 100644 --- a/cmd/minstrel/main.go +++ b/cmd/minstrel/main.go @@ -9,28 +9,30 @@ import ( "net/http" "os" "os/signal" + "slices" "syscall" "time" "github.com/jackc/pgx/v5/pgxpool" "git.fabledsword.com/bvandeusen/minstrel/internal/acoustid" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" "git.fabledsword.com/bvandeusen/minstrel/internal/config" "git.fabledsword.com/bvandeusen/minstrel/internal/coplay" "git.fabledsword.com/bvandeusen/minstrel/internal/coverart" "git.fabledsword.com/bvandeusen/minstrel/internal/db" "git.fabledsword.com/bvandeusen/minstrel/internal/eventbus" "git.fabledsword.com/bvandeusen/minstrel/internal/gc" + "git.fabledsword.com/bvandeusen/minstrel/internal/importer" "git.fabledsword.com/bvandeusen/minstrel/internal/library" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" + "git.fabledsword.com/bvandeusen/minstrel/internal/lifecycle" "git.fabledsword.com/bvandeusen/minstrel/internal/logging" "git.fabledsword.com/bvandeusen/minstrel/internal/mailer" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" "git.fabledsword.com/bvandeusen/minstrel/internal/playlists" "git.fabledsword.com/bvandeusen/minstrel/internal/reacquisition" "git.fabledsword.com/bvandeusen/minstrel/internal/recsettings" + "git.fabledsword.com/bvandeusen/minstrel/internal/requests" "git.fabledsword.com/bvandeusen/minstrel/internal/scrobble" "git.fabledsword.com/bvandeusen/minstrel/internal/scrobble/listenbrainz" "git.fabledsword.com/bvandeusen/minstrel/internal/server" @@ -38,6 +40,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/subsonic" syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" "git.fabledsword.com/bvandeusen/minstrel/internal/tags" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" ) func main() { @@ -125,7 +128,7 @@ func run() error { } defer pool.Close() - // Fingerprinting settings (M400 #3913): one instance, shared by the scanner, + // Fingerprinting settings: one instance, shared by the scanner, // the fingerprint backfill, the duplicate sweep and the admin API, so a save // reaches all of them without a restart. A load failure is logged, not fatal: // the service falls back to the shipped defaults. @@ -133,23 +136,37 @@ func run() error { if fpErr != nil { logger.Warn("fingerprint settings: using defaults", "err", fpErr) } - scanner := library.New(pool, logger, cfg.Library.ScanPaths, fpSettings) + // The library's folders: the ones saved in Settings, or + // MINSTREL_LIBRARY_SCAN_PATHS until any are. A load failure falls back to + // the env value. + roots, rootsSaved, rootsErr := library.LoadRoots(ctx, pool, cfg.Library.ScanPaths) + switch { + case rootsErr != nil: + logger.Warn("library folders: using MINSTREL_LIBRARY_SCAN_PATHS", "err", rootsErr) + case rootsSaved && len(cfg.Library.ScanPaths) > 0 && !slices.Equal(roots, cfg.Library.ScanPaths): + logger.Warn("library folders: the ones saved in Settings are used, not MINSTREL_LIBRARY_SCAN_PATHS", + "settings", roots, "env", cfg.Library.ScanPaths) + } + scanner := library.New(pool, logger, roots, fpSettings) - // Loudness analysis settings (M464 #4995): shared by the loudness backfill + // Loudness analysis settings: shared by the loudness backfill // and the admin API, and falls back to the defaults like the above. loudSettings, loudErr := library.NewLoudnessSettingsService(ctx, pool) if loudErr != nil { logger.Warn("loudness settings: using defaults", "err", loudErr) } - // AcoustID lookup settings (M401): shared by the lookup worker and the + // AcoustID lookup settings: shared by the lookup worker and the // admin API. A load failure leaves the lookup off. acoustIDSettings, acoustIDErr := library.NewAcoustIDSettingsService(ctx, pool) if acoustIDErr != nil { logger.Warn("acoustid settings: using defaults (lookup off)", "err", acoustIDErr) } + // One AcoustID client, so the lookup worker and the import's check share + // its pace (AcoustID allows three requests a second per key). + acoustIDClient := acoustid.New(acoustid.DefaultBaseURL) acoustIDLookup := library.NewAcoustIDLookupWorker(pool, logger.With("component", "acoustid_lookup"), - acoustIDSettings, acoustid.New(acoustid.DefaultBaseURL)) + acoustIDSettings, acoustIDClient) contact := cfg.Library.ContactEmail if contact == "" { @@ -177,10 +194,9 @@ func run() error { coverEnricher := coverart.NewEnricher(pool, logger.With("component", "coverart"), coverSettings) coverEnricher.DataDir = cfg.Storage.DataDir - // Recommendation tuning lab (#1250): seeds shipped defaults on first - // boot and pushes the daily_mix weights + taste config into package - // playlists — must precede the scheduler so the first builds score - // with the operator's tuned values, not the pre-push literals. + // Recommendation tuning: seeds the shipped defaults on first boot and + // pushes the daily-mix weights and taste config into package playlists. + // It must precede the scheduler, so the first builds use the tuned values. recSettings, err := recsettings.New(ctx, pool, logger.With("component", "recsettings")) if err != nil { logger.Error("recommendation settings service init failed", "err", err) @@ -190,13 +206,13 @@ func run() error { // One unified scan chain: library walk → MBID backfill → cover enrich. // Boot-time scan and manual-trigger scans share this path; results land // in scan_runs for the admin overview. - if cfg.Library.ScanOnStartup && len(cfg.Library.ScanPaths) > 0 { + if cfg.Library.ScanOnStartup && len(roots) > 0 { go func() { if _, err := library.RunScan(ctx, pool, scanner, coverEnricher, logger.With("component", "scan_run"), library.RunScanConfig{ BackfillCap: 5000, - EnrichCap: -1, // #388: no global cap; per-provider rate limits + EnrichCap: -1, // no global cap; each provider paces itself ArtistEnrichCap: -1, DataDir: cfg.Storage.DataDir, }, @@ -206,14 +222,13 @@ func run() error { }() } else { // No startup file walk, but still run backfill + enrich so cover - // progress doesn't stall just because the operator disabled - // scan-on-startup. + // art keeps arriving with scan-on-startup off. go func() { if _, err := library.RunScan(ctx, pool, nil, coverEnricher, logger.With("component", "scan_run"), library.RunScanConfig{ BackfillCap: 5000, - EnrichCap: -1, // #388: no global cap; per-provider rate limits + EnrichCap: -1, // no global cap; each provider paces itself ArtistEnrichCap: -1, DataDir: cfg.Storage.DataDir, }, @@ -223,46 +238,44 @@ func run() error { }() } - // Start the ListenBrainz scrobble worker. Per spec §M4a, runs every 30s - // and drains up to 50 pending rows per tick. Per-user gating happens - // inside the worker (rows from disabled users are skipped). + // ListenBrainz scrobble worker: every 30s, drains up to 50 pending rows. + // Rows from users with scrobbling off are skipped inside the worker. scrobbleWorker := scrobble.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "scrobble")) go scrobbleWorker.Run(ctx) - // Start the similarity ingest worker. Per spec §M4b, runs every 1h, drains - // up to 5 played tracks + 5 played artists per tick, weekly re-fetch cap - // per row. Public LB endpoints — no token required. + // Similarity ingest worker: hourly, fetches ListenBrainz similarity for up + // to 5 played tracks and 5 played artists, each re-fetched at most weekly. + // The endpoints are public, so no token is needed. similarityWorker := similarity.NewWorker(pool, listenbrainz.NewClient(), logger.With("component", "similarity")) go similarityWorker.Run(ctx) - // Start the household co-play worker (#1533). Recomputes artist–artist + // Household co-play worker: recomputes artist–artist // co-occurrence edges (source='user_cooccurrence') from play_events every // 6h — a collaborative candidate arm for the radio/mix pools. Pure local // SQL, no external calls; empty on single-user servers. go coplay.NewWorker(pool, logger.With("component", "coplay")).Run(ctx) - // Fingerprint backfill (M400 #3908): fingerprints the tracks the scan never - // will — everything imported before fingerprinting existed, and rows derived - // by an older method. A worker of its own rather than a scan stage; see - // internal/library/fingerprint_backfill.go for why. + // Fingerprint backfill: fingerprints the tracks the scan skips (unchanged + // files, and rows fingerprinted by an older method). A worker of its own + // rather than a scan stage; see internal/library/fingerprint_backfill.go. go library.NewFingerprintBackfillWorker(pool, logger.With("component", "fingerprint_backfill"), fpSettings).Run(ctx) - // Loudness backfill (M464 #4995): measures every track's loudness for + // Loudness backfill: measures every track's loudness for // normalization, new tracks included; the scan only drops a changed file's // measurement. See internal/library/loudness_backfill.go. go library.NewLoudnessBackfillWorker(pool, logger.With("component", "loudness_backfill"), loudSettings).Run(ctx) - // AcoustID lookup (M401): fills the recording MBIDs untagged tracks lack, - // so the similarity arms can see them. Idle until the operator adds a key - // and switches it on. See internal/library/acoustid_lookup.go. + // AcoustID lookup: fills the recording MBIDs untagged tracks lack, so the + // similarity arms can see them. Idle until an admin adds a key and + // switches it on. See internal/library/acoustid_lookup.go. go acoustIDLookup.Run(ctx) - // Duplicate sweep (M400 #3910): proposes groups of tracks holding one + // Duplicate sweep: proposes groups of tracks holding one // recording, from the fingerprints above. Sweeps only when fingerprints have // changed since the last sweep. go library.NewDuplicateSweepWorker(pool, logger.With("component", "duplicate_sweep"), fpSettings).Run(ctx) - // Start the tag-enrichment worker (#1490). Reconciles the compiled-in + // Tag-enrichment worker: reconciles the compiled-in // tag providers with tag_provider_settings, bumps the sources version if // the provider set changed (re-opening settled rows), then drains tracks // needing folksonomy tags on a periodic tick. Standalone (not in the file @@ -281,99 +294,100 @@ func run() error { tagEnricher := tags.NewEnricher(pool, logger.With("component", "tags"), tagSettings) go tags.NewWorker(tagEnricher, logger.With("component", "tags")).Run(ctx) - // Start the GC worker. Runs every 1h and sweeps lifecycle tables - // that have no writer-side close path or retention policy: - // orphan play_events, stale play_sessions, expired - // scrobble_queue failures, stuck system_playlist_runs, expired - // password_resets. Each sweep is idempotent — a row that's - // already clean is a no-op. Addresses drift audit findings - // #565 #566 #567 #574 #575 (Scribe parent #552). + // GC worker: hourly, sweeps the tables that have no writer-side close + // path or retention policy: orphan play_events, stale play_sessions, + // expired scrobble_queue failures, stuck system_playlist_runs and expired + // password_resets. Each sweep is idempotent. gcWorker := gc.NewWorker(pool, logger.With("component", "gc")) go gcWorker.Run(ctx) - // Start the Lidarr reconciler worker. Per spec §M5a, polls pending Lidarr - // import requests and reconciles them against the library. Short-circuits - // to no-op when lidarr_config.enabled = false. - lidarrCfg := lidarrconfig.New(pool) - // Live-event bus shared between SSE subscribers (api.Mount) and - // background workers that publish (reconciler today; scanner later). - // Constructed before any service that publishes so they all share the - // same instance. + // Live-event bus shared between SSE subscribers (api.Mount) and the + // background workers that publish. Built before any of them, so they all + // share one instance. bus := eventbus.New() // Scan-run lifecycle events use a package-level setter rather than // threading the bus through RunScan + TryStartScan + the Scheduler // + every test caller. Per-process singleton, set once at startup. library.SetEventBus(bus) - // Per-tick Lidarr client factory: re-reads config so an admin save - // takes effect without a restart, returning nil while Lidarr is - // disabled/unconfigured. Mirrors server.go's lidarrClientFn; the - // reconciler uses it to (re)send unconfirmed adds. - lidarrClientFn := func() *lidarr.Client { - c, cerr := lidarrCfg.Get(ctx) - if cerr != nil || !c.Enabled || c.BaseURL == "" || c.APIKey == "" { - return nil - } - return lidarr.NewClient(c.BaseURL, c.APIKey) - } - // The notifications inbox's one writer (M489), shared by every + // The notifications inbox's one writer, shared by every // background producer started here. The API builds its own over the // same pool and bus. notifier := notifications.New(pool, bus, logger.With("component", "notifications")) - lidarrReconciler := lidarrrequests.NewReconciler(pool, lidarrCfg, lidarrClientFn, logger.With("component", "lidarr"), bus) - lidarrReconciler.SetNotifier(notifier) + // The request reconciler completes approved requests once the library + // holds what they asked for. + requestReconciler := requests.NewReconciler(pool, logger.With("component", "requests"), bus) + requestReconciler.SetNotifier(notifier) library.SetNotifier(notifier) - go lidarrReconciler.Run(ctx) + go requestReconciler.Run(ctx) - // Duplicate resolver (M498): classifies the sweep's groups and, with the - // operator's auto-resolve on, merges copies Lidarr does not map and moves a - // Lidarr release that lists songs twice. Started after SetNotifier so its - // summaries reach admins. See internal/library/duplicate_resolve.go. - resolverLidarr := func() library.LidarrLibrary { - if c := lidarrClientFn(); c != nil { - return c - } - return nil // a nil *lidarr.Client must not become a non-nil interface + // Release catalogue: re-reads discographies older than a + // week from MusicBrainz, a batch an hour, through the process-wide + // MusicBrainz pacer it shares with tag enrichment. + cat := catalogue.New(pool, nil, logger.With("component", "catalogue")) + go cat.Run(ctx) + + // The wanted list: reads followed artists' discographies + // daily for new releases and marks wants held once the library has them. + go wanted.New(pool, cat, logger.With("component", "wanted")).Run(ctx) + + // Search and grab: every ten minutes, searches the + // configured indexers for wants due, paced by each source's + // searches-per-hour, and hands the best release to the download client + // when auto-grab is on. + go wanted.NewSearcher(pool, cat, logger.With("component", "search")).Run(ctx) + + // Import with verification: every minute, finished + // downloads are checked against their album (AcoustID, lengths, tags) + // and filed into the library with MusicBrainz tags, or refused. + var printer importer.Printer + if acoustIDSettings != nil { + printer = &importer.AcoustIDPrinter{Settings: acoustIDSettings, Client: acoustIDClient} } - go library.NewDuplicateResolveWorker(pool, logger.With("component", "duplicate_resolve"), - cfg.Storage.DataDir, fpSettings, resolverLidarr).Run(ctx) + imp := importer.New(pool, scanner, cat, printer, scanner.Roots, logger.With("component", "import")) + imp.DataDir = cfg.Storage.DataDir + go imp.Run(ctx) - // Missing-file re-acquisition (milestone #290). Turns albums whose files - // have been gone longer than the grace window into Lidarr requests, on an - // exponential per-album backoff. Hourly tick — the shortest meaningful - // backoff is measured in hours, so waking more often would only re-read - // settings and find nothing due. + // Download lifecycle: every ten minutes, a download that + // failed or that nobody shares is retried, then given up and its album + // searched again; an imported one seeds, then is removed. + go lifecycle.New(pool, logger.With("component", "lifecycle")).Run(ctx) + + // Duplicate resolver: classifies the sweep's groups and, with auto-resolve + // on, merges the copies that are safe to merge. + // Started after SetNotifier so its summaries reach admins. It asks the + // catalogue for the track list of an album whose wrong slot it must + // decide, a few a pass. See internal/library/duplicate_resolve.go. + go library.NewDuplicateResolveWorker(pool, logger.With("component", "duplicate_resolve"), + cfg.Storage.DataDir, fpSettings, library.ResolveSources{ + TrackLists: cat.RefreshReleaseGroup, Identify: acoustIDLookup.IdentifyRecordings, + }).Run(ctx) + + // Missing-file re-acquisition: wants back albums whose files have been + // gone longer than the grace window, for the search schedule to find. + // Hourly, because the grace window is measured in hours. // // A settings-load failure is logged, not fatal: NewSettingsService always - // returns a usable service holding the defaults, and running on defaults - // is far better than dropping the feature because the database hiccuped - // during boot. + // returns a usable service holding the defaults. reacqSettings, reacqErr := reacquisition.NewSettingsService( ctx, pool, logger.With("component", "reacquisition")) if reacqErr != nil { logger.Warn("reacquisition: using default settings", "err", reacqErr) } - reacqSweeper := reacquisition.NewSweeper( - pool, - reacqSettings, - lidarrrequests.NewService(pool, lidarrCfg, lidarrClientFn, nil), - logger.With("component", "reacquisition"), - ) - reacqSweeper.SetNotifier(notifier) - go reacqSweeper.Run(ctx) + go reacquisition.NewSweeper(pool, reacqSettings, logger.With("component", "reacquisition")).Run(ctx) - // library_changes compactor (#357 follow-up). Daily tick; deletes + // library_changes compactor: daily, deletes // rows older than the configured retention so the change-log table // doesn't grow unbounded. Clients that drop offline longer than // retention hit the /api/library/sync 410 fallback and resync. libraryChangesCompactor := syncpkg.NewCompactor(pool, logger.With("component", "library_changes_compactor")) go libraryChangesCompactor.Run(ctx) - // Notifications inbox retention (M489). Daily: read rows go after 90 + // Notifications inbox retention, daily: read rows go after 90 // days, anything at all after a year. go notifications.NewRetention(pool, logger.With("component", "notifications_retention")).Run(ctx) - // Notifications by email (M489 #5346), grouped, never one per event: new + // Notifications by email, grouped, never one per event: new // music as a daily summary at a local hour, everything else batched an // hour (admin-configurable) after the first item. go notifications.NewDigest( @@ -382,10 +396,8 @@ func run() error { logger.With("component", "notification_digest"), ).Run(ctx) - // Per-user system-playlist scheduler (#392 Half B). Fires each - // active user's daily build at 03:00 in their stored timezone. - // Replaces the 24h-anchored cron loop (removed in the next commit - // of this arc). + // Per-user system-playlist scheduler: runs each active user's daily + // build at 03:00 in their stored timezone. playlistScheduler, err := playlists.NewScheduler( pool, logger.With("component", "playlist_scheduler"), @@ -414,15 +426,14 @@ func run() error { scanCfg := library.RunScanConfig{ BackfillCap: 5000, - EnrichCap: -1, // #388: no global cap; per-provider rate limits + EnrichCap: -1, // no global cap; each provider paces itself ArtistEnrichCap: -1, DataDir: cfg.Storage.DataDir, } // Filesystem watcher: near-instant pickup of new/changed files via // fsnotify, scanning just the affected paths + enriching their albums // inline. Replaces the removed configurable scan scheduler. - watcher := library.NewWatcher(scanner, coverEnricher, - logger.With("component", "watcher"), cfg.Library.ScanPaths) + watcher := library.NewWatcher(scanner, coverEnricher, logger.With("component", "watcher")) go func() { if werr := watcher.Run(ctx); werr != nil { logger.Warn("library watcher exited", "err", werr) @@ -444,7 +455,7 @@ func run() error { srv.LoudnessSettings = loudSettings srv.AcoustIDLookup = acoustIDLookup // The sweeper above holds this same instance, so a save from the admin - // card changes what it does on its next tick (#3936). + // card changes what it does on its next tick. srv.ReacqSettings = reacqSettings srv.StreamSecret = cfg.StreamSecret httpServer := &http.Server{ @@ -457,6 +468,11 @@ func run() error { // server's limitRequestBody middleware instead. IdleTimeout: 120 * time.Second, } + // Shutdown waits for in-flight requests but never cancels them, and an + // SSE stream only ends when its client leaves. Closing the bus ends + // every stream as shutdown begins, so the drain waits only on real work + // such as an audio stream; clients reconnect to the next instance. + httpServer.RegisterOnShutdown(bus.Close) errCh := make(chan error, 1) go func() { diff --git a/config.example.yaml b/config.example.yaml index d56e1d40..b5b7bf7c 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -1,78 +1,54 @@ -# Minstrel example configuration. -# Every field is optional; defaults shown below match Default() in internal/config. -# Each setting can be overridden at runtime via MINSTREL_* environment variables -# (see internal/config/config.go for the full list). +# Minstrel example configuration. In the container image, mount it at +# /etc/smartmusic/config.yaml. Every field is optional; the values below are +# the defaults. Each MINSTREL_* variable named here overrides its field. server: - address: ":4533" + address: ":4533" # MINSTREL_SERVER_ADDRESS storage: - # On-disk root for cached runtime artifacts (currently playlist cover - # collages under /playlist_covers/). Default: "./data" relative - # to the working directory. Production deployments should set an absolute - # path on persistent storage. - # Env: MINSTREL_STORAGE_DATA_DIR - data_dir: "./data" + # Artwork, caches and the stream-signing key. Use persistent storage. + data_dir: "./data" # MINSTREL_STORAGE_DATA_DIR (the image sets /app/data) database: - # Postgres connection string, e.g. postgres://minstrel:minstrel@localhost:5432/minstrel?sslmode=disable - url: "" + # e.g. postgres://minstrel:minstrel@localhost:5432/minstrel?sslmode=disable + url: "" # MINSTREL_DATABASE_URL log: - level: "info" # debug | info | warn | error - format: "text" # text | json + level: "info" # debug | info | warn | error. MINSTREL_LOG_LEVEL + format: "text" # text | json. MINSTREL_LOG_FORMAT library: - # Filesystem roots to scan for music. Each path is walked recursively. - # Env: MINSTREL_LIBRARY_SCAN_PATHS (colon-separated, PATH-style) - scan_paths: [] - # Kick off a scan on server startup. On by default so a fresh install - # populates its library on first boot with no manual step. Scans are - # incremental (unchanged files are skipped by mtime), so the cost on - # every later restart is just a directory walk, not a full re-index. - # Set false if you'd rather scan only on a schedule / via the admin UI. - # Env: MINSTREL_LIBRARY_SCAN_ON_STARTUP - scan_on_startup: true + # Library folders, walked recursively. Used on first start only: once they + # are changed in Admin › Library, the saved ones win. Changing a folder there + # moves its tracks with it, so plays, likes and playlists survive a new mount. + scan_paths: [] # MINSTREL_LIBRARY_SCAN_PATHS (colon-separated) + # Scan on startup. Scans are incremental, so a restart only walks the folders. + scan_on_startup: true # MINSTREL_LIBRARY_SCAN_ON_STARTUP + # Contact sent to the Cover Art Archive in the User-Agent. Empty sends + # the project's URL. + contact_email: "" # MINSTREL_CONTACT_EMAIL -# Per-instance branding. Operators can rename their instance ("Family -# Jukebox", "Office Music", etc.) and override the OG share-preview -# description. Both fields default to sensible Minstrel values when -# unset; you only need this section to override. -# Env: MINSTREL_BRANDING_APP_NAME, MINSTREL_BRANDING_DESCRIPTION branding: - app_name: "Minstrel" - description: "Self-hosted music server with Subsonic compatibility, smart shuffle, and Lidarr integration." + app_name: "Minstrel" # MINSTREL_BRANDING_APP_NAME + description: "Self-hosted music server with Subsonic compatibility, smart shuffle, and music acquisition." # MINSTREL_BRANDING_DESCRIPTION subsonic: - # Gate for the legacy `p=` plaintext password parameter on /rest/*. - # Leave disabled unless a specific client requires it — Minstrel prefers - # apiKey (OpenSubsonic) and falls back to u+t+s token auth. - # Env: MINSTREL_SUBSONIC_ALLOW_PLAINTEXT_PASSWORD - allow_plaintext_password: false + # Accept the plaintext `p=` password on /rest/*. Leave off unless a client + # needs it; apiKey and token auth work without it. + allow_plaintext_password: false # MINSTREL_SUBSONIC_ALLOW_PLAINTEXT_PASSWORD events: - # Idle gap (minutes) between play_events that closes a play_session and - # starts a new one. Spec §6. + # Minutes of silence that end a listening session. session_timeout_minutes: 30 - # Skip rule (spec §6): a play counts as a SKIP if completion ratio is below - # this threshold AND duration played (ms) is below the next threshold. + # A play is a skip when it stops before this fraction of the track AND + # before this many milliseconds. skip_max_completion_ratio: 0.5 skip_max_duration_played_ms: 30000 recommendation: - # Base score every candidate gets before adjustments. Spec §6. - base_weight: 1.0 - # Bonus for tracks the user has liked (general_likes). - like_boost: 2.0 - # Multiplier on recency_decay (1.0 for tracks ≥ 30 days stale, 0 for fresh). - recency_weight: 1.0 - # Penalty multiplier on skip_ratio (skips/plays); 1.0 = full penalty. - skip_penalty: 1.0 - # ± random jitter applied to every candidate; breaks ties without dominating. - jitter_magnitude: 0.1 - # Hard-suppression window: tracks played within this many hours never appear. + # Scoring weights are tuned in Admin › Tuning, not here. + # Tracks played within this many hours are left out of radio. recently_played_hours: 1 - # Default radio size when ?limit= is not specified. + # Radio length when ?limit= is not given, and the most ?limit= may ask for. radio_size: 50 - # Maximum allowed ?limit= (server caps to this regardless). radio_size_max: 200 diff --git a/deploy/initdb/01-create-test-db.sql b/deploy/initdb/01-create-test-db.sql index e1c339f1..638792e0 100644 --- a/deploy/initdb/01-create-test-db.sql +++ b/deploy/initdb/01-create-test-db.sql @@ -1,6 +1,5 @@ -- Runs once, only on a fresh Postgres data volume (Docker entrypoint -- initdb hook). Creates the dedicated integration-test database so --- `go test` never truncates the operator's dev `minstrel` DB (Fable --- #339). For an already-initialised volume this script does NOT run; +-- `go test` never truncates the dev `minstrel` DB. For an already-initialised volume this script does NOT run; -- `make test-integration` creates the DB idempotently instead. CREATE DATABASE minstrel_test OWNER minstrel; diff --git a/docker-compose.yml b/docker-compose.yml index a56998cf..83437538 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,17 +1,8 @@ version: "3.9" -# Local development environment for Minstrel. -# -# Integration tests run against a SEPARATE database (minstrel_test) so a -# test run never truncates the dev `minstrel` DB (Fable #339). Use the -# Makefile target, which ensures the test DB exists then points the -# tests at it: -# make test-integration -# (CI runs the same suite against its own ephemeral Postgres service — -# see the `integration` job in .gitea/workflows/release.yml.) -# -# Full stack (server + db): -# docker compose up --build +# Local development stack: `docker compose up --build`. +# The database tests use a separate minstrel_test database, so they never +# truncate your dev data. Run them with `make test-integration`. services: postgres: @@ -26,8 +17,8 @@ services: # - "5432:5432" volumes: - minstrel-pgdata:/var/lib/postgresql/data - # Creates minstrel_test on a fresh volume (Fable #339). No-op on - # an existing volume — make test-integration ensures it instead. + # Creates minstrel_test on a fresh volume. On an existing volume, + # make test-integration creates it instead. - ./deploy/initdb:/docker-entrypoint-initdb.d:ro healthcheck: test: ["CMD-SHELL", "pg_isready -U minstrel -d minstrel"] @@ -51,12 +42,9 @@ services: ports: - "4533:4533" volumes: - # Point ./music at your test library (symlink, bind-mount, whatever). - # Read-only so a buggy scanner can't rewrite your files. + # Your test library, read-only so a scanner bug can't change your files. - /mnt/Media/Music:/music:ro - # Cached artifacts: playlist cover collages, artist art, album-cover - # fallbacks (when the music dir is RO). Persists across container - # recreates so the operator doesn't redownload art on every up/down. + # Artwork and caches, kept across container recreates. - minstrel-data:/app/data volumes: diff --git a/docs/hosting.md b/docs/hosting.md index 10ede42e..ab544713 100644 --- a/docs/hosting.md +++ b/docs/hosting.md @@ -13,7 +13,7 @@ Publish the port to the network and use `http://:4533`: ports: ['4533:4533'] ``` -If nothing sits in front of Minstrel, set **Admin → Integrations → Client IP +If nothing sits in front of Minstrel, set **Admin › Email & network → Client IP detection** to 0. It defaults to 1, which assumes one proxy (see below for why that matters). @@ -48,7 +48,7 @@ Three things, all required: the app is. Raise the proxy's read timeout well past a minute (nginx: `proxy_read_timeout 1h;`). -3. **Tell Minstrel where it lives**, in **Admin → Integrations**: +3. **Tell Minstrel where it lives**, in **Admin › Email & network**: - **Client IP detection:** the number of proxies in front of Minstrel. One proxy (Caddy, Traefik, nginx) is 1; Cloudflare in front of Traefik is 2. Minstrel uses this to find the real client address, for login rate diff --git a/docs/screenshots/README.md b/docs/screenshots/README.md index 3aae8e97..c7306c6f 100644 --- a/docs/screenshots/README.md +++ b/docs/screenshots/README.md @@ -11,7 +11,7 @@ across all shots for a consistent look. | `home.png` | `/` | The home page with a populated library (system playlists / recently added) | | `register.png` | `/register` | The account-creation form on a fresh instance | | `library-scan.png` | `/admin` | The library scan running in stages (grab it mid-scan so stage tallies show) | -| `integrations.png` | `/admin/integrations` | The ListenBrainz + Lidarr integration cards | +| `integrations.png` | `/admin/acquisition` | Indexers, download clients and searching | | `android-download.png` | `/settings` | The "Install the Android app" download card | | `invite-users.png` | `/admin/users` | The invite-token generation / users list | diff --git a/docs/smart-music-client-spec.md b/docs/smart-music-client-spec.md index d74f8a86..0ffa0f70 100644 --- a/docs/smart-music-client-spec.md +++ b/docs/smart-music-client-spec.md @@ -1,5 +1,7 @@ # Project Spec: Smart Music Client (companion app) +> **History.** This is the original design spec, kept as written. Where it describes an outside library manager doing search, adds and deletes, Minstrel now does that itself. + ## 1. Overview A mobile music player client built as the intentional companion to the Smart Music Server (see `smart-music-server-spec.md`). This is not a general-purpose Subsonic client — it is the client that knows how to consume the server's extended API and surface its novel features (contextual likes, session-aware radio, Lidarr integration, server-canonical state) as first-class UX. diff --git a/docs/smart-music-server-spec.md b/docs/smart-music-server-spec.md index def6d03f..ad380759 100644 --- a/docs/smart-music-server-spec.md +++ b/docs/smart-music-server-spec.md @@ -1,5 +1,7 @@ # Project Spec: Smart Music Server (working name) +> **History.** This is the original design spec, kept as written. Where it describes an outside library manager doing search, adds and deletes, Minstrel now does that itself. + ## 1. Overview A self-hosted music server and streaming backend built to close a real gap in the self-hosted music ecosystem: **state and intelligence belong on the server, not the client.** Existing Subsonic-compatible servers (Navidrome, Gonic, Airsonic-Advanced) defer all of the "smart" behavior — shuffle logic, radio, play count semantics, like management — to clients. The result is fragmented experience across devices: a song liked on one client is unknown to another; shuffle quality depends on which app you opened; play counts diverge. This project centralizes those decisions on the server so every client sees the same canonical state. diff --git a/internal/acoustid/client.go b/internal/acoustid/client.go index d6153dea..4cbfe5f3 100644 --- a/internal/acoustid/client.go +++ b/internal/acoustid/client.go @@ -1,13 +1,13 @@ // Package acoustid looks up MusicBrainz recording ids by acoustic fingerprint -// through the AcoustID web service (M401). +// through the AcoustID web service. // -// What it relies on, from https://acoustid.org/webservice (read 2026-10-06): +// What it relies on, from https://acoustid.org/webservice: // - POST https://api.acoustid.org/v2/lookup with client, duration (whole // seconds), fingerprint (fpcalc's compressed string) and meta. A gzip body // with Content-Encoding: gzip is accepted and preferred, because // fingerprints are long. // - meta=recordings returns each recording's title, duration and artists, -// which the worker needs to tell several candidates apart (M401 D5). +// which the worker needs to tell several candidates apart. // - "Do not make more than 3 requests per second." // - The service is free for non-commercial use only. // @@ -39,9 +39,8 @@ const ( // minInterval keeps to AcoustID's 3 requests a second with headroom: one // every 400ms is 2.5 a second. minInterval = 400 * time.Millisecond - // lookupTimeout bounds one request, connection included (rule 156). A - // lookup answers in well under a second; 20s is the line past which slow - // has become never. + // lookupTimeout bounds one request, connection included. A lookup answers + // in well under a second; 20s is the line past which slow has become never. lookupTimeout = 20 * time.Second // bodyLimit caps the response read. A lookup with meta=recordings for a // popular song runs to tens of kilobytes. @@ -59,7 +58,7 @@ const ( ) var ( - // ErrInvalidKey: the operator's application key was refused. No lookup + // ErrInvalidKey: the configured application key was refused. No lookup // can succeed until it is changed, so the worker stops and says so. ErrInvalidKey = errors.New("acoustid: the API key was refused") // ErrInvalidFingerprint: AcoustID rejected this track's fingerprint. A diff --git a/internal/acoustid/client_test.go b/internal/acoustid/client_test.go index 3226fcc4..437388ed 100644 --- a/internal/acoustid/client_test.go +++ b/internal/acoustid/client_test.go @@ -198,10 +198,14 @@ func TestLookup_KeepsToTheRateLimit(t *testing.T) { t.Fatal(err) } } + // The client spaces its sends by minInterval; the server sees arrivals, + // which transit can bunch by a few milliseconds (389ms under -race, CI run + // 9543). What AcoustID enforces is 3 a second at its end, so that is the + // bound: minInterval's headroom is what absorbs the bunching. + const serviceLimit = time.Second / 3 for i := 1; i < len(at); i++ { - // A little slack for the server's own clock reading. - if gap := at[i].Sub(at[i-1]); gap < minInterval-10*time.Millisecond { - t.Errorf("requests %d and %d were %s apart, want at least %s", i-1, i, gap, minInterval) + if gap := at[i].Sub(at[i-1]); gap < serviceLimit { + t.Errorf("requests %d and %d arrived %s apart, want at least %s", i-1, i, gap, serviceLimit) } } } diff --git a/internal/acquire/acquire_test.go b/internal/acquire/acquire_test.go new file mode 100644 index 00000000..ac0ebf16 --- /dev/null +++ b/internal/acquire/acquire_test.go @@ -0,0 +1,180 @@ +package acquire + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "testing" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/exchange" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" + "git.fabledsword.com/bvandeusen/minstrel/internal/prowlarr" +) + +func validUpdate() Update { + return Update{Label: "Prowlarr", BaseURL: "http://prowlarr:9696", TimeoutMs: 30000, LocalPath: "/downloads", SearchesPerHour: 120} +} + +func TestUpdateValidate(t *testing.T) { + if err := validUpdate().validate(); err != nil { + t.Fatalf("a valid form: %v", err) + } + for name, tc := range map[string]struct { + edit func(*Update) + field string + }{ + "no name": {func(u *Update) { u.Label = "" }, "label"}, + "switched on with no address": {func(u *Update) { u.Enabled, u.BaseURL = true, "" }, "base_url"}, + "an address with no scheme": {func(u *Update) { u.BaseURL = "prowlarr:9696" }, "base_url"}, + "a timeout too short": {func(u *Update) { u.TimeoutMs = 10 }, "timeout_ms"}, + "a relative downloads folder": {func(u *Update) { u.LocalPath = "downloads" }, "local_path"}, + "a relative client folder": {func(u *Update) { u.RemotePath = "data" }, "remote_path"}, + "too many searches": {func(u *Update) { u.SearchesPerHour = 4000 }, "searches_per_hour"}, + } { + u := validUpdate() + tc.edit(&u) + var invalid *InvalidError + if err := u.validate(); !errors.As(err, &invalid) || invalid.Field != tc.field { + t.Errorf("%s: err = %v, want an InvalidError on %s", name, err, tc.field) + } + } + // Switched off, an empty address is a row still being set up. + u := validUpdate() + u.BaseURL = "" + if err := u.validate(); err != nil { + t.Errorf("switched off with no address: %v", err) + } +} + +func TestSourceFor_BuildsTheClientTheProtocolNames(t *testing.T) { + for protocol, check := range map[string]func(Source) bool{ + ProtocolProwlarr: func(s Source) bool { _, ok := s.Search.(*prowlarr.Client); return ok && s.Fetch != nil }, + ProtocolTorznab: func(s Source) bool { _, ok := s.Search.(*indexer.Torznab); return ok && s.Fetch != nil }, + // Exchange's results are all magnets: nothing to fetch through it. + ProtocolExchange: func(s Source) bool { _, ok := s.Search.(*exchange.Client); return ok && s.Fetch == nil }, + } { + src := SourceFor(dbq.AcquisitionService{Code: protocol, Protocol: protocol, SearchesPerHour: 7}) + if !check(src) || src.Code != protocol || src.PerHour != 7 { + t.Errorf("%s: source %+v", protocol, src) + } + } +} + +func TestJobs_AnExchangeDoesWhatItsSwitchesAllow(t *testing.T) { + ex := dbq.AcquisitionService{Kind: KindDownload, Protocol: ProtocolExchange, ServesSearch: true} + if !Searches(ex) || Fetches(ex) { + t.Error("an exchange with only search on should search and not fetch") + } + // A qBittorrent's switches mean nothing: it fetches, whatever they say. + qb := dbq.AcquisitionService{Kind: KindDownload, Protocol: ProtocolQBittorrent, ServesSearch: true} + if Searches(qb) || !Fetches(qb) { + t.Error("a qBittorrent fetches and never searches") + } +} + +func TestTest_NoAddressIsNothingToReach(t *testing.T) { + r := Test(context.Background(), dbq.AcquisitionService{Protocol: ProtocolProwlarr, Kind: KindSearch}) + if r.OK || !strings.Contains(r.Detail, "No address") { + t.Fatalf("result = %+v", r) + } +} + +func TestTest_SearchReportsResultsAndARefusedKey(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("X-Api-Key") != "good" { + w.WriteHeader(http.StatusUnauthorized) + return + } + _, _ = w.Write([]byte(`[{"title": "Radiohead - OK Computer"}]`)) + })) + defer srv.Close() + + row := dbq.AcquisitionService{Kind: KindSearch, Protocol: ProtocolProwlarr, BaseUrl: srv.URL, ApiKey: "good", TimeoutMs: 5000} + if r := Test(context.Background(), row); !r.OK || !strings.Contains(r.Detail, "1 results") { + t.Errorf("a working key: %+v", r) + } + row.ApiKey = "bad" + if r := Test(context.Background(), row); r.OK || !strings.Contains(r.Detail, "refused") { + t.Errorf("a refused key: %+v", r) + } +} + +// fakeExchange answers the three native APIs. +func fakeExchange(t *testing.T) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.URL.Path == "/api/downloads": + _, _ = w.Write([]byte(`{"enabled": true, "running": true, "inbound_conns": 3, "downloads": []}`)) + case r.URL.Path == "/api/search": + _, _ = w.Write([]byte(`{"total": 0, "results": []}`)) + case strings.HasPrefix(r.URL.Path, "/api/torrents/"): + w.WriteHeader(http.StatusNotFound) + _, _ = w.Write([]byte(`{"error": "unknown torrent"}`)) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + return srv +} + +func TestTest_AnExchangeReportsEachJobSwitchedOn(t *testing.T) { + srv := fakeExchange(t) + row := dbq.AcquisitionService{ + Kind: KindDownload, Protocol: ProtocolExchange, BaseUrl: srv.URL, TimeoutMs: 5000, + ServesDownloads: true, ServesSearch: true, ServesEnrichment: true, LocalPath: t.TempDir(), + } + r := Test(context.Background(), row) + if !r.OK { + t.Fatalf("all three jobs: %+v", r) + } + for _, job := range []string{"Downloads:", "Search:", "Release details:"} { + if !strings.Contains(r.Detail, job) { + t.Errorf("detail %q does not report %s", r.Detail, job) + } + } + row.ServesDownloads, row.ServesEnrichment = false, false + if r := Test(context.Background(), row); strings.Contains(r.Detail, "Downloads:") || !strings.Contains(r.Detail, "Search:") { + t.Errorf("only search on: %+v", r) + } + row.ServesSearch = false + if r := Test(context.Background(), row); r.OK || !strings.Contains(r.Detail, "switched off") { + t.Errorf("every job off: %+v", r) + } +} + +func TestTest_ADownloadClientNeedsAFolderMinstrelCanSee(t *testing.T) { + srv := fakeExchange(t) + row := dbq.AcquisitionService{ + Kind: KindDownload, Protocol: ProtocolExchange, BaseUrl: srv.URL, TimeoutMs: 5000, + ServesDownloads: true, Category: DefaultCategory, + LocalPath: filepath.Join(t.TempDir(), "not-mounted"), + } + r := Test(context.Background(), row) + if r.OK || !strings.Contains(r.Detail, "cannot see its downloads folder") { + t.Fatalf("a missing folder: %+v", r) + } + row.LocalPath = t.TempDir() + if r := Test(context.Background(), row); !r.OK || !strings.Contains(r.Detail, `"minstrel" category`) { + t.Errorf("a visible, writable folder: %+v", r) + } +} + +func TestDownloadClient_WrapsThePathMapping(t *testing.T) { + d, err := DownloadClient(dbq.AcquisitionService{Protocol: ProtocolQBittorrent, LocalPath: "/downloads", RemotePath: "/data"}) + if err != nil { + t.Fatal(err) + } + if _, plain := d.(*downloader.Client); plain { + t.Error("a row naming two folders should come back wrapped in its mapping") + } + if _, err := DownloadClient(dbq.AcquisitionService{Protocol: ProtocolProwlarr}); err == nil { + t.Error("a search protocol is not a download client") + } +} diff --git a/internal/acquire/build.go b/internal/acquire/build.go new file mode 100644 index 00000000..882d528c --- /dev/null +++ b/internal/acquire/build.go @@ -0,0 +1,45 @@ +package acquire + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// Build makes a Service from the settings as they stand: every enabled +// search source behind one paced Fanout, the first enabled download client, +// and the quality profile. Built afresh for each pass and each request, so a +// changed setting takes effect on the next one. +func Build(ctx context.Context, pool *pgxpool.Pool) (*Service, dbq.AcquisitionSetting, error) { + q := dbq.New(pool) + settings, err := q.GetAcquisitionSettings(ctx) + if err != nil { + return nil, settings, fmt.Errorf("acquire: read settings: %w", err) + } + sources, err := Sources(ctx, pool) + if err != nil { + return nil, settings, fmt.Errorf("acquire: read search sources: %w", err) + } + clients, err := q.ListUsableAcquisitionServices(ctx, JobDownload) + if err != nil { + return nil, settings, fmt.Errorf("acquire: read download clients: %w", err) + } + s := &Service{Profile: release.NewProfile(settings.AllowedQualities), Memory: NewMemory(pool)} + fan := &Fanout{Sources: sources, Pace: SearchPace} + if len(sources) > 0 { + s.Search = fan + } + if len(clients) > 0 { + row := clients[0] + client, err := DownloadClient(row) + if err != nil { + return nil, settings, fmt.Errorf("acquire: download client %s: %w", row.Code, err) + } + s.Grabber = &Downloader{Client: client, Code: row.Code, Landing: row.LocalPath, Fetch: fan} + } + return s, settings, nil +} diff --git a/internal/acquire/fanout.go b/internal/acquire/fanout.go new file mode 100644 index 00000000..c4f607c8 --- /dev/null +++ b/internal/acquire/fanout.go @@ -0,0 +1,164 @@ +package acquire + +import ( + "context" + "errors" + "fmt" + "strings" + "sync" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// Searching every source at once. A source that is down narrows a search rather +// than failing it: the results come back with the list of who did not answer. + +// SourceFailure is one source that could not be reached. +type SourceFailure struct { + Code string + Label string + Err error +} + +// PartialError is a search that reached some sources and not others. It is +// returned alongside the results, not instead of them. +type PartialError struct { + Failures []SourceFailure + // Answered is how many sources did answer. Zero means the search failed. + Answered int +} + +func (e *PartialError) Error() string { + if e == nil || len(e.Failures) == 0 { + return "some sources did not answer" + } + parts := make([]string, 0, len(e.Failures)) + for _, f := range e.Failures { + parts = append(parts, fmt.Sprintf("%s: %s", f.Label, indexer.FirstLine(f.Err.Error()))) + } + return "some sources did not answer — " + strings.Join(parts, "; ") +} + +// Unwrap lets errors.Is find indexer.ErrUnreachable through a partial failure. +func (e *PartialError) Unwrap() []error { + if e == nil { + return nil + } + out := make([]error, 0, len(e.Failures)) + for _, f := range e.Failures { + out = append(out, f.Err) + } + return out +} + +// Fanout searches every source and merges what comes back. It is both the +// Searcher and the Fetcher, routing a .torrent fetch to the source that +// offered the result. +type Fanout struct { + // Sources in code order: the fold keeps a duplicate's first arrival, so + // the order decides which source a grab is fetched through. + Sources []Source + // Pace spaces each source's searches. Nil paces nothing. + Pace *Pacer +} + +// Search asks every source concurrently. Each client bounds its own call by +// its configured timeout, so there is no shared deadline here. +func (f *Fanout) Search(ctx context.Context, query indexer.Query) ([]indexer.Result, error) { + if f == nil || len(f.Sources) == 0 { + return nil, indexer.ErrUnconfigured + } + found := make([][]indexer.Result, len(f.Sources)) + errs := make([]error, len(f.Sources)) + var wg sync.WaitGroup + for i, src := range f.Sources { + if src.Search == nil { + continue + } + f.Pace.spend(src) + wg.Add(1) + go func(i int, src Source) { + defer wg.Done() + found[i], errs[i] = src.Search.Search(ctx, query) + }(i, src) + } + wg.Wait() + + out := make([]indexer.Result, 0, 32) + var failures []SourceFailure + answered, unconfigured := 0, 0 + for i, src := range f.Sources { + if src.Search == nil { + continue + } + if errors.Is(errs[i], indexer.ErrUnconfigured) { + unconfigured++ + continue + } + if rl, ok := indexer.AsRateLimited(errs[i]); ok { + f.Pace.hold(src.Code, rl) + } + if errs[i] != nil { + failures = append(failures, SourceFailure{Code: src.Code, Label: src.Label, Err: errs[i]}) + continue + } + answered++ + for j := range found[i] { + if found[i][j].Source == "" { + found[i][j].Source = src.Code + } + } + out = append(out, found[i]...) + } + switch { + case answered == 0 && len(failures) > 0: + return nil, &PartialError{Failures: failures} + case len(failures) > 0: + return out, &PartialError{Failures: failures, Answered: answered} + case answered == 0 && unconfigured > 0: + return nil, indexer.ErrUnconfigured + } + return out, nil +} + +// Download fetches a result's .torrent from the source that offered it: a +// Prowlarr link carries Prowlarr's key and only Prowlarr can redeem it. +func (f *Fanout) Download(ctx context.Context, r indexer.Result) ([]byte, error) { + if f == nil { + return nil, indexer.ErrUnconfigured + } + for _, src := range f.Sources { + if src.Code != r.Source { + continue + } + if src.Fetch == nil { + return nil, fmt.Errorf("%w: %s cannot fetch torrent files", indexer.ErrUnconfigured, src.Label) + } + return src.Fetch.Download(ctx, r) + } + if r.Source == "" { + return nil, fmt.Errorf("%w: this result does not say which source it came from", indexer.ErrUnconfigured) + } + return nil, fmt.Errorf("%w: %q is no longer a configured source", indexer.ErrUnconfigured, r.Source) +} + +// Narrowing is one source that did not answer, as a page shows it. +type Narrowing struct { + Source string `json:"source"` + Label string `json:"label"` + Reason string `json:"reason"` +} + +// narrowings reads a PartialError out of err, and whether any source +// answered. Nil when err is not a partial failure. +func narrowings(err error) ([]Narrowing, bool) { + var partial *PartialError + if !errors.As(err, &partial) || len(partial.Failures) == 0 { + return nil, false + } + out := make([]Narrowing, 0, len(partial.Failures)) + for _, f := range partial.Failures { + out = append(out, Narrowing{Source: f.Code, Label: f.Label, Reason: indexer.FirstLine(f.Err.Error())}) + } + return out, partial.Answered > 0 +} diff --git a/internal/acquire/grab.go b/internal/acquire/grab.go new file mode 100644 index 00000000..001814bd --- /dev/null +++ b/internal/acquire/grab.go @@ -0,0 +1,338 @@ +package acquire + +import ( + "context" + "errors" + "path" + "strings" + "sync" + "time" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// Taking a release. One live grab per want: a second is tie-broken against +// the download already running. + +// Tag goes on every torrent Minstrel adds, so they can be told apart from +// anything else in the client. +const Tag = "minstrel" + +// MissingClient is Taken.Missing when no download client is set up. +const MissingClient = "download_client" + +// nearlyDone is how far along a running download stays whatever would +// replace it. +const nearlyDone = 0.9 + +// Downloader is the grab half of a Service. +type Downloader struct { + Client downloader.Driver + // Code is the client's acquisition_services row. + Code string + // Landing is the client's downloads folder as Minstrel sees it; each + // grab lands in a folder of its own under it. + Landing string + // Fetch gets a result's .torrent from the source that offered it. + Fetch Fetcher +} + +// Origin says who took a release. +type Origin struct { + By string // OriginSchedule or OriginOperator + User pgtype.UUID +} + +// Contest is how a grab fared against the download already running. +type Contest struct { + // Kept: the running one stays and this grab was not made. Replaced: + // this grab took its place. + Kept, Replaced bool + Reason string + Rival dbq.Grab +} + +// Taken is what came of a grab. +type Taken struct { + Chosen *Candidate + // Added is the only field that means a download started. + Added bool + SavePath string + Grab dbq.Grab + // Gone: the search no longer offers the release. + Gone bool + // Refused: not taken; the reason is on Chosen. + Refused bool + // Unconfigured: Missing names what is not set up ("" for search). + Unconfigured bool + Missing string + Contest *Contest + Error string +} + +// Take searches again and grabs the candidate named by key: a person's +// grab. Searching again keeps the indexer's download link (which can carry +// its key) on the server, and finds a release gone since it was shown. +// force takes what the verdict would not. +func (s *Service) Take(ctx context.Context, want Want, key string, force bool, origin Origin) Taken { + if s.Grabber == nil { + return Taken{Unconfigured: true, Missing: MissingClient} + } + found := s.For(ctx, want) + switch { + case found.Unconfigured: + return Taken{Unconfigured: true} + case found.Error != "": + return Taken{Error: found.Error} + } + chosen := pick(found.Candidates, key) + if chosen == nil { + return Taken{Gone: true} + } + return s.Grab(ctx, want, chosen, found.Candidates, force, origin) +} + +// Grab hands a judged candidate to the download client. The schedule calls +// it with the candidate it has just judged; Take calls it after searching +// again. +func (s *Service) Grab(ctx context.Context, want Want, chosen *Candidate, all []Candidate, force bool, origin Origin) Taken { + out := Taken{Chosen: chosen} + if s.Grabber == nil || s.Grabber.Client == nil { + out.Unconfigured, out.Missing = true, MissingClient + return out + } + if chosen.Verdict == Blocked || (chosen.Verdict != Grab && !force) { + out.Refused = true + return out + } + + unlock := lockWant(want.ID) + defer unlock() + contest := s.contest(ctx, want, chosen, all, origin) + if contest != nil && contest.Kept { + out.Contest = contest + return out + } + + add := downloader.Add{Magnet: chosen.Source.Magnet, Tags: []string{Tag}} + hash := chosen.InfoHash + if add.Magnet == "" { + if s.Grabber.Fetch == nil { + out.Error = "This release is a .torrent link, and nothing is set up to fetch it." + return out + } + body, err := s.Grabber.Fetch.Download(ctx, chosen.Source) + if magnet, ok := indexer.AsMagnet(err); ok { + add.Magnet, err = magnet, nil + } + if err != nil { + out.Error = indexer.FirstLine(err.Error()) + return out + } + if add.Magnet == "" { + add.Torrent = body + if meta, err := release.ParseTorrent(body); err == nil { + if hash == "" { + hash = meta.InfoHash + } + if refused := s.judgeFiles(ctx, want, chosen, hash, meta.Files, force); refused { + out.Refused = true + return out + } + } + } + } + if hash == "" && add.Magnet != "" { + hash = indexer.MagnetHash(add.Magnet) + } + add.SavePath = landingFor(s.Grabber.Landing, hash, chosen.Name) + + err := s.Grabber.Client.AddTorrent(ctx, add) + if errors.Is(err, downloader.ErrDuplicate) && s.heldElsewhere(ctx, hash) { + why := "Already in the download client outside Minstrel's category, where Minstrel cannot follow it (" + + strings.TrimPrefix(indexer.FirstLine(err.Error()), downloader.ErrDuplicate.Error()+": ") + ")" + if force { + out.Error = why + "." + return out + } + set(chosen, Blocked, Hard, CodeBlocked, why+".") + if s.Memory != nil && hash != "" { + _ = s.Memory.Block(ctx, dbq.BlockReleaseParams{ + InfoHash: hash, Name: chosen.Name, Reason: BlockGone, Detail: why + ".", WantID: want.ID, + ExpiresAt: pgtype.Timestamptz{Time: time.Now().Add(GoneBlock), Valid: true}, + }) + } + out.Refused = true + return out + } + if err != nil && !errors.Is(err, downloader.ErrDuplicate) { + if errors.Is(err, downloader.ErrUnconfigured) { + out.Unconfigured, out.Missing = true, MissingClient + } else { + out.Error = indexer.FirstLine(err.Error()) + } + return out + } + out.Added, out.SavePath = true, add.SavePath + if s.Memory == nil { + return out + } + + var replaced pgtype.UUID + why := "" + if contest != nil && contest.Replaced { + out.Contest = contest + replaced, why = contest.Rival.ID, "Replaced by "+chosen.Name+": "+contest.Reason+"." + } + g, err := s.Memory.Grabbed(ctx, dbq.InsertGrabParams{ + WantID: want.ID, InfoHash: hash, Name: chosen.Name, Key: Key(*chosen), Indexer: chosen.Indexer, + Source: chosen.Source.Source, Client: s.Grabber.Code, SavePath: add.SavePath, + Quality: string(chosen.Tier), Origin: origin.By, Forced: force, GrabbedBy: origin.User, + }, replaced, why) + if err != nil { + out.Error = "The download started, but could not be recorded: " + indexer.FirstLine(err.Error()) + return out + } + out.Grab = g + // Removed only after the new download was added and recorded, so a + // failure never leaves the want with nothing running. + if replaced.Valid && contest.Rival.InfoHash != "" { + if err := s.Grabber.Client.Delete(ctx, true, contest.Rival.InfoHash); err != nil { + contest.Reason += ". The download it replaced could not be removed from the client: " + indexer.FirstLine(err.Error()) + } + } + return out +} + +// heldElsewhere reports whether a release the client says it already holds is +// one it does not list for Minstrel: held under another category. Recorded +// as a grab, it would read as gone half an hour later and be taken again, +// every pass. A client that +// cannot answer now is given the benefit of the doubt, as the add was. +func (s *Service) heldElsewhere(ctx context.Context, hash string) bool { + if hash == "" { + return false + } + listed, err := s.Grabber.Client.Torrents(ctx, indexer.NormalizeHash(hash)) + return err == nil && len(listed) == 0 +} + +// judgeFiles judges a chosen release again on its .torrent's file list, and +// sets it aside when the files say it is not the album, so no later search +// offers it again. force still takes it. +func (s *Service) judgeFiles(ctx context.Context, want Want, chosen *Candidate, hash string, files []release.File, force bool) bool { + chosen.Files = files + if chosen.FilesCount == 0 { + chosen.FilesCount = len(files) + } + veto, reason, _ := release.JudgeFiles(want.Want, files) + if veto == "" || force { + return false + } + set(chosen, Rejected, Hard, veto, reason) + if s.Memory != nil && hash != "" { + _ = s.Memory.Block(ctx, dbq.BlockReleaseParams{ + InfoHash: hash, Name: chosen.Name, Reason: BlockFiles, Detail: reason, WantID: want.ID, + }) + } + return true +} + +var wantLocks sync.Map // want id → *sync.Mutex + +// lockWant settles two grabs for one want one after the other. The +// one-live-grab index is the backstop for anything outside this process. +func lockWant(id pgtype.UUID) func() { + if !id.Valid { + return func() {} + } + m, _ := wantLocks.LoadOrStore(id.Bytes, &sync.Mutex{}) + mu := m.(*sync.Mutex) + mu.Lock() + return mu.Unlock +} + +// contest settles a grab against the download running for its want: a +// person's grab beats the schedule's; between two of a kind the better +// candidate wins; a tie, a download nearly done, or one that cannot be +// removed stays. Nil when nothing is running. +func (s *Service) contest(ctx context.Context, want Want, chosen *Candidate, all []Candidate, origin Origin) *Contest { + if s.Memory == nil || !want.ID.Valid { + return nil + } + running, ok, err := s.Memory.LiveGrab(ctx, want.ID) + if err != nil || !ok { + return nil + } + c := &Contest{Rival: running} + if running.InfoHash != "" && running.InfoHash == chosen.InfoHash { + c.Kept, c.Reason = true, "That release is already downloading for this album." + return c + } + switch { + case origin.By == OriginOperator && running.Origin == OriginSchedule: + c.Reason = "you chose it over the one the schedule started" + case origin.By == OriginSchedule && running.Origin == OriginOperator: + c.Kept, c.Reason = true, "A download you started for this album is already running." + return c + default: + var theirs *Candidate + for i := range all { + if (running.InfoHash != "" && all[i].InfoHash == running.InfoHash) || Key(all[i]) == running.Key { + theirs = &all[i] + break + } + } + if theirs == nil || !better(*chosen, *theirs) { + c.Kept, c.Reason = true, "A download for this album is already running, and this release is no better." + return c + } + c.Reason = "it ranks above the one that was running" + } + if running.InfoHash == "" { + c.Kept, c.Reason = true, "A download for this album is already running, and it cannot be identified to replace it." + return c + } + if ts, err := s.Grabber.Client.Torrents(ctx, running.InfoHash); err == nil && len(ts) > 0 && ts[0].Progress >= nearlyDone { + c.Kept, c.Reason = true, "A download for this album is already running and nearly done." + return c + } + c.Replaced = true + return c +} + +// landingFor is the folder a download lands in: one per torrent, named by +// its infohash, so two grabs never unpack into the same place. +func landingFor(root, hash, name string) string { + dir := hash + if dir == "" { + dir = slug(name) + } + if dir == "" { + dir = "unnamed" + } + return path.Join(root, dir) +} + +// slug keeps a folder name to characters every filesystem in the chain +// takes. +func slug(s string) string { + var b strings.Builder + for _, r := range strings.ToLower(strings.TrimSpace(s)) { + if b.Len() >= 80 { + break + } + switch { + case r >= 'a' && r <= 'z', r >= '0' && r <= '9': + b.WriteRune(r) + case r == '-', r == '_', r == ' ', r == '.': + b.WriteRune('-') + } + } + return strings.Trim(b.String(), "-") +} diff --git a/internal/acquire/memory.go b/internal/acquire/memory.go new file mode 100644 index 00000000..8e553e9f --- /dev/null +++ b/internal/acquire/memory.go @@ -0,0 +1,199 @@ +package acquire + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// Block reasons, matching release_blocks_reason_check. +const ( + BlockOperator = "operator" + BlockFiles = "files" + // BlockImport: the import found its files were not the album. + BlockImport = "import" + // BlockStalled: given up on because it would not finish; held + // back until the block's expires_at. + BlockStalled = "stalled" + // BlockGone: its download client would not let Minstrel follow it, held + // outside Minstrel's category or leaving the client twice; held + // back for GoneBlock. + BlockGone = "gone" +) + +// GoneBlock is how long a release its client lost track of is held back: +// long enough that a release which keeps vanishing is tried once a day rather +// than on every pass, short enough that it is taken once the client lets go. +const GoneBlock = 24 * time.Hour + +// Grab origins, matching grabs_origin_check. +const ( + OriginSchedule = "schedule" + OriginOperator = "operator" +) + +// Memory is what searches and grabs leave behind: every candidate of a +// want's last search, the grabs, and the releases set aside. +type Memory struct { + pool *pgxpool.Pool +} + +// NewMemory returns a Memory over pool. +func NewMemory(pool *pgxpool.Pool) *Memory { return &Memory{pool: pool} } + +// Keep replaces a want's candidates with this search's, in order. +func (m *Memory) Keep(ctx context.Context, wantID pgtype.UUID, cands []Candidate) error { + return pgx.BeginFunc(ctx, m.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + if err := q.DeleteWantCandidates(ctx, wantID); err != nil { + return err + } + for i, c := range cands { + reasons, err := json.Marshal(c.Reasons) + if err != nil { + return err + } + if c.Reasons == nil { + reasons = []byte("[]") + } + folded := c.FoldedFrom + if folded == nil { + folded = []string{} + } + if err := q.InsertWantCandidate(ctx, dbq.InsertWantCandidateParams{ + WantID: wantID, Key: Key(c), Name: c.Name, InfoHash: c.InfoHash, Indexer: c.Indexer, + Source: c.Source.Source, SizeBytes: c.SizeBytes, Seeders: int32(c.Seeders), + FilesCount: int32(c.FilesCount), Format: release.Describe(c.Release), Quality: string(c.Tier), + Year: int32(c.Year), Verdict: string(c.Verdict), Reason: c.Reason, Reasons: reasons, + FoldedFrom: folded, Rank: int32(i), + }); err != nil { + return err + } + } + return nil + }) +} + +// blocked is every release set aside, by infohash, with the sentence a +// candidate shows. A failed read blocks nothing: offering a release set +// aside costs one grab a person can undo, while blocking everything on a +// database blip would hide every result. +func (m *Memory) blocked(ctx context.Context) map[string]string { + if m == nil { + return nil + } + rows, err := dbq.New(m.pool).ListReleaseBlocks(ctx) + if err != nil { + return nil + } + out := make(map[string]string, len(rows)) + for _, r := range rows { + out[indexer.NormalizeHash(r.InfoHash)] = BlockSentence(r.Reason, r.Detail) + } + return out +} + +// BlockSentence is what a blocked candidate says. +func BlockSentence(reason, detail string) string { + switch reason { + case BlockFiles: + if detail != "" { + return "Set aside for what its files hold: " + detail + } + return "Set aside for what its files hold." + case BlockImport: + if detail != "" { + return "Refused on import: " + detail + } + return "Refused on import: its files were not the album." + case BlockGone: + if detail != "" { + return "Held back for now: " + detail + } + return "Held back for now: its download client lost track of it." + case BlockStalled: + if detail != "" { + return "Given up on for now: " + detail + } + return "Given up on for now: it would not finish." + } + return "You set this release aside." +} + +// Block sets a release aside and marks every kept candidate of it blocked. +func (m *Memory) Block(ctx context.Context, p dbq.BlockReleaseParams) error { + p.InfoHash = indexer.NormalizeHash(p.InfoHash) + if p.InfoHash == "" { + return errors.New("a release without an infohash cannot be set aside") + } + return pgx.BeginFunc(ctx, m.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + if err := q.BlockRelease(ctx, p); err != nil { + return err + } + return q.MarkCandidatesBlocked(ctx, dbq.MarkCandidatesBlockedParams{ + InfoHash: p.InfoHash, Reason: BlockSentence(p.Reason, p.Detail), + }) + }) +} + +// Unblock lets a release be taken again. Its kept candidates wait for a +// person until the next search judges them afresh. +func (m *Memory) Unblock(ctx context.Context, hash string) (bool, error) { + hash = indexer.NormalizeHash(hash) + var n int64 + err := pgx.BeginFunc(ctx, m.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + var err error + if n, err = q.UnblockRelease(ctx, hash); err != nil { + return err + } + return q.MarkCandidatesUnblocked(ctx, dbq.MarkCandidatesUnblockedParams{ + InfoHash: hash, Reason: "Allowed again. The next search judges it afresh.", + }) + }) + return n > 0, err +} + +// LiveGrab is the download running for a want, if there is one. +func (m *Memory) LiveGrab(ctx context.Context, wantID pgtype.UUID) (dbq.Grab, bool, error) { + g, err := dbq.New(m.pool).GetLiveGrab(ctx, wantID) + if errors.Is(err, pgx.ErrNoRows) { + return g, false, nil + } + if err != nil { + return g, false, err + } + return g, true, nil +} + +// Grabbed records a download started, ending the grab it replaced first so +// the one-live-grab index holds. +func (m *Memory) Grabbed(ctx context.Context, p dbq.InsertGrabParams, replaced pgtype.UUID, why string) (dbq.Grab, error) { + var g dbq.Grab + err := pgx.BeginFunc(ctx, m.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + if replaced.Valid { + if err := q.EndGrab(ctx, dbq.EndGrabParams{ID: replaced, EndReason: "superseded", EndDetail: why}); err != nil { + return err + } + } + var err error + g, err = q.InsertGrab(ctx, p) + return err + }) + if err != nil { + return g, fmt.Errorf("record grab: %w", err) + } + return g, nil +} diff --git a/internal/acquire/pace.go b/internal/acquire/pace.go new file mode 100644 index 00000000..fb2352e3 --- /dev/null +++ b/internal/acquire/pace.go @@ -0,0 +1,145 @@ +package acquire + +import ( + "context" + "sync" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// SearchPace is the one Pacer every search shares. Process-wide because a +// Fanout is built fresh for each search; a Pacer per Fanout would start empty +// and pace nothing. +var SearchPace = &Pacer{} + +// holdWithoutRetryAfter is how long a source that answered 429 without saying +// for how long is left alone: one schedule tick. +const holdWithoutRetryAfter = 10 * time.Minute + +// Pacer spaces the searches sent to each source by its searches_per_hour: a +// spacing, not a burst bucket, because Prowlarr answers 429 to a few requests +// in quick succession. Every search spends, the schedule's and a +// person's alike; only the schedule waits. +type Pacer struct { + // Now and Sleep are the clock, replaceable in tests. + Now func() time.Time + Sleep func(context.Context, time.Duration) error + + mu sync.Mutex + next map[string]time.Time +} + +func (p *Pacer) now() time.Time { + if p.Now != nil { + return p.Now() + } + return time.Now() +} + +func (p *Pacer) sleep(ctx context.Context, d time.Duration) error { + if p.Sleep != nil { + return p.Sleep(ctx, d) + } + t := time.NewTimer(d) + defer t.Stop() + select { + case <-t.C: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +// gap is the spacing a source's number asks for; zero is unpaced. +func gap(perHour int) time.Duration { + if perHour <= 0 { + return 0 + } + return time.Hour / time.Duration(perHour) +} + +// readyAt is when every source may next be asked. +func (p *Pacer) readyAt(sources []Source) time.Time { + p.mu.Lock() + defer p.mu.Unlock() + var at time.Time + for _, s := range sources { + if s.Search == nil { + continue + } + if n := p.next[s.Code]; n.After(at) { + at = n + } + } + return at +} + +// Wait holds until every source may be asked, and answers false, without +// waiting, when that is after the deadline or ctx ends first. All of them, +// not any: a search while one is held back would be narrowed, and the want +// recorded as looked for. +func (p *Pacer) Wait(ctx context.Context, sources []Source, deadline time.Time) bool { + if p == nil { + return true + } + at := p.readyAt(sources) + if !deadline.IsZero() && at.After(deadline) { + return false + } + if d := at.Sub(p.now()); d > 0 { + return p.sleep(ctx, d) == nil + } + return ctx.Err() == nil +} + +// spend takes one search's slot from a source. +func (p *Pacer) spend(s Source) { + g := gap(s.PerHour) + if p == nil || g == 0 { + return + } + p.mu.Lock() + defer p.mu.Unlock() + if p.next == nil { + p.next = map[string]time.Time{} + } + at := p.next[s.Code] + if t := p.now(); at.Before(t) { + at = t + } + p.next[s.Code] = at.Add(g) +} + +// hold leaves a source alone after it answered 429, for as long as it asked, +// whatever its searches_per_hour says. +func (p *Pacer) hold(code string, rl *indexer.RateLimited) { + if p == nil { + return + } + d := rl.RetryAfter + if d <= 0 { + d = holdWithoutRetryAfter + } + p.mu.Lock() + defer p.mu.Unlock() + if p.next == nil { + p.next = map[string]time.Time{} + } + if until := p.now().Add(d); until.After(p.next[code]) { + p.next[code] = until + } +} + +// Ready holds until every source this service searches may be asked again, +// and answers false when that is past the deadline. The schedule calls it +// before each want; a person's search never does. +func (s *Service) Ready(ctx context.Context, deadline time.Time) bool { + if s == nil { + return ctx.Err() == nil + } + if f, ok := s.Search.(*Fanout); ok && f != nil { + return f.Pace.Wait(ctx, f.Sources, deadline) + } + return ctx.Err() == nil +} diff --git a/internal/acquire/search.go b/internal/acquire/search.go new file mode 100644 index 00000000..73960b3a --- /dev/null +++ b/internal/acquire/search.go @@ -0,0 +1,408 @@ +package acquire + +import ( + "context" + "errors" + "sort" + "strings" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// One search for one want, and what could be done about each result. +// internal/release +// says whether a name is the album and what quality it is; this package +// asks in the right order and says why in words a person can act on. + +// Verdict is what can be done about one result. +type Verdict string + +const ( + // Grab: the album, in a quality the profile takes. + Grab Verdict = "grab" + // Review: maybe the album; a person decides. + Review Verdict = "review" + // Refused: the album, in a quality the profile does not take. The only + // verdict that names a setting. + Refused Verdict = "refused" + // Rejected: not the album. + Rejected Verdict = "rejected" + // Blocked: a release set aside, by a person or by its files. + Blocked Verdict = "blocked" +) + +// order sorts verdicts: what can be taken first, what was given up on last. +var order = map[Verdict]int{Grab: 0, Review: 1, Refused: 2, Rejected: 3, Blocked: 4} + +// Reason kinds: a hard reason is a fact about the bytes, a soft one about the +// moment or a setting. +const ( + Hard = "hard" + Soft = "soft" +) + +// Reason codes beyond internal/release's vetoes. +const ( + CodeBlocked = "blocked" + CodeProfile = "profile" + CodeNoSeeders = "no_seeders" + CodeDoubt = "doubt" + // CodeNotBetter: the album, but no better than the copy the library + // holds, for a want that is an upgrade. + CodeNotBetter = "not_better" +) + +// Reason is one thing a verdict rests on. +type Reason struct { + Kind string `json:"kind"` + Code string `json:"code"` + Text string `json:"text"` +} + +// Want is the album searched for: its wants row and how the catalogue +// describes it. +type Want struct { + ID pgtype.UUID + release.Want + // Floor is the tier of the copy the library holds, for an album wanted + // for a better one: a release must rank above it. "" for an + // album not held, and for a video rip, which any clean copy beats. + Floor release.Tier + // Held says the floor in words: "MP3, about 192 kbps". + Held string +} + +// Candidate is one result with everything decided about it. +type Candidate struct { + release.Release + // Source is the indexer's own result. It never leaves the server: its + // DownloadURL can carry the indexer's key. + Source indexer.Result + InfoHash string + Indexer string + SizeBytes int64 + Seeders int + SwarmKnown bool + Category int + FilesCount int + // Files is the torrent's file list once its .torrent has been read. + Files []release.File + FoldedFrom []string + + Tier release.Tier + YearAgrees bool + Verdict Verdict + Reason string + Reasons []Reason +} + +// Key is how a candidate is named across a search and a grab: its infohash, +// or its name when the source gave none. +func Key(c Candidate) string { + if c.InfoHash != "" { + return c.InfoHash + } + return c.Name +} + +// Result is one search. +type Result struct { + Query string + // Unconfigured: no search source is set up. + Unconfigured bool + // Error is a search that failed outright. + Error string + // Narrowed names sources that did not answer on a search that otherwise + // did. + Narrowed []Narrowing + Considered int + Folded int + // Looked is how many .torrent files were read for their file lists. + Looked int + Candidates []Candidate + // KeepError: the candidates could not be stored. + KeepError string +} + +// Grabbable is what may be taken, best first. +func (r Result) Grabbable() []Candidate { + var out []Candidate + for _, c := range r.Candidates { + if c.Verdict == Grab { + out = append(out, c) + } + } + return out +} + +// ForAPerson says whether anything found is worth a person's look: a +// release that may be the album, whatever its quality. +func (r Result) ForAPerson() bool { + for _, c := range r.Candidates { + if c.Verdict == Grab || c.Verdict == Review || c.Verdict == Refused { + return true + } + } + return false +} + +// Service searches and grabs. +type Service struct { + // Search is nil when no source is configured. + Search Searcher + // Grabber is nil when no download client is configured. + Grabber *Downloader + Profile release.Profile + // Memory stores candidates, grabs and blocks. Nil keeps nothing. + Memory *Memory +} + +// Query is the text sent to the indexers: the artist and the title, without +// a trailing bracketed part the release names usually leave off. +func Query(w release.Want) string { + title := w.Title + if i := strings.IndexAny(title, "(["); i > 0 { + title = title[:i] + } + return strings.Join(strings.Fields(strings.TrimSpace(w.Artist)+" "+strings.TrimSpace(title)), " ") +} + +// For searches for one want and judges everything that comes back. It never +// returns an error: a search that could not be made is a Result saying so. +func (s *Service) For(ctx context.Context, want Want) Result { + r := Result{Query: Query(want.Want)} + if s == nil || s.Search == nil { + r.Unconfigured = true + return r + } + found, err := s.Search.Search(ctx, indexer.Query{Text: r.Query, Categories: indexer.MusicCategories}) + // A partial search arrives as results and an error together. + if narrowed, answered := narrowings(err); narrowed != nil && answered { + r.Narrowed, err = narrowed, nil + } + switch { + case errors.Is(err, indexer.ErrUnconfigured): + r.Unconfigured = true + return r + case err != nil: + r.Error = indexer.FirstLine(err.Error()) + return r + } + r.Considered = len(found) + cands, folded := fold(found) + r.Folded = folded + + blocked := s.Memory.blocked(ctx) + for i := range cands { + decide(&cands[i], want, s.Profile, blocked) + } + sortCandidates(cands) + for _, i := range s.peek(ctx, cands) { + decide(&cands[i], want, s.Profile, blocked) + r.Looked++ + } + sortCandidates(cands) + r.Candidates = cands + if s.Memory != nil && want.ID.Valid { + if err := s.Memory.Keep(ctx, want.ID, cands); err != nil { + r.KeepError = indexer.FirstLine(err.Error()) + } + } + return r +} + +// decide works out one candidate's verdict, in order: given up on, then +// whether it is the album, then whether the profile takes its quality, then +// whether anything leaves it to a person. A quality refusal said about the +// wrong album would be noise, so identity comes first. +func decide(c *Candidate, want Want, profile release.Profile, blocked map[string]string) { + w := want.Want + c.Verdict, c.Reason, c.Reasons = "", "", nil + if why, ok := blocked[c.InfoHash]; ok && c.InfoHash != "" { + set(c, Blocked, Hard, CodeBlocked, why) + return + } + m := release.Judge(w, c.Release) + c.YearAgrees = m.YearAgrees + if m.Veto != "" { + set(c, Rejected, Hard, m.Veto, m.Reason) + return + } + doubt := m.Doubt + if len(c.Files) > 0 { + if in := release.Inspect(c.Files); in.Format != "" { + c.Format = in.Format + } + veto, reason, fileDoubt := release.JudgeFiles(w, c.Files) + if veto != "" { + set(c, Rejected, Hard, veto, reason) + return + } + if doubt == "" { + doubt = fileDoubt + } + } + c.Tier = release.TierOf(c.Release, c.Category) + if !profile.Allows(c.Tier) { + set(c, Refused, Soft, CodeProfile, "Your quality settings don’t take "+strings.ToLower(release.Label(c.Tier))+".") + return + } + if want.Floor != "" && release.Rank(c.Tier) <= release.Rank(want.Floor) { + held := want.Held + if held == "" { + held = strings.ToLower(release.Label(want.Floor)) + } + set(c, Refused, Soft, CodeNotBetter, "No better than the copy you have ("+held+").") + return + } + if doubt != "" { + set(c, Review, Soft, CodeDoubt, doubt) + return + } + if c.Seeders == 0 { + set(c, Review, Soft, CodeNoSeeders, "Nobody is sharing it right now.") + return + } + c.Verdict = Grab + c.Reason = "The album, " + strings.ToLower(release.Label(c.Tier)) + "." + if d := release.Describe(c.Release); d != "" { + c.Reason = "The album, " + d + "." + } +} + +func set(c *Candidate, v Verdict, kind, code, text string) { + c.Verdict, c.Reason = v, text + c.Reasons = append(c.Reasons, Reason{Kind: kind, Code: code, Text: text}) +} + +// sortCandidates puts the best first, stably so the same search proposes the +// same grab twice. +func sortCandidates(cands []Candidate) { + sort.SliceStable(cands, func(a, b int) bool { return better(cands[a], cands[b]) }) +} + +// better says whether x ranks above y: verdict, then quality, then how many +// share it, then whether its year agrees. +func better(x, y Candidate) bool { + if order[x.Verdict] != order[y.Verdict] { + return order[x.Verdict] < order[y.Verdict] + } + if rx, ry := release.Rank(x.Tier), release.Rank(y.Tier); rx != ry { + return rx > ry + } + if x.Seeders != y.Seeders { + return x.Seeders > y.Seeders + } + return x.YearAgrees && !y.YearAgrees +} + +// fold merges results that are the same torrent, by infohash. Prowlarr +// aggregates, so one release arrives once per tracker that lists it. A +// result without a hash is never folded: an absent hash is not a shared one. +func fold(found []indexer.Result) ([]Candidate, int) { + out := make([]Candidate, 0, len(found)) + at := make(map[string]int, len(found)) + folded := 0 + for _, f := range found { + if strings.TrimSpace(f.Title) == "" { + continue + } + hash := indexer.NormalizeHash(f.InfoHash) + if hash == "" && f.Magnet != "" { + hash = indexer.MagnetHash(f.Magnet) + } + if hash != "" { + if i, seen := at[hash]; seen { + folded++ + if f.Indexer != "" { + out[i].FoldedFrom = append(out[i].FoldedFrom, f.Indexer) + } + // A measured swarm beats a reported one; among like, the + // highest count. + switch { + case f.SwarmKnown && !out[i].SwarmKnown: + out[i].Seeders, out[i].SwarmKnown = f.Seeders, true + case f.SwarmKnown == out[i].SwarmKnown && f.Seeders > out[i].Seeders: + out[i].Seeders = f.Seeders + } + if f.FilesCount > 0 && out[i].FilesCount == 0 { + out[i].FilesCount = f.FilesCount + } + continue + } + at[hash] = len(out) + } + out = append(out, Candidate{ + Release: release.Parse(f.Title), Source: f, InfoHash: hash, Indexer: f.Indexer, + SizeBytes: f.SizeBytes, Seeders: f.Seeders, SwarmKnown: f.SwarmKnown, + Category: f.Category, FilesCount: f.FilesCount, + }) + } + return out, folded +} + +// maxPeeks is how many .torrent files one search reads. Few: each goes +// through Prowlarr to a tracker, and Prowlarr answers 429 to a burst. +const maxPeeks = 3 + +// peek reads the .torrent of candidates that are the album but whose name +// states no format, so their files can say what they are; it returns which +// now carry a file list. A failed fetch leaves the candidate as it was. +func (s *Service) peek(ctx context.Context, cands []Candidate) []int { + f, ok := s.Search.(Fetcher) + if !ok { + return nil + } + var looked []int + for i := range cands { + if len(looked) >= maxPeeks || ctx.Err() != nil { + break + } + c := &cands[i] + if c.Tier != release.TierUnknown || (c.Verdict != Grab && c.Verdict != Refused) || + c.Files != nil || c.Source.DownloadURL == "" || c.Source.Magnet != "" { + continue + } + body, err := f.Download(ctx, c.Source) + if err != nil { + continue + } + meta, err := release.ParseTorrent(body) + if err != nil { + continue + } + c.Files = meta.Files + if c.InfoHash == "" { + c.InfoHash = meta.InfoHash + } + if c.FilesCount == 0 { + c.FilesCount = len(meta.Files) + } + looked = append(looked, i) + } + return looked +} + +// pick finds a candidate by key: the infohash first, the exact name for a +// result that has none. +func pick(cands []Candidate, key string) *Candidate { + key = strings.TrimSpace(key) + if key == "" { + return nil + } + hash := indexer.NormalizeHash(key) + for i := range cands { + if cands[i].InfoHash != "" && cands[i].InfoHash == hash { + return &cands[i] + } + } + for i := range cands { + if cands[i].InfoHash == "" && cands[i].Name == key { + return &cands[i] + } + } + return nil +} diff --git a/internal/acquire/search_test.go b/internal/acquire/search_test.go new file mode 100644 index 00000000..63022d0e --- /dev/null +++ b/internal/acquire/search_test.go @@ -0,0 +1,291 @@ +package acquire + +import ( + "context" + "errors" + "fmt" + "strings" + "testing" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +type fakeSearch struct { + results []indexer.Result + err error + asked []indexer.Query +} + +func (f *fakeSearch) Search(_ context.Context, q indexer.Query) ([]indexer.Result, error) { + f.asked = append(f.asked, q) + return f.results, f.err +} + +var ram = Want{Want: release.Want{ + Artist: "Daft Punk", Title: "Random Access Memories", Year: 2013, PrimaryType: "Album", + MinTracks: 13, MaxTracks: 15, +}} + +var anyQuality = release.NewProfile([]string{ + "lossless_hires", "lossless", "lossy_high", "lossy_standard", "lossy_low", "lossy_poor", "lossy_trash", +}) + +func result(title, hash string, seeders int) indexer.Result { + return indexer.Result{Title: title, InfoHash: hash, Seeders: seeders, Indexer: "Knaben", Source: "prowlarr", + Magnet: "magnet:?xt=urn:btih:" + hash} +} + +func TestFor_JudgesSortsAndFolds(t *testing.T) { + search := &fakeSearch{results: []indexer.Result{ + result("Daft Punk - Random Access Memories (2013) [MP3 320]", "aaaa", 40), + result("Daft Punk Random Access Memories (Drumless Edition 2023) [Flac 24-88]", "bbbb", 90), + result("Daft Punk - Random Access Memories (2013) [FLAC]", "cccc", 12), + result("Daft Punk - Random Access Memories (2013) [FLAC]", "CCCC", 30), // the same torrent from another tracker + result("Daft Punk - Discovery (2001) [FLAC]", "dddd", 50), + }} + s := &Service{Search: search, Profile: anyQuality} + r := s.For(context.Background(), ram) + + if len(search.asked) != 1 || search.asked[0].Text != "Daft Punk Random Access Memories" { + t.Fatalf("asked %+v", search.asked) + } + if r.Considered != 5 || r.Folded != 1 || len(r.Candidates) != 4 { + t.Fatalf("considered %d folded %d candidates %d", r.Considered, r.Folded, len(r.Candidates)) + } + // Lossless before high-quality lossy among what can be taken; the + // folded copy keeps the higher seeder count. + first, second := r.Candidates[0], r.Candidates[1] + if first.InfoHash != "cccc" || first.Verdict != Grab || first.Seeders != 30 || first.Tier != release.TierLossless { + t.Errorf("first = %s %s %d %s", first.Name, first.Verdict, first.Seeders, first.Tier) + } + if second.InfoHash != "aaaa" || second.Verdict != Grab { + t.Errorf("second = %s %s", second.Name, second.Verdict) + } + for _, c := range r.Candidates[2:] { + if c.Verdict != Rejected || len(c.Reasons) != 1 || c.Reasons[0].Kind != Hard { + t.Errorf("%s: %s %+v", c.Name, c.Verdict, c.Reasons) + } + } + if len(r.Grabbable()) != 2 || !r.ForAPerson() { + t.Errorf("grabbable %d", len(r.Grabbable())) + } +} + +func TestFor_ProfileSeedersAndDoubt(t *testing.T) { + search := &fakeSearch{results: []indexer.Result{ + result("Daft Punk - Random Access Memories [MP3 128]", "aaaa", 5), + result("Daft Punk - Random Access Memories [FLAC]", "bbbb", 0), + result("Daft Punk - Random Access Memories", "cccc", 9), + }} + s := &Service{Search: search, Profile: release.NewProfile([]string{"lossless", "lossy_high"})} + r := s.For(context.Background(), ram) + verdicts := map[string]Verdict{} + for _, c := range r.Candidates { + verdicts[c.InfoHash] = c.Verdict + } + if verdicts["aaaa"] != Refused || verdicts["bbbb"] != Review || verdicts["cccc"] != Refused { + t.Errorf("verdicts = %v", verdicts) + } + if len(r.Grabbable()) != 0 || !r.ForAPerson() { + t.Error("nothing grabbable, but something for a person") + } +} + +func TestFor_BlockedReleasesAreSetAside(t *testing.T) { + c := Candidate{Release: release.Parse("Daft Punk - Random Access Memories [FLAC]"), InfoHash: "aaaa", Seeders: 3} + decide(&c, ram, anyQuality, map[string]string{"aaaa": "You set this release aside."}) + if c.Verdict != Blocked || c.Reason != "You set this release aside." { + t.Errorf("verdict %s %q", c.Verdict, c.Reason) + } +} + +func TestFor_UnconfiguredAndFailedSearches(t *testing.T) { + if r := (&Service{}).For(context.Background(), ram); !r.Unconfigured { + t.Error("no source is unconfigured") + } + r := (&Service{Search: &fakeSearch{err: indexer.ErrUnconfigured}}).For(context.Background(), ram) + if !r.Unconfigured { + t.Error("ErrUnconfigured is unconfigured") + } + r = (&Service{Search: &fakeSearch{err: errors.New("dial tcp: refused")}}).For(context.Background(), ram) + if r.Error == "" { + t.Error("a failed search says so") + } +} + +func TestFanout_ANarrowedSearchKeepsWhatAnswered(t *testing.T) { + up := &fakeSearch{results: []indexer.Result{result("Daft Punk - Random Access Memories [FLAC]", "aaaa", 3)}} + down := &fakeSearch{err: indexer.ErrUnreachable} + f := &Fanout{Sources: []Source{{Code: "prowlarr", Label: "Prowlarr", Search: up}, {Code: "torznab", Label: "Feed", Search: down}}} + r := (&Service{Search: f, Profile: anyQuality}).For(context.Background(), ram) + if r.Error != "" || len(r.Candidates) != 1 || len(r.Narrowed) != 1 || r.Narrowed[0].Source != "torznab" { + t.Errorf("narrowed search: error %q, %d candidates, narrowed %+v", r.Error, len(r.Candidates), r.Narrowed) + } + f.Sources = f.Sources[1:] + if r := (&Service{Search: f}).For(context.Background(), ram); r.Error == "" { + t.Error("every source down is a failed search") + } +} + +func TestPacer_SpacesSearchesAndHoldsAfterA429(t *testing.T) { + now := time.Unix(0, 0) + var slept time.Duration + p := &Pacer{Now: func() time.Time { return now }, Sleep: func(_ context.Context, d time.Duration) error { slept += d; return nil }} + src := []Source{{Code: "prowlarr", PerHour: 120, Search: &fakeSearch{}}} + p.spend(src[0]) + if !p.Wait(context.Background(), src, now.Add(time.Hour)) || slept != 30*time.Second { + t.Errorf("slept %s, want 30s", slept) + } + if p.Wait(context.Background(), src, now.Add(time.Second)) { + t.Error("waited past the deadline") + } + p.hold("prowlarr", &indexer.RateLimited{RetryAfter: 5 * time.Minute}) + if at := p.readyAt(src); at.Sub(now) != 5*time.Minute { + t.Errorf("held until %s", at.Sub(now)) + } +} + +type fakeClient struct { + downloader.Driver + added []downloader.Add + addErr error + deleted []string + progress float64 + // elsewhere: the client holds it but does not list it for Minstrel. + elsewhere bool +} + +func (c *fakeClient) AddTorrent(_ context.Context, a downloader.Add) error { + c.added = append(c.added, a) + return c.addErr +} + +func (c *fakeClient) Torrents(_ context.Context, hashes ...string) ([]downloader.Torrent, error) { + if c.elsewhere { + return nil, nil + } + return []downloader.Torrent{{Hash: hashes[0], Progress: c.progress}}, nil +} + +func (c *fakeClient) Delete(_ context.Context, _ bool, hashes ...string) error { + c.deleted = append(c.deleted, hashes...) + return nil +} + +func TestGrab_AddsAMagnetToItsOwnFolder(t *testing.T) { + client := &fakeClient{} + s := &Service{Search: &fakeSearch{results: []indexer.Result{ + result("Daft Punk - Random Access Memories (2013) [FLAC]", "ABCD", 8), + }}, Profile: anyQuality, Grabber: &Downloader{Client: client, Code: "qbittorrent", Landing: "/downloads"}} + r := s.For(context.Background(), ram) + took := s.Grab(context.Background(), ram, &r.Candidates[0], r.Candidates, false, Origin{By: OriginSchedule}) + if !took.Added || took.SavePath != "/downloads/abcd" { + t.Fatalf("took %+v", took) + } + if len(client.added) != 1 || !strings.Contains(strings.ToLower(client.added[0].Magnet), "abcd") || client.added[0].Tags[0] != Tag { + t.Errorf("added %+v", client.added) + } +} + +func TestGrab_RefusesWhatTheVerdictWouldNotTake(t *testing.T) { + client := &fakeClient{} + s := &Service{Grabber: &Downloader{Client: client, Landing: "/downloads"}} + review := Candidate{Verdict: Review, InfoHash: "aaaa", Source: indexer.Result{Magnet: "magnet:?xt=urn:btih:aaaa"}} + if took := s.Grab(context.Background(), ram, &review, nil, false, Origin{By: OriginSchedule}); !took.Refused { + t.Error("a review was taken unforced") + } + if took := s.Grab(context.Background(), ram, &review, nil, true, Origin{By: OriginOperator}); !took.Added { + t.Errorf("a forced review was not taken: %+v", took) + } + blocked := Candidate{Verdict: Blocked, InfoHash: "bbbb"} + if took := s.Grab(context.Background(), ram, &blocked, nil, true, Origin{By: OriginOperator}); !took.Refused { + t.Error("forcing took a release set aside") + } + if took := (&Service{}).Grab(context.Background(), ram, &review, nil, true, Origin{}); !took.Unconfigured || took.Missing != MissingClient { + t.Errorf("no client: %+v", took) + } +} + +func TestGrab_AClientThatAlreadyHasItCounts(t *testing.T) { + client := &fakeClient{addErr: downloader.ErrDuplicate} + s := &Service{Grabber: &Downloader{Client: client, Landing: "/downloads"}} + c := Candidate{Verdict: Grab, InfoHash: "aaaa", Source: indexer.Result{Magnet: "magnet:?xt=urn:btih:aaaa"}} + if took := s.Grab(context.Background(), ram, &c, nil, false, Origin{By: OriginSchedule}); !took.Added { + t.Errorf("a duplicate: %+v", took) + } +} + +// A duplicate the client does not list for Minstrel sits under another +// category, where it would read as gone half an hour later. It is set +// aside rather than taken; forced, it is refused with the client's reason. +func TestGrab_ADuplicateHeldOutsideMinstrelIsNotTaken(t *testing.T) { + client := &fakeClient{addErr: fmt.Errorf("%w: already downloading Ohio Players (lidarr)", downloader.ErrDuplicate), elsewhere: true} + s := &Service{Grabber: &Downloader{Client: client, Landing: "/downloads"}} + c := Candidate{Verdict: Grab, InfoHash: "aaaa", Source: indexer.Result{Magnet: "magnet:?xt=urn:btih:aaaa"}} + took := s.Grab(context.Background(), ram, &c, nil, false, Origin{By: OriginSchedule}) + if took.Added || !took.Refused { + t.Fatalf("took %+v", took) + } + if c.Verdict != Blocked || !strings.Contains(c.Reason, "(already downloading Ohio Players (lidarr))") { + t.Errorf("candidate %s: %q", c.Verdict, c.Reason) + } + + forced := Candidate{Verdict: Grab, InfoHash: "aaaa", Source: indexer.Result{Magnet: "magnet:?xt=urn:btih:aaaa"}} + took = s.Grab(context.Background(), ram, &forced, nil, true, Origin{By: OriginOperator}) + if took.Added || !strings.Contains(took.Error, "outside Minstrel's category") { + t.Errorf("forced: %+v", took) + } +} + +func TestBetter_QualityThenSeedersThenYear(t *testing.T) { + flac := Candidate{Verdict: Grab, Tier: release.TierLossless, Seeders: 2} + mp3 := Candidate{Verdict: Grab, Tier: release.TierLossyHigh, Seeders: 90} + if !better(flac, mp3) || better(mp3, flac) { + t.Error("quality first among grabs") + } + busy := Candidate{Verdict: Grab, Tier: release.TierLossless, Seeders: 20} + if !better(busy, flac) { + t.Error("then seeders") + } + dated := Candidate{Verdict: Grab, Tier: release.TierLossless, Seeders: 2, YearAgrees: true} + if !better(dated, flac) { + t.Error("then the year") + } + if better(Candidate{Verdict: Review}, flac) || !better(flac, Candidate{Verdict: Review}) { + t.Error("a grab ranks above a review") + } +} + +func TestLandingFor_OneFolderPerTorrent(t *testing.T) { + if got := landingFor("/downloads", "abcd", "x"); got != "/downloads/abcd" { + t.Error(got) + } + if got := landingFor("/downloads", "", "Daft Punk: R.A.M. [FLAC]"); got != "/downloads/daft-punk-r-a-m--flac" { + t.Error(got) + } +} + +func TestDecide_AnUpgradeTakesOnlyABetterCopy(t *testing.T) { + up := ram + up.Floor, up.Held = release.TierLossyLow, "MP3, about 192 kbps" + same := Candidate{Release: release.Parse("Daft Punk - Random Access Memories [MP3 192]"), InfoHash: "aaaa", Seeders: 3} + decide(&same, up, anyQuality, nil) + if same.Verdict != Refused || same.Reasons[0].Code != CodeNotBetter || + same.Reason != "No better than the copy you have (MP3, about 192 kbps)." { + t.Errorf("same quality: %s %q", same.Verdict, same.Reason) + } + flac := Candidate{Release: release.Parse("Daft Punk - Random Access Memories [FLAC]"), InfoHash: "bbbb", Seeders: 3} + decide(&flac, up, anyQuality, nil) + if flac.Verdict != Grab { + t.Errorf("a FLAC over a 192 MP3: %s %q", flac.Verdict, flac.Reason) + } + // A video rip has no floor: any clean copy is better. + rip := ram + decide(&same, rip, anyQuality, nil) + if same.Verdict != Grab { + t.Errorf("no floor: %s %q", same.Verdict, same.Reason) + } +} diff --git a/internal/acquire/settings.go b/internal/acquire/settings.go new file mode 100644 index 00000000..e71284a3 --- /dev/null +++ b/internal/acquire/settings.go @@ -0,0 +1,185 @@ +// Package acquire turns the configured indexers and download clients into +// working clients. +// +// ONE BUILDER PER KIND. SourceFor makes a search source from a row and +// DownloadClient makes a download client, path mapping included. The grab, +// the importer, the stall watchdog and the settings page's Test all build +// through them, so a Test that passes vouches for the path real traffic takes. +// Clients are rebuilt from the row on every use, so a changed address or a new +// key takes effect on the next call. +package acquire + +import ( + "context" + "errors" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/exchange" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" + "git.fabledsword.com/bvandeusen/minstrel/internal/prowlarr" +) + +// Kinds: which list a row appears in. +const ( + KindSearch = "search" + KindDownload = "download" +) + +// Jobs a row can do. An exchange row does each its switch allows; every +// other row does the one its kind names. +const ( + JobSearch = "search" + JobDownload = "download" + JobEnrichment = "enrichment" +) + +// Protocols: which client a row builds. +const ( + ProtocolProwlarr = "prowlarr" + ProtocolTorznab = "torznab" + ProtocolQBittorrent = "qbittorrent" + ProtocolExchange = "exchange" +) + +// DefaultCategory files everything Minstrel adds to a download client. +const DefaultCategory = "minstrel" + +// addable is what a new row of a protocol starts as. +type addable struct { + kind string + label string + category string + timeoutMs int32 + perHour int32 +} + +// catalogue is every protocol an admin can add. Exchange starts unpaced +// (it searches its own database); the others at 120 searches an hour, which +// keeps a Prowlarr's indexers under the rate most trackers allow. +var catalogue = map[string]addable{ + ProtocolProwlarr: {kind: KindSearch, label: "Prowlarr", timeoutMs: 60000, perHour: 120}, + ProtocolTorznab: {kind: KindSearch, label: "Torznab feed", timeoutMs: 30000, perHour: 120}, + ProtocolQBittorrent: {kind: KindDownload, label: "qBittorrent", category: DefaultCategory, timeoutMs: 15000, perHour: 0}, + ProtocolExchange: {kind: KindDownload, label: "FabledExchange", category: DefaultCategory, timeoutMs: 30000, perHour: 0}, +} + +// Addable says whether protocol is one an admin can add. +func Addable(protocol string) bool { + _, ok := catalogue[protocol] + return ok +} + +// Searches says whether a row is a search source. +func Searches(row dbq.AcquisitionService) bool { + if row.Protocol == ProtocolExchange { + return row.ServesSearch + } + return row.Kind == KindSearch +} + +// Fetches says whether a row is a download client. +func Fetches(row dbq.AcquisitionService) bool { + if row.Protocol == ProtocolExchange { + return row.ServesDownloads + } + return row.Kind == KindDownload +} + +// Searcher is a search source. +type Searcher interface { + Search(ctx context.Context, q indexer.Query) ([]indexer.Result, error) +} + +// Fetcher fetches a result's .torrent through the source that offered it. +type Fetcher interface { + Download(ctx context.Context, r indexer.Result) ([]byte, error) +} + +// Source is one configured search source. +type Source struct { + Code string + Label string + // PerHour is the most searches the schedule sends it an hour; 0 = unpaced. + PerHour int + Search Searcher + // Fetch is nil for a source whose results are all magnets. + Fetch Fetcher +} + +// SourceFor builds the search source a row describes. +func SourceFor(row dbq.AcquisitionService) Source { + src := Source{Code: row.Code, Label: row.Label, PerHour: int(row.SearchesPerHour)} + timeout := msDuration(row.TimeoutMs) + switch row.Protocol { + case ProtocolProwlarr: + c := &prowlarr.Client{Config: prowlarr.Config{ + Code: row.Code, BaseURL: row.BaseUrl, APIKey: row.ApiKey, Timeout: timeout, + }} + src.Search, src.Fetch = c, c + case ProtocolExchange: + src.Search = ExchangeFor(row) + default: + c := &indexer.Torznab{Config: indexer.TorznabConfig{ + Code: row.Code, FeedURL: row.BaseUrl, APIKey: row.ApiKey, Timeout: timeout, + }} + src.Search, src.Fetch = c, c + } + return src +} + +// ExchangeFor builds the search and enrichment client for an exchange row. +func ExchangeFor(row dbq.AcquisitionService) *exchange.Client { + return &exchange.Client{ + Code: row.Code, + Config: exchange.Config{ + BaseURL: row.BaseUrl, APIKey: row.ApiKey, Timeout: msDuration(row.TimeoutMs), + }, + } +} + +// DownloadClient builds the download client a row describes, wrapped in its +// path mapping. +func DownloadClient(row dbq.AcquisitionService) (downloader.Driver, error) { + d, err := downloader.For(row.Protocol, downloader.Config{ + BaseURL: row.BaseUrl, Username: row.Username, Password: row.Password, + APIKey: row.ApiKey, Category: row.Category, Timeout: msDuration(row.TimeoutMs), + }) + if err != nil { + return nil, err + } + return downloader.Mapped(d, downloader.PathMap{Local: row.LocalPath, Remote: row.RemotePath}), nil +} + +// DownloadClientFor builds the download client a grab went to, by the code +// of its acquisition_services row. +func DownloadClientFor(ctx context.Context, pool *pgxpool.Pool, code string) (downloader.Driver, error) { + row, err := dbq.New(pool).GetAcquisitionService(ctx, code) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + return nil, fmt.Errorf("the download client %q is no longer set up", code) + } + return nil, err + } + return DownloadClient(row) +} + +// Sources builds every enabled search source. +func Sources(ctx context.Context, pool *pgxpool.Pool) ([]Source, error) { + rows, err := dbq.New(pool).ListUsableAcquisitionServices(ctx, JobSearch) + if err != nil { + return nil, err + } + out := make([]Source, len(rows)) + for i, r := range rows { + out[i] = SourceFor(r) + } + return out, nil +} + +func msDuration(ms int32) time.Duration { return time.Duration(ms) * time.Millisecond } diff --git a/internal/acquire/store.go b/internal/acquire/store.go new file mode 100644 index 00000000..0245be01 --- /dev/null +++ b/internal/acquire/store.go @@ -0,0 +1,191 @@ +package acquire + +import ( + "context" + "errors" + "fmt" + "net/url" + "strings" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +// ErrNotFound is a code with no row. +var ErrNotFound = errors.New("no such service") + +// ErrTooMany is a twenty-first row of one protocol. +var ErrTooMany = errors.New("there are already twenty of these") + +// InvalidError is a setting an admin has to change, in words for the +// settings card. +type InvalidError struct { + Field string + Message string +} + +func (e *InvalidError) Error() string { return e.Message } + +func invalid(field, msg string) error { return &InvalidError{Field: field, Message: msg} } + +// maxPerProtocol bounds the codes tried for a new row. Nobody runs twenty +// Prowlarrs; the bound keeps a loop over a full table finite. +const maxPerProtocol = 20 + +// Store reads and writes acquisition_services. +type Store struct { + pool *pgxpool.Pool +} + +// NewStore returns a Store over pool. +func NewStore(pool *pgxpool.Pool) *Store { return &Store{pool: pool} } + +// List returns every row without its secrets. +func (s *Store) List(ctx context.Context) ([]dbq.ListAcquisitionServicesRow, error) { + return dbq.New(s.pool).ListAcquisitionServices(ctx) +} + +// Get returns one row, secrets included: for building a client only. +func (s *Store) Get(ctx context.Context, code string) (dbq.AcquisitionService, error) { + row, err := dbq.New(s.pool).GetAcquisitionService(ctx, code) + if errors.Is(err, pgx.ErrNoRows) { + return row, ErrNotFound + } + return row, err +} + +// Create adds a row of protocol, switched off and with no address. Its code +// is the protocol for the first, then protocol-2, protocol-3. +func (s *Store) Create(ctx context.Context, protocol string) (dbq.AcquisitionService, error) { + a, ok := catalogue[protocol] + if !ok { + return dbq.AcquisitionService{}, invalid("protocol", "One of prowlarr, torznab, qbittorrent, exchange.") + } + q := dbq.New(s.pool) + for n := 1; n <= maxPerProtocol; n++ { + code, label := protocol, a.label + if n > 1 { + code, label = fmt.Sprintf("%s-%d", protocol, n), fmt.Sprintf("%s %d", a.label, n) + } + row, err := q.CreateAcquisitionService(ctx, dbq.CreateAcquisitionServiceParams{ + Code: code, Kind: a.kind, Protocol: protocol, Label: label, + Category: a.category, TimeoutMs: a.timeoutMs, SearchesPerHour: a.perHour, + }) + if errors.Is(err, pgx.ErrNoRows) { + continue + } + return row, err + } + return dbq.AcquisitionService{}, ErrTooMany +} + +// Update is everything on the settings form except the credentials. +type Update struct { + Label string + Enabled bool + BaseURL string + Username string + Category string + TimeoutMs int32 + ServesSearch bool + ServesDownloads bool + ServesEnrichment bool + LocalPath string + RemotePath string + SearchesPerHour int32 +} + +// Save validates u and writes it to code's row. +func (s *Store) Save(ctx context.Context, code string, u Update) (dbq.AcquisitionService, error) { + u.Label = strings.TrimSpace(u.Label) + u.BaseURL = strings.TrimSpace(u.BaseURL) + u.Username = strings.TrimSpace(u.Username) + u.Category = strings.TrimSpace(u.Category) + u.LocalPath = strings.TrimSpace(u.LocalPath) + u.RemotePath = strings.TrimSpace(u.RemotePath) + if err := u.validate(); err != nil { + return dbq.AcquisitionService{}, err + } + row, err := dbq.New(s.pool).UpdateAcquisitionService(ctx, dbq.UpdateAcquisitionServiceParams{ + Code: code, Label: u.Label, Enabled: u.Enabled, BaseUrl: u.BaseURL, + Username: u.Username, Category: u.Category, TimeoutMs: u.TimeoutMs, + ServesSearch: u.ServesSearch, ServesDownloads: u.ServesDownloads, + ServesEnrichment: u.ServesEnrichment, LocalPath: u.LocalPath, + RemotePath: u.RemotePath, SearchesPerHour: u.SearchesPerHour, + }) + if errors.Is(err, pgx.ErrNoRows) { + return row, ErrNotFound + } + return row, err +} + +// validate mirrors the table's CHECKs, so a bad value is named in the card's +// own terms before Postgres refuses it. +func (u Update) validate() error { + if u.Label == "" { + return invalid("label", "Name: give it a name.") + } + if u.Enabled && u.BaseURL == "" { + return invalid("base_url", "Address: give the address before switching this on.") + } + if u.BaseURL != "" { + parsed, err := url.Parse(u.BaseURL) + if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Host == "" { + return invalid("base_url", "Address: start with http:// or https://.") + } + } + if u.TimeoutMs < 1000 || u.TimeoutMs > 300000 { + return invalid("timeout_ms", "Give up after: between 1000 and 300000 milliseconds.") + } + if !strings.HasPrefix(u.LocalPath, "/") { + return invalid("local_path", "Downloads folder as Minstrel sees it: a full path, starting with /.") + } + if u.RemotePath != "" && !strings.HasPrefix(u.RemotePath, "/") { + return invalid("remote_path", "Downloads folder as the client sees it: a full path starting with /, or empty when it is the same path.") + } + if u.SearchesPerHour < 0 || u.SearchesPerHour > 3600 { + return invalid("searches_per_hour", "Searches per hour: between 0 and 3600. 0 means no limit.") + } + return nil +} + +// Delete removes code's row. +func (s *Store) Delete(ctx context.Context, code string) error { + n, err := dbq.New(s.pool).DeleteAcquisitionService(ctx, code) + if err != nil { + return err + } + if n == 0 { + return ErrNotFound + } + return nil +} + +// SetAPIKey stores a key, trimmed; empty clears it. +func (s *Store) SetAPIKey(ctx context.Context, code, key string) error { + n, err := dbq.New(s.pool).SetAcquisitionServiceAPIKey(ctx, dbq.SetAcquisitionServiceAPIKeyParams{ + Code: code, ApiKey: strings.TrimSpace(key), + }) + return affected(n, err) +} + +// SetPassword stores a password as typed (spaces are part of a password); +// empty clears it. +func (s *Store) SetPassword(ctx context.Context, code, password string) error { + n, err := dbq.New(s.pool).SetAcquisitionServicePassword(ctx, dbq.SetAcquisitionServicePasswordParams{ + Code: code, Password: password, + }) + return affected(n, err) +} + +func affected(n int64, err error) error { + if err != nil { + return err + } + if n == 0 { + return ErrNotFound + } + return nil +} diff --git a/internal/acquire/store_test.go b/internal/acquire/store_test.go new file mode 100644 index 00000000..37c369da --- /dev/null +++ b/internal/acquire/store_test.go @@ -0,0 +1,108 @@ +package acquire + +import ( + "context" + "errors" + "io" + "log/slog" + "os" + "testing" + + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" +) + +func newPool(t *testing.T) *pgxpool.Pool { + t.Helper() + if testing.Short() { + t.Skip("skipping integration test in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(context.Background(), dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + dbtest.ResetDB(t, pool) + return pool +} + +func TestStore_AddsConfiguresAndRemoves_Integration(t *testing.T) { + ctx := context.Background() + s := NewStore(newPool(t)) + + first, err := s.Create(ctx, ProtocolExchange) + if err != nil { + t.Fatalf("create: %v", err) + } + second, err := s.Create(ctx, ProtocolExchange) + if err != nil { + t.Fatalf("create a second: %v", err) + } + if first.Code != "exchange" || second.Code != "exchange-2" || second.Label != "FabledExchange 2" { + t.Errorf("codes = %q, %q (%q); want exchange, exchange-2", first.Code, second.Code, second.Label) + } + // A new exchange is off, does every job, files under minstrel and is + // unpaced. + if first.Enabled || first.Kind != KindDownload || !first.ServesSearch || !first.ServesDownloads || + !first.ServesEnrichment || first.Category != DefaultCategory || first.SearchesPerHour != 0 || first.LocalPath != "/downloads" { + t.Errorf("a new exchange = %+v", first) + } + if _, err := s.Create(ctx, "transmission"); !errors.As(err, new(*InvalidError)) { + t.Errorf("an unknown protocol: err = %v", err) + } + + u := Update{ + Label: "Exchange", Enabled: true, BaseURL: "http://exchange:8080", Category: "minstrel", + TimeoutMs: 20000, ServesSearch: true, LocalPath: "/downloads", RemotePath: "/data/torrents", + } + saved, err := s.Save(ctx, first.Code, u) + if err != nil { + t.Fatalf("save: %v", err) + } + if !saved.Enabled || saved.ServesDownloads || saved.RemotePath != "/data/torrents" { + t.Errorf("saved = %+v", saved) + } + if _, err := s.Save(ctx, "nope", u); !errors.Is(err, ErrNotFound) { + t.Errorf("saving a missing row: err = %v", err) + } + + // Secrets are write-only: the list says whether one is set, never what. + if err := s.SetAPIKey(ctx, first.Code, " key "); err != nil { + t.Fatalf("set key: %v", err) + } + full, err := s.Get(ctx, first.Code) + if err != nil || full.ApiKey != "key" { + t.Errorf("stored key = %q (err %v), want it trimmed", full.ApiKey, err) + } + rows, err := s.List(ctx) + if err != nil || len(rows) != 2 { + t.Fatalf("list: %d rows, err %v", len(rows), err) + } + for _, r := range rows { + if r.Code == first.Code && !r.HasApiKey { + t.Error("the list should say a key is set") + } + } + + // Only the exchange with search switched on is a search source. + srcs, err := Sources(ctx, s.pool) + if err != nil || len(srcs) != 1 || srcs[0].Code != first.Code { + t.Errorf("sources = %+v, err %v; want only the enabled exchange", srcs, err) + } + + if err := s.Delete(ctx, second.Code); err != nil { + t.Fatalf("delete: %v", err) + } + if err := s.Delete(ctx, second.Code); !errors.Is(err, ErrNotFound) { + t.Errorf("deleting twice: err = %v", err) + } +} diff --git a/internal/acquire/test.go b/internal/acquire/test.go new file mode 100644 index 00000000..9ec6d07b --- /dev/null +++ b/internal/acquire/test.go @@ -0,0 +1,134 @@ +package acquire + +import ( + "context" + "errors" + "fmt" + "os" + "strings" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/exchange" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// TestResult is what the settings card says after Test. +type TestResult struct { + OK bool `json:"ok"` + Detail string `json:"detail"` +} + +// testQuery is a search every music indexer should answer. +const testQuery = "Radiohead" + +// Test reaches the service a row describes, through the same builders real +// traffic uses. It runs whether or not the row is switched on: paste the +// address, test it, then switch it on. +func Test(ctx context.Context, row dbq.AcquisitionService) TestResult { + if strings.TrimSpace(row.BaseUrl) == "" { + return TestResult{Detail: "No address is set, so there is nothing to reach yet."} + } + if row.Protocol == ProtocolExchange { + return testExchange(ctx, row) + } + if row.Kind == KindSearch { + return testSearch(ctx, row) + } + return testDownload(ctx, row) +} + +// testExchange runs each job the row has switched on and reports them all. +func testExchange(ctx context.Context, row dbq.AcquisitionService) TestResult { + var parts []string + ok := true + run := func(name string, on bool, test func() TestResult) { + if !on { + return + } + r := test() + ok = ok && r.OK + parts = append(parts, name+": "+r.Detail) + } + run("Downloads", row.ServesDownloads, func() TestResult { return testDownload(ctx, row) }) + run("Search", row.ServesSearch, func() TestResult { return testSearch(ctx, row) }) + run("Release details", row.ServesEnrichment, func() TestResult { return testEnrichment(ctx, row) }) + if len(parts) == 0 { + return TestResult{Detail: "Every job of this FabledExchange is switched off, so there is nothing to test."} + } + return TestResult{OK: ok, Detail: strings.Join(parts, " ")} +} + +// testSearch runs a real search. +func testSearch(ctx context.Context, row dbq.AcquisitionService) TestResult { + src := SourceFor(row) + results, err := src.Search.Search(ctx, indexer.Query{Text: testQuery, Categories: indexer.MusicCategories}) + switch { + case errors.Is(err, indexer.ErrUnconfigured): + return TestResult{Detail: "Add this service's address, and the API key it shows on its own settings page."} + case errors.Is(err, indexer.ErrAuth): + return TestResult{Detail: indexer.FirstLine(err.Error()) + ". Check it on the service's own settings page."} + case err != nil: + return TestResult{Detail: indexer.FirstLine(err.Error())} + case len(results) == 0: + return TestResult{OK: true, Detail: "Connected, but a test search for music found nothing. Check that it has an indexer with music categories."} + } + return TestResult{OK: true, Detail: fmt.Sprintf("Connected. %d results for a test search.", len(results))} +} + +// testDownload lists the client's torrents, which proves the session and not +// only the login, then checks the downloads folder from Minstrel's side. +func testDownload(ctx context.Context, row dbq.AcquisitionService) TestResult { + c, err := DownloadClient(row) + if err != nil { + return TestResult{Detail: err.Error()} + } + ts, err := c.Torrents(ctx) + if err != nil { + return TestResult{Detail: indexer.FirstLine(err.Error())} + } + detail := fmt.Sprintf("Connected. %d torrents.", len(ts)) + if row.Category != "" { + detail = fmt.Sprintf("Connected. %d torrents in the %q category.", len(ts), row.Category) + } + if up, err := c.Connected(ctx); err == nil && !up { + detail += " The client reports it is not connected to any peers, so downloads will stall." + } + if problem := landingProblem(row.LocalPath); problem != "" { + return TestResult{Detail: detail + " " + problem} + } + return TestResult{OK: true, Detail: detail} +} + +// landingProblem checks Minstrel can see and write the downloads folder: it +// moves finished downloads out of it. Only the local path can be checked; +// the client's own name for it cannot be seen from here. +func landingProblem(dir string) string { + info, err := os.Stat(dir) + switch { + case err != nil: + return fmt.Sprintf("But Minstrel cannot see its downloads folder at %s. Mount the download client's downloads folder there, or change where Minstrel looks for it.", dir) + case !info.IsDir(): + return fmt.Sprintf("But %s is a file, not the downloads folder.", dir) + } + probe, err := os.CreateTemp(dir, ".minstrel-probe-*") + if err != nil { + return fmt.Sprintf("But Minstrel cannot write to %s, so it could not move a finished download out of it. Check the mount is not read-only and Minstrel's user can write there.", dir) + } + name := probe.Name() + _ = probe.Close() + _ = os.Remove(name) + return "" +} + +// testEnrichment asks about a hash no torrent has. A JSON "not crawled" is a +// healthy answer; a 404 from something that is not Exchange is not. +func testEnrichment(ctx context.Context, row dbq.AcquisitionService) TestResult { + _, err := ExchangeFor(row).Torrent(ctx, strings.Repeat("0", 40)) + switch { + case err == nil, errors.Is(err, exchange.ErrUnknown): + return TestResult{OK: true, Detail: "Connected. The index answered."} + case errors.Is(err, exchange.ErrUnconfigured): + return TestResult{Detail: "No address is set, so there is nothing to reach yet."} + } + return TestResult{Detail: indexer.FirstLine(err.Error())} +} diff --git a/internal/api/admin_acoustid.go b/internal/api/admin_acoustid.go index 6a3a433a..69c88f17 100644 --- a/internal/api/admin_acoustid.go +++ b/internal/api/admin_acoustid.go @@ -11,7 +11,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/library" ) -// AcoustID recording-id lookup, admin side (M401 #3922). +// AcoustID recording-id lookup, admin side. // acoustIDUnsettledLimit caps the admin list of tracks a lookup could not // settle. It is a list to read, not to page through. @@ -96,7 +96,8 @@ type acoustIDSettingsReq struct { APIKey *string `json:"api_key"` } -// handleUpdateAcoustIDSettings implements PUT /api/admin/library/acoustid-settings. +// handleUpdateAcoustIDSettings implements PUT +// /api/admin/library/acoustid-settings. func (h *handlers) handleUpdateAcoustIDSettings(w http.ResponseWriter, r *http.Request) { var req acoustIDSettingsReq if err := json.NewDecoder(r.Body).Decode(&req); err != nil { @@ -143,9 +144,9 @@ type acoustIDUnsettledResp struct { LookedUpAt time.Time `json:"looked_up_at"` } -// handleListUnsettledAcoustID implements GET /api/admin/library/acoustid/unsettled: -// the tracks a lookup could not settle (ambiguous or failed), so they can be -// found and tagged by hand. +// handleListUnsettledAcoustID implements GET +// /api/admin/library/acoustid/unsettled: the tracks a lookup could not settle +// (ambiguous or failed), so they can be found and tagged by hand. func (h *handlers) handleListUnsettledAcoustID(w http.ResponseWriter, r *http.Request) { rows, err := dbq.New(h.pool).ListUnsettledAcoustIDLookups(r.Context(), acoustIDUnsettledLimit) if err != nil { diff --git a/internal/api/admin_acquisition.go b/internal/api/admin_acquisition.go new file mode 100644 index 00000000..f6144e61 --- /dev/null +++ b/internal/api/admin_acquisition.go @@ -0,0 +1,210 @@ +package api + +import ( + "encoding/json" + "errors" + "io" + "net/http" + + "github.com/go-chi/chi/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +// Indexers and download clients, admin side. Rows an admin +// adds and configures; nothing is seeded. Secrets are write-only: no response +// carries one, only whether it is set. + +// acquisitionBodyLimit caps a settings body. The largest is a full form of +// short strings. +const acquisitionBodyLimit = 64 << 10 + +type acquisitionServiceResp struct { + Code string `json:"code"` + Kind string `json:"kind"` + Protocol string `json:"protocol"` + Label string `json:"label"` + Enabled bool `json:"enabled"` + BaseURL string `json:"base_url"` + HasAPIKey bool `json:"has_api_key"` + HasPassword bool `json:"has_password"` + Username string `json:"username"` + Category string `json:"category"` + TimeoutMs int32 `json:"timeout_ms"` + ServesSearch bool `json:"serves_search"` + ServesDownloads bool `json:"serves_downloads"` + ServesEnrichment bool `json:"serves_enrichment"` + LocalPath string `json:"local_path"` + RemotePath string `json:"remote_path"` + SearchesPerHour int32 `json:"searches_per_hour"` + // Usable is switched on and addressed: the state the jobs read. + Usable bool `json:"usable"` +} + +func acquisitionRespOf(r dbq.AcquisitionService) acquisitionServiceResp { + return acquisitionServiceResp{ + Code: r.Code, Kind: r.Kind, Protocol: r.Protocol, Label: r.Label, + Enabled: r.Enabled, BaseURL: r.BaseUrl, + HasAPIKey: r.ApiKey != "", HasPassword: r.Password != "", + Username: r.Username, Category: r.Category, TimeoutMs: r.TimeoutMs, + ServesSearch: r.ServesSearch, ServesDownloads: r.ServesDownloads, + ServesEnrichment: r.ServesEnrichment, + LocalPath: r.LocalPath, RemotePath: r.RemotePath, + SearchesPerHour: r.SearchesPerHour, + Usable: r.Enabled && r.BaseUrl != "", + } +} + +// handleListAcquisitionServices implements GET /api/admin/acquisition/services. +func (h *handlers) handleListAcquisitionServices(w http.ResponseWriter, r *http.Request) { + rows, err := h.acquisition.List(r.Context()) + if err != nil { + writeErrWithLog(w, h.logger, "admin: list acquisition services", apierror.Internal(err)) + return + } + out := make([]acquisitionServiceResp, len(rows)) + for i, row := range rows { + out[i] = acquisitionRespOf(dbq.AcquisitionService{ + Code: row.Code, Kind: row.Kind, Protocol: row.Protocol, Label: row.Label, + Enabled: row.Enabled, BaseUrl: row.BaseUrl, Username: row.Username, + Category: row.Category, TimeoutMs: row.TimeoutMs, + ServesSearch: row.ServesSearch, ServesDownloads: row.ServesDownloads, + ServesEnrichment: row.ServesEnrichment, LocalPath: row.LocalPath, + RemotePath: row.RemotePath, SearchesPerHour: row.SearchesPerHour, + }) + out[i].HasAPIKey, out[i].HasPassword = row.HasApiKey, row.HasPassword + } + writeJSON(w, http.StatusOK, map[string]any{"services": out}) +} + +// handleCreateAcquisitionService implements POST +// /api/admin/acquisition/services with {"protocol": +// "prowlarr"|"torznab"|"qbittorrent"|"exchange"}. The row starts switched off, +// with no address. +func (h *handlers) handleCreateAcquisitionService(w http.ResponseWriter, r *http.Request) { + var body struct { + Protocol string `json:"protocol"` + } + if !decodeAcquisitionBody(w, r, &body) { + return + } + row, err := h.acquisition.Create(r.Context(), body.Protocol) + if err != nil { + h.writeAcquisitionErr(w, "admin: create acquisition service", err) + return + } + writeJSON(w, http.StatusCreated, acquisitionRespOf(row)) +} + +type updateAcquisitionBody struct { + Label string `json:"label"` + Enabled bool `json:"enabled"` + BaseURL string `json:"base_url"` + Username string `json:"username"` + Category string `json:"category"` + TimeoutMs int32 `json:"timeout_ms"` + ServesSearch bool `json:"serves_search"` + ServesDownloads bool `json:"serves_downloads"` + ServesEnrichment bool `json:"serves_enrichment"` + LocalPath string `json:"local_path"` + RemotePath string `json:"remote_path"` + SearchesPerHour int32 `json:"searches_per_hour"` +} + +// handleUpdateAcquisitionService implements PUT +// /api/admin/acquisition/services/{code}: the whole form except credentials. +func (h *handlers) handleUpdateAcquisitionService(w http.ResponseWriter, r *http.Request) { + var body updateAcquisitionBody + if !decodeAcquisitionBody(w, r, &body) { + return + } + row, err := h.acquisition.Save(r.Context(), chi.URLParam(r, "code"), acquire.Update(body)) + if err != nil { + h.writeAcquisitionErr(w, "admin: update acquisition service", err) + return + } + writeJSON(w, http.StatusOK, acquisitionRespOf(row)) +} + +// handleDeleteAcquisitionService implements DELETE +// /api/admin/acquisition/services/{code}. +func (h *handlers) handleDeleteAcquisitionService(w http.ResponseWriter, r *http.Request) { + if err := h.acquisition.Delete(r.Context(), chi.URLParam(r, "code")); err != nil { + h.writeAcquisitionErr(w, "admin: delete acquisition service", err) + return + } + w.WriteHeader(http.StatusNoContent) +} + +// handleTestAcquisitionService implements POST +// /api/admin/acquisition/services/{code}/test. POST because it reaches +// another machine, which a prefetch must not trigger. Always 200 with +// {ok, detail}: a failed connection is the answer, not an error. +func (h *handlers) handleTestAcquisitionService(w http.ResponseWriter, r *http.Request) { + row, err := h.acquisition.Get(r.Context(), chi.URLParam(r, "code")) + if err != nil { + h.writeAcquisitionErr(w, "admin: test acquisition service", err) + return + } + writeJSON(w, http.StatusOK, acquire.Test(r.Context(), row)) +} + +// handleSetAcquisitionSecret implements PUT and DELETE on +// /api/admin/acquisition/services/{code}/api-key and .../password. PUT takes +// {"value": "..."}; DELETE clears it. 204 with no body either way: echoing +// anything here is how a secret ends up in a response. +func (h *handlers) handleSetAcquisitionSecret(field string) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + value := "" + if r.Method == http.MethodPut { + var body struct { + Value string `json:"value"` + } + if !decodeAcquisitionBody(w, r, &body) { + return + } + if body.Value == "" { + writeErr(w, apierror.BadRequest("invalid_setting", "Paste a value, or use Remove to clear the one that is set.")) + return + } + value = body.Value + } + code := chi.URLParam(r, "code") + var err error + if field == "password" { + err = h.acquisition.SetPassword(r.Context(), code, value) + } else { + err = h.acquisition.SetAPIKey(r.Context(), code, value) + } + if err != nil { + h.writeAcquisitionErr(w, "admin: set acquisition "+field, err) + return + } + w.WriteHeader(http.StatusNoContent) + } +} + +func decodeAcquisitionBody(w http.ResponseWriter, r *http.Request, into any) bool { + err := json.NewDecoder(http.MaxBytesReader(w, r.Body, acquisitionBodyLimit)).Decode(into) + if err != nil && !errors.Is(err, io.EOF) { + writeErr(w, apierror.BadRequest("invalid_body", "malformed JSON")) + return false + } + return true +} + +func (h *handlers) writeAcquisitionErr(w http.ResponseWriter, msg string, err error) { + var invalid *acquire.InvalidError + switch { + case errors.As(err, &invalid): + writeErr(w, apierror.BadRequest("invalid_setting", invalid.Message)) + case errors.Is(err, acquire.ErrNotFound): + writeErr(w, apierror.NotFound("service")) + case errors.Is(err, acquire.ErrTooMany): + writeErr(w, apierror.Conflict("too_many_services", "There are already twenty of these. Remove one you no longer use first.")) + default: + writeErrWithLog(w, h.logger, msg, apierror.Internal(err)) + } +} diff --git a/internal/api/admin_cover_sources.go b/internal/api/admin_cover_sources.go index 5c9c1171..924eca90 100644 --- a/internal/api/admin_cover_sources.go +++ b/internal/api/admin_cover_sources.go @@ -70,7 +70,8 @@ type updateCoverSourceResp struct { VersionBumped bool `json:"version_bumped"` } -// handleUpdateCoverSource implements PATCH /api/admin/cover-sources/{provider_id}. +// handleUpdateCoverSource implements PATCH +// /api/admin/cover-sources/{provider_id}. func (h *handlers) handleUpdateCoverSource(w http.ResponseWriter, r *http.Request) { id := chi.URLParam(r, "provider_id") @@ -128,7 +129,8 @@ type testCoverSourceResp struct { Error string `json:"error,omitempty"` } -// handleTestCoverSource implements POST /api/admin/cover-sources/{provider_id}/test. +// handleTestCoverSource implements POST +// /api/admin/cover-sources/{provider_id}/test. func (h *handlers) handleTestCoverSource(w http.ResponseWriter, r *http.Request) { id := chi.URLParam(r, "provider_id") diff --git a/internal/api/admin_cover_sources_test.go b/internal/api/admin_cover_sources_test.go index 3a0de4e7..0da0d4f9 100644 --- a/internal/api/admin_cover_sources_test.go +++ b/internal/api/admin_cover_sources_test.go @@ -103,7 +103,8 @@ func TestAdminListCoverSources_ReturnsRegisteredProviders(t *testing.T) { if p.APIKeySet { t.Error("api_key_set = true, want false (no key set)") } - // apiTestAlbumProvider implements AlbumCoverProvider but not TestableProvider. + // apiTestAlbumProvider implements AlbumCoverProvider but not + // TestableProvider. if p.Testable { t.Error("testable = true, want false") } diff --git a/internal/api/admin_coverage.go b/internal/api/admin_coverage.go index 7bbe259c..851d32c7 100644 --- a/internal/api/admin_coverage.go +++ b/internal/api/admin_coverage.go @@ -39,11 +39,12 @@ func (h *handlers) handleGetLibraryCoverage(w http.ResponseWriter, r *http.Reque }) } -// fingerprintCoverageResp is the wire shape for GET /api/admin/library/fingerprints. -// fingerprinted + rejected + pending = total. Missing tracks are not counted: -// there is no file to fingerprint. Enabled travels with the counts because with -// fingerprinting off (#3913) pending never shrinks, and a gauge that implies -// progress would be promising work nothing is doing. +// fingerprintCoverageResp is the wire shape for GET +// /api/admin/library/fingerprints. fingerprinted + rejected + pending = total. +// Missing tracks are not counted: there is no file to fingerprint. Enabled +// travels with the counts because with fingerprinting off pending never +// shrinks, and a gauge that implies progress would be promising work nothing is +// doing. type fingerprintCoverageResp struct { Total int64 `json:"total"` Fingerprinted int64 `json:"fingerprinted"` @@ -70,7 +71,7 @@ func (h *handlers) fingerprintCoverage(ctx context.Context) (fingerprintCoverage } // handleGetFingerprintCoverage implements GET /api/admin/library/fingerprints: -// how far the fingerprint backfill (#3908) has got. The backfill is its own +// how far the fingerprint backfill has got. The backfill is its own // worker spanning many passes, with no scan run to attach a tally to, so its // progress is read live here. Always 200; zeros on an empty library. func (h *handlers) handleGetFingerprintCoverage(w http.ResponseWriter, r *http.Request) { diff --git a/internal/api/admin_coverage_test.go b/internal/api/admin_coverage_test.go index 4fbd4e0c..4a42c647 100644 --- a/internal/api/admin_coverage_test.go +++ b/internal/api/admin_coverage_test.go @@ -80,11 +80,9 @@ func TestAdminLibraryCoverage_MixedRowsReturnCorrectBuckets(t *testing.T) { mbid5 := "55555555-5555-5555-5555-555555555555" mbid6 := "66666666-6666-6666-6666-666666666666" mbid7 := "77777777-7777-7777-7777-777777777777" - // Cover every cover_art_source value the migrations allow so a future - // addition without updating the rollup query trips this test — - // regression guard for drift #557, the gap that hid #556 (deezer + - // lastfm omission introduced by migration 0020 but never wired into - // the rollup whitelist). + // Cover every cover_art_source value the migrations allow, so a source + // added to the CHECK but not to the rollup query's whitelist trips this + // test. rows := []seed{ {title: "WithArtSidecar", source: &sourceSidecar, mbid: &mbid1}, // with_art {title: "WithArtMbcaa", source: &sourceMbcaa, mbid: &mbid2}, // with_art diff --git a/internal/api/admin_covers.go b/internal/api/admin_covers.go index d354ce31..b9856f23 100644 --- a/internal/api/admin_covers.go +++ b/internal/api/admin_covers.go @@ -19,8 +19,9 @@ type adminCoverRefetchResp struct { CoverArtSource *string `json:"cover_art_source"` } -// handleAdminAlbumRefetchCover implements POST /api/admin/albums/{id}/cover/refetch. -// Synchronously clears + re-runs the enricher for one album. +// handleAdminAlbumRefetchCover implements POST +// /api/admin/albums/{id}/cover/refetch. Synchronously clears + re-runs the +// enricher for one album. func (h *handlers) handleAdminAlbumRefetchCover(w http.ResponseWriter, r *http.Request) { albumID, ok := parseUUID(chi.URLParam(r, "id")) if !ok { @@ -51,9 +52,9 @@ type adminBulkRefetchResp struct { Started bool `json:"started"` } -// handleAdminBulkRefetchCovers implements POST /api/admin/covers/refetch-missing. -// Returns immediately; the drain runs UNBOUNDED in a background goroutine -// (#388 — no global cap; remote providers self-throttle per provider, local +// handleAdminBulkRefetchCovers implements POST +// /api/admin/covers/refetch-missing. Returns immediately; the drain runs +// UNBOUNDED in a background goroutine (remote providers pace themselves, local // sources run at disk speed). The count is unknowable synchronously, so the // response just acknowledges the drain started. func (h *handlers) handleAdminBulkRefetchCovers(w http.ResponseWriter, _ *http.Request) { diff --git a/internal/api/admin_duplicates.go b/internal/api/admin_duplicates.go index 0bc48c42..970478f6 100644 --- a/internal/api/admin_duplicates.go +++ b/internal/api/admin_duplicates.go @@ -14,15 +14,15 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/audit" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" "git.fabledsword.com/bvandeusen/minstrel/internal/library" - "git.fabledsword.com/bvandeusen/minstrel/internal/tracks" ) // duplicateMemberView is one copy in a proposed duplicate group. LikeCount and // PlayCount span every user: the report is admin-only, and what a copy carries -// is the fact the operator weighs when choosing which to keep. +// is the fact an admin weighs when choosing which to keep. type duplicateMemberView struct { TrackID string `json:"track_id"` Title string `json:"title"` + ArtistID string `json:"artist_id"` ArtistName string `json:"artist_name"` AlbumID string `json:"album_id"` AlbumTitle string `json:"album_title"` @@ -33,17 +33,16 @@ type duplicateMemberView struct { AddedAt string `json:"added_at"` LikeCount int64 `json:"like_count"` PlayCount int64 `json:"play_count"` - // LidarrState is Lidarr's view of the file as of the resolver's last pass - // (M498): "tracked" (mapped to a track, so it cannot be removed without - // Lidarr downloading it again), "unmapped", or null when not asked. - LidarrState *string `json:"lidarr_state"` - DiscNumber *int32 `json:"disc_number"` - TrackNumber *int32 `json:"track_number"` + DiscNumber *int32 `json:"disc_number"` + TrackNumber *int32 `json:"track_number"` + // StreamURL lets the review play each copy, to hear whether they are the + // same song. + StreamURL string `json:"stream_url"` } // duplicateGroupView is one proposal. SurvivorTrackID and SurvivorReason are // the copy the report proposes keeping and the rule that chose it -// (library.ProposeSurvivor) — a default the merge (#3911) lets the operator +// (library.ProposeSurvivor) — a default the merge lets an admin // override. type duplicateGroupView struct { ID string `json:"id"` @@ -53,7 +52,7 @@ type duplicateGroupView struct { SurvivorTrackID string `json:"survivor_track_id"` SurvivorReason string `json:"survivor_reason"` // Class is the resolver's verdict (library.DuplicateClass), null until its - // first pass; ResolveNote is why it left the group for the operator. + // first pass; ResolveNote is why it left the group for an admin. Class *string `json:"class"` ResolveNote *string `json:"resolve_note"` Members []duplicateMemberView `json:"members"` @@ -79,8 +78,8 @@ type duplicateViewCounts struct { Resolved int64 `json:"resolved"` } -// Report views (M498). cross_release groups are the same song on different -// releases, kept on purpose; everything else pending needs the operator. +// Report views. cross_release groups are the same song on different +// releases, kept on purpose; everything else pending needs an admin. const ( duplicateViewReview = "review" duplicateViewCrossRelease = "cross_release" @@ -98,12 +97,13 @@ type adminDuplicatesResponse struct { Groups []duplicateGroupView `json:"groups"` } -// handleListDuplicates implements GET /api/admin/library/duplicates (#3912). -// ?view=review (the default) or cross_release picks the tab (M498). +// handleListDuplicates implements GET /api/admin/library/duplicates. +// ?view=review (the default) or cross_release picks the tab. // // Read-only. The sweep's state and the fingerprint backfill's progress travel // with the groups because an empty report means three different things — still -// fingerprinting, never swept, or swept and clean — and the page has to say which. +// fingerprinting, never swept, or swept and clean — and the page has to say +// which. func (h *handlers) handleListDuplicates(w http.ResponseWriter, r *http.Request) { limit, offset, err := parsePaging(r.URL.Query()) if err != nil { @@ -170,7 +170,9 @@ func (h *handlers) handleListDuplicates(w http.ResponseWriter, r *http.Request) } // resolvedActions are the audit actions the "Resolved automatically" tab lists. -var resolvedActions = []string{string(audit.ActionDuplicateMerge), string(audit.ActionLidarrReleaseChange)} +var resolvedActions = []string{ + string(audit.ActionDuplicateMerge), string(audit.ActionDuplicateDismiss), string(audit.ActionDuplicateWrongFile), +} func duplicateCounts(ctx context.Context, q *dbq.Queries) (duplicateViewCounts, error) { var c duplicateViewCounts @@ -185,9 +187,8 @@ func duplicateCounts(ctx context.Context, q *dbq.Queries) (duplicateViewCounts, return c, err } -// resolvedActivityView is one thing the resolver did on its own: a merge -// (action duplicate_merge) or a Lidarr release change (lidarr_release_change). -// Details is the audit row's metadata, which names the files or releases. +// resolvedActivityView is one merge the resolver made on its own (action +// duplicate_merge). Details is the audit row's metadata, which names the files. type resolvedActivityView struct { ID string `json:"id"` Action string `json:"action"` @@ -203,7 +204,7 @@ type resolvedActivityResponse struct { } // handleListResolvedDuplicates implements GET -// /api/admin/library/duplicates/resolved (M498): what the resolver did by +// /api/admin/library/duplicates/resolved: what the resolver did by // itself, newest first, read from the audit log so it outlives the groups. func (h *handlers) handleListResolvedDuplicates(w http.ResponseWriter, r *http.Request) { limit, offset, err := parsePaging(r.URL.Query()) @@ -281,6 +282,7 @@ func foldDuplicateGroups(rows []dbq.ListPendingDuplicateGroupMembersRow) []dupli groups[n].Members = append(groups[n].Members, duplicateMemberView{ TrackID: trackID, Title: row.Title, + ArtistID: uuidToString(row.ArtistID), ArtistName: row.ArtistName, AlbumID: uuidToString(row.AlbumID), AlbumTitle: row.AlbumTitle, @@ -291,14 +293,13 @@ func foldDuplicateGroups(rows []dbq.ListPendingDuplicateGroupMembersRow) []dupli AddedAt: formatTimestamp(row.AddedAt), LikeCount: row.LikeCount, PlayCount: row.PlayCount, - LidarrState: row.LidarrState, DiscNumber: row.DiscNumber, TrackNumber: row.TrackNumber, + StreamURL: streamURL(row.TrackID), }) candidates[n] = append(candidates[n], library.SurvivorCandidate{ TrackID: trackID, FileFormat: row.FileFormat, FileSize: row.FileSize, AddedAt: row.AddedAt.Time, - LidarrTracked: row.LidarrState != nil && *row.LidarrState == "tracked", - TagFit: library.TagFitScore(row.TrackNumber != nil, row.PositionClash, row.FilePath, row.HasMbid), + TagFit: library.TagFitScore(row.TrackNumber != nil, row.PositionClash, row.FilePath, row.HasMbid), }) } for i := range groups { @@ -332,7 +333,7 @@ func (h *handlers) handleRunDuplicateSweep(w http.ResponseWriter, _ *http.Reques // handleDismissDuplicateGroup implements POST // /api/admin/library/duplicates/{id}/dismiss: "these are not duplicates". The // sweep keeps the dismissal and will not propose that set of tracks again. On a -// cross-release group it also undoes the song link (M498 #5438): the copies +// cross-release group it also undoes the song link: the copies // stop sharing likes from here on. 404 duplicate_group_not_pending when the // group was already resolved or is gone. func (h *handlers) handleDismissDuplicateGroup(w http.ResponseWriter, r *http.Request) { @@ -377,31 +378,26 @@ func (h *handlers) dismissDuplicateGroup(ctx context.Context, id pgtype.UUID) (i // keeps the report's proposal. type mergeDuplicateRequest struct { SurvivorTrackID string `json:"survivor_track_id"` - Unmonitor bool `json:"unmonitor"` } // mergeDuplicateResponse reports what the merge removed. RemovedPaths are files -// deleted from disk; the operator reads them to know exactly what went. +// deleted from disk; an admin reads them to know exactly what went. type mergeDuplicateResponse struct { - SurvivorTrackID string `json:"survivor_track_id"` - RemovedPaths []string `json:"removed_paths"` - LidarrUnmonitorFailed *bool `json:"lidarr_unmonitor_failed,omitempty"` + SurvivorTrackID string `json:"survivor_track_id"` + RemovedPaths []string `json:"removed_paths"` } -// mergeRequestBodyLimit bounds the request body. It holds one id and a flag. +// mergeRequestBodyLimit bounds the request body. It holds one id. const mergeRequestBodyLimit = 1 << 16 -// handleMergeDuplicateGroup implements POST /api/admin/library/duplicates/{id}/merge -// (#3911): keep one copy, move the others' likes, plays and playlist entries onto -// it, and delete the others' files and rows. +// handleMergeDuplicateGroup implements POST +// /api/admin/library/duplicates/{id}/merge: keep one copy, move the others' +// likes, plays and playlist entries onto it, and delete the others' files and +// rows. // // Errors: // - 409 library_not_writable / 500 file_delete_failed when a file could not be // removed — nothing was changed (fileRemoveAPIError) -// - 409 copy_tracked_by_lidarr when a copy to remove is one Lidarr maps to a -// track: removing it would make Lidarr download it again (M498) -// - 503 lidarr_unavailable when Lidarr is on but did not answer, so that -// cannot be checked // - 404 duplicate_group_not_pending when the group was already resolved // - 400 survivor_not_in_group, invalid_id, invalid_body func (h *handlers) handleMergeDuplicateGroup(w http.ResponseWriter, r *http.Request) { @@ -427,7 +423,7 @@ func (h *handlers) handleMergeDuplicateGroup(w http.ResponseWriter, r *http.Requ } } - res, unmonitorFailed, err := h.tracks.MergeDuplicates(r.Context(), groupID, survivorID, admin.ID, body.Unmonitor) + res, err := h.tracks.MergeDuplicates(r.Context(), groupID, survivorID, admin.ID) if err != nil { if apiErr, ok := fileRemoveAPIError(err); ok { logFileRemoveFailure(h.logger, apiErr, "group_id", uuidToString(groupID)) @@ -439,11 +435,6 @@ func (h *handlers) handleMergeDuplicateGroup(w http.ResponseWriter, r *http.Requ writeAdminJSONErr(w, http.StatusNotFound, "duplicate_group_not_pending") case errors.Is(err, library.ErrSurvivorNotInGroup): writeAdminJSONErr(w, http.StatusBadRequest, "survivor_not_in_group") - case errors.Is(err, library.ErrCopyTrackedByLidarr): - writeAdminJSONErr(w, http.StatusConflict, "copy_tracked_by_lidarr") - case errors.Is(err, tracks.ErrLidarrUnavailable): - h.logger.Warn("admin: merge duplicate group: lidarr unavailable", "group_id", uuidToString(groupID), "err", err) - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_unavailable") default: h.logger.Error("admin: merge duplicate group", "group_id", uuidToString(groupID), "err", err) writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") @@ -458,9 +449,92 @@ func (h *handlers) handleMergeDuplicateGroup(w http.ResponseWriter, r *http.Requ for _, c := range res.Removed { resp.RemovedPaths = append(resp.RemovedPaths, c.FilePath) } - if body.Unmonitor && unmonitorFailed { - failed := true - resp.LidarrUnmonitorFailed = &failed + writeJSON(w, http.StatusOK, resp) +} + +// refetchDuplicateRequest is an admin's answer for one recording filling +// several of an album's slots. keep_track_id is the copy whose title the +// recording is; empty means they cannot say, and every copy goes. +type refetchDuplicateRequest struct { + KeepTrackID string `json:"keep_track_id"` +} + +// refetchDuplicateResponse reports what went. Blocked names the releases set +// aside; empty when the album came from before Minstrel downloaded anything. +type refetchDuplicateResponse struct { + KeptTrackID *string `json:"kept_track_id"` + RemovedPaths []string `json:"removed_paths"` + Blocked []string `json:"blocked"` +} + +// handleRefetchDuplicateGroup implements POST +// /api/admin/library/duplicates/{id}/refetch: settle a group of one +// recording in several slots of an album the way the resolver would, with an +// admin's answer. The copies in slots that are not the recording's are +// removed and marked missing, the release they came from is set aside, and the +// album is wanted again. Merge is the wrong action there: it keeps one title +// whether or not it is the audio's, and fetches nothing. +// +// Errors: +// - 409 library_not_writable / 500 file_delete_failed when a file could not be +// removed (fileRemoveAPIError) +// - 404 duplicate_group_not_pending when the group was already resolved +// - 400 not_wrong_slot when the group is not one recording in several slots +// - 400 survivor_not_in_group, invalid_id, invalid_body +func (h *handlers) handleRefetchDuplicateGroup(w http.ResponseWriter, r *http.Request) { + admin, ok := requireUser(w, r) + if !ok { + return + } + groupID, ok := parseUUID(chi.URLParam(r, "id")) + if !ok { + writeAdminJSONErr(w, http.StatusBadRequest, "invalid_id") + return + } + var body refetchDuplicateRequest + if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, mergeRequestBodyLimit)).Decode(&body); err != nil && !errors.Is(err, io.EOF) { + writeAdminJSONErr(w, http.StatusBadRequest, "invalid_body") + return + } + var keepID pgtype.UUID // invalid: every copy goes + if body.KeepTrackID != "" { + if keepID, ok = parseUUID(body.KeepTrackID); !ok { + writeAdminJSONErr(w, http.StatusBadRequest, "invalid_id") + return + } + } + + res, err := library.SettleWrongSlot(r.Context(), h.pool, h.logger, groupID, keepID, admin.ID) + if err != nil { + if apiErr, ok := fileRemoveAPIError(err); ok { + logFileRemoveFailure(h.logger, apiErr, "group_id", uuidToString(groupID)) + writeErr(w, apiErr) + return + } + switch { + case errors.Is(err, library.ErrDuplicateGroupNotPending): + writeAdminJSONErr(w, http.StatusNotFound, "duplicate_group_not_pending") + case errors.Is(err, library.ErrNotWrongSlot): + writeAdminJSONErr(w, http.StatusBadRequest, "not_wrong_slot") + case errors.Is(err, library.ErrSurvivorNotInGroup): + writeAdminJSONErr(w, http.StatusBadRequest, "survivor_not_in_group") + default: + h.logger.Error("admin: refetch duplicate group", "group_id", uuidToString(groupID), "err", err) + writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") + } + return + } + + resp := refetchDuplicateResponse{RemovedPaths: make([]string, 0, len(res.Removed)), Blocked: res.Blocked} + if resp.Blocked == nil { + resp.Blocked = []string{} + } + if res.Kept != nil { + id := uuidToString(res.Kept.TrackID) + resp.KeptTrackID = &id + } + for _, c := range res.Removed { + resp.RemovedPaths = append(resp.RemovedPaths, c.FilePath) } writeJSON(w, http.StatusOK, resp) } diff --git a/internal/api/admin_duplicates_test.go b/internal/api/admin_duplicates_test.go index e67cfe56..ba869714 100644 --- a/internal/api/admin_duplicates_test.go +++ b/internal/api/admin_duplicates_test.go @@ -64,28 +64,28 @@ func TestFoldDuplicateGroups(t *testing.T) { } } -// M498: the proposal shown beside a group follows the resolver's rule. The copy -// Lidarr tracks is proposed even when the other is lossless, because removing -// the tracked one would make Lidarr download it again. -func TestFoldDuplicateGroups_ProposesTheCopyLidarrTracks(t *testing.T) { +// the proposal shown beside a group follows the resolver's rule. The copy +// whose tags fit the album is proposed even when the other is larger (only a +// lossless copy outranks it). +func TestFoldDuplicateGroups_ProposesTheCopyWhoseTagsFit(t *testing.T) { at := time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC) - tracked, unmapped := "tracked", "unmapped" class := "same_release" + three := int32(3) rows := []dbq.ListPendingDuplicateGroupMembersRow{ {GroupID: dupUUID(3), Tier: "acoustic", DetectedAt: dupTS(at), TrackID: dupUUID(30), Class: &class, - Title: "Song", FileFormat: "flac", FileSize: 30_000_000, AddedAt: dupTS(at), LidarrState: &unmapped}, + Title: "Song", FileFormat: "m4a", FileSize: 30_000_000, AddedAt: dupTS(at)}, {GroupID: dupUUID(3), Tier: "acoustic", DetectedAt: dupTS(at), TrackID: dupUUID(31), Class: &class, - Title: "Song", FileFormat: "mp3", FileSize: 9_000_000, AddedAt: dupTS(at), LidarrState: &tracked}, + Title: "Song", FileFormat: "mp3", FileSize: 9_000_000, AddedAt: dupTS(at), TrackNumber: &three, HasMbid: true}, } got := foldDuplicateGroups(rows) if len(got) != 1 { t.Fatalf("folded %d groups, want 1", len(got)) } g := got[0] - if g.SurvivorTrackID != uuidToString(dupUUID(31)) || g.SurvivorReason != "the copy Lidarr tracks" { - t.Errorf("survivor = (%s, %q), want the tracked mp3", g.SurvivorTrackID, g.SurvivorReason) + if g.SurvivorTrackID != uuidToString(dupUUID(31)) || g.SurvivorReason != "tags fit the album" { + t.Errorf("survivor = (%s, %q), want the tagged mp3", g.SurvivorTrackID, g.SurvivorReason) } - if g.Class == nil || *g.Class != "same_release" || g.Members[1].LidarrState == nil || *g.Members[1].LidarrState != "tracked" { - t.Errorf("class or lidarr state not carried: %+v", g) + if g.Class == nil || *g.Class != "same_release" { + t.Errorf("class not carried: %+v", g) } } diff --git a/internal/api/admin_fingerprint_settings.go b/internal/api/admin_fingerprint_settings.go index d21bbbc0..cf27c54e 100644 --- a/internal/api/admin_fingerprint_settings.go +++ b/internal/api/admin_fingerprint_settings.go @@ -10,15 +10,15 @@ import ( ) // fingerprintSettingsBody is the wire shape for GET and PUT -// /api/admin/library/fingerprint-settings (M400 #3913). The threshold travels as -// the bit-error rate the matcher uses; the card presents it as a match percentage. +// /api/admin/library/fingerprint-settings. The threshold travels as the +// bit-error rate the matcher uses; the card presents it as a match percentage. type fingerprintSettingsBody struct { Enabled bool `json:"enabled"` ChromaprintLengthSec int32 `json:"chromaprint_length_sec"` AcousticMaxBitErrorRate float64 `json:"acoustic_max_bit_error_rate"` BackfillConcurrency int32 `json:"backfill_concurrency"` SweepIntervalHours int32 `json:"sweep_interval_hours"` - // AutoResolve is a pointer so a client that predates it (M498) and leaves + // AutoResolve is a pointer so a client that predates it and leaves // it out keeps the saved value rather than switching the resolver off. AutoResolve *bool `json:"auto_resolve"` } @@ -34,16 +34,18 @@ func fingerprintSettingsBodyOf(s library.FingerprintSettings) fingerprintSetting } } -// handleGetFingerprintSettings implements GET /api/admin/library/fingerprint-settings. +// handleGetFingerprintSettings implements GET +// /api/admin/library/fingerprint-settings. func (h *handlers) handleGetFingerprintSettings(w http.ResponseWriter, _ *http.Request) { writeJSON(w, http.StatusOK, fingerprintSettingsBodyOf(h.fingerprintSettings.Get())) } -// handleUpdateFingerprintSettings implements PUT /api/admin/library/fingerprint-settings. +// handleUpdateFingerprintSettings implements PUT +// /api/admin/library/fingerprint-settings. // -// A whole-row write. A body that leaves a field out decodes it as zero, which no -// field accepts, so a partial save is refused rather than zeroing what it omitted. -// auto_resolve is the exception: left out, it keeps its saved value. +// A whole-row write. A body that leaves a field out decodes it as zero, which +// no field accepts, so a partial save is refused rather than zeroing what it +// omitted. auto_resolve is the exception: left out, it keeps its saved value. // The saved settings reach the scanner and both workers at once: they share the // service instance. func (h *handlers) handleUpdateFingerprintSettings(w http.ResponseWriter, r *http.Request) { diff --git a/internal/api/admin_import.go b/internal/api/admin_import.go new file mode 100644 index 00000000..207283ef --- /dev/null +++ b/internal/api/admin_import.go @@ -0,0 +1,285 @@ +package api + +import ( + "encoding/json" + "errors" + "net/http" + + "github.com/jackc/pgx/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/importer" + "git.fabledsword.com/bvandeusen/minstrel/internal/lifecycle" +) + +// Downloads and their imports: the Wanted view's Activity tab, +// a person's "import anyway", "refuse" and "retry", and the Blocked tab. + +// downloadView is one grab and what its import made of it. +type downloadView struct { + ID string `json:"id"` + WantID *string `json:"want_id"` + WantTitle string `json:"want_title"` + WantArtist string `json:"want_artist"` + Name string `json:"name"` + Indexer string `json:"indexer"` + Client string `json:"client"` + Quality string `json:"quality"` + Origin string `json:"origin"` + GrabbedAt string `json:"grabbed_at"` + EndedAt *string `json:"ended_at"` + EndReason string `json:"end_reason"` + Progress float32 `json:"progress"` + ImportState string `json:"import_state"` + ImportReason string `json:"import_reason"` + ImportedAt *string `json:"imported_at"` + ImportedFiles int32 `json:"imported_files"` + Destination string `json:"destination"` + Edition string `json:"edition,omitempty"` + Evidence *importer.Evidence `json:"evidence,omitempty"` + Files []string `json:"files"` + Skipped int `json:"skipped"` + RehomedFrom string `json:"rehomed_from,omitempty"` + // EndDetail says why an ended download ended. + EndDetail string `json:"end_detail"` + // Seeding: imported and still sharing; Ratio its share ratio as last + // read; SeedReleasedAt when it was removed once it had shared enough. + Seeding bool `json:"seeding"` + Ratio float32 `json:"ratio"` + SeedReleasedAt *string `json:"seed_released_at"` + // Condition is what is wrong with it now, if anything. + Condition *conditionView `json:"condition"` +} + +// conditionView is an open row of the intervention ledger: what was seen, +// since when, and how many times it was retried. +type conditionView struct { + Trigger string `json:"trigger"` + Detail string `json:"detail"` + Retries int32 `json:"retries"` + Since string `json:"since"` +} + +func downloadFrom(g dbq.ListGrabsRow) downloadView { + v := downloadView{ + ID: uuidToString(g.ID), Name: g.Name, Indexer: g.Indexer, Client: g.Client, Quality: g.Quality, + Origin: g.Origin, GrabbedAt: formatTimestamp(g.GrabbedAt), EndReason: g.EndReason, + Progress: g.Progress, ImportState: g.ImportState, ImportReason: g.ImportReason, + ImportedFiles: g.ImportedFiles, Destination: g.Destination, Files: []string{}, + EndDetail: g.EndDetail, Seeding: g.Seeding, Ratio: g.Ratio, + } + if g.SeedReleasedAt.Valid { + at := formatTimestamp(g.SeedReleasedAt) + v.SeedReleasedAt = &at + } + if g.Condition != nil { + v.Condition = &conditionView{Trigger: *g.Condition, Detail: deref(g.ConditionDetail), + Since: formatTimestamp(g.ConditionSince)} + if g.ConditionRetries != nil { + v.Condition.Retries = *g.ConditionRetries + } + } + if g.WantID.Valid { + id := uuidToString(g.WantID) + v.WantID = &id + } + if g.WantTitle != nil { + v.WantTitle = *g.WantTitle + } + if g.WantArtist != nil { + v.WantArtist = *g.WantArtist + } + if g.EndedAt.Valid { + at := formatTimestamp(g.EndedAt) + v.EndedAt = &at + } + if g.ImportedAt.Valid { + at := formatTimestamp(g.ImportedAt) + v.ImportedAt = &at + } + var det importer.Detail + if len(g.ImportDetail) > 0 && json.Unmarshal(g.ImportDetail, &det) == nil && det.Verdict.Outcome != "" { + ev := det.Verdict.Evidence + v.Evidence, v.Edition, v.Skipped, v.RehomedFrom = &ev, det.Edition, det.Skipped, det.RehomedFrom + if det.Files != nil { + v.Files = det.Files + } + } + return v +} + +type downloadsPage struct { + Items []downloadView `json:"items"` + Limit int `json:"limit"` + Offset int `json:"offset"` + // ClientsDown are the download clients not answering now. + ClientsDown []clientDownView `json:"clients_down"` +} + +type clientDownView struct { + Client string `json:"client"` + Detail string `json:"detail"` + Since string `json:"since"` +} + +// handleListDownloads implements GET /api/admin/downloads?offset=: running +// downloads first, then the rest newest first. +func (h *handlers) handleListDownloads(w http.ResponseWriter, r *http.Request) { + limit, offset, err := parsePaging(r.URL.Query()) + if err != nil { + writeErr(w, apierror.BadRequest("invalid_paging", "limit and offset must be numbers")) + return + } + rows, err := dbq.New(h.pool).ListGrabs(r.Context(), dbq.ListGrabsParams{MaxRows: int32(limit), Skip: int32(offset)}) + if err != nil { + writeErrWithLog(w, h.logger, "admin downloads: list", apierror.Internal(err)) + return + } + down, err := dbq.New(h.pool).ListClientsDown(r.Context()) + if err != nil { + writeErrWithLog(w, h.logger, "admin downloads: clients down", apierror.Internal(err)) + return + } + out := downloadsPage{Items: make([]downloadView, 0, len(rows)), Limit: limit, Offset: offset, + ClientsDown: make([]clientDownView, 0, len(down))} + for _, g := range rows { + out.Items = append(out.Items, downloadFrom(g)) + } + for _, c := range down { + out.ClientsDown = append(out.ClientsDown, clientDownView{Client: c.Client, Detail: c.Detail, + Since: formatTimestamp(c.FirstSeen)}) + } + writeJSON(w, http.StatusOK, out) +} + +// handleAbandonDownload implements POST /api/admin/downloads/{id}/abandon: a +// running download is given up, its release set aside, the download removed +// and the album searched again. +func (h *handlers) handleAbandonDownload(w http.ResponseWriter, r *http.Request) { + admin, ok := requireUser(w, r) + if !ok { + return + } + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + if _, err := h.lifecycle.Abandon(r.Context(), id, admin.ID); err != nil { + switch { + case errors.Is(err, pgx.ErrNoRows), errors.Is(err, lifecycle.ErrNotAbandonable): + writeErr(w, apierror.Conflict("not_running", "that download is no longer running")) + default: + writeErrWithLog(w, h.logger, "admin downloads: abandon", apierror.Internal(err)) + } + return + } + w.WriteHeader(http.StatusNoContent) +} + +// handleImportAnyway implements POST /api/admin/downloads/{id}/import: a +// download the check held for a person (or one that failed) is imported on +// the next pass whatever the check says. +func (h *handlers) handleImportAnyway(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + if _, err := dbq.New(h.pool).ForceGrabImport(r.Context(), id); err != nil { + h.writeImportErr(w, "import anyway", err) + return + } + w.WriteHeader(http.StatusNoContent) +} + +// handleRetryImport implements POST /api/admin/downloads/{id}/retry. +func (h *handlers) handleRetryImport(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + if _, err := dbq.New(h.pool).RetryGrabImport(r.Context(), id); err != nil { + h.writeImportErr(w, "retry", err) + return + } + w.WriteHeader(http.StatusNoContent) +} + +// handleRefuseDownload implements POST /api/admin/downloads/{id}/refuse: the +// release is blocked, the download removed and the album searched again. +func (h *handlers) handleRefuseDownload(w http.ResponseWriter, r *http.Request) { + admin, ok := requireUser(w, r) + if !ok { + return + } + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + if _, err := h.importer.Refuse(r.Context(), id, admin.ID); err != nil { + h.writeImportErr(w, "refuse", err) + return + } + w.WriteHeader(http.StatusNoContent) +} + +func (h *handlers) writeImportErr(w http.ResponseWriter, what string, err error) { + switch { + case errors.Is(err, pgx.ErrNoRows), errors.Is(err, importer.ErrNotDecidable): + writeErr(w, apierror.Conflict("not_decidable", "that download is not waiting on a decision")) + default: + writeErrWithLog(w, h.logger, "admin downloads: "+what, apierror.Internal(err)) + } +} + +type releaseBlockView struct { + InfoHash string `json:"info_hash"` + Name string `json:"name"` + Reason string `json:"reason"` + Detail string `json:"detail"` + BlockedAt string `json:"blocked_at"` + // ExpiresAt is when a release given up on may be taken again; null is + // for good. + ExpiresAt *string `json:"expires_at"` + WantTitle string `json:"want_title"` + WantArtist string `json:"want_artist"` +} + +type releaseBlocksPage struct { + Items []releaseBlockView `json:"items"` + Limit int `json:"limit"` + Offset int `json:"offset"` +} + +// handleListReleaseBlocks implements GET /api/admin/release-blocks?offset=: +// every release set aside, by a person or by a check, newest first. +func (h *handlers) handleListReleaseBlocks(w http.ResponseWriter, r *http.Request) { + limit, offset, err := parsePaging(r.URL.Query()) + if err != nil { + writeErr(w, apierror.BadRequest("invalid_paging", "limit and offset must be numbers")) + return + } + rows, err := dbq.New(h.pool).ListReleaseBlocksPage(r.Context(), dbq.ListReleaseBlocksPageParams{ + MaxRows: int32(limit), Skip: int32(offset)}) + if err != nil { + writeErrWithLog(w, h.logger, "admin release blocks: list", apierror.Internal(err)) + return + } + out := releaseBlocksPage{Items: make([]releaseBlockView, 0, len(rows)), Limit: limit, Offset: offset} + for _, b := range rows { + v := releaseBlockView{InfoHash: b.InfoHash, Name: b.Name, Reason: b.Reason, Detail: b.Detail, + BlockedAt: formatTimestamp(b.BlockedAt)} + if b.ExpiresAt.Valid { + at := formatTimestamp(b.ExpiresAt) + v.ExpiresAt = &at + } + if b.WantTitle != nil { + v.WantTitle = *b.WantTitle + } + if b.WantArtist != nil { + v.WantArtist = *b.WantArtist + } + out.Items = append(out.Items, v) + } + writeJSON(w, http.StatusOK, out) +} diff --git a/internal/api/admin_library_folders.go b/internal/api/admin_library_folders.go new file mode 100644 index 00000000..0c2a4745 --- /dev/null +++ b/internal/api/admin_library_folders.go @@ -0,0 +1,111 @@ +package api + +import ( + "context" + "errors" + "net/http" + "os" + + "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/library" +) + +// The library's folders: where the scan, the watcher and the import +// look for music, set here rather than only by MINSTREL_LIBRARY_SCAN_PATHS. + +type libraryFolderView struct { + Path string `json:"path"` + // Present is whether the server can see the folder right now; a folder + // whose mount went away reads false here before anything else notices. + Present bool `json:"present"` + Tracks int64 `json:"tracks"` +} + +type libraryFoldersView struct { + Folders []libraryFolderView `json:"folders"` + // Saved is false while the folders still come from + // MINSTREL_LIBRARY_SCAN_PATHS. + Saved bool `json:"saved"` + // Moved is how many tracks a save moved to a new folder. + Moved int64 `json:"moved,omitempty"` +} + +func (h *handlers) libraryFolders(ctx context.Context) (libraryFoldersView, error) { + q := dbq.New(h.pool) + row, err := q.GetLibrarySettings(ctx) + if err != nil { + return libraryFoldersView{}, err + } + v := libraryFoldersView{Folders: []libraryFolderView{}, Saved: len(row.ScanPaths) > 0} + for _, root := range h.scanner.Roots() { + n, err := q.CountTracksUnder(ctx, root) + if err != nil { + return libraryFoldersView{}, err + } + info, statErr := os.Stat(root) + v.Folders = append(v.Folders, libraryFolderView{Path: root, Present: statErr == nil && info.IsDir(), Tracks: n}) + } + return v, nil +} + +// handleGetLibraryFolders implements GET /api/admin/library/folders. +func (h *handlers) handleGetLibraryFolders(w http.ResponseWriter, r *http.Request) { + v, err := h.libraryFolders(r.Context()) + if err != nil { + writeErrWithLog(w, h.logger, "admin library folders: get", apierror.Internal(err)) + return + } + writeJSON(w, http.StatusOK, v) +} + +type putLibraryFoldersBody struct { + Folders []struct { + Path string `json:"path"` + // Was is the folder this one replaces, whose files are now at Path; + // "" is a folder added. + Was string `json:"was"` + } `json:"folders"` +} + +// handlePutLibraryFolders implements PUT /api/admin/library/folders. +// +// The whole list. A folder whose path changed moves its tracks with it, so +// their plays, likes and playlists survive; one left out is no longer read, +// and its tracks are marked missing by the next scan. After a save a scan +// starts, to pick up anything the old folders did not hold. +func (h *handlers) handlePutLibraryFolders(w http.ResponseWriter, r *http.Request) { + var body putLibraryFoldersBody + if !decodeBody(w, r, &body) { + return + } + if h.scanner == nil { + writeErr(w, apierror.Internal(errors.New("no scanner"))) + return + } + changes := make([]library.RootChange, 0, len(body.Folders)) + for _, f := range body.Folders { + changes = append(changes, library.RootChange{Path: f.Path, Was: f.Was}) + } + res, err := h.scanner.SetRoots(r.Context(), changes) + if err != nil { + var re *library.RootsError + if errors.As(err, &re) { + writeErr(w, apierror.BadRequest("bad_folders", re.Error())) + return + } + writeErrWithLog(w, h.logger, "admin library folders: put", apierror.Internal(err)) + return + } + if _, _, err := library.TryStartScan(context.Background(), h.pool, h.scanner, h.coverart, + h.logger.With("source", "library_folders"), h.scanCfg); err != nil { + h.logger.Warn("admin library folders: scan after save", "err", err) + } + v, err := h.libraryFolders(r.Context()) + if err != nil { + writeErrWithLog(w, h.logger, "admin library folders: get", apierror.Internal(err)) + return + } + v.Moved = res.Tracks + writeJSON(w, http.StatusOK, v) +} diff --git a/internal/api/admin_library_missing.go b/internal/api/admin_library_missing.go index abbafb0b..30a3a4a8 100644 --- a/internal/api/admin_library_missing.go +++ b/internal/api/admin_library_missing.go @@ -7,7 +7,6 @@ import ( "github.com/jackc/pgx/v5/pgtype" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/reacquisition" ) // missingTrackView is one track whose file the scan could not find. @@ -30,28 +29,25 @@ type missingTrackView struct { LastPlayedAt *string `json:"last_played_at"` } -// reacquisitionStateView is what the sweeper has done about one album -// (milestone #290), attached to the group so the operator can tell "nothing -// has happened yet" from "asked twice, still nothing" without cross-checking -// the Requests queue. -// -// NextAttemptAt is computed rather than stored: the schedule is a function of -// the attempt count and the current settings, so persisting it would go stale -// the moment an operator edited the backoff. +// reacquisitionStateView is what the wanted list says about one album: when +// it was wanted back, and when the library +// held it again. A missing album with no want yet is still inside the grace +// window, or MusicBrainz cannot name it. Want is that want as the wanted list +// sends it, so the Missing page can show its search and download and act on +// it. type reacquisitionStateView struct { - Attempts int `json:"attempts"` - LastAttemptAt *string `json:"last_attempt_at"` - NextAttemptAt *string `json:"next_attempt_at"` - GaveUpAt *string `json:"gave_up_at"` + WantedAt *string `json:"wanted_at"` + HeldAt *string `json:"held_at"` + Want *wantView `json:"want"` } // missingGroupView is a directory's worth of missing tracks. // -// Grouping is the whole ergonomic argument for this surface. The case that -// produced #2523 was three reorganised albums showing up as ~40 individually -// missing files; presented flat that reads as forty problems, presented by -// folder it reads as three. MissingSince is the EARLIEST mark in the group, -// so a directory sorts and reads by when it first went away. +// Grouping is the whole ergonomic argument for this surface: a reorganised +// album shows up as dozens of individually missing files, which presented flat +// reads as dozens of problems and presented by folder reads as one. +// MissingSince is the EARLIEST mark in the group, so a directory sorts and +// reads by when it first went away. type missingGroupView struct { Directory string `json:"directory"` MissingSince string `json:"missing_since"` @@ -76,8 +72,8 @@ type adminMissingResponse struct { // // Read-only by design. Nothing on this surface deletes a track: a missing file // keeps its row, its history and its likes because it may come back, and if it -// comes back renamed the scanner adopts it (#2528). The surface exists so an -// operator can SEE what the library has lost and act on it deliberately. +// comes back renamed the scanner adopts it. The surface exists so an admin +// can SEE what the library has lost and act on it deliberately. func (h *handlers) handleListMissingTracks(w http.ResponseWriter, r *http.Request) { limit, offset, err := parsePaging(r.URL.Query()) if err != nil { @@ -159,15 +155,14 @@ func groupMissingByDirectory(rows []dbq.ListMissingTracksRow) []missingGroupView return groups } -// attachReacquisitionState decorates each group with what the sweeper has -// done about its album (milestone #290). +// attachReacquisitionState decorates each group with its album's want. // // Best-effort: this is context on a list whose primary job is showing what is // missing, so a failure here leaves the groups bare rather than failing the // page. One batched query for the whole page, not one per group. // -// A group is keyed by directory while re-acquisition is keyed by album, and -// those line up in practice (an album's files live in one folder) but are not +// A group is keyed by directory while a want is keyed by album, and those +// line up in practice (an album's files live in one folder) but are not // guaranteed to — a directory holding two albums takes the first album's // state, which is the same album its first track belongs to. func (h *handlers) attachReacquisitionState( @@ -183,7 +178,13 @@ func (h *handlers) attachReacquisitionState( // groupMissingByDirectory folded them in. dirAlbum := make(map[string]pgtype.UUID, len(groups)) ids := make([]pgtype.UUID, 0, len(groups)) + // The latest loss in each folder: a want held before it is the album + // as it was before its files went, not the library holding it again. + dirLost := make(map[string]time.Time, len(groups)) for _, row := range rows { + if row.MissingSince.Valid && row.MissingSince.Time.After(dirLost[row.Directory]) { + dirLost[row.Directory] = row.MissingSince.Time + } if _, seen := dirAlbum[row.Directory]; seen { continue } @@ -191,60 +192,71 @@ func (h *handlers) attachReacquisitionState( ids = append(ids, row.AlbumID) } - states, err := q.GetReacquisitionForAlbums(r.Context(), ids) + wants, err := q.GetWantsForAlbums(r.Context(), ids) if err != nil { - h.logger.Warn("admin missing: reacquisition state lookup failed", "err", err) + h.logger.Warn("admin missing: want lookup failed", "err", err) return } - byAlbum := make(map[string]dbq.MissingReacquisition, len(states)) - for _, s := range states { - byAlbum[uuidToString(s.AlbumID)] = s - } - - // The settings service is optional in contexts that only wire routing - // (tests), and the backoff projection is decoration on a list whose real - // job is elsewhere — so fall back to the shipped defaults rather than - // making this a nil-pointer waiting to happen (rule #48). - cfg := reacquisition.Defaults - if h.reacqSettings != nil { - cfg = h.reacqSettings.Get() + // Newest want first, so the first seen per album is its current one. + byAlbum := make(map[string]dbq.GetWantsForAlbumsRow, len(wants)) + for _, w := range wants { + if _, seen := byAlbum[uuidToString(w.AlbumID)]; !seen { + byAlbum[uuidToString(w.AlbumID)] = w + } } + full := h.wantViewsByID(r, q, byAlbum) for i := range groups { albumID, ok := dirAlbum[groups[i].Directory] if !ok { continue } - state, ok := byAlbum[uuidToString(albumID)] + w, ok := byAlbum[uuidToString(albumID)] if !ok { continue } - groups[i].Reacquisition = buildReacquisitionState(state, cfg.Backoff(state.Attempts)) + // Held before its files went (every library album is on the list, + // held): not wanted back yet, which the sweep does once + // the grace window has passed. + if w.FulfilledAt.Valid && !w.FulfilledAt.Time.After(dirLost[groups[i].Directory]) { + continue + } + view := &reacquisitionStateView{} + wantedAt := formatTimestamp(w.WantedAt) + view.WantedAt = &wantedAt + if w.FulfilledAt.Valid { + heldAt := formatTimestamp(w.FulfilledAt) + view.HeldAt = &heldAt + } + if v, ok := full[uuidToString(w.WantID)]; ok { + view.Want = &v + } + groups[i].Reacquisition = view } } -// buildReacquisitionState renders one album's attempt record, projecting the -// next attempt from the last one plus the backoff the current settings imply. -func buildReacquisitionState( - state dbq.MissingReacquisition, - backoff time.Duration, -) *reacquisitionStateView { - out := &reacquisitionStateView{Attempts: int(state.Attempts)} - if state.LastAttemptAt.Valid { - s := formatTimestamp(state.LastAttemptAt) - out.LastAttemptAt = &s - // Only meaningful while more attempts remain; an album that has given - // up has no next attempt to promise. - if !state.GaveUpAt.Valid { - next := formatTimestamp(pgtype.Timestamptz{ - Time: state.LastAttemptAt.Time.Add(backoff), - Valid: true, - }) - out.NextAttemptAt = &next - } +// wantViewsByID reads each album's current want in full, keyed by want id. +// Best-effort like the state it decorates: a failure leaves the groups with +// their dates and no want. +func (h *handlers) wantViewsByID( + r *http.Request, + q *dbq.Queries, + byAlbum map[string]dbq.GetWantsForAlbumsRow, +) map[string]wantView { + ids := make([]pgtype.UUID, 0, len(byAlbum)) + for _, w := range byAlbum { + ids = append(ids, w.WantID) } - if state.GaveUpAt.Valid { - s := formatTimestamp(state.GaveUpAt) - out.GaveUpAt = &s + out := make(map[string]wantView, len(ids)) + if len(ids) == 0 { + return out + } + rows, err := q.ListWants(r.Context(), dbq.ListWantsParams{State: "all", WantIds: ids, MaxRows: int32(len(ids))}) + if err != nil { + h.logger.Warn("admin missing: want detail lookup failed", "err", err) + return out + } + for _, row := range rows { + out[uuidToString(row.ID)] = wantViewOf(row) } return out } diff --git a/internal/api/admin_library_missing_wants_test.go b/internal/api/admin_library_missing_wants_test.go new file mode 100644 index 00000000..6710580d --- /dev/null +++ b/internal/api/admin_library_missing_wants_test.go @@ -0,0 +1,84 @@ +package api + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/jackc/pgx/v5/pgtype" +) + +// seedMissingWant marks a track's file missing for two days and puts its +// album on the wanted list, held at heldDaysAgo (0 for an open want) and +// last searched with nothing found. +func seedMissingWant(t *testing.T, h *handlers, trackID, albumID pgtype.UUID, group string, heldDaysAgo int) string { + t.Helper() + ctx := context.Background() + if _, err := h.pool.Exec(ctx, + `UPDATE tracks SET missing_since = now() - interval '2 days' WHERE id = $1`, trackID); err != nil { + t.Fatalf("mark missing: %v", err) + } + if _, err := h.pool.Exec(ctx, `DELETE FROM wants WHERE release_group_mbid = $1`, group); err != nil { + t.Fatalf("clear want: %v", err) + } + var id pgtype.UUID + if err := h.pool.QueryRow(ctx, + `INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, album_id, + wanted_at, fulfilled_at, searched_at, search_outcome) + VALUES ($1, 'artist-'||$1, 'Artist', 'Album '||$1, 'library', $2, + now() - interval '30 days', + CASE WHEN $3::int > 0 THEN now() - make_interval(days => $3::int) END, + now() - interval '1 hour', 'nothing') + RETURNING id`, group, albumID, heldDaysAgo).Scan(&id); err != nil { + t.Fatalf("insert want: %v", err) + } + return uuidToString(id) +} + +// A folder's album that was held before its files went is not wanted back +// yet (every library album is on the list, held); an open want comes with +// its search, so the Missing page can show it and act on it. +func TestListMissingTracks_WantState(t *testing.T) { + h, pool := testHandlers(t) + truncateLibrary(t, pool) + admin := seedUser(t, pool, "admin", "pw", true) + + heldTrack, _ := seedTrackOnDisk(t, h, "missheld") + openTrack, _ := seedTrackOnDisk(t, h, "missopen") + seedMissingWant(t, h, heldTrack.ID, heldTrack.AlbumID, "rg-missing-held", 10) + openID := seedMissingWant(t, h, openTrack.ID, openTrack.AlbumID, "rg-missing-open", 0) + + req := withUser(httptest.NewRequest(http.MethodGet, "/api/admin/library/missing", nil), admin) + w := httptest.NewRecorder() + h.handleListMissingTracks(w, req) + if w.Code != http.StatusOK { + t.Fatalf("status %d: %s", w.Code, w.Body.String()) + } + var resp adminMissingResponse + if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { + t.Fatalf("decode: %v", err) + } + if len(resp.Groups) != 2 { + t.Fatalf("groups = %d, want 2", len(resp.Groups)) + } + byTitle := map[string]missingGroupView{} + for _, g := range resp.Groups { + byTitle[g.Tracks[0].Title] = g + } + + if r := byTitle["AdmQ Track missheld"].Reacquisition; r != nil { + t.Errorf("held before the loss: reacquisition = %+v, want none", r) + } + r := byTitle["AdmQ Track missopen"].Reacquisition + if r == nil || r.WantedAt == nil || r.HeldAt != nil { + t.Fatalf("open want: reacquisition = %+v, want wanted and not held", r) + } + if r.Want == nil || r.Want.ID != openID { + t.Fatalf("open want: want = %+v, want id %s", r.Want, openID) + } + if r.Want.Search == nil || r.Want.Search.Outcome != "nothing" { + t.Errorf("open want: search = %+v, want outcome nothing", r.Want.Search) + } +} diff --git a/internal/api/admin_lidarr.go b/internal/api/admin_lidarr.go deleted file mode 100644 index 7eea6dfc..00000000 --- a/internal/api/admin_lidarr.go +++ /dev/null @@ -1,392 +0,0 @@ -package api - -import ( - "context" - "encoding/json" - "errors" - "net/http" - "net/url" - "sync" - - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" -) - -// lidarrConfigView is the JSON shape returned by GET /api/admin/lidarr/config -// and PUT /api/admin/lidarr/config. api_key is always masked as "***" when set. -type lidarrConfigView struct { - Enabled bool `json:"enabled"` - BaseURL string `json:"base_url"` - APIKey string `json:"api_key"` - DefaultQualityProfileID int `json:"default_quality_profile_id"` - DefaultMetadataProfileID int `json:"default_metadata_profile_id"` - DefaultRootFolderPath string `json:"default_root_folder_path"` -} - -// maskAPIKey converts an api_key for external response: non-empty keys become -// "***"; empty (never set) keys remain "". -func maskAPIKey(key string) string { - if key == "" { - return "" - } - return "***" -} - -// configToView converts a lidarrconfig.Config to a lidarrConfigView with the -// api_key masked. -func configToView(cfg lidarrconfig.Config) lidarrConfigView { - return lidarrConfigView{ - Enabled: cfg.Enabled, - BaseURL: cfg.BaseURL, - APIKey: maskAPIKey(cfg.APIKey), - DefaultQualityProfileID: cfg.DefaultQualityProfileID, - DefaultMetadataProfileID: cfg.DefaultMetadataProfileID, - DefaultRootFolderPath: cfg.DefaultRootFolderPath, - } -} - -// handleGetLidarrConfig implements GET /api/admin/lidarr/config. -// Returns the current Lidarr config with api_key masked. -func (h *handlers) handleGetLidarrConfig(w http.ResponseWriter, r *http.Request) { - cfg, err := h.lidarrCfg.Get(r.Context()) - if err != nil { - h.logger.Error("admin: get lidarr config", "err", err) - writeAdminJSONErr(w, http.StatusInternalServerError, "internal_error") - return - } - writeJSON(w, http.StatusOK, configToView(cfg)) -} - -// putLidarrConfigBody is the decoded JSON body for PUT /api/admin/lidarr/config. -type putLidarrConfigBody struct { - Enabled bool `json:"enabled"` - BaseURL string `json:"base_url"` - APIKey string `json:"api_key"` - DefaultQualityProfileID int `json:"default_quality_profile_id"` - DefaultMetadataProfileID int `json:"default_metadata_profile_id"` - DefaultRootFolderPath string `json:"default_root_folder_path"` -} - -// handlePutLidarrConfig implements PUT /api/admin/lidarr/config. -// Empty api_key in the body preserves the currently saved api_key. -func (h *handlers) handlePutLidarrConfig(w http.ResponseWriter, r *http.Request) { - var body putLidarrConfigBody - if err := json.NewDecoder(r.Body).Decode(&body); err != nil { - writeAdminJSONErr(w, http.StatusBadRequest, "bad_request") - return - } - - // Resolve the effective api_key: empty in body = preserve saved value. - apiKey := body.APIKey - if apiKey == "" { - saved, err := h.lidarrCfg.Get(r.Context()) - if err != nil { - h.logger.Error("admin: put lidarr config: load saved", "err", err) - writeAdminJSONErr(w, http.StatusInternalServerError, "internal_error") - return - } - apiKey = saved.APIKey - } - - // Validate: enabled=true requires base_url AND api_key. - if body.Enabled { - if body.BaseURL == "" || apiKey == "" { - writeAdminJSONErr(w, http.StatusBadRequest, "missing_required_field") - return - } - // Validate that the URL parses. - if _, err := url.ParseRequestURI(body.BaseURL); err != nil { - writeAdminJSONErr(w, http.StatusBadRequest, "invalid_url") - return - } - // Defaults must be present so future Approve calls don't dispatch - // invalid POST bodies to Lidarr (which produces 4xx with field - // errors). The frontend pre-selects first values so this almost - // never fires for non-malicious traffic, but it's the right gate. - if body.DefaultQualityProfileID == 0 || - body.DefaultMetadataProfileID == 0 || - body.DefaultRootFolderPath == "" { - writeAdminJSONErr(w, http.StatusBadRequest, "missing_defaults") - return - } - } - - cfg := lidarrconfig.Config{ - Enabled: body.Enabled, - BaseURL: body.BaseURL, - APIKey: apiKey, - DefaultQualityProfileID: body.DefaultQualityProfileID, - DefaultMetadataProfileID: body.DefaultMetadataProfileID, - DefaultRootFolderPath: body.DefaultRootFolderPath, - } - if err := h.lidarrCfg.Save(r.Context(), cfg); err != nil { - h.logger.Error("admin: put lidarr config: save", "err", err) - writeAdminJSONErr(w, http.StatusInternalServerError, "internal_error") - return - } - writeJSON(w, http.StatusOK, configToView(cfg)) -} - -// testLidarrBody is the optional JSON body for POST /api/admin/lidarr/test. -type testLidarrBody struct { - BaseURL string `json:"base_url"` - APIKey string `json:"api_key"` -} - -// testLidarrResponse is the JSON shape returned by POST /api/admin/lidarr/test. -// All fields except Ok are optional. On a failed connection (Ok=false), only -// Ok and Error are populated. On success (Ok=true), Version is populated; -// QualityProfiles / MetadataProfiles / RootFolders carry the live Lidarr -// data and ListErrors maps any per-list fetch failures by list name. The -// integrations page consumes the lists to pre-fill its dropdowns on -// first-time setup, fixing the chicken-and-egg where the dropdowns -// previously gated on cfg.Enabled. -type testLidarrResponse struct { - Ok bool `json:"ok"` - Version string `json:"version,omitempty"` - Error string `json:"error,omitempty"` - QualityProfiles []qualityProfileView `json:"quality_profiles,omitempty"` - MetadataProfiles []metadataProfileView `json:"metadata_profiles,omitempty"` - RootFolders []rootFolderView `json:"root_folders,omitempty"` - ListErrors map[string]string `json:"list_errors,omitempty"` -} - -// handleTestLidarrConnection implements POST /api/admin/lidarr/test. -// Always returns 200; the ok/error fields in the response body indicate -// connection success or failure. -func (h *handlers) handleTestLidarrConnection(w http.ResponseWriter, r *http.Request) { - var body testLidarrBody - // Ignore decode errors — an empty body is valid (all fields optional). - _ = json.NewDecoder(r.Body).Decode(&body) - - // Fall back to saved config for any empty field. - baseURL := body.BaseURL - apiKey := body.APIKey - if baseURL == "" || apiKey == "" { - saved, err := h.lidarrCfg.Get(r.Context()) - if err != nil { - h.logger.Error("admin: test lidarr: load config", "err", err) - writeJSON(w, http.StatusOK, map[string]any{"ok": false, "error": "internal_error"}) - return - } - if baseURL == "" { - baseURL = saved.BaseURL - } - if apiKey == "" { - apiKey = saved.APIKey - } - } - - client := lidarr.NewClient(baseURL, apiKey) - result, err := client.Ping(r.Context()) - if err != nil { - errCode := lidarrErrCode(err) - // Warn (not Error): a failed connection test is expected when the - // admin is configuring the integration and may have a typo'd URL - // or wrong key. Without this log line the underlying *net.DNSError - // / *net.OpError / TLS error is invisible to operators — they only - // see the bucket code in the UI. base_url is safe to log; api_key - // must NEVER be logged. - h.logger.Warn("admin: test lidarr ping failed", - "err", err, "base_url", baseURL, "code", errCode) - writeJSON(w, http.StatusOK, testLidarrResponse{Ok: false, Error: errCode}) - return - } - quality, metadata, folders, listErrs := fetchLidarrLists(r.Context(), client) - writeJSON(w, http.StatusOK, testLidarrResponse{ - Ok: true, - Version: result.Version, - QualityProfiles: quality, - MetadataProfiles: metadata, - RootFolders: folders, - ListErrors: listErrs, - }) -} - -// fetchLidarrLists fans out the three list calls in parallel and returns -// the lists plus a map of any per-list errors. Empty map (returned as nil -// to omit the JSON field) if all succeeded. Used by the test handler to -// pre-populate the integrations page dropdowns on the same round-trip as -// the connection check. -func fetchLidarrLists(ctx context.Context, client *lidarr.Client) ( - []qualityProfileView, []metadataProfileView, []rootFolderView, map[string]string, -) { - var ( - quality []qualityProfileView - metadata []metadataProfileView - folders []rootFolderView - errMu sync.Mutex - errs = map[string]string{} - wg sync.WaitGroup - ) - recordErr := func(name string, err error) { - errMu.Lock() - errs[name] = lidarrErrCode(err) - errMu.Unlock() - } - - wg.Add(3) - go func() { - defer wg.Done() - ps, err := client.ListQualityProfiles(ctx) - if err != nil { - recordErr("quality_profiles", err) - return - } - quality = make([]qualityProfileView, len(ps)) - for i, p := range ps { - quality[i] = qualityProfileView{ID: p.ID, Name: p.Name} - } - }() - go func() { - defer wg.Done() - ps, err := client.ListMetadataProfiles(ctx) - if err != nil { - recordErr("metadata_profiles", err) - return - } - metadata = make([]metadataProfileView, len(ps)) - for i, p := range ps { - metadata[i] = metadataProfileView{ID: p.ID, Name: p.Name} - } - }() - go func() { - defer wg.Done() - fs, err := client.ListRootFolders(ctx) - if err != nil { - recordErr("root_folders", err) - return - } - folders = make([]rootFolderView, len(fs)) - for i, f := range fs { - folders[i] = rootFolderView{Path: f.Path, Accessible: f.Accessible, FreeSpace: f.FreeSpace} - } - }() - wg.Wait() - - if len(errs) == 0 { - return quality, metadata, folders, nil - } - return quality, metadata, folders, errs -} - -// lidarrErrCode maps a lidarr client error to the stable string code used in -// admin API responses. -func lidarrErrCode(err error) string { - switch { - case errors.Is(err, lidarr.ErrUnreachable): - return "lidarr_unreachable" - case errors.Is(err, lidarr.ErrAuthFailed): - return "lidarr_auth_failed" - case errors.Is(err, lidarr.ErrLookupFailed): - return "lidarr_lookup_failed" - default: - return "internal_error" - } -} - -// qualityProfileView is the JSON shape for a single quality profile in the -// GET /api/admin/lidarr/quality-profiles response. -type qualityProfileView struct { - ID int `json:"id"` - Name string `json:"name"` -} - -// handleListQualityProfiles implements GET /api/admin/lidarr/quality-profiles. -func (h *handlers) handleListQualityProfiles(w http.ResponseWriter, r *http.Request) { - _, client, ok := h.lidarrClientFromConfig(w, r) - if !ok { - return - } - - profiles, err := client.ListQualityProfiles(r.Context()) - if err != nil { - writeAdminJSONErr(w, http.StatusServiceUnavailable, lidarrErrCode(err)) - return - } - out := make([]qualityProfileView, len(profiles)) - for i, p := range profiles { - out[i] = qualityProfileView{ID: p.ID, Name: p.Name} - } - writeJSON(w, http.StatusOK, out) -} - -// metadataProfileView is the JSON shape for a single metadata profile in the -// GET /api/admin/lidarr/metadata-profiles response. -type metadataProfileView struct { - ID int `json:"id"` - Name string `json:"name"` -} - -// handleListMetadataProfiles implements GET /api/admin/lidarr/metadata-profiles. -func (h *handlers) handleListMetadataProfiles(w http.ResponseWriter, r *http.Request) { - _, client, ok := h.lidarrClientFromConfig(w, r) - if !ok { - return - } - - profiles, err := client.ListMetadataProfiles(r.Context()) - if err != nil { - writeAdminJSONErr(w, http.StatusServiceUnavailable, lidarrErrCode(err)) - return - } - out := make([]metadataProfileView, len(profiles)) - for i, p := range profiles { - out[i] = metadataProfileView{ID: p.ID, Name: p.Name} - } - writeJSON(w, http.StatusOK, out) -} - -// rootFolderView is the JSON shape for a single root folder in the -// GET /api/admin/lidarr/root-folders response. -type rootFolderView struct { - Path string `json:"path"` - Accessible bool `json:"accessible"` - FreeSpace int64 `json:"free_space"` -} - -// handleListRootFolders implements GET /api/admin/lidarr/root-folders. -func (h *handlers) handleListRootFolders(w http.ResponseWriter, r *http.Request) { - _, client, ok := h.lidarrClientFromConfig(w, r) - if !ok { - return - } - - folders, err := client.ListRootFolders(r.Context()) - if err != nil { - writeAdminJSONErr(w, http.StatusServiceUnavailable, lidarrErrCode(err)) - return - } - out := make([]rootFolderView, len(folders)) - for i, f := range folders { - out[i] = rootFolderView{Path: f.Path, Accessible: f.Accessible, FreeSpace: f.FreeSpace} - } - writeJSON(w, http.StatusOK, out) -} - -// lidarrClientFromConfig is a shared helper for handlers that need to proxy a -// request to Lidarr. It loads the saved config, returns 503 when disabled, and -// constructs a Client. Returns (cfg, client, true) on success; (_, nil, false) -// when a response has already been written. -func (h *handlers) lidarrClientFromConfig(w http.ResponseWriter, r *http.Request) (lidarrconfig.Config, *lidarr.Client, bool) { - cfg, err := h.lidarrCfg.Get(r.Context()) - if err != nil { - h.logger.Error("admin: lidarr proxy: load config", "err", err) - writeAdminJSONErr(w, http.StatusInternalServerError, "internal_error") - return lidarrconfig.Config{}, nil, false - } - if !cfg.Enabled { - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_disabled") - return lidarrconfig.Config{}, nil, false - } - return cfg, lidarr.NewClient(cfg.BaseURL, cfg.APIKey), true -} - -// writeAdminJSONErr writes a flat {"error":""} JSON response. This -// mirrors the shape the RequireAdmin middleware uses and that the spec -// defines for /api/admin/* errors. -func writeAdminJSONErr(w http.ResponseWriter, status int, code string) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(status) - _ = json.NewEncoder(w).Encode(map[string]string{"error": code}) -} diff --git a/internal/api/admin_lidarr_test.go b/internal/api/admin_lidarr_test.go deleted file mode 100644 index 5db71172..00000000 --- a/internal/api/admin_lidarr_test.go +++ /dev/null @@ -1,455 +0,0 @@ -package api - -import ( - "bytes" - "context" - "encoding/json" - "net/http" - "net/http/httptest" - "testing" - - "github.com/go-chi/chi/v5" - - "git.fabledsword.com/bvandeusen/minstrel/internal/auth" - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" -) - -// newAdminLidarrRouter builds a test chi router with the five admin/lidarr -// endpoints. RequireAdmin middleware is applied. Tests inject a user into -// context directly (skipping RequireUser). -func newAdminLidarrRouter(h *handlers) chi.Router { - r := chi.NewRouter() - r.Route("/api/admin", func(admin chi.Router) { - admin.Use(auth.RequireAdmin()) - admin.Get("/lidarr/config", h.handleGetLidarrConfig) - admin.Put("/lidarr/config", h.handlePutLidarrConfig) - admin.Post("/lidarr/test", h.handleTestLidarrConnection) - admin.Get("/lidarr/quality-profiles", h.handleListQualityProfiles) - admin.Get("/lidarr/root-folders", h.handleListRootFolders) - }) - return r -} - -// seedAdminUser creates a test user with is_admin=true. -func seedAdminUser(t *testing.T, h *handlers) dbq.User { - t.Helper() - return seedUser(t, h.pool, "admin", "pw", true) -} - -// doAdminReq fires an HTTP request at the admin router with the given user -// injected into context (bypassing RequireUser). -func doAdminReq(t *testing.T, h *handlers, method, path string, body []byte, user dbq.User) *httptest.ResponseRecorder { - t.Helper() - var reqBody *bytes.Buffer - if body != nil { - reqBody = bytes.NewBuffer(body) - } else { - reqBody = bytes.NewBuffer(nil) - } - req := httptest.NewRequest(method, path, reqBody) - if body != nil { - req.Header.Set("Content-Type", "application/json") - } - req = withUser(req, user) - w := httptest.NewRecorder() - newAdminLidarrRouter(h).ServeHTTP(w, req) - return w -} - -// saveLidarrConfigFull writes a lidarr config with the given key directly -// (for admin tests that need precise api_key control). -func saveLidarrConfigFull(t *testing.T, h *handlers, cfg lidarrconfig.Config) { - t.Helper() - svc := lidarrconfig.New(h.pool) - if err := svc.Save(context.Background(), cfg); err != nil { - t.Fatalf("saveLidarrConfigFull: %v", err) - } -} - -// TestHandleGetLidarrConfig_MasksAPIKey verifies that a set api_key is -// returned as "***" and never in plaintext. -func TestHandleGetLidarrConfig_MasksAPIKey(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - saveLidarrConfigFull(t, h, lidarrconfig.Config{ - Enabled: true, - BaseURL: "http://lidarr.lan:8686", - APIKey: "secret123", - }) - - w := doAdminReq(t, h, http.MethodGet, "/api/admin/lidarr/config", nil, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var resp lidarrConfigView - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp.APIKey == "secret123" { - t.Error("api_key leaked in response; want ***") - } - if resp.APIKey != "***" { - t.Errorf("api_key = %q, want ***", resp.APIKey) - } -} - -// TestHandleGetLidarrConfig_EmptyKeyReturnsEmpty verifies that when api_key is -// unset in the DB, GET returns api_key="" (not "***"). -func TestHandleGetLidarrConfig_EmptyKeyReturnsEmpty(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - // Save config with empty api_key (null in DB via lidarrconfig.Service). - saveLidarrConfigFull(t, h, lidarrconfig.Config{ - Enabled: false, - BaseURL: "", - APIKey: "", - }) - - w := doAdminReq(t, h, http.MethodGet, "/api/admin/lidarr/config", nil, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var resp lidarrConfigView - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp.APIKey != "" { - t.Errorf("api_key = %q, want empty string", resp.APIKey) - } -} - -// TestHandlePutLidarrConfig_EmptyKeyPreservesSaved verifies that a PUT with -// api_key="" does not overwrite the saved api_key. -func TestHandlePutLidarrConfig_EmptyKeyPreservesSaved(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - // Seed a config with a known api_key. - saveLidarrConfigFull(t, h, lidarrconfig.Config{ - Enabled: true, - BaseURL: "http://lidarr.lan:8686", - APIKey: "originalkey", - }) - - // PUT with empty api_key — should preserve "originalkey". enabled=true - // requires the defaults gate (missing_defaults) to be satisfied. - body := []byte(`{"enabled":true,"base_url":"http://lidarr.lan:8686","api_key":"","default_quality_profile_id":1,"default_metadata_profile_id":1,"default_root_folder_path":"/music"}`) - w := doAdminReq(t, h, http.MethodPut, "/api/admin/lidarr/config", body, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - - // Confirm the saved api_key is still "originalkey" by reading raw config. - svc := lidarrconfig.New(h.pool) - saved, err := svc.Get(context.Background()) - if err != nil { - t.Fatalf("get saved config: %v", err) - } - if saved.APIKey != "originalkey" { - t.Errorf("saved api_key = %q, want originalkey", saved.APIKey) - } -} - -// TestHandlePutLidarrConfig_EnabledRequiresBaseURL_400 verifies that enabling -// Lidarr without a base_url returns 400 with missing_required_field. -func TestHandlePutLidarrConfig_EnabledRequiresBaseURL_400(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - body := []byte(`{"enabled":true,"base_url":"","api_key":"somekey"}`) - w := doAdminReq(t, h, http.MethodPut, "/api/admin/lidarr/config", body, admin) - if w.Code != http.StatusBadRequest { - t.Fatalf("status = %d, want 400; body = %s", w.Code, w.Body.String()) - } - var resp map[string]string - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp["error"] != "missing_required_field" { - t.Errorf("error = %q, want missing_required_field", resp["error"]) - } -} - -// TestHandlePutLidarrConfig_HappyPath verifies a valid PUT returns 200 with -// the updated config and the api_key masked. -func TestHandlePutLidarrConfig_HappyPath(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - body := []byte(`{"enabled":true,"base_url":"http://lidarr.lan:8686","api_key":"newkey","default_quality_profile_id":2,"default_metadata_profile_id":1,"default_root_folder_path":"/music"}`) - w := doAdminReq(t, h, http.MethodPut, "/api/admin/lidarr/config", body, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var resp lidarrConfigView - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if !resp.Enabled { - t.Error("enabled = false, want true") - } - if resp.BaseURL != "http://lidarr.lan:8686" { - t.Errorf("base_url = %q, want http://lidarr.lan:8686", resp.BaseURL) - } - if resp.APIKey != "***" { - t.Errorf("api_key = %q, want ***", resp.APIKey) - } - if resp.DefaultQualityProfileID != 2 { - t.Errorf("default_quality_profile_id = %d, want 2", resp.DefaultQualityProfileID) - } - if resp.DefaultRootFolderPath != "/music" { - t.Errorf("default_root_folder_path = %q, want /music", resp.DefaultRootFolderPath) - } -} - -// TestHandleTestLidarrConnection_HappyPath verifies that POST /test against a -// stub returning valid status JSON responds 200 with ok=true and the version. -func TestHandleTestLidarrConnection_HappyPath(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - switch r.URL.Path { - case "/api/v1/system/status": - _, _ = w.Write([]byte(`{"version":"2.0.5"}`)) - case "/api/v1/qualityprofile": - _, _ = w.Write([]byte(`[{"id":1,"name":"FLAC"},{"id":2,"name":"MP3-320"}]`)) - case "/api/v1/metadataprofile": - _, _ = w.Write([]byte(`[{"id":1,"name":"Standard"}]`)) - case "/api/v1/rootfolder": - _, _ = w.Write([]byte(`[{"path":"/music","accessible":true,"freeSpace":12345}]`)) - default: - http.NotFound(w, r) - } - })) - t.Cleanup(stub.Close) - - saveLidarrConfigFull(t, h, lidarrconfig.Config{ - Enabled: true, - BaseURL: stub.URL, - APIKey: "test-key", - }) - - // POST with empty body — falls back to saved config. - w := doAdminReq(t, h, http.MethodPost, "/api/admin/lidarr/test", []byte(`{}`), admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var resp map[string]any - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp["ok"] != true { - t.Errorf("ok = %v, want true", resp["ok"]) - } - if resp["version"] != "2.0.5" { - t.Errorf("version = %v, want 2.0.5", resp["version"]) - } - qp, ok := resp["quality_profiles"].([]any) - if !ok || len(qp) != 2 { - t.Errorf("quality_profiles = %v, want 2 entries", resp["quality_profiles"]) - } - mp, ok := resp["metadata_profiles"].([]any) - if !ok || len(mp) != 1 { - t.Errorf("metadata_profiles = %v, want 1 entry", resp["metadata_profiles"]) - } - rf, ok := resp["root_folders"].([]any) - if !ok || len(rf) != 1 { - t.Errorf("root_folders = %v, want 1 entry", resp["root_folders"]) - } - if _, present := resp["list_errors"]; present { - t.Errorf("list_errors should be omitted when all lists succeed; got %v", resp["list_errors"]) - } -} - -// TestHandleTestLidarrConnection_PartialListFailure verifies that POST /test -// against a Lidarr that responds OK to the ping + two of three lists still -// reports ok=true with the failing list omitted and named in list_errors. -func TestHandleTestLidarrConnection_PartialListFailure(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - switch r.URL.Path { - case "/api/v1/system/status": - _, _ = w.Write([]byte(`{"version":"2.0.5"}`)) - case "/api/v1/qualityprofile": - _, _ = w.Write([]byte(`[{"id":1,"name":"FLAC"}]`)) - case "/api/v1/metadataprofile": - http.Error(w, "boom", http.StatusInternalServerError) - case "/api/v1/rootfolder": - _, _ = w.Write([]byte(`[{"path":"/music","accessible":true,"freeSpace":12345}]`)) - default: - http.NotFound(w, r) - } - })) - t.Cleanup(stub.Close) - - body := []byte(`{"base_url":"` + stub.URL + `","api_key":"k"}`) - w := doAdminReq(t, h, http.MethodPost, "/api/admin/lidarr/test", body, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var resp map[string]any - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp["ok"] != true { - t.Errorf("ok = %v, want true", resp["ok"]) - } - if _, present := resp["metadata_profiles"]; present { - t.Errorf("metadata_profiles should be omitted on failure; got %v", resp["metadata_profiles"]) - } - listErrs, ok := resp["list_errors"].(map[string]any) - if !ok { - t.Fatalf("list_errors missing or wrong shape: %v", resp["list_errors"]) - } - if _, present := listErrs["metadata_profiles"]; !present { - t.Errorf("list_errors.metadata_profiles should be present, got %v", listErrs) - } -} - -// TestHandleTestLidarrConnection_Unreachable verifies that POST /test against -// an unreachable URL returns 200 with ok=false and lidarr_unreachable. -func TestHandleTestLidarrConnection_Unreachable(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - // Point at a URL that will immediately refuse connections. - body := []byte(`{"base_url":"http://127.0.0.1:1","api_key":"k"}`) - w := doAdminReq(t, h, http.MethodPost, "/api/admin/lidarr/test", body, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var resp map[string]any - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp["ok"] != false { - t.Errorf("ok = %v, want false", resp["ok"]) - } - if resp["error"] != "lidarr_unreachable" { - t.Errorf("error = %v, want lidarr_unreachable", resp["error"]) - } -} - -// TestHandleListQualityProfiles_HappyPath verifies that GET /quality-profiles -// proxies and normalizes the Lidarr response. -func TestHandleListQualityProfiles_HappyPath(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - stub := newLidarrStub(t, `[{"id":1,"name":"Lossless"},{"id":2,"name":"Standard"}]`) - saveLidarrConfigFull(t, h, lidarrconfig.Config{ - Enabled: true, - BaseURL: stub.URL, - APIKey: "test-key", - }) - - w := doAdminReq(t, h, http.MethodGet, "/api/admin/lidarr/quality-profiles", nil, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var profiles []qualityProfileView - if err := json.Unmarshal(w.Body.Bytes(), &profiles); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if len(profiles) != 2 { - t.Fatalf("len(profiles) = %d, want 2", len(profiles)) - } - if profiles[0].ID != 1 || profiles[0].Name != "Lossless" { - t.Errorf("profiles[0] = %+v, want {1 Lossless}", profiles[0]) - } - if profiles[1].ID != 2 || profiles[1].Name != "Standard" { - t.Errorf("profiles[1] = %+v, want {2 Standard}", profiles[1]) - } -} - -// TestHandleListRootFolders_HappyPath verifies that GET /root-folders proxies -// and normalizes the Lidarr response. -func TestHandleListRootFolders_HappyPath(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - admin := seedAdminUser(t, h) - - stub := newLidarrStub(t, `[{"path":"/music","accessible":true,"freeSpace":1234567890}]`) - saveLidarrConfigFull(t, h, lidarrconfig.Config{ - Enabled: true, - BaseURL: stub.URL, - APIKey: "test-key", - }) - - w := doAdminReq(t, h, http.MethodGet, "/api/admin/lidarr/root-folders", nil, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var folders []rootFolderView - if err := json.Unmarshal(w.Body.Bytes(), &folders); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if len(folders) != 1 { - t.Fatalf("len(folders) = %d, want 1", len(folders)) - } - f := folders[0] - if f.Path != "/music" { - t.Errorf("path = %q, want /music", f.Path) - } - if !f.Accessible { - t.Error("accessible = false, want true") - } - if f.FreeSpace != 1234567890 { - t.Errorf("free_space = %d, want 1234567890", f.FreeSpace) - } -} - -// TestAdminEndpoints_NonAdmin403 is a table-driven test verifying that every -// admin endpoint returns 403 with not_authorized for a non-admin user. -func TestAdminEndpoints_NonAdmin403(t *testing.T) { - h, pool := testHandlers(t) - resetLidarrState(t, h) - - nonAdmin := seedUser(t, pool, "regular", "pw", false) - - cases := []struct { - method string - path string - body []byte - }{ - {http.MethodGet, "/api/admin/lidarr/config", nil}, - {http.MethodPut, "/api/admin/lidarr/config", []byte(`{}`)}, - {http.MethodPost, "/api/admin/lidarr/test", []byte(`{}`)}, - {http.MethodGet, "/api/admin/lidarr/quality-profiles", nil}, - {http.MethodGet, "/api/admin/lidarr/root-folders", nil}, - } - - for _, tc := range cases { - tc := tc - t.Run(tc.method+" "+tc.path, func(t *testing.T) { - w := doAdminReq(t, h, tc.method, tc.path, tc.body, nonAdmin) - if w.Code != http.StatusForbidden { - t.Fatalf("status = %d, want 403; body = %s", w.Code, w.Body.String()) - } - var resp map[string]string - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp["error"] != "not_authorized" { - t.Errorf("error = %q, want not_authorized", resp["error"]) - } - }) - } -} diff --git a/internal/api/admin_loudness.go b/internal/api/admin_loudness.go index 3df88ffc..2e022651 100644 --- a/internal/api/admin_loudness.go +++ b/internal/api/admin_loudness.go @@ -9,9 +9,9 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/library" ) -// loudnessCoverageResp is the wire shape for GET /api/admin/library/loudness -// (M464 #4995). measured + silent + unreadable + pending = total; missing -// tracks are not counted. Enabled travels with the counts because with +// loudnessCoverageResp is the wire shape for GET /api/admin/library/loudness. +// measured + silent + unreadable + pending = total; missing tracks are not +// counted. Enabled travels with the counts because with // analysis off, pending never shrinks, and a gauge implying progress would be // promising work nothing is doing. type loudnessCoverageResp struct { @@ -57,14 +57,15 @@ func loudnessSettingsBodyOf(s library.LoudnessSettings) loudnessSettingsBody { } } -// handleGetLoudnessSettings implements GET /api/admin/library/loudness-settings. +// handleGetLoudnessSettings implements GET +// /api/admin/library/loudness-settings. func (h *handlers) handleGetLoudnessSettings(w http.ResponseWriter, _ *http.Request) { writeJSON(w, http.StatusOK, loudnessSettingsBodyOf(h.loudnessSettings.Get())) } -// handleUpdateLoudnessSettings implements PUT /api/admin/library/loudness-settings. -// A whole-row write: a body that leaves out the concurrency decodes it as zero, -// which is refused rather than saved. +// handleUpdateLoudnessSettings implements PUT +// /api/admin/library/loudness-settings. A whole-row write: a body that leaves +// out the concurrency decodes it as zero, which is refused rather than saved. func (h *handlers) handleUpdateLoudnessSettings(w http.ResponseWriter, r *http.Request) { var req loudnessSettingsBody if err := json.NewDecoder(r.Body).Decode(&req); err != nil { diff --git a/internal/api/admin_network.go b/internal/api/admin_network.go index e2e2a4be..b37d73d3 100644 --- a/internal/api/admin_network.go +++ b/internal/api/admin_network.go @@ -14,7 +14,7 @@ type networkSettingsResp struct { TrustedProxyHops int `json:"trusted_proxy_hops"` MaxHops int `json:"max_hops"` // DetectedClientIP is what the CURRENT setting resolves this very request - // to. It's the difference between a number the operator has to reason + // to. It's the difference between a number an admin has to reason // about and one they can verify: set the value, reload, and check the // address matches the machine you're sitting at. DetectedClientIP string `json:"detected_client_ip"` @@ -79,7 +79,7 @@ func (h *handlers) handleUpdateNetworkSettings(w http.ResponseWriter, r *http.Re } // Echo the payload recomputed under the NEW value, so the card can show // immediately what the change did to this request's own address rather - // than making the operator reload to find out. + // than making an admin reload to find out. writeJSON(w, http.StatusOK, h.networkSettingsPayload(r)) } diff --git a/internal/api/admin_notification_email.go b/internal/api/admin_notification_email.go index c4916856..3e750e5e 100644 --- a/internal/api/admin_notification_email.go +++ b/internal/api/admin_notification_email.go @@ -11,7 +11,7 @@ import ( ) // notificationEmailBody is the wire shape for GET and PUT -// /api/admin/notification-email (M489 #5346): when the grouped emails go out. +// /api/admin/notification-email: when the grouped emails go out. type notificationEmailBody struct { SummaryHour int32 `json:"summary_hour"` BatchWindowMinutes int32 `json:"batch_window_minutes"` diff --git a/internal/api/admin_quarantine.go b/internal/api/admin_quarantine.go index bd372d3f..07ab3202 100644 --- a/internal/api/admin_quarantine.go +++ b/internal/api/admin_quarantine.go @@ -10,22 +10,20 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/auth" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrquarantine" + "git.fabledsword.com/bvandeusen/minstrel/internal/quarantine" ) // adminQueueRowView is a single aggregated row in the admin queue response. type adminQueueRowView struct { - TrackID string `json:"track_id"` - TrackTitle string `json:"track_title"` - ArtistName string `json:"artist_name"` - AlbumTitle string `json:"album_title"` - AlbumID string `json:"album_id"` - LidarrAlbumMBID *string `json:"lidarr_album_mbid,omitempty"` - ReportCount int32 `json:"report_count"` - LatestAt string `json:"latest_at"` - ReasonCounts map[string]int `json:"reason_counts"` - Reports []adminQueueReportView `json:"reports"` + TrackID string `json:"track_id"` + TrackTitle string `json:"track_title"` + ArtistName string `json:"artist_name"` + AlbumTitle string `json:"album_title"` + AlbumID string `json:"album_id"` + ReportCount int32 `json:"report_count"` + LatestAt string `json:"latest_at"` + ReasonCounts map[string]int `json:"reason_counts"` + Reports []adminQueueReportView `json:"reports"` } // adminQueueReportView is one user's report under an aggregated admin row. @@ -47,7 +45,7 @@ func formatTimestamp(ts pgtype.Timestamptz) string { // handleListAdminQuarantine implements GET /api/admin/quarantine. func (h *handlers) handleListAdminQuarantine(w http.ResponseWriter, r *http.Request) { - rows, err := h.lidarrQuarantine.ListAdminQueue(r.Context()) + rows, err := h.quarantine.ListAdminQueue(r.Context()) if err != nil { h.logger.Error("admin: list quarantine", "err", err) writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") @@ -66,30 +64,29 @@ func (h *handlers) handleListAdminQuarantine(w http.ResponseWriter, r *http.Requ }) } out = append(out, adminQueueRowView{ - TrackID: uuidToString(row.TrackID), - TrackTitle: row.TrackTitle, - ArtistName: row.ArtistName, - AlbumTitle: row.AlbumTitle, - AlbumID: uuidToString(row.AlbumID), - LidarrAlbumMBID: row.LidarrAlbumMBID, - ReportCount: row.ReportCount, - LatestAt: formatTimestamp(row.LatestAt), - ReasonCounts: row.ReasonCounts, - Reports: reports, + TrackID: uuidToString(row.TrackID), + TrackTitle: row.TrackTitle, + ArtistName: row.ArtistName, + AlbumTitle: row.AlbumTitle, + AlbumID: uuidToString(row.AlbumID), + ReportCount: row.ReportCount, + LatestAt: formatTimestamp(row.LatestAt), + ReasonCounts: row.ReasonCounts, + Reports: reports, }) } writeJSON(w, http.StatusOK, out) } -// actionResultView is the response shape for the three admin destructive -// actions (resolve, delete-file, delete-via-lidarr). +// actionResultView is the response shape for the admin actions (resolve, +// delete-file). type actionResultView struct { - ActionID string `json:"action_id"` - AffectedUsers int32 `json:"affected_users"` - DeletedTrackCount *int `json:"deleted_track_count,omitempty"` + ActionID string `json:"action_id"` + AffectedUsers int32 `json:"affected_users"` } -// handleResolveQuarantine implements POST /api/admin/quarantine/{track_id}/resolve. +// handleResolveQuarantine implements POST +// /api/admin/quarantine/{track_id}/resolve. func (h *handlers) handleResolveQuarantine(w http.ResponseWriter, r *http.Request) { admin, ok := auth.UserFromContext(r.Context()) if !ok { @@ -101,9 +98,9 @@ func (h *handlers) handleResolveQuarantine(w http.ResponseWriter, r *http.Reques writeAdminJSONErr(w, http.StatusBadRequest, "invalid_id") return } - action, err := h.lidarrQuarantine.Resolve(r.Context(), id, admin.ID) + action, err := h.quarantine.Resolve(r.Context(), id, admin.ID) if err != nil { - if errors.Is(err, lidarrquarantine.ErrTrackNotFound) { + if errors.Is(err, quarantine.ErrTrackNotFound) { writeAdminJSONErr(w, http.StatusNotFound, "track_not_found") return } @@ -119,7 +116,8 @@ func (h *handlers) handleResolveQuarantine(w http.ResponseWriter, r *http.Reques }) } -// handleDeleteQuarantineFile implements POST /api/admin/quarantine/{track_id}/delete-file. +// handleDeleteQuarantineFile implements POST +// /api/admin/quarantine/{track_id}/delete-file. func (h *handlers) handleDeleteQuarantineFile(w http.ResponseWriter, r *http.Request) { admin, ok := auth.UserFromContext(r.Context()) if !ok { @@ -131,17 +129,17 @@ func (h *handlers) handleDeleteQuarantineFile(w http.ResponseWriter, r *http.Req writeAdminJSONErr(w, http.StatusBadRequest, "invalid_id") return } - action, err := h.lidarrQuarantine.DeleteFile(r.Context(), id, admin.ID) + action, err := h.quarantine.DeleteFile(r.Context(), id, admin.ID) if err != nil { - // Written in the enveloped shape, not writeAdminJSONErr's bare code: the - // message is the part that tells the operator which directory and uid. + // Written in the enveloped shape, not writeAdminJSONErr's bare code: + // the message is the part that tells an admin which directory and uid. if apiErr, ok := fileRemoveAPIError(err); ok { logFileRemoveFailure(h.logger, apiErr, "track_id", uuidToString(id)) writeErr(w, apiErr) return } switch { - case errors.Is(err, lidarrquarantine.ErrTrackNotFound): + case errors.Is(err, quarantine.ErrTrackNotFound): writeAdminJSONErr(w, http.StatusNotFound, "track_not_found") default: h.logger.Error("admin: delete file", "err", err) @@ -158,60 +156,19 @@ func (h *handlers) handleDeleteQuarantineFile(w http.ResponseWriter, r *http.Req }) } -// handleDeleteQuarantineViaLidarr implements POST /api/admin/quarantine/{track_id}/delete-via-lidarr. -func (h *handlers) handleDeleteQuarantineViaLidarr(w http.ResponseWriter, r *http.Request) { - admin, ok := auth.UserFromContext(r.Context()) - if !ok { - writeAdminJSONErr(w, http.StatusUnauthorized, "unauthorized") - return - } - id, ok := parseUUID(chi.URLParam(r, "track_id")) - if !ok { - writeAdminJSONErr(w, http.StatusBadRequest, "invalid_id") - return - } - action, deleted, err := h.lidarrQuarantine.DeleteViaLidarr(r.Context(), id, admin.ID) - if err != nil { - switch { - case errors.Is(err, lidarrquarantine.ErrLidarrDisabled): - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_disabled") - case errors.Is(err, lidarrquarantine.ErrTrackNotFound): - writeAdminJSONErr(w, http.StatusNotFound, "track_not_found") - case errors.Is(err, lidarrquarantine.ErrAlbumMBIDMissing): - writeAdminJSONErr(w, http.StatusNotFound, "album_mbid_missing") - case errors.Is(err, lidarrquarantine.ErrLidarrAlbumNotFound): - writeAdminJSONErr(w, http.StatusBadGateway, "lidarr_album_lookup_failed") - case errors.Is(err, lidarr.ErrUnreachable): - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_unreachable") - case errors.Is(err, lidarr.ErrAuthFailed): - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_auth_failed") - default: - h.logger.Error("admin: delete via lidarr", "err", err) - writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") - } - return - } - // Broadcast: same rationale as delete-file — track row gone everywhere. - h.publishQuarantineEvent("quarantine.deleted_via_lidarr", pgtype.UUID{}, id, true) - writeJSON(w, http.StatusOK, actionResultView{ - ActionID: uuidToString(action.ID), - AffectedUsers: action.AffectedUsers, - DeletedTrackCount: &deleted, - }) -} - // actionLogView is the row shape returned by GET /api/admin/quarantine/actions. type actionLogView struct { - ID string `json:"id"` - TrackID string `json:"track_id"` - TrackTitle string `json:"track_title"` - ArtistName string `json:"artist_name"` - AlbumTitle *string `json:"album_title,omitempty"` - Action string `json:"action"` - AdminID *string `json:"admin_id,omitempty"` - LidarrAlbumMBID *string `json:"lidarr_album_mbid,omitempty"` - AffectedUsers int32 `json:"affected_users"` - CreatedAt string `json:"created_at"` + ID string `json:"id"` + TrackID string `json:"track_id"` + TrackTitle string `json:"track_title"` + ArtistName string `json:"artist_name"` + AlbumTitle *string `json:"album_title,omitempty"` + Action string `json:"action"` + AdminID *string `json:"admin_id,omitempty"` + // AlbumMBID is set on the older deleted_album rows: the album removed. + AlbumMBID *string `json:"album_mbid,omitempty"` + AffectedUsers int32 `json:"affected_users"` + CreatedAt string `json:"created_at"` } // handleListQuarantineActions implements GET /api/admin/quarantine/actions. @@ -241,16 +198,16 @@ func (h *handlers) handleListQuarantineActions(w http.ResponseWriter, r *http.Re adminID = &s } out = append(out, actionLogView{ - ID: uuidToString(row.ID), - TrackID: uuidToString(row.TrackID), - TrackTitle: row.TrackTitle, - ArtistName: row.ArtistName, - AlbumTitle: row.AlbumTitle, - Action: string(row.Action), - AdminID: adminID, - LidarrAlbumMBID: row.LidarrAlbumMbid, - AffectedUsers: row.AffectedUsers, - CreatedAt: formatTimestamp(row.CreatedAt), + ID: uuidToString(row.ID), + TrackID: uuidToString(row.TrackID), + TrackTitle: row.TrackTitle, + ArtistName: row.ArtistName, + AlbumTitle: row.AlbumTitle, + Action: string(row.Action), + AdminID: adminID, + AlbumMBID: row.AlbumMbid, + AffectedUsers: row.AffectedUsers, + CreatedAt: formatTimestamp(row.CreatedAt), }) } writeJSON(w, http.StatusOK, out) diff --git a/internal/api/admin_quarantine_test.go b/internal/api/admin_quarantine_test.go index 5318d5b3..e412adfd 100644 --- a/internal/api/admin_quarantine_test.go +++ b/internal/api/admin_quarantine_test.go @@ -15,9 +15,6 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/auth" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrquarantine" ) // newAdminQuarantineRouter builds a test chi router with the five admin @@ -30,7 +27,6 @@ func newAdminQuarantineRouter(h *handlers) chi.Router { admin.Get("/quarantine", h.handleListAdminQuarantine) admin.Post("/quarantine/{track_id}/resolve", h.handleResolveQuarantine) admin.Post("/quarantine/{track_id}/delete-file", h.handleDeleteQuarantineFile) - admin.Post("/quarantine/{track_id}/delete-via-lidarr", h.handleDeleteQuarantineViaLidarr) admin.Get("/quarantine/actions", h.handleListQuarantineActions) }) return r @@ -56,27 +52,11 @@ func doAdminQuarantineReq(t *testing.T, h *handlers, method, path string, body [ return w } -// installQuarantineClientFn rewires h.lidarrQuarantine to a fresh Service -// with a real clientFn pointing at the (current) saved lidarr_config. This -// mirrors testHandlersWithClientFn for the requests test file. -func installQuarantineClientFn(t *testing.T, h *handlers) { - t.Helper() - cfg := lidarrconfig.New(h.pool) - clientFn := func() *lidarr.Client { - c, err := cfg.Get(context.Background()) - if err != nil || !c.Enabled || c.BaseURL == "" { - return nil - } - return lidarr.NewClient(c.BaseURL, c.APIKey) - } - h.lidarrQuarantine = lidarrquarantine.NewService(h.pool, cfg, clientFn, h.dataDir) -} - // flagDirect bypasses the HTTP handler to seed a quarantine row via the // Service for setup. Used to construct admin queue scenarios. func flagDirect(t *testing.T, h *handlers, userID, trackID pgtype.UUID, reason string) { t.Helper() - if _, err := h.lidarrQuarantine.Flag(context.Background(), userID, trackID, reason, ""); err != nil { + if _, err := h.quarantine.Flag(context.Background(), userID, trackID, reason, ""); err != nil { t.Fatalf("flagDirect: %v", err) } } @@ -110,49 +90,13 @@ func seedTrackOnDisk(t *testing.T, h *handlers, unique string) (dbq.Track, strin return tr, path } -// seedTrackWithAlbumMBID seeds an artist+album-with-mbid+track. The mbid is -// required for delete-via-lidarr scenarios. -func seedTrackWithAlbumMBID(t *testing.T, h *handlers, unique, albumMBID string) (dbq.Track, string) { - t.Helper() - artist := seedArtist(t, h.pool, "DvL Artist "+unique) - dir := t.TempDir() - path := filepath.Join(dir, "track-"+unique+".mp3") - if err := os.WriteFile(path, []byte("audio"), 0o644); err != nil { - t.Fatalf("write file: %v", err) - } - mbid := albumMBID - album, err := dbq.New(h.pool).UpsertAlbum(context.Background(), dbq.UpsertAlbumParams{ - Title: "DvL Album " + unique, SortTitle: "DvL Album " + unique, - ArtistID: artist.ID, Mbid: &mbid, - }) - if err != nil { - t.Fatalf("UpsertAlbum: %v", err) - } - one := int32(1) - tr, err := dbq.New(h.pool).UpsertTrack(context.Background(), dbq.UpsertTrackParams{ - Title: "DvL Track " + unique, - AlbumID: album.ID, - ArtistID: artist.ID, - TrackNumber: &one, - DurationMs: 30000, - FilePath: path, - FileSize: 5, - FileFormat: "mp3", - }) - if err != nil { - t.Fatalf("UpsertTrack: %v", err) - } - return tr, path -} - // TestListAdminQuarantine_AggregatedShape seeds 3 users flagging the same // track with mixed reasons. Verifies one aggregated row with report_count=3, // reason_counts populated, and a Reports list of length 3. func TestListAdminQuarantine_AggregatedShape(t *testing.T) { h, pool := testHandlers(t) truncateLibrary(t, pool) - resetLidarrState(t, h) - installQuarantineClientFn(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) bob := seedUser(t, pool, "bob", "pw", false) @@ -192,8 +136,7 @@ func TestListAdminQuarantine_AggregatedShape(t *testing.T) { func TestResolveQuarantine_HappyPath(t *testing.T) { h, pool := testHandlers(t) truncateLibrary(t, pool) - resetLidarrState(t, h) - installQuarantineClientFn(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) bob := seedUser(t, pool, "bob", "pw", false) @@ -227,7 +170,7 @@ func TestResolveQuarantine_HappyPath(t *testing.T) { if len(actions) != 1 { t.Fatalf("audit rows = %d, want 1", len(actions)) } - if actions[0].Action != dbq.LidarrQuarantineActionKindResolved { + if actions[0].Action != dbq.QuarantineActionKindResolved { t.Errorf("audit action = %v, want resolved", actions[0].Action) } } @@ -237,8 +180,7 @@ func TestResolveQuarantine_HappyPath(t *testing.T) { func TestDeleteQuarantineFile_HappyPath(t *testing.T) { h, pool := testHandlers(t) truncateLibrary(t, pool) - resetLidarrState(t, h) - installQuarantineClientFn(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) admin := seedUser(t, pool, "admin", "pw", true) @@ -268,141 +210,12 @@ func TestDeleteQuarantineFile_HappyPath(t *testing.T) { } } -// TestDeleteQuarantineViaLidarr_HappyPath uses a stub Lidarr that returns -// a single album for the lookup and 200 for the delete. Verifies the -// audit row is written and the local track row is deleted. -func TestDeleteQuarantineViaLidarr_HappyPath(t *testing.T) { - h, pool := testHandlers(t) - truncateLibrary(t, pool) - resetLidarrState(t, h) - installQuarantineClientFn(t, h) - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - if r.URL.Path == "/api/v1/album" && r.Method == http.MethodGet { - _, _ = w.Write([]byte(`[{"id":42,"foreignAlbumId":"al-mbid-dvl","title":"Al","artistId":7}]`)) - return - } - w.WriteHeader(http.StatusOK) - })) - t.Cleanup(stub.Close) - saveLidarrConfig(t, h, stub.URL, true) - installQuarantineClientFn(t, h) // re-install after config save so clientFn picks it up - - alice := seedUser(t, pool, "alice", "pw", false) - admin := seedUser(t, pool, "admin", "pw", true) - - track, _ := seedTrackWithAlbumMBID(t, h, "dvl-happy", "al-mbid-dvl") - flagDirect(t, h, alice.ID, track.ID, "bad_rip") - - w := doAdminQuarantineReq(t, h, http.MethodPost, - "/api/admin/quarantine/"+uuidToString(track.ID)+"/delete-via-lidarr", nil, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var got actionResultView - if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if got.AffectedUsers != 1 { - t.Errorf("affected_users = %d, want 1", got.AffectedUsers) - } - if got.DeletedTrackCount == nil || *got.DeletedTrackCount != 1 { - t.Errorf("deleted_track_count = %v, want 1", got.DeletedTrackCount) - } - - // Track row should be gone. - if _, err := dbq.New(pool).GetTrackByID(context.Background(), track.ID); err == nil { - t.Error("track row still exists after delete-via-lidarr") - } - // Audit row written with the lidarr_album_mbid. - actions, err := dbq.New(pool).ListQuarantineActions(context.Background(), 50) - if err != nil { - t.Fatalf("list actions: %v", err) - } - if len(actions) != 1 { - t.Fatalf("audit rows = %d, want 1", len(actions)) - } - if actions[0].Action != dbq.LidarrQuarantineActionKindDeletedViaLidarr { - t.Errorf("audit action = %v, want deleted_via_lidarr", actions[0].Action) - } - if actions[0].LidarrAlbumMbid == nil || *actions[0].LidarrAlbumMbid != "al-mbid-dvl" { - t.Errorf("audit lidarr_album_mbid = %v", actions[0].LidarrAlbumMbid) - } -} - -// TestDeleteQuarantineViaLidarr_LidarrDisabled verifies 503 when Lidarr is -// not configured. -func TestDeleteQuarantineViaLidarr_LidarrDisabled(t *testing.T) { - h, pool := testHandlers(t) - truncateLibrary(t, pool) - resetLidarrState(t, h) - installQuarantineClientFn(t, h) - - alice := seedUser(t, pool, "alice", "pw", false) - admin := seedUser(t, pool, "admin", "pw", true) - - track, _ := seedTrackWithAlbumMBID(t, h, "dvl-disabled", "al-mbid-disabled") - flagDirect(t, h, alice.ID, track.ID, "bad_rip") - - // Lidarr config is reset (disabled). - w := doAdminQuarantineReq(t, h, http.MethodPost, - "/api/admin/quarantine/"+uuidToString(track.ID)+"/delete-via-lidarr", nil, admin) - if w.Code != http.StatusServiceUnavailable { - t.Fatalf("status = %d, want 503; body = %s", w.Code, w.Body.String()) - } - var resp map[string]string - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp["error"] != "lidarr_disabled" { - t.Errorf("error = %q, want lidarr_disabled", resp["error"]) - } -} - -// TestDeleteQuarantineViaLidarr_AlbumMBIDMissing verifies 404 album_mbid_missing -// when the parent album has no mbid. -func TestDeleteQuarantineViaLidarr_AlbumMBIDMissing(t *testing.T) { - h, pool := testHandlers(t) - truncateLibrary(t, pool) - resetLidarrState(t, h) - installQuarantineClientFn(t, h) - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusOK) - })) - t.Cleanup(stub.Close) - saveLidarrConfig(t, h, stub.URL, true) - installQuarantineClientFn(t, h) - - alice := seedUser(t, pool, "alice", "pw", false) - admin := seedUser(t, pool, "admin", "pw", true) - - // Track whose album has NO mbid. - track, _ := seedTrackOnDisk(t, h, "no-mbid") - flagDirect(t, h, alice.ID, track.ID, "bad_rip") - - w := doAdminQuarantineReq(t, h, http.MethodPost, - "/api/admin/quarantine/"+uuidToString(track.ID)+"/delete-via-lidarr", nil, admin) - if w.Code != http.StatusNotFound { - t.Fatalf("status = %d, want 404; body = %s", w.Code, w.Body.String()) - } - var resp map[string]string - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if resp["error"] != "album_mbid_missing" { - t.Errorf("error = %q, want album_mbid_missing", resp["error"]) - } -} - // TestQuarantineAdminEndpointsRequire403ForNonAdmin verifies all five admin // endpoints reject non-admin callers with 403. func TestQuarantineAdminEndpointsRequire403ForNonAdmin(t *testing.T) { h, pool := testHandlers(t) truncateLibrary(t, pool) - resetLidarrState(t, h) - installQuarantineClientFn(t, h) + resetRequests(t, h) nonAdmin := seedUser(t, pool, "regular", "pw", false) fakeID := "00000000-0000-0000-0000-000000000001" @@ -414,7 +227,6 @@ func TestQuarantineAdminEndpointsRequire403ForNonAdmin(t *testing.T) { {http.MethodGet, "/api/admin/quarantine"}, {http.MethodPost, "/api/admin/quarantine/" + fakeID + "/resolve"}, {http.MethodPost, "/api/admin/quarantine/" + fakeID + "/delete-file"}, - {http.MethodPost, "/api/admin/quarantine/" + fakeID + "/delete-via-lidarr"}, {http.MethodGet, "/api/admin/quarantine/actions"}, } for _, tc := range cases { diff --git a/internal/api/admin_reacquisition.go b/internal/api/admin_reacquisition.go index b4f3b30a..01d841c1 100644 --- a/internal/api/admin_reacquisition.go +++ b/internal/api/admin_reacquisition.go @@ -10,31 +10,29 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/reacquisition" ) -// reacquisitionSettingsResp is the admin card's payload (milestone #290). +// reacquisitionSettingsResp is the admin card's payload. // // Carries more than the stored settings: [UnnameableAlbums] is the count of -// albums with missing files that can never be auto-requested because neither -// they nor their artist has an MBID. The card states that number rather than -// leaving the operator to wonder why some rows never get a request — a -// feature that silently does nothing for part of its input reads as broken. +// albums with missing files that can never be wanted automatically because +// neither they nor their artist has an MBID. The card states that number +// rather than leaving an admin to wonder why some rows never get wanted — +// a feature that silently does nothing for part of its input reads as broken. type reacquisitionSettingsResp struct { - Enabled bool `json:"enabled"` - GraceHours int32 `json:"grace_hours"` - BackoffBaseHours int32 `json:"backoff_base_hours"` - BackoffMaxHours int32 `json:"backoff_max_hours"` - MaxAttempts int32 `json:"max_attempts"` - MaxPerPass int32 `json:"max_per_pass"` - AutoApprove bool `json:"auto_approve"` + Enabled bool `json:"enabled"` + GraceHours int32 `json:"grace_hours"` + MaxPerPass int32 `json:"max_per_pass"` UnnameableAlbums int64 `json:"unnameable_albums"` } -// handleGetReacquisitionSettings implements GET /api/admin/library/reacquisition. +// handleGetReacquisitionSettings implements GET +// /api/admin/library/reacquisition. func (h *handlers) handleGetReacquisitionSettings(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, h.reacquisitionPayload(r)) } -// handleUpdateReacquisitionSettings implements PUT /api/admin/library/reacquisition. +// handleUpdateReacquisitionSettings implements PUT +// /api/admin/library/reacquisition. func (h *handlers) handleUpdateReacquisitionSettings(w http.ResponseWriter, r *http.Request) { var req reacquisitionSettingsResp if err := json.NewDecoder(r.Body).Decode(&req); err != nil { @@ -42,13 +40,9 @@ func (h *handlers) handleUpdateReacquisitionSettings(w http.ResponseWriter, r *h return } _, err := h.reacqSettings.Set(r.Context(), reacquisition.Settings{ - Enabled: req.Enabled, - GraceHours: req.GraceHours, - BackoffBaseHours: req.BackoffBaseHours, - BackoffMaxHours: req.BackoffMaxHours, - MaxAttempts: req.MaxAttempts, - MaxPerPass: req.MaxPerPass, - AutoApprove: req.AutoApprove, + Enabled: req.Enabled, + GraceHours: req.GraceHours, + MaxPerPass: req.MaxPerPass, }) if err != nil { // The Go-side validation mirrors migration 0056's CHECKs so the @@ -69,13 +63,9 @@ func (h *handlers) handleUpdateReacquisitionSettings(w http.ResponseWriter, r *h func (h *handlers) reacquisitionPayload(r *http.Request) reacquisitionSettingsResp { cur := h.reacqSettings.Get() out := reacquisitionSettingsResp{ - Enabled: cur.Enabled, - GraceHours: cur.GraceHours, - BackoffBaseHours: cur.BackoffBaseHours, - BackoffMaxHours: cur.BackoffMaxHours, - MaxAttempts: cur.MaxAttempts, - MaxPerPass: cur.MaxPerPass, - AutoApprove: cur.AutoApprove, + Enabled: cur.Enabled, + GraceHours: cur.GraceHours, + MaxPerPass: cur.MaxPerPass, } // Best-effort: the settings are the point of this endpoint, and failing // the whole card because a count query hiccuped would be the wrong trade. diff --git a/internal/api/admin_recommendation_trends.go b/internal/api/admin_recommendation_trends.go index d9e8922c..4d306b9c 100644 --- a/internal/api/admin_recommendation_trends.go +++ b/internal/api/admin_recommendation_trends.go @@ -1,6 +1,6 @@ -// Admin recommendation-trends endpoint (#1251): weekly per-surface -// outcome series with tuning-audit markers — the verify half of the -// tune→verify loop the tuning lab (#1250) opens. Aggregated across +// Admin recommendation-trends endpoint: weekly per-surface outcome +// series with tuning-audit markers — the verify half of the tune→verify +// loop the tuning card opens. Aggregated across // all users because the knobs are global; rows carry rates only. package api diff --git a/internal/api/admin_recommendation_tuning.go b/internal/api/admin_recommendation_tuning.go index 2d52633e..d5e69799 100644 --- a/internal/api/admin_recommendation_tuning.go +++ b/internal/api/admin_recommendation_tuning.go @@ -1,8 +1,8 @@ -// Admin recommendation-tuning endpoints (#1250): the defaults- -// discovery lab. GET returns current values + shipped defaults for +// Admin recommendation-tuning endpoints: the lab for finding good +// defaults. GET returns current values + shipped defaults for // every scope; PATCH applies a partial update to one scope; reset // restores a scope to shipped defaults. Every change writes an audit -// row (consumed by the metrics trend view, #1251). +// row (consumed by the metrics trend view). package api import ( @@ -67,7 +67,7 @@ func tasteRespFrom(t recsettings.TasteTuning) tasteTuningResp { } } -// discoverTuningResp is the Discover scope on the wire (#2377). +// discoverTuningResp is the Discover scope on the wire. type discoverTuningResp struct { TagOverlapWeight float64 `json:"tag_overlap_weight"` SnoozeDays float64 `json:"snooze_days"` @@ -113,7 +113,8 @@ func (h *handlers) tuningSnapshot() tuningSnapshot { return out } -// handleGetRecommendationTuning implements GET /api/admin/recommendation-tuning. +// handleGetRecommendationTuning implements GET +// /api/admin/recommendation-tuning. func (h *handlers) handleGetRecommendationTuning(w http.ResponseWriter, _ *http.Request) { writeJSON(w, http.StatusOK, h.tuningSnapshot()) } diff --git a/internal/api/admin_requests.go b/internal/api/admin_requests.go index 1d63e487..4fd395eb 100644 --- a/internal/api/admin_requests.go +++ b/internal/api/admin_requests.go @@ -1,18 +1,20 @@ package api import ( + "context" "encoding/json" "errors" "net/http" "strconv" "github.com/go-chi/chi/v5" + "github.com/jackc/pgx/v5/pgtype" "git.fabledsword.com/bvandeusen/minstrel/internal/auth" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" + "git.fabledsword.com/bvandeusen/minstrel/internal/requests" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" ) // validRequestStatuses is the set of allowed values for the ?status= param. @@ -50,7 +52,7 @@ func (h *handlers) handleListAdminRequests(w http.ResponseWriter, r *http.Reques limit = v } - rows, err := h.lidarrRequests.ListByStatus(r.Context(), status, int32(limit)) + rows, err := h.requests.ListByStatus(r.Context(), status, int32(limit)) if err != nil { h.logger.Error("admin: list requests", "err", err) writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") @@ -58,22 +60,46 @@ func (h *handlers) handleListAdminRequests(w http.ResponseWriter, r *http.Reques } q := dbq.New(h.pool) + names := requesterNames(r.Context(), q, rows) out := make([]requestView, 0, len(rows)) for _, row := range rows { v := requestViewFrom(row) + v.Username = names[row.UserID] fillProgress(r.Context(), q, row, &v) out = append(out, v) } writeJSON(w, http.StatusOK, out) } -// approveRequestBody is the optional JSON body for POST /api/admin/requests/:id/approve. -type approveRequestBody struct { - QualityProfileID int `json:"quality_profile_id"` - RootFolderPath string `json:"root_folder_path"` +// requesterNames maps each request's user id to that user's username, in one +// query for the whole page. Like fillProgress, a failed lookup is swallowed: +// the names are a reporting nicety, and the page falls back to the id. +func requesterNames(ctx context.Context, q *dbq.Queries, rows []dbq.Request) map[pgtype.UUID]string { + ids := make([]pgtype.UUID, 0, len(rows)) + for _, row := range rows { + ids = append(ids, row.UserID) + } + names := make(map[pgtype.UUID]string, len(rows)) + users, err := q.ListUsernamesByIDs(ctx, ids) + if err != nil { + return names + } + for _, u := range users { + names[u.ID] = u.Username + } + return names } -// handleApproveRequest implements POST /api/admin/requests/:id/approve. +// approveRequestBody is the optional JSON body for POST +// /api/admin/requests/:id/approve. profile_id picks the release-type profile an +// artist request is followed on; omitted, the default profile. +type approveRequestBody struct { + ProfileID string `json:"profile_id"` +} + +// handleApproveRequest implements POST /api/admin/requests/:id/approve. The +// request goes on the wanted list: an album or track request +// wants its album, an artist request follows the artist. func (h *handlers) handleApproveRequest(w http.ResponseWriter, r *http.Request) { admin, ok := auth.UserFromContext(r.Context()) if !ok { @@ -90,35 +116,25 @@ func (h *handlers) handleApproveRequest(w http.ResponseWriter, r *http.Request) var body approveRequestBody // Body is entirely optional; ignore decode errors. _ = json.NewDecoder(r.Body).Decode(&body) + var profile pgtype.UUID + if body.ProfileID != "" { + if profile, ok = parseUUID(body.ProfileID); !ok { + writeAdminJSONErr(w, http.StatusBadRequest, "invalid_profile") + return + } + } - row, err := h.lidarrRequests.Approve(r.Context(), id, admin.ID, lidarrrequests.ApproveOverrides{ - QualityProfileID: body.QualityProfileID, - RootFolderPath: body.RootFolderPath, - }) + row, err := h.requests.Approve(r.Context(), id, admin.ID, profile) if err != nil { switch { - case errors.Is(err, lidarrrequests.ErrLidarrDisabled): - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_disabled") - case errors.Is(err, lidarrrequests.ErrDefaultsIncomplete): - writeAdminJSONErr(w, http.StatusBadRequest, "lidarr_defaults_incomplete") - case errors.Is(err, lidarrrequests.ErrNotFound): + case errors.Is(err, requests.ErrNotFound): writeAdminJSONErr(w, http.StatusNotFound, "request_not_found") - case errors.Is(err, lidarrrequests.ErrNotPending): + case errors.Is(err, requests.ErrNotPending): writeAdminJSONErr(w, http.StatusConflict, "request_not_pending") - case errors.Is(err, lidarr.ErrUnreachable): - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_unreachable") - case errors.Is(err, lidarr.ErrAuthFailed): - writeAdminJSONErr(w, http.StatusServiceUnavailable, "lidarr_auth_failed") - case errors.Is(err, lidarr.ErrServerError): - // Lidarr itself 5xx'd — its response body is in err's wrapped - // message (see lidarr.readBodySnippet). Logged below for ops. - h.logger.Error("admin: approve request: lidarr 5xx", "err", err) - writeAdminJSONErr(w, http.StatusBadGateway, "lidarr_server_error") - case errors.Is(err, lidarr.ErrLookupFailed): - // Lidarr returned 4xx — usually "artist already exists" or - // "no metadata found." Log so ops can see the actual reason. - h.logger.Warn("admin: approve request: lidarr 4xx", "err", err) - writeAdminJSONErr(w, http.StatusBadGateway, "lidarr_rejected") + case errors.Is(err, wanted.ErrProfileNotFound): + writeAdminJSONErr(w, http.StatusBadRequest, "profile_not_found") + case errors.Is(err, wanted.ErrNotFollowable): + writeAdminJSONErr(w, http.StatusBadRequest, "not_followable") default: h.logger.Error("admin: approve request", "err", err) writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") @@ -131,7 +147,8 @@ func (h *handlers) handleApproveRequest(w http.ResponseWriter, r *http.Request) writeJSON(w, http.StatusOK, requestViewFrom(row)) } -// rejectRequestBody is the optional JSON body for POST /api/admin/requests/:id/reject. +// rejectRequestBody is the optional JSON body for POST +// /api/admin/requests/:id/reject. type rejectRequestBody struct { Notes string `json:"notes"` } @@ -153,12 +170,12 @@ func (h *handlers) handleRejectRequest(w http.ResponseWriter, r *http.Request) { var body rejectRequestBody _ = json.NewDecoder(r.Body).Decode(&body) - row, err := h.lidarrRequests.Reject(r.Context(), id, admin.ID, body.Notes) + row, err := h.requests.Reject(r.Context(), id, admin.ID, body.Notes) if err != nil { switch { - case errors.Is(err, lidarrrequests.ErrNotFound): + case errors.Is(err, requests.ErrNotFound): writeAdminJSONErr(w, http.StatusNotFound, "request_not_found") - case errors.Is(err, lidarrrequests.ErrNotPending): + case errors.Is(err, requests.ErrNotPending): writeAdminJSONErr(w, http.StatusConflict, "request_not_pending") default: h.logger.Error("admin: reject request", "err", err) diff --git a/internal/api/admin_requests_test.go b/internal/api/admin_requests_test.go index 2106b5ed..734eadfc 100644 --- a/internal/api/admin_requests_test.go +++ b/internal/api/admin_requests_test.go @@ -6,16 +6,12 @@ import ( "encoding/json" "net/http" "net/http/httptest" - "strings" "testing" "github.com/go-chi/chi/v5" "git.fabledsword.com/bvandeusen/minstrel/internal/auth" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" ) // newAdminRequestsRouter builds a test chi router with the three admin/requests @@ -52,35 +48,19 @@ func doAdminRequestReq(t *testing.T, h *handlers, method, path string, body []by return w } -// testHandlersWithClientFn creates a handlers instance with a real clientFn -// so Service.Approve actually calls Lidarr during tests. -func testHandlersWithClientFn(t *testing.T) (*handlers, *lidarrconfig.Service) { - t.Helper() - h, _ := testHandlers(t) - lidarrCfg := lidarrconfig.New(h.pool) - clientFn := func() *lidarr.Client { - cfg, err := lidarrCfg.Get(context.Background()) - if err != nil || !cfg.Enabled || cfg.BaseURL == "" { - return nil - } - return lidarr.NewClient(cfg.BaseURL, cfg.APIKey) - } - h.lidarrRequests = lidarrrequests.NewService(h.pool, lidarrCfg, clientFn, nil) - return h, lidarrCfg -} - -// seedPendingArtistRequest creates a pending artist-kind lidarr request row +// seedPendingArtistRequest creates a pending artist-kind request row // for the given user and returns its parsed requestView. func seedPendingArtistRequest(t *testing.T, h *handlers, user dbq.User, artistMBID, artistName string) requestView { t.Helper() return createArtistRequest(t, h, user, artistMBID, artistName) } -// TestHandleListAdminRequests_DefaultStatus_Pending seeds 2 pending + 1 rejected -// and verifies that GET without ?status returns only the 2 pending rows. +// TestHandleListAdminRequests_DefaultStatus_Pending seeds 2 pending + 1 +// rejected and verifies that GET without ?status returns only the 2 pending +// rows. func TestHandleListAdminRequests_DefaultStatus_Pending(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) + h, _ := testHandlers(t) + resetRequests(t, h) alice := seedUser(t, h.pool, "alice", "pw", false) admin := seedUser(t, h.pool, "admin", "pw", true) @@ -90,7 +70,7 @@ func TestHandleListAdminRequests_DefaultStatus_Pending(t *testing.T) { // Reject one request directly via service. rv3 := createArtistRequest(t, h, alice, "mbid-reject-1", "Reject Me") - _, err := h.lidarrRequests.Reject(context.Background(), rv3.ID, admin.ID, "test") + _, err := h.requests.Reject(context.Background(), rv3.ID, admin.ID, "test") if err != nil { t.Fatalf("reject seed: %v", err) } @@ -111,14 +91,17 @@ func TestHandleListAdminRequests_DefaultStatus_Pending(t *testing.T) { if rv.Status != "pending" { t.Errorf("result status = %q, want pending", rv.Status) } + if rv.Username != alice.Username { + t.Errorf("result username = %q, want %q (the requester)", rv.Username, alice.Username) + } } } -// TestHandleListAdminRequests_StatusFilter verifies GET ?status=rejected returns -// only rejected rows. +// TestHandleListAdminRequests_StatusFilter verifies GET ?status=rejected +// returns only rejected rows. func TestHandleListAdminRequests_StatusFilter(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) + h, _ := testHandlers(t) + resetRequests(t, h) alice := seedUser(t, h.pool, "alice", "pw", false) admin := seedUser(t, h.pool, "admin", "pw", true) @@ -126,7 +109,7 @@ func TestHandleListAdminRequests_StatusFilter(t *testing.T) { createArtistRequest(t, h, alice, "mbid-pending-f1", "Pending Filter 1") rv2 := createArtistRequest(t, h, alice, "mbid-rejected-f1", "Rejected Filter 1") - _, err := h.lidarrRequests.Reject(context.Background(), rv2.ID, admin.ID, "") + _, err := h.requests.Reject(context.Background(), rv2.ID, admin.ID, "") if err != nil { t.Fatalf("reject seed: %v", err) } @@ -151,8 +134,8 @@ func TestHandleListAdminRequests_StatusFilter(t *testing.T) { // TestHandleListAdminRequests_BadStatus_400 verifies GET with an invalid // ?status returns 400. func TestHandleListAdminRequests_BadStatus_400(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) + h, _ := testHandlers(t) + resetRequests(t, h) admin := seedUser(t, h.pool, "admin", "pw", true) @@ -169,190 +152,11 @@ func TestHandleListAdminRequests_BadStatus_400(t *testing.T) { } } -// TestHandleApproveRequest_HappyPath seeds a Lidarr stub, seeds a pending -// artist request, and verifies POST /approve → 200 with row status=approved. -func TestHandleApproveRequest_HappyPath(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - switch { - case strings.Contains(r.URL.Path, "/metadataprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Standard"}]`)) - case strings.Contains(r.URL.Path, "/qualityprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Lossless"}]`)) - default: - _, _ = w.Write([]byte(`{"id":1}`)) - } - })) - t.Cleanup(stub.Close) - - saveLidarrConfig(t, h, stub.URL, true) - - alice := seedUser(t, h.pool, "alice", "pw", false) - admin := seedUser(t, h.pool, "admin", "pw", true) - - rv := seedPendingArtistRequest(t, h, alice, "artist-mbid-approve", "Approve Me") - - w := doAdminRequestReq(t, h, http.MethodPost, "/api/admin/requests/"+uuidToString(rv.ID)+"/approve", nil, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - - var got requestView - if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if got.Status != "approved" { - t.Errorf("status = %q, want approved", got.Status) - } - if got.ID != rv.ID { - t.Errorf("id mismatch") - } -} - -// TestHandleApproveRequest_OverrideUsed seeds config with default qp=1/root="/m", -// approves with override qp=5/root="/other", verifies the stub received those -// values and the row reflects the overrides. -func TestHandleApproveRequest_OverrideUsed(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) - - var capturedBody []byte - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.Method == http.MethodPost { - buf := make([]byte, 4096) - n, _ := r.Body.Read(buf) - capturedBody = buf[:n] - } - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - switch { - case strings.Contains(r.URL.Path, "/metadataprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Standard"}]`)) - case strings.Contains(r.URL.Path, "/qualityprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Lossless"}]`)) - default: - _, _ = w.Write([]byte(`{"id":1}`)) - } - })) - t.Cleanup(stub.Close) - - // Save config with default qp=1, root="/m". - if err := lidarrconfig.New(h.pool).Save(context.Background(), lidarrconfig.Config{ - Enabled: true, - BaseURL: stub.URL, - APIKey: "test-key", - DefaultQualityProfileID: 1, - DefaultRootFolderPath: "/m", - }); err != nil { - t.Fatalf("save config: %v", err) - } - - alice := seedUser(t, h.pool, "alice", "pw", false) - admin := seedUser(t, h.pool, "admin", "pw", true) - - rv := seedPendingArtistRequest(t, h, alice, "artist-mbid-override", "Override Me") - - body := []byte(`{"quality_profile_id":5,"root_folder_path":"/other"}`) - w := doAdminRequestReq(t, h, http.MethodPost, "/api/admin/requests/"+uuidToString(rv.ID)+"/approve", body, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - - // Verify stub received the override values. - var stubPayload map[string]any - if err := json.Unmarshal(capturedBody, &stubPayload); err != nil { - t.Fatalf("decode stub payload: %v; body = %s", err, capturedBody) - } - if qp, _ := stubPayload["qualityProfileId"].(float64); int(qp) != 5 { - t.Errorf("stub qualityProfileId = %v, want 5", stubPayload["qualityProfileId"]) - } - if rf, _ := stubPayload["rootFolderPath"].(string); rf != "/other" { - t.Errorf("stub rootFolderPath = %q, want /other", rf) - } - - // Verify the returned row reflects the overrides. - var got requestView - if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { - t.Fatalf("decode response: %v; body = %s", err, w.Body.String()) - } - if got.QualityProfileID == nil || *got.QualityProfileID != 5 { - t.Errorf("quality_profile_id = %v, want 5", got.QualityProfileID) - } - if got.RootFolderPath == nil || *got.RootFolderPath != "/other" { - t.Errorf("root_folder_path = %v, want /other", got.RootFolderPath) - } -} - -// TestHandleApproveRequest_LidarrUnreachable_ApprovesDeferred points config -// at an unreachable port and verifies the durable-approve contract: POST -// /approve still succeeds (200) and the request becomes approved with the -// Lidarr add deferred (lidarr_add_confirmed_at NULL) for the reconciler to -// retry — the admin's decision is never lost to a transient Lidarr outage. -func TestHandleApproveRequest_LidarrUnreachable_ApprovesDeferred(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) - - saveLidarrConfig(t, h, "http://127.0.0.1:1", true) - - alice := seedUser(t, h.pool, "alice", "pw", false) - admin := seedUser(t, h.pool, "admin", "pw", true) - - rv := seedPendingArtistRequest(t, h, alice, "artist-mbid-unreachable", "Unreachable") - - w := doAdminRequestReq(t, h, http.MethodPost, "/api/admin/requests/"+uuidToString(rv.ID)+"/approve", nil, admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200 (durable approve despite Lidarr down); body = %s", w.Code, w.Body.String()) - } - - // Must no longer be pending. - pending, err := h.lidarrRequests.ListByStatus(context.Background(), "pending", 10) - if err != nil { - t.Fatalf("list pending: %v", err) - } - for _, r := range pending { - if r.ID == rv.ID { - t.Error("row still pending after durable approve") - } - } - - // Must be approved with the add deferred (confirmed_at NULL) so the - // reconciler retries it — not failed, not lost. - approved, err := h.lidarrRequests.ListByStatus(context.Background(), "approved", 10) - if err != nil { - t.Fatalf("list approved: %v", err) - } - var got *dbq.LidarrRequest - for i := range approved { - if approved[i].ID == rv.ID { - got = &approved[i] - break - } - } - if got == nil { - t.Fatalf("row %s not in approved after approve with Lidarr down", uuidToString(rv.ID)) - } - if got.LidarrAddConfirmedAt.Valid { - t.Error("lidarr_add_confirmed_at set despite Lidarr unreachable; want NULL (deferred to reconciler)") - } -} - // TestHandleApproveRequest_NotPending_409 pre-rejects a row and verifies // POST /approve → 409 request_not_pending. func TestHandleApproveRequest_NotPending_409(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) - - // Need an enabled Lidarr config so Approve reaches the not-pending check. - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte(`{"id":1}`)) - })) - t.Cleanup(stub.Close) - saveLidarrConfig(t, h, stub.URL, true) + h, _ := testHandlers(t) + resetRequests(t, h) alice := seedUser(t, h.pool, "alice", "pw", false) admin := seedUser(t, h.pool, "admin", "pw", true) @@ -360,7 +164,7 @@ func TestHandleApproveRequest_NotPending_409(t *testing.T) { rv := seedPendingArtistRequest(t, h, alice, "artist-mbid-notpending", "Not Pending") // Pre-reject the row. - _, err := h.lidarrRequests.Reject(context.Background(), rv.ID, admin.ID, "pre-rejected") + _, err := h.requests.Reject(context.Background(), rv.ID, admin.ID, "pre-rejected") if err != nil { t.Fatalf("pre-reject: %v", err) } @@ -381,8 +185,8 @@ func TestHandleApproveRequest_NotPending_409(t *testing.T) { // TestHandleRejectRequest_HappyPath verifies POST /reject with notes returns // 200 and the row transitions to rejected with notes saved. func TestHandleRejectRequest_HappyPath(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) + h, _ := testHandlers(t) + resetRequests(t, h) alice := seedUser(t, h.pool, "alice", "pw", false) admin := seedUser(t, h.pool, "admin", "pw", true) @@ -410,8 +214,8 @@ func TestHandleRejectRequest_HappyPath(t *testing.T) { // TestHandleRejectRequest_NoNotes_OK verifies POST /reject with empty body // returns 200 (notes are optional). func TestHandleRejectRequest_NoNotes_OK(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) + h, _ := testHandlers(t) + resetRequests(t, h) alice := seedUser(t, h.pool, "alice", "pw", false) admin := seedUser(t, h.pool, "admin", "pw", true) @@ -435,8 +239,8 @@ func TestHandleRejectRequest_NoNotes_OK(t *testing.T) { // TestAdminRequestsEndpoints_NonAdmin403 is a table-driven test verifying that // all 3 admin request endpoints reject non-admin users with 403. func TestAdminRequestsEndpoints_NonAdmin403(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) + h, _ := testHandlers(t) + resetRequests(t, h) nonAdmin := seedUser(t, h.pool, "regular", "pw", false) @@ -470,3 +274,62 @@ func TestAdminRequestsEndpoints_NonAdmin403(t *testing.T) { }) } } + +// TestHandleApproveRequest_AlbumBecomesAWant: approving an album request puts +// its release group on the wanted list, as the requester's. +func TestHandleApproveRequest_AlbumBecomesAWant(t *testing.T) { + h, _ := testHandlers(t) + resetRequests(t, h) + alice := seedUser(t, h.pool, "alice", "pw", false) + admin := seedUser(t, h.pool, "admin", "pw", true) + + w := doCreateRequest(h, alice, `{"kind":"album","artist_mbid":"artist-x","artist_name":"Gorillaz","album_mbid":"group-x","album_title":"Demon Days"}`) + if w.Code != http.StatusCreated { + t.Fatalf("create: %d %s", w.Code, w.Body.String()) + } + var rv requestView + _ = json.Unmarshal(w.Body.Bytes(), &rv) + + resp := doAdminRequestReq(t, h, http.MethodPost, "/api/admin/requests/"+uuidToString(rv.ID)+"/approve", nil, admin) + if resp.Code != http.StatusOK { + t.Fatalf("approve: %d %s", resp.Code, resp.Body.String()) + } + want, err := dbq.New(h.pool).GetWantByReleaseGroup(context.Background(), "group-x") + if err != nil { + t.Fatalf("no want recorded: %v", err) + } + if want.Title != "Demon Days" || want.CreatedBy != alice.ID || want.Reason != "request" { + t.Errorf("want = %+v", want) + } +} + +// TestHandleApproveRequest_ArtistOnTheChosenProfile: an artist request is +// followed on the profile the admin picked; an unknown profile is a 400. +func TestHandleApproveRequest_ArtistOnTheChosenProfile(t *testing.T) { + h, _ := testHandlers(t) + resetRequests(t, h) + alice := seedUser(t, h.pool, "alice", "pw", false) + admin := seedUser(t, h.pool, "admin", "pw", true) + ctx := context.Background() + + profile, err := dbq.New(h.pool).CreateReleaseTypeProfile(ctx, dbq.CreateReleaseTypeProfileParams{ + Name: "Albums only", PrimaryTypes: []string{"Album"}, SecondaryTypes: []string{}, + }) + if err != nil { + t.Fatal(err) + } + rv := seedPendingArtistRequest(t, h, alice, "artist-profiled", "Profiled") + path := "/api/admin/requests/" + uuidToString(rv.ID) + "/approve" + + if resp := doAdminRequestReq(t, h, http.MethodPost, path, []byte(`{"profile_id":"00000000-0000-4000-8000-000000000000"}`), admin); resp.Code != http.StatusBadRequest { + t.Fatalf("unknown profile: %d %s", resp.Code, resp.Body.String()) + } + body := []byte(`{"profile_id":"` + uuidToString(profile.ID) + `"}`) + if resp := doAdminRequestReq(t, h, http.MethodPost, path, body, admin); resp.Code != http.StatusOK { + t.Fatalf("approve: %d %s", resp.Code, resp.Body.String()) + } + f, err := dbq.New(h.pool).GetFollow(ctx, "artist-profiled") + if err != nil || f.ProfileID != profile.ID || !f.WantExisting { + t.Errorf("follow = %+v, %v: on Albums only, wanting what exists", f, err) + } +} diff --git a/internal/api/admin_scan.go b/internal/api/admin_scan.go index c2b5854e..93e3bf63 100644 --- a/internal/api/admin_scan.go +++ b/internal/api/admin_scan.go @@ -76,8 +76,8 @@ type scanTriggerResp struct { // handleTriggerScan implements POST /api/admin/scan/run. // Returns 202 on success, 409 if a scan is already in flight (and not stale), // 500 on DB errors. The lazy reaper inside library.TryStartScan handles -// stale (>1h) in-flight rows automatically — the operator no longer needs -// to manually clear stuck rows after a server crash. +// stale (>1h) in-flight rows automatically, so a row left stuck by a +// server crash needs no manual clearing. func (h *handlers) handleTriggerScan(w http.ResponseWriter, _ *http.Request) { bgCtx := context.Background() started, existing, err := library.TryStartScan( diff --git a/internal/api/admin_search.go b/internal/api/admin_search.go new file mode 100644 index 00000000..aaebdb50 --- /dev/null +++ b/internal/api/admin_search.go @@ -0,0 +1,340 @@ +package api + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "slices" + "time" + + "github.com/go-chi/chi/v5" + "github.com/jackc/pgx/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" +) + +// Searching for wants and grabbing releases: the schedule's +// settings, each want's candidates, and a person's Search, Grab and "never +// this one". + +// searchTimeout bounds a search a person waits on: a Prowlarr fanning out to +// many trackers can take most of a minute. +const searchTimeout = 2 * time.Minute + +type searchSettingsView struct { + AutoGrab bool `json:"auto_grab"` + SearchIntervalHours int32 `json:"search_interval_hours"` + AllowedQualities []string `json:"allowed_qualities"` + Tiers []release.TierInfo `json:"tiers"` + LastPassAt *string `json:"last_pass_at"` + LastPass json.RawMessage `json:"last_pass"` + // ImportRoot is the library folder new music is filed into; "" + // is the first of ScanPaths, the library's folders. + ImportRoot string `json:"import_root"` + ScanPaths []string `json:"scan_paths"` + // The download lifecycle: seed an imported download to this + // ratio or for this many days; give up on one nobody shares after this + // many hours. 0 is: remove once imported, no time limit, never. + SeedRatio float32 `json:"seed_ratio"` + SeedDays int32 `json:"seed_days"` + StallHours int32 `json:"stall_hours"` + // UpgradeUntil is the tier a held album is upgraded to; "" is + // no quality upgrades. + UpgradeUntil string `json:"upgrade_until"` +} + +func searchSettings(s dbq.AcquisitionSetting, roots []string) searchSettingsView { + v := searchSettingsView{ + AutoGrab: s.AutoGrab, SearchIntervalHours: s.SearchIntervalHours, + AllowedQualities: s.AllowedQualities, Tiers: release.Tiers, LastPass: s.LastPass, + ImportRoot: s.ImportRoot, ScanPaths: roots, + SeedRatio: s.SeedRatio, SeedDays: s.SeedDays, StallHours: s.StallHours, + UpgradeUntil: s.UpgradeUntil, + } + if v.ScanPaths == nil { + v.ScanPaths = []string{} + } + if v.AllowedQualities == nil { + v.AllowedQualities = []string{} + } + if len(v.LastPass) == 0 { + v.LastPass = json.RawMessage("{}") + } + if s.LastPassAt.Valid { + at := formatTimestamp(s.LastPassAt) + v.LastPassAt = &at + } + return v +} + +// handleGetSearchSettings implements GET /api/admin/acquisition/settings. +func (h *handlers) handleGetSearchSettings(w http.ResponseWriter, r *http.Request) { + s, err := dbq.New(h.pool).GetAcquisitionSettings(r.Context()) + if err != nil { + writeErrWithLog(w, h.logger, "admin search settings: get", apierror.Internal(err)) + return + } + writeJSON(w, http.StatusOK, searchSettings(s, h.importer.Roots())) +} + +type putSearchSettingsBody struct { + AutoGrab bool `json:"auto_grab"` + SearchIntervalHours int32 `json:"search_interval_hours"` + AllowedQualities []string `json:"allowed_qualities"` + ImportRoot string `json:"import_root"` + SeedRatio float32 `json:"seed_ratio"` + SeedDays int32 `json:"seed_days"` + StallHours int32 `json:"stall_hours"` + UpgradeUntil string `json:"upgrade_until"` +} + +// handlePutSearchSettings implements PUT /api/admin/acquisition/settings. +func (h *handlers) handlePutSearchSettings(w http.ResponseWriter, r *http.Request) { + var body putSearchSettingsBody + if !decodeBody(w, r, &body) { + return + } + if body.SearchIntervalHours < 0 || body.SearchIntervalHours > 720 { + writeErr(w, apierror.BadRequest("bad_interval", "search_interval_hours must be between 0 and 720")) + return + } + if len(body.AllowedQualities) == 0 { + writeErr(w, apierror.BadRequest("bad_quality", "allow at least one quality")) + return + } + for _, t := range body.AllowedQualities { + if !release.KnownTier(t) { + writeErr(w, apierror.BadRequest("bad_quality", "unknown quality "+t)) + return + } + } + if body.UpgradeUntil != "" && !release.KnownTier(body.UpgradeUntil) { + writeErr(w, apierror.BadRequest("bad_upgrade_until", "unknown quality "+body.UpgradeUntil)) + return + } + if body.ImportRoot != "" && !slices.Contains(h.importer.Roots(), body.ImportRoot) { + writeErr(w, apierror.BadRequest("bad_import_root", "file new music into one of the library's folders")) + return + } + if body.SeedRatio < 0 || body.SeedRatio > 100 || body.SeedDays < 0 || body.SeedDays > 365 || + body.StallHours < 0 || body.StallHours > 720 { + writeErr(w, apierror.BadRequest("bad_lifecycle", + "seed ratio must be 0 to 100, seed days 0 to 365 and stall hours 0 to 720")) + return + } + s, err := dbq.New(h.pool).UpdateAcquisitionSettings(r.Context(), dbq.UpdateAcquisitionSettingsParams{ + AutoGrab: body.AutoGrab, SearchIntervalHours: body.SearchIntervalHours, AllowedQualities: body.AllowedQualities, + ImportRoot: body.ImportRoot, SeedRatio: body.SeedRatio, SeedDays: body.SeedDays, StallHours: body.StallHours, + UpgradeUntil: body.UpgradeUntil, + }) + if err != nil { + writeErrWithLog(w, h.logger, "admin search settings: put", apierror.Internal(err)) + return + } + writeJSON(w, http.StatusOK, searchSettings(s, h.importer.Roots())) +} + +// candidateView is one release a want's last search found. +type candidateView struct { + Key string `json:"key"` + Name string `json:"name"` + InfoHash string `json:"info_hash"` + Indexer string `json:"indexer"` + SizeBytes int64 `json:"size_bytes"` + Seeders int32 `json:"seeders"` + FilesCount int32 `json:"files_count"` + Format string `json:"format"` + Quality string `json:"quality"` + Year int32 `json:"year"` + Verdict string `json:"verdict"` + Reason string `json:"reason"` + Reasons json.RawMessage `json:"reasons"` + FoldedFrom []string `json:"folded_from"` + SearchedAt string `json:"searched_at"` + Downloading bool `json:"downloading"` +} + +// handleListCandidates implements GET /api/admin/wanted/{id}/candidates. +func (h *handlers) handleListCandidates(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + q := dbq.New(h.pool) + rows, err := q.ListWantCandidates(r.Context(), id) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: candidates", apierror.Internal(err)) + return + } + live, err := q.GetLiveGrab(r.Context(), id) + if err != nil && !errors.Is(err, pgx.ErrNoRows) { + writeErrWithLog(w, h.logger, "admin wanted: live grab", apierror.Internal(err)) + return + } + out := make([]candidateView, 0, len(rows)) + for _, c := range rows { + v := candidateView{ + Key: c.Key, Name: c.Name, InfoHash: c.InfoHash, Indexer: c.Indexer, SizeBytes: c.SizeBytes, + Seeders: c.Seeders, FilesCount: c.FilesCount, Format: c.Format, Quality: c.Quality, Year: c.Year, + Verdict: c.Verdict, Reason: c.Reason, Reasons: c.Reasons, FoldedFrom: c.FoldedFrom, + SearchedAt: formatTimestamp(c.SearchedAt), + Downloading: live.ID.Valid && live.Key == c.Key, + } + if v.FoldedFrom == nil { + v.FoldedFrom = []string{} + } + if len(v.Reasons) == 0 { + v.Reasons = json.RawMessage("[]") + } + out = append(out, v) + } + writeJSON(w, http.StatusOK, map[string]any{"items": out}) +} + +// handleSearchWant implements POST /api/admin/wanted/{id}/search: search +// now, as the schedule would, taking the best release when auto-grab is on. +func (h *handlers) handleSearchWant(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + ctx, cancel := context.WithTimeout(r.Context(), searchTimeout) + defer cancel() + res, err := h.searcher.SearchNow(ctx, id) + if h.writeSearchErr(w, err) { + return + } + writeJSON(w, http.StatusOK, res) +} + +type grabBody struct { + Key string `json:"key"` + Force bool `json:"force"` +} + +// grabView is what came of a person's grab. +type grabView struct { + Added bool `json:"added"` + Refused bool `json:"refused"` + Gone bool `json:"gone"` + Unconfigured bool `json:"unconfigured"` + Missing string `json:"missing,omitempty"` + Kept bool `json:"kept"` + Replaced bool `json:"replaced"` + Verdict string `json:"verdict,omitempty"` + Reason string `json:"reason,omitempty"` + Error string `json:"error,omitempty"` +} + +// handleGrabCandidate implements POST /api/admin/wanted/{id}/grab: take the +// release named by key. The search runs again first, so the indexer's link +// never leaves the server; force takes what the verdict would not. +func (h *handlers) handleGrabCandidate(w http.ResponseWriter, r *http.Request) { + admin, ok := requireUser(w, r) + if !ok { + return + } + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + var body grabBody + if !decodeBody(w, r, &body) { + return + } + ctx, cancel := context.WithTimeout(r.Context(), searchTimeout) + defer cancel() + took, err := h.searcher.Take(ctx, id, body.Key, body.Force, admin.ID) + if h.writeSearchErr(w, err) { + return + } + v := grabView{ + Added: took.Added, Refused: took.Refused, Gone: took.Gone, Unconfigured: took.Unconfigured, + Missing: took.Missing, Error: took.Error, + } + if took.Chosen != nil { + v.Verdict, v.Reason = string(took.Chosen.Verdict), took.Chosen.Reason + } + if took.Contest != nil { + v.Kept, v.Replaced = took.Contest.Kept, took.Contest.Replaced + if took.Contest.Kept { + v.Reason = took.Contest.Reason + } + } + writeJSON(w, http.StatusOK, v) +} + +type blockBody struct { + Key string `json:"key"` +} + +// handleBlockCandidate implements POST /api/admin/wanted/{id}/block: never +// take this release again, for any want. +func (h *handlers) handleBlockCandidate(w http.ResponseWriter, r *http.Request) { + admin, ok := requireUser(w, r) + if !ok { + return + } + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + var body blockBody + if !decodeBody(w, r, &body) { + return + } + c, err := dbq.New(h.pool).GetWantCandidate(r.Context(), dbq.GetWantCandidateParams{WantID: id, Key: body.Key}) + if errors.Is(err, pgx.ErrNoRows) { + writeErr(w, apierror.NotFound("candidate")) + return + } + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: block", apierror.Internal(err)) + return + } + if c.InfoHash == "" { + writeErr(w, apierror.BadRequest("no_infohash", "this release has no infohash to set aside by")) + return + } + if err := acquire.NewMemory(h.pool).Block(r.Context(), dbq.BlockReleaseParams{ + InfoHash: c.InfoHash, Name: c.Name, Reason: acquire.BlockOperator, WantID: id, BlockedBy: admin.ID, + }); err != nil { + writeErrWithLog(w, h.logger, "admin wanted: block", apierror.Internal(err)) + return + } + w.WriteHeader(http.StatusNoContent) +} + +// handleUnblockRelease implements DELETE /api/admin/release-blocks/{hash}. +func (h *handlers) handleUnblockRelease(w http.ResponseWriter, r *http.Request) { + removed, err := acquire.NewMemory(h.pool).Unblock(r.Context(), chi.URLParam(r, "hash")) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: unblock", apierror.Internal(err)) + return + } + if !removed { + writeErr(w, apierror.NotFound("block")) + return + } + w.WriteHeader(http.StatusNoContent) +} + +// writeSearchErr answers a Searcher error, and says whether there was one. +func (h *handlers) writeSearchErr(w http.ResponseWriter, err error) bool { + switch { + case err == nil: + return false + case errors.Is(err, pgx.ErrNoRows): + writeErr(w, apierror.NotFound("want")) + case errors.Is(err, wanted.ErrWantHeld): + writeErr(w, apierror.Conflict("want_held", "the library already holds this album, and it isn't wanted for a better copy")) + default: + writeErrWithLog(w, h.logger, "admin wanted: search", apierror.Internal(err)) + } + return true +} diff --git a/internal/api/admin_status.go b/internal/api/admin_status.go new file mode 100644 index 00000000..17c6f8fa --- /dev/null +++ b/internal/api/admin_status.go @@ -0,0 +1,229 @@ +package api + +import ( + "net/http" + "syscall" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/buildinfo" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/logging" +) + +// The admin Overview's status sections: after the verdict, who is +// listening, what is downloading, what the library holds and how much room +// it has, and how the server itself is doing. One read for the page, polled +// while it is open. + +type listeningView struct { + Username string `json:"username"` + TrackID string `json:"track_id"` + Title string `json:"title"` + ArtistName string `json:"artist_name"` + AlbumID string `json:"album_id"` + AlbumTitle string `json:"album_title"` + DurationMs int32 `json:"duration_ms"` + StartedAt string `json:"started_at"` + Client string `json:"client"` + DeviceClass string `json:"device_class"` +} + +type listeningStatusView struct { + Now []listeningView `json:"now"` + // The last day: how many plays, by how many people, the latest when. + PlaysDay int64 `json:"plays_day"` + ListenersDay int64 `json:"listeners_day"` + LastPlayAt *string `json:"last_play_at"` +} + +type downloadsStatusView struct { + Running int64 `json:"running"` + NeedsYou int64 `json:"needs_you"` + NeedsYouSince *string `json:"needs_you_since"` + ImportedDay int64 `json:"imported_day"` + GivenUpDay int64 `json:"given_up_day"` + // Live is the downloads not yet ended, the ones needing a person first. + Live []downloadView `json:"live"` + ClientsDown []clientDownView `json:"clients_down"` +} + +// diskView is the filesystem one library folder, or the data folder, sits +// on. +type diskView struct { + Path string `json:"path"` + // Data marks Minstrel's own data folder (covers, fingerprints), beside + // the library's. + Data bool `json:"data,omitempty"` + TotalBytes uint64 `json:"total_bytes"` + FreeBytes uint64 `json:"free_bytes"` + // Error says why the folder could not be read, which is itself the + // news: a drive that did not mount. + Error string `json:"error,omitempty"` +} + +type libraryStatusView struct { + Tracks int64 `json:"tracks"` + Albums int64 `json:"albums"` + Artists int64 `json:"artists"` + Bytes int64 `json:"bytes"` + Disks []diskView `json:"disks"` +} + +type serverErrorView struct { + At string `json:"at"` + Message string `json:"message"` + Attrs string `json:"attrs"` +} + +type serverStatusView struct { + Version string `json:"version"` + Channel string `json:"channel"` + StartedAt string `json:"started_at"` + // Migration is the database schema's version, and Dirty whether its + // last migration stopped part way. + Migration int64 `json:"migration"` + Dirty bool `json:"dirty"` + // Errors is the newest few errors logged since the start, newest first; + // ErrorCount is all of them. + Errors []serverErrorView `json:"errors"` + ErrorCount int64 `json:"error_count"` +} + +type adminStatusView struct { + Listening listeningStatusView `json:"listening"` + Downloads downloadsStatusView `json:"downloads"` + Library libraryStatusView `json:"library"` + Server serverStatusView `json:"server"` +} + +// overviewLiveDownloads is how many running downloads the Overview lists; +// Activity has the rest. +const overviewLiveDownloads = 5 + +// overviewErrors is how many recent errors the Overview lists. +const overviewErrors = 10 + +// handleGetAdminStatus implements GET /api/admin/status. +func (h *handlers) handleGetAdminStatus(w http.ResponseWriter, r *http.Request) { + q := dbq.New(h.pool) + ctx := r.Context() + var out adminStatusView + + now, err := q.ListListeningNow(ctx) + if err != nil { + h.logger.Error("admin status: listening", "err", err) + writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") + return + } + out.Listening.Now = make([]listeningView, 0, len(now)) + for _, p := range now { + out.Listening.Now = append(out.Listening.Now, listeningView{ + Username: p.Username, TrackID: uuidToString(p.TrackID), Title: p.TrackTitle, + ArtistName: p.ArtistName, AlbumID: uuidToString(p.AlbumID), AlbumTitle: p.AlbumTitle, + DurationMs: p.DurationMs, StartedAt: formatTimestamp(p.StartedAt), + Client: deref(p.ClientID), DeviceClass: deref(p.DeviceClass), + }) + } + day, err := q.PlaysLastDay(ctx) + if err != nil { + h.logger.Error("admin status: plays", "err", err) + writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") + return + } + out.Listening.PlaysDay, out.Listening.ListenersDay = day.Plays, day.Listeners + if day.LastAt.Valid { + at := formatTimestamp(day.LastAt) + out.Listening.LastPlayAt = &at + } + + if err := h.downloadsStatus(r, q, &out.Downloads); err != nil { + h.logger.Error("admin status: downloads", "err", err) + writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") + return + } + + lib, err := q.LibraryStatus(ctx) + if err != nil { + h.logger.Error("admin status: library", "err", err) + writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") + return + } + out.Library = libraryStatusView{Tracks: lib.Tracks, Albums: lib.Albums, Artists: lib.Artists, + Bytes: lib.Bytes, Disks: []diskView{}} + if h.importer != nil { + for _, root := range h.importer.Roots() { + out.Library.Disks = append(out.Library.Disks, diskOf(root)) + } + } + if h.dataDir != "" { + d := diskOf(h.dataDir) + d.Data = true + out.Library.Disks = append(out.Library.Disks, d) + } + + out.Server = serverStatus(logging.Errors) + // golang-migrate's own table, outside sqlc's schema. + if err := h.pool.QueryRow(ctx, `SELECT version, dirty FROM schema_migrations LIMIT 1`). + Scan(&out.Server.Migration, &out.Server.Dirty); err != nil { + h.logger.Warn("admin status: migration version", "err", err) + } + writeJSON(w, http.StatusOK, out) +} + +func (h *handlers) downloadsStatus(r *http.Request, q *dbq.Queries, out *downloadsStatusView) error { + st, err := q.DownloadsStatus(r.Context()) + if err != nil { + return err + } + *out = downloadsStatusView{Running: st.Running, NeedsYou: st.NeedsYou, + ImportedDay: st.ImportedDay, GivenUpDay: st.GivenUpDay, + Live: []downloadView{}, ClientsDown: []clientDownView{}} + if st.NeedsYou > 0 && st.NeedsYouSince.Valid { + since := formatTimestamp(st.NeedsYouSince) + out.NeedsYouSince = &since + } + // ListGrabs puts the ones needing a person first, then the running ones. + rows, err := q.ListGrabs(r.Context(), dbq.ListGrabsParams{MaxRows: overviewLiveDownloads}) + if err != nil { + return err + } + for _, g := range rows { + if !g.EndedAt.Valid { + out.Live = append(out.Live, downloadFrom(g)) + } + } + down, err := q.ListClientsDown(r.Context()) + if err != nil { + return err + } + for _, c := range down { + out.ClientsDown = append(out.ClientsDown, clientDownView{Client: c.Client, Detail: c.Detail, + Since: formatTimestamp(c.FirstSeen)}) + } + return nil +} + +// diskOf reads the size and free space of the filesystem path is on. +func diskOf(path string) diskView { + var st syscall.Statfs_t + if err := syscall.Statfs(path, &st); err != nil { + return diskView{Path: path, Error: err.Error()} + } + bsize := uint64(st.Bsize) + return diskView{Path: path, TotalBytes: st.Blocks * bsize, FreeBytes: st.Bavail * bsize} +} + +func serverStatus(ring *logging.ErrorRing) serverStatusView { + kept, total := ring.Snapshot() + v := serverStatusView{Version: buildinfo.Version, Channel: buildinfo.Channel, + StartedAt: buildinfo.Started.UTC().Format(time.RFC3339), ErrorCount: total, + Errors: make([]serverErrorView, 0, min(len(kept), overviewErrors))} + for i, e := range kept { + if i == overviewErrors { + break + } + v.Errors = append(v.Errors, serverErrorView{At: e.At.UTC().Format(time.RFC3339), + Message: e.Message, Attrs: e.Attrs}) + } + return v +} diff --git a/internal/api/admin_status_test.go b/internal/api/admin_status_test.go new file mode 100644 index 00000000..4e2bcc07 --- /dev/null +++ b/internal/api/admin_status_test.go @@ -0,0 +1,89 @@ +package api + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/buildinfo" + "git.fabledsword.com/bvandeusen/minstrel/internal/logging" +) + +// openPlay records a play by user of track that started ago and has not +// ended. +func openPlay(t *testing.T, h *handlers, user, track pgtype.UUID, ago string) { + t.Helper() + ctx := context.Background() + var session pgtype.UUID + if err := h.pool.QueryRow(ctx, + `INSERT INTO play_sessions (user_id, started_at, last_event_at) VALUES ($1, now(), now()) RETURNING id`, + user).Scan(&session); err != nil { + t.Fatalf("session: %v", err) + } + if _, err := h.pool.Exec(ctx, + `INSERT INTO play_events (user_id, track_id, session_id, started_at, client_id) + VALUES ($1, $2, $3, now() - $4::interval, 'web')`, user, track, session, ago); err != nil { + t.Fatalf("play: %v", err) + } +} + +// The Overview's status: a play still inside its track's length is +// listening now; one long past it, never closed by its client, is not. +func TestAdminStatus_ListeningAndServer(t *testing.T) { + h, pool := testHandlers(t) + truncateLibrary(t, pool) + admin := seedUser(t, pool, "admin", "pw", true) + + fresh, _ := seedTrackOnDisk(t, h, "statusfresh") // 30 s long + stale, _ := seedTrackOnDisk(t, h, "statusstale") + openPlay(t, h, admin.ID, fresh.ID, "10 seconds") + openPlay(t, h, admin.ID, stale.ID, "1 hour") + + req := withUser(httptest.NewRequest(http.MethodGet, "/api/admin/status", nil), admin) + w := httptest.NewRecorder() + h.handleGetAdminStatus(w, req) + if w.Code != http.StatusOK { + t.Fatalf("status %d: %s", w.Code, w.Body.String()) + } + var got adminStatusView + if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { + t.Fatalf("decode: %v", err) + } + + var sawFresh, sawStale bool + for _, p := range got.Listening.Now { + switch p.TrackID { + case uuidToString(fresh.ID): + sawFresh = true + if p.Username != admin.Username || p.Client != "web" || p.Title != "AdmQ Track statusfresh" { + t.Errorf("listening row = %+v", p) + } + case uuidToString(stale.ID): + sawStale = true + } + } + if !sawFresh || sawStale { + t.Errorf("listening now: fresh %v (want true), stale %v (want false)", sawFresh, sawStale) + } + if got.Listening.PlaysDay < 2 || got.Listening.LastPlayAt == nil { + t.Errorf("last day: plays %d, last %v", got.Listening.PlaysDay, got.Listening.LastPlayAt) + } + + if got.Library.Tracks != 2 || got.Library.Albums != 2 || got.Library.Bytes != 10 { + t.Errorf("library = %+v, want 2 tracks, 2 albums, 10 bytes", got.Library) + } + if got.Downloads.Live == nil || got.Downloads.ClientsDown == nil || got.Library.Disks == nil { + t.Error("lists must be [], never null") + } + + if got.Server.Version != buildinfo.Version || got.Server.StartedAt == "" || got.Server.Migration < 92 { + t.Errorf("server = %+v", got.Server) + } + if _, total := logging.Errors.Snapshot(); got.Server.ErrorCount != total { + t.Errorf("error count = %d, ring says %d", got.Server.ErrorCount, total) + } +} diff --git a/internal/api/admin_summary.go b/internal/api/admin_summary.go new file mode 100644 index 00000000..9a870497 --- /dev/null +++ b/internal/api/admin_summary.go @@ -0,0 +1,57 @@ +package api + +import ( + "net/http" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +// adminQueueView is one queue that waits on a person: how many, and since +// when the oldest of them has waited (null when the queue is empty). +type adminQueueView struct { + Count int64 `json:"count"` + Since *string `json:"since"` +} + +// adminSummaryView is what the admin rail and the Overview's verdict read: +// every queue an admin acts on, in one response, so each admin page pays one +// cheap read for its rail. +type adminSummaryView struct { + Requests adminQueueView `json:"requests"` + Reports adminQueueView `json:"reports"` + Duplicates adminQueueView `json:"duplicates"` + Suspect adminQueueView `json:"suspect"` + Missing adminQueueView `json:"missing"` +} + +func queueView(count int64, since pgtype.Timestamptz) adminQueueView { + v := adminQueueView{Count: count} + if count > 0 && since.Valid { + s := formatTimestamp(since) + v.Since = &s + } + return v +} + +func adminSummaryFrom(row dbq.AdminSummaryRow) adminSummaryView { + return adminSummaryView{ + Requests: queueView(row.Requests, row.RequestsSince), + Reports: queueView(row.Reports, row.ReportsSince), + Duplicates: queueView(row.Duplicates, row.DuplicatesSince), + Suspect: queueView(row.Suspect, row.SuspectSince), + Missing: queueView(row.Missing, row.MissingSince), + } +} + +// handleGetAdminSummary implements GET /api/admin/summary. +func (h *handlers) handleGetAdminSummary(w http.ResponseWriter, r *http.Request) { + row, err := dbq.New(h.pool).AdminSummary(r.Context()) + if err != nil { + h.logger.Error("admin: summary", "err", err) + writeAdminJSONErr(w, http.StatusInternalServerError, "server_error") + return + } + writeJSON(w, http.StatusOK, adminSummaryFrom(row)) +} diff --git a/internal/api/admin_summary_test.go b/internal/api/admin_summary_test.go new file mode 100644 index 00000000..f6970512 --- /dev/null +++ b/internal/api/admin_summary_test.go @@ -0,0 +1,91 @@ +package api + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + + "github.com/go-chi/chi/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/auth" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +func getAdminSummary(t *testing.T, h *handlers, user dbq.User) (int, adminSummaryView) { + t.Helper() + r := chi.NewRouter() + r.Route("/api/admin", func(admin chi.Router) { + admin.Use(auth.RequireAdmin()) + admin.Get("/summary", h.handleGetAdminSummary) + }) + w := httptest.NewRecorder() + r.ServeHTTP(w, withUser(httptest.NewRequest(http.MethodGet, "/api/admin/summary", nil), user)) + var out adminSummaryView + if w.Code == http.StatusOK { + if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil { + t.Fatal(err) + } + } + return w.Code, out +} + +// The summary counts each queue an admin acts on and says since when the +// oldest item has waited; an empty queue has no since. +func TestAdminSummary_CountsTheQueuesAndSinceWhen(t *testing.T) { + h, _ := testHandlers(t) + admin := seedUser(t, h.pool, "admin", "pw", true) + listener := seedUser(t, h.pool, "listener", "pw", false) + ctx := context.Background() + + code, empty := getAdminSummary(t, h, admin) + if code != http.StatusOK { + t.Fatalf("status = %d", code) + } + if empty.Requests.Count != 0 || empty.Requests.Since != nil || empty.Missing.Count != 0 { + t.Errorf("empty summary = %+v", empty) + } + + if _, err := dbq.New(h.pool).CreateRequest(ctx, dbq.CreateRequestParams{ + UserID: listener.ID, Kind: dbq.RequestKindArtist, ArtistMbid: "a0000000-0000-4000-8000-000000000001", + ArtistName: "Annadale", + }); err != nil { + t.Fatal(err) + } + _, got := getAdminSummary(t, h, admin) + if got.Requests.Count != 1 || got.Requests.Since == nil { + t.Errorf("requests = %+v, want 1 with a since", got.Requests) + } + + if code, _ := getAdminSummary(t, h, listener); code != http.StatusForbidden { + t.Errorf("a listener got %d, want 403", code) + } +} + +// Duplicates count what needs review: a cross-release group is kept and +// linked by the resolver and waits on nobody. +func TestAdminSummary_DuplicatesCountOnlyWhatNeedsReview(t *testing.T) { + h, _ := testHandlers(t) + truncateLibrary(t, h.pool) + admin := seedUser(t, h.pool, "dupsum-admin", "pw", true) + ctx := context.Background() + a := seedQuarantineTrack(t, h, "dupsum-a") + b := seedQuarantineTrack(t, h, "dupsum-b") + for _, g := range []struct{ key, class string }{{"dupsum-review", "review"}, {"dupsum-cross", "cross_release"}} { + var id [16]byte + if err := h.pool.QueryRow(ctx, + `INSERT INTO duplicate_groups (member_key, tier, class) VALUES ($1, 'acoustic', $2) RETURNING id`, + g.key, g.class).Scan(&id); err != nil { + t.Fatal(err) + } + if _, err := h.pool.Exec(ctx, `INSERT INTO duplicate_group_members (group_id, track_id) VALUES ($1, $2), ($1, $3)`, + id, a.ID, b.ID); err != nil { + t.Fatal(err) + } + } + _, got := getAdminSummary(t, h, admin) + if got.Duplicates.Count != 1 { + t.Errorf("duplicates = %d, want 1: the cross-release group needs nobody", got.Duplicates.Count) + } +} diff --git a/internal/api/admin_suspect_sources.go b/internal/api/admin_suspect_sources.go index cca97934..8e5a0e90 100644 --- a/internal/api/admin_suspect_sources.go +++ b/internal/api/admin_suspect_sources.go @@ -12,7 +12,7 @@ import ( ) // suspectTrackView is one flagged track. FilePath is the evidence: the -// markers are read from its basename, and the operator needs to see it to +// markers are read from its basename, and an admin needs to see it to // judge whether the flag is right. type suspectTrackView struct { TrackID string `json:"track_id"` @@ -27,10 +27,14 @@ type suspectTrackView struct { TrackNumber *int32 `json:"track_number"` Markers []string `json:"markers"` // Verdict is "suspect" while the track is held back from radio and the - // mixes, "fine" once the operator let it back, null before the resolver's - // first pass (M498 #5439). VerdictAt is when it was last set. + // mixes, "fine" once an admin let it back, null before the resolver's + // first pass. VerdictAt is when it was last set. Verdict *string `json:"verdict"` VerdictAt *string `json:"verdict_at"` + // Replacing: the album is on the wanted list for a clean copy; + // SearchedAt is when one was last searched for, null before the first. + Replacing bool `json:"replacing"` + SearchedAt *string `json:"searched_at"` } // suspectGroupView is a folder's worth of flagged tracks. As on the @@ -53,8 +57,8 @@ type adminSuspectSourcesResponse struct { // // A marker is a reason to look, not proof: a band can title a song "Music // Video". The duplicate resolver holds a flagged track back from radio and the -// mixes and merges it away where a clean copy exists (M498 #5439); this report -// is where the operator sees that and can say a track is fine. +// mixes and merges it away where a clean copy exists; this report +// is where an admin sees that and can say a track is fine. func (h *handlers) handleListSuspectSources(w http.ResponseWriter, r *http.Request) { limit, offset, err := parsePaging(r.URL.Query()) if err != nil { @@ -112,6 +116,11 @@ func groupSuspectByDirectory(rows []dbq.ListSuspectSourceTracksRow) []suspectGro at := row.SourceVerdictAt.Time.UTC().Format(time.RFC3339) t.VerdictAt = &at } + t.Replacing = row.WantUpgrade == "rip" + if t.Replacing && row.SearchedAt.Valid { + at := row.SearchedAt.Time.UTC().Format(time.RFC3339) + t.SearchedAt = &at + } if n := len(groups); n > 0 && groups[n-1].Directory == row.Directory { groups[n-1].Tracks = append(groups[n-1].Tracks, t) continue diff --git a/internal/api/admin_tag_sources.go b/internal/api/admin_tag_sources.go index cc9b534b..412a1a42 100644 --- a/internal/api/admin_tag_sources.go +++ b/internal/api/admin_tag_sources.go @@ -15,7 +15,7 @@ import ( // tagSourceResp is the wire shape for one tag-enrichment provider in the // GET /api/admin/tag-sources list. Mirrors the cover-sources admin surface // (admin_cover_sources.go) — a separate, independent settings card so a new -// tag source is added without touching art settings (#1490). +// tag source is added without touching art settings. type tagSourceResp struct { ID string `json:"id"` DisplayName string `json:"display_name"` @@ -114,7 +114,8 @@ type testTagSourceResp struct { Error string `json:"error,omitempty"` } -// handleTestTagSource implements POST /api/admin/tag-sources/{provider_id}/test. +// handleTestTagSource implements POST +// /api/admin/tag-sources/{provider_id}/test. func (h *handlers) handleTestTagSource(w http.ResponseWriter, r *http.Request) { id := chi.URLParam(r, "provider_id") diff --git a/internal/api/admin_tracks.go b/internal/api/admin_tracks.go index 7630fa27..e944a257 100644 --- a/internal/api/admin_tracks.go +++ b/internal/api/admin_tracks.go @@ -3,7 +3,6 @@ package api import ( "errors" "net/http" - "strconv" "github.com/go-chi/chi/v5" @@ -11,27 +10,22 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/tracks" ) -// removeTrackResponse is the wire shape from spec §5 (M7 #372). The three +// removeTrackResponse is DELETE /api/admin/tracks/{id}'s answer. The two // optional fields are omitted unless the corresponding cleanup actually // happened, so the response stays minimal in the common case. type removeTrackResponse struct { - DeletedTrackID string `json:"deleted_track_id"` - DeletedAlbumID *string `json:"deleted_album_id,omitempty"` - DeletedArtistID *string `json:"deleted_artist_id,omitempty"` - LidarrUnmonitorFailed *bool `json:"lidarr_unmonitor_failed,omitempty"` + DeletedTrackID string `json:"deleted_track_id"` + DeletedAlbumID *string `json:"deleted_album_id,omitempty"` + DeletedArtistID *string `json:"deleted_artist_id,omitempty"` } -// handleRemoveTrack implements DELETE /api/admin/tracks/{id}?unmonitor=true|false. +// handleRemoveTrack implements DELETE /api/admin/tracks/{id}. // -// Admin-only (gated by auth.RequireAdmin on the /admin route group). Deletes the -// file, then the DB row, and runs the album/artist cascade tidy-up — and deletes -// nothing at all when the file cannot be removed (#3918). When -// unmonitor=true and the track has an mbid, also calls Lidarr.UnmonitorTrack -// — failure there is non-fatal (the destructive part already completed) and -// surfaces as `lidarr_unmonitor_failed: true` in the success envelope. +// Admin-only (gated by auth.RequireAdmin on the /admin route group). Deletes +// the file, then the DB row, and runs the album/artist cascade tidy-up — and +// deletes nothing at all when the file cannot be removed. // -// Per spec §5, Lidarr-side errors during the unmonitor step do NOT map to -// wire error codes. The codes this handler emits are not_found, +// The codes this handler emits are not_found, // library_not_writable (409) and file_delete_failed when the file could not be // removed, server_error, plus the auth codes the middleware emits upstream. func (h *handlers) handleRemoveTrack(w http.ResponseWriter, r *http.Request) { @@ -46,13 +40,6 @@ func (h *handlers) handleRemoveTrack(w http.ResponseWriter, r *http.Request) { return } - unmonitor := false - if v := r.URL.Query().Get("unmonitor"); v != "" { - if parsed, err := strconv.ParseBool(v); err == nil { - unmonitor = parsed - } - } - // Defensive: RequireUser+RequireAdmin should have run upstream. If // we got here without a user in context the routing is broken. admin, ok := requireUser(w, r) @@ -60,9 +47,7 @@ func (h *handlers) handleRemoveTrack(w http.ResponseWriter, r *http.Request) { return } - deletedAlbum, deletedArtist, lidarrUnmonitorFailed, err := h.tracks.RemoveTrack( - r.Context(), trackID, admin.ID, unmonitor, - ) + deletedAlbum, deletedArtist, err := h.tracks.RemoveTrack(r.Context(), trackID, admin.ID) if err != nil { if errors.Is(err, tracks.ErrNotFound) { writeErr(w, &apierror.Error{Status: http.StatusNotFound, Code: "not_found", Message: "track not found"}) @@ -87,13 +72,5 @@ func (h *handlers) handleRemoveTrack(w http.ResponseWriter, r *http.Request) { s := uuidToString(*deletedArtist) resp.DeletedArtistID = &s } - // Only emit the flag when unmonitor was requested AND it failed. - // Avoids "false" cluttering the wire when the operator didn't ask - // for an unmonitor in the first place. - if unmonitor && lidarrUnmonitorFailed { - t := true - resp.LidarrUnmonitorFailed = &t - } - writeJSON(w, http.StatusOK, resp) } diff --git a/internal/api/admin_tracks_test.go b/internal/api/admin_tracks_test.go index 36637850..0dd9a4cb 100644 --- a/internal/api/admin_tracks_test.go +++ b/internal/api/admin_tracks_test.go @@ -4,7 +4,6 @@ import ( "bytes" "context" "encoding/json" - "errors" "net/http" "net/http/httptest" "os" @@ -15,41 +14,12 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/auth" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/tracks" ) -// fakeLidarrUnmonitorer captures UnmonitorTrack calls for assertion and lets -// tests dictate the return error. Used to drive both the Lidarr-success and -// Lidarr-failure branches without spinning up a stub HTTP server. -type fakeLidarrUnmonitorer struct { - calls []fakeUnmonitorCall - err error -} - -type fakeUnmonitorCall struct { - trackMbid string - albumMbid string -} - -func (f *fakeLidarrUnmonitorer) UnmonitorTrack(_ context.Context, trackMbid, albumMbid string) error { - f.calls = append(f.calls, fakeUnmonitorCall{trackMbid: trackMbid, albumMbid: albumMbid}) - return f.err -} - -// installTracksLidarrStub replaces h.tracks with a fresh tracks.Service whose -// LidarrUnmonitorer is the supplied fake. Returns the fake so tests can -// assert call counts and inject error returns. -func installTracksLidarrStub(t *testing.T, h *handlers) *fakeLidarrUnmonitorer { - t.Helper() - fake := &fakeLidarrUnmonitorer{} - h.tracks = tracks.NewService(h.pool, h.logger, fake, h.dataDir) - return fake -} - // newAdminTracksRouter builds a test chi router with the admin-tracks // endpoint. RequireAdmin middleware is applied; tests inject the user into // context manually (RequireUser is bypassed). Mirrors the pattern from -// admin_lidarr_test.go and admin_quarantine_test.go. +// admin_quarantine_test.go. func newAdminTracksRouter(h *handlers) chi.Router { r := chi.NewRouter() r.Route("/api/admin", func(admin chi.Router) { @@ -90,8 +60,8 @@ func doAdminTracksReqNoUser(t *testing.T, h *handlers, method, path, idParam str } // seedTrackForRemoveTest creates a fresh artist+album+track on disk and -// returns the track row. The mbid string is set on both track and album -// when non-empty so the unmonitor branch has the data it needs. +// returns the track row. The mbids are set on the track and album when +// non-empty. func seedTrackForRemoveTest(t *testing.T, h *handlers, unique, trackMbid, albumMbid string) (dbq.Track, string) { t.Helper() artist := seedArtist(t, h.pool, "RemoveTest Artist "+unique) @@ -137,10 +107,8 @@ func seedTrackForRemoveTest(t *testing.T, h *handlers, unique, trackMbid, albumM } // TestHandleRemoveTrack_NonAdmin403 verifies the RequireAdmin middleware -// rejects non-admin callers. Asserts the existing middleware shape -// ({"error":"not_authorized"}) — spec §5 says `unauthorized`, but the -// shared middleware emits not_authorized today, and this handler doesn't -// own that envelope. +// rejects non-admin callers, with the shared middleware's +// {"error":"not_authorized"} envelope, which this handler doesn't own. func TestHandleRemoveTrack_NonAdmin403(t *testing.T) { h, pool := testHandlers(t) truncateLibrary(t, pool) @@ -206,11 +174,9 @@ func TestHandleRemoveTrack_UnknownID404(t *testing.T) { } } -// TestHandleRemoveTrack_HappyPath_NoUnmonitor verifies the success envelope -// when the operator does NOT request unmonitor: 200 with deleted_track_id -// (and the optional cascade fields when they fired) but NO -// lidarr_unmonitor_failed key in the body. -func TestHandleRemoveTrack_HappyPath_NoUnmonitor(t *testing.T) { +// TestHandleRemoveTrack_HappyPath verifies the success envelope: 200 with +// deleted_track_id and the optional cascade fields when they fired. +func TestHandleRemoveTrack_HappyPath(t *testing.T) { h, pool := testHandlers(t) truncateLibrary(t, pool) admin := seedUser(t, pool, "admin", "pw", true) @@ -237,21 +203,6 @@ func TestHandleRemoveTrack_HappyPath_NoUnmonitor(t *testing.T) { if got.DeletedArtistID == nil { t.Error("deleted_artist_id missing; expected sole-track cascade") } - if got.LidarrUnmonitorFailed != nil { - t.Errorf("lidarr_unmonitor_failed should be omitted when unmonitor not requested; got %v", - *got.LidarrUnmonitorFailed) - } - - // Verify the raw JSON has no `lidarr_unmonitor_failed` key — omitempty - // on a *bool drops it, but we double-check the wire to catch - // regressions. - var raw map[string]any - if err := json.Unmarshal(w.Body.Bytes(), &raw); err != nil { - t.Fatalf("decode raw: %v", err) - } - if _, ok := raw["lidarr_unmonitor_failed"]; ok { - t.Error("wire body contained lidarr_unmonitor_failed when unmonitor was not requested") - } // File is gone, DB row is gone. if _, err := os.Stat(path); !os.IsNotExist(err) { @@ -261,50 +212,3 @@ func TestHandleRemoveTrack_HappyPath_NoUnmonitor(t *testing.T) { t.Error("track row still exists") } } - -// TestHandleRemoveTrack_UnmonitorFailureFlag verifies that when the -// operator sends ?unmonitor=true AND the Lidarr call fails, the response -// is still 200 (file + DB delete completed) but carries -// lidarr_unmonitor_failed: true. -func TestHandleRemoveTrack_UnmonitorFailureFlag(t *testing.T) { - h, pool := testHandlers(t) - truncateLibrary(t, pool) - admin := seedUser(t, pool, "admin", "pw", true) - - fake := installTracksLidarrStub(t, h) - fake.err = errors.New("lidarr exploded") - - tr, path := seedTrackForRemoveTest(t, h, "umon-fail", "track-mbid-x", "album-mbid-x") - w := doAdminTracksReq(t, h, http.MethodDelete, - "/api/admin/tracks/"+uuidToString(tr.ID)+"?unmonitor=true", admin) - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - - var got removeTrackResponse - if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { - t.Fatalf("decode: %v; body = %s", err, w.Body.String()) - } - if got.DeletedTrackID != uuidToString(tr.ID) { - t.Errorf("deleted_track_id = %q, want %q", got.DeletedTrackID, uuidToString(tr.ID)) - } - if got.LidarrUnmonitorFailed == nil || !*got.LidarrUnmonitorFailed { - t.Errorf("lidarr_unmonitor_failed = %v, want true", got.LidarrUnmonitorFailed) - } - - // Lidarr was actually called with the right mbids. - if len(fake.calls) != 1 { - t.Fatalf("UnmonitorTrack call count = %d, want 1", len(fake.calls)) - } - if fake.calls[0].trackMbid != "track-mbid-x" || fake.calls[0].albumMbid != "album-mbid-x" { - t.Errorf("unmonitor call = %+v, want {track-mbid-x album-mbid-x}", fake.calls[0]) - } - - // Destructive part still happened. - if _, err := os.Stat(path); !os.IsNotExist(err) { - t.Errorf("file still exists: %v", err) - } - if _, err := dbq.New(pool).GetTrackByID(context.Background(), tr.ID); err == nil { - t.Error("track row still exists") - } -} diff --git a/internal/api/admin_wanted.go b/internal/api/admin_wanted.go new file mode 100644 index 00000000..0cb28b80 --- /dev/null +++ b/internal/api/admin_wanted.go @@ -0,0 +1,821 @@ +package api + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "strconv" + "strings" + + "github.com/go-chi/chi/v5" + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" +) + +// The wanted list's admin surface: wants, follows and +// release-type profiles. + +// wantView is one row of GET /api/admin/wanted. +type wantView struct { + ID string `json:"id"` + ReleaseGroupMBID string `json:"release_group_mbid"` + Title string `json:"title"` + ArtistMBID string `json:"artist_mbid"` + ArtistName string `json:"artist_name"` + ArtistID string `json:"artist_id,omitempty"` + PrimaryType string `json:"primary_type"` + SecondaryTypes []string `json:"secondary_types"` + Year string `json:"year"` + Reason string `json:"reason"` + RequestedBy string `json:"requested_by,omitempty"` + AlbumID string `json:"album_id,omitempty"` + WantedAt string `json:"wanted_at"` + HeldAt *string `json:"held_at"` + ImageURL string `json:"image_url"` + // Upgrade: why a held album is wanted again, quality or rip, and + // HeldQuality what the library's copy is. + Upgrade string `json:"upgrade,omitempty"` + HeldQuality string `json:"held_quality,omitempty"` + // Search is the schedule's last look; Grab the download running. + Search *wantSearchView `json:"search"` + Grab *wantGrabView `json:"grab"` +} + +type wantSearchView struct { + SearchedAt string `json:"searched_at"` + Outcome string `json:"outcome"` + Considered int32 `json:"considered"` + BestRelease string `json:"best_release"` + BestVerdict string `json:"best_verdict"` + Error string `json:"error"` +} + +// wantGrabView is the want's latest download, running or ended, and what +// its import made of it. +type wantGrabView struct { + ID string `json:"id"` + Name string `json:"name"` + GrabbedAt string `json:"grabbed_at"` + Origin string `json:"origin"` + Quality string `json:"quality"` + Live bool `json:"live"` + EndReason string `json:"end_reason"` + Progress float32 `json:"progress"` + ImportState string `json:"import_state"` + ImportReason string `json:"import_reason"` + EndDetail string `json:"end_detail"` + ImportedAt *string `json:"imported_at"` + ImportedFiles int32 `json:"imported_files"` +} + +// wantCounts sizes each tab of the Wanted page. Artists: how many artists +// the list holds albums of, its main page. +type wantCounts struct { + Artists int64 `json:"artists"` + // Unmatched: library albums waiting to be matched by hand. + Unmatched int64 `json:"unmatched"` + Wanted int64 `json:"wanted"` + Missing int64 `json:"missing"` + Held int64 `json:"held"` + // Upgrades: held albums wanted for a better copy. + Upgrades int64 `json:"upgrades"` + Following int64 `json:"following"` + // Activity: downloads running; Attention: of those, the ones waiting on + // a person; Blocked: releases set aside. + Activity int64 `json:"activity"` + Attention int64 `json:"attention"` + Blocked int64 `json:"blocked"` +} + +type wantsPage struct { + Items []wantView `json:"items"` + Total int64 `json:"total"` + Limit int `json:"limit"` + Offset int `json:"offset"` + Counts wantCounts `json:"counts"` + // Library is the one-time listing of the library's albums. + Library libraryListingView `json:"library"` + // Artist heads one artist's page, when the list is narrowed to them. + Artist *wantArtistView `json:"artist,omitempty"` +} + +// wantArtistView is one artist on the Wanted view's main page: how +// many of their albums the list holds, of how many. +type wantArtistView struct { + ArtistMBID string `json:"artist_mbid"` + ArtistName string `json:"artist_name"` + SortName string `json:"sort_name"` + ArtistID string `json:"artist_id,omitempty"` + ImageURL string `json:"image_url"` + Albums int32 `json:"albums"` + Held int32 `json:"held"` + Wanted int32 `json:"wanted"` + Missing int32 `json:"missing"` + Downloading int32 `json:"downloading"` + Following bool `json:"following"` +} + +type wantArtistsPage struct { + Items []wantArtistView `json:"items"` + Total int64 `json:"total"` + Limit int `json:"limit"` + Offset int `json:"offset"` + Counts wantCounts `json:"counts"` + Library libraryListingView `json:"library"` +} + +// libraryListingView: whether the library's albums are all on the list +// yet, and how many could not be matched to MusicBrainz; those are +// listed in the Collection's Unmatched section. +type libraryListingView struct { + ListedAt *string `json:"listed_at"` + Unlisted int32 `json:"unlisted"` +} + +// unmatchedAlbumView is one album in the Collection's Unmatched section: why +// no MusicBrainz album matched it, or that a person left it off. +type unmatchedAlbumView struct { + AlbumID string `json:"album_id"` + Title string `json:"title"` + Artist string `json:"artist"` + Detail string `json:"detail"` + Tracks int32 `json:"tracks"` + LeftOff bool `json:"left_off"` + DecidedAt string `json:"decided_at"` +} + +// unmatchedShown bounds the Unmatched section: a library a listing could +// not match more of than this has a bigger problem than one page shows. +const unmatchedShown = 500 + +const wantsPageSize = 50 + +// wantArtistsPageSize fills a few rows of the artist grid a page. +const wantArtistsPageSize = 96 + +// wantsHeader is what every Collection page carries: each section's count, +// and how the one-time library listing went. +func (h *handlers) wantsHeader(r *http.Request, q *dbq.Queries) (wantCounts, libraryListingView, error) { + counts, err := q.CountWants(r.Context()) + if err != nil { + return wantCounts{}, libraryListingView{}, err + } + listing, err := q.GetLibraryListing(r.Context()) + if err != nil { + return wantCounts{}, libraryListingView{}, err + } + lib := libraryListingView{Unlisted: listing.LibraryUnlisted} + if listing.LibraryListedAt.Valid { + at := formatTimestamp(listing.LibraryListedAt) + lib.ListedAt = &at + } + return wantCounts{Artists: counts.Artists, Unmatched: counts.Unmatched, Wanted: counts.Wanted, Missing: counts.Missing, Held: counts.Held, + Upgrades: counts.Upgrades, Following: counts.Following, Activity: counts.Activity, Attention: counts.Attention, Blocked: counts.Blocked}, lib, nil +} + +func offsetParam(r *http.Request) int { + offset, _ := strconv.Atoi(r.URL.Query().Get("offset")) + return max(offset, 0) +} + +func wantArtistViewOf(row dbq.ListWantArtistsRow) wantArtistView { + v := wantArtistView{ + ArtistMBID: row.ArtistMbid, ArtistName: row.ArtistName, SortName: row.SortName, + ImageURL: artistArtURL(row.ArtistMbid, row.ArtistName), + Albums: row.Albums, Held: row.Held, Wanted: row.Wanted, Missing: row.Missing, + Downloading: row.Downloading, Following: row.Following, + } + if row.LibraryArtistID.Valid { + v.ArtistID = uuidToString(row.LibraryArtistID) + } + return v +} + +// handleListWantArtists implements GET /api/admin/wanted/artists?q=&offset=: +// the Wanted view's main page, every artist on the list A–Z. +func (h *handlers) handleListWantArtists(w http.ResponseWriter, r *http.Request) { + offset := offsetParam(r) + q := dbq.New(h.pool) + rows, err := q.ListWantArtists(r.Context(), dbq.ListWantArtistsParams{ + Q: strings.TrimSpace(r.URL.Query().Get("q")), Skip: int32(offset), MaxRows: wantArtistsPageSize}) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: artists", apierror.Internal(err)) + return + } + counts, lib, err := h.wantsHeader(r, q) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: counts", apierror.Internal(err)) + return + } + out := wantArtistsPage{Items: make([]wantArtistView, 0, len(rows)), Limit: wantArtistsPageSize, Offset: offset, + Counts: counts, Library: lib} + for _, row := range rows { + out.Total = row.Total + out.Items = append(out.Items, wantArtistViewOf(row)) + } + writeJSON(w, http.StatusOK, out) +} + +// handleListWants implements GET +// /api/admin/wanted?state=wanted|missing|upgrade|held|all&artist=&q=&offset=. +// wanted is everything open but missing files; upgrade the held albums wanted +// for a better copy. Each list runs artist by artist, A–Z, like the library; q +// narrows it to an artist or album, and artist to one artist's page, which it +// then heads. +func (h *handlers) handleListWants(w http.ResponseWriter, r *http.Request) { + state := r.URL.Query().Get("state") + if state == "" { + state = "wanted" + } + if state != "wanted" && state != "missing" && state != "upgrade" && state != "held" && state != "all" { + writeErr(w, apierror.BadRequest("bad_state", "state must be wanted, missing, upgrade, held or all")) + return + } + offset := offsetParam(r) + artist := strings.TrimSpace(r.URL.Query().Get("artist")) + q := dbq.New(h.pool) + rows, err := q.ListWants(r.Context(), dbq.ListWantsParams{State: state, ArtistMbid: artist, + Q: strings.TrimSpace(r.URL.Query().Get("q")), Skip: int32(offset), MaxRows: wantsPageSize}) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: list", apierror.Internal(err)) + return + } + counts, lib, err := h.wantsHeader(r, q) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: counts", apierror.Internal(err)) + return + } + out := wantsPage{Library: lib, Counts: counts, + Items: make([]wantView, 0, len(rows)), Limit: wantsPageSize, Offset: offset} + if artist != "" { + heads, err := q.ListWantArtists(r.Context(), dbq.ListWantArtistsParams{ArtistMbid: artist, MaxRows: 1}) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: artist", apierror.Internal(err)) + return + } + if len(heads) == 0 { + writeErr(w, apierror.NotFound("artist")) + return + } + v := wantArtistViewOf(heads[0]) + out.Artist = &v + } + for _, row := range rows { + out.Total = row.Total + out.Items = append(out.Items, wantViewOf(row)) + } + writeJSON(w, http.StatusOK, out) +} + +// wantViewOf is one ListWants row as the wanted list and the Missing page +// send it. +func wantViewOf(row dbq.ListWantsRow) wantView { + v := wantView{ + ID: uuidToString(row.ID), ReleaseGroupMBID: row.ReleaseGroupMbid, Title: row.Title, + ArtistMBID: row.ArtistMbid, ArtistName: row.ArtistName, + PrimaryType: row.PrimaryType, SecondaryTypes: row.SecondaryTypes, + Year: first4(row.FirstReleaseDate), Reason: row.Reason, + WantedAt: formatTimestamp(row.WantedAt), Upgrade: row.Upgrade, HeldQuality: row.HeldQuality, + ImageURL: groupArtURL(musicbrainz.ReleaseGroup{MBID: row.ReleaseGroupMbid, Title: row.Title, ArtistCredit: row.ArtistName}), + } + if row.LibraryArtistID.Valid { + v.ArtistID = uuidToString(row.LibraryArtistID) + } + if row.AlbumID.Valid { + v.AlbumID = uuidToString(row.AlbumID) + } + if row.RequestedBy != nil { + v.RequestedBy = *row.RequestedBy + } + if row.FulfilledAt.Valid { + held := formatTimestamp(row.FulfilledAt) + v.HeldAt = &held + } + if v.SecondaryTypes == nil { + v.SecondaryTypes = []string{} + } + if row.SearchedAt.Valid { + v.Search = &wantSearchView{ + SearchedAt: formatTimestamp(row.SearchedAt), Outcome: row.SearchOutcome, + Considered: row.SearchConsidered, BestRelease: row.BestRelease, + BestVerdict: row.BestVerdict, Error: row.SearchError, + } + } + if row.GrabName != nil { + v.Grab = &wantGrabView{ID: uuidToString(row.GrabID), Name: *row.GrabName, + GrabbedAt: formatTimestamp(row.GrabbedAt), Live: !row.GrabEndedAt.Valid, + Origin: deref(row.GrabOrigin), Quality: deref(row.GrabQuality), EndReason: deref(row.GrabEndReason), + ImportState: deref(row.GrabImportState), ImportReason: deref(row.GrabImportReason), + EndDetail: deref(row.GrabEndDetail)} + if row.GrabProgress != nil { + v.Grab.Progress = *row.GrabProgress + } + if row.GrabImportedFiles != nil { + v.Grab.ImportedFiles = *row.GrabImportedFiles + } + if row.GrabImportedAt.Valid { + at := formatTimestamp(row.GrabImportedAt) + v.Grab.ImportedAt = &at + } + } + return v +} + +// wantTracksView is an album's track list for the Wanted view: the +// library's own tracks for an album it holds or held, else MusicBrainz's +// standard edition of what is wanted. +type wantTracksView struct { + // library, musicbrainz, or none when MusicBrainz lists no edition. + Source string `json:"source"` + // Edition names the MusicBrainz release the list is from. + Edition string `json:"edition,omitempty"` + Tracks []wantTrackView `json:"tracks"` +} + +type wantTrackView struct { + Disc int32 `json:"disc"` + Number int32 `json:"number"` + Title string `json:"title"` + LengthMs int32 `json:"length_ms"` + // Missing: the library's file for it is gone. + Missing bool `json:"missing"` +} + +// handleWantTracks implements GET /api/admin/wanted/{id}/tracks. +func (h *handlers) handleWantTracks(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + q := dbq.New(h.pool) + want, err := q.GetWantAlbum(r.Context(), id) + if errors.Is(err, pgx.ErrNoRows) { + writeErr(w, apierror.NotFound("want")) + return + } + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: tracks", apierror.Internal(err)) + return + } + out := wantTracksView{Tracks: []wantTrackView{}} + if want.AlbumID.Valid { + rows, err := q.ListLibraryAlbumTracks(r.Context(), want.AlbumID) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: library tracks", apierror.Internal(err)) + return + } + out.Source = "library" + for _, t := range rows { + v := wantTrackView{Title: t.Title, LengthMs: t.DurationMs, Missing: t.Missing} + if t.DiscNumber != nil { + v.Disc = *t.DiscNumber + } + if t.TrackNumber != nil { + v.Number = *t.TrackNumber + } + out.Tracks = append(out.Tracks, v) + } + writeJSON(w, http.StatusOK, out) + return + } + eds, err := h.catalogue.Releases(r.Context(), want.ReleaseGroupMbid) + switch { + case errors.Is(err, catalogue.ErrNotFound): + out.Source = "none" + writeJSON(w, http.StatusOK, out) + return + case errors.Is(err, musicbrainz.ErrUnreachable), errors.Is(err, musicbrainz.ErrBadResponse): + writeErr(w, &apierror.Error{Status: 503, Code: "musicbrainz_unreachable", Message: "MusicBrainz can't be reached", Cause: err}) + return + case err != nil: + writeErrWithLog(w, h.logger, "admin wanted: editions", apierror.Internal(err)) + return + } + e, ok := standardEdition(eds.Editions) + if !ok { + out.Source = "none" + writeJSON(w, http.StatusOK, out) + return + } + out.Source = "musicbrainz" + out.Edition = editionLabel(e.Release) + for _, t := range e.Tracks { + out.Tracks = append(out.Tracks, wantTrackView{Disc: t.MediumPosition, Number: t.Position, Title: t.Title, LengthMs: t.LengthMs}) + } + writeJSON(w, http.StatusOK, out) +} + +// standardEdition is the edition a person would call the album: an official +// one with the fewest tracks, else any with the fewest. The catalogue lists +// official editions first by date, so a tie goes to the earliest. +func standardEdition(eds []catalogue.Edition) (catalogue.Edition, bool) { + best := -1 + for i, e := range eds { + if len(e.Tracks) == 0 { + continue + } + if best < 0 { + best = i + continue + } + b := eds[best] + bo, eo := b.Release.Status == "Official", e.Release.Status == "Official" + if (eo && !bo) || (eo == bo && len(e.Tracks) < len(b.Tracks)) { + best = i + } + } + if best < 0 { + return catalogue.Edition{}, false + } + return eds[best], true +} + +// editionLabel names a release for a person: "2023 · CD · GB". +func editionLabel(r dbq.MbRelease) string { + var parts []string + for _, p := range []string{first4(r.ReleaseDate), r.Formats, r.Country} { + if p != "" { + parts = append(parts, p) + } + } + return strings.Join(parts, " · ") +} + +func first4(date string) string { + if len(date) < 4 { + return "" + } + return date[:4] +} + +// handleRemoveWant implements DELETE /api/admin/wanted/{id}. +func (h *handlers) handleRemoveWant(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + removed, err := h.wanted.RemoveWant(r.Context(), id) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: remove", apierror.Internal(err)) + return + } + if !removed { + writeErr(w, apierror.NotFound("want")) + return + } + w.WriteHeader(http.StatusNoContent) +} + +// followView is a followed artist. +type followView struct { + ArtistMBID string `json:"artist_mbid"` + ArtistName string `json:"artist_name"` + ArtistID string `json:"artist_id,omitempty"` + ProfileID string `json:"profile_id"` + ProfileName string `json:"profile_name"` + FollowedAt string `json:"followed_at"` + CheckedAt *string `json:"checked_at"` + OpenWants int64 `json:"open_wants"` + ImageURL string `json:"image_url"` +} + +// handleListFollows implements GET /api/admin/follows. +func (h *handlers) handleListFollows(w http.ResponseWriter, r *http.Request) { + rows, err := dbq.New(h.pool).ListFollows(r.Context()) + if err != nil { + writeErrWithLog(w, h.logger, "admin follows: list", apierror.Internal(err)) + return + } + out := make([]followView, 0, len(rows)) + for _, f := range rows { + v := followView{ + ArtistMBID: f.ArtistMbid, ArtistName: f.ArtistName, + ProfileID: uuidToString(f.ProfileID), ProfileName: f.ProfileName, + FollowedAt: formatTimestamp(f.FollowedAt), OpenWants: f.OpenWants, + ImageURL: artistArtURL(f.ArtistMbid, f.ArtistName), + } + if f.ArtistID.Valid { + v.ArtistID = uuidToString(f.ArtistID) + } + if f.CheckedAt.Valid { + checked := formatTimestamp(f.CheckedAt) + v.CheckedAt = &checked + } + out = append(out, v) + } + writeJSON(w, http.StatusOK, out) +} + +// followStateView answers whether one artist is followed, for the control on +// an artist page. +type followStateView struct { + Following bool `json:"following"` + ProfileID string `json:"profile_id,omitempty"` + ProfileName string `json:"profile_name,omitempty"` +} + +// handleGetFollow implements GET /api/admin/follows/{mbid}. +func (h *handlers) handleGetFollow(w http.ResponseWriter, r *http.Request) { + f, err := dbq.New(h.pool).GetFollow(r.Context(), chi.URLParam(r, "mbid")) + if errors.Is(err, pgx.ErrNoRows) { + writeJSON(w, http.StatusOK, followStateView{}) + return + } + if err != nil { + writeErrWithLog(w, h.logger, "admin follows: get", apierror.Internal(err)) + return + } + writeJSON(w, http.StatusOK, followStateView{ + Following: true, ProfileID: uuidToString(f.ProfileID), ProfileName: f.ProfileName, + }) +} + +type putFollowBody struct { + ArtistName string `json:"artist_name"` + ProfileID string `json:"profile_id"` + WantExisting bool `json:"want_existing"` +} + +// handlePutFollow implements PUT /api/admin/follows/{mbid}: follow an artist, +// or change the profile it is followed on. want_existing also wants what the +// artist has already released that the profile passes and isn't held. +func (h *handlers) handlePutFollow(w http.ResponseWriter, r *http.Request) { + admin, ok := requireUser(w, r) + if !ok { + return + } + var body putFollowBody + if !decodeBody(w, r, &body) { + return + } + var profile pgtype.UUID + if body.ProfileID != "" { + if profile, ok = parseUUID(body.ProfileID); !ok { + writeErr(w, apierror.BadRequest("bad_profile", "profile_id is not an id")) + return + } + } + mbid := chi.URLParam(r, "mbid") + if _, err := h.wanted.Follow(r.Context(), wanted.FollowParams{ + ArtistMBID: mbid, ArtistName: body.ArtistName, ProfileID: profile, + By: admin.ID, WantExisting: body.WantExisting, + }); err != nil { + switch { + case errors.Is(err, wanted.ErrNotFollowable): + writeErr(w, apierror.BadRequest("not_followable", "Various Artists and MusicBrainz placeholders can't be followed")) + case errors.Is(err, wanted.ErrProfileNotFound): + writeErr(w, apierror.BadRequest("profile_not_found", "no such profile")) + default: + writeErrWithLog(w, h.logger, "admin follows: put", apierror.Internal(err)) + } + return + } + h.handleGetFollow(w, r) +} + +// handleDeleteFollow implements DELETE /api/admin/follows/{mbid}. What the +// follow already wanted stays on the list. +func (h *handlers) handleDeleteFollow(w http.ResponseWriter, r *http.Request) { + removed, err := h.wanted.Unfollow(r.Context(), chi.URLParam(r, "mbid")) + if err != nil { + writeErrWithLog(w, h.logger, "admin follows: delete", apierror.Internal(err)) + return + } + if !removed { + writeErr(w, apierror.NotFound("follow")) + return + } + w.WriteHeader(http.StatusNoContent) +} + +// releaseProfileView is a release-type profile. +type releaseProfileView struct { + ID string `json:"id"` + Name string `json:"name"` + PrimaryTypes []string `json:"primary_types"` + SecondaryTypes []string `json:"secondary_types"` + IsDefault bool `json:"is_default"` + FollowCount int64 `json:"follow_count"` +} + +// releaseProfilesResponse also carries MusicBrainz's types, so the editor +// offers exactly what a profile may hold. +type releaseProfilesResponse struct { + Profiles []releaseProfileView `json:"profiles"` + PrimaryTypes []string `json:"primary_types"` + SecondaryTypes []string `json:"secondary_types"` +} + +// handleListReleaseProfiles implements GET /api/admin/release-profiles. +func (h *handlers) handleListReleaseProfiles(w http.ResponseWriter, r *http.Request) { + rows, err := dbq.New(h.pool).ListReleaseTypeProfiles(r.Context()) + if err != nil { + writeErrWithLog(w, h.logger, "admin release profiles: list", apierror.Internal(err)) + return + } + out := releaseProfilesResponse{ + Profiles: make([]releaseProfileView, 0, len(rows)), + PrimaryTypes: wanted.PrimaryTypes, SecondaryTypes: wanted.SecondaryTypes, + } + for _, p := range rows { + out.Profiles = append(out.Profiles, releaseProfileView{ + ID: uuidToString(p.ID), Name: p.Name, PrimaryTypes: p.PrimaryTypes, + SecondaryTypes: p.SecondaryTypes, IsDefault: p.IsDefault, FollowCount: p.FollowCount, + }) + } + writeJSON(w, http.StatusOK, out) +} + +type releaseProfileBody struct { + Name string `json:"name"` + PrimaryTypes []string `json:"primary_types"` + SecondaryTypes []string `json:"secondary_types"` + IsDefault bool `json:"is_default"` +} + +func (b releaseProfileBody) input() wanted.ProfileInput { + return wanted.ProfileInput{ + Name: b.Name, PrimaryTypes: b.PrimaryTypes, SecondaryTypes: b.SecondaryTypes, IsDefault: b.IsDefault, + } +} + +// handleCreateReleaseProfile implements POST /api/admin/release-profiles. +func (h *handlers) handleCreateReleaseProfile(w http.ResponseWriter, r *http.Request) { + var body releaseProfileBody + if !decodeBody(w, r, &body) { + return + } + p, err := h.wanted.CreateProfile(r.Context(), body.input()) + if err != nil { + h.writeProfileErr(w, err) + return + } + writeJSON(w, http.StatusCreated, releaseProfileView{ + ID: uuidToString(p.ID), Name: p.Name, PrimaryTypes: p.PrimaryTypes, + SecondaryTypes: p.SecondaryTypes, IsDefault: p.IsDefault, + }) +} + +// handleUpdateReleaseProfile implements PUT /api/admin/release-profiles/{id}. +// A follow on the profile judges new releases by the new types from its next +// check; releases it has already judged are not judged again. +func (h *handlers) handleUpdateReleaseProfile(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + var body releaseProfileBody + if !decodeBody(w, r, &body) { + return + } + p, err := h.wanted.UpdateProfile(r.Context(), id, body.input()) + if err != nil { + h.writeProfileErr(w, err) + return + } + writeJSON(w, http.StatusOK, releaseProfileView{ + ID: uuidToString(p.ID), Name: p.Name, PrimaryTypes: p.PrimaryTypes, + SecondaryTypes: p.SecondaryTypes, IsDefault: p.IsDefault, + }) +} + +// handleDeleteReleaseProfile implements DELETE +// /api/admin/release-profiles/{id}. +func (h *handlers) handleDeleteReleaseProfile(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + if err := h.wanted.DeleteProfile(r.Context(), id); err != nil { + h.writeProfileErr(w, err) + return + } + w.WriteHeader(http.StatusNoContent) +} + +func (h *handlers) writeProfileErr(w http.ResponseWriter, err error) { + switch { + case errors.Is(err, wanted.ErrProfileInvalid): + writeErr(w, apierror.BadRequest("profile_invalid", err.Error())) + case errors.Is(err, wanted.ErrProfileNotFound): + writeErr(w, apierror.NotFound("profile")) + case errors.Is(err, wanted.ErrProfileNameUsed): + writeErr(w, apierror.Conflict("profile_name_taken", "a profile already has that name")) + case errors.Is(err, wanted.ErrProfileInUse): + writeErr(w, apierror.Conflict("profile_in_use", "artists are followed on this profile; move them first")) + case errors.Is(err, wanted.ErrProfileDefault): + writeErr(w, apierror.Conflict("profile_is_default", "the default profile can't be deleted")) + default: + writeErrWithLog(w, h.logger, "admin release profiles", apierror.Internal(err)) + } +} + +// deref is a nullable column's text, or "". +func deref(s *string) string { + if s == nil { + return "" + } + return *s +} + +// handleListUnmatched implements GET /api/admin/wanted/unmatched: the +// Collection's Unmatched section, the albums no MusicBrainz album +// matched and, after them, the ones a person left off. +func (h *handlers) handleListUnmatched(w http.ResponseWriter, r *http.Request) { + rows, err := dbq.New(h.pool).ListUnmatchedLibraryAlbums(r.Context(), unmatchedShown) + if err != nil { + writeErrWithLog(w, h.logger, "admin wanted: unmatched", apierror.Internal(err)) + return + } + out := struct { + Items []unmatchedAlbumView `json:"items"` + }{Items: make([]unmatchedAlbumView, 0, len(rows))} + for _, u := range rows { + out.Items = append(out.Items, unmatchedAlbumView{ + AlbumID: uuidToString(u.AlbumID), Title: u.Title, Artist: u.ArtistName, Detail: u.Detail, + Tracks: u.Tracks, LeftOff: u.Outcome == "dismissed", DecidedAt: formatTimestamp(u.DecidedAt), + }) + } + writeJSON(w, http.StatusOK, out) +} + +type manualMatchView struct { + ReleaseGroupMBID string `json:"release_group_mbid"` + Title string `json:"title"` + Artist string `json:"artist"` + Agree int `json:"agree"` + Tracks int `json:"tracks"` +} + +// handleMatchUnmatched implements POST /api/admin/wanted/unmatched/{id}/match +// {"ref": ""}: the album goes on the list as held, +// as that MusicBrainz album. +func (h *handlers) handleMatchUnmatched(w http.ResponseWriter, r *http.Request) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + var body struct { + Ref string `json:"ref"` + } + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + writeErr(w, apierror.BadRequest("bad_body", "expected {\"ref\": \"…\"}")) + return + } + m, err := h.wanted.MatchAlbumTo(r.Context(), id, body.Ref) + switch { + case errors.Is(err, wanted.ErrMatchRef): + writeErr(w, apierror.BadRequest("bad_ref", "That isn't a MusicBrainz album link or id.")) + return + case errors.Is(err, wanted.ErrMatchNoGroup): + writeErr(w, &apierror.Error{Status: http.StatusUnprocessableEntity, Code: "no_such_album", + Message: "MusicBrainz has no album at that link.", Cause: err}) + return + case errors.Is(err, wanted.ErrAlbumNotUnmatched): + writeErr(w, apierror.NotFound("unmatched album")) + return + case errors.Is(err, musicbrainz.ErrUnreachable), errors.Is(err, musicbrainz.ErrBadResponse): + writeErr(w, &apierror.Error{Status: http.StatusServiceUnavailable, Code: "musicbrainz_unreachable", + Message: "MusicBrainz can't be reached", Cause: err}) + return + case err != nil: + writeErrWithLog(w, h.logger, "admin wanted: match by hand", apierror.Internal(err)) + return + } + writeJSON(w, http.StatusOK, manualMatchView{ReleaseGroupMBID: m.Group, Title: m.Title, Artist: m.Artist, + Agree: m.Agree, Tracks: m.Tracks}) +} + +// handleLeaveUnmatchedOff implements POST +// /api/admin/wanted/unmatched/{id}/leave-off, and handlePutUnmatchedBack its +// undoing, POST …/{id}/put-back. +func (h *handlers) handleLeaveUnmatchedOff(w http.ResponseWriter, r *http.Request) { + h.setUnmatched(w, r, h.wanted.LeaveAlbumOff) +} + +func (h *handlers) handlePutUnmatchedBack(w http.ResponseWriter, r *http.Request) { + h.setUnmatched(w, r, h.wanted.PutAlbumBack) +} + +func (h *handlers) setUnmatched(w http.ResponseWriter, r *http.Request, set func(context.Context, pgtype.UUID) error) { + id, ok := requireURLUUID(w, r, "id") + if !ok { + return + } + err := set(r.Context(), id) + switch { + case errors.Is(err, wanted.ErrAlbumNotUnmatched): + writeErr(w, apierror.NotFound("unmatched album")) + case err != nil: + writeErrWithLog(w, h.logger, "admin wanted: unmatched", apierror.Internal(err)) + default: + w.WriteHeader(http.StatusNoContent) + } +} diff --git a/internal/api/admin_wanted_test.go b/internal/api/admin_wanted_test.go new file mode 100644 index 00000000..e80a2edb --- /dev/null +++ b/internal/api/admin_wanted_test.go @@ -0,0 +1,371 @@ +package api + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" + + "github.com/go-chi/chi/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/auth" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" +) + +func doWantedReq(t *testing.T, h *handlers, path string, user dbq.User) *httptest.ResponseRecorder { + t.Helper() + return doWanted(t, h, http.MethodGet, path, "", user) +} + +func doWanted(t *testing.T, h *handlers, method, path, body string, user dbq.User) *httptest.ResponseRecorder { + t.Helper() + r := chi.NewRouter() + r.Route("/api/admin", func(admin chi.Router) { + admin.Use(auth.RequireAdmin()) + admin.Get("/wanted", h.handleListWants) + admin.Get("/wanted/artists", h.handleListWantArtists) + admin.Get("/wanted/unmatched", h.handleListUnmatched) + admin.Post("/wanted/unmatched/{id}/match", h.handleMatchUnmatched) + admin.Post("/wanted/unmatched/{id}/leave-off", h.handleLeaveUnmatchedOff) + admin.Post("/wanted/unmatched/{id}/put-back", h.handlePutUnmatchedBack) + admin.Get("/wanted/{id}/tracks", h.handleWantTracks) + }) + w := httptest.NewRecorder() + r.ServeHTTP(w, withUser(httptest.NewRequest(method, path, strings.NewReader(body)), user)) + return w +} + +func seedWant(t *testing.T, h *handlers, group, artistMBID, artist, title, date string) dbq.Want { + t.Helper() + w, err := wanted.WantWith(context.Background(), dbq.New(h.pool), wanted.WantParams{ + GroupMBID: group, ArtistMBID: artistMBID, ArtistName: artist, Title: title, + PrimaryType: "Album", FirstReleaseDate: date, Reason: wanted.ReasonFollow, + }) + if err != nil { + t.Fatal(err) + } + return w +} + +func listWanted(t *testing.T, h *handlers, user dbq.User, q string) wantsPage { + t.Helper() + w := doWantedReq(t, h, "/api/admin/wanted?"+url.Values{"q": {q}}.Encode(), user) + if w.Code != http.StatusOK { + t.Fatalf("status = %d; body = %s", w.Code, w.Body.String()) + } + var page wantsPage + if err := json.Unmarshal(w.Body.Bytes(), &page); err != nil { + t.Fatal(err) + } + return page +} + +// The list runs like the library: artists A–Z by the library's sort +// name where the library holds the artist, each artist's albums oldest +// first; q narrows it to an artist or an album. +func TestListWants_ArtistsAToZThenTheirAlbumsByDate(t *testing.T) { + h, _ := testHandlers(t) + admin := seedUser(t, h.pool, "admin", "pw", true) + ctx := context.Background() + beatles := "a0000000-0000-4000-8000-000000000002" + if _, err := dbq.New(h.pool).UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "The Beatles", SortName: "Beatles, The", Mbid: &beatles}); err != nil { + t.Fatal(err) + } + seedWant(t, h, "g0000000-0000-4000-8000-000000000001", "a0000000-0000-4000-8000-000000000003", "Zeds", "Zed Album", "2001") + seedWant(t, h, "g0000000-0000-4000-8000-000000000002", beatles, "The Beatles", "Abbey Road", "1969-09-26") + seedWant(t, h, "g0000000-0000-4000-8000-000000000003", beatles, "The Beatles", "Please Please Me", "1963-03-22") + seedWant(t, h, "g0000000-0000-4000-8000-000000000004", "a0000000-0000-4000-8000-000000000001", "Camel", "Moonmadness", "1976") + + page := listWanted(t, h, admin, "") + var got []string + for _, it := range page.Items { + got = append(got, it.ArtistName+"/"+it.Title) + } + want := "[The Beatles/Please Please Me The Beatles/Abbey Road Camel/Moonmadness Zeds/Zed Album]" + if fmt.Sprint(got) != want { + t.Errorf("order = %v, want %s", got, want) + } + if page.Items[0].ArtistID == "" || page.Items[2].ArtistID != "" { + t.Errorf("artist ids = %q, %q: only an artist the library holds has one", page.Items[0].ArtistID, page.Items[2].ArtistID) + } + + page = listWanted(t, h, admin, "BEATLES") + if len(page.Items) != 2 || page.Total != 2 { + t.Errorf("q=BEATLES: %d items, total %d; want the two Beatles albums", len(page.Items), page.Total) + } + page = listWanted(t, h, admin, "moon") + if len(page.Items) != 1 || page.Items[0].Title != "Moonmadness" || page.Total != 1 { + t.Errorf("q=moon: %+v, total %d", page.Items, page.Total) + } + if page = listWanted(t, h, admin, "nothing like it"); len(page.Items) != 0 || page.Total != 0 { + t.Errorf("no match: %d items, total %d", len(page.Items), page.Total) + } +} + +// The Collection's catalogue: every artist on the list A–Z, each +// with how many of their albums are held; an artist's page lists every album +// of theirs, held or not, under that artist. +func TestListWantArtists_CatalogueAndAnArtistsPage(t *testing.T) { + h, _ := testHandlers(t) + admin := seedUser(t, h.pool, "admin", "pw", true) + ctx := context.Background() + q := dbq.New(h.pool) + beatles := "a0000000-0000-4000-8000-000000000002" + camel := "a0000000-0000-4000-8000-000000000001" + if _, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "The Beatles", SortName: "Beatles, The", Mbid: &beatles}); err != nil { + t.Fatal(err) + } + seedWant(t, h, "g0000000-0000-4000-8000-000000000002", beatles, "The Beatles", "Abbey Road", "1969") + held := seedWant(t, h, "g0000000-0000-4000-8000-000000000003", beatles, "The Beatles", "Please Please Me", "1963") + if _, err := h.pool.Exec(ctx, `UPDATE wants SET fulfilled_at = now() WHERE id = $1`, held.ID); err != nil { + t.Fatal(err) + } + seedWant(t, h, "g0000000-0000-4000-8000-000000000004", camel, "Camel", "Moonmadness", "1976") + if _, err := h.pool.Exec(ctx, `INSERT INTO follows (artist_mbid, artist_name, profile_id) + SELECT $1, 'Camel', id FROM release_type_profiles WHERE is_default`, camel); err != nil { + t.Fatal(err) + } + + w := doWantedReq(t, h, "/api/admin/wanted/artists", admin) + if w.Code != http.StatusOK { + t.Fatalf("status = %d; body = %s", w.Code, w.Body.String()) + } + var page wantArtistsPage + if err := json.Unmarshal(w.Body.Bytes(), &page); err != nil { + t.Fatal(err) + } + if len(page.Items) != 2 || page.Total != 2 || page.Counts.Artists != 2 { + t.Fatalf("catalogue = %+v, total %d, artists %d", page.Items, page.Total, page.Counts.Artists) + } + b, c := page.Items[0], page.Items[1] + if b.ArtistName != "The Beatles" || b.SortName != "Beatles, The" || b.Albums != 2 || b.Held != 1 || b.Wanted != 1 || + b.ArtistID == "" || b.Following || b.ImageURL == "" { + t.Errorf("Beatles = %+v", b) + } + if c.ArtistName != "Camel" || c.Albums != 1 || c.Held != 0 || !c.Following || c.ArtistID != "" { + t.Errorf("Camel = %+v", c) + } + + // q finds an artist by an album's title. + w = doWantedReq(t, h, "/api/admin/wanted/artists?q=moon", admin) + page = wantArtistsPage{} + if err := json.Unmarshal(w.Body.Bytes(), &page); err != nil { + t.Fatal(err) + } + if len(page.Items) != 1 || page.Items[0].ArtistName != "Camel" { + t.Errorf("q=moon: %+v", page.Items) + } + + // The Beatles' page: both albums, held or not, headed by the artist. + w = doWantedReq(t, h, "/api/admin/wanted?state=all&artist="+beatles, admin) + var albums wantsPage + if err := json.Unmarshal(w.Body.Bytes(), &albums); err != nil { + t.Fatal(err) + } + if len(albums.Items) != 2 || albums.Items[0].Title != "Please Please Me" || albums.Items[0].HeldAt == nil || + albums.Artist == nil || albums.Artist.Held != 1 { + t.Errorf("artist page = %+v, head %+v", albums.Items, albums.Artist) + } + if w := doWantedReq(t, h, "/api/admin/wanted?state=all&artist=nobody", admin); w.Code != http.StatusNotFound { + t.Errorf("an artist with nothing on the list: status %d, want 404", w.Code) + } +} + +// fakeEditions is a MusicBrainz holding one group with two editions, the +// shorter official one the standard, or 503 when down is set. +func fakeEditions(t *testing.T, group string, down bool) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if down { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + switch { + case r.URL.Path == "/release-group/"+group: + _, _ = fmt.Fprintf(w, `{"id": %q, "title": "Moonmadness", "primary-type": "Album", "first-release-date": "1976"}`, group) + case r.URL.Path == "/release" && r.URL.Query().Get("release-group") == group: + _, _ = fmt.Fprint(w, `{"release-count": 2, "releases": [ + {"id": "r-deluxe", "title": "Moonmadness", "status": "Official", "date": "2009", "country": "GB", + "media": [{"position": 1, "format": "CD", "tracks": [ + {"position": 1, "number": "1", "title": "Aristillus", "length": 120000, "recording": {"id": "rec-1"}}, + {"position": 2, "number": "2", "title": "Song Within a Song", "length": 433000, "recording": {"id": "rec-2"}}, + {"position": 3, "number": "3", "title": "Bonus", "length": 200000, "recording": {"id": "rec-3"}}]}]}, + {"id": "r-orig", "title": "Moonmadness", "status": "Official", "date": "1976-03", "country": "GB", + "media": [{"position": 1, "format": "12\" Vinyl", "tracks": [ + {"position": 1, "number": "A1", "title": "Aristillus", "length": 120000, "recording": {"id": "rec-1"}}, + {"position": 2, "number": "A2", "title": "Song Within a Song", "length": 433000, "recording": {"id": "rec-2"}}]}]}]}`) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + return srv +} + +func wantTracks(t *testing.T, h *handlers, user dbq.User, id string) (int, wantTracksView) { + t.Helper() + w := doWantedReq(t, h, "/api/admin/wanted/"+id+"/tracks", user) + var out wantTracksView + if w.Code == http.StatusOK { + if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil { + t.Fatal(err) + } + } + return w.Code, out +} + +// An album not yet held shows MusicBrainz's standard edition: the shortest +// official one. +func TestWantTracks_StandardEditionFromMusicBrainz(t *testing.T) { + const group = "g0000000-0000-4000-8000-000000000010" + for _, down := range []bool{false, true} { + h, pool := testHandlers(t) + admin := seedUser(t, h.pool, "admin", "pw", true) + h.catalogue = catalogue.New(pool, &musicbrainz.Client{BaseURL: fakeEditions(t, group, down).URL, + Pace: &musicbrainz.Pacer{}, Retries: -1}, h.logger) + w := seedWant(t, h, group, "a0000000-0000-4000-8000-000000000001", "Camel", "Moonmadness", "1976") + + code, out := wantTracks(t, h, admin, uuidToString(w.ID)) + if down { + if code != http.StatusServiceUnavailable { + t.Errorf("MusicBrainz down: status %d, want 503", code) + } + continue + } + if code != http.StatusOK || out.Source != "musicbrainz" || len(out.Tracks) != 2 { + t.Fatalf("status %d, %+v: want the two-track original", code, out) + } + if out.Edition != "1976 · 12\" Vinyl · GB" { + t.Errorf("edition = %q", out.Edition) + } + if tr := out.Tracks[1]; tr.Disc != 1 || tr.Number != 2 || tr.Title != "Song Within a Song" || tr.LengthMs != 433000 { + t.Errorf("track 2 = %+v", tr) + } + } +} + +// A held album shows the library's own tracks, a lost file marked. +func TestWantTracks_HeldAlbumShowsTheLibrarysTracks(t *testing.T) { + h, _ := testHandlers(t) + admin := seedUser(t, h.pool, "admin", "pw", true) + ctx := context.Background() + q := dbq.New(h.pool) + mbid := "a0000000-0000-4000-8000-000000000001" + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "Camel", SortName: "Camel", Mbid: &mbid}) + if err != nil { + t.Fatal(err) + } + album, err := q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: "Moonmadness", SortTitle: "Moonmadness", ArtistID: artist.ID}) + if err != nil { + t.Fatal(err) + } + for i, title := range []string{"Aristillus", "Song Within a Song"} { + num := int32(i + 1) + if _, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: title, AlbumID: album.ID, ArtistID: artist.ID, TrackNumber: &num, DurationMs: 120000, + FilePath: fmt.Sprintf("/music/camel/%02d.flac", i+1), FileSize: 1, FileFormat: "flac", + }); err != nil { + t.Fatal(err) + } + } + if _, err := h.pool.Exec(ctx, `UPDATE tracks SET missing_since = now() WHERE title = 'Aristillus'`); err != nil { + t.Fatal(err) + } + w := seedWant(t, h, "g0000000-0000-4000-8000-000000000011", mbid, "Camel", "Moonmadness", "1976") + if err := q.FulfillWant(ctx, dbq.FulfillWantParams{ID: w.ID, AlbumID: album.ID}); err != nil { + t.Fatal(err) + } + + code, out := wantTracks(t, h, admin, uuidToString(w.ID)) + if code != http.StatusOK || out.Source != "library" || len(out.Tracks) != 2 { + t.Fatalf("status %d, %+v", code, out) + } + if !out.Tracks[0].Missing || out.Tracks[1].Missing || out.Tracks[1].Number != 2 { + t.Errorf("tracks = %+v: the first is missing, the second on disk", out.Tracks) + } +} + +// The Unmatched section: the albums matching could not place, a +// local band's album left off and put back, and a link that names no +// MusicBrainz album refused before MusicBrainz is asked. +func TestUnmatched_ListLeaveOffPutBack(t *testing.T) { + h, _ := testHandlers(t) + admin := seedUser(t, h.pool, "admin", "pw", true) + ctx := context.Background() + q := dbq.New(h.pool) + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "Annadale", SortName: "Annadale"}) + if err != nil { + t.Fatal(err) + } + album, err := q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: "Dial Tone", SortTitle: "Dial Tone", ArtistID: artist.ID}) + if err != nil { + t.Fatal(err) + } + if _, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: "Intro", AlbumID: album.ID, ArtistID: artist.ID, DurationMs: 120000, + FilePath: "/music/annadale/01.flac", FileSize: 1, FileFormat: "flac", + }); err != nil { + t.Fatal(err) + } + if err := q.RecordAlbumMatch(ctx, dbq.RecordAlbumMatchParams{ + AlbumID: album.ID, Outcome: "unmatched", Detail: "MusicBrainz has no album by that name", Matcher: 1}); err != nil { + t.Fatal(err) + } + id := uuidToString(album.ID) + + list := func() []unmatchedAlbumView { + t.Helper() + w := doWantedReq(t, h, "/api/admin/wanted/unmatched", admin) + if w.Code != http.StatusOK { + t.Fatalf("status = %d; body = %s", w.Code, w.Body.String()) + } + var out struct { + Items []unmatchedAlbumView `json:"items"` + } + if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil { + t.Fatal(err) + } + return out.Items + } + if got := list(); len(got) != 1 || got[0].AlbumID != id || got[0].Artist != "Annadale" || got[0].Tracks != 1 || + got[0].LeftOff || got[0].Detail == "" || got[0].DecidedAt == "" { + t.Fatalf("unmatched = %+v", got) + } + + for _, c := range []struct { + path, body string + code int + }{ + {"/api/admin/wanted/unmatched/" + id + "/match", `{"ref":"Dial Tone"}`, http.StatusBadRequest}, + {"/api/admin/wanted/unmatched/" + id + "/match", `nope`, http.StatusBadRequest}, + {"/api/admin/wanted/unmatched/not-a-uuid/leave-off", "", http.StatusBadRequest}, + {"/api/admin/wanted/unmatched/" + id + "/put-back", "", http.StatusNotFound}, + {"/api/admin/wanted/unmatched/" + id + "/leave-off", "", http.StatusNoContent}, + {"/api/admin/wanted/unmatched/" + id + "/leave-off", "", http.StatusNotFound}, + } { + if w := doWanted(t, h, http.MethodPost, c.path, c.body, admin); w.Code != c.code { + t.Errorf("POST %s %s = %d, want %d; body = %s", c.path, c.body, w.Code, c.code, w.Body.String()) + } + } + if got := list(); len(got) != 1 || !got[0].LeftOff { + t.Fatalf("left off = %+v", got) + } + if w := doWanted(t, h, http.MethodPost, "/api/admin/wanted/unmatched/"+id+"/put-back", "", admin); w.Code != http.StatusNoContent { + t.Fatalf("put back = %d; body = %s", w.Code, w.Body.String()) + } + if got := list(); len(got) != 1 || got[0].LeftOff { + t.Errorf("put back = %+v", got) + } + + user := seedUser(t, h.pool, "listener", "pw", false) + if w := doWantedReq(t, h, "/api/admin/wanted/unmatched", user); w.Code != http.StatusForbidden { + t.Errorf("a listener reads the unmatched section: %d", w.Code) + } +} diff --git a/internal/api/api.go b/internal/api/api.go index 1162ec25..bb222fe6 100644 --- a/internal/api/api.go +++ b/internal/api/api.go @@ -1,5 +1,5 @@ // Package api implements Minstrel's native JSON surface under /api. It is -// consumed by the built-in web SPA and (eventually) the Flutter client. +// consumed by the built-in web SPA and the Android app. // Subsonic-compatible endpoints under /rest are intentionally separate — // see internal/subsonic — and the two packages must not depend on each other. package api @@ -13,30 +13,34 @@ import ( "github.com/go-chi/chi/v5" "github.com/jackc/pgx/v5/pgxpool" + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" "git.fabledsword.com/bvandeusen/minstrel/internal/auth" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" "git.fabledsword.com/bvandeusen/minstrel/internal/config" "git.fabledsword.com/bvandeusen/minstrel/internal/coverart" "git.fabledsword.com/bvandeusen/minstrel/internal/eventbus" + "git.fabledsword.com/bvandeusen/minstrel/internal/importer" "git.fabledsword.com/bvandeusen/minstrel/internal/library" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrquarantine" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" + "git.fabledsword.com/bvandeusen/minstrel/internal/lifecycle" "git.fabledsword.com/bvandeusen/minstrel/internal/mailer" "git.fabledsword.com/bvandeusen/minstrel/internal/netsettings" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" "git.fabledsword.com/bvandeusen/minstrel/internal/playevents" "git.fabledsword.com/bvandeusen/minstrel/internal/playlists" + "git.fabledsword.com/bvandeusen/minstrel/internal/quarantine" "git.fabledsword.com/bvandeusen/minstrel/internal/reacquisition" "git.fabledsword.com/bvandeusen/minstrel/internal/recommendation" "git.fabledsword.com/bvandeusen/minstrel/internal/recsettings" + "git.fabledsword.com/bvandeusen/minstrel/internal/requests" "git.fabledsword.com/bvandeusen/minstrel/internal/tags" "git.fabledsword.com/bvandeusen/minstrel/internal/tracks" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" ) // Mount attaches /api/* handlers to r. Public endpoints (login) are outside // RequireUser; everything else is gated by the middleware. The events writer // is shared with the Subsonic mount so /rest/scrobble feeds the same store. -func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playevents.Writer, recCfg config.RecommendationConfig, recSettings *recsettings.Service, lidarrCfg *lidarrconfig.Service, lidarrReqs *lidarrrequests.Service, lidarrQuar *lidarrquarantine.Service, tracksSvc *tracks.Service, playlistsSvc *playlists.Service, coverEnricher *coverart.Enricher, coverSettings *coverart.SettingsService, tagSettings *tags.SettingsService, scanner *library.Scanner, scanCfg library.RunScanConfig, dataDir string, sender mailer.Sender, bus *eventbus.Bus, playlistScheduler *playlists.Scheduler, streamSecret []byte, netSettings *netsettings.Service, reacqSettings *reacquisition.SettingsService, fpSettings *library.FingerprintSettingsService, loudSettings *library.LoudnessSettingsService, acoustIDLookup *library.AcoustIDLookupWorker) { +func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playevents.Writer, recCfg config.RecommendationConfig, recSettings *recsettings.Service, requestSvc *requests.Service, quarantineSvc *quarantine.Service, tracksSvc *tracks.Service, playlistsSvc *playlists.Service, coverEnricher *coverart.Enricher, coverSettings *coverart.SettingsService, tagSettings *tags.SettingsService, scanner *library.Scanner, scanCfg library.RunScanConfig, dataDir string, sender mailer.Sender, bus *eventbus.Bus, playlistScheduler *playlists.Scheduler, streamSecret []byte, netSettings *netsettings.Service, reacqSettings *reacquisition.SettingsService, fpSettings *library.FingerprintSettingsService, loudSettings *library.LoudnessSettingsService, acoustIDLookup *library.AcoustIDLookupWorker) { rng := rand.New(rand.NewSource(rand.Int63())) setupToken, err := auth.NewSetupToken() if err != nil { @@ -45,13 +49,13 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev panic("api: mint setup token: " + err.Error()) } logSetupTokenIfNeeded(pool, logger, setupToken) + cat := catalogue.New(pool, nil, logger) h := &handlers{ pool: pool, logger: logger, events: events, recCfg: recCfg, recSettings: recSettings, rng: rng.Float64, - lidarrCfg: lidarrCfg, - lidarrRequests: lidarrReqs, - lidarrQuarantine: lidarrQuar, + requests: requestSvc, + quarantine: quarantineSvc, tracks: tracksSvc, playlists: playlistsSvc, coverart: coverEnricher, @@ -70,6 +74,14 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev fingerprintSettings: fpSettings, loudnessSettings: loudSettings, acoustIDLookup: acoustIDLookup, + acquisition: acquire.NewStore(pool), + catalogue: cat, + wanted: wanted.New(pool, cat, logger), + searcher: wanted.NewSearcher(pool, cat, logger), + importer: importer.New(pool, scanner, cat, nil, scannerRoots(scanner), logger), + lifecycle: lifecycle.New(pool, logger), + catalogueSearches: newSearchCache(), + catalogueArt: catalogue.NewArt(dataDir, artSources(coverSettings)), leveled: newLeveledRenderer(dataDir, loudSettings, logger), librarySize: recommendation.NewLibrarySize(nil), loginGuard: auth.NewLoginGuard(), @@ -93,15 +105,13 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev // query token (UPnP / Sonos path; see streamAuthOk). The // middleware attaches user to context when a valid cookie / // bearer is present but does NOT 401 on absence; the handler's - // own streamAuthOk performs the actual auth check. See the - // design at - // docs/superpowers/specs/2026-06-03-android-output-picker-upnp-design.md. + // own streamAuthOk performs the actual auth check. api.With(auth.OptionalUser(pool, logger)).Get("/tracks/{id}/stream", h.handleGetStream) // Extension-bearing alias so Sonos's URL probe can identify the // audio format from the path. The {ext} param is consumed by chi - // and ignored by the handler (which keys off {id}). See task #610. + // and ignored by the handler (which keys off {id}). api.With(auth.OptionalUser(pool, logger)).Get("/tracks/{id}/stream.{ext}", h.handleGetStream) - // The leveled stream for Sonos/UPnP (M464 #5001): session or a + // The leveled stream for Sonos/UPnP: session or a // leveled token, like the plain stream. api.With(auth.OptionalUser(pool, logger)).Get("/tracks/{id}/leveled.flac", h.handleGetLeveledStream) @@ -142,7 +152,7 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev authed.Get("/albums/{id}/cover", h.handleGetCover) authed.Get("/library/shuffle", h.handleLibraryShuffle) authed.Get("/library/albums", h.handleListLibraryAlbums) - // Browse indexes (#367). Genre filtering rides + // Browse indexes. Genre filtering rides // /library/albums?genre= rather than a path segment, because raw // ID3 genres contain slashes ("Rock/Pop") that a path can't carry. authed.Get("/library/genres", h.handleListGenres) @@ -157,8 +167,8 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev authed.Get("/search", h.handleSearch) authed.Get("/radio", h.handleRadio) authed.Get("/discover/suggestions", h.handleListSuggestions) - // Snooze = "not right now", time-boxed and self-expiring - // (#2374). Not a dislike — see the migration for why. + // Snooze = "not right now", time-boxed and self-expiring. + // Not a dislike — see the migration for why. authed.Post("/discover/suggestions/{mbid}/snooze", h.handleSnoozeSuggestion) authed.Delete("/discover/suggestions/{mbid}/snooze", h.handleUnsnoozeSuggestion) authed.Get("/discover/snoozes", h.handleListSuggestionSnoozes) @@ -166,10 +176,8 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev authed.Get("/home/index", h.handleGetHomeIndex) authed.Post("/events", h.handleEvents) authed.Get("/events/stream", h.handleEventsStream) - // UPnP / Sonos cast slice: issue a short-lived HMAC stream URL - // the speaker can fetch without the user's session. See the - // design at - // docs/superpowers/specs/2026-06-03-android-output-picker-upnp-design.md. + // UPnP / Sonos casting: issue a short-lived HMAC stream URL the + // speaker can fetch without the user's session. authed.Post("/cast/stream-token", h.handleCastStreamToken) authed.Post("/likes/tracks/{id}", h.handleLikeTrack) authed.Delete("/likes/tracks/{id}", h.handleUnlikeTrack) @@ -182,7 +190,11 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev authed.Get("/likes/artists", h.handleListLikedArtists) authed.Get("/likes/ids", h.handleGetLikedIDs) - authed.Get("/lidarr/search", h.handleLidarrSearch) + // The release catalogue: request search asks + // MusicBrainz, and art for what isn't held comes through + // Minstrel from the enabled cover-art sources. + authed.Get("/catalogue/search", h.handleCatalogueSearch) + authed.Get("/catalogue/art/{kind}/{mbid}", h.handleCatalogueArt) authed.Post("/requests", h.handleCreateRequest) authed.Get("/requests", h.handleListRequests) @@ -197,13 +209,13 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev // load failures). Admin-only inbox; any user can report. authed.Post("/playback-errors", h.handleReportPlaybackError) - // Device diagnostics ingest (M9). Any signed-in user can + // Device diagnostics ingest. Any signed-in user can // POST a batch, but events are only stored when the // account's debug_mode_enabled flag is on (handler no-ops // otherwise). Admin views live under /admin/diagnostics. authed.Post("/diagnostics", h.handleReportDiagnostics) - // Self-hosted in-app update channel (#397). Auth-gated to + // Self-hosted in-app update channel. Auth-gated to // prevent anonymous bandwidth abuse on the APK stream; // /apk additionally per-user rate-limited. authed.Get("/client/version", h.handleClientVersion) @@ -211,23 +223,63 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev authed.Route("/admin", func(admin chi.Router) { admin.Use(auth.RequireAdmin()) - admin.Get("/lidarr/config", h.handleGetLidarrConfig) - admin.Put("/lidarr/config", h.handlePutLidarrConfig) - admin.Post("/lidarr/test", h.handleTestLidarrConnection) - admin.Get("/lidarr/quality-profiles", h.handleListQualityProfiles) - admin.Get("/lidarr/metadata-profiles", h.handleListMetadataProfiles) - admin.Get("/lidarr/root-folders", h.handleListRootFolders) + admin.Get("/acquisition/services", h.handleListAcquisitionServices) + admin.Post("/acquisition/services", h.handleCreateAcquisitionService) + admin.Put("/acquisition/services/{code}", h.handleUpdateAcquisitionService) + admin.Delete("/acquisition/services/{code}", h.handleDeleteAcquisitionService) + admin.Post("/acquisition/services/{code}/test", h.handleTestAcquisitionService) + admin.Put("/acquisition/services/{code}/api-key", h.handleSetAcquisitionSecret("api_key")) + admin.Delete("/acquisition/services/{code}/api-key", h.handleSetAcquisitionSecret("api_key")) + admin.Put("/acquisition/services/{code}/password", h.handleSetAcquisitionSecret("password")) + admin.Delete("/acquisition/services/{code}/password", h.handleSetAcquisitionSecret("password")) + // The queues that wait on an admin, for the rail and the + // Overview. + admin.Get("/summary", h.handleGetAdminSummary) + admin.Get("/status", h.handleGetAdminStatus) admin.Get("/requests", h.handleListAdminRequests) admin.Post("/requests/{id}/approve", h.handleApproveRequest) admin.Post("/requests/{id}/reject", h.handleRejectRequest) + // The wanted list: what Minstrel is looking + // for, the artists it follows, and the release-type + // profiles that decide which of their releases count. + admin.Get("/wanted", h.handleListWants) + admin.Get("/wanted/artists", h.handleListWantArtists) + admin.Get("/wanted/unmatched", h.handleListUnmatched) + admin.Post("/wanted/unmatched/{id}/match", h.handleMatchUnmatched) + admin.Post("/wanted/unmatched/{id}/leave-off", h.handleLeaveUnmatchedOff) + admin.Post("/wanted/unmatched/{id}/put-back", h.handlePutUnmatchedBack) + admin.Delete("/wanted/{id}", h.handleRemoveWant) + // Searching and grabbing. + admin.Get("/acquisition/settings", h.handleGetSearchSettings) + admin.Put("/acquisition/settings", h.handlePutSearchSettings) + admin.Get("/wanted/{id}/candidates", h.handleListCandidates) + admin.Get("/wanted/{id}/tracks", h.handleWantTracks) + admin.Post("/wanted/{id}/search", h.handleSearchWant) + admin.Post("/wanted/{id}/grab", h.handleGrabCandidate) + admin.Post("/wanted/{id}/block", h.handleBlockCandidate) + admin.Delete("/release-blocks/{hash}", h.handleUnblockRelease) + admin.Get("/release-blocks", h.handleListReleaseBlocks) + admin.Get("/downloads", h.handleListDownloads) + admin.Post("/downloads/{id}/import", h.handleImportAnyway) + admin.Post("/downloads/{id}/retry", h.handleRetryImport) + admin.Post("/downloads/{id}/refuse", h.handleRefuseDownload) + admin.Post("/downloads/{id}/abandon", h.handleAbandonDownload) + admin.Get("/follows", h.handleListFollows) + admin.Get("/follows/{mbid}", h.handleGetFollow) + admin.Put("/follows/{mbid}", h.handlePutFollow) + admin.Delete("/follows/{mbid}", h.handleDeleteFollow) + admin.Get("/release-profiles", h.handleListReleaseProfiles) + admin.Post("/release-profiles", h.handleCreateReleaseProfile) + admin.Put("/release-profiles/{id}", h.handleUpdateReleaseProfile) + admin.Delete("/release-profiles/{id}", h.handleDeleteReleaseProfile) + admin.Get("/quarantine", h.handleListAdminQuarantine) admin.Post("/quarantine/{track_id}/resolve", h.handleResolveQuarantine) admin.Get("/playback-errors", h.handleListAdminPlaybackErrors) admin.Post("/playback-errors/{id}/resolve", h.handleResolvePlaybackError) admin.Post("/quarantine/{track_id}/delete-file", h.handleDeleteQuarantineFile) - admin.Post("/quarantine/{track_id}/delete-via-lidarr", h.handleDeleteQuarantineViaLidarr) admin.Get("/quarantine/actions", h.handleListQuarantineActions) admin.Delete("/tracks/{id}", h.handleRemoveTrack) @@ -238,19 +290,23 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev admin.Get("/network-settings", h.handleGetNetworkSettings) admin.Put("/network-settings", h.handleUpdateNetworkSettings) - // Policy for turning a missing file back into a Lidarr - // request (#290). Sits beside the missing-files list it - // governs rather than under /lidarr, because the operator - // meets it on the missing-files surface. + // Policy for wanting back an album whose files went missing + // Sits beside the missing-files list it governs, because an + // admin meets it on that surface. admin.Get("/library/reacquisition", h.handleGetReacquisitionSettings) admin.Put("/library/reacquisition", h.handleUpdateReacquisitionSettings) + // Where the library lives. A changed folder moves its + // tracks rather than re-adding them. + admin.Get("/library/folders", h.handleGetLibraryFolders) + admin.Put("/library/folders", h.handlePutLibraryFolders) + admin.Get("/scan/status", h.handleGetScanStatus) admin.Post("/scan/run", h.handleTriggerScan) // Sits under /library rather than /tracks because what it // reports is a property of the library's relationship to disk, // and because the destructive /tracks/{id} route above must - // not be mistaken for it (#2527). + // not be mistaken for it. admin.Get("/library/missing", h.handleListMissingTracks) admin.Get("/library/suspect-sources", h.handleListSuspectSources) admin.Put("/library/suspect-sources/{id}", h.handleSetSuspectSourceVerdict) @@ -266,14 +322,16 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev admin.Put("/library/acoustid-settings", h.handleUpdateAcoustIDSettings) admin.Post("/library/acoustid/run", h.handleRunAcoustIDLookup) admin.Get("/library/acoustid/unsettled", h.handleListUnsettledAcoustID) - // Duplicates report (#3912): proposals from the duplicate sweep, a - // trigger to sweep now, dismissal, and the merge (#3911), which deletes - // the removed copies' files after moving their history onto the kept one. + // Duplicates report: proposals from the duplicate sweep, a + // trigger to sweep now, dismissal, and the merge, which deletes + // the removed copies' files after moving their history onto the + // kept one. admin.Get("/library/duplicates", h.handleListDuplicates) admin.Get("/library/duplicates/resolved", h.handleListResolvedDuplicates) admin.Post("/library/duplicates/sweep", h.handleRunDuplicateSweep) admin.Post("/library/duplicates/{id}/dismiss", h.handleDismissDuplicateGroup) admin.Post("/library/duplicates/{id}/merge", h.handleMergeDuplicateGroup) + admin.Post("/library/duplicates/{id}/refetch", h.handleRefetchDuplicateGroup) admin.Get("/invites", h.handleListInvites) admin.Post("/invites", h.handleCreateInvite) @@ -286,7 +344,7 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev admin.Put("/users/{id}/auto-approve", h.handleAdminAutoApproveToggle) admin.Put("/users/{id}/debug-mode", h.handleAdminDebugModeToggle) - // Device diagnostics timeline + device overview (M9). + // Device diagnostics timeline + device overview. admin.Get("/diagnostics", h.handleListAdminDiagnostics) admin.Get("/diagnostics/devices", h.handleListAdminDiagnosticDevices) @@ -306,12 +364,12 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, events *playev admin.Get("/notification-email", h.handleGetNotificationEmail) admin.Put("/notification-email", h.handleUpdateNotificationEmail) - // Recommendation tuning lab (#1250): scoring-weight + // Recommendation tuning: scoring-weight // profiles + taste-build knobs, DB-backed, live effect. admin.Get("/recommendation-tuning", h.handleGetRecommendationTuning) admin.Patch("/recommendation-tuning/{scope}", h.handlePatchRecommendationTuning) admin.Post("/recommendation-tuning/{scope}/reset", h.handleResetRecommendationTuning) - // Weekly outcome trends + knob-turn markers (#1251). + // Weekly outcome trends + knob-turn markers. admin.Get("/recommendation-trends", h.handleGetRecommendationTrends) }) @@ -338,43 +396,62 @@ type handlers struct { recSettings *recsettings.Service rng func() float64 // librarySize memoises the track count that sizes the candidate pool - // (#3880). Held here rather than counted per request: the count is a + // Held here rather than counted per request: the count is a // full table scan, and library size only moves when a scan runs. - librarySize *recommendation.LibrarySize - lidarrCfg *lidarrconfig.Service - lidarrRequests *lidarrrequests.Service - lidarrQuarantine *lidarrquarantine.Service - tracks *tracks.Service - playlists *playlists.Service - coverart *coverart.Enricher - coverSettings *coverart.SettingsService - tagSettings *tags.SettingsService - scanner *library.Scanner - scanCfg library.RunScanConfig - dataDir string - mailer mailer.Sender - eventbus *eventbus.Bus - // notifier writes the notifications inbox (M489). Nil-safe: a nil + librarySize *recommendation.LibrarySize + requests *requests.Service + quarantine *quarantine.Service + tracks *tracks.Service + playlists *playlists.Service + coverart *coverart.Enricher + coverSettings *coverart.SettingsService + tagSettings *tags.SettingsService + scanner *library.Scanner + scanCfg library.RunScanConfig + dataDir string + mailer mailer.Sender + eventbus *eventbus.Bus + // notifier writes the notifications inbox. Nil-safe: a nil // notifier records nothing, which is what most handler tests want. notifier *notifications.Notifier playlistScheduler *playlists.Scheduler // reacqSettings is the DB-backed policy for auto re-acquisition of - // missing files (milestone #290) — grace window, backoff, attempt caps. + // missing files — grace window, backoff, attempt caps. // Cached in the service, so the admin card reads it without a query. reacqSettings *reacquisition.SettingsService - // fingerprintSettings is the fingerprinting policy (M400 #3913), the same + // fingerprintSettings is the fingerprinting policy, the same // instance the scanner and the fingerprint workers read, so a save from the // admin card reaches them without a restart. Nil serves the defaults. fingerprintSettings *library.FingerprintSettingsService - // loudnessSettings is the loudness analysis policy (M464 #4995), the same + // loudnessSettings is the loudness analysis policy, the same // instance the loudness backfill reads. Nil serves the defaults. loudnessSettings *library.LoudnessSettingsService - // acoustIDLookup is the AcoustID lookup worker (M401): its settings, its + // acoustIDLookup is the AcoustID lookup worker: its settings, its // status and "look up now". Nil serves the defaults (off). acoustIDLookup *library.AcoustIDLookupWorker - // leveled renders the leveled streams handed to Sonos/UPnP speakers - // (M464 #5001). Nil when its cache directory cannot be made: a level - // request then gets the plain stream. + // acquisition is the configured indexers and download clients. + acquisition *acquire.Store + // catalogue is the release catalogue read from MusicBrainz, and + // catalogueArt the images for what isn't held. + catalogue *catalogue.Service + catalogueArt *catalogue.Art + // wanted is the wanted list: follows, release-type profiles and wants. + // Its worker runs in main; the API reads and edits. + wanted *wanted.Service + // searcher searches for wants and grabs releases for them. + searcher *wanted.Searcher + // importer refuses a download at a person's word. The import + // loop itself runs in main; this one only acts on a request. + importer *importer.Importer + // lifecycle gives up on a download at a person's word; its loop + // runs in main. + lifecycle *lifecycle.Service + // catalogueSearches reuses recent MusicBrainz answers; nil asks every + // time, which is what handler tests get. + catalogueSearches *searchCache + // leveled renders the leveled streams handed to Sonos/UPnP speakers. Nil + // when its cache directory cannot be made: a level request then gets the + // plain stream. leveled *library.LeveledRenderer // setupToken must accompany the first registration while no users exist // (see auth.SetupToken). requireSetupToken is set by Mount, the only @@ -394,13 +471,10 @@ type handlers struct { // middleware on every request and edited from the admin network card. netSettings *netsettings.Service // streamSecret is the HMAC key used by SignStreamToken / - // VerifyStreamToken to authenticate the UPnP-speaker stream path - // (see internal/api/stream_token.go and the design at - // docs/superpowers/specs/2026-06-03-android-output-picker-upnp-design.md). - // nil in slice 1; slice 2 wires the env-var-with-app_preferences- - // fallback loader. A nil secret leaves the cookie path intact and - // makes the token path unreachable (HMAC of empty key won't match - // anything a client mints), which is the desired slice-1 default. + // VerifyStreamToken to authenticate the UPnP-speaker stream path (see + // internal/api/stream_token.go), from config.Config.StreamSecret. A nil + // secret leaves the cookie path intact and makes the token path + // unreachable (HMAC of empty key won't match anything a client mints). streamSecret []byte } @@ -414,3 +488,24 @@ func newLeveledRenderer(dataDir string, settings *library.LoudnessSettingsServic } return r } + +// artSources hands the art cache the cover-art settings, or nothing when +// there are none: a nil *SettingsService inside the interface would not +// compare equal to nil and would be called. +func artSources(s *coverart.SettingsService) catalogue.ArtSources { + if s == nil { + return nil + } + return s +} + +// scannerRoots reads the library folders, or none when the API runs without +// a scanner (tests). +func scannerRoots(s *library.Scanner) func() []string { + return func() []string { + if s == nil { + return nil + } + return s.Roots() + } +} diff --git a/internal/api/auth.go b/internal/api/auth.go index a36dea87..89bcc13f 100644 --- a/internal/api/auth.go +++ b/internal/api/auth.go @@ -140,7 +140,7 @@ func (h *handlers) handleLogin(w http.ResponseWriter, r *http.Request) { Value: token, Path: "/", HttpOnly: true, - Secure: auth.IsHTTPS(r, h.netSettings.Hops()), // plain HTTP and LAN stay functional (rule 94) + Secure: auth.IsHTTPS(r, h.netSettings.Hops()), // plain HTTP and LAN stay functional SameSite: http.SameSiteStrictMode, MaxAge: int(sessionCookieMaxAge.Seconds()), }) diff --git a/internal/api/auth_forgot.go b/internal/api/auth_forgot.go index 6d652437..57baf6b7 100644 --- a/internal/api/auth_forgot.go +++ b/internal/api/auth_forgot.go @@ -130,15 +130,15 @@ func (h *handlers) sendResetEmail(ctx context.Context, user dbq.User) error { return h.mailer.Send(ctx, *user.Email, mailer.ResetEmailSubject, textBody, htmlBody) } -// errNoPublicURL stops a reset email from going out before the operator has +// errNoPublicURL stops a reset email from going out before an admin has // said where Minstrel lives. It surfaces in the log, not the response, which // stays the same opaque 200 either way. -var errNoPublicURL = errors.New("forgot-password: no public URL set (Admin → Integrations → Public address); reset email not sent") +var errNoPublicURL = errors.New("forgot-password: no public URL set (Admin › Email & network › Public address); reset email not sent") -// buildResetURL builds the emailed link from the operator-set public URL. +// buildResetURL builds the emailed link from an admin-set public URL. // It deliberately ignores the request: the Host header is whatever the -// requester sent, and building from it let a forged Host plant a real reset -// token on a link to someone else's server. +// requester sent, and building from it would let a forged Host plant a real +// reset token on a link to someone else's server. func buildResetURL(publicURL, token string) (string, error) { if publicURL == "" { return "", errNoPublicURL diff --git a/internal/api/auth_register.go b/internal/api/auth_register.go index eec2fd72..02d142f5 100644 --- a/internal/api/auth_register.go +++ b/internal/api/auth_register.go @@ -175,7 +175,8 @@ func (h *handlers) handleRegister(w http.ResponseWriter, r *http.Request) { return } - // Atomic invite redeem (if applicable). Now we have user.ID for redeemed_by. + // Atomic invite redeem (if applicable). Now we have user.ID for + // redeemed_by. if usedInviteToken != "" { rows, rerr := q.RedeemInvite(r.Context(), dbq.RedeemInviteParams{ Token: usedInviteToken, @@ -214,7 +215,7 @@ func (h *handlers) handleRegister(w http.ResponseWriter, r *http.Request) { Value: sessionToken, Path: "/", HttpOnly: true, - Secure: auth.IsHTTPS(r, h.netSettings.Hops()), // plain HTTP and LAN stay functional (rule 94) + Secure: auth.IsHTTPS(r, h.netSettings.Hops()), // plain HTTP and LAN stay functional SameSite: http.SameSiteStrictMode, MaxAge: int(sessionCookieMaxAge.Seconds()), }) diff --git a/internal/api/auth_register_test.go b/internal/api/auth_register_test.go index d64de07e..34b51587 100644 --- a/internal/api/auth_register_test.go +++ b/internal/api/auth_register_test.go @@ -158,7 +158,8 @@ func TestRegister_OpenMode_NoTokenNeeded(t *testing.T) { h, pool := testHandlers(t) seedUser(t, pool, "preexisting", "pw", true) // Switch to open mode; restore invite_only afterwards so other tests are - // not affected (registration_settings is a singleton not truncated by ResetDB). + // not affected (registration_settings is a singleton not truncated by + // ResetDB). if _, err := pool.Exec(context.Background(), "UPDATE registration_settings SET mode = 'open' WHERE id = true"); err != nil { t.Fatalf("set mode: %v", err) diff --git a/internal/api/auth_test.go b/internal/api/auth_test.go index feb137ae..a09a1a0a 100644 --- a/internal/api/auth_test.go +++ b/internal/api/auth_test.go @@ -18,20 +18,21 @@ import ( "golang.org/x/crypto/bcrypt" "git.fabledsword.com/bvandeusen/minstrel/internal/auth" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" "git.fabledsword.com/bvandeusen/minstrel/internal/config" "git.fabledsword.com/bvandeusen/minstrel/internal/db" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" "git.fabledsword.com/bvandeusen/minstrel/internal/library" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrquarantine" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" "git.fabledsword.com/bvandeusen/minstrel/internal/mailer" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" "git.fabledsword.com/bvandeusen/minstrel/internal/playevents" "git.fabledsword.com/bvandeusen/minstrel/internal/playlists" + "git.fabledsword.com/bvandeusen/minstrel/internal/quarantine" "git.fabledsword.com/bvandeusen/minstrel/internal/recsettings" + "git.fabledsword.com/bvandeusen/minstrel/internal/requests" "git.fabledsword.com/bvandeusen/minstrel/internal/tracks" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" ) // testHandlers spins up a handlers instance against MINSTREL_TEST_DATABASE_URL. @@ -64,22 +65,19 @@ func testHandlers(t *testing.T) (*handlers, *pgxpool.Pool) { if err != nil { t.Fatalf("recsettings: %v", err) } - lidarrCfg := lidarrconfig.New(pool) - lidarrReqs := lidarrrequests.NewService(pool, lidarrCfg, nil, nil) - lidarrQuar := lidarrquarantine.NewService(pool, lidarrCfg, nil, "") - // tracks.Service has no Lidarr unmonitorer in tests by default; the - // admin-tracks tests below override h.tracks via installTracksLidarrStub - // when they need a stubbed Lidarr. + requestSvc := requests.NewService(pool) + cat := catalogue.New(pool, nil, logger) + quarantineSvc := quarantine.NewService(pool, "") dataDir := t.TempDir() - tracksSvc := tracks.NewService(pool, logger, nil, dataDir) + tracksSvc := tracks.NewService(pool, logger, dataDir) playlistsSvc := playlists.NewService(pool, logger, dataDir) - h := &handlers{pool: pool, logger: logger, events: w, recCfg: recCfg, recSettings: recSettings, rng: func() float64 { return 0.5 }, lidarrCfg: lidarrCfg, lidarrRequests: lidarrReqs, lidarrQuarantine: lidarrQuar, tracks: tracksSvc, playlists: playlistsSvc, dataDir: dataDir, scanner: nil, scanCfg: library.RunScanConfig{}, mailer: &mailer.FakeSender{}, notifier: notifications.New(pool, nil, nil)} + h := &handlers{pool: pool, logger: logger, events: w, recCfg: recCfg, recSettings: recSettings, rng: func() float64 { return 0.5 }, requests: requestSvc, catalogue: cat, wanted: wanted.New(pool, cat, logger), quarantine: quarantineSvc, tracks: tracksSvc, playlists: playlistsSvc, dataDir: dataDir, scanner: nil, scanCfg: library.RunScanConfig{}, mailer: &mailer.FakeSender{}, notifier: notifications.New(pool, nil, nil)} return h, pool } // seedUser creates a test user. The supplied username is automatically // prefixed with dbtest.TestUserPrefix so dbtest.ResetDB cleans it up -// without touching the operator's admin row. Callers that need to send +// without touching the developer's admin row. Callers that need to send // the username in HTTP bodies or compare against API responses must // include the same prefix in their literals. func seedUser(t *testing.T, pool *pgxpool.Pool, username, password string, isAdmin bool) dbq.User { diff --git a/internal/api/cast_token.go b/internal/api/cast_token.go index 8930f2c5..2705948c 100644 --- a/internal/api/cast_token.go +++ b/internal/api/cast_token.go @@ -20,7 +20,7 @@ const ( type castTokenRequest struct { TrackID string `json:"trackId"` ExpSeconds int `json:"expSeconds,omitempty"` - // Level asks for the leveled stream (M464 #5001): the track rendered at + // Level asks for the leveled stream: the track rendered at // the user's loudness gain. AsAlbum says the track is playing as part of // its album in order, which picks album gain in auto mode; only the // client holding the queue knows it. @@ -68,7 +68,7 @@ func mimeForFormat(format string) string { // extension (Content-Type header alone is insufficient) -- without a // recognizable extension, Sonos reports TrackDuration=0 and seeks // trigger auto-advance because every position past 0 looks past-the- -// end. Defaults to "mp3" for unknown formats. See task #610. +// end. Defaults to "mp3" for unknown formats. func extForFormat(format string) string { switch strings.ToLower(strings.TrimSpace(format)) { case "mp3", "mpeg": @@ -101,9 +101,6 @@ func extForFormat(format string) string { // // expSeconds is clamped to [60, 86400]; default 21600 (6h) — long enough // to play through any typical track without re-minting mid-playback. -// -// Part of the output-picker UPnP slice. See -// docs/superpowers/specs/2026-06-03-android-output-picker-upnp-design.md. func (h *handlers) handleCastStreamToken(w http.ResponseWriter, r *http.Request) { user, ok := requireUser(w, r) if !ok { diff --git a/internal/api/cast_token_test.go b/internal/api/cast_token_test.go index 526f50ed..b08bd1f2 100644 --- a/internal/api/cast_token_test.go +++ b/internal/api/cast_token_test.go @@ -54,7 +54,7 @@ func TestCastStreamToken_HappyPath(t *testing.T) { t.Fatalf("URL missing stream path: %s", resp.URL) } // Stream URL must carry a file extension so Sonos's URL probe can - // identify the audio format (see task #610). Track seeded above is + // identify the audio format. Track seeded above is // .flac via seedTrack's default file_format. if !strings.Contains(resp.URL, "/stream.flac?") { t.Fatalf("URL missing file-extension segment: %s", resp.URL) diff --git a/internal/api/catalogue.go b/internal/api/catalogue.go new file mode 100644 index 00000000..9b883600 --- /dev/null +++ b/internal/api/catalogue.go @@ -0,0 +1,284 @@ +package api + +import ( + "context" + "errors" + "net/http" + "net/url" + "slices" + "strings" + "sync" + "time" + + "github.com/go-chi/chi/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" +) + +// catalogueSearchLimit is how many results one search shows. +const catalogueSearchLimit = 25 + +// catalogueSearchResult is one row of GET /api/catalogue/search, read by the +// Discover page and request rows. artist_mbid and album_mbid are the parents a +// request needs: empty on an artist, the artist on an album, both on a track. +// An album is a MusicBrainz release group. +type catalogueSearchResult struct { + MBID string `json:"mbid"` + Name string `json:"name"` + SecondaryText string `json:"secondary_text"` + ImageURL string `json:"image_url"` + ArtistMBID string `json:"artist_mbid"` + AlbumMBID string `json:"album_mbid"` + InLibrary bool `json:"in_library"` + Requested bool `json:"requested"` +} + +// handleCatalogueSearch implements GET +// /api/catalogue/search?q=&kind=artist|album|track asking MusicBrainz directly. +// An unreachable MusicBrainz is a 503 the page shows as such, not an empty +// result. +func (h *handlers) handleCatalogueSearch(w http.ResponseWriter, r *http.Request) { + kind := strings.TrimSpace(r.URL.Query().Get("kind")) + switch kind { + case "artist", "album", "track": + default: + writeErr(w, apierror.BadRequest("bad_kind", "kind must be artist, album, or track")) + return + } + q := strings.TrimSpace(r.URL.Query().Get("q")) + if q == "" { + writeErr(w, apierror.BadRequest("missing_query", "q is required")) + return + } + + out, err := h.catalogueSearches.get(r.Context(), kind, q, h.searchCatalogue) + if err != nil { + switch { + case errors.Is(err, musicbrainz.ErrUnreachable), errors.Is(err, musicbrainz.ErrBadResponse): + writeErr(w, &apierror.Error{Status: 503, Code: "musicbrainz_unreachable", Message: "MusicBrainz can't be reached", Cause: err}) + case r.Context().Err() != nil: + return + default: + h.logger.Error("api: catalogue search", "kind", kind, "err", err) + writeErr(w, apierror.InternalMsg("search failed", err)) + } + return + } + + dbQ := dbq.New(h.pool) + for i := range out { + inLib, err := h.catalogueInLibrary(r.Context(), kind, out[i].MBID) + if err != nil { + writeErr(w, apierror.InternalMsg("library check failed", err)) + return + } + requested, err := dbQ.HasNonTerminalRequestForMBID(r.Context(), out[i].MBID) + if err != nil { + writeErr(w, apierror.InternalMsg("request check failed", err)) + return + } + out[i].InLibrary, out[i].Requested = inLib, requested + } + writeJSON(w, http.StatusOK, out) +} + +func (h *handlers) searchCatalogue(ctx context.Context, kind, q string) ([]catalogueSearchResult, error) { + mb := h.catalogue.MusicBrainz() + out := []catalogueSearchResult{} + switch kind { + case "artist": + artists, err := mb.SearchArtists(ctx, q, catalogueSearchLimit) + if err != nil { + return nil, err + } + for _, a := range artists { + out = append(out, catalogueSearchResult{ + MBID: a.MBID, Name: a.Name, SecondaryText: artistLine(a), + ImageURL: artistArtURL(a.MBID, a.Name), + }) + } + case "album": + groups, err := mb.SearchReleaseGroups(ctx, q, catalogueSearchLimit) + if err != nil { + return nil, err + } + for _, g := range groups { + out = append(out, catalogueSearchResult{ + MBID: g.MBID, Name: g.Title, SecondaryText: joinNonEmpty(g.ArtistCredit, g.Year(), groupType(g)), + ImageURL: groupArtURL(g), ArtistMBID: first(g.ArtistMBIDs), + }) + } + case "track": + recs, err := mb.SearchRecordings(ctx, q, catalogueSearchLimit) + if err != nil { + return nil, err + } + for _, rec := range recs { + res := catalogueSearchResult{ + MBID: rec.MBID, Name: rec.Title, + SecondaryText: joinNonEmpty(rec.ReleaseGroup.Title, rec.ArtistCredit), + ArtistMBID: first(rec.ArtistMBIDs), AlbumMBID: rec.ReleaseGroup.MBID, + } + if rec.ReleaseGroup.MBID != "" { + res.ImageURL = groupArtURL(rec.ReleaseGroup) + } + out = append(out, res) + } + } + return out, nil +} + +// catalogueSearchTTL is how long a search's MusicBrainz answer is reused. +// MusicBrainz allows a request a second, and Discover searches as the user +// types, so the same words are often asked again within moments. +const catalogueSearchTTL = 10 * time.Minute + +// catalogueSearchCacheMax bounds the cache; past it, the cache starts over. +const catalogueSearchCacheMax = 500 + +// searchCache keeps recent MusicBrainz answers by kind and query. A nil +// cache asks every time. Whether a result is held or requested is not +// cached: the handler works that out on every request. +type searchCache struct { + mu sync.Mutex + entries map[string]searchEntry + now func() time.Time +} + +type searchEntry struct { + at time.Time + results []catalogueSearchResult +} + +func newSearchCache() *searchCache { + return &searchCache{entries: map[string]searchEntry{}, now: time.Now} +} + +func (c *searchCache) get(ctx context.Context, kind, q string, + ask func(context.Context, string, string) ([]catalogueSearchResult, error), +) ([]catalogueSearchResult, error) { + if c == nil { + return ask(ctx, kind, q) + } + key := kind + "\x00" + strings.ToLower(q) + c.mu.Lock() + e, ok := c.entries[key] + c.mu.Unlock() + if ok && c.now().Sub(e.at) < catalogueSearchTTL { + return slices.Clone(e.results), nil + } + results, err := ask(ctx, kind, q) + if err != nil { + return nil, err + } + c.mu.Lock() + if len(c.entries) >= catalogueSearchCacheMax { + c.entries = map[string]searchEntry{} + } + c.entries[key] = searchEntry{at: c.now(), results: slices.Clone(results)} + c.mu.Unlock() + return results, nil +} + +// artistLine is what tells two artists of one name apart: MusicBrainz's own +// disambiguation when it has one, else the kind of artist and the country. +func artistLine(a musicbrainz.Artist) string { + if a.Disambiguation != "" { + return a.Disambiguation + } + return joinNonEmpty(a.Type, a.Country) +} + +// groupType names an album's kind the way people do: "Album", "EP", +// "Album · Live", "Single · Remix". +func groupType(g musicbrainz.ReleaseGroup) string { + return joinNonEmpty(append([]string{g.PrimaryType}, g.SecondaryTypes...)...) +} + +func joinNonEmpty(parts ...string) string { + kept := parts[:0:0] + for _, p := range parts { + if p != "" { + kept = append(kept, p) + } + } + return strings.Join(kept, " · ") +} + +func first(s []string) string { + if len(s) == 0 { + return "" + } + return s[0] +} + +// artistArtURL and groupArtURL point at Minstrel's own art endpoints. The +// names ride along for the sources that look art up by name. +func artistArtURL(mbid, name string) string { + if mbid == "" { + return "" + } + return "/api/catalogue/art/artist/" + mbid + "?" + url.Values{"name": {name}}.Encode() +} + +func groupArtURL(g musicbrainz.ReleaseGroup) string { + return "/api/catalogue/art/release-group/" + g.MBID + "?" + + url.Values{"artist": {g.ArtistCredit}, "title": {g.Title}}.Encode() +} + +// catalogueInLibrary reports whether a result is already held: an artist by +// its MBID, an album by its release group (or a release id that happens to be +// the same), a track by its recording. +func (h *handlers) catalogueInLibrary(ctx context.Context, kind, mbid string) (bool, error) { + var query string + switch kind { + case "artist": + query = `SELECT EXISTS(SELECT 1 FROM artists WHERE mbid = $1)` + case "album": + query = `SELECT EXISTS(SELECT 1 FROM albums WHERE release_group_mbid = $1 OR mbid = $1)` + case "track": + query = `SELECT EXISTS(SELECT 1 FROM tracks WHERE mbid = $1)` + default: + return false, nil + } + var exists bool + err := h.pool.QueryRow(ctx, query, mbid).Scan(&exists) + return exists, err +} + +// handleCatalogueArt implements GET /api/catalogue/art/{kind}/{mbid} for an +// artist or a release group Minstrel doesn't hold. The image comes from the +// enabled cover-art sources and is kept on disk, so the browser +// never fetches a third-party host. A 404 shows the placeholder. +func (h *handlers) handleCatalogueArt(w http.ResponseWriter, r *http.Request) { + kind, mbid := chi.URLParam(r, "kind"), chi.URLParam(r, "mbid") + if !catalogue.ValidMBID(mbid) { + writeErr(w, apierror.NotFound("art")) + return + } + var ( + body []byte + err error + ) + switch kind { + case "artist": + body, err = h.catalogueArt.Artist(r.Context(), mbid, r.URL.Query().Get("name")) + case "release-group": + body, err = h.catalogueArt.ReleaseGroup(r.Context(), mbid, r.URL.Query().Get("artist"), r.URL.Query().Get("title")) + default: + writeErr(w, apierror.NotFound("art")) + return + } + if err != nil { + if r.Context().Err() == nil { + writeErr(w, apierror.NotFound("art")) + } + return + } + w.Header().Set("Content-Type", http.DetectContentType(body)) + w.Header().Set("Cache-Control", "private, max-age=86400") + _, _ = w.Write(body) +} diff --git a/internal/api/catalogue_test.go b/internal/api/catalogue_test.go new file mode 100644 index 00000000..0dfdf7f6 --- /dev/null +++ b/internal/api/catalogue_test.go @@ -0,0 +1,254 @@ +package api + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "testing" + "time" + + "github.com/go-chi/chi/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/coverart" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" +) + +const ( + searchArtistMBID = "a74b1b7f-71a5-4011-9441-d0b5e4122711" + searchGroupMBID = "b1392450-e666-3926-a536-22c65f834433" + searchRecMBID = "c1a0e1b2-0000-4000-8000-000000000001" +) + +// fakeMusicBrainz answers the three searches with one Radiohead result each, +// or 503 when down is set. +func fakeMusicBrainz(t *testing.T, down bool) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if down { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + credit := `[{"name": "Radiohead", "artist": {"id": "` + searchArtistMBID + `"}}]` + group := `{"id": "` + searchGroupMBID + `", "title": "OK Computer", "primary-type": "Album", + "first-release-date": "1997-05-21", "artist-credit": ` + credit + `}` + switch r.URL.Path { + case "/artist": + _, _ = w.Write([]byte(`{"artists": [{"id": "` + searchArtistMBID + `", "name": "Radiohead", "type": "Group", "country": "GB"}]}`)) + case "/release-group": + _, _ = w.Write([]byte(`{"release-groups": [` + group + `]}`)) + case "/recording": + _, _ = w.Write([]byte(`{"recordings": [{"id": "` + searchRecMBID + `", "title": "Karma Police", + "artist-credit": ` + credit + `, + "releases": [{"id": "r1", "status": "Official", "release-group": ` + group + `}]}]}`)) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + return srv +} + +func catalogueHandlers(t *testing.T, down bool) *handlers { + t.Helper() + h, pool := testHandlers(t) + resetRequests(t, h) + mb := &musicbrainz.Client{BaseURL: fakeMusicBrainz(t, down).URL, Pace: &musicbrainz.Pacer{}, Retries: -1} + h.catalogue = catalogue.New(pool, mb, h.logger) + return h +} + +func doCatalogueSearch(t *testing.T, h *handlers, q, kind string) *httptest.ResponseRecorder { + t.Helper() + req := httptest.NewRequest(http.MethodGet, "/api/catalogue/search?"+url.Values{"q": {q}, "kind": {kind}}.Encode(), nil) + w := httptest.NewRecorder() + h.handleCatalogueSearch(w, req) + return w +} + +func decodeResults(t *testing.T, w *httptest.ResponseRecorder) []catalogueSearchResult { + t.Helper() + if w.Code != http.StatusOK { + t.Fatalf("status = %d, body = %s", w.Code, w.Body.String()) + } + var out []catalogueSearchResult + if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil { + t.Fatalf("decode: %v; body = %s", err, w.Body.String()) + } + return out +} + +func errorCode(t *testing.T, w *httptest.ResponseRecorder) string { + t.Helper() + var resp struct { + Error struct { + Code string `json:"code"` + } `json:"error"` + } + if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { + t.Fatalf("decode: %v; body = %s", err, w.Body.String()) + } + return resp.Error.Code +} + +func TestCatalogueSearch_RefusesABadKindOrNoQuery(t *testing.T) { + h := catalogueHandlers(t, false) + if w := doCatalogueSearch(t, h, "x", "playlist"); w.Code != http.StatusBadRequest || errorCode(t, w) != "bad_kind" { + t.Errorf("bad kind: %d %s", w.Code, w.Body.String()) + } + if w := doCatalogueSearch(t, h, " ", "artist"); w.Code != http.StatusBadRequest || errorCode(t, w) != "missing_query" { + t.Errorf("no query: %d %s", w.Code, w.Body.String()) + } +} + +func TestCatalogueSearch_EachKindAsksMusicBrainz(t *testing.T) { + h := catalogueHandlers(t, false) + + artists := decodeResults(t, doCatalogueSearch(t, h, "radiohead", "artist")) + if len(artists) != 1 || artists[0].MBID != searchArtistMBID || artists[0].SecondaryText != "Group · GB" { + t.Fatalf("artists = %+v", artists) + } + if artists[0].ImageURL != "/api/catalogue/art/artist/"+searchArtistMBID+"?name=Radiohead" { + t.Errorf("artist art = %q: it must be Minstrel's own endpoint", artists[0].ImageURL) + } + + albums := decodeResults(t, doCatalogueSearch(t, h, "ok computer", "album")) + if len(albums) != 1 || albums[0].MBID != searchGroupMBID || albums[0].ArtistMBID != searchArtistMBID || + albums[0].SecondaryText != "Radiohead · 1997 · Album" { + t.Fatalf("albums = %+v", albums) + } + + tracks := decodeResults(t, doCatalogueSearch(t, h, "karma police", "track")) + if len(tracks) != 1 || tracks[0].AlbumMBID != searchGroupMBID || tracks[0].ArtistMBID != searchArtistMBID || + tracks[0].SecondaryText != "OK Computer · Radiohead" { + t.Fatalf("tracks = %+v", tracks) + } +} + +func TestCatalogueSearch_MarksWhatIsHeldAndWhatIsRequested(t *testing.T) { + h := catalogueHandlers(t, false) + ctx := context.Background() + q := dbq.New(h.pool) + + artistMBID := searchArtistMBID + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: dbtest.TestUserPrefix + "Radiohead", SortName: "Radiohead", Mbid: &artistMBID}) + if err != nil { + t.Fatal(err) + } + // The album is held under a release id; its release group is what matches. + if _, err := h.pool.Exec(ctx, + `INSERT INTO albums (title, sort_title, artist_id, mbid, release_group_mbid) VALUES ('OK Computer', 'OK Computer', $1, 'some-release-id', $2)`, + artist.ID, searchGroupMBID); err != nil { + t.Fatal(err) + } + if got := decodeResults(t, doCatalogueSearch(t, h, "radiohead", "artist")); !got[0].InLibrary { + t.Error("the artist is held") + } + if got := decodeResults(t, doCatalogueSearch(t, h, "ok computer", "album")); !got[0].InLibrary { + t.Error("the album is held, matched by release group") + } + + user := seedUser(t, h.pool, "requester", "pw", false) + track := searchRecMBID + if _, err := q.CreateRequest(ctx, dbq.CreateRequestParams{ + UserID: user.ID, Kind: dbq.RequestKindTrack, ArtistMbid: searchArtistMBID, + TrackMbid: &track, ArtistName: "Radiohead", + }); err != nil { + t.Fatal(err) + } + got := decodeResults(t, doCatalogueSearch(t, h, "karma police", "track")) + if got[0].InLibrary || !got[0].Requested { + t.Errorf("track = %+v, want requested and not held", got[0]) + } +} + +func TestCatalogueSearch_UnreachableMusicBrainzIs503(t *testing.T) { + h := catalogueHandlers(t, true) + w := doCatalogueSearch(t, h, "radiohead", "artist") + if w.Code != http.StatusServiceUnavailable || errorCode(t, w) != "musicbrainz_unreachable" { + t.Fatalf("status %d, body %s", w.Code, w.Body.String()) + } +} + +// artSource is an artist-art source with one image. +type artSource struct{ apiTestAlbumProvider } + +func (artSource) FetchArtistArt(_ context.Context, _ coverart.ArtistRef) ([]byte, []byte, error) { + return []byte("\x89PNG\r\n\x1a\nrest"), nil, nil +} + +type fakeArtSources struct{ artists []coverart.ArtistArtProvider } + +func (s fakeArtSources) EnabledArtistProviders() []coverart.ArtistArtProvider { return s.artists } +func (fakeArtSources) EnabledAlbumProviders() []coverart.AlbumCoverProvider { return nil } + +func TestCatalogueArt_ServesTheImageOrAPlaceholder404(t *testing.T) { + h := &handlers{catalogueArt: catalogue.NewArt(t.TempDir(), fakeArtSources{artists: []coverart.ArtistArtProvider{&artSource{}}})} + r := chi.NewRouter() + r.Get("/api/catalogue/art/{kind}/{mbid}", h.handleCatalogueArt) + + get := func(path string) *httptest.ResponseRecorder { + w := httptest.NewRecorder() + r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil)) + return w + } + if w := get("/api/catalogue/art/artist/" + searchArtistMBID + "?name=Radiohead"); w.Code != http.StatusOK || w.Header().Get("Content-Type") != "image/png" { + t.Errorf("artist: %d %q", w.Code, w.Header().Get("Content-Type")) + } + // No album source is enabled: a 404, which the page shows as its + // placeholder. + if w := get("/api/catalogue/art/release-group/" + searchGroupMBID); w.Code != http.StatusNotFound { + t.Errorf("release group with no source: %d", w.Code) + } + for _, bad := range []string{"/api/catalogue/art/artist/not-an-id", "/api/catalogue/art/label/" + searchArtistMBID} { + if w := get(bad); w.Code != http.StatusNotFound { + t.Errorf("%s: %d", bad, w.Code) + } + } +} + +func TestSuggestionArt_PointsAtMinstrelsOwnEndpoint(t *testing.T) { + views := []suggestionView{{MBID: searchArtistMBID, Name: "Sigur Rós"}, {Name: "No MBID"}} + resolveSuggestionArt(views) + if views[0].ImageURL != "/api/catalogue/art/artist/"+searchArtistMBID+"?name=Sigur+R%C3%B3s" { + t.Errorf("image_url = %q", views[0].ImageURL) + } + if views[1].ImageURL != "" { + t.Errorf("no MBID, no image: %q", views[1].ImageURL) + } +} + +func TestSearchCache_ReusesAnAnswerButNotTheFlags(t *testing.T) { + c := newSearchCache() + now := time.Now() + c.now = func() time.Time { return now } + asks := 0 + ask := func(context.Context, string, string) ([]catalogueSearchResult, error) { + asks++ + return []catalogueSearchResult{{MBID: "m"}}, nil + } + first, _ := c.get(context.Background(), "artist", "Radiohead", ask) + first[0].InLibrary = true // the handler sets flags on what it is given + again, _ := c.get(context.Background(), "artist", "radiohead", ask) + if asks != 1 { + t.Errorf("asks = %d: the same words in another case are the same search", asks) + } + if again[0].InLibrary { + t.Error("a flag set on one answer leaked into the cache") + } + if _, _ = c.get(context.Background(), "album", "radiohead", ask); asks != 2 { + t.Error("another kind is another search") + } + now = now.Add(catalogueSearchTTL + time.Second) + if _, _ = c.get(context.Background(), "artist", "radiohead", ask); asks != 3 { + t.Error("an old answer is asked again") + } + var none *searchCache + if _, _ = none.get(context.Background(), "artist", "x", ask); asks != 4 { + t.Error("a nil cache asks every time") + } +} diff --git a/internal/api/client_assets.go b/internal/api/client_assets.go index 5f7e8c39..0ce6f507 100644 --- a/internal/api/client_assets.go +++ b/internal/api/client_assets.go @@ -1,6 +1,6 @@ package api -// In-app update endpoints (#397). The Android APK ships bundled with +// In-app update endpoints. The Android APK ships bundled with // the server image so the client can self-update without an external // app store. CI sequencing bakes the APK + sidecar version file into // /app/client/ at image build time. @@ -13,15 +13,15 @@ package api // download every 60s. Real install flows fire one download per // update; anything tighter is scripted/abusive. // -// Returns 404 gracefully when the APK isn't present (dev environments, -// pre-CI-wiring); the client treats 404 as "no update channel available." -// -// (These paragraphs said "the Flutter client" until 2026-09-10. That client -// was deleted in v2026.08.18 — the Android app is the only one now.) +// Returns 404 gracefully when the APK isn't present (a local build); the +// client treats 404 as "no update channel available." import ( + "crypto/sha256" + "encoding/hex" "encoding/json" "errors" + "io" "net/http" "os" "path/filepath" @@ -65,6 +65,41 @@ var ( clientAPKLastDownloadMu sync.Mutex ) +// clientAPKSum is the bundled APK's sha256, kept until the file changes +// under it (another size or modification time), so a version check every +// few minutes from every phone does not re-read tens of megabytes each time. +type clientAPKSum struct { + size int64 + mod time.Time + hex string +} + +var ( + clientAPKSumCache clientAPKSum + clientAPKSumCacheMu sync.Mutex +) + +// clientAPKDigest returns the sha256 of the APK at path, in hex. +func clientAPKDigest(path string, st os.FileInfo) (string, error) { + clientAPKSumCacheMu.Lock() + defer clientAPKSumCacheMu.Unlock() + c := clientAPKSumCache + if c.hex != "" && c.size == st.Size() && c.mod.Equal(st.ModTime()) { + return c.hex, nil + } + f, err := os.Open(path) + if err != nil { + return "", err + } + defer func() { _ = f.Close() }() + h := sha256.New() + if _, err := io.Copy(h, f); err != nil { + return "", err + } + clientAPKSumCache = clientAPKSum{size: st.Size(), mod: st.ModTime(), hex: hex.EncodeToString(h.Sum(nil))} + return clientAPKSumCache.hex, nil +} + func testResetClientAPKRateLimit() { clientAPKLastDownloadMu.Lock() defer clientAPKLastDownloadMu.Unlock() @@ -96,10 +131,10 @@ func clientAPKAllowDownload(userID string, now time.Time) time.Duration { // installed over that?" — the name never does. // - Channel is a SIBLING FIELD, never a suffix inside the name. // -// JSON rather than a positional line on purpose. The obvious growth path for -// the old one-value file was " ", which a first-space split -// silently mangles the moment a third field appears: the code stops parsing, -// and the reader falls back to name comparison WITHOUT erroring. +// JSON rather than a positional line on purpose: a " " line split +// on the first space silently mangles the moment a third field appears — the +// code stops parsing, and the reader falls back to name comparison WITHOUT +// erroring. type clientVersionSidecar struct { Name string `json:"name"` // Pointer, not int64: absent must stay distinguishable from zero. An @@ -118,6 +153,11 @@ type clientVersionResponse struct { Channel string `json:"channel,omitempty"` APKURL string `json:"apk_url"` SizeBytes int64 `json:"size_bytes"` + // SHA256 is worked out from the file being served, not copied from the + // sidecar, so it cannot disagree with the bytes the phone receives. The + // app downloads ahead on Wi-Fi and installs later; this is how it knows + // the file it kept is the whole build that was offered. + SHA256 string `json:"sha256"` } // handleClientVersion returns the bundled Android client version + a @@ -162,12 +202,19 @@ func (h *handlers) handleClientVersion(w http.ResponseWriter, _ *http.Request) { return } + sum, err := clientAPKDigest(apkPath, stat) + if err != nil { + writeErrWithLog(w, h.logger, "client_version: digest apk", err) + return + } + writeJSON(w, http.StatusOK, clientVersionResponse{ Version: strings.TrimSpace(sidecar.Name), Code: sidecar.Code, Channel: strings.TrimSpace(sidecar.Channel), APKURL: "/api/client/apk", SizeBytes: stat.Size(), + SHA256: sum, }) } diff --git a/internal/api/client_assets_test.go b/internal/api/client_assets_test.go index fdd735ec..eb8a77a1 100644 --- a/internal/api/client_assets_test.go +++ b/internal/api/client_assets_test.go @@ -2,6 +2,8 @@ package api import ( "context" + "crypto/sha256" + "encoding/hex" "encoding/json" "io" "log/slog" @@ -128,6 +130,39 @@ func TestClientVersion_200WithBothFiles(t *testing.T) { if resp.SizeBytes != size { t.Errorf("size_bytes: want %d, got %d", size, resp.SizeBytes) } + want := sha256.Sum256([]byte("fake apk content")) + if resp.SHA256 != hex.EncodeToString(want[:]) { + t.Errorf("sha256: want the served file's digest %x, got %q", want, resp.SHA256) + } +} + +// The digest is cached by size and modification time; a new APK dropped in +// place must be answered with its own digest, not the old one's. +func TestClientVersion_DigestFollowsAReplacedAPK(t *testing.T) { + writeClientAssets(t, `{"name":"a","code":1}`) + first := decodeClientVersion(t, getClientVersion(t)) + dir := os.Getenv("MINSTREL_CLIENT_APK_DIR") + apk := filepath.Join(dir, clientAPKFilename) + if err := os.WriteFile(apk, []byte("another, longer build"), 0o644); err != nil { + t.Fatal(err) + } + second := decodeClientVersion(t, getClientVersion(t)) + want := sha256.Sum256([]byte("another, longer build")) + if second.SHA256 != hex.EncodeToString(want[:]) || second.SHA256 == first.SHA256 { + t.Errorf("sha256 after replace: want %x, got %q (before: %q)", want, second.SHA256, first.SHA256) + } +} + +func decodeClientVersion(t *testing.T, rr *httptest.ResponseRecorder) clientVersionResponse { + t.Helper() + if rr.Code != http.StatusOK { + t.Fatalf("want 200, got %d (body: %s)", rr.Code, rr.Body.String()) + } + var resp clientVersionResponse + if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil { + t.Fatal(err) + } + return resp } // A release published before ordering keys were recorded has a name and diff --git a/internal/api/convert.go b/internal/api/convert.go index 2ea8b1da..a48ba868 100644 --- a/internal/api/convert.go +++ b/internal/api/convert.go @@ -55,9 +55,9 @@ func yearFromDate(d pgtype.Date) int { return d.Time.Year() } -// durationMsToSec converts millisecond durations (as stored in tracks.duration_ms) -// to seconds, rounding to nearest. Callers display seconds; sub-second precision -// is not useful in UI. +// durationMsToSec converts millisecond durations (as stored in +// tracks.duration_ms) to seconds, rounding to nearest. Callers display seconds; +// sub-second precision is not useful in UI. func durationMsToSec(ms int32) int { return int((ms + 500) / 1000) } @@ -79,7 +79,7 @@ func streamURL(trackID pgtype.UUID) string { // cast tokens. Sonos's URL probe gates duration detection on a recognizable // audio file extension; the bare `/stream` shape reports TrackDuration=0 // and breaks seek/auto-advance. The bare /stream route stays mounted as an -// alias for legacy / web / Subsonic clients. See task #610. +// alias for web and Subsonic clients. func streamURLWithExt(trackID pgtype.UUID, ext string) string { return streamURL(trackID) + "." + ext } diff --git a/internal/api/errors.go b/internal/api/errors.go index be7c0427..71b81d3c 100644 --- a/internal/api/errors.go +++ b/internal/api/errors.go @@ -51,7 +51,7 @@ func writeRateLimited(w http.ResponseWriter, wait time.Duration) { } // writeErrWithLog logs the error at Error level and writes the response. -// Use for 500-class errors where the operator needs the cause in logs. +// Use for 500-class errors whose cause belongs in the logs. func writeErrWithLog(w http.ResponseWriter, logger *slog.Logger, msg string, err error) { if logger != nil { logger.Error(msg, "err", err) @@ -67,3 +67,12 @@ func writeJSON(w http.ResponseWriter, status int, body any) { w.WriteHeader(status) _ = json.NewEncoder(w).Encode(body) } + +// writeAdminJSONErr writes a flat {"error":""} JSON response. This +// mirrors the shape the RequireAdmin middleware uses and that the spec +// defines for /api/admin/* errors. +func writeAdminJSONErr(w http.ResponseWriter, status int, code string) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _ = json.NewEncoder(w).Encode(map[string]string{"error": code}) +} diff --git a/internal/api/events.go b/internal/api/events.go index 191d1688..5946d78b 100644 --- a/internal/api/events.go +++ b/internal/api/events.go @@ -22,16 +22,16 @@ type eventRequest struct { At *string `json:"at"` ClientID *string `json:"client_id"` // Source tags which surface a play_started came from. "for_you" - // / "discover" feed the system-playlist rotation dedup (#415); + // / "discover" feed the system-playlist rotation dedup; // absent / "" for library, user-playlist, radio, Subsonic. Source *string `json:"source"` - // DeviceClass is the client's self-reported device class (#1551), + // DeviceClass is the client's self-reported device class, // normalized + stored per play to condition radio on the current device. DeviceClass *string `json:"device_class"` } // knownDeviceClasses whitelists the device classes the context-affinity facet -// buckets by (#1551). A client sends one; anything unrecognized normalizes to +// buckets by. A client sends one; anything unrecognized normalizes to // "other", and empty/absent to "" (stored NULL → excluded from the device // dimension). Kept permissive (no DB CHECK) so a new client class is one // whitelist entry, not a migration. @@ -66,9 +66,9 @@ const eventFutureSkew = 5 * time.Minute // clampEventTime bounds a client-supplied event timestamp to sanity: // no earlier than the account's creation (offline replays can // legitimately be days old, but no play can predate the user) and no -// later than now + a small skew allowance. Unbounded client clocks -// previously let a skewed device write arbitrarily old plays, which -// poisoned Rediscover's "not played in 6 months" ordering (#1246). +// later than now + a small skew allowance. An unbounded client clock would +// let a skewed device write arbitrarily old plays, poisoning Rediscover's +// "not played in 6 months" ordering. func clampEventTime(at, userCreatedAt, now time.Time) time.Time { if at.Before(userCreatedAt) { return userCreatedAt @@ -157,11 +157,11 @@ func (h *handlers) handleEventPlayStarted( } // handleEventPlayOffline records a complete play that happened with -// no/flaky connectivity, replayed from the Flutter offline mutation -// queue (#426 part B). `at` is the original (client) play-start time -// — parsed by handleEvents from req.At — so history lands on the -// real timeline, not replay time. duration_played_ms drives the skip -// rule; source advances #415 rotation for system-playlist plays. +// no or flaky connectivity, replayed from a client's offline queue. +// `at` is the original (client) play-start time — parsed by handleEvents +// from req.At — so history lands on the real timeline, not replay time. +// duration_played_ms drives the skip rule; source advances the +// system-playlist rotation. func (h *handlers) handleEventPlayOffline( w http.ResponseWriter, r *http.Request, user dbq.User, req eventRequest, at time.Time, clientID string, diff --git a/internal/api/events_publish.go b/internal/api/events_publish.go index 7b77edb4..40eadb46 100644 --- a/internal/api/events_publish.go +++ b/internal/api/events_publish.go @@ -1,7 +1,7 @@ -// Small helpers for posting live events to the SSE bus (#392). Producers -// call these from their handler success paths so the wire shape stays in -// one place. Each helper is a no-op when h.eventbus is nil (older callers -// or tests that don't construct a bus); production always has one. +// Small helpers for posting live events to the SSE bus. Producers call +// these from their handler success paths so the wire shape stays in one +// place. Each helper is a no-op when h.eventbus is nil (tests that don't +// construct a bus); production always has one. package api @@ -48,8 +48,8 @@ func (h *handlers) publishLikeEvent(userID, entityID pgtype.UUID, entityType str } // publishQuarantineEvent broadcasts a quarantine action. User-side -// flag/unflag are scoped to the user; admin-side resolve / delete-file / -// delete-via-lidarr are broadcast (empty UserID) because a single admin +// flag/unflag are scoped to the user; admin-side resolve / delete-file +// are broadcast (empty UserID) because a single admin // action can affect every user who'd flagged that track — every client // invalidates and the non-affected ones no-op when the query result // matches their cache. @@ -103,12 +103,12 @@ func (h *handlers) publishSystemRebuilt(userID pgtype.UUID) { }) } -// publishRequestStatusChanged broadcasts a Lidarr request status flip to +// publishRequestStatusChanged broadcasts a request status flip to // the request's original requester so their /requests page reflects the // new state without manual refresh. Admin actors (approve / reject) still // route to the requester's user_id, not the admin's — the original user // is who needs to see the update. -func (h *handlers) publishRequestStatusChanged(row dbq.LidarrRequest) { +func (h *handlers) publishRequestStatusChanged(row dbq.Request) { if h.eventbus == nil { return } diff --git a/internal/api/events_stream.go b/internal/api/events_stream.go index ff58deb0..f9fccf8f 100644 --- a/internal/api/events_stream.go +++ b/internal/api/events_stream.go @@ -1,6 +1,6 @@ // Live event stream — Server-Sent Events at GET /api/events/stream. // -// Clients (today: Flutter) subscribe to be notified of state changes +// The web app and the Android app subscribe to be notified of state changes // elsewhere in the system (likes, request status flips, quarantine // resolutions, scan progress, playlist mutations) so static screens // can invalidate cached data without explicit user action. @@ -105,7 +105,8 @@ func (h *handlers) handleEventsStream(w http.ResponseWriter, r *http.Request) { case e, ok := <-ch: if !ok { - // Bus closed our subscription (rare; only on full shutdown). + // The bus closed at shutdown. Ending here lets the + // server drain; the client reconnects to the next one. return } // Scope filter: forward only events scoped to this user or diff --git a/internal/api/file_remove_error.go b/internal/api/file_remove_error.go index 3b2441c2..f9cfb670 100644 --- a/internal/api/file_remove_error.go +++ b/internal/api/file_remove_error.go @@ -10,17 +10,17 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/library" ) -// fileRemoveAPIError answers a delete that could not reach the track's file -// (#3918). Both delete endpoints use it, so the operator gets the same +// fileRemoveAPIError answers a delete that could not reach the track's file. +// Both delete endpoints use it, so an admin gets the same // explanation from the admin remove-track action and from quarantine's Delete // file. // // The unwritable case is a 409 rather than a 500 because nothing is broken: the // request conflicts with how the library is mounted, and the fix is the -// operator's. The message names the directory — removal writes to the parent, +// host's. The message names the directory — removal writes to the parent, // not the file — and the uid/gid the process runs as, which is the half of a // permission problem invisible from the host. Every case says nothing was -// deleted, because that is exactly what the operator will be worried about. +// deleted, because that is exactly what an admin will be worried about. func fileRemoveAPIError(err error) (*apierror.Error, bool) { var fre *library.FileRemoveError if !errors.As(err, &fre) { @@ -46,7 +46,7 @@ func fileRemoveAPIError(err error) (*apierror.Error, bool) { } // logFileRemoveFailure records a delete that could not reach its file. An -// unwritable library is an environment fact the operator can fix, so it is a +// unwritable library is an environment fact an admin can fix, so it is a // Warn; anything else is a real fault. func logFileRemoveFailure(logger *slog.Logger, apiErr *apierror.Error, attrs ...any) { attrs = append(attrs, "code", apiErr.Code, "err", apiErr.Cause) diff --git a/internal/api/file_remove_error_test.go b/internal/api/file_remove_error_test.go index 1da062da..f0925865 100644 --- a/internal/api/file_remove_error_test.go +++ b/internal/api/file_remove_error_test.go @@ -15,7 +15,7 @@ import ( const removeTestPath = "/music/Moe Shop/WWW (2020)/01 - WWW.mp3" // removeFailure builds the error a delete service returns when the file would -// not go, wrapped the way lidarrquarantine.DeleteFile and tracks.RemoveTrack +// not go, wrapped the way quarantine.DeleteFile and tracks.RemoveTrack // wrap it — the mapping has to see through that. func removeFailure(errno syscall.Errno) error { return fmt.Errorf("delete file: %w", &library.FileRemoveError{ @@ -72,7 +72,7 @@ func TestFileRemoveAPIError(t *testing.T) { } // The unwritable answer must name the DIRECTORY. Removal needs write access to -// the parent, so a message naming the file would send the operator to fix the +// the parent, so a message naming the file would send an admin to fix the // wrong permissions. The directory is a prefix of the file path, which is why a // plain "contains the directory" check could never catch that regression. func TestFileRemoveAPIError_NotWritableNamesTheDirectoryNotTheFile(t *testing.T) { diff --git a/internal/api/home.go b/internal/api/home.go index 0e2daec5..22014c84 100644 --- a/internal/api/home.go +++ b/internal/api/home.go @@ -59,15 +59,14 @@ func (h *handlers) handleGetHome(w http.ResponseWriter, r *http.Request) { // handleGetHomeIndex implements GET /api/home/index. Returns the same // five sections /api/home computes, but as flat slices of entity IDs -// rather than denormalized objects. Drives the Flutter per-item +// rather than denormalized objects. Drives the Android per-item // rendering path — the client hydrates each tile against the // per-entity endpoints, so this response is tiny (a few KB at most) // and the cold-visit round-trip is correspondingly short. // // Shares the recommendation.HomeData computation with /api/home; the -// DB cost is identical. A future optimisation could add ID-only -// query variants, but the JSON savings already shrink the wire -// payload by roughly an order of magnitude on populated libraries. +// DB cost is identical, and the wire payload is roughly an order of +// magnitude smaller on populated libraries. func (h *handlers) handleGetHomeIndex(w http.ResponseWriter, r *http.Request) { user, ok := requireUser(w, r) if !ok { diff --git a/internal/api/leveled_stream.go b/internal/api/leveled_stream.go index 6c01f708..30e9fb3b 100644 --- a/internal/api/leveled_stream.go +++ b/internal/api/leveled_stream.go @@ -21,7 +21,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/library" ) -// The leveled stream (M464 #5001): a track rendered with the user's loudness +// The leveled stream: a track rendered with the user's loudness // gain applied, for the Sonos and UPnP speakers that fetch their own audio. // See internal/library/leveled.go for how it is rendered. @@ -49,7 +49,7 @@ func VerifyLeveledStreamToken(secret []byte, trackID string, exp int64, g librar } // leveledStreamPath is the leveled stream's path. It ends in .flac because -// Sonos reads the format from the URL's extension (task #610). +// Sonos reads the format from the URL's extension. func leveledStreamPath(trackID pgtype.UUID) string { return "/api/tracks/" + uuidToString(trackID) + "/leveled.flac" } diff --git a/internal/api/library.go b/internal/api/library.go index a4e2c99c..685bf952 100644 --- a/internal/api/library.go +++ b/internal/api/library.go @@ -119,7 +119,8 @@ func (h *handlers) handleGetArtist(w http.ResponseWriter, r *http.Request) { } refs := make([]AlbumRef, 0, len(rows)) for _, row := range rows { - // durationSec=0: not aggregated for nested album lists per spec data flow. + // durationSec=0: not aggregated for nested album lists per spec data + // flow. refs = append(refs, albumRefFrom(row.Album, artist.Name, int(row.TrackCount), 0)) } genres, err := q.ListGenresForArtist(r.Context(), artist.ID) @@ -132,6 +133,9 @@ func (h *handlers) handleGetArtist(w http.ResponseWriter, r *http.Request) { Albums: refs, Genres: nonNilStrings(genres), } + if artist.Mbid != nil { + detail.MBID = *artist.Mbid + } writeJSON(w, http.StatusOK, detail) } @@ -254,7 +258,7 @@ func (h *handlers) handleGetArtistTopTracks(w http.ResponseWriter, r *http.Reque // handleLibraryShuffle implements GET /api/library/shuffle?limit=N — // the online source for the client's always-present "Shuffle all" -// (#427 S4). N random tracks across the whole library, per-user +// N random tracks across the whole library, per-user // quarantine filtered. limit defaults to 100, clamped to 1..500. // Offline, the client shuffles its local cache instead and never // calls this. diff --git a/internal/api/library_albums.go b/internal/api/library_albums.go index f480d311..e57f0e8b 100644 --- a/internal/api/library_albums.go +++ b/internal/api/library_albums.go @@ -39,11 +39,11 @@ type yearFilter struct { // the SPA infinite-scrolls against this endpoint via TanStack // createInfiniteQuery. // -// Optional filters (#367): `genre` and `year_from`/`year_to`. +// Optional filters: `genre` and `year_from`/`year_to`. // // Genre arrives as a QUERY parameter rather than a path segment on purpose. -// Raw ID3 genres routinely contain a slash — "Rock/Pop" is a real tag, and -// the one the task itself cites — which cannot survive a path segment: Go +// Raw ID3 genres routinely contain a slash — "Rock/Pop" is a real tag — +// which cannot survive a path segment: Go // normalises %2F and the router would split the value into two segments. func (h *handlers) handleListLibraryAlbums(w http.ResponseWriter, r *http.Request) { limit, offset, err := parsePaging(r.URL.Query()) diff --git a/internal/api/library_browse.go b/internal/api/library_browse.go index e4c98d85..ce65632e 100644 --- a/internal/api/library_browse.go +++ b/internal/api/library_browse.go @@ -7,18 +7,13 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// genreCount is one row of the genre browse index (#367). +// genreCount is one row of the genre browse index. // // Genres are the tag's own strings, split on [;,] but otherwise untouched — no // case folding and no synonym mapping. So "Rock" and "rock" can both appear, -// as can "Rock/Pop" alongside "Rock" and "Pop". That's deliberate for v1: the +// as can "Rock/Pop" alongside "Rock" and "Pop". That's deliberate: the // alternative is a normalisation table to invent and maintain, and the raw // spread has to be visible before anyone can judge whether it's a problem. -// -// The first look at that spread found it dominated by welded tokens like -// "Alternative RockRock" — the scanner's own bug, not the operator's tagging -// (#2499). Judge the "is a taxonomy needed" question (#2468) only against a -// library re-scanned since that fix. type genreCount struct { Genre string `json:"genre"` TrackCount int `json:"track_count"` diff --git a/internal/api/library_browse_test.go b/internal/api/library_browse_test.go index 1b591a5d..9d8988fe 100644 --- a/internal/api/library_browse_test.go +++ b/internal/api/library_browse_test.go @@ -83,10 +83,9 @@ func TestParseYearFilter(t *testing.T) { } } -// The crux of #367: a track tagged "Rock;Pop" must be reachable from BOTH -// genres. An exact-string match — which is what ListAlbumsByGenre did before -// this task — makes every multi-genre track invisible from either of its -// genres, so the index would list a genre whose page is empty. +// A track tagged "Rock;Pop" must be reachable from BOTH genres. An +// exact-string match makes every multi-genre track invisible from either of +// its genres, so the index would list a genre whose page is empty. func TestListGenres_SplitsMultiGenreTags(t *testing.T) { h, pool := testHandlers(t) artist := seedArtist(t, pool, "Genre Splitter") @@ -121,7 +120,7 @@ func TestListGenres_SplitsMultiGenreTags(t *testing.T) { // Splitting produces leading spaces on every fragment after the first, and // showing " Pop" as a genre distinct from "Pop" would be a bug. Trimming is a -// repair for our own splitting, not normalisation of the operator's tags. +// repair for our own splitting, not normalisation of the library's tags. func TestListGenres_TrimsFragmentWhitespace(t *testing.T) { h, pool := testHandlers(t) artist := seedArtist(t, pool, "Spacey Tags") @@ -223,8 +222,8 @@ func TestListLibraryAlbums_GenreWithSlashSurvives(t *testing.T) { func TestListLibraryAlbums_YearRangeFilter(t *testing.T) { h, pool := testHandlers(t) artist := seedArtist(t, pool, "Chronology") - // Each album needs a playable track: since #2702 the year axis lists only - // albums with something to play, matching what the genre axis already did. + // Each album needs a playable track: the year axis, like the genre axis, + // lists only albums with something to play. for _, a := range []struct { title string year int @@ -341,10 +340,9 @@ func keysOf(m map[string]bool) []string { return out } -// #2702: an album whose every file has gone leaves the year axis, matching -// what the genre axis already did. Before this the two disagreed — the same -// album was absent from genre and still listed under its year, where opening -// it found nothing playable. +// An album whose every file has gone leaves the year axis, as it leaves the +// genre axis: otherwise the same album would be absent from its genre and +// still listed under its year, where opening it finds nothing playable. func TestListAlbumYears_ExcludesFullyMissingAlbums(t *testing.T) { h, pool := testHandlers(t) q := dbq.New(pool) @@ -387,8 +385,8 @@ func TestListAlbumYears_ExcludesFullyMissingAlbums(t *testing.T) { } } -// The index, the list and the count must agree. #367 needed care for exactly -// this reason at the genre level: if they diverge, a year leads to an empty +// The index, the list and the count must agree, as at the genre level: if +// they diverge, a year leads to an empty // page or "Load more" promises rows that never arrive. func TestListLibraryAlbums_YearFilterAndCountAgreeOnMissing(t *testing.T) { h, pool := testHandlers(t) diff --git a/internal/api/library_sync_test.go b/internal/api/library_sync_test.go index 6ffc795c..4539a4a4 100644 --- a/internal/api/library_sync_test.go +++ b/internal/api/library_sync_test.go @@ -41,9 +41,8 @@ func TestLibrarySync_AfterArtistUpsert_ReturnsHydratedPayload(t *testing.T) { truncateLibrary(t, pool) u := seedUser(t, pool, "alice", "x", false) - // Seed an artist + write a change row in the same tx — that's the - // invariant the scanner wiring (Task 5) will enforce, and the - // handler relies on. + // Seed an artist + write a change row in the same tx — the invariant + // library writes keep, and the handler relies on. ctx := context.Background() tx, err := pool.Begin(ctx) if err != nil { @@ -122,8 +121,8 @@ func TestLibrarySync_DeleteIsReflected(t *testing.T) { u := seedUser(t, pool, "alice", "x", false) // Just write a delete change row directly (no underlying mutation). - // Scanner wiring (Task 5) is what couples real deletes to LogChange; - // here we test the handler reflects what's in the log. + // Library writes couple real deletes to LogChange; here we test the + // handler reflects what's in the log. ctx := context.Background() tx, _ := pool.Begin(ctx) _ = sync.LogChange(ctx, tx, sync.EntityArtist, "deadbeef-dead-beef-dead-beefdeadbeef", sync.OpDelete) diff --git a/internal/api/library_sync_views.go b/internal/api/library_sync_views.go index c9038c3c..f3cbd994 100644 --- a/internal/api/library_sync_views.go +++ b/internal/api/library_sync_views.go @@ -3,8 +3,8 @@ package api // Wire shapes for /api/library/sync upserts. The sqlc-generated row // structs (dbq.Artist, dbq.Album, dbq.Track, dbq.Playlist) have no // JSON tags (sqlc.yaml: emit_json_tags=false), so a raw json.Marshal -// of them produces PascalCase field names that the Flutter sync -// controller (which reads snake_case keys) can't parse. +// of them produces PascalCase field names, and clients read snake_case +// keys. // // These view structs add the JSON tag layer + flatten pgtype.UUID and // pgtype.Timestamptz / Date into strings, mirroring the pattern @@ -50,7 +50,7 @@ type albumSyncView struct { ReleaseDate *string `json:"release_date"` CoverArtPath *string `json:"cover_art_path"` Mbid *string `json:"mbid"` - // AlbumGain and AlbumPeak are the album's ReplayGain 2.0 values (#4997): + // AlbumGain and AlbumPeak are the album's ReplayGain 2.0 values: // dB to the -18 LUFS reference, and a linear peak. Null until every track // on the album is measured. AlbumGain *float32 `json:"album_gain"` @@ -87,20 +87,20 @@ type trackSyncView struct { FilePath string `json:"file_path"` FileFormat string `json:"file_format"` Genre *string `json:"genre"` - // Missing reports that the file is currently absent from disk (#2704). + // Missing reports that the file is currently absent from disk. // // Shipped as state rather than filtered out of the feed, because a // missing file is expected to come back: the scanner clears the mark - // when it does, and adopts the row if it returns under a new name - // (#2528). Dropping the row instead would mean a delete-and-recreate on - // every client for what is often a transient unmount, churning caches - // and discarding the identity #2528 works to preserve. + // when it does, and adopts the row if it returns under a new name. + // Dropping the row instead would mean a delete-and-recreate on every + // client for what is often a transient unmount, churning caches and + // discarding the track's identity. // // A bool rather than the timestamp: clients need it to decide whether a // track is playable, which is a yes/no. The "gone since" clock is an - // operator concern and lives on the admin surface. + // admin concern and lives on the admin surface. Missing bool `json:"missing"` - // TrackGain and TrackPeak are the track's ReplayGain 2.0 values (#4997). + // TrackGain and TrackPeak are the track's ReplayGain 2.0 values. // Null until the track is measured. The album's pair rides on the album // view, since it changes when the album does, not when this track does. TrackGain *float32 `json:"track_gain"` diff --git a/internal/api/library_sync_views_test.go b/internal/api/library_sync_views_test.go index fe627d45..0000d109 100644 --- a/internal/api/library_sync_views_test.go +++ b/internal/api/library_sync_views_test.go @@ -81,9 +81,9 @@ func TestTrackSyncView_WireKeys(t *testing.T) { }) } -// The mark is what the client filters on, so a wrong value here silently -// re-introduces #2704: a present track marked missing vanishes from the -// client's library, a missing one stays playable and fails at the speaker. +// The mark is what the client filters on, so a wrong value here is silent: a +// present track marked missing vanishes from the client's library, a missing +// one stays playable and fails at the speaker. func TestTrackSyncView_MissingReflectsTheMark(t *testing.T) { present := dbq.Track{ID: validUUID, AlbumID: validUUID, ArtistID: validUUID} if toTrackSyncView(present, library.ReplayGain{}).Missing { @@ -97,7 +97,7 @@ func TestTrackSyncView_MissingReflectsTheMark(t *testing.T) { } } -// The gains are what Android levels playback by, offline included (#4997): +// The gains are what Android levels playback by, offline included: // the track's pair rides on the track view and the album's on the album view. func TestSyncViews_CarryReplayGain(t *testing.T) { gain, peak := float32(-3.25), float32(0.9441) diff --git a/internal/api/library_test.go b/internal/api/library_test.go index 7bb8dc83..ec40a475 100644 --- a/internal/api/library_test.go +++ b/internal/api/library_test.go @@ -169,7 +169,8 @@ func TestHandleGetAlbum_NoTracks(t *testing.T) { if len(got.Tracks) != 0 { t.Errorf("tracks len = %d, want 0", len(got.Tracks)) } - // Year omitempty: releaseYear=0 => Year=0 => field should be absent from JSON + // Year omitempty: releaseYear=0 => Year=0 => field should be absent from + // JSON if got.Year != 0 { t.Errorf("year = %d, want 0", got.Year) } @@ -226,7 +227,8 @@ func TestHandleGetArtist_HappyPath(t *testing.T) { if len(got.Albums) != 2 { t.Fatalf("albums len = %d, want 2", len(got.Albums)) } - // ListAlbumsByArtist orders by release_date then sort_title; Abbey Road (1969) first. + // ListAlbumsByArtist orders by release_date then sort_title; Abbey Road + // (1969) first. if got.Albums[0].Title != "Abbey Road" { t.Errorf("first album = %q, want Abbey Road", got.Albums[0].Title) } @@ -465,7 +467,7 @@ func TestRoutesRegisteredInMount(t *testing.T) { r := chi.NewRouter() w := playevents.NewWriter(h.pool, slog.New(slog.NewTextHandler(io.Discard, nil)), 30*time.Minute, 0.5, 30000) - Mount(r, h.pool, h.logger, w, config.RecommendationConfig{RadioSize: 50, RadioSizeMax: 200, RecentlyPlayedHours: 1}, h.recSettings, h.lidarrCfg, h.lidarrRequests, h.lidarrQuarantine, h.tracks, h.playlists, h.coverart, h.coverSettings, h.tagSettings, h.scanner, h.scanCfg, h.dataDir, nil, eventbus.New(), nil, nil, h.netSettings, nil, nil, nil, nil) + Mount(r, h.pool, h.logger, w, config.RecommendationConfig{RadioSize: 50, RadioSizeMax: 200, RecentlyPlayedHours: 1}, h.recSettings, h.requests, h.quarantine, h.tracks, h.playlists, h.coverart, h.coverSettings, h.tagSettings, h.scanner, h.scanCfg, h.dataDir, nil, eventbus.New(), nil, nil, h.netSettings, nil, nil, nil, nil) paths := []string{ "/api/artists", @@ -475,10 +477,10 @@ func TestRoutesRegisteredInMount(t *testing.T) { "/api/tracks/00000000-0000-0000-0000-000000000001", "/api/tracks/00000000-0000-0000-0000-000000000001/stream", "/api/search?q=x", - // Browse indexes (#367). + // Browse indexes. "/api/library/genres", "/api/library/years", - // Admin surfaces for the missing-file lifecycle (#2527). Both must + // Admin surfaces for the missing-file lifecycle. Both must // 401 at the middleware rather than 404 — they live inside the // admin group, so reaching the auth check is what proves they are // wired. @@ -561,7 +563,7 @@ func TestGetArtistTracks_HonorsQuarantine(t *testing.T) { FileSize: 1, FileFormat: "mp3", }) if _, err := pool.Exec(context.Background(), - `INSERT INTO lidarr_quarantine (user_id, track_id, reason) VALUES ($1, $2, 'bad_rip')`, + `INSERT INTO quarantine (user_id, track_id, reason) VALUES ($1, $2, 'bad_rip')`, user.ID, track.ID, ); err != nil { t.Fatalf("quarantine insert: %v", err) diff --git a/internal/api/lidarr.go b/internal/api/lidarr.go deleted file mode 100644 index 59e0dda6..00000000 --- a/internal/api/lidarr.go +++ /dev/null @@ -1,138 +0,0 @@ -package api - -import ( - "context" - "errors" - "net/http" - "strings" - - "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" -) - -// lidarrSearchResult is the JSON shape returned by GET /api/lidarr/search. -// Field names follow the lowercase snake_case convention used by all /api/* -// responses. artist_mbid and album_mbid are the parent identifiers surfaced -// by LookupAlbum / LookupTrack; they are empty strings on artist results. -type lidarrSearchResult struct { - MBID string `json:"mbid"` - Name string `json:"name"` - SecondaryText string `json:"secondary_text"` - ImageURL string `json:"image_url"` - ArtistMBID string `json:"artist_mbid"` - AlbumMBID string `json:"album_mbid"` - InLibrary bool `json:"in_library"` - Requested bool `json:"requested"` -} - -// handleLidarrSearch implements GET /api/lidarr/search?q=&kind=artist|album|track. -// -// Steps: -// 1. Validate kind and q params. -// 2. Load lidarr config; 503 when disabled. -// 3. Call the matching Lidarr lookup. -// 4. For each result, compute in_library (EXISTS by MBID) and requested -// (HasNonTerminalRequestForMBID). -// 5. Return normalized JSON array. -func (h *handlers) handleLidarrSearch(w http.ResponseWriter, r *http.Request) { - kind := strings.TrimSpace(r.URL.Query().Get("kind")) - switch kind { - case "artist", "album", "track": - // valid - default: - writeErr(w, apierror.BadRequest("bad_kind", "kind must be artist, album, or track")) - return - } - - q := strings.TrimSpace(r.URL.Query().Get("q")) - if q == "" { - writeErr(w, apierror.BadRequest("missing_query", "q is required")) - return - } - - cfg, err := h.lidarrCfg.Get(r.Context()) - if err != nil { - h.logger.Error("api: lidarr search: load config", "err", err) - writeErr(w, &apierror.Error{Status: 500, Code: "lidarr_lookup_failed", Message: "failed to load lidarr config", Cause: err}) - return - } - if !cfg.Enabled { - writeErr(w, &apierror.Error{Status: 503, Code: "lidarr_disabled", Message: "Lidarr integration is not enabled"}) - return - } - - client := lidarr.NewClient(cfg.BaseURL, cfg.APIKey) - - var results []lidarr.LookupResult - switch kind { - case "artist": - results, err = client.LookupArtist(r.Context(), q) - case "album": - results, err = client.LookupAlbum(r.Context(), q) - case "track": - results, err = client.LookupTrack(r.Context(), q) - } - if err != nil { - switch { - case errors.Is(err, lidarr.ErrUnreachable): - writeErr(w, &apierror.Error{Status: 503, Code: "lidarr_unreachable", Message: "Lidarr is unreachable", Cause: err}) - case errors.Is(err, lidarr.ErrAuthFailed): - writeErr(w, &apierror.Error{Status: 503, Code: "lidarr_auth_failed", Message: "Lidarr authentication failed", Cause: err}) - default: - h.logger.Error("api: lidarr search: lookup failed", "kind", kind, "err", err) - writeErr(w, &apierror.Error{Status: 500, Code: "lidarr_lookup_failed", Message: "Lidarr lookup failed", Cause: err}) - } - return - } - - dbQ := dbq.New(h.pool) - out := make([]lidarrSearchResult, 0, len(results)) - for _, res := range results { - inLib, libErr := h.lidarrInLibrary(r.Context(), kind, res.MBID) - if libErr != nil { - h.logger.Error("api: lidarr search: in_library check failed", "err", libErr) - writeErr(w, &apierror.Error{Status: 500, Code: "lidarr_lookup_failed", Message: "library check failed", Cause: libErr}) - return - } - requested, reqErr := dbQ.HasNonTerminalRequestForMBID(r.Context(), res.MBID) - if reqErr != nil { - h.logger.Error("api: lidarr search: requested check failed", "err", reqErr) - writeErr(w, &apierror.Error{Status: 500, Code: "lidarr_lookup_failed", Message: "request check failed", Cause: reqErr}) - return - } - out = append(out, lidarrSearchResult{ - MBID: res.MBID, - Name: res.Name, - SecondaryText: res.Secondary, - ImageURL: res.ImageURL, - ArtistMBID: res.ArtistMBID, - AlbumMBID: res.AlbumMBID, - InLibrary: inLib, - Requested: requested, - }) - } - writeJSON(w, http.StatusOK, out) -} - -// lidarrInLibrary checks whether the given MBID exists in the corresponding -// local library table based on kind. Runs a direct EXISTS query since -// these per-kind MBID lookups are not in the sqlc-generated query set. -func (h *handlers) lidarrInLibrary(ctx context.Context, kind, mbid string) (bool, error) { - var query string - switch kind { - case "artist": - query = `SELECT EXISTS(SELECT 1 FROM artists WHERE mbid = $1)` - case "album": - query = `SELECT EXISTS(SELECT 1 FROM albums WHERE mbid = $1)` - case "track": - query = `SELECT EXISTS(SELECT 1 FROM tracks WHERE mbid = $1)` - default: - return false, nil - } - var exists bool - if err := h.pool.QueryRow(ctx, query, mbid).Scan(&exists); err != nil { - return false, err - } - return exists, nil -} diff --git a/internal/api/lidarr_test.go b/internal/api/lidarr_test.go deleted file mode 100644 index d800292e..00000000 --- a/internal/api/lidarr_test.go +++ /dev/null @@ -1,327 +0,0 @@ -package api - -import ( - "context" - "encoding/json" - "fmt" - "net/http" - "net/http/httptest" - "testing" - "time" - - "github.com/jackc/pgx/v5/pgtype" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" -) - -// resetLidarrState clears lidarr_requests rows and resets lidarr_config -// to the factory-default state (enabled=false, nulls) between tests. -func resetLidarrState(t *testing.T, h *handlers) { - t.Helper() - ctx := context.Background() - if _, err := h.pool.Exec(ctx, - "DELETE FROM lidarr_requests; UPDATE lidarr_config SET enabled=false, base_url=NULL, api_key=NULL WHERE id=1", - ); err != nil { - t.Fatalf("resetLidarrState: %v", err) - } -} - -// saveLidarrConfig writes a test lidarr_config row pointing at the given stub URL. -func saveLidarrConfig(t *testing.T, h *handlers, baseURL string, enabled bool) { - t.Helper() - svc := lidarrconfig.New(h.pool) - err := svc.Save(context.Background(), lidarrconfig.Config{ - Enabled: enabled, - BaseURL: baseURL, - APIKey: "test-key", - DefaultQualityProfileID: 1, - DefaultRootFolderPath: "/music", - }) - if err != nil { - t.Fatalf("saveLidarrConfig: %v", err) - } -} - -// lidarrArtistStubBody returns a minimal Lidarr artist lookup JSON payload. -func lidarrArtistStubBody(mbid, name string) string { - return fmt.Sprintf(`[{"foreignArtistId":%q,"artistName":%q,"genres":["Rock"],"albumCount":5,"images":[]}]`, mbid, name) -} - -// newLidarrStub creates an httptest.Server that always returns the given body -// with 200 OK for any /api/v1/* path. -func newLidarrStub(t *testing.T, body string) *httptest.Server { - t.Helper() - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte(body)) - })) - t.Cleanup(srv.Close) - return srv -} - -// seedArtistWithMBID inserts an artist row with a specific MBID for in_library testing. -func seedArtistWithMBID(t *testing.T, h *handlers, name, mbid string) dbq.Artist { - t.Helper() - a, err := dbq.New(h.pool).UpsertArtist(context.Background(), dbq.UpsertArtistParams{ - Name: name, - SortName: name, - Mbid: &mbid, - }) - if err != nil { - t.Fatalf("seedArtistWithMBID: %v", err) - } - return a -} - -// seedRequestForMBID inserts a pending lidarr_requests row for the given MBID. -func seedRequestForMBID(t *testing.T, h *handlers, userID pgtype.UUID, kind dbq.LidarrRequestKind, artistMBID, mbid string) { - t.Helper() - var albumMBID, trackMBID *string - switch kind { - case dbq.LidarrRequestKindAlbum: - albumMBID = &mbid - case dbq.LidarrRequestKindTrack: - trackMBID = &mbid - } - _, err := dbq.New(h.pool).CreateLidarrRequest(context.Background(), dbq.CreateLidarrRequestParams{ - UserID: userID, - Kind: kind, - LidarrArtistMbid: artistMBID, - LidarrAlbumMbid: albumMBID, - LidarrTrackMbid: trackMBID, - ArtistName: "Test Artist", - AlbumTitle: nil, - TrackTitle: nil, - }) - if err != nil { - t.Fatalf("seedRequestForMBID: %v", err) - } -} - -// doLidarrSearch fires a GET /api/lidarr/search request directly at the handler. -// It simulates the RequireUser middleware by injecting a dummy user context. -func doLidarrSearch(t *testing.T, h *handlers, q, kind string) *httptest.ResponseRecorder { - t.Helper() - url := fmt.Sprintf("/api/lidarr/search?q=%s&kind=%s", q, kind) - req := httptest.NewRequest(http.MethodGet, url, nil) - // Inject a minimal user context the same way other handler tests do it. - // The lidarr search handler doesn't use the user from context, so any - // non-nil user value keeps the handler from panicking. - w := httptest.NewRecorder() - h.handleLidarrSearch(w, req) - return w -} - -// TestHandleLidarrSearch_ValidationErrors covers bad kind and missing query -// in a table-driven test. -func TestHandleLidarrSearch_ValidationErrors(t *testing.T) { - h, _ := testHandlers(t) - - cases := []struct { - name string - q string - kind string - wantStatus int - wantCode string - }{ - { - name: "bad kind playlist", - q: "Beatles", - kind: "playlist", - wantStatus: http.StatusBadRequest, - wantCode: "bad_kind", - }, - { - name: "missing query", - q: "", - kind: "artist", - wantStatus: http.StatusBadRequest, - wantCode: "missing_query", - }, - } - - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - url := fmt.Sprintf("/api/lidarr/search?q=%s&kind=%s", tc.q, tc.kind) - req := httptest.NewRequest(http.MethodGet, url, nil) - w := httptest.NewRecorder() - h.handleLidarrSearch(w, req) - - if w.Code != tc.wantStatus { - t.Fatalf("status = %d, want %d; body = %s", w.Code, tc.wantStatus, w.Body.String()) - } - var resp struct { - Error struct { - Code string `json:"code"` - } `json:"error"` - } - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode body: %v; body=%s", err, w.Body.String()) - } - if resp.Error.Code != tc.wantCode { - t.Errorf("error.code = %q, want %q", resp.Error.Code, tc.wantCode) - } - }) - } -} - -// TestHandleLidarrSearch_DisabledReturns503 verifies that a disabled Lidarr -// config causes the handler to return 503 lidarr_disabled. -func TestHandleLidarrSearch_DisabledReturns503(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - - // Config is already disabled after resetLidarrState; ensure it's explicit. - saveLidarrConfig(t, h, "http://127.0.0.1:1", false) - - req := httptest.NewRequest(http.MethodGet, "/api/lidarr/search?q=Beatles&kind=artist", nil) - w := httptest.NewRecorder() - h.handleLidarrSearch(w, req) - - if w.Code != http.StatusServiceUnavailable { - t.Fatalf("status = %d, want 503; body = %s", w.Code, w.Body.String()) - } - var resp struct { - Error struct { - Code string `json:"code"` - } `json:"error"` - } - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode body: %v; body=%s", err, w.Body.String()) - } - if resp.Error.Code != "lidarr_disabled" { - t.Errorf("error.code = %q, want lidarr_disabled", resp.Error.Code) - } -} - -// TestHandleLidarrSearch_HappyPath_RequestableResult checks a result that is -// neither in the library nor already requested. -func TestHandleLidarrSearch_HappyPath_RequestableResult(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - - const mbid = "artist-mbid-abc123" - stub := newLidarrStub(t, lidarrArtistStubBody(mbid, "The Beatles")) - saveLidarrConfig(t, h, stub.URL, true) - - w := doLidarrSearch(t, h, "Beatles", "artist") - - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var results []lidarrSearchResult - if err := json.Unmarshal(w.Body.Bytes(), &results); err != nil { - t.Fatalf("decode: %v; body=%s", err, w.Body.String()) - } - if len(results) != 1 { - t.Fatalf("len(results) = %d, want 1", len(results)) - } - r := results[0] - if r.MBID != mbid { - t.Errorf("mbid = %q, want %q", r.MBID, mbid) - } - if r.InLibrary { - t.Errorf("in_library = true, want false") - } - if r.Requested { - t.Errorf("requested = true, want false") - } -} - -// TestHandleLidarrSearch_HappyPath_InLibraryResult checks that a result whose -// MBID matches an artist in the local DB has in_library=true. -func TestHandleLidarrSearch_HappyPath_InLibraryResult(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - - const mbid = "artist-mbid-inlib" - stub := newLidarrStub(t, lidarrArtistStubBody(mbid, "Library Artist")) - saveLidarrConfig(t, h, stub.URL, true) - - // Seed the artist with the matching MBID so in_library becomes true. - seedArtistWithMBID(t, h, dbtest.TestUserPrefix+"LibraryArtist", mbid) - - w := doLidarrSearch(t, h, "Library+Artist", "artist") - - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var results []lidarrSearchResult - if err := json.Unmarshal(w.Body.Bytes(), &results); err != nil { - t.Fatalf("decode: %v; body=%s", err, w.Body.String()) - } - if len(results) != 1 { - t.Fatalf("len(results) = %d, want 1", len(results)) - } - if !results[0].InLibrary { - t.Errorf("in_library = false, want true") - } -} - -// TestHandleLidarrSearch_HappyPath_AlreadyRequested verifies that a result -// whose MBID matches a pending lidarr_requests row has requested=true. -func TestHandleLidarrSearch_HappyPath_AlreadyRequested(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - - const mbid = "artist-mbid-requested" - stub := newLidarrStub(t, lidarrArtistStubBody(mbid, "Requested Artist")) - saveLidarrConfig(t, h, stub.URL, true) - - // Seed a user and a pending request for the artist MBID. - user := seedUser(t, h.pool, "requester", "pw", false) - seedRequestForMBID(t, h, user.ID, dbq.LidarrRequestKindArtist, mbid, mbid) - - w := doLidarrSearch(t, h, "Requested+Artist", "artist") - - if w.Code != http.StatusOK { - t.Fatalf("status = %d, want 200; body = %s", w.Code, w.Body.String()) - } - var results []lidarrSearchResult - if err := json.Unmarshal(w.Body.Bytes(), &results); err != nil { - t.Fatalf("decode: %v; body=%s", err, w.Body.String()) - } - if len(results) != 1 { - t.Fatalf("len(results) = %d, want 1", len(results)) - } - if !results[0].Requested { - t.Errorf("requested = false, want true") - } -} - -// TestHandleLidarrSearch_LidarrUnreachableReturns503 verifies that a -// connection-refused Lidarr base URL returns 503 lidarr_unreachable. -func TestHandleLidarrSearch_LidarrUnreachableReturns503(t *testing.T) { - h, _ := testHandlers(t) - resetLidarrState(t, h) - - // Port 1 is reserved and will refuse connections immediately. - saveLidarrConfig(t, h, "http://127.0.0.1:1", true) - - req := httptest.NewRequest(http.MethodGet, "/api/lidarr/search?q=Beatles&kind=artist", nil) - // Use a short-deadline context so the test doesn't hang waiting for TCP. - ctx, cancel := context.WithTimeout(req.Context(), 3*time.Second) - defer cancel() - req = req.WithContext(ctx) - - w := httptest.NewRecorder() - h.handleLidarrSearch(w, req) - - if w.Code != http.StatusServiceUnavailable { - t.Fatalf("status = %d, want 503; body = %s", w.Code, w.Body.String()) - } - var resp struct { - Error struct { - Code string `json:"code"` - } `json:"error"` - } - if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil { - t.Fatalf("decode: %v; body=%s", err, w.Body.String()) - } - if resp.Error.Code != "lidarr_unreachable" { - t.Errorf("error.code = %q, want lidarr_unreachable", resp.Error.Code) - } -} diff --git a/internal/api/likes.go b/internal/api/likes.go index aad2a663..955ea06a 100644 --- a/internal/api/likes.go +++ b/internal/api/likes.go @@ -16,9 +16,9 @@ import ( // logLikeChange writes a library_changes row for a like / unlike. Best- // effort: a failure here means the next sync may miss the row, but the -// HTTP response still succeeds. Cheap consequence — operator's other -// device will see the like once it manually pulls or once a future -// like/unlike on the same entity emits a fresh log row. +// HTTP response still succeeds. Cheap consequence — the user's other +// device sees the like once it manually pulls, or once a later like/unlike +// on the same entity emits a fresh log row. func (h *handlers) logLikeChange(r *http.Request, et syncpkg.EntityType, userID, entityID pgtype.UUID, op syncpkg.Op) { if err := syncpkg.LogChange(r.Context(), h.pool, et, syncpkg.EncodeLikeID(syncpkg.FormatUUID(userID), syncpkg.FormatUUID(entityID)), op); err != nil { @@ -60,7 +60,7 @@ func (h *handlers) handleLikeTrack(w http.ResponseWriter, r *http.Request) { if slices.Contains(liked, id) { _ = playevents.CaptureContextualLikeIfPlaying(r.Context(), q, user.ID, id, h.logger) } - // The like is on the song (M498 #5438): every copy it reached changes on + // The like is on the song: every copy it reached changes on // the client too. for _, t := range withTrack(id, liked) { h.logLikeChange(r, syncpkg.EntityLikeTrack, user.ID, t, syncpkg.OpUpsert) diff --git a/internal/api/me.go b/internal/api/me.go index bc33cd8f..231626ac 100644 --- a/internal/api/me.go +++ b/internal/api/me.go @@ -10,20 +10,14 @@ import ( // handleGetMe returns the authenticated user's profile shape — id, // username, display_name, email, is_admin. Mirrors the response shape -// of PUT /api/me/profile so the Android Settings → Profile screen -// (and Flutter's equivalent) can read its starting state from a GET -// before the user has done a first save. +// of PUT /api/me/profile so the Android Settings → Profile screen can +// read its starting state from a GET before the user has done a first +// save. // -// Drift #578 fix: this previously emitted the narrower UserView -// (id, username, is_admin only). Android's MeApi.getProfile() deser- -// ialised into MyProfileWire and saw display_name=null, email=null -// on every read — wiping the form fields on every Settings open. -// Saving from that blank state then submitted empty strings, which -// handleUpdateMyProfile treats as "clear to NULL" — DESTROYING the -// user's stored display_name + email. Returning the full profile -// shape here closes the loop. Web (User type narrower than this -// response) keeps working via TypeScript structural typing — the -// extra fields are ignored. +// The full profile matters: a form seeded with display_name=null and +// email=null would, on save, submit empty strings, which +// handleUpdateMyProfile treats as "clear to NULL" — destroying the stored +// values. The web app's narrower User type ignores the extra fields. func (h *handlers) handleGetMe(w http.ResponseWriter, r *http.Request) { user, ok := auth.UserFromContext(r.Context()) if !ok { diff --git a/internal/api/me_history_test.go b/internal/api/me_history_test.go index 96e224c0..57bcfeb7 100644 --- a/internal/api/me_history_test.go +++ b/internal/api/me_history_test.go @@ -65,7 +65,7 @@ func seedPlaySession(t *testing.T, pool *pgxpool.Pool, userID pgtype.UUID, start func quarantineTrack(t *testing.T, pool *pgxpool.Pool, userID, trackID pgtype.UUID) { t.Helper() _, err := pool.Exec(context.Background(), - `INSERT INTO lidarr_quarantine (user_id, track_id, reason) VALUES ($1, $2, 'bad_rip')`, + `INSERT INTO quarantine (user_id, track_id, reason) VALUES ($1, $2, 'bad_rip')`, userID, trackID, ) if err != nil { diff --git a/internal/api/me_normalization.go b/internal/api/me_normalization.go index 4b41b9fe..0b1f9061 100644 --- a/internal/api/me_normalization.go +++ b/internal/api/me_normalization.go @@ -11,7 +11,7 @@ import ( ) // GET /api/me/normalization returns the caller's loudness-normalization -// preference, or the defaults when they have never set one (M464 #4998). +// preference, or the defaults when they have never set one. func (h *handlers) handleGetMyNormalization(w http.ResponseWriter, r *http.Request) { caller, ok := requireUser(w, r) if !ok { diff --git a/internal/api/me_notifications.go b/internal/api/me_notifications.go index 8c7ae485..48d078e5 100644 --- a/internal/api/me_notifications.go +++ b/internal/api/me_notifications.go @@ -18,7 +18,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" ) -// The notifications inbox (M489). Rows are written by internal/notifications; +// The notifications inbox. Rows are written by internal/notifications; // this surface lists them, counts the unread, marks them read, and holds each // user's per-kind settings. @@ -125,8 +125,8 @@ func (h *handlers) handleListMyNotifications(w http.ResponseWriter, r *http.Requ writeJSON(w, http.StatusOK, out) } -// handleMyUnreadNotificationCount implements GET /api/me/notifications/unread-count, -// the cheap call behind the badge. +// handleMyUnreadNotificationCount implements GET +// /api/me/notifications/unread-count, the cheap call behind the badge. func (h *handlers) handleMyUnreadNotificationCount(w http.ResponseWriter, r *http.Request) { user, ok := requireUser(w, r) if !ok { @@ -165,7 +165,8 @@ func (h *handlers) handleMarkMyNotificationRead(w http.ResponseWriter, r *http.R w.WriteHeader(http.StatusNoContent) } -// handleMarkAllMyNotificationsRead implements POST /api/me/notifications/read-all. +// handleMarkAllMyNotificationsRead implements POST +// /api/me/notifications/read-all. func (h *handlers) handleMarkAllMyNotificationsRead(w http.ResponseWriter, r *http.Request) { user, ok := requireUser(w, r) if !ok { diff --git a/internal/api/me_profile.go b/internal/api/me_profile.go index ce5f64cf..aa7e021b 100644 --- a/internal/api/me_profile.go +++ b/internal/api/me_profile.go @@ -23,7 +23,7 @@ type updateProfileReq struct { // nil = leave unchanged; non-nil empty string = clear; non-nil // non-empty = update to that value. Note: distinguishing // omit-vs-clear at JSON-decode time requires custom unmarshaling - // or a sentinel; for v1 we use the simpler "empty string clears" + // or a sentinel; we use the simpler "empty string clears" // semantics: any field present in the JSON body is applied. DisplayName *string `json:"display_name"` Email *string `json:"email"` @@ -37,7 +37,7 @@ type meProfileResp struct { IsAdmin bool `json:"is_admin"` // DebugModeEnabled is the account's diagnostics opt-in. The client // reads it here to decide whether to run the on-device diagnostics - // reporter (M9). Admin-set; the client also has a local OFF switch. + // reporter. Admin-set; the client also has a local OFF switch. DebugModeEnabled bool `json:"debug_mode_enabled"` } diff --git a/internal/api/me_recommendation_metrics.go b/internal/api/me_recommendation_metrics.go index 7716d9f3..ae2a796e 100644 --- a/internal/api/me_recommendation_metrics.go +++ b/internal/api/me_recommendation_metrics.go @@ -21,7 +21,7 @@ const ( recMetricsLowVolume = 20 ) -// Surface intents (milestone #127): each family is judged against its +// Surface intents: each family is judged against its // job, not one global bar — discovery mixes are EXPECTED to run higher // skip rates than the go-to surfaces. const ( @@ -40,15 +40,15 @@ type surfaceMetric struct { AvgCompletion float64 `json:"avg_completion"` // mean completion ratio, [0,1] LowConfidence bool `json:"low_confidence"` // plays < recMetricsLowVolume // SkipDelta / CompletionDelta are this row's difference from the manual - // baseline WITH its margin of error (#2495). nil on the baseline row + // baseline WITH its margin of error. nil on the baseline row // itself, and whenever the samples are too thin for a margin to mean // anything. Computed server-side so both clients read the same arithmetic - // instead of each re-deriving it — and so `low_confidence` is no longer - // mistaken for a decision threshold, which it never was. + // instead of each re-deriving it. `low_confidence` is a display + // threshold, not a decision one. SkipDelta *metricDelta `json:"skip_delta,omitempty"` CompletionDelta *metricDelta `json:"completion_delta,omitempty"` // Breakdown splits the family into the pick-kind populations its - // builder stamped (#1249, generalized #1270): For You's taste/fresh, + // builder stamped: For You's taste/fresh, // Discover's buckets, tier1-3 for tiered mixes — plus earlier plays // that predate attribution. Present only when the family has at // least one attributed play; the parent row remains the sum of its @@ -128,7 +128,7 @@ type familyAccum struct { // reduces to a single weighted division at the end. completionSum float64 // completionSqSum is the sum of squared completion ratios, which is what - // makes the variance mergeable across raw source rows (#2495). Standard + // makes the variance mergeable across raw source rows. Standard // deviations cannot be combined; sums of squares add exactly. completionSqSum float64 } diff --git a/internal/api/me_recommendation_metrics_test.go b/internal/api/me_recommendation_metrics_test.go index a5a6946e..4199fb5b 100644 --- a/internal/api/me_recommendation_metrics_test.go +++ b/internal/api/me_recommendation_metrics_test.go @@ -23,7 +23,7 @@ func newMetricsRouter(h *handlers) chi.Router { // seedSourcedPlay inserts a play_event with an explicit source + completion + // skip flag. A nil source inserts NULL (manual library play → baseline). -// pickKind is the For You taste/fresh attribution (#1249); nil everywhere +// pickKind is the For You taste/fresh attribution; nil everywhere // except attributed for_you plays. func seedSourcedPlay( t *testing.T, h *handlers, userID, trackID, sessionID pgtype.UUID, @@ -169,7 +169,7 @@ func TestBucketMetricsResponse_ForYouBreakdown(t *testing.T) { } func TestBucketMetricsResponse_DiscoverBucketBreakdown(t *testing.T) { - // Provenance is standard (#1270): Discover's bucket stamps surface as + // Provenance is standard: Discover's bucket stamps surface as // a breakdown exactly like For You's taste/fresh — this is what makes // the bucket allocation judgeable instead of a guess. src := func(s string) *string { return &s } @@ -225,7 +225,7 @@ func TestRecommendationMetrics_BucketsWithBaseline(t *testing.T) { radioA := "radio:11111111-1111-1111-1111-111111111111" taste, fresh := "taste", "fresh" // for_you: 3 plays, 1 skipped; completions 1.0, 0.95, 0.05 → mean 0.6667. - // Two attributed as taste picks, the skipped one as a fresh pick (#1249). + // Two attributed as taste picks, the skipped one as a fresh pick. seedSourcedPlay(t, h, user.ID, tk.ID, session, &forYou, &taste, 1.0, false) seedSourcedPlay(t, h, user.ID, tk.ID, session, &forYou, &taste, 0.95, false) seedSourcedPlay(t, h, user.ID, tk.ID, session, &forYou, &fresh, 0.05, true) @@ -265,7 +265,7 @@ func TestRecommendationMetrics_BucketsWithBaseline(t *testing.T) { if fy.AvgCompletion < 0.66 || fy.AvgCompletion > 0.67 { t.Errorf("for_you avg_completion = %.4f, want ~0.6667", fy.AvgCompletion) } - // Pick-kind attribution surfaces as the For You breakdown (#1249). + // Pick-kind attribution surfaces as the For You breakdown. if len(fy.Breakdown) != 2 { t.Fatalf("for_you breakdown rows = %d, want 2 (taste, fresh)", len(fy.Breakdown)) } diff --git a/internal/api/me_sessions_test.go b/internal/api/me_sessions_test.go index 8f9c3d6d..5473e608 100644 --- a/internal/api/me_sessions_test.go +++ b/internal/api/me_sessions_test.go @@ -49,8 +49,8 @@ func withURLParam(r *http.Request, key, value string) *http.Request { return r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, rctx)) } -// The rule #47 assertion. A delete keyed only on session id would let any -// household member revoke any other member's session by id — this pins that +// Household data stays per-user: a delete keyed only on session id would let +// any household member revoke any other member's session by id — this pins that // the user scope is actually in the WHERE clause and not just intended. func TestRevokeMySession_CannotRevokeAnotherUsersSession(t *testing.T) { h, pool := testHandlers(t) diff --git a/internal/api/me_subsonic_password.go b/internal/api/me_subsonic_password.go index 62143d33..6a0009fe 100644 --- a/internal/api/me_subsonic_password.go +++ b/internal/api/me_subsonic_password.go @@ -15,8 +15,7 @@ import ( // itself, so it is stored readable (migration 0003). That is why Minstrel // generates it rather than letting the user choose one: a generated value // can never be a login password reused from somewhere else, so a leaked -// users table gives up access to this server's /rest API and nothing more -// (M462 #5026). +// users table gives up access to this server's /rest API and nothing more. const subsonicPasswordBytes = 18 // 24 base64url characters diff --git a/internal/api/me_system_playlists.go b/internal/api/me_system_playlists.go index 91903927..a4c3cde4 100644 --- a/internal/api/me_system_playlists.go +++ b/internal/api/me_system_playlists.go @@ -19,9 +19,10 @@ type systemPlaylistsStatusResp struct { LastError *string `json:"last_error"` } -// handleGetSystemPlaylistsStatus implements GET /api/me/system-playlists-status. -// Returns the caller's system_playlist_runs row, or zero-values when no row -// exists yet (i.e. the user has never had a build attempted). +// handleGetSystemPlaylistsStatus implements GET +// /api/me/system-playlists-status. Returns the caller's system_playlist_runs +// row, or zero-values when no row exists yet (i.e. the user has never had a +// build attempted). func (h *handlers) handleGetSystemPlaylistsStatus(w http.ResponseWriter, r *http.Request) { caller, ok := requireUser(w, r) if !ok { diff --git a/internal/api/me_test.go b/internal/api/me_test.go index a9515e6c..680b9827 100644 --- a/internal/api/me_test.go +++ b/internal/api/me_test.go @@ -37,13 +37,10 @@ func TestHandleGetMe_ReturnsAuthenticatedUser(t *testing.T) { } } -// Drift #578 regression guard: a user whose profile fields ARE set -// must see them in the /api/me response. Previously this handler -// emitted the narrower UserView so Android Settings → Profile saw -// display_name=null and email=null on every read, wiped the form -// fields, and submitting from that blank state destroyed the user's -// stored values via the "empty string clears to NULL" semantics of -// PUT /api/me/profile. +// A user whose profile fields ARE set must see them in the /api/me +// response: a Settings → Profile form seeded without them would, on save, +// clear the stored values through PUT /api/me/profile's "empty string +// clears to NULL". func TestHandleGetMe_IncludesDisplayNameAndEmail(t *testing.T) { h, pool := testHandlers(t) user := seedUser(t, pool, "alice", "hunter2", false) diff --git a/internal/api/me_timezone.go b/internal/api/me_timezone.go index 60cfe06e..edd316c8 100644 --- a/internal/api/me_timezone.go +++ b/internal/api/me_timezone.go @@ -1,11 +1,8 @@ // PUT /api/me/timezone — accept the client's IANA timezone and store -// it on the user row. The system-playlist scheduler (#392 Half B) -// reads this column to fire each user's daily 03:00 build in their -// local time. -// -// Scheduler.Refresh(ctx, userID) is invoked in a later commit (Task 3) -// so a timezone change takes effect synchronously rather than waiting -// for the hourly reconciliation pass. +// it on the user row. The system-playlist scheduler reads this column +// to fire each user's daily 03:00 build in their local time. The +// handler calls Scheduler.Refresh so a change takes effect at once rather +// than at the hourly reconciliation pass. package api diff --git a/internal/api/media.go b/internal/api/media.go index a4009a44..4a577a05 100644 --- a/internal/api/media.go +++ b/internal/api/media.go @@ -125,17 +125,14 @@ func (h *handlers) handleGetCover(w http.ResponseWriter, r *http.Request) { // stream — either via the standard session path (cookie OR bearer token // resolved to a user-in-context by auth.OptionalUser, the middleware // the route is wrapped with in api.go) OR via a valid short-lived HMAC -// token in the ?token=&exp= query (UPnP / Sonos path, new for the -// output-picker UPnP slice). +// token in the ?token=&exp= query (the UPnP / Sonos path). // // The token path lets network speakers fetch the stream URL without -// carrying the user's session cookie — they cannot. See the design at -// docs/superpowers/specs/2026-06-03-android-output-picker-upnp-design.md. +// carrying the user's session cookie — they cannot. // -// When h.streamSecret is nil (slice-1 default, until slice 2 wires the -// loader), the token path always rejects because the HMAC of an empty -// key won't match any token a client could mint. The session path keeps -// working. +// When h.streamSecret is nil the token path always rejects, because the +// HMAC of an empty key won't match any token a client could mint. The +// session path keeps working. func (h *handlers) streamAuthOk(r *http.Request, trackID string) bool { if _, ok := auth.UserFromContext(r.Context()); ok { return true @@ -194,9 +191,9 @@ func (h *handlers) handleGetStream(w http.ResponseWriter, r *http.Request) { w.Header().Set("Accept-Ranges", "bytes") // Track bytes are immutable for a given track id (the scanner // indexes by file_path; re-encoded files take new ids). One-year - // max-age + immutable lets the client cache (LockCachingAudioSource - // on the Flutter side, browser cache on web) skip even the - // conditional GET on repeat plays. + // max-age + immutable lets the client cache (Media3's cache on Android, + // the browser cache on web) skip even the conditional GET on repeat + // plays. w.Header().Set("Cache-Control", "private, max-age=31536000, immutable") http.ServeContent(w, r, filepath.Base(track.FilePath), info.ModTime(), f) } diff --git a/internal/api/metrics_significance.go b/internal/api/metrics_significance.go index 4bc1d048..fdfcfdf7 100644 --- a/internal/api/metrics_significance.go +++ b/internal/api/metrics_significance.go @@ -2,27 +2,19 @@ package api import "math" -// Uncertainty on the deltas the recommendation-metrics card shows (#2495). +// Uncertainty on the deltas the recommendation-metrics card shows. // -// Why this exists: the card had exactly one volume threshold, -// recMetricsLowVolume = 20, and it was doing two jobs. Twenty plays is enough to -// be worth DISPLAYING — below that a skip rate is anecdote — but it is nowhere -// near enough to ACT on. Detecting the ~13pp differences that actually matter -// needs roughly 133 plays per arm for 80% power at α=0.05. +// recMetricsLowVolume = 20 answers "is this worth DISPLAYING" — below that a +// skip rate is anecdote — but twenty plays is nowhere near enough to ACT on. +// Detecting the ~13pp differences that actually matter needs roughly 133 +// plays per arm for 80% power at α=0.05, and two arms of 60-70 plays each +// can look like a bold difference while sitting at p ≈ 0.06. // -// So Discover's taste-matched (59 plays) and random-unheard (70) both rendered as -// full-confidence rows with a bold delta beside them, and that comparison sits at -// p ≈ 0.06. The card said "signal"; the arithmetic said "maybe". It led directly -// to a recommendation the data didn't support, and any reader with the same -// numbers would have made the same call. -// -// The fix is to publish the margin of error next to the delta and flag when the -// delta is smaller than it — i.e. not distinguishable from zero. Computed here, -// server-side, so both clients agree rather than each re-deriving it. -// -// recMetricsLowVolume stays exactly as it was. This is a second, independent -// signal, not a replacement: "too thin to show" and "too thin to act on" are -// different questions and deserve different answers. +// So each delta carries its margin of error, and a flag when the delta is +// smaller than it — not distinguishable from zero. Computed here, +// server-side, so both clients agree rather than each re-deriving it. "Too +// thin to show" and "too thin to act on" are different questions and get +// different answers. // deltaZ is the two-sided 95% normal critical value. Normal rather than // Student's t: at the sample sizes where a delta is worth acting on (n in the @@ -50,7 +42,8 @@ type metricDelta struct { // proportionDelta compares two rates (skips/plays) as a two-proportion // difference. Returns nil when either sample is empty, or when either rate is // degenerate (0 or 1) — a rate with no observed variation has an SE of 0 on its -// side, which would report a spuriously narrow margin rather than an honest one. +// side, which would report a spuriously narrow margin rather than an honest +// one. func proportionDelta(rate1 float64, n1 int64, rate2 float64, n2 int64) *metricDelta { if n1 <= 0 || n2 <= 0 { return nil @@ -98,8 +91,8 @@ func newDelta(deltaPP, marginPP float64) *metricDelta { } // sampleVariance recovers the sample variance from the aggregates the SQL -// returns. sum is mean×n rather than a selected column, which keeps the query to -// one extra expression. +// returns. sum is mean×n rather than a selected column, which keeps the query +// to one extra expression. // // The subtraction can go very slightly negative through floating-point // cancellation when every observation is identical, so the result is clamped — diff --git a/internal/api/metrics_significance_test.go b/internal/api/metrics_significance_test.go index c8e39ca8..f07e33f9 100644 --- a/internal/api/metrics_significance_test.go +++ b/internal/api/metrics_significance_test.go @@ -6,9 +6,8 @@ import ( ) func TestProportionDelta_ReproducesTheDiscoverCase(t *testing.T) { - // The comparison that motivated #2495: Discover taste-matched (59 plays, - // 15.3% skip) vs random-unheard (70 plays, 28.6%). A 13.3pp gap that the old - // card rendered as a confident coloured number, sitting at p ≈ 0.06. + // 59 plays at 15.3% skip vs 70 at 28.6%: a 13.3pp gap that looks + // confident and sits at p ≈ 0.06. d := proportionDelta(0.153, 59, 0.286, 70) if d == nil { t.Fatal("expected a delta for two real samples") @@ -16,8 +15,8 @@ func TestProportionDelta_ReproducesTheDiscoverCase(t *testing.T) { if math.Abs(d.DeltaPP-(-13.3)) > 0.1 { t.Errorf("DeltaPP = %.2f, want ≈ -13.3", d.DeltaPP) } - // This is the assertion the whole task exists for: at these sample sizes the - // margin swallows the difference. + // This is the assertion the whole task exists for: at these sample sizes + // the margin swallows the difference. if d.Distinguishable { t.Errorf("13.3pp on n=59/70 reported as distinguishable (margin %.2f) — "+ "this is exactly the false confidence #2495 set out to remove", d.MarginPP) @@ -70,7 +69,8 @@ func TestProportionDelta_DegenerateRates(t *testing.T) { if d := proportionDelta(1, 50, 1, 50); d != nil { t.Errorf("both rates 1 produced a delta: %+v", d) } - // One degenerate side is still informative — the other side carries variance. + // One degenerate side is still informative — the other side carries + // variance. if d := proportionDelta(0, 200, 0.3, 200); d == nil { t.Error("one degenerate rate should still yield a delta") } diff --git a/internal/api/notify_producers.go b/internal/api/notify_producers.go index e690f581..3cd2e5fc 100644 --- a/internal/api/notify_producers.go +++ b/internal/api/notify_producers.go @@ -6,11 +6,12 @@ import ( "github.com/jackc/pgx/v5/pgtype" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" + "git.fabledsword.com/bvandeusen/minstrel/internal/library" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" + "git.fabledsword.com/bvandeusen/minstrel/internal/requests" ) -// Helpers for the handlers that produce notifications (M489). Every call is +// Helpers for the handlers that produce notifications. Every call is // NotifyLogged: a notification never fails the action that caused it. // userLabel is how a user is named to someone else: their display name, or @@ -22,17 +23,17 @@ func userLabel(u dbq.User) string { return u.Username } -func requestPayload(row dbq.LidarrRequest) notifications.Payload { +func requestPayload(row dbq.Request) notifications.Payload { return notifications.Payload{ RequestID: uuidToString(row.ID), RequestKind: string(row.Kind), - Name: lidarrrequests.DisplayName(row), + Name: requests.DisplayName(row), } } // notifyRequestDecided tells the requester an admin approved or declined // their request. An admin deciding their own request needs no notice. -func (h *handlers) notifyRequestDecided(ctx context.Context, kind notifications.Kind, admin dbq.User, row dbq.LidarrRequest, reason string) { +func (h *handlers) notifyRequestDecided(ctx context.Context, kind notifications.Kind, admin dbq.User, row dbq.Request, reason string) { if row.UserID == admin.ID { return } @@ -66,21 +67,19 @@ func (h *handlers) trackLabel(ctx context.Context, trackID pgtype.UUID) string { if err != nil { return "" } - if a, aerr := q.GetArtistByID(ctx, t.ArtistID); aerr == nil && a.Name != "" { - return a.Name + " – " + t.Title + if a, aerr := q.GetArtistByID(ctx, t.ArtistID); aerr == nil { + return library.TrackLabel(a.Name, t.Title) } return t.Title } -// notifyPlaybackErrors tells admins how many playback errors await review. -// The notice coalesces, so while it is unread a new report only updates the -// count. -func (h *handlers) notifyPlaybackErrors(ctx context.Context) { - n, err := dbq.New(h.pool).CountUnresolvedPlaybackErrors(ctx) - if err != nil { - h.logger.Warn("api: playback_error: count unresolved", "err", err) - return +// notifyPlaybackErrors tells admins a track failed to play, naming it and who +// it failed for. The notice coalesces: while it is unread each report +// adds one to its count and its name to the front of the list. +func (h *handlers) notifyPlaybackErrors(ctx context.Context, trackID pgtype.UUID, user dbq.User) { + p := notifications.Payload{Count: 1} + if name := h.trackLabel(ctx, trackID); name != "" { + p.Items = []string{name + " (" + userLabel(user) + ")"} } - h.notifier.NotifyLogged(ctx, notifications.KindPlaybackErrors, notifications.ToAdmins(pgtype.UUID{}), - notifications.Payload{Count: n}.Map()) + h.notifier.NotifyLogged(ctx, notifications.KindPlaybackErrors, notifications.ToAdmins(pgtype.UUID{}), p.Map()) } diff --git a/internal/api/notify_producers_test.go b/internal/api/notify_producers_test.go index d5f7d0e7..bee25be8 100644 --- a/internal/api/notify_producers_test.go +++ b/internal/api/notify_producers_test.go @@ -15,8 +15,8 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" ) -// inboxItem is one thing the notifier left for one user: kind and decoded payload, -// newest first. +// inboxItem is one thing the notifier left for one user: kind and decoded +// payload, newest first. type inboxItem struct { Kind string Payload notifications.Payload @@ -53,27 +53,9 @@ func wantInbox(t *testing.T, who string, got []inboxItem, kinds ...notifications } } -// newApprovingLidarrStub answers every Lidarr call an approval makes. -func newApprovingLidarrStub(t *testing.T) *httptest.Server { - t.Helper() - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - switch { - case strings.Contains(r.URL.Path, "/metadataprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Standard"}]`)) - case strings.Contains(r.URL.Path, "/qualityprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Lossless"}]`)) - default: - _, _ = w.Write([]byte(`{"id":1}`)) - } - })) - t.Cleanup(stub.Close) - return stub -} - func TestNotify_NewRequestReachesOtherAdminsOnce(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "np-alice", "pw", false) admin := seedUser(t, pool, "np-admin", "pw", true) @@ -99,9 +81,8 @@ func TestNotify_NewRequestReachesOtherAdminsOnce(t *testing.T) { } func TestNotify_AutoApprovedRequestWaitsOnNobody(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) - saveLidarrConfig(t, h, newApprovingLidarrStub(t).URL, true) + h, _ := testHandlers(t) + resetRequests(t, h) alice := seedUser(t, h.pool, "np-auto", "pw", false) admin := seedUser(t, h.pool, "np-auto-admin", "pw", true) @@ -123,9 +104,8 @@ func TestNotify_AutoApprovedRequestWaitsOnNobody(t *testing.T) { } func TestNotify_DecisionsReachTheRequester(t *testing.T) { - h, _ := testHandlersWithClientFn(t) - resetLidarrState(t, h) - saveLidarrConfig(t, h, newApprovingLidarrStub(t).URL, true) + h, _ := testHandlers(t) + resetRequests(t, h) alice := seedUser(t, h.pool, "np-dec-alice", "pw", false) admin := seedUser(t, h.pool, "np-dec-admin", "pw", true) @@ -231,8 +211,9 @@ func TestNotify_PlaybackErrorsCoalesceIntoOneCount(t *testing.T) { got := inboxOf(t, pool, admin) wantInbox(t, "admin", got, notifications.KindPlaybackErrors) - if got[0].Payload.Count != 2 { - t.Errorf("count = %d, want 2 unresolved", got[0].Payload.Count) + if got[0].Payload.Count != 2 || len(got[0].Payload.Items) != 2 || + !strings.HasSuffix(got[0].Payload.Items[0], "np-play-alice)") { + t.Errorf("payload = %+v, want two reports, each naming the track and alice", got[0].Payload) } wantInbox(t, "reporter", inboxOf(t, pool, alice)) } diff --git a/internal/api/playback_errors.go b/internal/api/playback_errors.go index ccc358c7..99296522 100644 --- a/internal/api/playback_errors.go +++ b/internal/api/playback_errors.go @@ -130,7 +130,7 @@ func (h *handlers) handleReportPlaybackError(w http.ResponseWriter, r *http.Requ writeErr(w, apierror.Internal(err)) return } - h.notifyPlaybackErrors(r.Context()) + h.notifyPlaybackErrors(r.Context(), trackID, user) writeJSON(w, http.StatusCreated, map[string]string{"id": uuidToString(row.ID)}) } diff --git a/internal/api/playlists.go b/internal/api/playlists.go index eab70042..a1b22da7 100644 --- a/internal/api/playlists.go +++ b/internal/api/playlists.go @@ -60,7 +60,7 @@ type playlistTrackView struct { DurationSec int32 `json:"duration_sec"` StreamURL *string `json:"stream_url"` AddedAt string `json:"added_at"` - // Unavailable marks an entry whose file is missing from disk (#2527). + // Unavailable marks an entry whose file is missing from disk. // The track is still a real, known track — its history and likes are // intact and the file may come back — so the entry keeps its place and // its text; clients render it greyed out and skip over it on playback. @@ -316,8 +316,9 @@ func (h *handlers) handleAppendTracks(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, playlistDetailToView(detail)) } -// handleRemovePlaylistTrack implements DELETE /api/playlists/{id}/tracks/{position}. -// Owner only. Returns the post-mutation detail like AppendTracks. +// handleRemovePlaylistTrack implements DELETE +// /api/playlists/{id}/tracks/{position}. Owner only. Returns the post-mutation +// detail like AppendTracks. func (h *handlers) handleRemovePlaylistTrack(w http.ResponseWriter, r *http.Request) { caller, ok := requireUser(w, r) if !ok { diff --git a/internal/api/playlists_system_shuffle.go b/internal/api/playlists_system_shuffle.go index 84be5c1a..a1315c93 100644 --- a/internal/api/playlists_system_shuffle.go +++ b/internal/api/playlists_system_shuffle.go @@ -14,15 +14,14 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/playlists" ) -// Generic registry-driven system-playlist endpoints (#411 R2), -// replacing the former per-kind handlers. {kind} is the raw +// Generic registry-driven system-playlist endpoints. {kind} is the raw // system_variant ('for_you' | 'discover' | future singleton kinds); // only singleton kinds (playlists.RefreshableSystemKind) are // addressable here — songs_like_artist is multi-per-user and played // by playlist id, so it 404s. // // GET /api/playlists/system/{kind}/shuffle — the caller's playlist -// for that kind with tracks in rotation-aware order (#415): unheard +// for that kind with tracks in rotation-aware order: unheard // this rotation first (shuffled), then heard; resets when exhausted. // Same JSON shape as GET /api/playlists/{id} so clients reuse track // parsing. Intentionally uncached — it varies per play, unlike the @@ -73,7 +72,7 @@ func (h *handlers) handleSystemPlaylistRefresh(w http.ResponseWriter, r *http.Re writeErr(w, apierror.InternalMsg("build failed", err)) return } - // #968: announce the rebuild so the user's other clients refresh. + // Announce the rebuild so the user's other clients refresh. h.publishSystemRebuilt(user.ID) q := dbq.New(h.pool) v := kind diff --git a/internal/api/playlists_system_test.go b/internal/api/playlists_system_test.go index 621775fd..7664fbf4 100644 --- a/internal/api/playlists_system_test.go +++ b/internal/api/playlists_system_test.go @@ -10,8 +10,7 @@ import ( "github.com/go-chi/chi/v5" ) -// Generic registry-driven system-playlist endpoints (#411 R2). -// Replaces the former per-kind refresh handler tests. +// Generic registry-driven system-playlist endpoints. func newSystemPlaylistRouter(h *handlers) chi.Router { r := chi.NewRouter() r.Route("/api", func(api chi.Router) { diff --git a/internal/api/playlists_test.go b/internal/api/playlists_test.go index 1c77d1fa..0930a2e3 100644 --- a/internal/api/playlists_test.go +++ b/internal/api/playlists_test.go @@ -571,13 +571,15 @@ func TestListPlaylists_BadKindReturns400(t *testing.T) { } // TestEditEndpoints_403OnSystemPlaylist verifies that all five edit verbs -// return 403 system_playlist_readonly when the target playlist has kind='system'. +// return 403 system_playlist_readonly when the target playlist has +// kind='system'. func TestEditEndpoints_403OnSystemPlaylist(t *testing.T) { h, pool := testHandlers(t) u := seedUser(t, pool, "sysedit", "pw", false) // Seed a system playlist directly via SQL (BuildSystemPlaylists requires - // play_events seeding which we already exercise in playlists package tests). + // play_events seeding which we already exercise in playlists package + // tests). var sid string if err := pool.QueryRow(context.Background(), ` INSERT INTO playlists (user_id, name, description, is_public, kind, system_variant) diff --git a/internal/api/quarantine.go b/internal/api/quarantine.go index c5df1b22..091d24e7 100644 --- a/internal/api/quarantine.go +++ b/internal/api/quarantine.go @@ -8,8 +8,8 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrquarantine" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" + "git.fabledsword.com/bvandeusen/minstrel/internal/quarantine" ) // quarantineView is the JSON shape returned by the user-facing endpoints @@ -22,7 +22,7 @@ type quarantineView struct { CreatedAt pgtype.Timestamptz `json:"created_at"` } -func quarantineViewFrom(row dbq.LidarrQuarantine) quarantineView { +func quarantineViewFrom(row dbq.Quarantine) quarantineView { return quarantineView{ UserID: uuidToString(row.UserID), TrackID: uuidToString(row.TrackID), @@ -55,12 +55,12 @@ func (h *handlers) handleFlag(w http.ResponseWriter, r *http.Request) { writeErr(w, apierror.BadRequest("bad_request", "invalid track id")) return } - row, err := h.lidarrQuarantine.Flag(r.Context(), user.ID, trackID, body.Reason, body.Notes) + row, err := h.quarantine.Flag(r.Context(), user.ID, trackID, body.Reason, body.Notes) if err != nil { switch { - case errors.Is(err, lidarrquarantine.ErrBadReason): + case errors.Is(err, quarantine.ErrBadReason): writeErr(w, apierror.BadRequest("bad_reason", "invalid reason")) - case errors.Is(err, lidarrquarantine.ErrTrackNotFound): + case errors.Is(err, quarantine.ErrTrackNotFound): writeErr(w, &apierror.Error{Status: 404, Code: "track_not_found", Message: "track does not exist"}) default: h.logger.Error("api: flag", "err", err) @@ -71,7 +71,7 @@ func (h *handlers) handleFlag(w http.ResponseWriter, r *http.Request) { // Broadcast: the flagging user's other clients invalidate their // Hidden tab; admins' clients invalidate their quarantine queue. h.publishQuarantineEvent("quarantine.flagged", user.ID, trackID, true) - // Admins review flags (M489). An admin flagging a track needs no notice + // Admins review flags. An admin flagging a track needs no notice // of their own flag. h.notifier.NotifyLogged(r.Context(), notifications.KindQuarantineFlagged, notifications.ToAdmins(user.ID), notifications.Payload{Name: h.trackLabel(r.Context(), trackID), Actor: userLabel(user), Reason: quarantineReasonLabel(body.Reason)}.Map()) @@ -88,8 +88,8 @@ func (h *handlers) handleUnflag(w http.ResponseWriter, r *http.Request) { if !ok { return } - if err := h.lidarrQuarantine.Unflag(r.Context(), user.ID, id); err != nil { - if errors.Is(err, lidarrquarantine.ErrQuarantineNotFound) { + if err := h.quarantine.Unflag(r.Context(), user.ID, id); err != nil { + if errors.Is(err, quarantine.ErrQuarantineNotFound) { writeErr(w, &apierror.Error{Status: 404, Code: "quarantine_not_found", Message: "no quarantine for that track"}) return } @@ -125,7 +125,7 @@ func (h *handlers) handleListMyQuarantine(w http.ResponseWriter, r *http.Request if !ok { return } - rows, err := h.lidarrQuarantine.ListMine(r.Context(), user.ID) + rows, err := h.quarantine.ListMine(r.Context(), user.ID) if err != nil { h.logger.Error("api: list mine", "err", err) writeErr(w, apierror.InternalMsg("list failed", err)) @@ -134,10 +134,10 @@ func (h *handlers) handleListMyQuarantine(w http.ResponseWriter, r *http.Request out := make([]quarantineMineView, 0, len(rows)) for _, row := range rows { out = append(out, quarantineMineView{ - TrackID: uuidToString(row.LidarrQuarantine.TrackID), - Reason: string(row.LidarrQuarantine.Reason), - Notes: row.LidarrQuarantine.Notes, - CreatedAt: row.LidarrQuarantine.CreatedAt, + TrackID: uuidToString(row.Quarantine.TrackID), + Reason: string(row.Quarantine.Reason), + Notes: row.Quarantine.Notes, + CreatedAt: row.Quarantine.CreatedAt, TrackTitle: row.TrackTitle, TrackDurationMs: row.TrackDurationMs, AlbumID: uuidToString(row.AlbumID), diff --git a/internal/api/radio.go b/internal/api/radio.go index d8558bb8..641ceb58 100644 --- a/internal/api/radio.go +++ b/internal/api/radio.go @@ -82,16 +82,14 @@ func (h *handlers) handleRadio(w http.ResponseWriter, r *http.Request) { } currentVec := loadCurrentSessionVector(r, q, user.ID, h.logger) - // Condition on the current device (#1551): the latest play's device is a + // Condition on the current device: the latest play's device is a // cheap, request-free proxy for what the user is on right now. currentVec.DeviceClass = latestDeviceClass(r.Context(), q, user.ID, h.logger) exclude := parseExcludeParam(r.URL.Query().Get("exclude")) - // Size the pool to the library (#3880). A fixed ~170 candidates samples a - // shrinking fraction of a growing collection, which is what made the - // recommendations feel less relevant as the library grew. Degrades to the - // base limits if the count is unavailable — never fails the request over a - // sizing hint. + // Size the pool to the library: a fixed ~170 candidates samples a shrinking + // fraction of a growing collection. Degrades to the base limits if the + // count is unavailable — never fails the request over a sizing hint. librarySize := h.librarySize.Get(r.Context(), func(ctx context.Context) (int64, error) { return recommendation.CountLibraryTracks(ctx, q) }) @@ -101,7 +99,7 @@ func (h *handlers) handleRadio(w http.ResponseWriter, r *http.Request) { candidates, err := recommendation.LoadCandidatesFromSimilarity( r.Context(), q, user.ID, seedID, h.recCfg.RecentlyPlayedHours, currentVec, exclude, limits, - // A fresh seed per request (#3889): radio is a new session each time + // A fresh seed per request: radio is a new session each time // and SHOULD draw differently. The system mixes are the surfaces that // promise repeatability; this is not one of them. strconv.FormatInt(time.Now().UnixNano(), 36), @@ -119,11 +117,11 @@ func (h *handlers) handleRadio(w http.ResponseWriter, r *http.Request) { } } - // Scoring weights come from the DB-backed tuning lab (#1250) — + // Scoring weights come from the DB-backed tuning — // read per request so an admin change takes effect live. weights := h.recSettings.Weights(recsettings.ScopeRadio) - // Diversity caps (#3882). Radio had none while every sibling surface did, - // which is how a whole session could come back from one artist. Scaled to + // Diversity caps, so a whole session cannot come back from one artist. + // Scaled to // the requested length so a 20-track radio and a 200-track one are capped // alike; Shuffle relaxes them rather than returning a short radio. // @@ -177,7 +175,7 @@ func loadCurrentSessionVector(r *http.Request, q *dbq.Queries, userID pgtype.UUI } // latestDeviceClass returns the device_class of the user's most recent play as -// the "current device" for context conditioning (#1551), or "" when unknown +// the "current device" for context conditioning, or "" when unknown // (no plays yet, or the latest play predates device capture). Best-effort: a // lookup failure yields a device-agnostic ("") affinity cell. func latestDeviceClass(ctx context.Context, q *dbq.Queries, userID pgtype.UUID, logger *slog.Logger) string { diff --git a/internal/api/replay_gain.go b/internal/api/replay_gain.go index 38009db1..c2807031 100644 --- a/internal/api/replay_gain.go +++ b/internal/api/replay_gain.go @@ -19,8 +19,8 @@ type replayGainResp struct { Items map[string]library.ReplayGain `json:"items"` } -// handleGetReplayGain implements GET /api/tracks/replay-gain?ids=a,b,c (M464 -// #4997): ReplayGain 2.0 values (dB to -18 LUFS, linear peaks) for each track, +// handleGetReplayGain implements GET /api/tracks/replay-gain?ids=a,b,c: +// ReplayGain 2.0 values (dB to -18 LUFS, linear peaks) for each track, // and for its album when the whole album is measured. // // A lookup rather than fields on TrackRef. Tracks reach the player through diff --git a/internal/api/requests.go b/internal/api/requests.go index 0c1edd47..887da75b 100644 --- a/internal/api/requests.go +++ b/internal/api/requests.go @@ -9,34 +9,35 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" + "git.fabledsword.com/bvandeusen/minstrel/internal/requests" ) // requestView is the JSON shape returned by all /api/requests handlers. // Field names follow the lowercase snake_case convention for /api/* responses. type requestView struct { - ID pgtype.UUID `json:"id"` - UserID pgtype.UUID `json:"user_id"` - Status string `json:"status"` - Kind string `json:"kind"` - LidarrArtistMBID string `json:"lidarr_artist_mbid"` - LidarrAlbumMBID *string `json:"lidarr_album_mbid,omitempty"` - LidarrTrackMBID *string `json:"lidarr_track_mbid,omitempty"` - ArtistName string `json:"artist_name"` - AlbumTitle *string `json:"album_title,omitempty"` - TrackTitle *string `json:"track_title,omitempty"` - QualityProfileID *int32 `json:"quality_profile_id,omitempty"` - RootFolderPath *string `json:"root_folder_path,omitempty"` - DecidedAt pgtype.Timestamptz `json:"decided_at,omitempty"` - DecidedBy pgtype.UUID `json:"decided_by,omitempty"` - Notes *string `json:"notes,omitempty"` - CompletedAt pgtype.Timestamptz `json:"completed_at,omitempty"` - MatchedTrackID pgtype.UUID `json:"matched_track_id,omitempty"` - MatchedAlbumID pgtype.UUID `json:"matched_album_id,omitempty"` - MatchedArtistID pgtype.UUID `json:"matched_artist_id,omitempty"` - RequestedAt pgtype.Timestamptz `json:"requested_at"` - UpdatedAt pgtype.Timestamptz `json:"updated_at"` + ID pgtype.UUID `json:"id"` + UserID pgtype.UUID `json:"user_id"` + // Username names the requester. Only the admin list fills it; a user's + // own /requests page has no need to name them back to themselves. + Username string `json:"username,omitempty"` + Status string `json:"status"` + Kind string `json:"kind"` + ArtistMBID string `json:"artist_mbid"` + AlbumMBID *string `json:"album_mbid,omitempty"` + TrackMBID *string `json:"track_mbid,omitempty"` + ArtistName string `json:"artist_name"` + AlbumTitle *string `json:"album_title,omitempty"` + TrackTitle *string `json:"track_title,omitempty"` + DecidedAt pgtype.Timestamptz `json:"decided_at,omitempty"` + DecidedBy pgtype.UUID `json:"decided_by,omitempty"` + Notes *string `json:"notes,omitempty"` + CompletedAt pgtype.Timestamptz `json:"completed_at,omitempty"` + MatchedTrackID pgtype.UUID `json:"matched_track_id,omitempty"` + MatchedAlbumID pgtype.UUID `json:"matched_album_id,omitempty"` + MatchedArtistID pgtype.UUID `json:"matched_artist_id,omitempty"` + RequestedAt pgtype.Timestamptz `json:"requested_at"` + UpdatedAt pgtype.Timestamptz `json:"updated_at"` // Progress counters, computed at response time from the matched // entity's children. 0 when nothing has been ingested yet (or when // the request has no matched entity yet). For artist requests both @@ -45,29 +46,27 @@ type requestView struct { ImportedTrackCount int `json:"imported_track_count"` } -func requestViewFrom(row dbq.LidarrRequest) requestView { +func requestViewFrom(row dbq.Request) requestView { return requestView{ - ID: row.ID, - UserID: row.UserID, - Status: string(row.Status), - Kind: string(row.Kind), - LidarrArtistMBID: row.LidarrArtistMbid, - LidarrAlbumMBID: row.LidarrAlbumMbid, - LidarrTrackMBID: row.LidarrTrackMbid, - ArtistName: row.ArtistName, - AlbumTitle: row.AlbumTitle, - TrackTitle: row.TrackTitle, - QualityProfileID: row.QualityProfileID, - RootFolderPath: row.RootFolderPath, - DecidedAt: row.DecidedAt, - DecidedBy: row.DecidedBy, - Notes: row.Notes, - CompletedAt: row.CompletedAt, - MatchedTrackID: row.MatchedTrackID, - MatchedAlbumID: row.MatchedAlbumID, - MatchedArtistID: row.MatchedArtistID, - RequestedAt: row.RequestedAt, - UpdatedAt: row.UpdatedAt, + ID: row.ID, + UserID: row.UserID, + Status: string(row.Status), + Kind: string(row.Kind), + ArtistMBID: row.ArtistMbid, + AlbumMBID: row.AlbumMbid, + TrackMBID: row.TrackMbid, + ArtistName: row.ArtistName, + AlbumTitle: row.AlbumTitle, + TrackTitle: row.TrackTitle, + DecidedAt: row.DecidedAt, + DecidedBy: row.DecidedBy, + Notes: row.Notes, + CompletedAt: row.CompletedAt, + MatchedTrackID: row.MatchedTrackID, + MatchedAlbumID: row.MatchedAlbumID, + MatchedArtistID: row.MatchedArtistID, + RequestedAt: row.RequestedAt, + UpdatedAt: row.UpdatedAt, } } @@ -81,9 +80,9 @@ func requestViewFrom(row dbq.LidarrRequest) requestView { // N+1 against the request list: acceptable at admin scale (dozens of // in-flight requests at most). Errors are swallowed — progress is a // reporting nicety, not a correctness gate. -func fillProgress(ctx context.Context, q *dbq.Queries, row dbq.LidarrRequest, view *requestView) { +func fillProgress(ctx context.Context, q *dbq.Queries, row dbq.Request, view *requestView) { switch row.Kind { - case dbq.LidarrRequestKindArtist: + case dbq.RequestKindArtist: if !row.MatchedArtistID.Valid { return } @@ -93,7 +92,7 @@ func fillProgress(ctx context.Context, q *dbq.Queries, row dbq.LidarrRequest, vi if n, err := q.CountTracksByArtist(ctx, row.MatchedArtistID); err == nil { view.ImportedTrackCount = int(n) } - case dbq.LidarrRequestKindAlbum: + case dbq.RequestKindAlbum: if !row.MatchedAlbumID.Valid { return } @@ -101,22 +100,23 @@ func fillProgress(ctx context.Context, q *dbq.Queries, row dbq.LidarrRequest, vi if n, err := q.CountTracksByAlbum(ctx, row.MatchedAlbumID); err == nil { view.ImportedTrackCount = int(n) } - case dbq.LidarrRequestKindTrack: + case dbq.RequestKindTrack: if row.MatchedTrackID.Valid { view.ImportedTrackCount = 1 } } } -// createRequestBody is the decoded JSON for POST /api/requests. +// createRequestBody is the decoded JSON for POST /api/requests. album_mbid +// is the album's MusicBrainz release group, as catalogue search returns it. type createRequestBody struct { - Kind string `json:"kind"` - LidarrArtistMBID string `json:"lidarr_artist_mbid"` - LidarrAlbumMBID string `json:"lidarr_album_mbid"` - LidarrTrackMBID string `json:"lidarr_track_mbid"` - ArtistName string `json:"artist_name"` - AlbumTitle string `json:"album_title"` - TrackTitle string `json:"track_title"` + Kind string `json:"kind"` + ArtistMBID string `json:"artist_mbid"` + AlbumMBID string `json:"album_mbid"` + TrackMBID string `json:"track_mbid"` + ArtistName string `json:"artist_name"` + AlbumTitle string `json:"album_title"` + TrackTitle string `json:"track_title"` } // handleCreateRequest implements POST /api/requests. @@ -131,17 +131,17 @@ func (h *handlers) handleCreateRequest(w http.ResponseWriter, r *http.Request) { return } - row, created, err := h.lidarrRequests.CreateTracked(r.Context(), user.ID, lidarrrequests.CreateParams{ - Kind: body.Kind, - LidarrArtistMBID: body.LidarrArtistMBID, - LidarrAlbumMBID: body.LidarrAlbumMBID, - LidarrTrackMBID: body.LidarrTrackMBID, - ArtistName: body.ArtistName, - AlbumTitle: body.AlbumTitle, - TrackTitle: body.TrackTitle, + row, created, err := h.requests.CreateTracked(r.Context(), user.ID, requests.CreateParams{ + Kind: body.Kind, + ArtistMBID: body.ArtistMBID, + AlbumMBID: body.AlbumMBID, + TrackMBID: body.TrackMBID, + ArtistName: body.ArtistName, + AlbumTitle: body.AlbumTitle, + TrackTitle: body.TrackTitle, }) if err != nil { - if errors.Is(err, lidarrrequests.ErrInvalidKindFields) { + if errors.Is(err, requests.ErrInvalidKindFields) { writeErr(w, apierror.BadRequest("mbid_required", err.Error())) return } @@ -150,22 +150,14 @@ func (h *handlers) handleCreateRequest(w http.ResponseWriter, r *http.Request) { return } - // Auto-approve flow (#355 / #376 U2.5): when the user has the - // auto_approve_requests flag set by an admin, transition the - // just-created pending row through Approve so it dispatches to - // Lidarr immediately. Failures here (Lidarr disabled, defaults - // missing, network error) leave the row pending — same fallback - // as a manual admin approve that hits the same error path. The - // user gets a 201 with whatever status Approve produced; admin - // can still resolve the row in /admin/requests if it stayed - // pending. - // - // The "actor" id passed to Approve is the user themselves — the - // auto-approval is the user acting under the privilege the admin - // granted them via the toggle. The trail of "admin set the flag" - // already lives in audit_log (ActionAutoApproveToggle from U2). + // Auto-approve: when an admin has set the user's + // auto_approve_requests flag, the new request goes straight onto the + // wanted list as if approved. A failure leaves it pending for an admin, + // the same as a manual approval hitting the same error. The approver is + // the user themselves, acting under the privilege the admin granted; the + // grant is in audit_log (ActionAutoApproveToggle). if user.AutoApproveRequests { - approved, aerr := h.lidarrRequests.Approve(r.Context(), row.ID, user.ID, lidarrrequests.ApproveOverrides{}) + approved, aerr := h.requests.Approve(r.Context(), row.ID, user.ID, pgtype.UUID{}) if aerr == nil { row = approved } else { @@ -176,11 +168,11 @@ func (h *handlers) handleCreateRequest(w http.ResponseWriter, r *http.Request) { } } - // A new request still pending after any auto-approval waits on an admin - // (M489). One that deduped into a request already in flight was - // announced when it was first made, and the requester, if they are an - // admin themselves, needs no notice of their own request. - if created && row.Status == dbq.LidarrRequestStatusPending { + // A new request still pending after any auto-approval waits on an admin. + // One that deduped into a request already in flight was announced when it + // was first made, and the requester, if they are an admin themselves, + // needs no notice of their own request. + if created && row.Status == dbq.RequestStatusPending { p := requestPayload(row) p.Actor = userLabel(user) h.notifier.NotifyLogged(r.Context(), notifications.KindRequestPending, notifications.ToAdmins(user.ID), p.Map()) @@ -190,14 +182,15 @@ func (h *handlers) handleCreateRequest(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusCreated, requestViewFrom(row)) } -// handleListRequests implements GET /api/requests — returns caller's own requests. +// handleListRequests implements GET /api/requests — returns caller's own +// requests. func (h *handlers) handleListRequests(w http.ResponseWriter, r *http.Request) { user, ok := requireUser(w, r) if !ok { return } - rows, err := h.lidarrRequests.ListForUser(r.Context(), user.ID, 100) + rows, err := h.requests.ListForUser(r.Context(), user.ID, 100) if err != nil { h.logger.Error("api: list requests", "err", err) writeErr(w, apierror.InternalMsg("list failed", err)) @@ -222,7 +215,7 @@ func (h *handlers) handleGetRequest(w http.ResponseWriter, r *http.Request) { return } - row, apiErr := resolveByID(r, "id", dbq.New(h.pool).GetLidarrRequestByID, "request") + row, apiErr := resolveByID(r, "id", dbq.New(h.pool).GetRequestByID, "request") if apiErr != nil { writeErr(w, apiErr) return @@ -252,12 +245,12 @@ func (h *handlers) handleCancelRequest(w http.ResponseWriter, r *http.Request) { return } - row, err := h.lidarrRequests.Cancel(r.Context(), id, user.ID) + row, err := h.requests.Cancel(r.Context(), id, user.ID) if err != nil { switch { - case errors.Is(err, lidarrrequests.ErrNotPending): + case errors.Is(err, requests.ErrNotPending): writeErr(w, apierror.Conflict("request_not_pending", "request is not pending")) - case errors.Is(err, lidarrrequests.ErrNotFound): + case errors.Is(err, requests.ErrNotFound): writeErr(w, apierror.NotFound("request")) default: h.logger.Error("api: cancel request", "err", err) diff --git a/internal/api/requests_test.go b/internal/api/requests_test.go index c3f95a44..3e922eec 100644 --- a/internal/api/requests_test.go +++ b/internal/api/requests_test.go @@ -27,12 +27,14 @@ func newRequestsRouter(h *handlers) chi.Router { return r } -// withUser injects a user into the request context the same way RequireUser does. +// withUser injects a user into the request context the same way RequireUser +// does. func withUser(req *http.Request, user dbq.User) *http.Request { return req.WithContext(context.WithValue(req.Context(), userCtxKeyForTest(), user)) } -// doCreateRequest fires POST /api/requests with the given JSON body as the given user. +// doCreateRequest fires POST /api/requests with the given JSON body as the +// given user. func doCreateRequest(h *handlers, user dbq.User, body string) *httptest.ResponseRecorder { req := httptest.NewRequest(http.MethodPost, "/api/requests", bytes.NewBufferString(body)) req.Header.Set("Content-Type", "application/json") @@ -69,11 +71,11 @@ func doCancelRequest(h *handlers, user dbq.User, id pgtype.UUID) *httptest.Respo return w } -// createArtistRequest is a convenience wrapper that seeds a valid artist request -// via the handler and returns the parsed response. +// createArtistRequest is a convenience wrapper that seeds a valid artist +// request via the handler and returns the parsed response. func createArtistRequest(t *testing.T, h *handlers, user dbq.User, artistMBID, artistName string) requestView { t.Helper() - body := fmt.Sprintf(`{"kind":"artist","lidarr_artist_mbid":%q,"artist_name":%q}`, artistMBID, artistName) + body := fmt.Sprintf(`{"kind":"artist","artist_mbid":%q,"artist_name":%q}`, artistMBID, artistName) w := doCreateRequest(h, user, body) if w.Code != http.StatusCreated { t.Fatalf("createArtistRequest: status = %d, want 201; body = %s", w.Code, w.Body.String()) @@ -85,15 +87,12 @@ func createArtistRequest(t *testing.T, h *handlers, user dbq.User, artistMBID, a return rv } -// TestHandleCreateRequest_AutoApprove_LidarrDisabled_StaysPending verifies -// that a user with auto_approve_requests=true still receives a 201 Created -// when Lidarr is disabled in the test environment — the auto-approve attempt -// fails inside Service.Approve with ErrLidarrDisabled, which the handler -// catches and logs, leaving the row in pending state. This is the U2.5 -// fallback contract: a misconfigured Lidarr must not break request creation. -func TestHandleCreateRequest_AutoApprove_LidarrDisabled_StaysPending(t *testing.T) { +// TestHandleCreateRequest_AutoApprove_GoesStraightOnTheWantedList verifies +// that a user with auto_approve_requests=true gets their request approved on +// creation: an artist request follows the artist at once. +func TestHandleCreateRequest_AutoApprove_GoesStraightOnTheWantedList(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) if _, err := pool.Exec(context.Background(), @@ -105,13 +104,9 @@ func TestHandleCreateRequest_AutoApprove_LidarrDisabled_StaysPending(t *testing. if err != nil { t.Fatalf("reload user: %v", err) } - if !refreshed.AutoApproveRequests { - t.Fatalf("auto_approve flag did not persist — bailing") - } - body := `{"kind":"artist","lidarr_artist_mbid":"auto-mbid-1","artist_name":"Autoband"}` + body := `{"kind":"artist","artist_mbid":"auto-mbid-1","artist_name":"Autoband"}` w := doCreateRequest(h, refreshed, body) - if w.Code != http.StatusCreated { t.Fatalf("status = %d, want 201; body = %s", w.Code, w.Body.String()) } @@ -119,8 +114,11 @@ func TestHandleCreateRequest_AutoApprove_LidarrDisabled_StaysPending(t *testing. if err := json.Unmarshal(w.Body.Bytes(), &rv); err != nil { t.Fatalf("decode: %v", err) } - if rv.Status != "pending" { - t.Errorf("status = %q, want pending (auto-approve falls back when Lidarr disabled)", rv.Status) + if rv.Status != "approved" { + t.Errorf("status = %q, want approved", rv.Status) + } + if _, err := dbq.New(pool).GetFollow(context.Background(), "auto-mbid-1"); err != nil { + t.Errorf("the artist should be followed: %v", err) } } @@ -128,10 +126,10 @@ func TestHandleCreateRequest_AutoApprove_LidarrDisabled_StaysPending(t *testing. // returns 201 with kind=artist and status=pending. func TestHandleCreateRequest_Artist_HappyPath(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) - body := `{"kind":"artist","lidarr_artist_mbid":"artist-mbid-1","artist_name":"The Beatles"}` + body := `{"kind":"artist","artist_mbid":"artist-mbid-1","artist_name":"The Beatles"}` w := doCreateRequest(h, alice, body) if w.Code != http.StatusCreated { @@ -147,8 +145,8 @@ func TestHandleCreateRequest_Artist_HappyPath(t *testing.T) { if rv.Status != "pending" { t.Errorf("status = %q, want pending", rv.Status) } - if rv.LidarrArtistMBID != "artist-mbid-1" { - t.Errorf("lidarr_artist_mbid = %q", rv.LidarrArtistMBID) + if rv.ArtistMBID != "artist-mbid-1" { + t.Errorf("artist_mbid = %q", rv.ArtistMBID) } if rv.ArtistName != "The Beatles" { t.Errorf("artist_name = %q", rv.ArtistName) @@ -159,14 +157,14 @@ func TestHandleCreateRequest_Artist_HappyPath(t *testing.T) { } // TestHandleCreateRequest_AlbumWithoutAlbumMBID_400 verifies that sending -// kind=album without lidarr_album_mbid returns 400 with mbid_required. +// kind=album without album_mbid returns 400 with mbid_required. func TestHandleCreateRequest_AlbumWithoutAlbumMBID_400(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) - // Missing lidarr_album_mbid and album_title. - body := `{"kind":"album","lidarr_artist_mbid":"artist-mbid-1","artist_name":"The Beatles"}` + // Missing album_mbid and album_title. + body := `{"kind":"album","artist_mbid":"artist-mbid-1","artist_name":"The Beatles"}` w := doCreateRequest(h, alice, body) if w.Code != http.StatusBadRequest { @@ -189,11 +187,11 @@ func TestHandleCreateRequest_AlbumWithoutAlbumMBID_400(t *testing.T) { // with missing track_mbid/track_title returns 400 with mbid_required. func TestHandleCreateRequest_TrackWithoutTrackFields_400(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) // Provides album fields but misses track_mbid and track_title. - body := `{"kind":"track","lidarr_artist_mbid":"artist-mbid-1","artist_name":"Beatles","lidarr_album_mbid":"album-mbid-1","album_title":"Abbey Road"}` + body := `{"kind":"track","artist_mbid":"artist-mbid-1","artist_name":"Beatles","album_mbid":"album-mbid-1","album_title":"Abbey Road"}` w := doCreateRequest(h, alice, body) if w.Code != http.StatusBadRequest { @@ -216,7 +214,7 @@ func TestHandleCreateRequest_TrackWithoutTrackFields_400(t *testing.T) { // requests, not bob's. func TestHandleListRequests_OnlyOwnRequests(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) bob := seedUser(t, pool, "bob", "pw", false) @@ -243,10 +241,11 @@ func TestHandleListRequests_OnlyOwnRequests(t *testing.T) { } } -// TestHandleGetRequest_OwnReturns200 verifies a caller can fetch their own request. +// TestHandleGetRequest_OwnReturns200 verifies a caller can fetch their own +// request. func TestHandleGetRequest_OwnReturns200(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) rv := createArtistRequest(t, h, alice, "artist-mbid-get", "Get Artist") @@ -268,7 +267,7 @@ func TestHandleGetRequest_OwnReturns200(t *testing.T) { // cannot see another user's request. func TestHandleGetRequest_OtherUserNotAdmin_404(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) bob := seedUser(t, pool, "bob", "pw", false) charlie := seedUser(t, pool, "charlie", "pw", false) @@ -297,7 +296,7 @@ func TestHandleGetRequest_OtherUserNotAdmin_404(t *testing.T) { // user's request. func TestHandleGetRequest_OtherUserAsAdmin_200(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) bob := seedUser(t, pool, "bob", "pw", false) admin := seedUser(t, pool, "admin", "pw", true) @@ -322,7 +321,7 @@ func TestHandleGetRequest_OtherUserAsAdmin_200(t *testing.T) { // request returns 200 with status=rejected. func TestHandleCancelRequest_PendingHappyPath(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) rv := createArtistRequest(t, h, alice, "cancel-mbid-1", "Cancel Me") @@ -344,7 +343,7 @@ func TestHandleCancelRequest_PendingHappyPath(t *testing.T) { // rejected request returns 409 with request_not_pending. func TestHandleCancelRequest_NonPending_409(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice", "pw", false) rv := createArtistRequest(t, h, alice, "cancel-mbid-2", "Cancel Twice") @@ -355,10 +354,11 @@ func TestHandleCancelRequest_NonPending_409(t *testing.T) { t.Fatalf("first cancel: status = %d, want 200; body = %s", w1.Code, w1.Body.String()) } - // Second cancel on a now-rejected row → ErrNotPending → 409. - // The cancel SQL checks user_id = $2 AND status = 'pending'; zero rows → ErrNotPending. - // For this test we use dbq directly to reject via admin so Cancel returns ErrNotPending - // rather than the ownership miss. We already did first cancel so row is rejected. + // Second cancel on a now-rejected row → ErrNotPending → 409. The cancel SQL + // checks user_id = $2 AND status = 'pending'; zero rows → ErrNotPending. + // For this test we use dbq directly to reject via admin so Cancel returns + // ErrNotPending rather than the ownership miss. We already did first cancel + // so row is rejected. w2 := doCancelRequest(h, alice, rv.ID) if w2.Code != http.StatusConflict { t.Fatalf("second cancel: status = %d, want 409; body = %s", w2.Code, w2.Body.String()) @@ -380,14 +380,15 @@ func TestHandleCancelRequest_NonPending_409(t *testing.T) { // another user's request returns 409 (not_pending from the ownership miss). func TestHandleCancelRequest_WrongUser_409(t *testing.T) { h, pool := testHandlers(t) - resetLidarrState(t, h) + resetRequests(t, h) alice := seedUser(t, pool, "alice-cancel-owner", "pw", false) bob := seedUser(t, pool, "bob-cancel-other", "pw", false) rv := createArtistRequest(t, h, alice, "cancel-wrong-user-mbid", "Alice's Artist") - // Bob tries to cancel Alice's request: SQL WHERE user_id=bob AND status=pending → 0 rows. + // Bob tries to cancel Alice's request: SQL WHERE user_id=bob AND + // status=pending → 0 rows. w := doCancelRequest(h, bob, rv.ID) if w.Code != http.StatusConflict { t.Fatalf("status = %d, want 409; body = %s", w.Code, w.Body.String()) diff --git a/internal/api/reset_test.go b/internal/api/reset_test.go new file mode 100644 index 00000000..fd4e4c82 --- /dev/null +++ b/internal/api/reset_test.go @@ -0,0 +1,14 @@ +package api + +import ( + "context" + "testing" +) + +// resetRequests clears requests rows between tests. +func resetRequests(t *testing.T, h *handlers) { + t.Helper() + if _, err := h.pool.Exec(context.Background(), "DELETE FROM requests"); err != nil { + t.Fatalf("resetRequests: %v", err) + } +} diff --git a/internal/api/search.go b/internal/api/search.go index d0ba6575..a0fa2765 100644 --- a/internal/api/search.go +++ b/internal/api/search.go @@ -127,7 +127,8 @@ func (h *handlers) resolveAlbumRefs(ctx context.Context, q *dbq.Queries, albums h.logger.Error("api: count tracks for album failed", "err", cerr) return nil, cerr } - // durationSec=0: not aggregated for search album refs per spec data flow. + // durationSec=0: not aggregated for search album refs per spec data + // flow. items = append(items, albumRefFrom(a, names[uuidToString(a.ArtistID)], int(count), 0)) } return items, nil @@ -135,7 +136,7 @@ func (h *handlers) resolveAlbumRefs(ctx context.Context, q *dbq.Queries, albums // resolveTrackRefs builds TrackRefs from dbq.Track rows: looks up the // parent album + artist by id for each track (pair of queries per track). -// At limit=200 this is 400 queries worst case — acceptable for M1.5. +// At limit=200 this is 400 queries worst case. func (h *handlers) resolveTrackRefs(ctx context.Context, q *dbq.Queries, tracks []dbq.Track) ([]TrackRef, error) { items := make([]TrackRef, 0, len(tracks)) for _, t := range tracks { diff --git a/internal/api/stream_token.go b/internal/api/stream_token.go index 9f1d70f9..b43fc1a2 100644 --- a/internal/api/stream_token.go +++ b/internal/api/stream_token.go @@ -15,10 +15,9 @@ import ( // trackID is the canonical track UUID. exp is the Unix time after which // the token is invalid; the handler checks exp before serving. // -// The secret comes from MINSTREL_STREAM_SECRET (or its auto-generated + -// persisted fallback in app_preferences). The loader lands in slice 2 -// (POST /api/cast/stream-token); this file is the primitive both the -// loader and the stream handler share. +// The secret comes from MINSTREL_STREAM_SECRET, or its auto-generated +// fallback persisted under the data directory. This file is the primitive +// the token endpoint and the stream handler share. func SignStreamToken(secret []byte, trackID string, exp int64) string { mac := hmac.New(sha256.New, secret) // hash.Hash.Write is documented as never returning an error. diff --git a/internal/api/suggestions.go b/internal/api/suggestions.go index 977d571f..96cca9dd 100644 --- a/internal/api/suggestions.go +++ b/internal/api/suggestions.go @@ -1,21 +1,18 @@ package api import ( - "context" "encoding/json" "errors" "io" "net/http" "strconv" "strings" - "sync" "github.com/go-chi/chi/v5" "github.com/jackc/pgx/v5/pgtype" "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" "git.fabledsword.com/bvandeusen/minstrel/internal/recommendation" ) @@ -26,15 +23,15 @@ type suggestionView struct { Score float64 `json:"score"` Attribution []seedContributionView `json:"attribution"` // MatchedTags are the candidate's tags that overlap the user's taste - // profile, strongest first (#2377) — the "matches: shoegaze, melancholic" + // profile, strongest first — the "matches: shoegaze, melancholic" // line. Omitted when empty, which is common: tag coverage for - // out-of-library artists is permanently partial (#2376), and the card - // falls back to the seed attribution it has always shown. + // out-of-library artists is permanently partial, and the card falls back + // to the seed attribution. MatchedTags []string `json:"matched_tags,omitempty"` - // ImageURL is resolved on-demand from Lidarr (out-of-library - // artists have no local art row). Omitted when Lidarr is disabled - // or has no match — the client falls back to a placeholder. Not - // cached: a remote URL Lidarr surfaced, fetched by the browser. + // ImageURL is Minstrel's catalogue art endpoint for the candidate + // (out-of-library artists have no local art row). It + // answers 404 when no enabled source has an image, and the client + // falls back to a placeholder. ImageURL string `json:"image_url,omitempty"` } @@ -78,7 +75,7 @@ func (h *handlers) handleListSuggestions(w http.ResponseWriter, r *http.Request) } // Read the tuned weight per request so an admin change takes effect on the - // next refresh, no restart (rule #25). + // next refresh, no restart. tagWeight := h.recSettings.Discover().TagOverlapWeight suggestions, err := recommendation.SuggestArtists( r.Context(), h.pool, user.ID, halfLife, limit, tagWeight) @@ -105,13 +102,13 @@ func (h *handlers) handleListSuggestions(w http.ResponseWriter, r *http.Request) MatchedTags: s.MatchedTags, }) } - h.resolveSuggestionArt(r.Context(), out) + resolveSuggestionArt(out) writeJSON(w, http.StatusOK, out) } // maxSnoozeDays caps a client-supplied duration. The DEFAULT is not here: it's -// a DB-backed knob on the admin tuning card (rule #25), read per request via -// recSettings.Discover().SnoozeDays. See #2377. +// a DB-backed knob on the admin tuning card, read per request via +// recSettings.Discover().SnoozeDays. const maxSnoozeDays = 365.0 // snoozeRequest is the POST body. Both fields are optional in the JSON sense @@ -139,8 +136,8 @@ type snoozeView struct { // snoozing an already-snoozed candidate extends it rather than conflicting. // // This is NOT negative feedback. It records no verdict on the artist and is -// never read by internal/taste — see 0049_suggestion_snoozes.up.sql for the -// rule #101 reasoning. Returns 204. +// never read by internal/taste — see 0049_suggestion_snoozes.up.sql for why. +// Returns 204. func (h *handlers) handleSnoozeSuggestion(w http.ResponseWriter, r *http.Request) { user, ok := requireUser(w, r) if !ok { @@ -225,7 +222,7 @@ func (h *handlers) handleUnsnoozeSuggestion(w http.ResponseWriter, r *http.Reque // // The un-snooze surface needs this: a parked candidate is by definition // absent from the suggestion deck, so without a list there is no way to -// reach the DELETE above. Scoped to the caller (rule #47). Expired rows are +// reach the DELETE above. Scoped to the caller. Expired rows are // already filtered by the query — the hourly gc sweep only reclaims space. func (h *handlers) handleListSuggestionSnoozes(w http.ResponseWriter, r *http.Request) { user, ok := requireUser(w, r) @@ -250,48 +247,15 @@ func (h *handlers) handleListSuggestionSnoozes(w http.ResponseWriter, r *http.Re writeJSON(w, http.StatusOK, out) } -// resolveSuggestionArt fills ImageURL on-demand from Lidarr's artist -// lookup, matched by MBID (foreignArtistId). Best-effort and cache-free: -// Lidarr is the only source — when it's disabled, unreachable, or has -// no match for a candidate, that entry keeps an empty ImageURL and the -// client renders its placeholder. Lookups run with bounded concurrency -// so a full Discover page doesn't serialize ~12 round-trips. Never -// fails the request; the suggestions list is the contract, art is a -// nicety. -func (h *handlers) resolveSuggestionArt(ctx context.Context, views []suggestionView) { - if len(views) == 0 { - return - } - cfg, err := h.lidarrCfg.Get(ctx) - if err != nil || !cfg.Enabled || cfg.BaseURL == "" || cfg.APIKey == "" { - return // Lidarr off / unconfigured → placeholders only - } - client := lidarr.NewClient(cfg.BaseURL, cfg.APIKey) - - const maxConcurrent = 6 - sem := make(chan struct{}, maxConcurrent) - var wg sync.WaitGroup +// resolveSuggestionArt points each suggestion's ImageURL at Minstrel's +// catalogue art endpoint. Nothing is fetched here: the image is +// looked up from the enabled cover-art sources when the browser asks for it, +// and kept, so listing suggestions makes no outbound call. A candidate with +// no MBID keeps an empty ImageURL and the client renders its placeholder. +func resolveSuggestionArt(views []suggestionView) { for i := range views { - if views[i].MBID == "" || views[i].Name == "" { - continue + if views[i].MBID != "" { + views[i].ImageURL = artistArtURL(views[i].MBID, views[i].Name) } - wg.Add(1) - sem <- struct{}{} - go func(idx int) { - defer wg.Done() - defer func() { <-sem }() - results, lerr := client.LookupArtist(ctx, views[idx].Name) - if lerr != nil { - return // best-effort: no art on lookup failure - } - for _, res := range results { - if res.MBID == views[idx].MBID && res.ImageURL != "" { - // Distinct slice index per goroutine → race-free. - views[idx].ImageURL = res.ImageURL - return - } - } - }(i) } - wg.Wait() } diff --git a/internal/api/types.go b/internal/api/types.go index bcab22cf..b68594c5 100644 --- a/internal/api/types.go +++ b/internal/api/types.go @@ -7,7 +7,7 @@ import ( ) // LoginRequest is the POST /api/auth/login body. Kept boring on purpose — -// web and Flutter both send the same payload. +// web and Android send the same payload. type LoginRequest struct { Username string `json:"username"` Password string `json:"password"` @@ -15,7 +15,8 @@ type LoginRequest struct { // LoginResponse carries the opaque session token AND the authenticated user // so the SPA can hydrate its auth store in one round-trip. The token is also -// set as a cookie; SPAs ignore the body token, Flutter uses it as bearer. +// set as a cookie; the SPA ignores the body token, a native client may send +// it as a bearer. type LoginResponse struct { Token string `json:"token"` User UserView `json:"user"` @@ -82,7 +83,7 @@ type TrackRef struct { DiscNumber int `json:"disc_number,omitempty"` DurationSec int `json:"duration_sec"` StreamURL string `json:"stream_url"` - // Unavailable reports that the file is missing from disk (#2704). + // Unavailable reports that the file is missing from disk. // // Present on every TrackRef rather than only where it can be true: the // selection surfaces (recommendation, discover, mixes, browse) filter @@ -99,16 +100,19 @@ type TrackRef struct { type ArtistDetail struct { ArtistRef Albums []AlbumRef `json:"albums"` - // Genres carried by this artist's tracks, for quick-jump chips (#367). + // Genres carried by this artist's tracks, for quick-jump chips. // Always non-nil at JSON so the client can iterate without a null check. Genres []string `json:"genres"` + // MBID is the artist's MusicBrainz id, which following is keyed by. + // Empty when the tags never named one. + MBID string `json:"mbid"` } // AlbumDetail is the response body of GET /api/albums/{id}. type AlbumDetail struct { AlbumRef Tracks []TrackRef `json:"tracks"` - // Genres carried by this album's tracks, for quick-jump chips (#367). + // Genres carried by this album's tracks, for quick-jump chips. // Derived from the tracks rather than stored on the album, because genre // lives on tracks and an album's tracks can disagree. Non-nil at JSON. Genres []string `json:"genres"` diff --git a/internal/audit/audit.go b/internal/audit/audit.go index 3097bc23..07b37ba2 100644 --- a/internal/audit/audit.go +++ b/internal/audit/audit.go @@ -20,14 +20,13 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Action is the discriminator stored in audit_log.action. New values -// added in U2/U3 + future tasks; declare them here so callers don't -// stringly-type and so a future audit-search UI has a single source -// of truth for the vocabulary. +// Action is the discriminator stored in audit_log.action. Every value is +// declared here, so callers never stringly-type an action and the vocabulary +// has one source. type Action string const ( - // U1 + // Accounts and invites. ActionRegister Action = "register" ActionPromoteAdmin Action = "promote_admin" ActionDemoteAdmin Action = "demote_admin" @@ -35,38 +34,41 @@ const ( ActionInviteRedeem Action = "invite_redeem" ActionInviteRevoke Action = "invite_revoke" - // U2 (declared early; callers in U1 don't use them yet, but - // having them here means U2's diff is purely additive on the - // caller side, not also touching this file). + // Admin user management. ActionCreateUserAdmin Action = "create_user_admin" ActionDeleteUser Action = "delete_user" ActionPasswordResetAdmin Action = "password_reset_admin" ActionAutoApproveToggle Action = "auto_approve_toggle" - // U3 + // Credentials. ActionPasswordChangeSelf Action = "password_change_self" ActionTokenRegenerate Action = "token_regenerate" ActionForgotPasswordInit Action = "forgot_password_initiated" ActionPasswordResetByEmail Action = "password_reset_via_email" - // Active-sessions surface (#370). Worth auditing rather than silent: - // revoking sessions is what a user does when they think an account is - // compromised, so the audit trail is most useful precisely when it's - // exercised. + // Revoking sessions is what a user does when they think an account is + // compromised, so it is audited: the trail matters most exactly then. ActionSessionRevoke Action = "session_revoke" ActionSessionRevokeOthers Action = "session_revoke_others" - // Duplicate merge (#3911). Irreversible: a copy's file and row are removed + // Duplicate merge. Irreversible: a copy's file and row are removed // and its history moved onto the copy kept. The metadata names both, so the // log can answer "where did that file go" long after the report is gone. ActionDuplicateMerge Action = "duplicate_merge" + // The resolver settling a group without a merge: + // dismissing two versions of a song that are not duplicates, and removing + // copies that filled another song's track slot (their tracks are kept, + // marked missing, and fetched again). + ActionDuplicateDismiss Action = "duplicate_dismiss" + ActionDuplicateWrongFile Action = "duplicate_wrong_file" - // Lidarr release change (M498 #5437): the duplicate resolver moved an - // album's monitored release in Lidarr to one that lists each song once. - // Written with no actor; the metadata names the album and both releases. - ActionLidarrReleaseChange Action = "lidarr_release_change" + // An admin removing a track: its file and row are deleted, with every + // like, play and playlist entry that hung off the row. The metadata names + // the track and its file, so the log can say what went after the row is + // gone. + ActionTrackRemove Action = "track_remove" - // Subsonic password (#5026): generated in Settings for t/s-only clients. + // Subsonic password: generated in Settings for t/s-only clients. ActionSubsonicPasswordSet Action = "subsonic_password_set" ActionSubsonicPasswordClear Action = "subsonic_password_clear" ) diff --git a/internal/audit/audit_test.go b/internal/audit/audit_test.go index 46998384..19b0c07e 100644 --- a/internal/audit/audit_test.go +++ b/internal/audit/audit_test.go @@ -169,6 +169,8 @@ func TestWrite_AllActionConstantsArePersisted(t *testing.T) { audit.ActionForgotPasswordInit, audit.ActionPasswordResetByEmail, audit.ActionDuplicateMerge, + audit.ActionDuplicateDismiss, + audit.ActionDuplicateWrongFile, audit.ActionSubsonicPasswordSet, audit.ActionSubsonicPasswordClear, } diff --git a/internal/auth/clientip.go b/internal/auth/clientip.go index 3b741dc4..be7b450d 100644 --- a/internal/auth/clientip.go +++ b/internal/auth/clientip.go @@ -7,7 +7,7 @@ import ( ) // ClientIP returns the caller's address, reading through trustedProxyHops -// reverse proxies (#2453). +// reverse proxies. // // X-Forwarded-For grows left-to-right: every proxy APPENDS the peer it // received the request from. For client -> CDN -> own-proxy -> Minstrel the @@ -20,13 +20,12 @@ import ( // honestly claim is the address it told us about // hops 2 -> XFF[0] = client // -// This replaces an earlier heuristic that ignored XFF whenever RemoteAddr was -// public. That was safe but useless in the deployment that matters: a proxy -// on a public address (separate host, or a CDN) meant every session recorded -// the proxy, so the active-sessions surface could never show an address -// change (#370). +// Ignoring XFF whenever RemoteAddr is public would be safe but useless: behind +// a proxy on a public address (a separate host, or a CDN) every session would +// record the proxy, and the active-sessions list could never show an address +// change. // -// # What the operator is asserting +// # What the hop count asserts // // hops >= 1 is a DECLARATION that a proxy sits in front. Two ways to get it // wrong, both worth understanding rather than papering over: @@ -36,9 +35,9 @@ import ( // - Set HIGHER than the real chain: the index runs past the proxy-written // entries into attacker-supplied ones, same result. // -// Both are inherent to the trusted-hop model — Rails, Caddy, Traefik and -// nginx all behave this way — which is why 0 is a first-class value and the -// admin card tells the operator to count their proxies. +// Both are inherent to the trusted-hop model (Rails, Caddy, Traefik and nginx +// behave the same way), which is why 0 is the default and the admin card asks +// for the number of proxies in front. func ClientIP(r *http.Request, trustedProxyHops int) string { remote := hostOf(r.RemoteAddr) if trustedProxyHops <= 0 { @@ -51,7 +50,7 @@ func ClientIP(r *http.Request, trustedProxyHops int) string { return remote } // Clamp rather than reject: a chain shorter than the configured depth - // means the operator over-counted, and the leftmost entry is the closest + // means the hops were over-counted, and the leftmost entry is the closest // thing to a client on offer. The caveat above covers the risk. idx := len(chain) - trustedProxyHops if idx < 0 { @@ -68,8 +67,8 @@ func ClientIP(r *http.Request, trustedProxyHops int) string { // IsHTTPS reports whether the client reached us over HTTPS, reading through // trustedProxyHops reverse proxies by the same rule as ClientIP. // -// TLS terminates at the operator's proxy (rule 94), so r.TLS is nil in every -// real deployment and only X-Forwarded-Proto can say what the client used. +// TLS terminates at the reverse proxy, so r.TLS is nil in every real +// deployment and only X-Forwarded-Proto can say what the client used. // That header is believed only when hops >= 1: with no trusted proxy it is // just something any client can send. Proxies that append rather than // overwrite produce a comma list, read positionally like X-Forwarded-For. diff --git a/internal/auth/clientip_test.go b/internal/auth/clientip_test.go index 50a330d5..b462e7e4 100644 --- a/internal/auth/clientip_test.go +++ b/internal/auth/clientip_test.go @@ -34,8 +34,8 @@ func TestClientIP(t *testing.T) { }, { // The common case: one TLS-terminating proxy. Note RemoteAddr is - // PUBLIC here — a proxy on its own host — which the previous - // private-range heuristic got wrong. + // PUBLIC here — a proxy on its own host — which a private-range + // heuristic would get wrong. name: "one proxy on a public address yields the client", hops: 1, remoteAddr: "203.0.113.200:40000", @@ -68,8 +68,8 @@ func TestClientIP(t *testing.T) { want: "198.51.100.7", }, { - // A client prepending a lie is only reachable if the operator - // over-counts their proxies; at the correct depth it's skipped. + // A client prepending a lie is only reachable if the hops are + // over-counted; at the correct depth it's skipped. name: "forged prefix is not reached at the correct depth", hops: 1, remoteAddr: "172.18.0.1:40000", @@ -178,7 +178,7 @@ func TestIsHTTPS(t *testing.T) { tls bool want bool }{ - // The three cases rule 94 names. + // No trusted proxy, a trusted proxy saying https, and one saying http. {name: "hops 0 ignores a forwarded https", hops: 0, proto: "https", want: false}, {name: "hops 1 believes a forwarded https", hops: 1, proto: "https", want: true}, {name: "plain request is not https", hops: 1, want: false}, diff --git a/internal/auth/session.go b/internal/auth/session.go index 8f2ecdfa..2d4c45df 100644 --- a/internal/auth/session.go +++ b/internal/auth/session.go @@ -68,8 +68,8 @@ const SessionCookieName = "minstrel_session" // leak whether the username existed (matches the /rest/* auth posture). // // trustedHops supplies the reverse-proxy depth used to record the session's -// current address (#2453). It's a func rather than an int because the value -// is operator-editable at runtime and this middleware is constructed once at +// current address. It's a func rather than an int because the value +// is editable at runtime and this middleware is constructed once at // boot — reading it per request is what makes an admin change take effect // without a restart. Passing nil means "trust nothing", i.e. the socket peer. func RequireUser(pool *pgxpool.Pool, trustedHops func() int) func(http.Handler) http.Handler { @@ -155,7 +155,7 @@ func SessionIDCtxKeyForTest() any { return sessionIDCtxKey } // // Database lookup failures fall through too — a transient DB blip should not // 5xx a stream request that may have a perfectly valid signed token. The -// error is logged so the operator can correlate. +// error is logged so it can be correlated. func OptionalUser(pool *pgxpool.Pool, logger *slog.Logger) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { diff --git a/internal/auth/setup.go b/internal/auth/setup.go index 9d576be7..cb0b6848 100644 --- a/internal/auth/setup.go +++ b/internal/auth/setup.go @@ -7,10 +7,10 @@ import ( ) // SetupToken guards the first-admin registration. Until the first account -// exists, register makes whoever calls it the admin, so a fresh instance on a -// public address belonged to whoever found it first. Now that call also has -// to carry this token, which is generated at startup and written only to the -// server log: proof that the caller can read the operator's logs. +// exists, register makes whoever calls it the admin, so without this a fresh +// instance on a public address would belong to whoever found it first. The +// call must carry this token, which is generated at startup and written only +// to the server log: proof that the caller can read the server's logs. // // The token lives in memory. A restart mints a new one and logs it again, // which is the behaviour wanted: an old token from a log line someone else diff --git a/internal/buildinfo/buildinfo.go b/internal/buildinfo/buildinfo.go new file mode 100644 index 00000000..67bc4d8b --- /dev/null +++ b/internal/buildinfo/buildinfo.go @@ -0,0 +1,31 @@ +// Package buildinfo says which build is running and since when. It is the +// one place the version lives, so /healthz, the Subsonic envelope and the +// admin Overview cannot disagree about it. +package buildinfo + +import "time" + +// Version is the build's own version name — YYYY.MM.DD.HHMM, derived from +// the commit it was built from by ci/version.sh. Defaults to "dev" for local +// builds; overridden at link time via: +// +// -ldflags="-X 'git.fabledsword.com/bvandeusen/minstrel/internal/buildinfo.Version=2026.09.10.1449'" +// +// release.yml passes it through the MINSTREL_VERSION build-arg → Dockerfile +// ldflag. +// +// Image tags carry no version, so this is the only thing that says which +// build is running. +var Version = "dev" + +// Channel is which line this build came off — "stable" or "dev", or "local" +// for a plain `docker build`. +// +// A sibling field, never a suffix inside Version. The same +// commit built on both lanes reports the same version and differs only here; +// folding the two together is what makes a version string stop being +// comparable. +var Channel = "local" + +// Started is when this process started, for the admin Overview's uptime. +var Started = time.Now() diff --git a/internal/catalogue/art.go b/internal/catalogue/art.go new file mode 100644 index 00000000..0d1a1b63 --- /dev/null +++ b/internal/catalogue/art.go @@ -0,0 +1,214 @@ +package catalogue + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "regexp" + "sync" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/coverart" +) + +// NoArtRetry is how long "no source has art for this" is believed before +// the sources are asked again. +const NoArtRetry = 7 * 24 * time.Hour + +// fetchTimeout bounds one look-up across every source. +const fetchTimeout = 30 * time.Second + +// ErrNoArt: no enabled source has an image, or the sources could not be +// reached. The caller shows its placeholder. +var ErrNoArt = errors.New("catalogue: no art") + +var mbidPattern = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`) + +// ValidMBID reports whether s is shaped like a MusicBrainz id. It is the +// guard that keeps a request path out of the cache directory's parents. +func ValidMBID(s string) bool { return mbidPattern.MatchString(s) } + +// Art serves images for artists and release groups that are not in the +// library (search results, Discover suggestions, wanted albums), from the +// enabled cover-art sources. Each image is fetched once and kept +// under the data directory, so the browser only ever talks to Minstrel. +type Art struct { + dir string + artistSources func() []coverart.ArtistArtProvider + albumSources func() []coverart.AlbumCoverProvider + now func() time.Time + + mu sync.Mutex + inflight map[string]*artCall +} + +type artCall struct { + done chan struct{} + body []byte + err error +} + +// ArtSources lists the image sources to ask, in order. The cover-art +// settings service is one: it answers with the sources an admin enabled. +type ArtSources interface { + EnabledArtistProviders() []coverart.ArtistArtProvider + EnabledAlbumProviders() []coverart.AlbumCoverProvider +} + +// NewArt keeps images under dataDir/catalogue-art and asks the given +// sources. Nil sources ask nothing, so every image is the placeholder. +func NewArt(dataDir string, sources ArtSources) *Art { + a := &Art{dir: filepath.Join(dataDir, "catalogue-art"), now: time.Now, inflight: map[string]*artCall{}} + if sources != nil { + a.artistSources = sources.EnabledArtistProviders + a.albumSources = sources.EnabledAlbumProviders + } + return a +} + +// Artist returns an image of the artist: the first enabled artist source +// with one wins, its thumbnail preferred over its backdrop. +func (a *Art) Artist(ctx context.Context, mbid, name string) ([]byte, error) { + return a.cached(ctx, "artist", mbid, func(ctx context.Context) ([]byte, error) { + if a.artistSources == nil { + return nil, coverart.ErrNotFound + } + ref := coverart.ArtistRef{MBID: mbid, Name: name} + return firstOf(a.artistSources(), func(p coverart.ArtistArtProvider) ([]byte, error) { + thumb, fanart, err := p.FetchArtistArt(ctx, ref) + if err == nil && len(thumb) == 0 { + thumb = fanart + } + return thumb, err + }) + }) +} + +// ReleaseGroup returns a cover for the release group: a source that can +// answer by group (the Cover Art Archive) is asked by its id, the rest by +// artist and title. +func (a *Art) ReleaseGroup(ctx context.Context, mbid, artist, title string) ([]byte, error) { + return a.cached(ctx, "release-group", mbid, func(ctx context.Context) ([]byte, error) { + if a.albumSources == nil { + return nil, coverart.ErrNotFound + } + byName := coverart.AlbumRef{ArtistName: artist, AlbumTitle: title} + return firstOf(a.albumSources(), func(p coverart.AlbumCoverProvider) ([]byte, error) { + if g, ok := p.(coverart.ReleaseGroupCoverProvider); ok { + return g.FetchReleaseGroupCover(ctx, mbid) + } + if artist == "" || title == "" { + return nil, coverart.ErrNotFound + } + return p.FetchAlbumCover(ctx, byName) + }) + }) +} + +// firstOf asks each source in turn. It reports ErrNotFound only when every +// source said so; if any could not be reached, the miss is transient. +func firstOf[P any](sources []P, ask func(P) ([]byte, error)) ([]byte, error) { + transient := false + for _, p := range sources { + body, err := ask(p) + if err == nil && len(body) > 0 { + return body, nil + } + if err != nil && !errors.Is(err, coverart.ErrNotFound) { + transient = true + } + } + if transient { + return nil, coverart.ErrTransient + } + return nil, coverart.ErrNotFound +} + +// cached serves kind/mbid from disk, or fetches it once however many +// requests ask at the same moment. A definite "nobody has one" is written +// down and believed for NoArtRetry; an unreachable source is not, so the +// next request asks again. +func (a *Art) cached(ctx context.Context, kind, mbid string, fetch func(context.Context) ([]byte, error)) ([]byte, error) { + if !ValidMBID(mbid) { + return nil, ErrNoArt + } + dir := filepath.Join(a.dir, kind) + imgPath := filepath.Join(dir, mbid) + nonePath := imgPath + ".none" + + if body, err := os.ReadFile(imgPath); err == nil && len(body) > 0 { + return body, nil + } + if fi, err := os.Stat(nonePath); err == nil && a.now().Sub(fi.ModTime()) < NoArtRetry { + return nil, ErrNoArt + } + + key := kind + "/" + mbid + a.mu.Lock() + if call, ok := a.inflight[key]; ok { + a.mu.Unlock() + select { + case <-call.done: + return call.body, call.err + case <-ctx.Done(): + return nil, ctx.Err() + } + } + call := &artCall{done: make(chan struct{})} + a.inflight[key] = call + a.mu.Unlock() + + defer func() { + a.mu.Lock() + delete(a.inflight, key) + a.mu.Unlock() + close(call.done) + }() + + // The fetch outlives a browser that navigates away: a second request is + // likely, and the image is worth keeping. + fctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), fetchTimeout) + defer cancel() + body, err := fetch(fctx) + switch { + case err == nil && len(body) > 0: + if werr := writeFile(dir, imgPath, body); werr != nil { + // Serving it uncached is still better than a placeholder. + call.body = body + return body, nil + } + _ = os.Remove(nonePath) + call.body = body + return body, nil + case err == nil || errors.Is(err, coverart.ErrNotFound): + _ = writeFile(dir, nonePath, nil) + } + call.err = ErrNoArt + return nil, ErrNoArt +} + +func writeFile(dir, path string, body []byte) error { + if err := os.MkdirAll(dir, 0o755); err != nil { + return fmt.Errorf("catalogue art: %w", err) + } + tmp, err := os.CreateTemp(dir, ".part-*") + if err != nil { + return fmt.Errorf("catalogue art: %w", err) + } + if _, err := tmp.Write(body); err != nil { + _ = tmp.Close() + _ = os.Remove(tmp.Name()) + return fmt.Errorf("catalogue art: %w", err) + } + if err := tmp.Close(); err != nil { + _ = os.Remove(tmp.Name()) + return fmt.Errorf("catalogue art: %w", err) + } + if err := os.Rename(tmp.Name(), path); err != nil { + _ = os.Remove(tmp.Name()) + return fmt.Errorf("catalogue art: %w", err) + } + return nil +} diff --git a/internal/catalogue/art_test.go b/internal/catalogue/art_test.go new file mode 100644 index 00000000..70d099e9 --- /dev/null +++ b/internal/catalogue/art_test.go @@ -0,0 +1,165 @@ +package catalogue + +import ( + "context" + "errors" + "os" + "path/filepath" + "sync" + "sync/atomic" + "testing" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/coverart" +) + +const someMBID = "e21857d5-3256-4547-afb3-4b6ded592596" + +// fakeSource is an artist and album source answering with fixed results. +type fakeSource struct { + thumb, fanart, cover []byte + err error + calls atomic.Int32 + gate chan struct{} // when set, each fetch waits for it + lastRef coverart.AlbumRef + mu sync.Mutex +} + +func (f *fakeSource) ID() string { return "fake" } +func (f *fakeSource) DisplayName() string { return "Fake" } +func (f *fakeSource) RequiresAPIKey() bool { return false } +func (f *fakeSource) DefaultEnabled() bool { return true } +func (f *fakeSource) Configure(_ coverart.ProviderSettings) error { return nil } + +func (f *fakeSource) wait() { + f.calls.Add(1) + if f.gate != nil { + <-f.gate + } +} + +func (f *fakeSource) FetchArtistArt(_ context.Context, _ coverart.ArtistRef) ([]byte, []byte, error) { + f.wait() + return f.thumb, f.fanart, f.err +} + +func (f *fakeSource) FetchAlbumCover(_ context.Context, ref coverart.AlbumRef) ([]byte, error) { + f.wait() + f.mu.Lock() + f.lastRef = ref + f.mu.Unlock() + return f.cover, f.err +} + +// groupSource can also answer by release group, as the Cover Art Archive does. +type groupSource struct{ *fakeSource } + +func (g groupSource) FetchReleaseGroupCover(_ context.Context, _ string) ([]byte, error) { + g.wait() + return []byte("by-group"), nil +} + +type sources struct { + artists []coverart.ArtistArtProvider + albums []coverart.AlbumCoverProvider +} + +func (s sources) EnabledArtistProviders() []coverart.ArtistArtProvider { return s.artists } +func (s sources) EnabledAlbumProviders() []coverart.AlbumCoverProvider { return s.albums } + +func TestArt_FetchesOnceAndKeepsIt(t *testing.T) { + src := &fakeSource{fanart: []byte("backdrop")} + a := NewArt(t.TempDir(), sources{artists: []coverart.ArtistArtProvider{src}}) + + body, err := a.Artist(context.Background(), someMBID, "Gorillaz") + if err != nil || string(body) != "backdrop" { + t.Fatalf("with only a backdrop, use it: %q, %v", body, err) + } + if body, _ := a.Artist(context.Background(), someMBID, "Gorillaz"); string(body) != "backdrop" || src.calls.Load() != 1 { + t.Errorf("the second ask should come from disk: %q after %d calls", body, src.calls.Load()) + } +} + +func TestArt_NobodyHasOneIsBelievedForAWeek(t *testing.T) { + src := &fakeSource{err: coverart.ErrNotFound} + a := NewArt(t.TempDir(), sources{artists: []coverart.ArtistArtProvider{src}}) + + if _, err := a.Artist(context.Background(), someMBID, "x"); !errors.Is(err, ErrNoArt) { + t.Fatalf("err = %v, want ErrNoArt", err) + } + if _, err := a.Artist(context.Background(), someMBID, "x"); !errors.Is(err, ErrNoArt) || src.calls.Load() != 1 { + t.Errorf("a definite miss is remembered: %d calls", src.calls.Load()) + } + a.now = func() time.Time { return time.Now().Add(NoArtRetry + time.Hour) } + src.err, src.thumb = nil, []byte("thumb") + if body, err := a.Artist(context.Background(), someMBID, "x"); err != nil || string(body) != "thumb" { + t.Errorf("after a week the sources are asked again: %q, %v", body, err) + } +} + +func TestArt_AnUnreachableSourceIsAskedAgain(t *testing.T) { + src := &fakeSource{err: coverart.ErrTransient} + a := NewArt(t.TempDir(), sources{artists: []coverart.ArtistArtProvider{src}}) + for range 2 { + if _, err := a.Artist(context.Background(), someMBID, "x"); !errors.Is(err, ErrNoArt) { + t.Fatalf("err = %v", err) + } + } + if src.calls.Load() != 2 { + t.Errorf("calls = %d: an outage must not be written down as 'no art'", src.calls.Load()) + } +} + +func TestArt_ReleaseGroupAsksByGroupThenByName(t *testing.T) { + byName := &fakeSource{cover: []byte("by-name")} + caa := groupSource{&fakeSource{}} + a := NewArt(t.TempDir(), sources{albums: []coverart.AlbumCoverProvider{caa, byName}}) + if body, _ := a.ReleaseGroup(context.Background(), someMBID, "Gorillaz", "Demon Days"); string(body) != "by-group" { + t.Errorf("a source that answers by group is asked by group: %q", body) + } + + other := "0c4d5b5f-2f3c-3d31-9c4e-4b4f6b4c1a01" + a = NewArt(t.TempDir(), sources{albums: []coverart.AlbumCoverProvider{byName}}) + if body, _ := a.ReleaseGroup(context.Background(), other, "Gorillaz", "Demon Days"); string(body) != "by-name" { + t.Errorf("by name: %q", body) + } + if byName.lastRef.MBID != "" || byName.lastRef.AlbumTitle != "Demon Days" { + t.Errorf("a release-group id is not a release id, so it is not passed as one: %+v", byName.lastRef) + } +} + +func TestArt_ConcurrentAsksShareOneFetch(t *testing.T) { + src := &fakeSource{thumb: []byte("t"), gate: make(chan struct{})} + a := NewArt(t.TempDir(), sources{artists: []coverart.ArtistArtProvider{src}}) + var wg sync.WaitGroup + for range 5 { + wg.Add(1) + go func() { + defer wg.Done() + if body, err := a.Artist(context.Background(), someMBID, "x"); err != nil || string(body) != "t" { + t.Errorf("%q, %v", body, err) + } + }() + } + // Let the fetch through once every caller has had time to queue on it. + time.Sleep(50 * time.Millisecond) + close(src.gate) + wg.Wait() + if src.calls.Load() != 1 { + t.Errorf("calls = %d, want one fetch for five asks", src.calls.Load()) + } +} + +func TestArt_RefusesWhatIsNotAnMBID(t *testing.T) { + dir := t.TempDir() + a := NewArt(dir, sources{artists: []coverart.ArtistArtProvider{&fakeSource{thumb: []byte("t")}}}) + if _, err := a.Artist(context.Background(), "../../etc/passwd", "x"); !errors.Is(err, ErrNoArt) { + t.Fatalf("err = %v", err) + } + if _, err := os.Stat(filepath.Join(dir, "catalogue-art")); !os.IsNotExist(err) { + t.Error("nothing should be written for a refused id") + } + if _, err := NewArt(dir, nil).Artist(context.Background(), someMBID, "x"); !errors.Is(err, ErrNoArt) { + t.Errorf("no sources: err = %v", err) + } +} diff --git a/internal/catalogue/catalogue.go b/internal/catalogue/catalogue.go new file mode 100644 index 00000000..ada4fe47 --- /dev/null +++ b/internal/catalogue/catalogue.go @@ -0,0 +1,318 @@ +// Package catalogue is Minstrel's own record of what artists have released, +// read from MusicBrainz and kept in the database. The wanted +// list, search and import read it; MusicBrainz is asked only when what is +// held is missing or old. +package catalogue + +import ( + "context" + "errors" + "fmt" + "log/slog" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" +) + +const ( + // DiscographyMaxAge is how long an artist's list of release groups is + // trusted before it is read again. New releases appear on this cadence. + DiscographyMaxAge = 7 * 24 * time.Hour + // ReleasesMaxAge is how long a release group's editions and track lists + // are trusted. They change rarely once a group is out. + ReleasesMaxAge = 30 * 24 * time.Hour +) + +// ErrNotFound: MusicBrainz has no artist or release group under that id. +var ErrNotFound = errors.New("catalogue: not in MusicBrainz") + +// Service reads and refreshes the catalogue. +type Service struct { + pool *pgxpool.Pool + mb *musicbrainz.Client + logger *slog.Logger + now func() time.Time +} + +// New makes a Service. A nil client talks to the public MusicBrainz. +func New(pool *pgxpool.Pool, mb *musicbrainz.Client, logger *slog.Logger) *Service { + if mb == nil { + mb = &musicbrainz.Client{} + } + if logger == nil { + logger = slog.Default() + } + return &Service{pool: pool, mb: mb, logger: logger, now: time.Now} +} + +// MusicBrainz is the client the service asks, for callers that search. +func (s *Service) MusicBrainz() *musicbrainz.Client { return s.mb } + +// Discography is an artist and the release groups credited to them. +type Discography struct { + Artist dbq.MbArtist + Groups []dbq.MbReleaseGroup + // Stale: MusicBrainz could not be asked, so this is the last copy read. + Stale bool +} + +// Discography returns the artist's release groups, reading them from +// MusicBrainz first when they have never been read or are older than +// DiscographyMaxAge. When MusicBrainz is unreachable, a copy already held is +// returned marked Stale; with nothing held, the error is returned. +func (s *Service) Discography(ctx context.Context, artistMBID string) (Discography, error) { + return s.DiscographyWithin(ctx, artistMBID, DiscographyMaxAge) +} + +// DiscographyWithin is Discography with the age past which the held copy is +// read again: a followed artist is checked for new releases more often than +// one the catalogue merely knows. +func (s *Service) DiscographyWithin(ctx context.Context, artistMBID string, maxAge time.Duration) (Discography, error) { + q := dbq.New(s.pool) + artist, err := q.GetMBArtist(ctx, artistMBID) + held := err == nil && artist.DiscographyRefreshedAt.Valid + if err != nil && !errors.Is(err, pgx.ErrNoRows) { + return Discography{}, fmt.Errorf("catalogue: read artist: %w", err) + } + stale := false + if !held || s.now().Sub(artist.DiscographyRefreshedAt.Time) > maxAge { + if rerr := s.RefreshArtist(ctx, artistMBID); rerr != nil { + if !held || !errors.Is(rerr, musicbrainz.ErrUnreachable) { + return Discography{}, rerr + } + stale = true + } + if artist, err = q.GetMBArtist(ctx, artistMBID); err != nil { + return Discography{}, fmt.Errorf("catalogue: read artist: %w", err) + } + } + groups, err := q.ListMBArtistReleaseGroups(ctx, artistMBID) + if err != nil { + return Discography{}, fmt.Errorf("catalogue: read discography: %w", err) + } + return Discography{Artist: artist, Groups: groups, Stale: stale}, nil +} + +// RefreshArtist reads the artist and every release group credited to them, +// and replaces the held discography in one transaction. +func (s *Service) RefreshArtist(ctx context.Context, artistMBID string) error { + artist, err := s.mb.Artist(ctx, artistMBID) + if err != nil { + return mbErr(err) + } + groups, err := s.mb.ArtistReleaseGroups(ctx, artistMBID) + if err != nil { + return mbErr(err) + } + return pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + if err := q.UpsertMBArtist(ctx, dbq.UpsertMBArtistParams{ + Mbid: artist.MBID, Name: artist.Name, SortName: artist.SortName, + Disambiguation: artist.Disambiguation, ArtistType: artist.Type, Country: artist.Country, + }); err != nil { + return fmt.Errorf("catalogue: save artist: %w", err) + } + if err := q.DeleteMBArtistReleaseGroups(ctx, artist.MBID); err != nil { + return fmt.Errorf("catalogue: clear discography: %w", err) + } + for _, g := range groups { + if err := upsertGroup(ctx, q, g); err != nil { + return err + } + if err := q.AddMBArtistReleaseGroup(ctx, dbq.AddMBArtistReleaseGroupParams{ + ArtistMbid: artist.MBID, ReleaseGroupMbid: g.MBID, + }); err != nil { + return fmt.Errorf("catalogue: link release group: %w", err) + } + } + return q.MarkMBArtistDiscographyRefreshed(ctx, artist.MBID) + }) +} + +func upsertGroup(ctx context.Context, q *dbq.Queries, g musicbrainz.ReleaseGroup) error { + secondary := g.SecondaryTypes + if secondary == nil { + secondary = []string{} + } + if err := q.UpsertMBReleaseGroup(ctx, dbq.UpsertMBReleaseGroupParams{ + Mbid: g.MBID, Title: g.Title, PrimaryType: g.PrimaryType, SecondaryTypes: secondary, + FirstReleaseDate: g.FirstReleaseDate, ArtistCredit: g.ArtistCredit, + }); err != nil { + return fmt.Errorf("catalogue: save release group: %w", err) + } + return nil +} + +// Edition is one release of a group with its track list. +type Edition struct { + Release dbq.MbRelease + Tracks []dbq.MbReleaseTrack +} + +// Editions is a release group and its releases. +type Editions struct { + Group dbq.MbReleaseGroup + Editions []Edition + Stale bool +} + +// Releases returns a release group's releases with their track lists, read +// from MusicBrainz first when never read or older than ReleasesMaxAge. Like +// Discography, an unreachable MusicBrainz serves the held copy as Stale. +func (s *Service) Releases(ctx context.Context, groupMBID string) (Editions, error) { + q := dbq.New(s.pool) + group, err := q.GetMBReleaseGroup(ctx, groupMBID) + if err != nil && !errors.Is(err, pgx.ErrNoRows) { + return Editions{}, fmt.Errorf("catalogue: read release group: %w", err) + } + held := err == nil && group.ReleasesRefreshedAt.Valid + stale := false + if !held || s.now().Sub(group.ReleasesRefreshedAt.Time) > ReleasesMaxAge { + if rerr := s.RefreshReleaseGroup(ctx, groupMBID); rerr != nil { + if !held || !errors.Is(rerr, musicbrainz.ErrUnreachable) { + return Editions{}, rerr + } + stale = true + } + if group, err = q.GetMBReleaseGroup(ctx, groupMBID); err != nil { + return Editions{}, fmt.Errorf("catalogue: read release group: %w", err) + } + } + releases, err := q.ListMBReleasesForGroup(ctx, groupMBID) + if err != nil { + return Editions{}, fmt.Errorf("catalogue: read releases: %w", err) + } + tracks, err := q.ListMBReleaseTracksForGroup(ctx, groupMBID) + if err != nil { + return Editions{}, fmt.Errorf("catalogue: read track lists: %w", err) + } + byRelease := make(map[string][]dbq.MbReleaseTrack, len(releases)) + for _, t := range tracks { + byRelease[t.ReleaseMbid] = append(byRelease[t.ReleaseMbid], t) + } + out := Editions{Group: group, Stale: stale} + for _, r := range releases { + out.Editions = append(out.Editions, Edition{Release: r, Tracks: byRelease[r.Mbid]}) + } + return out, nil +} + +// RefreshReleaseGroup reads a group's releases and track lists and replaces +// the held copy in one transaction. A group not yet held is looked up first. +func (s *Service) RefreshReleaseGroup(ctx context.Context, groupMBID string) error { + q := dbq.New(s.pool) + var group *musicbrainz.ReleaseGroup + if _, err := q.GetMBReleaseGroup(ctx, groupMBID); errors.Is(err, pgx.ErrNoRows) { + g, gerr := s.mb.ReleaseGroup(ctx, groupMBID) + if gerr != nil { + return mbErr(gerr) + } + group = &g + } else if err != nil { + return fmt.Errorf("catalogue: read release group: %w", err) + } + releases, err := s.mb.ReleaseGroupReleases(ctx, groupMBID) + if err != nil { + return mbErr(err) + } + return pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + tq := dbq.New(tx) + if group != nil { + if err := upsertGroup(ctx, tq, *group); err != nil { + return err + } + } + if err := tq.DeleteMBReleasesForGroup(ctx, groupMBID); err != nil { + return fmt.Errorf("catalogue: clear releases: %w", err) + } + for _, r := range releases { + if err := tq.InsertMBRelease(ctx, dbq.InsertMBReleaseParams{ + Mbid: r.MBID, ReleaseGroupMbid: groupMBID, Title: r.Title, Status: r.Status, + ReleaseDate: r.Date, Country: r.Country, Disambiguation: r.Disambiguation, + Formats: r.Formats(), TrackCount: int32(r.TrackCount()), + }); err != nil { + return fmt.Errorf("catalogue: save release: %w", err) + } + for _, m := range r.Media { + for _, t := range m.Tracks { + if err := tq.InsertMBReleaseTrack(ctx, dbq.InsertMBReleaseTrackParams{ + ReleaseMbid: r.MBID, MediumPosition: int32(m.Position), Position: int32(t.Position), + MediumFormat: m.Format, Number: t.Number, Title: t.Title, + LengthMs: int32(t.LengthMs), RecordingMbid: t.RecordingMBID, + }); err != nil { + return fmt.Errorf("catalogue: save track: %w", err) + } + } + } + } + return tq.MarkMBReleaseGroupReleasesRefreshed(ctx, groupMBID) + }) +} + +// RefreshDue re-reads up to limit discographies that are older than +// DiscographyMaxAge, never-read first. It stops at the first unreachable +// MusicBrainz, since the rest would fail the same way. +func (s *Service) RefreshDue(ctx context.Context, limit int) (refreshed int, err error) { + due, err := dbq.New(s.pool).ListMBArtistsDueForRefresh(ctx, dbq.ListMBArtistsDueForRefreshParams{ + Before: pgtype.Timestamptz{Time: s.now().Add(-DiscographyMaxAge), Valid: true}, + MaxRows: int32(limit), + }) + if err != nil { + return 0, fmt.Errorf("catalogue: list due artists: %w", err) + } + for _, mbid := range due { + if err := s.RefreshArtist(ctx, mbid); err != nil { + if errors.Is(err, musicbrainz.ErrUnreachable) || ctx.Err() != nil { + return refreshed, err + } + // One artist MusicBrainz cannot describe (merged away, say) + // must not stop the rest. + s.logger.Warn("catalogue: refresh artist", "mbid", mbid, "err", err) + continue + } + refreshed++ + } + return refreshed, nil +} + +// RefreshInterval is how often Run looks for discographies due a re-read, +// and RefreshBatch how many it reads per look: 20 artists is about a minute +// of MusicBrainz's budget. +const ( + RefreshInterval = time.Hour + RefreshBatch = 20 +) + +// Run refreshes due discographies every RefreshInterval until ctx ends. +func (s *Service) Run(ctx context.Context) { + ticker := time.NewTicker(RefreshInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + n, err := s.RefreshDue(ctx, RefreshBatch) + switch { + case err != nil && ctx.Err() == nil: + s.logger.Warn("catalogue: refresh pass stopped", "refreshed", n, "err", err) + case n > 0: + s.logger.Info("catalogue: refresh pass complete", "refreshed", n) + } + } +} + +// mbErr maps MusicBrainz's not-found to the catalogue's; anything else passes +// through so callers can tell an unreachable MusicBrainz apart. +func mbErr(err error) error { + if errors.Is(err, musicbrainz.ErrNotFound) { + return fmt.Errorf("%w: %v", ErrNotFound, err) + } + return err +} diff --git a/internal/catalogue/catalogue_test.go b/internal/catalogue/catalogue_test.go new file mode 100644 index 00000000..5b495122 --- /dev/null +++ b/internal/catalogue/catalogue_test.go @@ -0,0 +1,247 @@ +package catalogue + +import ( + "context" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "os" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" +) + +func newPool(t *testing.T) *pgxpool.Pool { + t.Helper() + if testing.Short() { + t.Skip("skipping integration test in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(context.Background(), dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + dbtest.ResetDB(t, pool) + return pool +} + +const ( + gorillaz = "e21857d5-3256-4547-afb3-4b6ded592596" + deLaSoul = "4f2a6b9e-9ae6-4f34-9a41-3e9f2f7c4c1d" + demonDays = "0c4d5b5f-2f3c-3d31-9c4e-4b4f6b4c1a01" + feelGood = "0c4d5b5f-2f3c-3d31-9c4e-4b4f6b4c1a02" // credited to both + demonDaysR = "1a2b3c4d-0000-4000-8000-000000000001" + demonDaysJ = "1a2b3c4d-0000-4000-8000-000000000002" +) + +// fakeMB is a MusicBrainz with two artists, a collaboration between them, +// and one album with two editions. down makes every request a 503. +type fakeMB struct { + srv *httptest.Server + calls atomic.Int32 + down atomic.Bool +} + +func newFakeMB(t *testing.T) *fakeMB { + t.Helper() + f := &fakeMB{} + f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + f.calls.Add(1) + if f.down.Load() { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + q := r.URL.Query() + switch { + case r.URL.Path == "/artist/"+gorillaz: + _, _ = fmt.Fprintf(w, `{"id": %q, "name": "Gorillaz", "sort-name": "Gorillaz", "type": "Group", "country": "GB"}`, gorillaz) + case r.URL.Path == "/artist/"+deLaSoul: + _, _ = fmt.Fprintf(w, `{"id": %q, "name": "De La Soul", "type": "Group", "country": "US"}`, deLaSoul) + case r.URL.Path == "/release-group" && q.Get("artist") == gorillaz: + _, _ = fmt.Fprintf(w, `{"release-group-count": 2, "release-groups": [ + {"id": %q, "title": "Demon Days", "primary-type": "Album", "first-release-date": "2005-05-11", + "artist-credit": [{"name": "Gorillaz", "artist": {"id": %q}}]}, + {"id": %q, "title": "Feel Good Inc.", "primary-type": "Single", "secondary-types": [], "first-release-date": "2005-05-09", + "artist-credit": [{"name": "Gorillaz", "joinphrase": " feat. ", "artist": {"id": %q}}, {"name": "De La Soul", "artist": {"id": %q}}]} + ]}`, demonDays, gorillaz, feelGood, gorillaz, deLaSoul) + case r.URL.Path == "/release-group" && q.Get("artist") == deLaSoul: + _, _ = fmt.Fprintf(w, `{"release-group-count": 1, "release-groups": [ + {"id": %q, "title": "Feel Good Inc.", "primary-type": "Single", "first-release-date": "2005-05-09"} + ]}`, feelGood) + case r.URL.Path == "/release-group/"+demonDays: + _, _ = fmt.Fprintf(w, `{"id": %q, "title": "Demon Days", "primary-type": "Album", "first-release-date": "2005-05-11", + "artist-credit": [{"name": "Gorillaz", "artist": {"id": %q}}]}`, demonDays, gorillaz) + case r.URL.Path == "/release" && q.Get("release-group") == demonDays: + _, _ = fmt.Fprintf(w, `{"release-count": 2, "releases": [ + {"id": %q, "title": "Demon Days", "status": "Promotion", "date": "2005-04", "country": "JP", + "media": [{"position": 1, "format": "CD", "tracks": [ + {"position": 1, "number": "1", "title": "Intro", "length": 63000, "recording": {"id": "rec-intro"}}]}]}, + {"id": %q, "title": "Demon Days", "status": "Official", "date": "2005-05-11", "country": "GB", + "media": [{"position": 1, "format": "CD", "tracks": [ + {"position": 1, "number": "1", "title": "Intro", "length": 63000, "recording": {"id": "rec-intro"}}, + {"position": 2, "number": "2", "title": "Last Living Souls", "length": 190000, "recording": {"id": "rec-lls"}}]}]} + ]}`, demonDaysJ, demonDaysR) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(f.srv.Close) + return f +} + +func (f *fakeMB) client() *musicbrainz.Client { + return &musicbrainz.Client{BaseURL: f.srv.URL, Pace: &musicbrainz.Pacer{}, Retries: -1} +} + +func TestDiscography_ReadsOnceThenServesTheHeldCopy_Integration(t *testing.T) { + ctx := context.Background() + mb := newFakeMB(t) + s := New(newPool(t), mb.client(), nil) + + d, err := s.Discography(ctx, gorillaz) + if err != nil { + t.Fatal(err) + } + if d.Artist.Name != "Gorillaz" || d.Artist.Country != "GB" || len(d.Groups) != 2 || d.Stale { + t.Fatalf("discography = %+v", d) + } + // Oldest first: the single came out two days before the album. + if d.Groups[0].Mbid != feelGood || d.Groups[1].PrimaryType != "Album" { + t.Errorf("order = %s, %s", d.Groups[0].Title, d.Groups[1].Title) + } + if d.Groups[0].ArtistCredit != "Gorillaz feat. De La Soul" { + t.Errorf("credit = %q", d.Groups[0].ArtistCredit) + } + + before := mb.calls.Load() + if _, err := s.Discography(ctx, gorillaz); err != nil { + t.Fatal(err) + } + if mb.calls.Load() != before { + t.Error("a fresh discography should be served without asking MusicBrainz") + } + + // The collaboration belongs to both artists. + dls, err := s.Discography(ctx, deLaSoul) + if err != nil { + t.Fatal(err) + } + if len(dls.Groups) != 1 || dls.Groups[0].Mbid != feelGood { + t.Errorf("De La Soul's discography = %+v", dls.Groups) + } + if again, _ := s.Discography(ctx, gorillaz); len(again.Groups) != 2 { + t.Error("refreshing one artist must not take a shared group from the other") + } +} + +func TestDiscography_UnreachableServesStaleOrFails_Integration(t *testing.T) { + ctx := context.Background() + mb := newFakeMB(t) + pool := newPool(t) + s := New(pool, mb.client(), nil) + + mb.down.Store(true) + if _, err := s.Discography(ctx, gorillaz); !errors.Is(err, musicbrainz.ErrUnreachable) { + t.Fatalf("nothing held and MusicBrainz down: err = %v, want ErrUnreachable", err) + } + + mb.down.Store(false) + if _, err := s.Discography(ctx, gorillaz); err != nil { + t.Fatal(err) + } + // A week and a day later, MusicBrainz is down: the held copy is served. + s.now = func() time.Time { return time.Now().Add(DiscographyMaxAge + 24*time.Hour) } + mb.down.Store(true) + d, err := s.Discography(ctx, gorillaz) + if err != nil || !d.Stale || len(d.Groups) != 2 { + t.Fatalf("stale copy: %+v, %v", d, err) + } +} + +func TestDiscography_UnknownArtistIsNotFound_Integration(t *testing.T) { + s := New(newPool(t), newFakeMB(t).client(), nil) + if _, err := s.Discography(context.Background(), "00000000-0000-4000-8000-000000000000"); !errors.Is(err, ErrNotFound) { + t.Fatalf("err = %v, want ErrNotFound", err) + } +} + +func TestReleases_ReadsEditionsAndTrackLists_Integration(t *testing.T) { + ctx := context.Background() + mb := newFakeMB(t) + s := New(newPool(t), mb.client(), nil) + + // The group isn't held yet: it is looked up first. + e, err := s.Releases(ctx, demonDays) + if err != nil { + t.Fatal(err) + } + if e.Group.Title != "Demon Days" || len(e.Editions) != 2 { + t.Fatalf("editions = %+v", e) + } + official := e.Editions[0] + if official.Release.Mbid != demonDaysR || official.Release.TrackCount != 2 || official.Release.Formats != "CD" { + t.Errorf("the official edition should come first: %+v", official.Release) + } + if len(official.Tracks) != 2 || official.Tracks[1].RecordingMbid != "rec-lls" || official.Tracks[1].LengthMs != 190000 { + t.Errorf("tracks = %+v", official.Tracks) + } + + before := mb.calls.Load() + if _, err := s.Releases(ctx, demonDays); err != nil { + t.Fatal(err) + } + if mb.calls.Load() != before { + t.Error("fresh editions should be served without asking MusicBrainz") + } + // Refreshing replaces rather than adds. + if err := s.RefreshReleaseGroup(ctx, demonDays); err != nil { + t.Fatal(err) + } + if again, _ := s.Releases(ctx, demonDays); len(again.Editions) != 2 || len(again.Editions[0].Tracks) != 2 { + t.Errorf("after a refresh: %+v", again) + } +} + +func TestRefreshDue_ReadsWhatIsOldAndStopsWhenMusicBrainzIsDown_Integration(t *testing.T) { + ctx := context.Background() + mb := newFakeMB(t) + s := New(newPool(t), mb.client(), nil) + for _, a := range []string{gorillaz, deLaSoul} { + if err := s.RefreshArtist(ctx, a); err != nil { + t.Fatal(err) + } + } + if n, err := s.RefreshDue(ctx, 10); err != nil || n != 0 { + t.Fatalf("nothing is due yet: %d, %v", n, err) + } + + s.now = func() time.Time { return time.Now().Add(DiscographyMaxAge + time.Hour) } + if n, err := s.RefreshDue(ctx, 1); err != nil || n != 1 { + t.Errorf("one due, limit one: %d, %v", n, err) + } + mb.down.Store(true) + n, err := s.RefreshDue(ctx, 10) + if !errors.Is(err, musicbrainz.ErrUnreachable) || n != 0 { + t.Errorf("MusicBrainz down: %d, %v", n, err) + } + if !strings.Contains(err.Error(), "503") { + t.Errorf("the error should say what MusicBrainz answered: %v", err) + } +} diff --git a/internal/config/config.go b/internal/config/config.go index dd2e4ac8..645a21e3 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -24,12 +24,10 @@ type Config struct { Recommendation RecommendationConfig `yaml:"recommendation"` Branding BrandingConfig `yaml:"branding"` // StreamSecret is the base64-decoded HMAC key used to sign UPnP / - // Sonos stream URLs (see internal/api/stream_token.go and - // docs/superpowers/specs/2026-06-03-android-output-picker-upnp-design.md). - // Sourced from MINSTREL_STREAM_SECRET (base64-encoded), with a - // per-machine fallback persisted at /stream_secret. - // Not serialized to YAML — operator-machine-scoped runtime state, not - // a user-facing setting. Never log the contents. + // Sonos stream URLs (see internal/api/stream_token.go). Sourced from + // MINSTREL_STREAM_SECRET (base64-encoded), with a per-machine fallback + // persisted at /stream_secret. Not read from YAML: it + // is per-machine state, not a setting. Never log the contents. StreamSecret []byte `yaml:"-"` } @@ -37,12 +35,8 @@ type ServerConfig struct { Address string `yaml:"address"` } -// StorageConfig points at on-disk locations the server uses for cached -// runtime artifacts. Currently: -// - DataDir hosts /playlist_covers/{playlist_id}.jpg (M7 #352). -// -// More entries may grow here (e.g., transcode cache) without breaking -// existing layouts. +// StorageConfig points at the on-disk root for generated files: playlist +// cover collages, artwork, caches and the stream secret. type StorageConfig struct { // DataDir is the root for cached artifacts. Default: "./data" (relative // to the working directory). Operators in production should set an @@ -88,10 +82,9 @@ type EventsConfig struct { SkipMaxDurationPlayedMs int `yaml:"skip_max_duration_played_ms"` } -// RecommendationConfig holds the radio path's operational knobs. The -// scoring WEIGHTS moved to DB-backed admin settings (#1250, -// internal/recsettings) — YAML is bootstrap-only; anything an operator -// tunes lives in the UI with live effect. +// RecommendationConfig holds the radio path's operational knobs. The scoring +// weights are DB-backed admin settings (internal/recsettings), tuned in the +// UI with live effect. type RecommendationConfig struct { RecentlyPlayedHours int `yaml:"recently_played_hours"` RadioSize int `yaml:"radio_size"` @@ -106,7 +99,7 @@ func Default() Config { Log: LogConfig{Level: "info", Format: "text"}, Branding: BrandingConfig{ AppName: "Minstrel", - Description: "Self-hosted music server with Subsonic compatibility, smart shuffle, and Lidarr integration.", + Description: "Self-hosted music server with Subsonic compatibility, smart shuffle, and music acquisition.", }, Events: EventsConfig{ SessionTimeoutMinutes: 30, @@ -119,10 +112,10 @@ func Default() Config { RadioSizeMax: 200, }, Library: LibraryConfig{ - // On by default so a fresh install populates its library on - // first boot without the operator hunting for a "scan now" - // button. Scans are incremental (unchanged files skipped by - // mtime), so every later restart is just a cheap directory walk. + // On by default so a fresh install populates its library on first + // boot with no manual step. Scans are incremental (unchanged files + // skipped by mtime), so every later restart is just a cheap + // directory walk. ScanOnStartup: true, CoverArtFromMBCAA: true, }, @@ -159,15 +152,15 @@ const streamSecretBytes = 64 const streamSecretFile = "stream_secret" // resolveStreamSecret populates cfg.StreamSecret using, in order: -// 1. MINSTREL_STREAM_SECRET env var (operator-supplied, base64-url -// encoded — accepts either padded or raw form). +// 1. MINSTREL_STREAM_SECRET env var (base64-url encoded — accepts either +// padded or raw form). // 2. /stream_secret if present (auto-generated on a // previous boot; survives restarts so signed URLs stay valid). // 3. Auto-generate streamSecretBytes random bytes, persist them to that // file at 0600, then use them. // -// Logs (only) when an auto-generation happens so the operator can spot -// it during first-boot. Never logs the contents. +// Logs (only) when an auto-generation happens, so it shows on first boot. Never +// logs the contents. // // The loader runs before slog is initialized in cmd/minstrel/main.go; // log.Printf is the project's pre-logger convention. diff --git a/internal/coplay/worker.go b/internal/coplay/worker.go index ed2c0435..732508e9 100644 --- a/internal/coplay/worker.go +++ b/internal/coplay/worker.go @@ -1,11 +1,10 @@ -// Package coplay builds household co-play similarity edges (#1533, milestone -// #160 Opt 5). Minstrel is a single shared-library, multi-user server (no -// per-user library ACL), so the "household" is the whole instance's user set. -// A periodic worker recomputes artist–artist co-occurrence from play_events -// entirely in SQL (no external API) and stores the edges in artist_similarity -// under source='user_cooccurrence' — the pre-provisioned co-occurrence source -// from the 0009 schema — which the radio/mix candidate pool consumes as a -// collaborative arm. Single-user servers produce no edges (the >= 2 co-player +// Package coplay builds household co-play similarity edges. Minstrel is a +// single shared-library, multi-user server (no per-user library ACL), so the +// "household" is the whole instance's user set. A periodic worker recomputes +// artist–artist co-occurrence from play_events entirely in SQL (no external +// API) and stores the edges in artist_similarity under +// source='user_cooccurrence', which the radio/mix candidate pool consumes as +// a collaborative arm. Single-user servers produce no edges (the >= 2 co-player // gate), so the feature is a graceful no-op there. package coplay diff --git a/internal/coverart/artist_enricher.go b/internal/coverart/artist_enricher.go index 48187a42..2af0a052 100644 --- a/internal/coverart/artist_enricher.go +++ b/internal/coverart/artist_enricher.go @@ -141,11 +141,11 @@ func (e *Enricher) recordArtistArt(ctx context.Context, artistID pgtype.UUID, th // // Emits a single Info log line at end-of-batch with a category // breakdown (succeeded / settled-none / no-MBID / left-NULL / -// errored). The JSON tally written to scan_runs keeps the existing -// processed/succeeded/failed shape for backwards compat with the -// admin overview UI; the log line is the operator's diagnostic. -// limit: 0 = stage disabled; >0 = bounded; <0 = unbounded (#388 — -// no global cap; remote providers self-throttle per provider). +// errored). The JSON tally written to scan_runs keeps the +// processed/succeeded/failed shape the admin overview reads; the log line +// is the diagnostic. +// limit: 0 = stage disabled; >0 = bounded; <0 = unbounded (remote +// providers pace themselves). func (e *Enricher) EnrichArtistBatch(ctx context.Context, limit int, dataDir string, progressCb func(processed, succeeded, failed int)) (processed, succeeded, failed int, err error) { if limit == 0 { diff --git a/internal/coverart/embedded.go b/internal/coverart/embedded.go index ce00e731..f49ef1a9 100644 --- a/internal/coverart/embedded.go +++ b/internal/coverart/embedded.go @@ -23,7 +23,7 @@ var ErrNoEmbeddedArt = errors.New("coverart: no embedded album art") // usable picture; returns a wrapped error for I/O or tag-parse // failures so callers can distinguish "no art" from "couldn't read". // -// Picard- and Lidarr-tagged libraries typically carry the canonical +// Picard-tagged libraries typically carry the canonical // art in this block already — extracting it beats every external API // guess for accuracy and avoids a network round-trip. func ExtractEmbeddedAlbumArt(path string) ([]byte, error) { diff --git a/internal/coverart/enricher.go b/internal/coverart/enricher.go index cb7dda3e..bcce89c1 100644 --- a/internal/coverart/enricher.go +++ b/internal/coverart/enricher.go @@ -91,7 +91,7 @@ func (e *Enricher) EnrichAlbum(ctx context.Context, albumID pgtype.UUID) error { // Embedded layer: read the picture stored inside the audio file's // tag block (ID3v2 APIC, FLAC METADATA_BLOCK_PICTURE, MP4 covr). - // Picard/Lidarr-tagged libraries typically carry the canonical art + // Picard-tagged libraries typically carry the canonical art // here, so this beats every external provider for accuracy and // avoids the network round-trip. if row.TrackFilePath != nil && *row.TrackFilePath != "" { @@ -210,13 +210,12 @@ func (e *Enricher) recordAlbumCover(ctx context.Context, albumID pgtype.UUID, pa // // Emits a single Info log line at end-of-batch with a category // breakdown (succeeded / settled-none / left-NULL / errored). The -// JSON tally written to scan_runs keeps the existing -// processed/succeeded/failed shape for backwards compat with the -// admin overview UI; the log line is the operator's diagnostic. +// JSON tally written to scan_runs keeps the processed/succeeded/failed +// shape the admin overview reads; the log line is the diagnostic. // limit semantics: 0 = stage disabled (skip); >0 = bounded batch; -// <0 = unbounded — walk every album needing art. The global cap was -// removed (#388); external providers self-throttle via their -// per-provider httpClient MinInterval, local sources run at disk speed. +// <0 = unbounded — walk every album needing art. External providers pace +// themselves via their httpClient MinInterval; local sources run at disk +// speed. func (e *Enricher) EnrichBatch(ctx context.Context, limit int, progressCb func(processed, succeeded, failed int)) (processed, succeeded, failed int, err error) { if limit == 0 { @@ -302,7 +301,7 @@ func (e *Enricher) logBatchSummary(stage string, eligible, processed, succeeded, // admin bulk-retry endpoint. Clears 'none' to NULL first so // EnrichAlbum picks them up. // limit: 0 = no-op; >0 = bounded; <0 = unbounded (retry every -// missing/none album). Admin bulk-retry passes <0 post-#388. +// missing/none album). Admin bulk-retry passes <0. func (e *Enricher) EnrichRetryMissing(ctx context.Context, limit int) (int, error) { if limit == 0 { return 0, nil @@ -346,9 +345,8 @@ func (e *Enricher) RetryAlbum(ctx context.Context, albumID pgtype.UUID) error { } return fmt.Errorf("get album: %w", err) } - // Delete the file only if we previously wrote it (mbcaa or - // theaudiodb). Operator sidecars stay; embedded extracts (future) - // live in a managed cache with its own lifecycle. + // Delete the file only if we wrote it (mbcaa or theaudiodb). Sidecar + // images placed beside the music stay. if row.CoverArtSource != nil && (*row.CoverArtSource == "mbcaa" || *row.CoverArtSource == "theaudiodb") && row.CoverArtPath != nil && *row.CoverArtPath != "" { @@ -364,7 +362,7 @@ func (e *Enricher) RetryAlbum(ctx context.Context, albumID pgtype.UUID) error { return e.EnrichAlbum(ctx, albumID) } -// uuidString carries over from the previous enricher. +// uuidString renders a UUID as text for logs, or "" when it is not set. func uuidString(u pgtype.UUID) string { if !u.Valid { return "" diff --git a/internal/coverart/provider.go b/internal/coverart/provider.go index 92303ece..f7b07e44 100644 --- a/internal/coverart/provider.go +++ b/internal/coverart/provider.go @@ -16,8 +16,7 @@ import ( type Provider interface { // ID is a stable lowercase token persisted in DB columns // (cover_art_source / artist_art_source) and the - // cover_art_provider_settings.provider_id key, plus referenced - // in the operator-facing UI. + // cover_art_provider_settings.provider_id key, and shown in the admin UI. ID() string // DisplayName is the human-readable label shown in the admin @@ -33,10 +32,10 @@ type Provider interface { RequiresAPIKey() bool // DefaultEnabled is whether a freshly-installed Minstrel ships - // with this provider on. Both v1 providers return true. + // with this provider on. Providers that need an API key ship off. DefaultEnabled() bool - // Configure receives the operator's settings (api_key, enabled + // Configure receives the admin's settings (api_key, enabled // flag) and applies them. Called at boot during reconciliation // and on every admin PATCH. Implementations must be idempotent // and goroutine-safe (the enricher may be reading concurrently). @@ -82,6 +81,14 @@ type AlbumCoverProvider interface { FetchAlbumCover(ctx context.Context, ref AlbumRef) ([]byte, error) } +// ReleaseGroupCoverProvider is an opt-in capability for sources that can +// answer by MusicBrainz release group, which is how the release catalogue +// names an album before any edition of it is held. +type ReleaseGroupCoverProvider interface { + AlbumCoverProvider + FetchReleaseGroupCover(ctx context.Context, groupMBID string) ([]byte, error) +} + // ArtistArtProvider is an opt-in capability: providers implementing // it can fetch artist images. Returns thumb + fanart atomically at // the JSON metadata step (one round-trip yields both URLs); the two @@ -95,8 +102,8 @@ type ArtistArtProvider interface { // TestableProvider is an opt-in capability: providers that can answer // "is my config working?" without performing a full enrichment cycle. -// Used by the admin Test-Connection button. v1: both MBCAA and -// TheAudioDB implement this. +// Used by the admin Test-Connection button. MBCAA and TheAudioDB +// implement this. type TestableProvider interface { Provider TestConnection(ctx context.Context) error diff --git a/internal/coverart/provider_mbcaa.go b/internal/coverart/provider_mbcaa.go index a448555d..0bcc1f25 100644 --- a/internal/coverart/provider_mbcaa.go +++ b/internal/coverart/provider_mbcaa.go @@ -11,7 +11,7 @@ import ( "time" ) -// FetcherConfig assembles the operator-tunable knobs for the MBCAA +// FetcherConfig assembles the tunable knobs for the MBCAA // HTTP client. Kept exported because main.go's boot wiring constructs // one to pass into NewMBCAAProviderFromConfig. type FetcherConfig struct { @@ -92,10 +92,22 @@ func (p *mbcaaProvider) Configure(s ProviderSettings) error { // release MBID. MBCAA is MBID-only; returns ErrNotFound when // ref.MBID is empty. Returns ErrNotFound on 404 or ErrTransient otherwise. func (p *mbcaaProvider) FetchAlbumCover(ctx context.Context, ref AlbumRef) ([]byte, error) { + return p.fetchFront(ctx, "release", ref.MBID) +} + +// FetchReleaseGroupCover retrieves the 500px front cover the Cover Art +// Archive chose for a release group: what a search result or a wanted album +// shows before any edition of it is held. +func (p *mbcaaProvider) FetchReleaseGroupCover(ctx context.Context, groupMBID string) ([]byte, error) { + return p.fetchFront(ctx, "release-group", groupMBID) +} + +// fetchFront fetches /{entity}/{mbid}/front-500, honouring the rate limit. +func (p *mbcaaProvider) fetchFront(ctx context.Context, entity, mbid string) ([]byte, error) { if !p.enabled.Load() { return nil, ErrNotFound // defensive; enricher already filters on enabled } - if ref.MBID == "" { + if mbid == "" { return nil, ErrNotFound // MBCAA is MBID-only } @@ -119,7 +131,7 @@ func (p *mbcaaProvider) FetchAlbumCover(ctx context.Context, ref AlbumRef) ([]by p.lastCall = time.Now() p.mu.Unlock() - url := fmt.Sprintf("%s/release/%s/front-500", cfg.BaseURL, ref.MBID) + url := fmt.Sprintf("%s/%s/%s/front-500", cfg.BaseURL, entity, mbid) req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { return nil, fmt.Errorf("coverart: build request: %w", err) @@ -157,3 +169,5 @@ func (p *mbcaaProvider) TestConnection(ctx context.Context) error { } return err } + +var _ ReleaseGroupCoverProvider = (*mbcaaProvider)(nil) diff --git a/internal/coverart/provider_mbcaa_test.go b/internal/coverart/provider_mbcaa_test.go index b837617e..18090712 100644 --- a/internal/coverart/provider_mbcaa_test.go +++ b/internal/coverart/provider_mbcaa_test.go @@ -166,3 +166,24 @@ func TestNewMBCAAProviderFromConfig_ReplacesRegisteredConfig(t *testing.T) { t.Errorf("cfg.UserAgent = %q, want %q", gotUA, "Minstrel/test-replaced (x)") } } + +func TestMBCAAProvider_FetchReleaseGroupCover_AsksByGroup(t *testing.T) { + var path string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + path = r.URL.Path + _, _ = w.Write([]byte("jpeg")) + })) + defer srv.Close() + p := newMBCAAProviderForTest(t, srv.URL) + + body, err := p.FetchReleaseGroupCover(context.Background(), "rg-1") + if err != nil || string(body) != "jpeg" { + t.Fatalf("body %q, err %v", body, err) + } + if path != "/release-group/rg-1/front-500" { + t.Errorf("path = %q", path) + } + if _, err := p.FetchReleaseGroupCover(context.Background(), ""); !errors.Is(err, ErrNotFound) { + t.Errorf("no id: err = %v, want ErrNotFound", err) + } +} diff --git a/internal/coverart/provider_theaudiodb.go b/internal/coverart/provider_theaudiodb.go index 984502e1..2d9d8bff 100644 --- a/internal/coverart/provider_theaudiodb.go +++ b/internal/coverart/provider_theaudiodb.go @@ -23,8 +23,8 @@ const ( // theAudioDBMinPeriod enforces TheAudioDB's documented free/test // rate limit (2 req/sec). Patreon supporter keys allow higher - // rates; we don't differentiate in v1 — operators with supporter - // keys experience the conservative 2 req/sec ceiling. + // rates; we don't differentiate — supporter keys get the same + // conservative 2 req/sec ceiling. theAudioDBMinPeriod = 500 * time.Millisecond // theAudioDBMaxRetries is the number of retry attempts on 429 / diff --git a/internal/coverart/settings.go b/internal/coverart/settings.go index ab025cb7..d2e51b14 100644 --- a/internal/coverart/settings.go +++ b/internal/coverart/settings.go @@ -270,10 +270,9 @@ func enabledSetSignature(enabledIDs map[string]bool) string { // unchanged, returns (currentVersion, false, nil) without touching the DB. // // Used at boot to ensure rows previously settled to 'none' get re-tried -// once after a provider chain change (e.g. operator updates the binary -// and a new provider is registered, or one is removed). The version -// bump triggers eligibility for 'none' rows on the next enrichment pass -// without requiring a manual operator action. +// once after a provider chain change (an update registers a new provider, +// or removes one). The version bump makes 'none' rows eligible on the next +// enrichment pass with no manual step. func (s *SettingsService) BumpVersionIfProvidersChanged(ctx context.Context) (version int32, bumped bool, err error) { hash := registeredProvidersHash() q := dbq.New(s.pool) @@ -313,7 +312,7 @@ func registeredProvidersHash() string { // BumpVersion unconditionally increments cover_art_sources_meta.current_version // and returns the new value. Used by the manual "Re-search missing art" -// admin endpoint when the operator wants every 'none' row re-attempted +// admin endpoint, to re-attempt every 'none' row // on the next enrichment pass. // // Distinct from BumpVersionIfProvidersChanged: that method is the boot- diff --git a/internal/db/dbq/acoustid.sql.go b/internal/db/dbq/acoustid.sql.go index 7aba72b8..6315d96e 100644 --- a/internal/db/dbq/acoustid.sql.go +++ b/internal/db/dbq/acoustid.sql.go @@ -120,7 +120,7 @@ type ListTracksNeedingAcoustIDLookupRow struct { DurationMs int32 } -// AcoustID recording-id lookup (M401). Migration 0069 says what each +// AcoustID recording-id lookup. Migration 0069 says what each // lookup state means. // The lookup worker's queue: present tracks with no tag MBID and no lookup // since their file last changed. A looked-up id is included once its row is @@ -257,7 +257,7 @@ type SetTrackMbidFromAcoustIDParams struct { ID pgtype.UUID } -// Never over a tag id (D4): the guard is here, not only in the worker, so a +// Never over a tag id: the guard is here, not only in the worker, so a // tag the scan read between the worker's list and this write still wins. // updated_at is left alone: it gates the scan's mtime skip, and moving it // past a file's mtime would hide a change made just before the lookup. diff --git a/internal/db/dbq/acquisition.sql.go b/internal/db/dbq/acquisition.sql.go new file mode 100644 index 00000000..d5ff1b48 --- /dev/null +++ b/internal/db/dbq/acquisition.sql.go @@ -0,0 +1,362 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: acquisition.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const createAcquisitionService = `-- name: CreateAcquisitionService :one +INSERT INTO acquisition_services + (code, kind, protocol, label, category, timeout_ms, searches_per_hour) +VALUES + ($1, $2, $3, $4, + $5, $6, $7) +ON CONFLICT (code) DO NOTHING +RETURNING code, kind, protocol, label, enabled, base_url, api_key, username, password, category, timeout_ms, serves_search, serves_downloads, serves_enrichment, local_path, remote_path, searches_per_hour, created_at, updated_at +` + +type CreateAcquisitionServiceParams struct { + Code string + Kind string + Protocol string + Label string + Category string + TimeoutMs int32 + SearchesPerHour int32 +} + +// ON CONFLICT so the caller can try the next code when this one is taken. +func (q *Queries) CreateAcquisitionService(ctx context.Context, arg CreateAcquisitionServiceParams) (AcquisitionService, error) { + row := q.db.QueryRow(ctx, createAcquisitionService, + arg.Code, + arg.Kind, + arg.Protocol, + arg.Label, + arg.Category, + arg.TimeoutMs, + arg.SearchesPerHour, + ) + var i AcquisitionService + err := row.Scan( + &i.Code, + &i.Kind, + &i.Protocol, + &i.Label, + &i.Enabled, + &i.BaseUrl, + &i.ApiKey, + &i.Username, + &i.Password, + &i.Category, + &i.TimeoutMs, + &i.ServesSearch, + &i.ServesDownloads, + &i.ServesEnrichment, + &i.LocalPath, + &i.RemotePath, + &i.SearchesPerHour, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const deleteAcquisitionService = `-- name: DeleteAcquisitionService :execrows +DELETE FROM acquisition_services WHERE code = $1 +` + +func (q *Queries) DeleteAcquisitionService(ctx context.Context, code string) (int64, error) { + result, err := q.db.Exec(ctx, deleteAcquisitionService, code) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const getAcquisitionService = `-- name: GetAcquisitionService :one +SELECT code, kind, protocol, label, enabled, base_url, api_key, username, password, category, timeout_ms, serves_search, serves_downloads, serves_enrichment, local_path, remote_path, searches_per_hour, created_at, updated_at FROM acquisition_services WHERE code = $1 +` + +// The full row, credentials included. For building a client, never for a +// response. +func (q *Queries) GetAcquisitionService(ctx context.Context, code string) (AcquisitionService, error) { + row := q.db.QueryRow(ctx, getAcquisitionService, code) + var i AcquisitionService + err := row.Scan( + &i.Code, + &i.Kind, + &i.Protocol, + &i.Label, + &i.Enabled, + &i.BaseUrl, + &i.ApiKey, + &i.Username, + &i.Password, + &i.Category, + &i.TimeoutMs, + &i.ServesSearch, + &i.ServesDownloads, + &i.ServesEnrichment, + &i.LocalPath, + &i.RemotePath, + &i.SearchesPerHour, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const listAcquisitionServices = `-- name: ListAcquisitionServices :many + +SELECT code, kind, protocol, label, enabled, base_url, + (api_key <> '')::boolean AS has_api_key, + (password <> '')::boolean AS has_password, + username, category, timeout_ms, + serves_search, serves_downloads, serves_enrichment, + local_path, remote_path, searches_per_hour, + created_at, updated_at +FROM acquisition_services +ORDER BY kind, label, code +` + +type ListAcquisitionServicesRow struct { + Code string + Kind string + Protocol string + Label string + Enabled bool + BaseUrl string + HasApiKey bool + HasPassword bool + Username string + Category string + TimeoutMs int32 + ServesSearch bool + ServesDownloads bool + ServesEnrichment bool + LocalPath string + RemotePath string + SearchesPerHour int32 + CreatedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz +} + +// Indexers and download clients. +// Returns no secret, only whether one is set: this feeds the settings page. +func (q *Queries) ListAcquisitionServices(ctx context.Context) ([]ListAcquisitionServicesRow, error) { + rows, err := q.db.Query(ctx, listAcquisitionServices) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListAcquisitionServicesRow + for rows.Next() { + var i ListAcquisitionServicesRow + if err := rows.Scan( + &i.Code, + &i.Kind, + &i.Protocol, + &i.Label, + &i.Enabled, + &i.BaseUrl, + &i.HasApiKey, + &i.HasPassword, + &i.Username, + &i.Category, + &i.TimeoutMs, + &i.ServesSearch, + &i.ServesDownloads, + &i.ServesEnrichment, + &i.LocalPath, + &i.RemotePath, + &i.SearchesPerHour, + &i.CreatedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listUsableAcquisitionServices = `-- name: ListUsableAcquisitionServices :many +SELECT code, kind, protocol, label, enabled, base_url, api_key, username, password, category, timeout_ms, serves_search, serves_downloads, serves_enrichment, local_path, remote_path, searches_per_hour, created_at, updated_at FROM acquisition_services +WHERE enabled AND base_url <> '' + AND CASE WHEN protocol = 'exchange' THEN + CASE $1::text + WHEN 'search' THEN serves_search + WHEN 'download' THEN serves_downloads + WHEN 'enrichment' THEN serves_enrichment + ELSE false + END + ELSE kind = $1::text + END +ORDER BY code +` + +// Every enabled, addressed service doing one job. An exchange row does a job +// when its switch for it is on; any other row does the job its kind names. +// Ordered by code so a fan-out is deterministic. +func (q *Queries) ListUsableAcquisitionServices(ctx context.Context, job string) ([]AcquisitionService, error) { + rows, err := q.db.Query(ctx, listUsableAcquisitionServices, job) + if err != nil { + return nil, err + } + defer rows.Close() + var items []AcquisitionService + for rows.Next() { + var i AcquisitionService + if err := rows.Scan( + &i.Code, + &i.Kind, + &i.Protocol, + &i.Label, + &i.Enabled, + &i.BaseUrl, + &i.ApiKey, + &i.Username, + &i.Password, + &i.Category, + &i.TimeoutMs, + &i.ServesSearch, + &i.ServesDownloads, + &i.ServesEnrichment, + &i.LocalPath, + &i.RemotePath, + &i.SearchesPerHour, + &i.CreatedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const setAcquisitionServiceAPIKey = `-- name: SetAcquisitionServiceAPIKey :execrows +UPDATE acquisition_services SET api_key = $1, updated_at = now() +WHERE code = $2 +` + +type SetAcquisitionServiceAPIKeyParams struct { + ApiKey string + Code string +} + +func (q *Queries) SetAcquisitionServiceAPIKey(ctx context.Context, arg SetAcquisitionServiceAPIKeyParams) (int64, error) { + result, err := q.db.Exec(ctx, setAcquisitionServiceAPIKey, arg.ApiKey, arg.Code) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const setAcquisitionServicePassword = `-- name: SetAcquisitionServicePassword :execrows +UPDATE acquisition_services SET password = $1, updated_at = now() +WHERE code = $2 +` + +type SetAcquisitionServicePasswordParams struct { + Password string + Code string +} + +func (q *Queries) SetAcquisitionServicePassword(ctx context.Context, arg SetAcquisitionServicePasswordParams) (int64, error) { + result, err := q.db.Exec(ctx, setAcquisitionServicePassword, arg.Password, arg.Code) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const updateAcquisitionService = `-- name: UpdateAcquisitionService :one +UPDATE acquisition_services SET + label = $1, + enabled = $2, + base_url = $3, + username = $4, + category = $5, + timeout_ms = $6, + serves_search = $7, + serves_downloads = $8, + serves_enrichment = $9, + local_path = $10, + remote_path = $11, + searches_per_hour = $12, + updated_at = now() +WHERE code = $13 +RETURNING code, kind, protocol, label, enabled, base_url, api_key, username, password, category, timeout_ms, serves_search, serves_downloads, serves_enrichment, local_path, remote_path, searches_per_hour, created_at, updated_at +` + +type UpdateAcquisitionServiceParams struct { + Label string + Enabled bool + BaseUrl string + Username string + Category string + TimeoutMs int32 + ServesSearch bool + ServesDownloads bool + ServesEnrichment bool + LocalPath string + RemotePath string + SearchesPerHour int32 + Code string +} + +// Everything on the settings form except the credentials, which have their +// own write-only calls. +func (q *Queries) UpdateAcquisitionService(ctx context.Context, arg UpdateAcquisitionServiceParams) (AcquisitionService, error) { + row := q.db.QueryRow(ctx, updateAcquisitionService, + arg.Label, + arg.Enabled, + arg.BaseUrl, + arg.Username, + arg.Category, + arg.TimeoutMs, + arg.ServesSearch, + arg.ServesDownloads, + arg.ServesEnrichment, + arg.LocalPath, + arg.RemotePath, + arg.SearchesPerHour, + arg.Code, + ) + var i AcquisitionService + err := row.Scan( + &i.Code, + &i.Kind, + &i.Protocol, + &i.Label, + &i.Enabled, + &i.BaseUrl, + &i.ApiKey, + &i.Username, + &i.Password, + &i.Category, + &i.TimeoutMs, + &i.ServesSearch, + &i.ServesDownloads, + &i.ServesEnrichment, + &i.LocalPath, + &i.RemotePath, + &i.SearchesPerHour, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} diff --git a/internal/db/dbq/admin_status.sql.go b/internal/db/dbq/admin_status.sql.go new file mode 100644 index 00000000..84b47755 --- /dev/null +++ b/internal/db/dbq/admin_status.sql.go @@ -0,0 +1,168 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: admin_status.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const downloadsStatus = `-- name: DownloadsStatus :one +SELECT COUNT(*) FILTER (WHERE ended_at IS NULL AND import_state IN ('', 'downloading', 'waiting', 'importing'))::bigint AS running, + COUNT(*) FILTER (WHERE ended_at IS NULL AND import_state IN ('review', 'failed'))::bigint AS needs_you, + MIN(COALESCE(import_updated_at, grabbed_at)) FILTER (WHERE ended_at IS NULL AND import_state IN ('review', 'failed'))::timestamptz AS needs_you_since, + COUNT(*) FILTER (WHERE imported_at > now() - INTERVAL '24 hours')::bigint AS imported_day, + COUNT(*) FILTER (WHERE ended_at > now() - INTERVAL '24 hours' + AND end_reason IN ('stalled', 'failed', 'gone', 'abandoned'))::bigint AS given_up_day + FROM grabs + WHERE ended_at IS NULL + OR ended_at > now() - INTERVAL '24 hours' + OR imported_at > now() - INTERVAL '24 hours' +` + +type DownloadsStatusRow struct { + Running int64 + NeedsYou int64 + NeedsYouSince pgtype.Timestamptz + ImportedDay int64 + GivenUpDay int64 +} + +// Downloads at a glance: running, waiting on a person (and since when), and +// the last day's imports and give-ups. +func (q *Queries) DownloadsStatus(ctx context.Context) (DownloadsStatusRow, error) { + row := q.db.QueryRow(ctx, downloadsStatus) + var i DownloadsStatusRow + err := row.Scan( + &i.Running, + &i.NeedsYou, + &i.NeedsYouSince, + &i.ImportedDay, + &i.GivenUpDay, + ) + return i, err +} + +const libraryStatus = `-- name: LibraryStatus :one +SELECT COUNT(*)::bigint AS tracks, + COUNT(DISTINCT album_id)::bigint AS albums, + COUNT(DISTINCT artist_id)::bigint AS artists, + COALESCE(SUM(file_size), 0)::bigint AS bytes + FROM tracks + WHERE missing_since IS NULL +` + +type LibraryStatusRow struct { + Tracks int64 + Albums int64 + Artists int64 + Bytes int64 +} + +// What the library holds on disk: tracks whose files are present, the +// albums and artists they make, and their size. +func (q *Queries) LibraryStatus(ctx context.Context) (LibraryStatusRow, error) { + row := q.db.QueryRow(ctx, libraryStatus) + var i LibraryStatusRow + err := row.Scan( + &i.Tracks, + &i.Albums, + &i.Artists, + &i.Bytes, + ) + return i, err +} + +const listListeningNow = `-- name: ListListeningNow :many + +SELECT pe.started_at, pe.client_id, pe.device_class, + u.username, + t.id AS track_id, t.title AS track_title, t.duration_ms, + ar.name AS artist_name, + al.id AS album_id, al.title AS album_title + FROM play_events pe + JOIN users u ON u.id = pe.user_id + JOIN tracks t ON t.id = pe.track_id + JOIN albums al ON al.id = t.album_id + JOIN artists ar ON ar.id = t.artist_id + WHERE pe.ended_at IS NULL + AND pe.started_at > now() - INTERVAL '24 hours' + AND pe.started_at + make_interval(secs => t.duration_ms / 1000.0) + INTERVAL '5 minutes' > now() + ORDER BY pe.started_at DESC + LIMIT 20 +` + +type ListListeningNowRow struct { + StartedAt pgtype.Timestamptz + ClientID *string + DeviceClass *string + Username string + TrackID pgtype.UUID + TrackTitle string + DurationMs int32 + ArtistName string + AlbumID pgtype.UUID + AlbumTitle string +} + +// The admin Overview's status sections: who is listening, what is +// downloading, what the library holds. Each is one cheap read. +// Plays in progress: open play events still inside their track's length, +// with a few minutes' grace for a pause. A client that never says a play +// ended would otherwise read as listening until the 24-hour sweep closes it. +func (q *Queries) ListListeningNow(ctx context.Context) ([]ListListeningNowRow, error) { + rows, err := q.db.Query(ctx, listListeningNow) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListListeningNowRow + for rows.Next() { + var i ListListeningNowRow + if err := rows.Scan( + &i.StartedAt, + &i.ClientID, + &i.DeviceClass, + &i.Username, + &i.TrackID, + &i.TrackTitle, + &i.DurationMs, + &i.ArtistName, + &i.AlbumID, + &i.AlbumTitle, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const playsLastDay = `-- name: PlaysLastDay :one +SELECT COUNT(*)::bigint AS plays, + COUNT(DISTINCT user_id)::bigint AS listeners, + MAX(started_at)::timestamptz AS last_at + FROM play_events + WHERE started_at > now() - INTERVAL '24 hours' +` + +type PlaysLastDayRow struct { + Plays int64 + Listeners int64 + LastAt pgtype.Timestamptz +} + +// The last day's listening, for the line under Listening now. +func (q *Queries) PlaysLastDay(ctx context.Context) (PlaysLastDayRow, error) { + row := q.db.QueryRow(ctx, playsLastDay) + var i PlaysLastDayRow + err := row.Scan(&i.Plays, &i.Listeners, &i.LastAt) + return i, err +} diff --git a/internal/db/dbq/admin_summary.sql.go b/internal/db/dbq/admin_summary.sql.go new file mode 100644 index 00000000..91860ec5 --- /dev/null +++ b/internal/db/dbq/admin_summary.sql.go @@ -0,0 +1,77 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: admin_summary.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const adminSummary = `-- name: AdminSummary :one +SELECT + (SELECT count(*) FROM requests WHERE status = 'pending')::bigint AS requests, + (SELECT min(requested_at) FROM requests WHERE status = 'pending')::timestamptz AS requests_since, + -- A broken track is one report however it was found: a listener flagged + -- it, or it failed to play. Counted by track, as the Reports + -- page lists them. + (SELECT count(*) FROM ( + SELECT track_id FROM quarantine + UNION + SELECT track_id FROM playback_errors WHERE resolved_at IS NULL) r)::bigint AS reports, + (SELECT min(at) FROM ( + SELECT created_at AS at FROM quarantine + UNION ALL + SELECT occurred_at FROM playback_errors WHERE resolved_at IS NULL) r)::timestamptz AS reports_since, + -- Duplicates needing review: a cross-release group is kept and linked by + -- the resolver and waits on nobody (the report's review view). + (SELECT count(*) FROM duplicate_groups g + WHERE g.status = 'pending' AND g.class IS DISTINCT FROM 'cross_release' + AND (SELECT count(*) FROM duplicate_group_members m WHERE m.group_id = g.id) >= 2)::bigint AS duplicates, + (SELECT min(g.detected_at) FROM duplicate_groups g + WHERE g.status = 'pending' AND g.class IS DISTINCT FROM 'cross_release' + AND (SELECT count(*) FROM duplicate_group_members m WHERE m.group_id = g.id) >= 2)::timestamptz AS duplicates_since, + (SELECT count(*) FROM tracks WHERE source_verdict = 'suspect' AND missing_since IS NULL)::bigint AS suspect, + (SELECT min(source_verdict_at) FROM tracks WHERE source_verdict = 'suspect' AND missing_since IS NULL)::timestamptz AS suspect_since, + (SELECT count(*) FROM tracks WHERE missing_since IS NOT NULL)::bigint AS missing, + (SELECT min(missing_since) FROM tracks WHERE missing_since IS NOT NULL)::timestamptz AS missing_since +` + +type AdminSummaryRow struct { + Requests int64 + RequestsSince pgtype.Timestamptz + Reports int64 + ReportsSince pgtype.Timestamptz + Duplicates int64 + DuplicatesSince pgtype.Timestamptz + Suspect int64 + SuspectSince pgtype.Timestamptz + Missing int64 + MissingSince pgtype.Timestamptz +} + +// What the admin rail and the Overview's verdict read: each queue +// that waits on a person, with how many and since when. One round trip, so +// the rail costs one cheap read on every admin page. The "since" is the +// oldest item still waiting: how long the queue has been left, which is what +// tells an admin a count of 2 is fine today and wrong after a month. +func (q *Queries) AdminSummary(ctx context.Context) (AdminSummaryRow, error) { + row := q.db.QueryRow(ctx, adminSummary) + var i AdminSummaryRow + err := row.Scan( + &i.Requests, + &i.RequestsSince, + &i.Reports, + &i.ReportsSince, + &i.Duplicates, + &i.DuplicatesSince, + &i.Suspect, + &i.SuspectSince, + &i.Missing, + &i.MissingSince, + ) + return i, err +} diff --git a/internal/db/dbq/albums.sql.go b/internal/db/dbq/albums.sql.go index baaf1f56..fc3c8768 100644 --- a/internal/db/dbq/albums.sql.go +++ b/internal/db/dbq/albums.sql.go @@ -15,7 +15,7 @@ const countAlbums = `-- name: CountAlbums :one SELECT COUNT(*) FROM albums ` -// M6a: total album count for the /api/library/albums envelope. +// Total album count for the /api/library/albums envelope. func (q *Queries) CountAlbums(ctx context.Context) (int64, error) { row := q.db.QueryRow(ctx, countAlbums) var count int64 @@ -59,7 +59,7 @@ type DeleteAlbumIfEmptyRow struct { ArtistID pgtype.UUID } -// M7 #372: deletes the album row only when it has no remaining tracks. +// Deletes the album row only when it has no remaining tracks. // Used after a track delete to tidy up orphaned albums. RETURNING gives // us the artist_id so the service can chain into the artist-empty check. // Returns no rows when the album still has tracks; the service treats @@ -72,17 +72,39 @@ func (q *Queries) DeleteAlbumIfEmpty(ctx context.Context, id pgtype.UUID) (Delet } const getAlbumByArtistAndTitle = `-- name: GetAlbumByArtistAndTitle :one -SELECT id, title, sort_title, artist_id, release_date, mbid, cover_art_path, created_at, updated_at, cover_art_source, cover_art_sources_version, release_group_mbid FROM albums WHERE artist_id = $1 AND title = $2 LIMIT 1 +SELECT albums.id, albums.title, albums.sort_title, albums.artist_id, albums.release_date, albums.mbid, albums.cover_art_path, albums.created_at, albums.updated_at, albums.cover_art_source, albums.cover_art_sources_version, albums.release_group_mbid FROM albums + WHERE albums.artist_id = $1 AND albums.title = $2 + AND ($3::text IS NULL OR albums.release_group_mbid IS NULL + OR albums.release_group_mbid = $3) + ORDER BY (albums.release_group_mbid IS NOT DISTINCT FROM $3) DESC, + (SELECT count(*) FROM tracks t + WHERE t.album_id = albums.id AND starts_with(t.file_path, $4::text || '/')) DESC, + albums.id + LIMIT 1 ` type GetAlbumByArtistAndTitleParams struct { - ArtistID pgtype.UUID - Title string + ArtistID pgtype.UUID + Title string + ReleaseGroupMbid *string + Dir string } -// Scanner uses this for the no-mbid dedupe path: resolve-or-create. +// Scanner uses this for the no-mbid dedupe path: resolve-or-create. An album +// of the same name in another release group is a different album (one +// artist's self-titled albums), so a row whose +// group is known and differs is not returned. A row of the file's own group +// comes first, then one with no group yet, then the one holding the most +// tracks from the file's folder: a file tagged with no group (a live bonus +// track beside its album) belongs with its folder, not the oldest +// row, and a few strays already misfiled do not outvote the folder. func (q *Queries) GetAlbumByArtistAndTitle(ctx context.Context, arg GetAlbumByArtistAndTitleParams) (Album, error) { - row := q.db.QueryRow(ctx, getAlbumByArtistAndTitle, arg.ArtistID, arg.Title) + row := q.db.QueryRow(ctx, getAlbumByArtistAndTitle, + arg.ArtistID, + arg.Title, + arg.ReleaseGroupMbid, + arg.Dir, + ) var i Album err := row.Scan( &i.ID, @@ -144,7 +166,7 @@ type GetAlbumCoverageRollupRow struct { PendingNoMbid int64 } -// M7 coverage-gauge: library-wide cover-art coverage rollup for the +// Library-wide cover-art coverage rollup for the // admin gauge. Single sequential scan with FILTER aggregates — // sub-millisecond on realistic libraries. pending_no_mbid is a SUBSET // of pending; the UI surfaces it separately so operators can see how @@ -154,12 +176,9 @@ type GetAlbumCoverageRollupRow struct { // Invariant: with_art + pending + settled = total. (pending_no_mbid is // not part of the sum — it's a subset of pending, surfaced separately.) // -// The IN list below must stay in sync with the cover_art_source CHECK -// constraint — currently relaxed by migration 0020 to include 'deezer' -// and 'lastfm', and migration 0030 further relaxed it to any non-empty -// string. If a new source value is added without updating this query, -// with_art will silently undercount. Drift #556 caught the deezer + -// lastfm omission introduced by migration 0020. +// The IN list below must name every source the enrichers write (the +// cover_art_source CHECK accepts any non-empty string since migration +// 0030). A source missing here makes with_art silently undercount. func (q *Queries) GetAlbumCoverageRollup(ctx context.Context) (GetAlbumCoverageRollupRow, error) { row := q.db.QueryRow(ctx, getAlbumCoverageRollup) var i GetAlbumCoverageRollupRow @@ -214,7 +233,7 @@ SELECT id, title, sort_title, artist_id, release_date, mbid, cover_art_path, cre ` // Batched lookup used by /api/library/sync to hydrate upsert payloads -// (#357). Mirror of GetArtistsByIDs. +// . Mirror of GetArtistsByIDs. func (q *Queries) GetAlbumsByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([]Album, error) { rows, err := q.db.Query(ctx, getAlbumsByIDs, dollar_1) if err != nil { @@ -255,7 +274,7 @@ SELECT release_group_mbid::text LIMIT 1 ` -// #5241: the release group the library already knows for a release id, so a +// The release group the library already knows for a release id, so a // request carrying a release id can be repaired without asking MusicBrainz. func (q *Queries) GetReleaseGroupForReleaseMbid(ctx context.Context, mbid *string) (string, error) { row := q.db.QueryRow(ctx, getReleaseGroupForReleaseMbid, mbid) @@ -378,7 +397,7 @@ type ListAlbumsAlphaWithArtistRow struct { ArtistName string } -// M6a: alpha-sorted album list joined with artist_name. Used by +// Alpha-sorted album list joined with artist_name. Used by // /api/library/albums for the wrapping-grid page. Stable id-tiebreak. func (q *Queries) ListAlbumsAlphaWithArtist(ctx context.Context, arg ListAlbumsAlphaWithArtistParams) ([]ListAlbumsAlphaWithArtistRow, error) { rows, err := q.db.Query(ctx, listAlbumsAlphaWithArtist, arg.Limit, arg.Offset) @@ -525,12 +544,10 @@ type ListAlbumsByGenreParams struct { // Album "belongs to" a genre if any of its tracks carry that genre. // Serves Subsonic getAlbumList?type=byGenre. // -// Splits tracks.genre on [;,] as of #367. It previously compared the whole -// column verbatim, so a track tagged "Rock;Pop" was unreachable from EITHER -// "Rock" or "Pop" — a Subsonic client asking for a genre silently missed -// every multi-genre track. This also aligns the endpoint with -// recommendation.sql / discover.sql, which have always split, and with the -// genre browse index that #367 adds. +// Splits tracks.genre on [;,], so a track tagged "Rock;Pop" is reachable from +// both "Rock" and "Pop"; comparing the whole column would hide every +// multi-genre track from a Subsonic client asking for a genre. The same split +// as recommendation.sql, discover.sql and the genre browse index. // // EXISTS rather than JOIN + DISTINCT ON: the lateral split emits one row per // (track, genre-fragment), so a join would multiply rows per album and lean @@ -568,6 +585,69 @@ func (q *Queries) ListAlbumsByGenre(ctx context.Context, arg ListAlbumsByGenrePa return items, nil } +const listAlbumsFrequentlyPlayedByUser = `-- name: ListAlbumsFrequentlyPlayedByUser :many +WITH plays AS ( + SELECT track_id, count(*) AS cnt + FROM play_events + WHERE user_id = $3 AND was_skipped = false + GROUP BY track_id +), per_album AS ( + SELECT t.album_id, sum(p.cnt) AS cnt + FROM plays p + JOIN tracks t ON t.id = p.track_id + WHERE t.missing_since IS NULL + GROUP BY t.album_id +) +SELECT albums.id, albums.title, albums.sort_title, albums.artist_id, albums.release_date, albums.mbid, albums.cover_art_path, albums.created_at, albums.updated_at, albums.cover_art_source, albums.cover_art_sources_version, albums.release_group_mbid +FROM per_album pa +JOIN albums ON albums.id = pa.album_id +ORDER BY pa.cnt DESC, albums.id +LIMIT $2 OFFSET $1 +` + +type ListAlbumsFrequentlyPlayedByUserParams struct { + Off int32 + Lim int32 + UserID pgtype.UUID +} + +// The user's albums, most played first: an album's count is the user's +// completed plays across its tracks. Skips do not count, so skipping +// through an album does not make it a favourite. Plays of tracks whose file +// is gone do not count. Serves Subsonic getAlbumList2?type=frequent. +func (q *Queries) ListAlbumsFrequentlyPlayedByUser(ctx context.Context, arg ListAlbumsFrequentlyPlayedByUserParams) ([]Album, error) { + rows, err := q.db.Query(ctx, listAlbumsFrequentlyPlayedByUser, arg.Off, arg.Lim, arg.UserID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Album + for rows.Next() { + var i Album + if err := rows.Scan( + &i.ID, + &i.Title, + &i.SortTitle, + &i.ArtistID, + &i.ReleaseDate, + &i.Mbid, + &i.CoverArtPath, + &i.CreatedAt, + &i.UpdatedAt, + &i.CoverArtSource, + &i.CoverArtSourcesVersion, + &i.ReleaseGroupMbid, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + const listAlbumsMissingMbidWithTrack = `-- name: ListAlbumsMissingMbidWithTrack :many SELECT a.id AS album_id, a.artist_id AS artist_id, @@ -598,7 +678,7 @@ type ListAlbumsMissingMbidWithTrackRow struct { // one of its tracks' file_path so the worker can re-read tags. LIMIT // supplied by caller for batching/progress purposes. The track read is // one still on disk; an album with none left is skipped until a scan -// finds its files again (#5139). +// finds its files again. func (q *Queries) ListAlbumsMissingMbidWithTrack(ctx context.Context, limit int32) ([]ListAlbumsMissingMbidWithTrackRow, error) { rows, err := q.db.Query(ctx, listAlbumsMissingMbidWithTrack, limit) if err != nil { @@ -703,6 +783,73 @@ func (q *Queries) ListAlbumsRandom(ctx context.Context, limit int32) ([]Album, e return items, nil } +const listAlbumsRecentlyPlayedByUser = `-- name: ListAlbumsRecentlyPlayedByUser :many +WITH plays AS ( + SELECT track_id, max(started_at) AS last_played + FROM play_events + WHERE user_id = $3 + GROUP BY track_id +), per_album AS ( + SELECT t.album_id, max(p.last_played) AS last_played + FROM plays p + JOIN tracks t ON t.id = p.track_id + WHERE t.missing_since IS NULL + GROUP BY t.album_id +) +SELECT albums.id, albums.title, albums.sort_title, albums.artist_id, albums.release_date, albums.mbid, albums.cover_art_path, albums.created_at, albums.updated_at, albums.cover_art_source, albums.cover_art_sources_version, albums.release_group_mbid +FROM per_album pa +JOIN albums ON albums.id = pa.album_id +ORDER BY pa.last_played DESC, albums.id +LIMIT $2 OFFSET $1 +` + +type ListAlbumsRecentlyPlayedByUserParams struct { + Off int32 + Lim int32 + UserID pgtype.UUID +} + +// The user's albums, most recently played first: an album's time is its +// latest play of any of its tracks, skips included, since a skip is still +// the user reaching for that album. Plays of tracks whose file is gone do +// not count. Serves Subsonic getAlbumList2?type=recent. +// +// Aggregates play_events per track first (play_events_user_track_idx), then +// folds tracks into albums, so the join runs over distinct tracks rather +// than every play. +func (q *Queries) ListAlbumsRecentlyPlayedByUser(ctx context.Context, arg ListAlbumsRecentlyPlayedByUserParams) ([]Album, error) { + rows, err := q.db.Query(ctx, listAlbumsRecentlyPlayedByUser, arg.Off, arg.Lim, arg.UserID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Album + for rows.Next() { + var i Album + if err := rows.Scan( + &i.ID, + &i.Title, + &i.SortTitle, + &i.ArtistID, + &i.ReleaseDate, + &i.Mbid, + &i.CoverArtPath, + &i.CreatedAt, + &i.UpdatedAt, + &i.CoverArtSource, + &i.CoverArtSourcesVersion, + &i.ReleaseGroupMbid, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + const listRecentlyAddedAlbumsWithArtist = `-- name: ListRecentlyAddedAlbumsWithArtist :many SELECT albums.id, albums.title, albums.sort_title, albums.artist_id, albums.release_date, albums.mbid, albums.cover_art_path, albums.created_at, albums.updated_at, albums.cover_art_source, albums.cover_art_sources_version, albums.release_group_mbid, artists.name AS artist_name FROM albums @@ -716,7 +863,7 @@ type ListRecentlyAddedAlbumsWithArtistRow struct { ArtistName string } -// M6a: recently-added albums joined with artist_name + artist_id for the +// Recently-added albums joined with artist_name + artist_id for the // home-page section. created_at is the row-insert timestamp from the // scanner — the closest proxy to "added to library". Stable secondary // ordering by id keeps pagination tie-break deterministic. @@ -816,7 +963,7 @@ type SetAlbumCoverWithVersionParams struct { CoverArtSourcesVersion int32 } -// M7 cover-sources: records a successful or settled enrichment with the +// Records a successful or settled enrichment with the // current sources_version stamp. cover_art_path is NULL when source is // 'none' (no file written). Atomic; called per album per pass. func (q *Queries) SetAlbumCoverWithVersion(ctx context.Context, arg SetAlbumCoverWithVersionParams) error { @@ -841,7 +988,7 @@ type SetAlbumMbidIfNullParams struct { Mbid *string } -// M7 #379: heal MBID on existing album rows during rescan. Idempotent — +// Heal MBID on existing album rows during rescan. Idempotent — // only updates when the existing mbid is NULL. func (q *Queries) SetAlbumMbidIfNull(ctx context.Context, arg SetAlbumMbidIfNullParams) error { _, err := q.db.Exec(ctx, setAlbumMbidIfNull, arg.ID, arg.Mbid) @@ -860,7 +1007,7 @@ type SetAlbumReleaseGroupMbidIfNullParams struct { ReleaseGroupMbid *string } -// #5241: heal the release-group id on an existing album row, from a rescan's +// Heal the release-group id on an existing album row, from a rescan's // tags or from the re-acquisition sweeper's MusicBrainz lookup. Only fills a // NULL, so a tag never gets overwritten by a later guess or vice versa. func (q *Queries) SetAlbumReleaseGroupMbidIfNull(ctx context.Context, arg SetAlbumReleaseGroupMbidIfNullParams) error { @@ -880,7 +1027,7 @@ type SetReleaseGroupForReleaseMbidIfNullParams struct { ReleaseMbid string } -// #5241: cache a MusicBrainz-resolved release group onto the album row that +// Cache a MusicBrainz-resolved release group onto the album row that // holds the release, filling only a NULL. func (q *Queries) SetReleaseGroupForReleaseMbidIfNull(ctx context.Context, arg SetReleaseGroupForReleaseMbidIfNullParams) error { _, err := q.db.Exec(ctx, setReleaseGroupForReleaseMbidIfNull, arg.ReleaseGroupMbid, arg.ReleaseMbid) diff --git a/internal/db/dbq/artists.sql.go b/internal/db/dbq/artists.sql.go index a96aec5e..9011384a 100644 --- a/internal/db/dbq/artists.sql.go +++ b/internal/db/dbq/artists.sql.go @@ -19,7 +19,7 @@ UPDATE artists AND artist_art_source = 'none' ` -// M7 cover-sources: parallel of ClearAlbumCoverNone. Used by the +// Parallel of ClearAlbumCoverNone. Used by the // eligibility loop to flip stale 'none' rows back to NULL. func (q *Queries) ClearArtistArtNone(ctx context.Context, id pgtype.UUID) error { _, err := q.db.Exec(ctx, clearArtistArtNone, id) @@ -56,7 +56,7 @@ WHERE a.id = $1 RETURNING a.id ` -// M7 #372: deletes the artist row only when it has no remaining albums +// Deletes the artist row only when it has no remaining albums // AND no remaining tracks (defensive — tracks can in principle exist // without an album, though the scanner doesn't write that shape today). func (q *Queries) DeleteArtistIfEmpty(ctx context.Context, id pgtype.UUID) (pgtype.UUID, error) { @@ -253,7 +253,7 @@ type ListArtistsAlphaWithCoversRow struct { AlbumCount int64 } -// M6a: alpha-sorted artist list with derived cover_album_id (most-recent +// Alpha-sorted artist list with derived cover_album_id (most-recent // album whose cover_art_path is set). Replaces ListArtistsAlpha for the // /api/artists?sort=alpha branch — the new column is appended, so the // alpha branch is purely additive on the wire. album_count is computed @@ -305,7 +305,7 @@ type ListArtistsMissingArtParams struct { Limit int32 } -// M7 cover-sources: returns artist rows the artist-art enricher should +// Returns artist rows the artist-art enricher should // attempt. Eligible: artist_art_source IS NULL OR (= 'none' AND stale // artist_art_sources_version). MBID-less artists are returned but the // enricher skips them at the row.mbid == nil guard inside the chain. @@ -432,7 +432,7 @@ type SetArtistArtWithVersionParams struct { ArtistArtSourcesVersion int32 } -// M7 cover-sources: records a successful or settled artist-art +// Records a successful or settled artist-art // enrichment with the current sources_version stamp. thumb_path / // fanart_path are NULL on a 'none' result, or independently NULL if // the provider returned only one of the two image types (e.g. @@ -460,7 +460,7 @@ type SetArtistMbidIfNullParams struct { Mbid *string } -// M7 #379: heal MBID on existing artist rows during rescan. Idempotent — +// Heal MBID on existing artist rows during rescan. Idempotent — // only updates when the existing mbid is NULL, so concurrent scans don't // overwrite an existing value with a stale one. func (q *Queries) SetArtistMbidIfNull(ctx context.Context, arg SetArtistMbidIfNullParams) error { diff --git a/internal/db/dbq/audit_log.sql.go b/internal/db/dbq/audit_log.sql.go index 1a69e07b..873c0c4d 100644 --- a/internal/db/dbq/audit_log.sql.go +++ b/internal/db/dbq/audit_log.sql.go @@ -84,8 +84,8 @@ type ListAuditLogByActionsParams struct { } // One page of the given actions, newest first: the duplicates report's -// "resolved automatically" activity (M498) reads the resolver's merges and -// Lidarr release changes from here. +// "resolved automatically" activity reads the resolver's merges from +// here. func (q *Queries) ListAuditLogByActions(ctx context.Context, arg ListAuditLogByActionsParams) ([]AuditLog, error) { rows, err := q.db.Query(ctx, listAuditLogByActions, arg.Actions, diff --git a/internal/db/dbq/browse.sql.go b/internal/db/dbq/browse.sql.go index 0477431f..88f9502d 100644 --- a/internal/db/dbq/browse.sql.go +++ b/internal/db/dbq/browse.sql.go @@ -75,7 +75,7 @@ type ListAlbumYearsWithCountRow struct { AlbumCount int64 } -// Year browse index (#367). Only albums with a release_date appear — an +// Year browse index. Only albums with a release_date appear — an // album with no date isn't "year unknown" as a browsable bucket, it's absent // from this axis, and the UI says so rather than inventing a 0 row. // Newest first: recent releases are the likelier browse target. @@ -244,8 +244,7 @@ ORDER BY trim(g.genre) // Distinct genres carried by an album's tracks, for the album detail page's // quick-jump chips. Split and trimmed identically to ListGenresWithCount, so a -// chip always leads to a page that actually contains this album — the two -// diverging is exactly the bug #367 had to fix in ListAlbumsByGenre. +// chip always leads to a page that actually contains this album. func (q *Queries) ListGenresForAlbum(ctx context.Context, albumID pgtype.UUID) ([]string, error) { rows, err := q.db.Query(ctx, listGenresForAlbum, albumID) if err != nil { @@ -314,30 +313,24 @@ type ListGenresWithCountRow struct { TrackCount int64 } -// Every query in this file filters `tracks.missing_since IS NULL` (#2523). -// A row whose file has vanished keeps its genre forever — the scanner walks the -// filesystem, so it never revisits a path that no longer exists — which is how -// pre-#2499 welded genres survived a full re-scan and kept showing in the index. -// Browsing is a way of finding something to play, so a track that cannot play +// Every query in this file filters `tracks.missing_since IS NULL`. A row whose +// file has vanished keeps its genre forever — the scanner walks the +// filesystem, so it never revisits a path that no longer exists — and +// browsing is a way of finding something to play, so a track that cannot play // should not shape it. // -// The year axis filters too (#2702). It used to join albums only, on the -// reasoning that an album is a real release even when some of its tracks are -// gone — true, but it left the two browse axes disagreeing: the genre queries -// above filter tracks.missing_since, so an album whose every file had vanished -// was already absent from genre while still listed under its year, where -// opening it found nothing playable. +// The year axis filters too, so the two axes agree: an album whose every file +// has vanished is absent from both, rather than absent from its genre and +// still listed under its year, where opening it would find nothing playable. // -// Hiding is the answer rather than showing-and-marking because browsing is how -// you go looking for something to play, and the operator's rule for that case -// is to take it out of view; the admin missing-files surface is where absence -// is reported. It also means changing the axis that was inconsistent rather -// than the one that was already right. +// Hiding rather than showing-and-marking, because browsing is how you go +// looking for something to play; the admin missing-files surface is where +// absence is reported. // // The predicate is "has at least one playable track", so it also excludes an // album with no tracks at all. That is the same answer for the same reason: // nothing to play, nothing to browse to. -// Genre browse index (#367). +// Genre browse index. // // Genres live inline on tracks.genre as a delimited string, so this splits on // the same [;,] pattern already used by recommendation.sql and discover.sql — @@ -348,14 +341,14 @@ type ListGenresWithCountRow struct { // trim() but deliberately NO lower(): trimming repairs an artifact of OUR // splitting ("Rock; Pop" yields " Pop", and showing that as a distinct genre // would be a bug), whereas case is what the tag actually says. Raw ID3 is -// exposed as-is for v1, so "Rock" and "rock" appear as separate rows. +// exposed as-is, so "Rock" and "rock" appear as separate rows. // // COUNT(DISTINCT) because a sloppy tag like "Rock;Rock" would otherwise // inflate its own row. // // Ordered by count first: raw ID3 data has a long tail of one-off junk tags, -// so alphabetical would bury the handful of genres an operator actually has a -// library's worth of. Name breaks ties for a stable order. +// so alphabetical would bury the handful of genres a library actually has +// a lot of. Name breaks ties for a stable order. // Ordered by the expression, not the output alias: `ORDER BY genre` is // ambiguous between the alias and tracks.genre, and sqlc rejects it. func (q *Queries) ListGenresWithCount(ctx context.Context) ([]ListGenresWithCountRow, error) { diff --git a/internal/db/dbq/candidate_artist_tags.sql.go b/internal/db/dbq/candidate_artist_tags.sql.go index a733e4a2..12a9ceba 100644 --- a/internal/db/dbq/candidate_artist_tags.sql.go +++ b/internal/db/dbq/candidate_artist_tags.sql.go @@ -110,7 +110,7 @@ SELECT candidate_mbid, tag, weight WHERE candidate_mbid = ANY($1::text[]) ` -// Cached tags for a set of candidates, for slice 6's taste-overlap ranking. +// Cached tags for a set of candidates, for the taste-overlap ranking. // One row per (candidate, tag). func (q *Queries) ListCandidateArtistTagsForMbids(ctx context.Context, dollar_1 []string) ([]CandidateArtistTag, error) { rows, err := q.db.Query(ctx, listCandidateArtistTagsForMbids, dollar_1) @@ -160,7 +160,7 @@ type ListCandidateArtistsMissingTagsRow struct { TotalScore float64 } -// Folksonomy tags for out-of-library Discover candidates (#2376). Parallel to +// Folksonomy tags for out-of-library Discover candidates. Parallel to // track_tags.sql, but keyed by MBID because the artist has no local row. See // 0050_candidate_artist_tags.up.sql for why the bookkeeping is its own table. // Candidates eligible for tag enrichment: never processed (no state row) or diff --git a/internal/db/dbq/catalogue.sql.go b/internal/db/dbq/catalogue.sql.go new file mode 100644 index 00000000..018c0727 --- /dev/null +++ b/internal/db/dbq/catalogue.sql.go @@ -0,0 +1,402 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: catalogue.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const addMBArtistReleaseGroup = `-- name: AddMBArtistReleaseGroup :exec +INSERT INTO mb_artist_release_groups (artist_mbid, release_group_mbid) +VALUES ($1, $2) +ON CONFLICT DO NOTHING +` + +type AddMBArtistReleaseGroupParams struct { + ArtistMbid string + ReleaseGroupMbid string +} + +func (q *Queries) AddMBArtistReleaseGroup(ctx context.Context, arg AddMBArtistReleaseGroupParams) error { + _, err := q.db.Exec(ctx, addMBArtistReleaseGroup, arg.ArtistMbid, arg.ReleaseGroupMbid) + return err +} + +const deleteMBArtistReleaseGroups = `-- name: DeleteMBArtistReleaseGroups :exec +DELETE FROM mb_artist_release_groups WHERE artist_mbid = $1 +` + +func (q *Queries) DeleteMBArtistReleaseGroups(ctx context.Context, artistMbid string) error { + _, err := q.db.Exec(ctx, deleteMBArtistReleaseGroups, artistMbid) + return err +} + +const deleteMBReleasesForGroup = `-- name: DeleteMBReleasesForGroup :exec +DELETE FROM mb_releases WHERE release_group_mbid = $1 +` + +// Tracks go with their releases (ON DELETE CASCADE). +func (q *Queries) DeleteMBReleasesForGroup(ctx context.Context, releaseGroupMbid string) error { + _, err := q.db.Exec(ctx, deleteMBReleasesForGroup, releaseGroupMbid) + return err +} + +const getMBArtist = `-- name: GetMBArtist :one +SELECT mbid, name, sort_name, disambiguation, artist_type, country, discography_refreshed_at, created_at, updated_at FROM mb_artists WHERE mbid = $1 +` + +func (q *Queries) GetMBArtist(ctx context.Context, mbid string) (MbArtist, error) { + row := q.db.QueryRow(ctx, getMBArtist, mbid) + var i MbArtist + err := row.Scan( + &i.Mbid, + &i.Name, + &i.SortName, + &i.Disambiguation, + &i.ArtistType, + &i.Country, + &i.DiscographyRefreshedAt, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const getMBReleaseGroup = `-- name: GetMBReleaseGroup :one +SELECT mbid, title, primary_type, secondary_types, first_release_date, artist_credit, releases_refreshed_at, created_at, updated_at FROM mb_release_groups WHERE mbid = $1 +` + +func (q *Queries) GetMBReleaseGroup(ctx context.Context, mbid string) (MbReleaseGroup, error) { + row := q.db.QueryRow(ctx, getMBReleaseGroup, mbid) + var i MbReleaseGroup + err := row.Scan( + &i.Mbid, + &i.Title, + &i.PrimaryType, + &i.SecondaryTypes, + &i.FirstReleaseDate, + &i.ArtistCredit, + &i.ReleasesRefreshedAt, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const insertMBRelease = `-- name: InsertMBRelease :exec +INSERT INTO mb_releases (mbid, release_group_mbid, title, status, release_date, country, + disambiguation, formats, track_count) +VALUES ($1, $2, $3, $4, + $5, $6, $7, $8, + $9) +ON CONFLICT (mbid) DO UPDATE SET + release_group_mbid = EXCLUDED.release_group_mbid, + title = EXCLUDED.title, + status = EXCLUDED.status, + release_date = EXCLUDED.release_date, + country = EXCLUDED.country, + disambiguation = EXCLUDED.disambiguation, + formats = EXCLUDED.formats, + track_count = EXCLUDED.track_count +` + +type InsertMBReleaseParams struct { + Mbid string + ReleaseGroupMbid string + Title string + Status string + ReleaseDate string + Country string + Disambiguation string + Formats string + TrackCount int32 +} + +func (q *Queries) InsertMBRelease(ctx context.Context, arg InsertMBReleaseParams) error { + _, err := q.db.Exec(ctx, insertMBRelease, + arg.Mbid, + arg.ReleaseGroupMbid, + arg.Title, + arg.Status, + arg.ReleaseDate, + arg.Country, + arg.Disambiguation, + arg.Formats, + arg.TrackCount, + ) + return err +} + +const insertMBReleaseTrack = `-- name: InsertMBReleaseTrack :exec +INSERT INTO mb_release_tracks (release_mbid, medium_position, position, medium_format, number, + title, length_ms, recording_mbid) +VALUES ($1, $2, $3, + $4, $5, $6, $7, + $8) +` + +type InsertMBReleaseTrackParams struct { + ReleaseMbid string + MediumPosition int32 + Position int32 + MediumFormat string + Number string + Title string + LengthMs int32 + RecordingMbid string +} + +func (q *Queries) InsertMBReleaseTrack(ctx context.Context, arg InsertMBReleaseTrackParams) error { + _, err := q.db.Exec(ctx, insertMBReleaseTrack, + arg.ReleaseMbid, + arg.MediumPosition, + arg.Position, + arg.MediumFormat, + arg.Number, + arg.Title, + arg.LengthMs, + arg.RecordingMbid, + ) + return err +} + +const listMBArtistReleaseGroups = `-- name: ListMBArtistReleaseGroups :many +SELECT g.mbid, g.title, g.primary_type, g.secondary_types, g.first_release_date, g.artist_credit, g.releases_refreshed_at, g.created_at, g.updated_at FROM mb_release_groups g +JOIN mb_artist_release_groups ag ON ag.release_group_mbid = g.mbid +WHERE ag.artist_mbid = $1 +ORDER BY g.first_release_date = '', g.first_release_date, g.title, g.mbid +` + +// An artist's discography, oldest first; undated groups last. +func (q *Queries) ListMBArtistReleaseGroups(ctx context.Context, artistMbid string) ([]MbReleaseGroup, error) { + rows, err := q.db.Query(ctx, listMBArtistReleaseGroups, artistMbid) + if err != nil { + return nil, err + } + defer rows.Close() + var items []MbReleaseGroup + for rows.Next() { + var i MbReleaseGroup + if err := rows.Scan( + &i.Mbid, + &i.Title, + &i.PrimaryType, + &i.SecondaryTypes, + &i.FirstReleaseDate, + &i.ArtistCredit, + &i.ReleasesRefreshedAt, + &i.CreatedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listMBArtistsDueForRefresh = `-- name: ListMBArtistsDueForRefresh :many +SELECT mbid FROM mb_artists +WHERE discography_refreshed_at IS NULL OR discography_refreshed_at < $1 +ORDER BY discography_refreshed_at NULLS FIRST, mbid +LIMIT $2 +` + +type ListMBArtistsDueForRefreshParams struct { + Before pgtype.Timestamptz + MaxRows int32 +} + +// Artists whose discography has never been read, or was read before the +// cutoff: never-read first, then oldest first. +func (q *Queries) ListMBArtistsDueForRefresh(ctx context.Context, arg ListMBArtistsDueForRefreshParams) ([]string, error) { + rows, err := q.db.Query(ctx, listMBArtistsDueForRefresh, arg.Before, arg.MaxRows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []string + for rows.Next() { + var mbid string + if err := rows.Scan(&mbid); err != nil { + return nil, err + } + items = append(items, mbid) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listMBReleaseTracksForGroup = `-- name: ListMBReleaseTracksForGroup :many +SELECT t.release_mbid, t.medium_position, t.position, t.medium_format, t.number, t.title, t.length_ms, t.recording_mbid FROM mb_release_tracks t +JOIN mb_releases r ON r.mbid = t.release_mbid +WHERE r.release_group_mbid = $1 +ORDER BY t.release_mbid, t.medium_position, t.position +` + +func (q *Queries) ListMBReleaseTracksForGroup(ctx context.Context, releaseGroupMbid string) ([]MbReleaseTrack, error) { + rows, err := q.db.Query(ctx, listMBReleaseTracksForGroup, releaseGroupMbid) + if err != nil { + return nil, err + } + defer rows.Close() + var items []MbReleaseTrack + for rows.Next() { + var i MbReleaseTrack + if err := rows.Scan( + &i.ReleaseMbid, + &i.MediumPosition, + &i.Position, + &i.MediumFormat, + &i.Number, + &i.Title, + &i.LengthMs, + &i.RecordingMbid, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listMBReleasesForGroup = `-- name: ListMBReleasesForGroup :many +SELECT mbid, release_group_mbid, title, status, release_date, country, disambiguation, formats, track_count, created_at FROM mb_releases +WHERE release_group_mbid = $1 +ORDER BY status <> 'Official', release_date = '', release_date, title, mbid +` + +// Official editions first, then by date; undated last. +func (q *Queries) ListMBReleasesForGroup(ctx context.Context, releaseGroupMbid string) ([]MbRelease, error) { + rows, err := q.db.Query(ctx, listMBReleasesForGroup, releaseGroupMbid) + if err != nil { + return nil, err + } + defer rows.Close() + var items []MbRelease + for rows.Next() { + var i MbRelease + if err := rows.Scan( + &i.Mbid, + &i.ReleaseGroupMbid, + &i.Title, + &i.Status, + &i.ReleaseDate, + &i.Country, + &i.Disambiguation, + &i.Formats, + &i.TrackCount, + &i.CreatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const markMBArtistDiscographyRefreshed = `-- name: MarkMBArtistDiscographyRefreshed :exec +UPDATE mb_artists SET discography_refreshed_at = now() WHERE mbid = $1 +` + +func (q *Queries) MarkMBArtistDiscographyRefreshed(ctx context.Context, mbid string) error { + _, err := q.db.Exec(ctx, markMBArtistDiscographyRefreshed, mbid) + return err +} + +const markMBReleaseGroupReleasesRefreshed = `-- name: MarkMBReleaseGroupReleasesRefreshed :exec +UPDATE mb_release_groups SET releases_refreshed_at = now() WHERE mbid = $1 +` + +func (q *Queries) MarkMBReleaseGroupReleasesRefreshed(ctx context.Context, mbid string) error { + _, err := q.db.Exec(ctx, markMBReleaseGroupReleasesRefreshed, mbid) + return err +} + +const upsertMBArtist = `-- name: UpsertMBArtist :exec + +INSERT INTO mb_artists (mbid, name, sort_name, disambiguation, artist_type, country) +VALUES ($1, $2, $3, $4, + $5, $6) +ON CONFLICT (mbid) DO UPDATE SET + name = EXCLUDED.name, + sort_name = EXCLUDED.sort_name, + disambiguation = EXCLUDED.disambiguation, + artist_type = EXCLUDED.artist_type, + country = EXCLUDED.country, + updated_at = now() +` + +type UpsertMBArtistParams struct { + Mbid string + Name string + SortName string + Disambiguation string + ArtistType string + Country string +} + +// The release catalogue read from MusicBrainz. +func (q *Queries) UpsertMBArtist(ctx context.Context, arg UpsertMBArtistParams) error { + _, err := q.db.Exec(ctx, upsertMBArtist, + arg.Mbid, + arg.Name, + arg.SortName, + arg.Disambiguation, + arg.ArtistType, + arg.Country, + ) + return err +} + +const upsertMBReleaseGroup = `-- name: UpsertMBReleaseGroup :exec +INSERT INTO mb_release_groups (mbid, title, primary_type, secondary_types, first_release_date, artist_credit) +VALUES ($1, $2, $3, $4::text[], + $5, $6) +ON CONFLICT (mbid) DO UPDATE SET + title = EXCLUDED.title, + primary_type = EXCLUDED.primary_type, + secondary_types = EXCLUDED.secondary_types, + first_release_date = EXCLUDED.first_release_date, + artist_credit = EXCLUDED.artist_credit, + updated_at = now() +` + +type UpsertMBReleaseGroupParams struct { + Mbid string + Title string + PrimaryType string + SecondaryTypes []string + FirstReleaseDate string + ArtistCredit string +} + +// Leaves releases_refreshed_at alone: re-reading a discography says nothing +// about whether the group's track lists are current. +func (q *Queries) UpsertMBReleaseGroup(ctx context.Context, arg UpsertMBReleaseGroupParams) error { + _, err := q.db.Exec(ctx, upsertMBReleaseGroup, + arg.Mbid, + arg.Title, + arg.PrimaryType, + arg.SecondaryTypes, + arg.FirstReleaseDate, + arg.ArtistCredit, + ) + return err +} diff --git a/internal/db/dbq/coplay.sql.go b/internal/db/dbq/coplay.sql.go index f19fef6a..381a18cb 100644 --- a/internal/db/dbq/coplay.sql.go +++ b/internal/db/dbq/coplay.sql.go @@ -14,7 +14,7 @@ const deleteArtistCoplayEdges = `-- name: DeleteArtistCoplayEdges :exec DELETE FROM artist_similarity WHERE source = 'user_cooccurrence' ` -// Household co-play similarity (#1533, milestone #160 Opt 5). Minstrel is a +// Household co-play similarity. Minstrel is a // single shared-library, multi-user server (no per-user library ACL), so the // "household" is the whole instance's user set. These queries recompute // artist–artist co-occurrence edges from play_events and store them in diff --git a/internal/db/dbq/coverart_settings.sql.go b/internal/db/dbq/coverart_settings.sql.go index c87455b4..c6c32aad 100644 --- a/internal/db/dbq/coverart_settings.sql.go +++ b/internal/db/dbq/coverart_settings.sql.go @@ -77,7 +77,7 @@ SELECT provider_id, enabled, api_key, display_order, created_at, updated_at ORDER BY display_order, provider_id ` -// M7 cover-sources: queries for the cover_art_provider_settings table +// Queries for the cover_art_provider_settings table // and the cover_art_sources_meta singleton. The SettingsService // consumes these. // Returns all rows from cover_art_provider_settings. Used by the @@ -155,7 +155,7 @@ type UpsertProviderSettingsParams struct { // INSERT a default row for a newly-registered provider, or update an // existing row's display_order on boot reconciliation. Does NOT -// overwrite enabled / api_key on conflict — the operator's settings +// overwrite enabled / api_key on conflict — an admin's settings // persist across restarts. func (q *Queries) UpsertProviderSettings(ctx context.Context, arg UpsertProviderSettingsParams) error { _, err := q.db.Exec(ctx, upsertProviderSettings, arg.ProviderID, arg.Enabled, arg.DisplayOrder) diff --git a/internal/db/dbq/covers.sql.go b/internal/db/dbq/covers.sql.go index 4ea11961..ee99d700 100644 --- a/internal/db/dbq/covers.sql.go +++ b/internal/db/dbq/covers.sql.go @@ -31,7 +31,7 @@ UPDATE albums WHERE id = $1 AND cover_art_source = 'none' ` -// M7 #380: when MBID backfill heals an album, clear its 'none' cover_art_source +// When MBID backfill heals an album, clear its 'none' cover_art_source // back to NULL so the enricher's next batch retries the MBCAA fetch. // Idempotent — only updates rows currently stamped 'none'. func (q *Queries) ClearAlbumCoverNone(ctx context.Context, id pgtype.UUID) error { @@ -128,8 +128,8 @@ type ListAlbumsMissingCoverParams struct { Limit int32 } -// M7 #353: album cover enrichment queries. -// M7 cover-sources: returns rows the enricher should attempt. Eligible: +// Album cover enrichment queries. +// Returns rows the enricher should attempt. Eligible: // cover_art_source IS NULL (never tried), OR cover_art_source = 'none' // AND cover_art_sources_version != current (settled under an older // source set; recheck against the current chain). LIMIT supplied by diff --git a/internal/db/dbq/discover.sql.go b/internal/db/dbq/discover.sql.go index 3689aeba..87744461 100644 --- a/internal/db/dbq/discover.sql.go +++ b/internal/db/dbq/discover.sql.go @@ -15,7 +15,7 @@ const listCrossUserLikedTracksForDiscover = `-- name: ListCrossUserLikedTracksFo SELECT t.id, t.album_id, t.artist_id FROM general_likes gl JOIN tracks t ON t.id = gl.track_id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND gl.user_id != $1 AND NOT EXISTS ( SELECT 1 FROM play_events pe @@ -28,7 +28,7 @@ SELECT t.id, t.album_id, t.artist_id WHERE ml.user_id = $1 AND ml.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) GROUP BY t.id, t.album_id, t.artist_id @@ -55,8 +55,8 @@ type ListCrossUserLikedTracksForDiscoverRow struct { // // GROUP BY (not SELECT DISTINCT) so the md5 ORDER BY expression need // not appear in the select list — Postgres rejects DISTINCT + ORDER BY -// by-expression at plan time (SQLSTATE 42P10), which previously caused -// the entire Discover build to fail silently. +// by-expression at plan time (SQLSTATE 42P10), which would fail the +// entire Discover build. func (q *Queries) ListCrossUserLikedTracksForDiscover(ctx context.Context, arg ListCrossUserLikedTracksForDiscoverParams) ([]ListCrossUserLikedTracksForDiscoverRow, error) { rows, err := q.db.Query(ctx, listCrossUserLikedTracksForDiscover, arg.UserID, arg.Column2) if err != nil { @@ -96,7 +96,7 @@ dormant_artists AS ( SELECT t.id, t.album_id, t.artist_id FROM tracks t JOIN dormant_artists da ON da.id = t.artist_id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( SELECT 1 FROM play_events pe WHERE pe.user_id = $1 @@ -108,7 +108,7 @@ SELECT t.id, t.album_id, t.artist_id WHERE gl.user_id = $1 AND gl.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY md5(t.id::text || $2::text) @@ -161,7 +161,7 @@ func (q *Queries) ListDormantArtistTracksForDiscover(ctx context.Context, arg Li const listRandomUnheardTracksForDiscover = `-- name: ListRandomUnheardTracksForDiscover :many SELECT t.id, t.album_id, t.artist_id FROM tracks t - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( SELECT 1 FROM play_events pe WHERE pe.user_id = $1 @@ -173,7 +173,7 @@ SELECT t.id, t.album_id, t.artist_id WHERE gl.user_id = $1 AND gl.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY md5(t.id::text || $2::text) @@ -223,7 +223,7 @@ WITH scored AS ( FROM tracks t JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true JOIN taste_profile_tags nt ON nt.user_id = $3 AND trim(g_split.g) = nt.tag - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND nt.weight > 0 AND trim(g_split.g) <> '' AND NOT EXISTS ( @@ -237,7 +237,7 @@ WITH scored AS ( WHERE gl.user_id = $3 AND gl.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $3 AND q.track_id = t.id ) GROUP BY t.id, t.album_id, t.artist_id @@ -274,7 +274,7 @@ type ListTasteUnheardTracksForDiscoverRow struct { } // Taste-targeted novelty: unheard tracks whose genres overlap the user's -// taste-profile tags (taste_profile_tags, #796), ranked by summed tag +// taste-profile tags (taste_profile_tags), ranked by summed tag // weight — "new to you, but your vibe" rather than the crude random arm. // Genres live inline on tracks.genre as a delimited string, split the // same way the radio tag_overlap arm does (regexp_split_to_table on @@ -282,13 +282,13 @@ type ListTasteUnheardTracksForDiscoverRow struct { // when the user has no taste tags yet (cold start), so the caller // redistributes its slots to the other buckets. Stamped 'taste_unheard'. // -// The per-album and per-artist caps apply BEFORE the LIMIT (#5356). Summed -// weight rewards a track for carrying many of the user's tags, so a few -// artists whose every track is tagged with the whole profile take every row -// of a plain LIMIT; the caller's caps then left 6 of 120 on the deploy, and -// the best-performing arm handed its slots to the others. Ranking within -// album, then within artist over what the album cap kept, is the same walk -// capByAlbumAndArtist makes, so the caller's caps keep everything here. +// The per-album and per-artist caps apply BEFORE the LIMIT. Summed weight +// rewards a track for carrying many of the user's tags, so a few artists +// whose every track is tagged with the whole profile would take every row of +// a plain LIMIT, the caller's caps would then leave a handful, and the arm +// would hand its slots to the others. Ranking within album, then within +// artist over what the album cap kept, is the same walk capByAlbumAndArtist +// makes, so the caller's caps keep everything here. func (q *Queries) ListTasteUnheardTracksForDiscover(ctx context.Context, arg ListTasteUnheardTracksForDiscoverParams) ([]ListTasteUnheardTracksForDiscoverRow, error) { rows, err := q.db.Query(ctx, listTasteUnheardTracksForDiscover, arg.MaxPerArtist, diff --git a/internal/db/dbq/duplicates.sql.go b/internal/db/dbq/duplicates.sql.go index 51aaf8be..bed92a34 100644 --- a/internal/db/dbq/duplicates.sql.go +++ b/internal/db/dbq/duplicates.sql.go @@ -27,24 +27,6 @@ func (q *Queries) AddDuplicateGroupMember(ctx context.Context, arg AddDuplicateG return err } -const countDuplicateGroupsDetectedSince = `-- name: CountDuplicateGroupsDetectedSince :one -SELECT count(*)::bigint - FROM duplicate_groups - WHERE status = 'pending' - AND detected_at >= $1 -` - -// Pending proposals first made at or after `since`: what a sweep that started -// then found for the first time. A refreshed proposal keeps its detected_at, -// so a sweep that only re-finds known groups counts none (M489: admins are -// told about new duplicates, not reminded of the same ones every sweep). -func (q *Queries) CountDuplicateGroupsDetectedSince(ctx context.Context, since pgtype.Timestamptz) (int64, error) { - row := q.db.QueryRow(ctx, countDuplicateGroupsDetectedSince, since) - var column_1 int64 - err := row.Scan(&column_1) - return column_1, err -} - const countPendingDuplicateGroups = `-- name: CountPendingDuplicateGroups :one SELECT count(*)::bigint FROM duplicate_groups g @@ -73,9 +55,9 @@ SELECT count(*)::bigint END ` -// CountPendingDuplicateGroups narrowed to one view of the report (M498): +// CountPendingDuplicateGroups narrowed to one view of the report: // -// review what the operator still has to decide: every class but +// review what an admin still has to decide: every class but // cross_release, and groups not yet classified // cross_release the same song on different releases, which is kept, not merged func (q *Queries) CountPendingDuplicateGroupsInView(ctx context.Context, view string) (int64, error) { @@ -210,32 +192,6 @@ func (q *Queries) GetLatestFingerprintComputedAt(ctx context.Context) (pgtype.Ti return latest, err } -const listAlbumPresentTrackTitles = `-- name: ListAlbumPresentTrackTitles :many -SELECT title FROM tracks WHERE album_id = $1 AND missing_since IS NULL -` - -// The titles an album has on disk, for choosing the Lidarr release that covers -// them best (M498 #5437). -func (q *Queries) ListAlbumPresentTrackTitles(ctx context.Context, albumID pgtype.UUID) ([]string, error) { - rows, err := q.db.Query(ctx, listAlbumPresentTrackTitles, albumID) - if err != nil { - return nil, err - } - defer rows.Close() - var items []string - for rows.Next() { - var title string - if err := rows.Scan(&title); err != nil { - return nil, err - } - items = append(items, title) - } - if err := rows.Err(); err != nil { - return nil, err - } - return items, nil -} - const listDismissedDuplicateMemberSets = `-- name: ListDismissedDuplicateMemberSets :many SELECT g.id, array_agg(m.track_id ORDER BY m.track_id)::uuid[] AS track_ids FROM duplicate_groups g @@ -249,7 +205,7 @@ type ListDismissedDuplicateMemberSetsRow struct { TrackIds []pgtype.UUID } -// What the operator has already said are not duplicates. A new proposal whose +// What an admin has already said are not duplicates. A new proposal whose // every member sat together in one of these is not proposed again. func (q *Queries) ListDismissedDuplicateMemberSets(ctx context.Context) ([]ListDismissedDuplicateMemberSetsRow, error) { rows, err := q.db.Query(ctx, listDismissedDuplicateMemberSets) @@ -280,7 +236,7 @@ SELECT t.id, t.duration_ms, f.audio_stream_sha256, f.chromaprint AND f.chromaprint IS NOT NULL -- Only chromaprints taken at the current length: prints at two lengths are not -- comparable, and after a length change the backfill is still re-deriving the - -- rest (#3913). + -- rest. AND f.chromaprint_length_sec = $2 AND (t.duration_ms, t.id) > ($3::integer, $4::uuid) ORDER BY t.duration_ms, t.id @@ -340,60 +296,81 @@ func (q *Queries) ListDuplicateCandidates(ctx context.Context, arg ListDuplicate const listDuplicateGroupsForResolve = `-- name: ListDuplicateGroupsForResolve :many SELECT g.id AS group_id, g.tier, + g.worst_bit_error_rate, + g.class AS prior_class, t.id AS track_id, t.album_id, t.title, + t.source_verdict, artists.name AS artist_name, t.file_path, t.file_format, t.file_size, + t.duration_ms, t.added_at, t.disc_number, t.track_number, (t.mbid IS NOT NULL)::boolean AS has_mbid, - albums.release_group_mbid, - albums.title AS album_title, EXISTS (SELECT 1 FROM tracks o WHERE o.album_id = t.album_id AND o.id <> t.id AND o.missing_since IS NULL AND t.track_number IS NOT NULL AND o.disc_number IS NOT DISTINCT FROM t.disc_number - AND o.track_number = t.track_number)::boolean AS position_clash + AND o.track_number = t.track_number)::boolean AS position_clash, + COALESCE(NULLIF(al.release_group_mbid, ''), lam.release_group_mbid, '')::text AS release_group_mbid, + l.state AS acoustid_state, + l.recording_mbid AS acoustid_recording FROM duplicate_groups g JOIN duplicate_group_members m ON m.group_id = g.id JOIN tracks t ON t.id = m.track_id - JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id + JOIN albums al ON al.id = t.album_id + LEFT JOIN library_album_matches lam ON lam.album_id = t.album_id AND lam.outcome = 'listed' + LEFT JOIN track_acoustid_lookups l ON l.track_id = t.id WHERE g.status = 'pending' AND t.missing_since IS NULL + AND ($1::uuid IS NULL OR g.id = $1::uuid) ORDER BY g.id, t.id ` type ListDuplicateGroupsForResolveRow struct { - GroupID pgtype.UUID - Tier string - TrackID pgtype.UUID - AlbumID pgtype.UUID - Title string - ArtistName string - FilePath string - FileFormat string - FileSize int64 - AddedAt pgtype.Timestamptz - DiscNumber *int32 - TrackNumber *int32 - HasMbid bool - ReleaseGroupMbid *string - AlbumTitle string - PositionClash bool + GroupID pgtype.UUID + Tier string + WorstBitErrorRate *float32 + PriorClass *string + TrackID pgtype.UUID + AlbumID pgtype.UUID + Title string + SourceVerdict *string + ArtistName string + FilePath string + FileFormat string + FileSize int64 + DurationMs int32 + AddedAt pgtype.Timestamptz + DiscNumber *int32 + TrackNumber *int32 + HasMbid bool + PositionClash bool + ReleaseGroupMbid string + AcoustidState *string + AcoustidRecording *string } -// Every pending group's present copies, for the resolver (M498 #5435): what it +// Every pending group's present copies, for the resolver: what it // needs to classify the group and to choose the copy to keep. One row per copy, // grouped by group. position_clash is true when another present track on the // same album claims the same disc and track number: tags that do not fit the -// album, which the survivor rule weighs against a copy. -func (q *Queries) ListDuplicateGroupsForResolve(ctx context.Context) ([]ListDuplicateGroupsForResolveRow, error) { - rows, err := q.db.Query(ctx, listDuplicateGroupsForResolve) +// album, which the survivor rule weighs against a copy. source_verdict says +// whether the copy is a video rip ('suspect') or one an admin kept ('fine'). +// prior_class is the class the last pass gave the group, NULL for a group no +// pass has seen. release_group_mbid is the album's MusicBrainz group, from its +// tags or the library listing's match; ” when neither names one. +// acoustid_state and acoustid_recording are what AcoustID said of the file, +// when it was looked up: which song a recording cut to several slots is. +// group_id narrows it to one group, for an admin settling it; NULL reads them +// all. +func (q *Queries) ListDuplicateGroupsForResolve(ctx context.Context, groupID pgtype.UUID) ([]ListDuplicateGroupsForResolveRow, error) { + rows, err := q.db.Query(ctx, listDuplicateGroupsForResolve, groupID) if err != nil { return nil, err } @@ -404,20 +381,25 @@ func (q *Queries) ListDuplicateGroupsForResolve(ctx context.Context) ([]ListDupl if err := rows.Scan( &i.GroupID, &i.Tier, + &i.WorstBitErrorRate, + &i.PriorClass, &i.TrackID, &i.AlbumID, &i.Title, + &i.SourceVerdict, &i.ArtistName, &i.FilePath, &i.FileFormat, &i.FileSize, + &i.DurationMs, &i.AddedAt, &i.DiscNumber, &i.TrackNumber, &i.HasMbid, - &i.ReleaseGroupMbid, - &i.AlbumTitle, &i.PositionClash, + &i.ReleaseGroupMbid, + &i.AcoustidState, + &i.AcoustidRecording, ); err != nil { return nil, err } @@ -487,7 +469,6 @@ SELECT p.id AS group_id, p.detected_at, p.class, p.resolve_note, - m.lidarr_state, t.disc_number, t.track_number, (t.mbid IS NOT NULL)::boolean AS has_mbid, @@ -498,6 +479,7 @@ SELECT p.id AS group_id, AND o.track_number = t.track_number)::boolean AS position_clash, t.id AS track_id, t.title, + artists.id AS artist_id, artists.name AS artist_name, albums.id AS album_id, albums.title AS album_title, @@ -529,13 +511,13 @@ type ListPendingDuplicateGroupMembersRow struct { DetectedAt pgtype.Timestamptz Class *string ResolveNote *string - LidarrState *string DiscNumber *int32 TrackNumber *int32 HasMbid bool PositionClash bool TrackID pgtype.UUID Title string + ArtistID pgtype.UUID ArtistName string AlbumID pgtype.UUID AlbumTitle string @@ -551,9 +533,8 @@ type ListPendingDuplicateGroupMembersRow struct { // One page of proposals in one view (see CountPendingDuplicateGroupsInView), // newest first, flattened to one row per member so the handler folds them // without a query per group. What each copy carries — likes and plays from -// every user — is here because it is what the operator weighs when deciding -// which copy to keep; Lidarr's view of it, because a copy Lidarr tracks cannot -// be removed without Lidarr downloading it again. +// every user — is here because it is what an admin weighs when deciding +// which copy to keep. func (q *Queries) ListPendingDuplicateGroupMembers(ctx context.Context, arg ListPendingDuplicateGroupMembersParams) ([]ListPendingDuplicateGroupMembersRow, error) { rows, err := q.db.Query(ctx, listPendingDuplicateGroupMembers, arg.View, arg.PageOffset, arg.PageLimit) if err != nil { @@ -570,13 +551,13 @@ func (q *Queries) ListPendingDuplicateGroupMembers(ctx context.Context, arg List &i.DetectedAt, &i.Class, &i.ResolveNote, - &i.LidarrState, &i.DiscNumber, &i.TrackNumber, &i.HasMbid, &i.PositionClash, &i.TrackID, &i.Title, + &i.ArtistID, &i.ArtistName, &i.AlbumID, &i.AlbumTitle, @@ -632,28 +613,25 @@ func (q *Queries) SetDuplicateGroupClasses(ctx context.Context, arg SetDuplicate return err } -const setDuplicateMemberLidarrStates = `-- name: SetDuplicateMemberLidarrStates :exec -UPDATE duplicate_group_members m - SET lidarr_state = NULLIF(v.state, '') - FROM (SELECT unnest($1::uuid[]) AS group_id, - unnest($2::uuid[]) AS track_id, - unnest($3::text[]) AS state) v - WHERE m.group_id = v.group_id - AND m.track_id = v.track_id - AND m.lidarr_state IS DISTINCT FROM NULLIF(v.state, '') +const settleDuplicateGroupAutomatically = `-- name: SettleDuplicateGroupAutomatically :execrows +UPDATE duplicate_groups + SET status = $1, resolved_at = now(), resolved_automatically = true + WHERE id = $2 AND status = 'pending' ` -type SetDuplicateMemberLidarrStatesParams struct { - GroupIds []pgtype.UUID - TrackIds []pgtype.UUID - States []string +type SettleDuplicateGroupAutomaticallyParams struct { + Status string + ID pgtype.UUID } -// Lidarr's view of each copy as of this pass; an empty state clears it (Lidarr -// not consulted). Unchanged rows are left alone. -func (q *Queries) SetDuplicateMemberLidarrStates(ctx context.Context, arg SetDuplicateMemberLidarrStatesParams) error { - _, err := q.db.Exec(ctx, setDuplicateMemberLidarrStates, arg.GroupIds, arg.TrackIds, arg.States) - return err +// The resolver closing a group itself: 'dismissed' for versions that are not +// duplicates, 'merged' when it removed the wrong copies. Only a pending group. +func (q *Queries) SettleDuplicateGroupAutomatically(ctx context.Context, arg SettleDuplicateGroupAutomaticallyParams) (int64, error) { + result, err := q.db.Exec(ctx, settleDuplicateGroupAutomatically, arg.Status, arg.ID) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil } const startDuplicateSweep = `-- name: StartDuplicateSweep :one diff --git a/internal/db/dbq/events.sql.go b/internal/db/dbq/events.sql.go index ab656653..5b1ba57a 100644 --- a/internal/db/dbq/events.sql.go +++ b/internal/db/dbq/events.sql.go @@ -131,7 +131,7 @@ type GetSystemPickKindForTrackParams struct { // Looks a track up in the user's CURRENT snapshot of the given system // variant and returns its pick_kind. Used at play-ingestion time to // freeze provenance onto the play_event — snapshots rebuild daily, so -// attribution can't be reconstructed at read time (#1249/#1270). No +// attribution can't be reconstructed at read time. No // row = track not in today's snapshot (caller stores NULL); a row with // NULL pick_kind = the variant doesn't stamp (yet). songs_like_artist // is non-singleton (up to 3 mixes/user); a track in two of them takes @@ -166,7 +166,7 @@ type InsertPlayEventParams struct { // pick_kind is non-NULL only for system-playlist plays whose track was // found (with a stamped kind) in the user's live snapshot for that -// variant at ingestion time (#1249, generalized in #1270). +// variant at ingestion time. func (q *Queries) InsertPlayEvent(ctx context.Context, arg InsertPlayEventParams) (PlayEvent, error) { row := q.db.QueryRow(ctx, insertPlayEvent, arg.UserID, @@ -261,7 +261,7 @@ func (q *Queries) InsertSkipEvent(ctx context.Context, arg InsertSkipEventParams } const listRecentSessionTracks = `-- name: ListRecentSessionTracks :many -SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at FROM tracks t +SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at FROM tracks t JOIN play_events pe ON pe.track_id = t.id WHERE pe.session_id = $1 AND pe.started_at < $2 @@ -312,6 +312,7 @@ func (q *Queries) ListRecentSessionTracks(ctx context.Context, arg ListRecentSes &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ); err != nil { return nil, err } diff --git a/internal/db/dbq/fingerprints.sql.go b/internal/db/dbq/fingerprints.sql.go index f92f0cb2..581daed7 100644 --- a/internal/db/dbq/fingerprints.sql.go +++ b/internal/db/dbq/fingerprints.sql.go @@ -81,7 +81,7 @@ SELECT t.id, t.file_path LEFT JOIN track_fingerprints f ON f.track_id = t.id WHERE t.missing_since IS NULL -- A row taken at another length is as stale as one from an older method: - -- chromaprints at two lengths cannot be compared (#3913). + -- chromaprints at two lengths cannot be compared. AND (f.track_id IS NULL OR f.fingerprint_version < $1 OR f.chromaprint_length_sec <> $2) @@ -102,7 +102,7 @@ type ListTracksNeedingFingerprintRow struct { FilePath string } -// The backfill's work queue (#3908): tracks with no fingerprint, or one derived +// The backfill's work queue: tracks with no fingerprint, or one derived // by an older method. Keyset-paged on id so a pass visits each track at most // once. That cursor is load-bearing: an inconclusive attempt writes no row, so // without it a file that keeps timing out would be listed again straight away @@ -157,7 +157,7 @@ type UpsertTrackFingerprintParams struct { } // Written whenever a track's fingerprint is derived: by the scan when a file is -// new or its bytes changed, and by the backfill (#3908) for rows derived by an +// new or its bytes changed, and by the backfill for rows derived by an // older method. Replaces the row wholesale — a fingerprint of the old bytes has // no standing once the file has changed. func (q *Queries) UpsertTrackFingerprint(ctx context.Context, arg UpsertTrackFingerprintParams) error { diff --git a/internal/db/dbq/gc.sql.go b/internal/db/dbq/gc.sql.go index 957a6ef5..2cc7f360 100644 --- a/internal/db/dbq/gc.sql.go +++ b/internal/db/dbq/gc.sql.go @@ -20,10 +20,9 @@ UPDATE play_sessions ) ` -// #565: play_sessions.ended_at was added but never populated by any -// writer. Close sessions whose last_event_at is older than 6h — -// treating that as "user moved on" the same way audio_service does -// after grace periods. Sessions with NO events (track_count = 0) +// play_sessions.ended_at has no other writer. Close sessions whose +// last_event_at is older than 6h — treating that as "user moved on" the +// same way audio_service does after grace periods. Sessions with NO events (track_count = 0) // older than 1h are also closed (stale handshakes from clients that // never recorded a play). ended_at is set to last_event_at so the // session's duration reads naturally. @@ -46,12 +45,11 @@ UPDATE play_events AND started_at < now() - INTERVAL '24 hours' ` -// Background garbage-collector / lifecycle queries. All five address -// drift findings from the 2026-06-02 audit (Scribe parent #552): -// #565 #566 #567 #574 #575. Sweeper runs on a long tick (default 1h) +// Background garbage-collector / lifecycle queries: rows nothing else closes +// or removes. Sweeper runs on a long tick (default 1h) // so per-query cost is amortised; each is idempotent (re-running on // already-closed/-deleted rows is a no-op). -// #566: play_events rows opened more than 24h ago that never got a +// Play_events rows opened more than 24h ago that never got a // play_ended. Client crashed mid-track, network dropped, etc. We // synthesize ended_at = started_at + duration_played_ms when present, // otherwise leave duration_played_ms null and stamp ended_at = now() @@ -72,7 +70,7 @@ DELETE FROM password_resets OR (used_at IS NULL AND expires_at < now() - INTERVAL '1 hour') ` -// #575: password_resets accumulates expired + used rows forever. The +// Password_resets accumulates expired + used rows forever. The // validation path already rejects them; this just keeps the table // from growing unbounded. Used rows are kept for 7 days for audit; // unused expired rows go immediately. @@ -106,7 +104,7 @@ DELETE FROM scrobble_queue AND enqueued_at < now() - INTERVAL '14 days' ` -// #567: scrobble_queue rows that have been in status='failed' for +// Scrobble_queue rows that have been in status='failed' for // more than 14 days. The worker stops retrying after maxAttempts; // failed rows accumulate forever otherwise. CASCADE from play_events // already drops the row when the underlying event is deleted, so this @@ -127,7 +125,7 @@ UPDATE system_playlist_runs AND last_run_at < now() - INTERVAL '10 minutes' ` -// #574: system_playlist_runs.in_flight = true can wedge on a +// System_playlist_runs.in_flight = true can wedge on a // goroutine panic between SET in_flight=true and SET in_flight=false. // The duplicate-prevention check refuses to start a fresh regen while // in_flight, so a stuck row blocks all future regens for that user. diff --git a/internal/db/dbq/history.sql.go b/internal/db/dbq/history.sql.go index 32943d28..4edbbed0 100644 --- a/internal/db/dbq/history.sql.go +++ b/internal/db/dbq/history.sql.go @@ -14,7 +14,7 @@ import ( const listUserHistory = `-- name: ListUserHistory :many SELECT pe.id AS event_id, pe.started_at, - t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, + t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at, albums.title AS album_title, artists.name AS artist_name FROM play_events pe @@ -24,7 +24,7 @@ SELECT pe.id AS event_id, WHERE pe.user_id = $1 AND pe.was_skipped = false AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY pe.started_at DESC @@ -45,7 +45,7 @@ type ListUserHistoryRow struct { ArtistName string } -// M7 #365: track-level listening history. Returns one row per +// Track-level listening history. Returns one row per // play_events entry, newest first, joined with tracks/albums/artists // so the SPA can render a TrackRef without follow-up fetches. Filters // out skipped events and tracks the user has quarantined (per-user @@ -86,6 +86,7 @@ func (q *Queries) ListUserHistory(ctx context.Context, arg ListUserHistoryParams &i.Track.SongKey, &i.Track.SourceVerdict, &i.Track.SourceVerdictAt, + &i.Track.EmptiedAt, &i.AlbumTitle, &i.ArtistName, ); err != nil { diff --git a/internal/db/dbq/imports.sql.go b/internal/db/dbq/imports.sql.go new file mode 100644 index 00000000..b7ce6aec --- /dev/null +++ b/internal/db/dbq/imports.sql.go @@ -0,0 +1,902 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: imports.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const claimGrabImport = `-- name: ClaimGrabImport :one +UPDATE grabs + SET import_state = 'importing', import_attempts = import_attempts + 1, + import_updated_at = now(), progress = 1 + WHERE id = $1 AND ended_at IS NULL AND import_state IN ('', 'downloading', 'waiting') +RETURNING id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at +` + +// Takes a finished download for import. The update is the lock: a second +// pass finds nothing to claim. +func (q *Queries) ClaimGrabImport(ctx context.Context, id pgtype.UUID) (Grab, error) { + row := q.db.QueryRow(ctx, claimGrabImport, id) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const clearTrackSourceVerdict = `-- name: ClearTrackSourceVerdict :exec +UPDATE tracks SET source_verdict = NULL, source_verdict_at = NULL + WHERE id = $1 AND source_verdict = 'suspect' +` + +// A track whose file was replaced in place is no longer the rip it was. +func (q *Queries) ClearTrackSourceVerdict(ctx context.Context, id pgtype.UUID) error { + _, err := q.db.Exec(ctx, clearTrackSourceVerdict, id) + return err +} + +const finishUpgrade = `-- name: FinishUpgrade :exec +UPDATE wants + SET album_id = COALESCE($1, album_id), + upgrade = '', + upgrade_checked_at = NULL, + updated_at = now() + WHERE id = $2 +` + +type FinishUpgradeParams struct { + AlbumID pgtype.UUID + ID pgtype.UUID +} + +// An upgrade was imported: the want holds the album the new tracks are in, +// and is judged again by the next upgrade check. +func (q *Queries) FinishUpgrade(ctx context.Context, arg FinishUpgradeParams) error { + _, err := q.db.Exec(ctx, finishUpgrade, arg.AlbumID, arg.ID) + return err +} + +const forceGrabImport = `-- name: ForceGrabImport :one +UPDATE grabs + SET import_state = '', import_forced = true, import_reason = '', import_updated_at = now() + WHERE id = $1 AND ended_at IS NULL AND import_state IN ('review', 'failed') +RETURNING id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at +` + +// "Import anyway": the next pass imports the download whatever the check said. +func (q *Queries) ForceGrabImport(ctx context.Context, id pgtype.UUID) (Grab, error) { + row := q.db.QueryRow(ctx, forceGrabImport, id) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const getGrab = `-- name: GetGrab :one +SELECT id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at FROM grabs WHERE id = $1 +` + +func (q *Queries) GetGrab(ctx context.Context, id pgtype.UUID) (Grab, error) { + row := q.db.QueryRow(ctx, getGrab, id) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const heldPositions = `-- name: HeldPositions :many +SELECT DISTINCT COALESCE(t.disc_number, 1)::int AS disc, t.track_number::int AS track + FROM tracks t + JOIN albums a ON a.id = t.album_id + WHERE a.release_group_mbid = $1 + AND t.missing_since IS NULL AND t.track_number IS NOT NULL +` + +type HeldPositionsRow struct { + Disc int32 + Track int32 +} + +// The disc and track positions the library holds for an album of this group. +func (q *Queries) HeldPositions(ctx context.Context, releaseGroupMbid *string) ([]HeldPositionsRow, error) { + rows, err := q.db.Query(ctx, heldPositions, releaseGroupMbid) + if err != nil { + return nil, err + } + defer rows.Close() + var items []HeldPositionsRow + for rows.Next() { + var i HeldPositionsRow + if err := rows.Scan(&i.Disc, &i.Track); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const heldRecordings = `-- name: HeldRecordings :many +SELECT DISTINCT mbid::text AS mbid FROM tracks + WHERE mbid = ANY($1::text[]) AND missing_since IS NULL +` + +// Which of these recordings the library holds a file for. +func (q *Queries) HeldRecordings(ctx context.Context, recordings []string) ([]string, error) { + rows, err := q.db.Query(ctx, heldRecordings, recordings) + if err != nil { + return nil, err + } + defer rows.Close() + var items []string + for rows.Next() { + var mbid string + if err := rows.Scan(&mbid); err != nil { + return nil, err + } + items = append(items, mbid) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listGrabs = `-- name: ListGrabs :many +SELECT g.id, g.want_id, g.name, g.indexer, g.client, g.quality, g.origin, g.forced, + g.grabbed_at, g.ended_at, g.end_reason, g.end_detail, g.progress, g.import_state, + g.import_reason, g.import_detail, g.import_updated_at, g.imported_at, g.imported_files, + g.destination, g.seeding, g.ratio, g.seed_released_at, + w.title AS want_title, w.artist_name AS want_artist, + di.trigger AS condition, di.detail AS condition_detail, di.remediations AS condition_retries, + di.first_seen AS condition_since + FROM grabs g + LEFT JOIN wants w ON w.id = g.want_id + LEFT JOIN download_interventions di + ON di.id = (SELECT x.id FROM download_interventions x + WHERE x.grab_id = g.id AND x.outcome = 'open' + ORDER BY x.last_seen DESC LIMIT 1) + ORDER BY (g.ended_at IS NULL AND g.import_state IN ('review', 'failed')) DESC, + g.ended_at IS NULL DESC, g.grabbed_at DESC, g.id + LIMIT $2 OFFSET $1 +` + +type ListGrabsParams struct { + Skip int32 + MaxRows int32 +} + +type ListGrabsRow struct { + ID pgtype.UUID + WantID pgtype.UUID + Name string + Indexer string + Client string + Quality string + Origin string + Forced bool + GrabbedAt pgtype.Timestamptz + EndedAt pgtype.Timestamptz + EndReason string + EndDetail string + Progress float32 + ImportState string + ImportReason string + ImportDetail []byte + ImportUpdatedAt pgtype.Timestamptz + ImportedAt pgtype.Timestamptz + ImportedFiles int32 + Destination string + Seeding bool + Ratio float32 + SeedReleasedAt pgtype.Timestamptz + WantTitle *string + WantArtist *string + Condition *string + ConditionDetail *string + ConditionRetries *int32 + ConditionSince pgtype.Timestamptz +} + +// The Wanted view's Activity tab: downloads waiting on a person first, then +// the rest still running, then the ended ones newest first. Each carries +// what is wrong with it now, if anything. +func (q *Queries) ListGrabs(ctx context.Context, arg ListGrabsParams) ([]ListGrabsRow, error) { + rows, err := q.db.Query(ctx, listGrabs, arg.Skip, arg.MaxRows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListGrabsRow + for rows.Next() { + var i ListGrabsRow + if err := rows.Scan( + &i.ID, + &i.WantID, + &i.Name, + &i.Indexer, + &i.Client, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + &i.WantTitle, + &i.WantArtist, + &i.Condition, + &i.ConditionDetail, + &i.ConditionRetries, + &i.ConditionSince, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listImportedGrabsHolding = `-- name: ListImportedGrabsHolding :many +SELECT id, info_hash, name, want_id + FROM grabs + WHERE import_state = 'imported' + AND info_hash <> '' + AND destination <> '' + AND starts_with($1::text, rtrim(destination, '/') || '/') +` + +type ListImportedGrabsHoldingRow struct { + ID pgtype.UUID + InfoHash string + Name string + WantID pgtype.UUID +} + +// The downloads Minstrel imported into the folder holding path: the releases a +// file there came from, to block when the files proved not to be the album +// . A file imported before Minstrel downloaded anything has none. +func (q *Queries) ListImportedGrabsHolding(ctx context.Context, path string) ([]ListImportedGrabsHoldingRow, error) { + rows, err := q.db.Query(ctx, listImportedGrabsHolding, path) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListImportedGrabsHoldingRow + for rows.Next() { + var i ListImportedGrabsHoldingRow + if err := rows.Scan( + &i.ID, + &i.InfoHash, + &i.Name, + &i.WantID, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listLiveGrabs = `-- name: ListLiveGrabs :many + +SELECT g.id, g.want_id, g.info_hash, g.name, g.key, g.indexer, g.source, g.client, g.save_path, g.quality, g.origin, g.forced, g.grabbed_by, g.grabbed_at, g.ended_at, g.end_reason, g.end_detail, g.progress, g.import_state, g.import_reason, g.import_detail, g.import_forced, g.import_attempts, g.import_updated_at, g.imported_at, g.imported_files, g.destination, g.seeding, g.ratio, g.seed_released_at, w.release_group_mbid, w.artist_mbid, w.artist_name, w.title AS want_title, + w.reason AS want_reason, w.album_id AS want_album_id + FROM grabs g + LEFT JOIN wants w ON w.id = g.want_id + WHERE g.ended_at IS NULL + ORDER BY g.grabbed_at, g.id +` + +type ListLiveGrabsRow struct { + ID pgtype.UUID + WantID pgtype.UUID + InfoHash string + Name string + Key string + Indexer string + Source string + Client string + SavePath string + Quality string + Origin string + Forced bool + GrabbedBy pgtype.UUID + GrabbedAt pgtype.Timestamptz + EndedAt pgtype.Timestamptz + EndReason string + EndDetail string + Progress float32 + ImportState string + ImportReason string + ImportDetail []byte + ImportForced bool + ImportAttempts int32 + ImportUpdatedAt pgtype.Timestamptz + ImportedAt pgtype.Timestamptz + ImportedFiles int32 + Destination string + Seeding bool + Ratio float32 + SeedReleasedAt pgtype.Timestamptz + ReleaseGroupMbid *string + ArtistMbid *string + ArtistName *string + WantTitle *string + WantReason *string + WantAlbumID pgtype.UUID +} + +// Import with verification. +// Every download still running or waiting on its import, oldest first. +func (q *Queries) ListLiveGrabs(ctx context.Context) ([]ListLiveGrabsRow, error) { + rows, err := q.db.Query(ctx, listLiveGrabs) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListLiveGrabsRow + for rows.Next() { + var i ListLiveGrabsRow + if err := rows.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.WantTitle, + &i.WantReason, + &i.WantAlbumID, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listReleaseBlocksPage = `-- name: ListReleaseBlocksPage :many +SELECT b.info_hash, b.name, b.reason, b.detail, b.blocked_at, b.expires_at, w.title AS want_title, + w.artist_name AS want_artist + FROM release_blocks b + LEFT JOIN wants w ON w.id = b.want_id + WHERE b.expires_at IS NULL OR b.expires_at > now() + ORDER BY b.blocked_at DESC, b.info_hash + LIMIT $2 OFFSET $1 +` + +type ListReleaseBlocksPageParams struct { + Skip int32 + MaxRows int32 +} + +type ListReleaseBlocksPageRow struct { + InfoHash string + Name string + Reason string + Detail string + BlockedAt pgtype.Timestamptz + ExpiresAt pgtype.Timestamptz + WantTitle *string + WantArtist *string +} + +func (q *Queries) ListReleaseBlocksPage(ctx context.Context, arg ListReleaseBlocksPageParams) ([]ListReleaseBlocksPageRow, error) { + rows, err := q.db.Query(ctx, listReleaseBlocksPage, arg.Skip, arg.MaxRows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListReleaseBlocksPageRow + for rows.Next() { + var i ListReleaseBlocksPageRow + if err := rows.Scan( + &i.InfoHash, + &i.Name, + &i.Reason, + &i.Detail, + &i.BlockedAt, + &i.ExpiresAt, + &i.WantTitle, + &i.WantArtist, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listUpgradeTracks = `-- name: ListUpgradeTracks :many +SELECT id, file_path, mbid, COALESCE(disc_number, 1)::int AS disc, track_number, + file_format, file_size, duration_ms, source_verdict + FROM tracks + WHERE album_id = $1 AND missing_since IS NULL +` + +type ListUpgradeTracksRow struct { + ID pgtype.UUID + FilePath string + Mbid *string + Disc int32 + TrackNumber *int32 + FileFormat string + FileSize int64 + DurationMs int32 + SourceVerdict *string +} + +// The tracks an upgrade may replace: the present tracks of the album +// the want holds. +func (q *Queries) ListUpgradeTracks(ctx context.Context, albumID pgtype.UUID) ([]ListUpgradeTracksRow, error) { + rows, err := q.db.Query(ctx, listUpgradeTracks, albumID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListUpgradeTracksRow + for rows.Next() { + var i ListUpgradeTracksRow + if err := rows.Scan( + &i.ID, + &i.FilePath, + &i.Mbid, + &i.Disc, + &i.TrackNumber, + &i.FileFormat, + &i.FileSize, + &i.DurationMs, + &i.SourceVerdict, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const markWantSearched = `-- name: MarkWantSearched :exec +UPDATE wants SET searched_at = now() WHERE id = $1 +` + +// Imported: nothing to search for until the wanted pass settles the want. +func (q *Queries) MarkWantSearched(ctx context.Context, id pgtype.UUID) error { + _, err := q.db.Exec(ctx, markWantSearched, id) + return err +} + +const openWantsForGroups = `-- name: OpenWantsForGroups :many +SELECT w.id, w.release_group_mbid, w.artist_mbid, w.artist_name, w.title, w.primary_type, w.first_release_date, w.reason, w.album_id, w.created_by, w.wanted_at, w.fulfilled_at, w.created_at, w.updated_at, w.searched_at, w.search_outcome, w.search_considered, w.best_release, w.best_verdict, w.search_error, w.upgrade, w.held_tier, w.held_quality, w.upgrade_checked_at FROM wants w + WHERE w.fulfilled_at IS NULL AND w.release_group_mbid = ANY($1::text[]) + AND NOT EXISTS (SELECT 1 FROM grabs g WHERE g.want_id = w.id AND g.ended_at IS NULL) +` + +// Open wants for these release groups with no download running. +func (q *Queries) OpenWantsForGroups(ctx context.Context, groups []string) ([]Want, error) { + rows, err := q.db.Query(ctx, openWantsForGroups, groups) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Want + for rows.Next() { + var i Want + if err := rows.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.Title, + &i.PrimaryType, + &i.FirstReleaseDate, + &i.Reason, + &i.AlbumID, + &i.CreatedBy, + &i.WantedAt, + &i.FulfilledAt, + &i.CreatedAt, + &i.UpdatedAt, + &i.SearchedAt, + &i.SearchOutcome, + &i.SearchConsidered, + &i.BestRelease, + &i.BestVerdict, + &i.SearchError, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.UpgradeCheckedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const reclaimStaleImports = `-- name: ReclaimStaleImports :execrows +UPDATE grabs + SET import_state = 'waiting', + import_reason = 'The last import was cut short, so it is being tried again.' + WHERE ended_at IS NULL AND import_state = 'importing' + AND import_updated_at < now() - make_interval(mins => $1::int) +` + +// An import cut short (a restart mid-copy) is tried again. +func (q *Queries) ReclaimStaleImports(ctx context.Context, minutes int32) (int64, error) { + result, err := q.db.Exec(ctx, reclaimStaleImports, minutes) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const recordImport = `-- name: RecordImport :one +UPDATE grabs + SET import_state = $1, + import_reason = $2, + import_detail = $3, + import_updated_at = now(), + import_forced = false, + imported_at = CASE WHEN $1::text = 'imported' THEN now() ELSE imported_at END, + -- The files were copied, so an imported download goes on sharing + -- until the lifecycle removes it. + seeding = $1::text = 'imported', + imported_files = $4, + destination = $5, + ended_at = CASE WHEN $6::text <> '' THEN now() ELSE ended_at END, + end_reason = $6, + end_detail = CASE WHEN $6::text <> '' THEN $2::text ELSE end_detail END + WHERE id = $7 +RETURNING id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at +` + +type RecordImportParams struct { + ImportState string + ImportReason string + ImportDetail []byte + ImportedFiles int32 + Destination string + EndReason string + ID pgtype.UUID +} + +// The import's outcome. An end reason ends the grab: imported, refused or +// held. Without one the grab stays live: review and failed wait for a person. +func (q *Queries) RecordImport(ctx context.Context, arg RecordImportParams) (Grab, error) { + row := q.db.QueryRow(ctx, recordImport, + arg.ImportState, + arg.ImportReason, + arg.ImportDetail, + arg.ImportedFiles, + arg.Destination, + arg.EndReason, + arg.ID, + ) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const rehomeGrab = `-- name: RehomeGrab :one +UPDATE grabs g + SET want_id = $1 + WHERE g.id = $2 AND g.ended_at IS NULL + AND NOT EXISTS (SELECT 1 FROM grabs o WHERE o.want_id = $1 AND o.ended_at IS NULL) +RETURNING id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at +` + +type RehomeGrabParams struct { + WantID pgtype.UUID + ID pgtype.UUID +} + +// A download that turned out to be another wanted album is that want's grab, +// when the want has none running. +func (q *Queries) RehomeGrab(ctx context.Context, arg RehomeGrabParams) (Grab, error) { + row := q.db.QueryRow(ctx, rehomeGrab, arg.WantID, arg.ID) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const retryGrabImport = `-- name: RetryGrabImport :one +UPDATE grabs + SET import_state = '', import_reason = '', import_updated_at = now() + WHERE id = $1 AND ended_at IS NULL AND import_state = 'failed' +RETURNING id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at +` + +func (q *Queries) RetryGrabImport(ctx context.Context, id pgtype.UUID) (Grab, error) { + row := q.db.QueryRow(ctx, retryGrabImport, id) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const searchWantAgain = `-- name: SearchWantAgain :exec +UPDATE wants SET searched_at = NULL, search_outcome = '' WHERE id = $1 +` + +// Refused: search again on the next pass. +func (q *Queries) SearchWantAgain(ctx context.Context, id pgtype.UUID) error { + _, err := q.db.Exec(ctx, searchWantAgain, id) + return err +} + +const setGrabProgress = `-- name: SetGrabProgress :exec +UPDATE grabs + SET progress = $1, + import_state = $2, + import_reason = $3, + import_updated_at = now() + WHERE id = $4 AND ended_at IS NULL + AND import_state IN ('', 'downloading', 'waiting') +` + +type SetGrabProgressParams struct { + Progress float32 + ImportState string + ImportReason string + ID pgtype.UUID +} + +// What the download client says of a download not yet finished, or finished +// and waiting for what the import needs. Leaves a claimed, decided or failed +// import alone. +func (q *Queries) SetGrabProgress(ctx context.Context, arg SetGrabProgressParams) error { + _, err := q.db.Exec(ctx, setGrabProgress, + arg.Progress, + arg.ImportState, + arg.ImportReason, + arg.ID, + ) + return err +} diff --git a/internal/db/dbq/library_changes.sql.go b/internal/db/dbq/library_changes.sql.go index 0eeda6c4..1d38222d 100644 --- a/internal/db/dbq/library_changes.sql.go +++ b/internal/db/dbq/library_changes.sql.go @@ -116,11 +116,10 @@ type InsertLibraryChangesParams struct { Op string } -// Batch form, for a mutation that touches many rows at once (#2704: the scan's -// reconcile pass can mark up to a quarter of a library missing in one go). -// Every caller before this one changed a single entity, so per-row inserts -// were the right shape; a loop here would be thousands of round-trips inside -// a scan that is already the slow path. +// Batch form, for a mutation that touches many rows at once (the scan's +// reconcile pass can mark up to a quarter of a library missing in one go). A +// loop of single inserts would be thousands of round-trips inside a scan that +// is already the slow path. func (q *Queries) InsertLibraryChanges(ctx context.Context, arg InsertLibraryChangesParams) error { _, err := q.db.Exec(ctx, insertLibraryChanges, arg.EntityType, arg.EntityIds, arg.Op) return err diff --git a/internal/db/dbq/library_settings.sql.go b/internal/db/dbq/library_settings.sql.go new file mode 100644 index 00000000..7287bdb4 --- /dev/null +++ b/internal/db/dbq/library_settings.sql.go @@ -0,0 +1,162 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: library_settings.sql + +package dbq + +import ( + "context" +) + +const countTracksUnder = `-- name: CountTracksUnder :one +SELECT count(*) FROM tracks WHERE starts_with(file_path, $1::text || '/') +` + +func (q *Queries) CountTracksUnder(ctx context.Context, root string) (int64, error) { + row := q.db.QueryRow(ctx, countTracksUnder, root) + var count int64 + err := row.Scan(&count) + return count, err +} + +const getLibrarySettings = `-- name: GetLibrarySettings :one +SELECT id, scan_paths, updated_at FROM library_settings WHERE id = true +` + +func (q *Queries) GetLibrarySettings(ctx context.Context) (LibrarySetting, error) { + row := q.db.QueryRow(ctx, getLibrarySettings) + var i LibrarySetting + err := row.Scan(&i.ID, &i.ScanPaths, &i.UpdatedAt) + return i, err +} + +const rerootAlbumCoverPaths = `-- name: RerootAlbumCoverPaths :execrows +UPDATE albums + SET cover_art_path = $1::text || substr(cover_art_path, length($2::text) + 1) + WHERE starts_with(cover_art_path, $2::text || '/') +` + +type RerootAlbumCoverPathsParams struct { + NewRoot string + OldRoot string +} + +func (q *Queries) RerootAlbumCoverPaths(ctx context.Context, arg RerootAlbumCoverPathsParams) (int64, error) { + result, err := q.db.Exec(ctx, rerootAlbumCoverPaths, arg.NewRoot, arg.OldRoot) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const rerootGrabDestinations = `-- name: RerootGrabDestinations :execrows +UPDATE grabs + SET destination = $1::text || substr(destination, length($2::text) + 1) + WHERE destination = $2::text + OR starts_with(destination, $2::text || '/') +` + +type RerootGrabDestinationsParams struct { + NewRoot string + OldRoot string +} + +func (q *Queries) RerootGrabDestinations(ctx context.Context, arg RerootGrabDestinationsParams) (int64, error) { + result, err := q.db.Exec(ctx, rerootGrabDestinations, arg.NewRoot, arg.OldRoot) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const rerootImportRoot = `-- name: RerootImportRoot :execrows +UPDATE acquisition_settings + SET import_root = $1::text + WHERE import_root = $2::text +` + +type RerootImportRootParams struct { + NewRoot string + OldRoot string +} + +func (q *Queries) RerootImportRoot(ctx context.Context, arg RerootImportRootParams) (int64, error) { + result, err := q.db.Exec(ctx, rerootImportRoot, arg.NewRoot, arg.OldRoot) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const rerootTrackPaths = `-- name: RerootTrackPaths :execrows + +UPDATE tracks + SET file_path = $1::text || substr(file_path, length($2::text) + 1) + WHERE starts_with(file_path, $2::text || '/') +` + +type RerootTrackPathsParams struct { + NewRoot string + OldRoot string +} + +// A library folder that moved: every path under old_root is rewritten +// to the same place under new_root, so the tracks keep their ids and with them +// their plays, likes and playlists. updated_at is left alone on purpose: the +// scan compares it with each file's mtime, and the files have not changed. +func (q *Queries) RerootTrackPaths(ctx context.Context, arg RerootTrackPathsParams) (int64, error) { + result, err := q.db.Exec(ctx, rerootTrackPaths, arg.NewRoot, arg.OldRoot) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const sampleTrackPathsUnder = `-- name: SampleTrackPathsUnder :many +SELECT file_path FROM tracks + WHERE starts_with(file_path, $1::text || '/') AND missing_since IS NULL + ORDER BY random() + LIMIT $2::int +` + +type SampleTrackPathsUnderParams struct { + Root string + N int32 +} + +// A few present tracks under a folder, to check a move before rewriting it. +func (q *Queries) SampleTrackPathsUnder(ctx context.Context, arg SampleTrackPathsUnderParams) ([]string, error) { + rows, err := q.db.Query(ctx, sampleTrackPathsUnder, arg.Root, arg.N) + if err != nil { + return nil, err + } + defer rows.Close() + var items []string + for rows.Next() { + var file_path string + if err := rows.Scan(&file_path); err != nil { + return nil, err + } + items = append(items, file_path) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const updateLibraryScanPaths = `-- name: UpdateLibraryScanPaths :one +UPDATE library_settings + SET scan_paths = $1::text[], + updated_at = now() + WHERE id = true +RETURNING id, scan_paths, updated_at +` + +func (q *Queries) UpdateLibraryScanPaths(ctx context.Context, scanPaths []string) (LibrarySetting, error) { + row := q.db.QueryRow(ctx, updateLibraryScanPaths, scanPaths) + var i LibrarySetting + err := row.Scan(&i.ID, &i.ScanPaths, &i.UpdatedAt) + return i, err +} diff --git a/internal/db/dbq/lidarr_config.sql.go b/internal/db/dbq/lidarr_config.sql.go deleted file mode 100644 index 0f849d4a..00000000 --- a/internal/db/dbq/lidarr_config.sql.go +++ /dev/null @@ -1,109 +0,0 @@ -// Code generated by sqlc. DO NOT EDIT. -// versions: -// sqlc v1.31.1 -// source: lidarr_config.sql - -package dbq - -import ( - "context" - - "github.com/jackc/pgx/v5/pgtype" -) - -const getLidarrConfig = `-- name: GetLidarrConfig :one -SELECT id, enabled, base_url, api_key, default_quality_profile_id, - default_metadata_profile_id, default_root_folder_path, - created_at, updated_at -FROM lidarr_config -WHERE id = 1 -` - -type GetLidarrConfigRow struct { - ID int16 - Enabled bool - BaseUrl *string - ApiKey *string - DefaultQualityProfileID *int32 - DefaultMetadataProfileID *int32 - DefaultRootFolderPath *string - CreatedAt pgtype.Timestamptz - UpdatedAt pgtype.Timestamptz -} - -func (q *Queries) GetLidarrConfig(ctx context.Context) (GetLidarrConfigRow, error) { - row := q.db.QueryRow(ctx, getLidarrConfig) - var i GetLidarrConfigRow - err := row.Scan( - &i.ID, - &i.Enabled, - &i.BaseUrl, - &i.ApiKey, - &i.DefaultQualityProfileID, - &i.DefaultMetadataProfileID, - &i.DefaultRootFolderPath, - &i.CreatedAt, - &i.UpdatedAt, - ) - return i, err -} - -const updateLidarrConfig = `-- name: UpdateLidarrConfig :one -UPDATE lidarr_config - SET enabled = $1, - base_url = $2, - api_key = $3, - default_quality_profile_id = $4, - default_metadata_profile_id = $5, - default_root_folder_path = $6, - updated_at = now() - WHERE id = 1 - RETURNING id, enabled, base_url, api_key, default_quality_profile_id, - default_metadata_profile_id, default_root_folder_path, - created_at, updated_at -` - -type UpdateLidarrConfigParams struct { - Enabled bool - BaseUrl *string - ApiKey *string - DefaultQualityProfileID *int32 - DefaultMetadataProfileID *int32 - DefaultRootFolderPath *string -} - -type UpdateLidarrConfigRow struct { - ID int16 - Enabled bool - BaseUrl *string - ApiKey *string - DefaultQualityProfileID *int32 - DefaultMetadataProfileID *int32 - DefaultRootFolderPath *string - CreatedAt pgtype.Timestamptz - UpdatedAt pgtype.Timestamptz -} - -func (q *Queries) UpdateLidarrConfig(ctx context.Context, arg UpdateLidarrConfigParams) (UpdateLidarrConfigRow, error) { - row := q.db.QueryRow(ctx, updateLidarrConfig, - arg.Enabled, - arg.BaseUrl, - arg.ApiKey, - arg.DefaultQualityProfileID, - arg.DefaultMetadataProfileID, - arg.DefaultRootFolderPath, - ) - var i UpdateLidarrConfigRow - err := row.Scan( - &i.ID, - &i.Enabled, - &i.BaseUrl, - &i.ApiKey, - &i.DefaultQualityProfileID, - &i.DefaultMetadataProfileID, - &i.DefaultRootFolderPath, - &i.CreatedAt, - &i.UpdatedAt, - ) - return i, err -} diff --git a/internal/db/dbq/lidarr_requests.sql.go b/internal/db/dbq/lidarr_requests.sql.go deleted file mode 100644 index ecaac112..00000000 --- a/internal/db/dbq/lidarr_requests.sql.go +++ /dev/null @@ -1,582 +0,0 @@ -// Code generated by sqlc. DO NOT EDIT. -// versions: -// sqlc v1.31.1 -// source: lidarr_requests.sql - -package dbq - -import ( - "context" - - "github.com/jackc/pgx/v5/pgtype" -) - -const approveLidarrRequest = `-- name: ApproveLidarrRequest :one -UPDATE lidarr_requests - SET status = 'approved', - quality_profile_id = $2, - root_folder_path = $3, - decided_at = now(), - decided_by = $4, - updated_at = now() - WHERE id = $1 AND status = 'pending' - RETURNING id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at -` - -type ApproveLidarrRequestParams struct { - ID pgtype.UUID - QualityProfileID *int32 - RootFolderPath *string - DecidedBy pgtype.UUID -} - -func (q *Queries) ApproveLidarrRequest(ctx context.Context, arg ApproveLidarrRequestParams) (LidarrRequest, error) { - row := q.db.QueryRow(ctx, approveLidarrRequest, - arg.ID, - arg.QualityProfileID, - arg.RootFolderPath, - arg.DecidedBy, - ) - var i LidarrRequest - err := row.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ) - return i, err -} - -const cancelLidarrRequest = `-- name: CancelLidarrRequest :one -UPDATE lidarr_requests - SET status = 'rejected', - notes = 'cancelled by user', - decided_at = now(), - decided_by = $2, - updated_at = now() - WHERE id = $1 AND user_id = $2 AND status = 'pending' - RETURNING id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at -` - -type CancelLidarrRequestParams struct { - ID pgtype.UUID - DecidedBy pgtype.UUID -} - -// $2 serves as both the ownership guard (user_id = $2) and the -// decided_by audit field. The generated parameter struct names it -// DecidedBy — callers must pass the cancelling user's UUID, and -// a wrong UUID will silently no-op (zero rows updated) rather than -// erroring at the SQL layer. -func (q *Queries) CancelLidarrRequest(ctx context.Context, arg CancelLidarrRequestParams) (LidarrRequest, error) { - row := q.db.QueryRow(ctx, cancelLidarrRequest, arg.ID, arg.DecidedBy) - var i LidarrRequest - err := row.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ) - return i, err -} - -const completeLidarrRequest = `-- name: CompleteLidarrRequest :one -UPDATE lidarr_requests - SET status = 'completed', - matched_track_id = $2, - matched_album_id = $3, - matched_artist_id = $4, - completed_at = now(), - updated_at = now() - WHERE id = $1 AND status = 'approved' - RETURNING id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at -` - -type CompleteLidarrRequestParams struct { - ID pgtype.UUID - MatchedTrackID pgtype.UUID - MatchedAlbumID pgtype.UUID - MatchedArtistID pgtype.UUID -} - -// Reconciler transitions an approved request to completed when its -// target track/album/artist has appeared in the library. Callers -// should set ONLY the matched_*_id corresponding to the request's -// kind and pass pgtype.UUID{} (Valid: false) for the others — -// per-kind one-of-three is the contract, NULL for the others -// preserves auditability if the matched row is later evicted. -func (q *Queries) CompleteLidarrRequest(ctx context.Context, arg CompleteLidarrRequestParams) (LidarrRequest, error) { - row := q.db.QueryRow(ctx, completeLidarrRequest, - arg.ID, - arg.MatchedTrackID, - arg.MatchedAlbumID, - arg.MatchedArtistID, - ) - var i LidarrRequest - err := row.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ) - return i, err -} - -const createLidarrRequest = `-- name: CreateLidarrRequest :one -INSERT INTO lidarr_requests ( - user_id, kind, - lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, - artist_name, album_title, track_title -) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) -RETURNING id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at -` - -type CreateLidarrRequestParams struct { - UserID pgtype.UUID - Kind LidarrRequestKind - LidarrArtistMbid string - LidarrAlbumMbid *string - LidarrTrackMbid *string - ArtistName string - AlbumTitle *string - TrackTitle *string -} - -func (q *Queries) CreateLidarrRequest(ctx context.Context, arg CreateLidarrRequestParams) (LidarrRequest, error) { - row := q.db.QueryRow(ctx, createLidarrRequest, - arg.UserID, - arg.Kind, - arg.LidarrArtistMbid, - arg.LidarrAlbumMbid, - arg.LidarrTrackMbid, - arg.ArtistName, - arg.AlbumTitle, - arg.TrackTitle, - ) - var i LidarrRequest - err := row.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ) - return i, err -} - -const getLidarrRequestByID = `-- name: GetLidarrRequestByID :one -SELECT id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at FROM lidarr_requests WHERE id = $1 -` - -func (q *Queries) GetLidarrRequestByID(ctx context.Context, id pgtype.UUID) (LidarrRequest, error) { - row := q.db.QueryRow(ctx, getLidarrRequestByID, id) - var i LidarrRequest - err := row.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ) - return i, err -} - -const getNonTerminalRequestForMBID = `-- name: GetNonTerminalRequestForMBID :one -SELECT id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at FROM lidarr_requests - WHERE status IN ('pending', 'approved', 'completed') - AND ((kind = 'artist' AND lidarr_artist_mbid = $1) - OR (kind = 'album' AND lidarr_album_mbid = $1) - OR (kind = 'track' AND lidarr_track_mbid = $1)) - ORDER BY requested_at ASC - LIMIT 1 -` - -// Returns the oldest non-terminal (pending/approved/completed) request -// whose own-kind MBID matches @mbid, or pgx.ErrNoRows when none. Create() -// uses this to dedupe: a re-request for something already in flight -// returns the existing row instead of inserting a duplicate. MBIDs are -// globally unique per entity type, so the cross-column OR cannot -// false-match (an artist MBID never collides with an album/track MBID). -func (q *Queries) GetNonTerminalRequestForMBID(ctx context.Context, mbid string) (LidarrRequest, error) { - row := q.db.QueryRow(ctx, getNonTerminalRequestForMBID, mbid) - var i LidarrRequest - err := row.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ) - return i, err -} - -const hasNonTerminalRequestForMBID = `-- name: HasNonTerminalRequestForMBID :one -SELECT EXISTS ( - SELECT 1 FROM lidarr_requests - WHERE status IN ('pending', 'approved', 'completed') - AND ((kind = 'artist' AND lidarr_artist_mbid = $1) - OR (kind = 'album' AND lidarr_album_mbid = $1) - OR (kind = 'track' AND lidarr_track_mbid = $1)) -) -` - -// Returns true if any user has a pending/approved/completed request -// whose MBID matches at the given level. Used to set the `requested` -// flag on /api/lidarr/search responses. Terminal-status (rejected, -// failed) rows do not count. The same MBID parameter is checked -// against artist/album/track columns based on kind. -func (q *Queries) HasNonTerminalRequestForMBID(ctx context.Context, mbid string) (bool, error) { - row := q.db.QueryRow(ctx, hasNonTerminalRequestForMBID, mbid) - var exists bool - err := row.Scan(&exists) - return exists, err -} - -const listApprovedLidarrRequestsForReconcile = `-- name: ListApprovedLidarrRequestsForReconcile :many -SELECT id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at FROM lidarr_requests -WHERE status = 'approved' -ORDER BY decided_at ASC -LIMIT $1 -` - -func (q *Queries) ListApprovedLidarrRequestsForReconcile(ctx context.Context, limit int32) ([]LidarrRequest, error) { - rows, err := q.db.Query(ctx, listApprovedLidarrRequestsForReconcile, limit) - if err != nil { - return nil, err - } - defer rows.Close() - var items []LidarrRequest - for rows.Next() { - var i LidarrRequest - if err := rows.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ); err != nil { - return nil, err - } - items = append(items, i) - } - if err := rows.Err(); err != nil { - return nil, err - } - return items, nil -} - -const listLidarrRequestsByStatus = `-- name: ListLidarrRequestsByStatus :many -SELECT id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at FROM lidarr_requests -WHERE status = $1 -ORDER BY requested_at DESC -LIMIT $2 -` - -type ListLidarrRequestsByStatusParams struct { - Status LidarrRequestStatus - Limit int32 -} - -func (q *Queries) ListLidarrRequestsByStatus(ctx context.Context, arg ListLidarrRequestsByStatusParams) ([]LidarrRequest, error) { - rows, err := q.db.Query(ctx, listLidarrRequestsByStatus, arg.Status, arg.Limit) - if err != nil { - return nil, err - } - defer rows.Close() - var items []LidarrRequest - for rows.Next() { - var i LidarrRequest - if err := rows.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ); err != nil { - return nil, err - } - items = append(items, i) - } - if err := rows.Err(); err != nil { - return nil, err - } - return items, nil -} - -const listLidarrRequestsForUser = `-- name: ListLidarrRequestsForUser :many -SELECT id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at FROM lidarr_requests -WHERE user_id = $1 -ORDER BY requested_at DESC -LIMIT $2 -` - -type ListLidarrRequestsForUserParams struct { - UserID pgtype.UUID - Limit int32 -} - -func (q *Queries) ListLidarrRequestsForUser(ctx context.Context, arg ListLidarrRequestsForUserParams) ([]LidarrRequest, error) { - rows, err := q.db.Query(ctx, listLidarrRequestsForUser, arg.UserID, arg.Limit) - if err != nil { - return nil, err - } - defer rows.Close() - var items []LidarrRequest - for rows.Next() { - var i LidarrRequest - if err := rows.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ); err != nil { - return nil, err - } - items = append(items, i) - } - if err := rows.Err(); err != nil { - return nil, err - } - return items, nil -} - -const markLidarrRequestAddConfirmed = `-- name: MarkLidarrRequestAddConfirmed :exec -UPDATE lidarr_requests - SET lidarr_add_confirmed_at = now(), - updated_at = now() - WHERE id = $1 - AND status = 'approved' - AND lidarr_add_confirmed_at IS NULL -` - -// Idempotently records that Lidarr accepted the add for an approved -// request. The lidarr_add_confirmed_at IS NULL guard makes a redundant -// reconciler retry a no-op rather than bumping updated_at every tick. -func (q *Queries) MarkLidarrRequestAddConfirmed(ctx context.Context, id pgtype.UUID) error { - _, err := q.db.Exec(ctx, markLidarrRequestAddConfirmed, id) - return err -} - -const rejectLidarrRequest = `-- name: RejectLidarrRequest :one -UPDATE lidarr_requests - SET status = 'rejected', - notes = $2, - decided_at = now(), - decided_by = $3, - updated_at = now() - WHERE id = $1 AND status = 'pending' - RETURNING id, user_id, status, kind, lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, artist_name, album_title, track_title, quality_profile_id, root_folder_path, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at, lidarr_add_confirmed_at -` - -type RejectLidarrRequestParams struct { - ID pgtype.UUID - Notes *string - DecidedBy pgtype.UUID -} - -func (q *Queries) RejectLidarrRequest(ctx context.Context, arg RejectLidarrRequestParams) (LidarrRequest, error) { - row := q.db.QueryRow(ctx, rejectLidarrRequest, arg.ID, arg.Notes, arg.DecidedBy) - var i LidarrRequest - err := row.Scan( - &i.ID, - &i.UserID, - &i.Status, - &i.Kind, - &i.LidarrArtistMbid, - &i.LidarrAlbumMbid, - &i.LidarrTrackMbid, - &i.ArtistName, - &i.AlbumTitle, - &i.TrackTitle, - &i.QualityProfileID, - &i.RootFolderPath, - &i.DecidedAt, - &i.DecidedBy, - &i.Notes, - &i.CompletedAt, - &i.MatchedTrackID, - &i.MatchedAlbumID, - &i.MatchedArtistID, - &i.RequestedAt, - &i.UpdatedAt, - &i.LidarrAddConfirmedAt, - ) - return i, err -} - -const setLidarrRequestAlbumMbid = `-- name: SetLidarrRequestAlbumMbid :exec -UPDATE lidarr_requests - SET lidarr_album_mbid = $2, - updated_at = now() - WHERE id = $1 -` - -type SetLidarrRequestAlbumMbidParams struct { - ID pgtype.UUID - LidarrAlbumMbid *string -} - -// #5241: repoint a request at the album's release group. Requests made before -// the release-group read named albums by MusicBrainz release id, which Lidarr -// does not index; the reconciler rewrites them once it has resolved the group. -func (q *Queries) SetLidarrRequestAlbumMbid(ctx context.Context, arg SetLidarrRequestAlbumMbidParams) error { - _, err := q.db.Exec(ctx, setLidarrRequestAlbumMbid, arg.ID, arg.LidarrAlbumMbid) - return err -} diff --git a/internal/db/dbq/lifecycle.sql.go b/internal/db/dbq/lifecycle.sql.go new file mode 100644 index 00000000..09aa6f5a --- /dev/null +++ b/internal/db/dbq/lifecycle.sql.go @@ -0,0 +1,408 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: lifecycle.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const countGoneGrabs = `-- name: CountGoneGrabs :one +SELECT count(*) FROM grabs + WHERE info_hash = $1 AND end_reason = 'gone' AND ended_at > $2 +` + +type CountGoneGrabsParams struct { + InfoHash string + Since pgtype.Timestamptz +} + +// How many of a release's grabs ended gone since a time: a release that keeps +// leaving its client is held back rather than taken on every pass. +func (q *Queries) CountGoneGrabs(ctx context.Context, arg CountGoneGrabsParams) (int64, error) { + row := q.db.QueryRow(ctx, countGoneGrabs, arg.InfoHash, arg.Since) + var count int64 + err := row.Scan(&count) + return count, err +} + +const countRemediation = `-- name: CountRemediation :one +UPDATE download_interventions + SET remediations = remediations + 1, + action = $1, + last_seen = now(), + outcome = CASE WHEN $1::text IN ('given_up', 'removed') THEN 'resolved' ELSE outcome END, + settled_at = CASE WHEN $1::text IN ('given_up', 'removed') THEN now() ELSE settled_at END + WHERE client = $2 AND info_hash = $3 + AND trigger = $4 AND outcome = 'open' +RETURNING remediations +` + +type CountRemediationParams struct { + Action string + Client string + InfoHash string + Trigger string +} + +// Minstrel acted on an open condition. A retry climbs the ladder; giving up +// or removing ends the condition, since what it was about is gone. Written +// before the act, so a retry that happened is never uncounted. +func (q *Queries) CountRemediation(ctx context.Context, arg CountRemediationParams) (int32, error) { + row := q.db.QueryRow(ctx, countRemediation, + arg.Action, + arg.Client, + arg.InfoHash, + arg.Trigger, + ) + var remediations int32 + err := row.Scan(&remediations) + return remediations, err +} + +const deleteExpiredBlocks = `-- name: DeleteExpiredBlocks :execrows +DELETE FROM release_blocks WHERE expires_at IS NOT NULL AND expires_at <= now() +` + +// A release given up on is held back for a while; after that the search may +// take it again. +func (q *Queries) DeleteExpiredBlocks(ctx context.Context) (int64, error) { + result, err := q.db.Exec(ctx, deleteExpiredBlocks) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const giveUpGrab = `-- name: GiveUpGrab :one +UPDATE grabs + SET ended_at = now(), end_reason = $1, end_detail = $2, + import_updated_at = now() + WHERE id = $3 AND ended_at IS NULL AND import_state = ANY($4::text[]) +RETURNING id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at +` + +type GiveUpGrabParams struct { + EndReason string + EndDetail string + ID pgtype.UUID + States []string +} + +// Ends a running grab, given up on. Only from the states named: the update +// is the lock against an import that claimed the download meanwhile. +func (q *Queries) GiveUpGrab(ctx context.Context, arg GiveUpGrabParams) (Grab, error) { + row := q.db.QueryRow(ctx, giveUpGrab, + arg.EndReason, + arg.EndDetail, + arg.ID, + arg.States, + ) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const listClientsDown = `-- name: ListClientsDown :many +SELECT client, detail, first_seen FROM download_interventions + WHERE trigger = 'client_unreachable' AND outcome = 'open' + ORDER BY client +` + +type ListClientsDownRow struct { + Client string + Detail string + FirstSeen pgtype.Timestamptz +} + +// Download clients not answering now, for the Activity tab. +func (q *Queries) ListClientsDown(ctx context.Context) ([]ListClientsDownRow, error) { + rows, err := q.db.Query(ctx, listClientsDown) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListClientsDownRow + for rows.Next() { + var i ListClientsDownRow + if err := rows.Scan(&i.Client, &i.Detail, &i.FirstSeen); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listLeftovers = `-- name: ListLeftovers :many +SELECT id, client, info_hash, grab_id, name, trigger, action, outcome, detail, observations, remediations, first_seen, last_seen, settled_at FROM download_interventions + WHERE trigger = 'not_removed' AND outcome = 'open' AND info_hash <> '' + ORDER BY first_seen +` + +// Downloads given up on that the client would not remove; each pass asks +// again. +func (q *Queries) ListLeftovers(ctx context.Context) ([]DownloadIntervention, error) { + rows, err := q.db.Query(ctx, listLeftovers) + if err != nil { + return nil, err + } + defer rows.Close() + var items []DownloadIntervention + for rows.Next() { + var i DownloadIntervention + if err := rows.Scan( + &i.ID, + &i.Client, + &i.InfoHash, + &i.GrabID, + &i.Name, + &i.Trigger, + &i.Action, + &i.Outcome, + &i.Detail, + &i.Observations, + &i.Remediations, + &i.FirstSeen, + &i.LastSeen, + &i.SettledAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listSeedingGrabs = `-- name: ListSeedingGrabs :many +SELECT id, client, info_hash, name, imported_at FROM grabs + WHERE seeding AND seed_released_at IS NULL + ORDER BY imported_at, id +` + +type ListSeedingGrabsRow struct { + ID pgtype.UUID + Client string + InfoHash string + Name string + ImportedAt pgtype.Timestamptz +} + +// Imported downloads still sharing, oldest import first. +func (q *Queries) ListSeedingGrabs(ctx context.Context) ([]ListSeedingGrabsRow, error) { + rows, err := q.db.Query(ctx, listSeedingGrabs) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListSeedingGrabsRow + for rows.Next() { + var i ListSeedingGrabsRow + if err := rows.Scan( + &i.ID, + &i.Client, + &i.InfoHash, + &i.Name, + &i.ImportedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const liveGrabOnHash = `-- name: LiveGrabOnHash :one +SELECT EXISTS (SELECT 1 FROM grabs WHERE info_hash = $1 AND ended_at IS NULL)::bool +` + +func (q *Queries) LiveGrabOnHash(ctx context.Context, infoHash string) (bool, error) { + row := q.db.QueryRow(ctx, liveGrabOnHash, infoHash) + var column_1 bool + err := row.Scan(&column_1) + return column_1, err +} + +const raiseIntervention = `-- name: RaiseIntervention :one + +INSERT INTO download_interventions (client, info_hash, grab_id, name, trigger, detail) +VALUES ($1, $2, $3, $4, + $5, $6) +ON CONFLICT (client, info_hash, trigger) WHERE outcome = 'open' DO UPDATE SET + observations = download_interventions.observations + 1, + last_seen = now(), + detail = EXCLUDED.detail, + grab_id = COALESCE(EXCLUDED.grab_id, download_interventions.grab_id), + name = CASE WHEN EXCLUDED.name <> '' THEN EXCLUDED.name ELSE download_interventions.name END +RETURNING id, client, info_hash, grab_id, name, trigger, action, outcome, detail, observations, remediations, first_seen, last_seen, settled_at +` + +type RaiseInterventionParams struct { + Client string + InfoHash string + GrabID pgtype.UUID + Name string + Trigger string + Detail string +} + +// The download lifecycle: the intervention ledger, giving a +// download up, and seeding after import. +// A condition seen, or seen again: a second pass lands on the open row, so +// a download stuck for a week is one row that says how long. The name only +// improves, and the detail is the latest word. +func (q *Queries) RaiseIntervention(ctx context.Context, arg RaiseInterventionParams) (DownloadIntervention, error) { + row := q.db.QueryRow(ctx, raiseIntervention, + arg.Client, + arg.InfoHash, + arg.GrabID, + arg.Name, + arg.Trigger, + arg.Detail, + ) + var i DownloadIntervention + err := row.Scan( + &i.ID, + &i.Client, + &i.InfoHash, + &i.GrabID, + &i.Name, + &i.Trigger, + &i.Action, + &i.Outcome, + &i.Detail, + &i.Observations, + &i.Remediations, + &i.FirstSeen, + &i.LastSeen, + &i.SettledAt, + ) + return i, err +} + +const releaseSeed = `-- name: ReleaseSeed :exec +UPDATE grabs SET seed_released_at = now() WHERE id = $1 AND seed_released_at IS NULL +` + +// The seeding download was removed, or found gone. +func (q *Queries) ReleaseSeed(ctx context.Context, id pgtype.UUID) error { + _, err := q.db.Exec(ctx, releaseSeed, id) + return err +} + +const setGrabRatio = `-- name: SetGrabRatio :exec +UPDATE grabs SET ratio = $1 WHERE id = $2 +` + +type SetGrabRatioParams struct { + Ratio float32 + ID pgtype.UUID +} + +func (q *Queries) SetGrabRatio(ctx context.Context, arg SetGrabRatioParams) error { + _, err := q.db.Exec(ctx, setGrabRatio, arg.Ratio, arg.ID) + return err +} + +const settleEndedGrabs = `-- name: SettleEndedGrabs :execrows +UPDATE download_interventions di + SET outcome = 'resolved', settled_at = now(), last_seen = now() + FROM grabs g + WHERE g.id = di.grab_id AND g.ended_at IS NOT NULL + AND di.outcome = 'open' AND di.trigger <> 'not_removed' +` + +// What was wrong with a download whose grab has ended (imported, refused, +// given up) is over, except a download left in the client to remove. +func (q *Queries) SettleEndedGrabs(ctx context.Context) (int64, error) { + result, err := q.db.Exec(ctx, settleEndedGrabs) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const settleIntervention = `-- name: SettleIntervention :execrows +UPDATE download_interventions + SET outcome = 'resolved', settled_at = now(), last_seen = now() + WHERE client = $1 AND info_hash = $2 + AND trigger = $3 AND outcome = 'open' +` + +type SettleInterventionParams struct { + Client string + InfoHash string + Trigger string +} + +// The condition cleared. +func (q *Queries) SettleIntervention(ctx context.Context, arg SettleInterventionParams) (int64, error) { + result, err := q.db.Exec(ctx, settleIntervention, arg.Client, arg.InfoHash, arg.Trigger) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const settleInterventionsFor = `-- name: SettleInterventionsFor :execrows +UPDATE download_interventions + SET outcome = 'resolved', settled_at = now(), last_seen = now() + WHERE client = $1 AND info_hash = $2 AND outcome = 'open' +` + +type SettleInterventionsForParams struct { + Client string + InfoHash string +} + +// Everything open about one download is over: it is gone from the client. +func (q *Queries) SettleInterventionsFor(ctx context.Context, arg SettleInterventionsForParams) (int64, error) { + result, err := q.db.Exec(ctx, settleInterventionsFor, arg.Client, arg.InfoHash) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} diff --git a/internal/db/dbq/likes.sql.go b/internal/db/dbq/likes.sql.go index d7480f00..5c0137ec 100644 --- a/internal/db/dbq/likes.sql.go +++ b/internal/db/dbq/likes.sql.go @@ -93,7 +93,7 @@ type LikeTrackParams struct { TrackID pgtype.UUID } -// A like is on the song (M498 #5438): every copy of it sharing the track's +// A like is on the song: every copy of it sharing the track's // song_key is liked. Returns the tracks newly liked, for the sync log; empty // when the song was already liked. func (q *Queries) LikeTrack(ctx context.Context, arg LikeTrackParams) ([]pgtype.UUID, error) { @@ -281,7 +281,7 @@ func (q *Queries) ListLikedTrackIDs(ctx context.Context, userID pgtype.UUID) ([] } const listLikedTrackRows = `-- name: ListLikedTrackRows :many -SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at FROM tracks t +SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at FROM tracks t JOIN (SELECT DISTINCT ON (lt.song_key) l.track_id, l.liked_at FROM general_likes l JOIN tracks lt ON lt.id = l.track_id @@ -335,6 +335,7 @@ func (q *Queries) ListLikedTrackRows(ctx context.Context, arg ListLikedTrackRows &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ); err != nil { return nil, err } diff --git a/internal/db/dbq/loudness.sql.go b/internal/db/dbq/loudness.sql.go index c7d1df1d..9dfef31b 100644 --- a/internal/db/dbq/loudness.sql.go +++ b/internal/db/dbq/loudness.sql.go @@ -164,7 +164,7 @@ type GetReplayGainByTrackIDsRow struct { } // Track and album loudness for a set of tracks, for every surface that hands -// gains to a client (#4997). A measurement from an older analysis version is +// gains to a client. A measurement from an older analysis version is // still delivered: it is a better gain than none until the backfill redoes it. func (q *Queries) GetReplayGainByTrackIDs(ctx context.Context, ids []pgtype.UUID) ([]GetReplayGainByTrackIDsRow, error) { rows, err := q.db.Query(ctx, getReplayGainByTrackIDs, ids) @@ -280,7 +280,7 @@ type ListAlbumsNeedingLoudnessRow struct { Digest string } -// The album pass's work queue (#4996): albums whose present tracks or their +// The album pass's work queue: albums whose present tracks or their // measurements have changed since album loudness was last computed, or that // never had it. The digest is over every present track's id and the // measurement it holds at the current version ('-' for none), so a track @@ -484,7 +484,7 @@ type UpsertTrackLoudnessParams struct { AnalysisVersion int16 } -// Written when the backfill measures a track (#4995). Replaces the row +// Written when the backfill measures a track. Replaces the row // wholesale: a measurement of the old bytes has no standing once the file has // changed. func (q *Queries) UpsertTrackLoudness(ctx context.Context, arg UpsertTrackLoudnessParams) error { diff --git a/internal/db/dbq/merge.sql.go b/internal/db/dbq/merge.sql.go index 542eb77b..eee18b55 100644 --- a/internal/db/dbq/merge.sql.go +++ b/internal/db/dbq/merge.sql.go @@ -12,11 +12,9 @@ import ( ) const listDuplicateGroupMergeMembers = `-- name: ListDuplicateGroupMergeMembers :many -SELECT t.id, t.file_path, t.file_format, t.file_size, t.added_at, t.album_id, - t.mbid, albums.mbid AS album_mbid +SELECT t.id, t.file_path, t.file_format, t.file_size, t.added_at, t.album_id FROM duplicate_group_members m - JOIN tracks t ON t.id = m.track_id - JOIN albums ON albums.id = t.album_id + JOIN tracks t ON t.id = m.track_id WHERE m.group_id = $1 ORDER BY t.id ` @@ -28,8 +26,6 @@ type ListDuplicateGroupMergeMembersRow struct { FileSize int64 AddedAt pgtype.Timestamptz AlbumID pgtype.UUID - Mbid *string - AlbumMbid *string } func (q *Queries) ListDuplicateGroupMergeMembers(ctx context.Context, groupID pgtype.UUID) ([]ListDuplicateGroupMergeMembersRow, error) { @@ -48,8 +44,6 @@ func (q *Queries) ListDuplicateGroupMergeMembers(ctx context.Context, groupID pg &i.FileSize, &i.AddedAt, &i.AlbumID, - &i.Mbid, - &i.AlbumMbid, ); err != nil { return nil, err } @@ -75,7 +69,7 @@ type LockDuplicateGroupForMergeRow struct { Status string } -// Duplicate merge (Scribe #3911). Every statement here runs inside the one +// Duplicate merge. Every statement here runs inside the one // transaction library.MergeDuplicateGroup opens, after the removed copy's file // is already gone. The loser's own track row is deleted last with DeleteTrack; // what these do is move everything it carries onto the survivor first, so that @@ -233,24 +227,6 @@ func (q *Queries) MergeRepointContextualLikes(ctx context.Context, arg MergeRepo return result.RowsAffected(), nil } -const mergeRepointLidarrRequests = `-- name: MergeRepointLidarrRequests :execrows -UPDATE lidarr_requests SET matched_track_id = $1::uuid - WHERE matched_track_id = $2::uuid -` - -type MergeRepointLidarrRequestsParams struct { - SurvivorID pgtype.UUID - LoserID pgtype.UUID -} - -func (q *Queries) MergeRepointLidarrRequests(ctx context.Context, arg MergeRepointLidarrRequestsParams) (int64, error) { - result, err := q.db.Exec(ctx, mergeRepointLidarrRequests, arg.SurvivorID, arg.LoserID) - if err != nil { - return 0, err - } - return result.RowsAffected(), nil -} - const mergeRepointPlayEvents = `-- name: MergeRepointPlayEvents :execrows UPDATE play_events SET track_id = $1::uuid WHERE track_id = $2::uuid @@ -321,6 +297,24 @@ func (q *Queries) MergeRepointPlaylistTracks(ctx context.Context, arg MergeRepoi return items, nil } +const mergeRepointRequests = `-- name: MergeRepointRequests :execrows +UPDATE requests SET matched_track_id = $1::uuid + WHERE matched_track_id = $2::uuid +` + +type MergeRepointRequestsParams struct { + SurvivorID pgtype.UUID + LoserID pgtype.UUID +} + +func (q *Queries) MergeRepointRequests(ctx context.Context, arg MergeRepointRequestsParams) (int64, error) { + result, err := q.db.Exec(ctx, mergeRepointRequests, arg.SurvivorID, arg.LoserID) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + const mergeRepointSkipEvents = `-- name: MergeRepointSkipEvents :execrows UPDATE skip_events SET track_id = $1::uuid WHERE track_id = $2::uuid ` diff --git a/internal/db/dbq/missing_pairs.sql.go b/internal/db/dbq/missing_pairs.sql.go index 07042637..408f1e43 100644 --- a/internal/db/dbq/missing_pairs.sql.go +++ b/internal/db/dbq/missing_pairs.sql.go @@ -35,6 +35,7 @@ cand AS ( AND lower(p.title) = lower(m.title) WHERE m.missing_since IS NOT NULL AND (m.mbid IS NULL OR p.mbid IS NULL OR m.mbid = p.mbid) + AND (m.emptied_at IS DISTINCT FROM m.missing_since OR p.added_at > m.emptied_at) ), pairs AS ( -- A pair both rules found is one candidate, credited to the stronger rule. @@ -62,31 +63,35 @@ type ListMissingTrackPairsRow struct { PresentCandidates int32 } -// Missing-pair pass (M485). A track marked missing whose replacement is already -// on disk under another row — most often a Lidarr quality upgrade, mp3 → flac at -// a new path — is folded into that replacement, so its history moves across and +// Missing-pair pass. A track marked missing whose replacement is already +// on disk under another row — most often a quality upgrade, mp3 → flac at a new +// path — is folded into that replacement, so its history moves across and // re-acquisition stops chasing music the library already has. // -// Move adoption (#2528) cannot catch these: it links a new file to a missing row +// Move adoption cannot catch these: it links a new file to a missing row // at the moment the file is first inserted, and the replacement arrives as a // re-encode with no recording MBID, so it has nothing to match on. AcoustID may // give it an MBID later; by then both rows exist and only a fold can join them. // Every candidate (missing, present) pair, with how many candidates each side -// has. Two rules, measured on the operator's library on 2026-10-07 (#5274): +// has. Two rules: // // mbid the same recording MBID, within the album group (the album row, or -// albums sharing a release group). 142 of 359 pairs. -// title the same album row and the same title, ignoring case. 213 more. The +// albums sharing a release group). +// title the same album row and the same title, ignoring case. The // replacements are often not the original file — another edition, // re-encoded, untagged — so the metadata is what is left to go on. // -// Deliberately NOT gated on duration or track position. On the 142 pairs known -// to be one recording, 18% differed by over 2s (old mp3 durations are poor -// estimates), and the poorly tagged set is exactly where numbering is broken. +// Deliberately NOT gated on duration or track position. Pairs known to be one +// recording often differ by over 2s (old mp3 durations are poor estimates), +// and the poorly tagged set is exactly where numbering is broken. // // A pair whose MBIDs are both set and differ is not a title match: the tags say // they are different recordings. The caller folds a pair only when each side has // exactly one candidate; anything else is ambiguous and left alone. +// +// A slot the duplicate resolver emptied (emptied_at, migration 0095) pairs by +// title only with a track added since: the album held no copy of the song as +// long as it is, so one already there under the title is another song. func (q *Queries) ListMissingTrackPairs(ctx context.Context) ([]ListMissingTrackPairsRow, error) { rows, err := q.db.Query(ctx, listMissingTrackPairs) if err != nil { diff --git a/internal/db/dbq/models.go b/internal/db/dbq/models.go index 556c54c9..96e155aa 100644 --- a/internal/db/dbq/models.go +++ b/internal/db/dbq/models.go @@ -11,180 +11,180 @@ import ( "github.com/jackc/pgx/v5/pgtype" ) -type LidarrQuarantineActionKind string +type QuarantineActionKind string const ( - LidarrQuarantineActionKindResolved LidarrQuarantineActionKind = "resolved" - LidarrQuarantineActionKindDeletedFile LidarrQuarantineActionKind = "deleted_file" - LidarrQuarantineActionKindDeletedViaLidarr LidarrQuarantineActionKind = "deleted_via_lidarr" + QuarantineActionKindResolved QuarantineActionKind = "resolved" + QuarantineActionKindDeletedFile QuarantineActionKind = "deleted_file" + QuarantineActionKindDeletedAlbum QuarantineActionKind = "deleted_album" ) -func (e *LidarrQuarantineActionKind) Scan(src interface{}) error { +func (e *QuarantineActionKind) Scan(src interface{}) error { switch s := src.(type) { case []byte: - *e = LidarrQuarantineActionKind(s) + *e = QuarantineActionKind(s) case string: - *e = LidarrQuarantineActionKind(s) + *e = QuarantineActionKind(s) default: - return fmt.Errorf("unsupported scan type for LidarrQuarantineActionKind: %T", src) + return fmt.Errorf("unsupported scan type for QuarantineActionKind: %T", src) } return nil } -type NullLidarrQuarantineActionKind struct { - LidarrQuarantineActionKind LidarrQuarantineActionKind - Valid bool // Valid is true if LidarrQuarantineActionKind is not NULL +type NullQuarantineActionKind struct { + QuarantineActionKind QuarantineActionKind + Valid bool // Valid is true if QuarantineActionKind is not NULL } // Scan implements the Scanner interface. -func (ns *NullLidarrQuarantineActionKind) Scan(value interface{}) error { +func (ns *NullQuarantineActionKind) Scan(value interface{}) error { if value == nil { - ns.LidarrQuarantineActionKind, ns.Valid = "", false + ns.QuarantineActionKind, ns.Valid = "", false return nil } ns.Valid = true - return ns.LidarrQuarantineActionKind.Scan(value) + return ns.QuarantineActionKind.Scan(value) } // Value implements the driver Valuer interface. -func (ns NullLidarrQuarantineActionKind) Value() (driver.Value, error) { +func (ns NullQuarantineActionKind) Value() (driver.Value, error) { if !ns.Valid { return nil, nil } - return string(ns.LidarrQuarantineActionKind), nil + return string(ns.QuarantineActionKind), nil } -type LidarrQuarantineReason string +type QuarantineReason string const ( - LidarrQuarantineReasonBadRip LidarrQuarantineReason = "bad_rip" - LidarrQuarantineReasonWrongFile LidarrQuarantineReason = "wrong_file" - LidarrQuarantineReasonWrongTags LidarrQuarantineReason = "wrong_tags" - LidarrQuarantineReasonDuplicate LidarrQuarantineReason = "duplicate" - LidarrQuarantineReasonOther LidarrQuarantineReason = "other" + QuarantineReasonBadRip QuarantineReason = "bad_rip" + QuarantineReasonWrongFile QuarantineReason = "wrong_file" + QuarantineReasonWrongTags QuarantineReason = "wrong_tags" + QuarantineReasonDuplicate QuarantineReason = "duplicate" + QuarantineReasonOther QuarantineReason = "other" ) -func (e *LidarrQuarantineReason) Scan(src interface{}) error { +func (e *QuarantineReason) Scan(src interface{}) error { switch s := src.(type) { case []byte: - *e = LidarrQuarantineReason(s) + *e = QuarantineReason(s) case string: - *e = LidarrQuarantineReason(s) + *e = QuarantineReason(s) default: - return fmt.Errorf("unsupported scan type for LidarrQuarantineReason: %T", src) + return fmt.Errorf("unsupported scan type for QuarantineReason: %T", src) } return nil } -type NullLidarrQuarantineReason struct { - LidarrQuarantineReason LidarrQuarantineReason - Valid bool // Valid is true if LidarrQuarantineReason is not NULL +type NullQuarantineReason struct { + QuarantineReason QuarantineReason + Valid bool // Valid is true if QuarantineReason is not NULL } // Scan implements the Scanner interface. -func (ns *NullLidarrQuarantineReason) Scan(value interface{}) error { +func (ns *NullQuarantineReason) Scan(value interface{}) error { if value == nil { - ns.LidarrQuarantineReason, ns.Valid = "", false + ns.QuarantineReason, ns.Valid = "", false return nil } ns.Valid = true - return ns.LidarrQuarantineReason.Scan(value) + return ns.QuarantineReason.Scan(value) } // Value implements the driver Valuer interface. -func (ns NullLidarrQuarantineReason) Value() (driver.Value, error) { +func (ns NullQuarantineReason) Value() (driver.Value, error) { if !ns.Valid { return nil, nil } - return string(ns.LidarrQuarantineReason), nil + return string(ns.QuarantineReason), nil } -type LidarrRequestKind string +type RequestKind string const ( - LidarrRequestKindArtist LidarrRequestKind = "artist" - LidarrRequestKindAlbum LidarrRequestKind = "album" - LidarrRequestKindTrack LidarrRequestKind = "track" + RequestKindArtist RequestKind = "artist" + RequestKindAlbum RequestKind = "album" + RequestKindTrack RequestKind = "track" ) -func (e *LidarrRequestKind) Scan(src interface{}) error { +func (e *RequestKind) Scan(src interface{}) error { switch s := src.(type) { case []byte: - *e = LidarrRequestKind(s) + *e = RequestKind(s) case string: - *e = LidarrRequestKind(s) + *e = RequestKind(s) default: - return fmt.Errorf("unsupported scan type for LidarrRequestKind: %T", src) + return fmt.Errorf("unsupported scan type for RequestKind: %T", src) } return nil } -type NullLidarrRequestKind struct { - LidarrRequestKind LidarrRequestKind - Valid bool // Valid is true if LidarrRequestKind is not NULL +type NullRequestKind struct { + RequestKind RequestKind + Valid bool // Valid is true if RequestKind is not NULL } // Scan implements the Scanner interface. -func (ns *NullLidarrRequestKind) Scan(value interface{}) error { +func (ns *NullRequestKind) Scan(value interface{}) error { if value == nil { - ns.LidarrRequestKind, ns.Valid = "", false + ns.RequestKind, ns.Valid = "", false return nil } ns.Valid = true - return ns.LidarrRequestKind.Scan(value) + return ns.RequestKind.Scan(value) } // Value implements the driver Valuer interface. -func (ns NullLidarrRequestKind) Value() (driver.Value, error) { +func (ns NullRequestKind) Value() (driver.Value, error) { if !ns.Valid { return nil, nil } - return string(ns.LidarrRequestKind), nil + return string(ns.RequestKind), nil } -type LidarrRequestStatus string +type RequestStatus string const ( - LidarrRequestStatusPending LidarrRequestStatus = "pending" - LidarrRequestStatusApproved LidarrRequestStatus = "approved" - LidarrRequestStatusRejected LidarrRequestStatus = "rejected" - LidarrRequestStatusCompleted LidarrRequestStatus = "completed" - LidarrRequestStatusFailed LidarrRequestStatus = "failed" + RequestStatusPending RequestStatus = "pending" + RequestStatusApproved RequestStatus = "approved" + RequestStatusRejected RequestStatus = "rejected" + RequestStatusCompleted RequestStatus = "completed" + RequestStatusFailed RequestStatus = "failed" ) -func (e *LidarrRequestStatus) Scan(src interface{}) error { +func (e *RequestStatus) Scan(src interface{}) error { switch s := src.(type) { case []byte: - *e = LidarrRequestStatus(s) + *e = RequestStatus(s) case string: - *e = LidarrRequestStatus(s) + *e = RequestStatus(s) default: - return fmt.Errorf("unsupported scan type for LidarrRequestStatus: %T", src) + return fmt.Errorf("unsupported scan type for RequestStatus: %T", src) } return nil } -type NullLidarrRequestStatus struct { - LidarrRequestStatus LidarrRequestStatus - Valid bool // Valid is true if LidarrRequestStatus is not NULL +type NullRequestStatus struct { + RequestStatus RequestStatus + Valid bool // Valid is true if RequestStatus is not NULL } // Scan implements the Scanner interface. -func (ns *NullLidarrRequestStatus) Scan(value interface{}) error { +func (ns *NullRequestStatus) Scan(value interface{}) error { if value == nil { - ns.LidarrRequestStatus, ns.Valid = "", false + ns.RequestStatus, ns.Valid = "", false return nil } ns.Valid = true - return ns.LidarrRequestStatus.Scan(value) + return ns.RequestStatus.Scan(value) } // Value implements the driver Valuer interface. -func (ns NullLidarrRequestStatus) Value() (driver.Value, error) { +func (ns NullRequestStatus) Value() (driver.Value, error) { if !ns.Valid { return nil, nil } - return string(ns.LidarrRequestStatus), nil + return string(ns.RequestStatus), nil } type AcoustidSetting struct { @@ -195,6 +195,46 @@ type AcoustidSetting struct { UpdatedAt pgtype.Timestamptz } +type AcquisitionService struct { + Code string + Kind string + Protocol string + Label string + Enabled bool + BaseUrl string + ApiKey string + Username string + Password string + Category string + TimeoutMs int32 + ServesSearch bool + ServesDownloads bool + ServesEnrichment bool + LocalPath string + RemotePath string + SearchesPerHour int32 + CreatedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz +} + +type AcquisitionSetting struct { + ID bool + AutoGrab bool + SearchIntervalHours int32 + AllowedQualities []string + LastPassAt pgtype.Timestamptz + LastPass []byte + UpdatedAt pgtype.Timestamptz + ImportRoot string + SeedRatio float32 + SeedDays int32 + StallHours int32 + LibraryListingStartedAt pgtype.Timestamptz + LibraryListedAt pgtype.Timestamptz + LibraryUnlisted int32 + UpgradeUntil string +} + type Album struct { ID pgtype.UUID Title string @@ -322,6 +362,23 @@ type DiscoverTuning struct { UpdatedAt pgtype.Timestamptz } +type DownloadIntervention struct { + ID pgtype.UUID + Client string + InfoHash string + GrabID pgtype.UUID + Name string + Trigger string + Action string + Outcome string + Detail string + Observations int32 + Remediations int32 + FirstSeen pgtype.Timestamptz + LastSeen pgtype.Timestamptz + SettledAt pgtype.Timestamptz +} + type DuplicateGroup struct { ID pgtype.UUID MemberKey string @@ -337,9 +394,8 @@ type DuplicateGroup struct { } type DuplicateGroupMember struct { - GroupID pgtype.UUID - TrackID pgtype.UUID - LidarrState *string + GroupID pgtype.UUID + TrackID pgtype.UUID } type DuplicateSweep struct { @@ -363,6 +419,24 @@ type FingerprintSetting struct { AutoResolve bool } +type Follow struct { + ArtistMbid string + ArtistName string + ProfileID pgtype.UUID + FollowedBy pgtype.UUID + FollowedAt pgtype.Timestamptz + BaselineAt pgtype.Timestamptz + WantExisting bool + CheckedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz +} + +type FollowSeenGroup struct { + ArtistMbid string + ReleaseGroupMbid string + SeenAt pgtype.Timestamptz +} + type GeneralLike struct { UserID pgtype.UUID TrackID pgtype.UUID @@ -381,6 +455,49 @@ type GeneralLikesArtist struct { LikedAt pgtype.Timestamptz } +type Grab struct { + ID pgtype.UUID + WantID pgtype.UUID + InfoHash string + Name string + Key string + Indexer string + Source string + Client string + SavePath string + Quality string + Origin string + Forced bool + GrabbedBy pgtype.UUID + GrabbedAt pgtype.Timestamptz + EndedAt pgtype.Timestamptz + EndReason string + EndDetail string + Progress float32 + ImportState string + ImportReason string + ImportDetail []byte + ImportForced bool + ImportAttempts int32 + ImportUpdatedAt pgtype.Timestamptz + ImportedAt pgtype.Timestamptz + ImportedFiles int32 + Destination string + Seeding bool + Ratio float32 + SeedReleasedAt pgtype.Timestamptz +} + +type LibraryAlbumMatch struct { + AlbumID pgtype.UUID + Outcome string + Method string + ReleaseGroupMbid string + Detail string + DecidedAt pgtype.Timestamptz + Matcher int32 +} + type LibraryChange struct { ID int64 EntityType string @@ -389,62 +506,10 @@ type LibraryChange struct { ChangedAt pgtype.Timestamptz } -type LidarrConfig struct { - ID int16 - Enabled bool - BaseUrl *string - ApiKey *string - DefaultQualityProfileID *int32 - DefaultRootFolderPath *string - CreatedAt pgtype.Timestamptz - UpdatedAt pgtype.Timestamptz - DefaultMetadataProfileID *int32 -} - -type LidarrQuarantine struct { - UserID pgtype.UUID - TrackID pgtype.UUID - Reason LidarrQuarantineReason - Notes *string - CreatedAt pgtype.Timestamptz -} - -type LidarrQuarantineAction struct { - ID pgtype.UUID - TrackID pgtype.UUID - TrackTitle string - ArtistName string - AlbumTitle *string - Action LidarrQuarantineActionKind - AdminID pgtype.UUID - LidarrAlbumMbid *string - AffectedUsers int32 - CreatedAt pgtype.Timestamptz -} - -type LidarrRequest struct { - ID pgtype.UUID - UserID pgtype.UUID - Status LidarrRequestStatus - Kind LidarrRequestKind - LidarrArtistMbid string - LidarrAlbumMbid *string - LidarrTrackMbid *string - ArtistName string - AlbumTitle *string - TrackTitle *string - QualityProfileID *int32 - RootFolderPath *string - DecidedAt pgtype.Timestamptz - DecidedBy pgtype.UUID - Notes *string - CompletedAt pgtype.Timestamptz - MatchedTrackID pgtype.UUID - MatchedAlbumID pgtype.UUID - MatchedArtistID pgtype.UUID - RequestedAt pgtype.Timestamptz - UpdatedAt pgtype.Timestamptz - LidarrAddConfirmedAt pgtype.Timestamptz +type LibrarySetting struct { + ID bool + ScanPaths []string + UpdatedAt pgtype.Timestamptz } type ListenbrainzSimilarRecording struct { @@ -461,14 +526,57 @@ type LoudnessSetting struct { LeveledCacheMb int32 } -type MissingReacquisition struct { - AlbumID pgtype.UUID - Attempts int32 - LastAttemptAt pgtype.Timestamptz - LastRequestID pgtype.UUID - GaveUpAt pgtype.Timestamptz - CreatedAt pgtype.Timestamptz - UpdatedAt pgtype.Timestamptz +type MbArtist struct { + Mbid string + Name string + SortName string + Disambiguation string + ArtistType string + Country string + DiscographyRefreshedAt pgtype.Timestamptz + CreatedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz +} + +type MbArtistReleaseGroup struct { + ArtistMbid string + ReleaseGroupMbid string +} + +type MbRelease struct { + Mbid string + ReleaseGroupMbid string + Title string + Status string + ReleaseDate string + Country string + Disambiguation string + Formats string + TrackCount int32 + CreatedAt pgtype.Timestamptz +} + +type MbReleaseGroup struct { + Mbid string + Title string + PrimaryType string + SecondaryTypes []string + FirstReleaseDate string + ArtistCredit string + ReleasesRefreshedAt pgtype.Timestamptz + CreatedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz +} + +type MbReleaseTrack struct { + ReleaseMbid string + MediumPosition int32 + Position int32 + MediumFormat string + Number string + Title string + LengthMs int32 + RecordingMbid string } type NetworkSetting struct { @@ -561,15 +669,32 @@ type PlaylistTrack struct { PickKind *string } +type Quarantine struct { + UserID pgtype.UUID + TrackID pgtype.UUID + Reason QuarantineReason + Notes *string + CreatedAt pgtype.Timestamptz +} + +type QuarantineAction struct { + ID pgtype.UUID + TrackID pgtype.UUID + TrackTitle string + ArtistName string + AlbumTitle *string + Action QuarantineActionKind + AdminID pgtype.UUID + AlbumMbid *string + AffectedUsers int32 + CreatedAt pgtype.Timestamptz +} + type ReacquisitionSetting struct { - ID bool - Enabled bool - GraceHours int32 - BackoffBaseHours int32 - BackoffMaxHours int32 - MaxAttempts int32 - MaxPerPass int32 - AutoApprove bool + ID bool + Enabled bool + GraceHours int32 + MaxPerPass int32 } type RecommendationTuningAudit struct { @@ -599,6 +724,49 @@ type RegistrationSetting struct { Mode string } +type ReleaseBlock struct { + InfoHash string + Name string + Reason string + Detail string + WantID pgtype.UUID + BlockedBy pgtype.UUID + BlockedAt pgtype.Timestamptz + ExpiresAt pgtype.Timestamptz +} + +type ReleaseTypeProfile struct { + ID pgtype.UUID + Name string + PrimaryTypes []string + SecondaryTypes []string + IsDefault bool + CreatedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz +} + +type Request struct { + ID pgtype.UUID + UserID pgtype.UUID + Status RequestStatus + Kind RequestKind + ArtistMbid string + AlbumMbid *string + TrackMbid *string + ArtistName string + AlbumTitle *string + TrackTitle *string + DecidedAt pgtype.Timestamptz + DecidedBy pgtype.UUID + Notes *string + CompletedAt pgtype.Timestamptz + MatchedTrackID pgtype.UUID + MatchedAlbumID pgtype.UUID + MatchedArtistID pgtype.UUID + RequestedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz +} + type ScanRun struct { ID pgtype.UUID StartedAt pgtype.Timestamptz @@ -758,6 +926,7 @@ type Track struct { SongKey pgtype.UUID SourceVerdict *string SourceVerdictAt pgtype.Timestamptz + EmptiedAt pgtype.Timestamptz } type TrackAcoustidLookup struct { @@ -876,6 +1045,54 @@ type UserNotificationPref struct { UpdatedAt pgtype.Timestamptz } +type Want struct { + ID pgtype.UUID + ReleaseGroupMbid string + ArtistMbid string + ArtistName string + Title string + PrimaryType string + FirstReleaseDate string + Reason string + AlbumID pgtype.UUID + CreatedBy pgtype.UUID + WantedAt pgtype.Timestamptz + FulfilledAt pgtype.Timestamptz + CreatedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz + SearchedAt pgtype.Timestamptz + SearchOutcome string + SearchConsidered int32 + BestRelease string + BestVerdict string + SearchError string + Upgrade string + HeldTier string + HeldQuality string + UpgradeCheckedAt pgtype.Timestamptz +} + +type WantCandidate struct { + WantID pgtype.UUID + Key string + Name string + InfoHash string + Indexer string + Source string + SizeBytes int64 + Seeders int32 + FilesCount int32 + Format string + Quality string + Year int32 + Verdict string + Reason string + Reasons []byte + FoldedFrom []string + Rank int32 + SearchedAt pgtype.Timestamptz +} + type YouMightLikeAlbum struct { UserID pgtype.UUID AlbumID pgtype.UUID diff --git a/internal/db/dbq/notifications.sql.go b/internal/db/dbq/notifications.sql.go index d5c64d7f..4ea5343d 100644 --- a/internal/db/dbq/notifications.sql.go +++ b/internal/db/dbq/notifications.sql.go @@ -54,7 +54,7 @@ type InsertNotificationParams struct { EmailWanted bool } -// M489 notifications inbox (#726). Rows are written only through +// Notifications inbox. Rows are written only through // internal/notifications.Notifier, which decides recipients and honours each // recipient's inbox preference before it reaches these. // email_wanted is the recipient's email channel for this kind as of now. A @@ -122,7 +122,7 @@ type ListEmailPendingNotificationsRow struct { ReadAsOf pgtype.Timestamptz } -// Email digest (#5346) ------------------------------------------------------ +// Email digest ------------------------------------------------------ // Every unread, un-emailed row of a user who has an address, oldest first. // Read rows are never selected: the user has seen them, so they are not news. // read_as_of is the database's clock at the read, handed back to @@ -346,8 +346,9 @@ type MarkAllNotificationsReadParams struct { // up_to, when set, limits it to what existed when the user asked: a "mark // all read" queued offline and replayed later must not mark notices that -// arrived in between, which the user never saw. A coalesced row updated -// since then carries a newer created_at, so it stays unread too. +// arrived in between, which the user never saw. A coalesced row that grew +// since then is marked read with it: it arrived before they asked, and its +// growth is a quiet update to a notice they had seen. func (q *Queries) MarkAllNotificationsRead(ctx context.Context, arg MarkAllNotificationsReadParams) (int64, error) { result, err := q.db.Exec(ctx, markAllNotificationsRead, arg.UserID, arg.UpTo) if err != nil { @@ -457,15 +458,15 @@ VALUES ($1, $2, $3, $4, CASE WHEN $5::boolean THEN NULL ELSE now() END) ON CONFLICT (user_id, coalesce_key) WHERE read_at IS NULL AND coalesce_key IS NOT NULL DO UPDATE SET - payload = CASE - WHEN $6::boolean THEN - EXCLUDED.payload || jsonb_build_object('count', - COALESCE((user_notifications.payload->>'count')::bigint, 0) - + COALESCE((EXCLUDED.payload->>'count')::bigint, 0)) - ELSE EXCLUDED.payload - END, - created_at = now(), - emailed_at = EXCLUDED.emailed_at + payload = EXCLUDED.payload || jsonb_build_object( + 'count', COALESCE((user_notifications.payload->>'count')::bigint, 0) + + COALESCE((EXCLUDED.payload->>'count')::bigint, 0), + 'items', (SELECT COALESCE(jsonb_agg(e.item ORDER BY e.n), '[]'::jsonb) + FROM jsonb_array_elements( + COALESCE(EXCLUDED.payload->'items', '[]'::jsonb) + || COALESCE(user_notifications.payload->'items', '[]'::jsonb) + ) WITH ORDINALITY AS e(item, n) + WHERE e.n <= 10)) RETURNING id ` @@ -475,20 +476,18 @@ type UpsertCoalescedNotificationParams struct { Payload []byte CoalesceKey *string EmailWanted bool - SumCount bool } // One unread row per (user, coalesce_key). A new event while that row is -// unread updates it in place and moves it back to the top of the inbox. +// unread updates it in place: its `count` adds to the row's, and its `items` +// (the names of what changed) go in front of the row's, keeping the newest +// ten. Every coalesced kind's event is "N more happened", never a restated +// total, so the row always says what is new since it was last read. // -// sum_count: the payload's `count` adds to the unread row's count rather than -// replacing it. For kinds whose event is "N more happened" (tracks marked -// missing). Kinds whose event states the whole current total (pending -// duplicate groups) pass false and the payload simply replaces. -// -// emailed_at is cleared so the newer state goes out in the next batch; an -// emailed-but-unread row that keeps growing is news the user hasn't seen. -// Unless email_wanted is false, as in InsertNotification. +// created_at and emailed_at are left alone: an update changes what the +// notice says, not when it arrived. Moving created_at would make the phone +// announce it again and clearing emailed_at would send it again, so a notice +// that keeps growing would alert on every update. func (q *Queries) UpsertCoalescedNotification(ctx context.Context, arg UpsertCoalescedNotificationParams) (pgtype.UUID, error) { row := q.db.QueryRow(ctx, upsertCoalescedNotification, arg.UserID, @@ -496,7 +495,6 @@ func (q *Queries) UpsertCoalescedNotification(ctx context.Context, arg UpsertCoa arg.Payload, arg.CoalesceKey, arg.EmailWanted, - arg.SumCount, ) var id pgtype.UUID err := row.Scan(&id) diff --git a/internal/db/dbq/playback_errors.sql.go b/internal/db/dbq/playback_errors.sql.go index 53988c4e..607441b9 100644 --- a/internal/db/dbq/playback_errors.sql.go +++ b/internal/db/dbq/playback_errors.sql.go @@ -11,17 +11,6 @@ import ( "github.com/jackc/pgx/v5/pgtype" ) -const countUnresolvedPlaybackErrors = `-- name: CountUnresolvedPlaybackErrors :one -SELECT count(*)::bigint FROM playback_errors WHERE resolved_at IS NULL -` - -func (q *Queries) CountUnresolvedPlaybackErrors(ctx context.Context) (int64, error) { - row := q.db.QueryRow(ctx, countUnresolvedPlaybackErrors) - var column_1 int64 - err := row.Scan(&column_1) - return column_1, err -} - const insertPlaybackError = `-- name: InsertPlaybackError :one INSERT INTO playback_errors (track_id, user_id, client_id, kind, detail) VALUES ($1, $2, $3, $4, $5) diff --git a/internal/db/dbq/playlists.sql.go b/internal/db/dbq/playlists.sql.go index ce928088..78c134bd 100644 --- a/internal/db/dbq/playlists.sql.go +++ b/internal/db/dbq/playlists.sql.go @@ -38,7 +38,7 @@ type AppendPlaylistTrackParams struct { // Inserts at the next available position. Snapshot fields are copied // from the tracks/albums/artists join at insert time. tracks.duration_ms // is converted to seconds for the snapshot. pick_kind is NULL for manual -// playlists; the For You builder stamps 'taste'/'fresh' (#1249). +// playlists; the For You builder stamps 'taste'/'fresh'. func (q *Queries) AppendPlaylistTrack(ctx context.Context, arg AppendPlaylistTrackParams) (PlaylistTrack, error) { row := q.db.QueryRow(ctx, appendPlaylistTrack, arg.PlaylistID, arg.PickKind, arg.TrackID) var i PlaylistTrack @@ -182,7 +182,7 @@ SELECT id, user_id, name, description, is_public, cover_path, track_count, durat ` // Batched lookup used by /api/library/sync to hydrate upsert payloads -// (#357). Mirror of GetArtistsByIDs. +// . Mirror of GetArtistsByIDs. func (q *Queries) GetPlaylistsByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([]Playlist, error) { rows, err := q.db.Query(ctx, getPlaylistsByIDs, dollar_1) if err != nil { @@ -295,7 +295,7 @@ type ListPlaylistTracksRow struct { // (track was removed from the library). The denormalized snapshot fields // on playlist_tracks remain authoritative for title/artist/album text. // -// Deliberately NOT filtered on missing_since (#2527), unlike every browse / +// Deliberately NOT filtered on missing_since, unlike every browse / // discover / mix query. A playlist entry is something the user put here on // purpose, so a missing file stays in the list and renders as a dead row // rather than silently vanishing; missing_since rides along so the service diff --git a/internal/db/dbq/lidarr_quarantine.sql.go b/internal/db/dbq/quarantine.sql.go similarity index 81% rename from internal/db/dbq/lidarr_quarantine.sql.go rename to internal/db/dbq/quarantine.sql.go index eac2ac6d..2e814f34 100644 --- a/internal/db/dbq/lidarr_quarantine.sql.go +++ b/internal/db/dbq/quarantine.sql.go @@ -1,7 +1,7 @@ // Code generated by sqlc. DO NOT EDIT. // versions: // sqlc v1.31.1 -// source: lidarr_quarantine.sql +// source: quarantine.sql package dbq @@ -12,7 +12,7 @@ import ( ) const countQuarantineForTrack = `-- name: CountQuarantineForTrack :one -SELECT count(*)::int FROM lidarr_quarantine WHERE track_id = $1 +SELECT count(*)::int FROM quarantine WHERE track_id = $1 ` // Reads affected_users for the audit row before the delete fires. @@ -24,7 +24,7 @@ func (q *Queries) CountQuarantineForTrack(ctx context.Context, trackID pgtype.UU } const deleteQuarantine = `-- name: DeleteQuarantine :one -DELETE FROM lidarr_quarantine +DELETE FROM quarantine WHERE user_id = $1 AND track_id = $2 RETURNING user_id, track_id, reason, notes, created_at ` @@ -36,9 +36,9 @@ type DeleteQuarantineParams struct { // Removes the caller's row. Returns the deleted row so the handler can // distinguish "no row existed" (zero rows -> ErrNoRows) from success. -func (q *Queries) DeleteQuarantine(ctx context.Context, arg DeleteQuarantineParams) (LidarrQuarantine, error) { +func (q *Queries) DeleteQuarantine(ctx context.Context, arg DeleteQuarantineParams) (Quarantine, error) { row := q.db.QueryRow(ctx, deleteQuarantine, arg.UserID, arg.TrackID) - var i LidarrQuarantine + var i Quarantine err := row.Scan( &i.UserID, &i.TrackID, @@ -50,7 +50,7 @@ func (q *Queries) DeleteQuarantine(ctx context.Context, arg DeleteQuarantinePara } const deleteQuarantineForTrack = `-- name: DeleteQuarantineForTrack :exec -DELETE FROM lidarr_quarantine WHERE track_id = $1 +DELETE FROM quarantine WHERE track_id = $1 ` // Clears all per-user rows for a given track. Used by Resolve and the @@ -68,26 +68,24 @@ SELECT ar.name AS artist_name, al.title AS album_title, al.id AS album_id, - al.mbid AS lidarr_album_mbid, count(q.user_id)::int AS report_count, max(q.created_at)::timestamptz AS latest_at -FROM lidarr_quarantine q +FROM quarantine q JOIN tracks t ON t.id = q.track_id JOIN albums al ON al.id = t.album_id JOIN artists ar ON ar.id = t.artist_id -GROUP BY t.id, ar.name, al.title, al.id, al.mbid +GROUP BY t.id, ar.name, al.title, al.id ORDER BY max(q.created_at) DESC ` type ListAdminQuarantineQueueRow struct { - TrackID pgtype.UUID - TrackTitle string - ArtistName string - AlbumTitle string - AlbumID pgtype.UUID - LidarrAlbumMbid *string - ReportCount int32 - LatestAt pgtype.Timestamptz + TrackID pgtype.UUID + TrackTitle string + ArtistName string + AlbumTitle string + AlbumID pgtype.UUID + ReportCount int32 + LatestAt pgtype.Timestamptz } // Aggregated admin queue. One row per track. The handler post-processes @@ -108,7 +106,6 @@ func (q *Queries) ListAdminQuarantineQueue(ctx context.Context) ([]ListAdminQuar &i.ArtistName, &i.AlbumTitle, &i.AlbumID, - &i.LidarrAlbumMbid, &i.ReportCount, &i.LatestAt, ); err != nil { @@ -123,20 +120,20 @@ func (q *Queries) ListAdminQuarantineQueue(ctx context.Context) ([]ListAdminQuar } const listQuarantineActions = `-- name: ListQuarantineActions :many -SELECT id, track_id, track_title, artist_name, album_title, action, admin_id, lidarr_album_mbid, affected_users, created_at FROM lidarr_quarantine_actions +SELECT id, track_id, track_title, artist_name, album_title, action, admin_id, album_mbid, affected_users, created_at FROM quarantine_actions ORDER BY created_at DESC LIMIT $1 ` -func (q *Queries) ListQuarantineActions(ctx context.Context, limit int32) ([]LidarrQuarantineAction, error) { +func (q *Queries) ListQuarantineActions(ctx context.Context, limit int32) ([]QuarantineAction, error) { rows, err := q.db.Query(ctx, listQuarantineActions, limit) if err != nil { return nil, err } defer rows.Close() - var items []LidarrQuarantineAction + var items []QuarantineAction for rows.Next() { - var i LidarrQuarantineAction + var i QuarantineAction if err := rows.Scan( &i.ID, &i.TrackID, @@ -145,7 +142,7 @@ func (q *Queries) ListQuarantineActions(ctx context.Context, limit int32) ([]Lid &i.AlbumTitle, &i.Action, &i.AdminID, - &i.LidarrAlbumMbid, + &i.AlbumMbid, &i.AffectedUsers, &i.CreatedAt, ); err != nil { @@ -170,7 +167,7 @@ SELECT al.cover_art_path AS album_cover_art_path, ar.id AS artist_id, ar.name AS artist_name -FROM lidarr_quarantine q +FROM quarantine q JOIN tracks t ON t.id = q.track_id JOIN albums al ON al.id = t.album_id JOIN artists ar ON ar.id = t.artist_id @@ -179,7 +176,7 @@ ORDER BY q.created_at DESC ` type ListQuarantineForUserRow struct { - LidarrQuarantine LidarrQuarantine + Quarantine Quarantine TrackIDJoin pgtype.UUID TrackTitle string TrackDurationMs int32 @@ -204,11 +201,11 @@ func (q *Queries) ListQuarantineForUser(ctx context.Context, userID pgtype.UUID) for rows.Next() { var i ListQuarantineForUserRow if err := rows.Scan( - &i.LidarrQuarantine.UserID, - &i.LidarrQuarantine.TrackID, - &i.LidarrQuarantine.Reason, - &i.LidarrQuarantine.Notes, - &i.LidarrQuarantine.CreatedAt, + &i.Quarantine.UserID, + &i.Quarantine.TrackID, + &i.Quarantine.Reason, + &i.Quarantine.Notes, + &i.Quarantine.CreatedAt, &i.TrackIDJoin, &i.TrackTitle, &i.TrackDurationMs, @@ -235,7 +232,7 @@ SELECT q.reason, q.notes, q.created_at -FROM lidarr_quarantine q +FROM quarantine q JOIN users u ON u.id = q.user_id WHERE q.track_id = $1 ORDER BY q.created_at DESC @@ -244,7 +241,7 @@ ORDER BY q.created_at DESC type ListQuarantineReportsForTrackRow struct { UserID pgtype.UUID Username string - Reason LidarrQuarantineReason + Reason QuarantineReason Notes *string CreatedAt pgtype.Timestamptz } @@ -278,7 +275,7 @@ func (q *Queries) ListQuarantineReportsForTrack(ctx context.Context, trackID pgt } const upsertQuarantine = `-- name: UpsertQuarantine :one -INSERT INTO lidarr_quarantine (user_id, track_id, reason, notes) +INSERT INTO quarantine (user_id, track_id, reason, notes) VALUES ($1, $2, $3, $4) ON CONFLICT (user_id, track_id) DO UPDATE SET reason = EXCLUDED.reason, @@ -290,20 +287,20 @@ RETURNING user_id, track_id, reason, notes, created_at type UpsertQuarantineParams struct { UserID pgtype.UUID TrackID pgtype.UUID - Reason LidarrQuarantineReason + Reason QuarantineReason Notes *string } // Insert a new quarantine row, or update reason/notes if the user has // already flagged this track. -func (q *Queries) UpsertQuarantine(ctx context.Context, arg UpsertQuarantineParams) (LidarrQuarantine, error) { +func (q *Queries) UpsertQuarantine(ctx context.Context, arg UpsertQuarantineParams) (Quarantine, error) { row := q.db.QueryRow(ctx, upsertQuarantine, arg.UserID, arg.TrackID, arg.Reason, arg.Notes, ) - var i LidarrQuarantine + var i Quarantine err := row.Scan( &i.UserID, &i.TrackID, @@ -315,26 +312,25 @@ func (q *Queries) UpsertQuarantine(ctx context.Context, arg UpsertQuarantinePara } const writeQuarantineAction = `-- name: WriteQuarantineAction :one -INSERT INTO lidarr_quarantine_actions ( +INSERT INTO quarantine_actions ( track_id, track_title, artist_name, album_title, - action, admin_id, lidarr_album_mbid, affected_users -) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) -RETURNING id, track_id, track_title, artist_name, album_title, action, admin_id, lidarr_album_mbid, affected_users, created_at + action, admin_id, affected_users +) VALUES ($1, $2, $3, $4, $5, $6, $7) +RETURNING id, track_id, track_title, artist_name, album_title, action, admin_id, album_mbid, affected_users, created_at ` type WriteQuarantineActionParams struct { - TrackID pgtype.UUID - TrackTitle string - ArtistName string - AlbumTitle *string - Action LidarrQuarantineActionKind - AdminID pgtype.UUID - LidarrAlbumMbid *string - AffectedUsers int32 + TrackID pgtype.UUID + TrackTitle string + ArtistName string + AlbumTitle *string + Action QuarantineActionKind + AdminID pgtype.UUID + AffectedUsers int32 } // Audit row for an admin destructive action. -func (q *Queries) WriteQuarantineAction(ctx context.Context, arg WriteQuarantineActionParams) (LidarrQuarantineAction, error) { +func (q *Queries) WriteQuarantineAction(ctx context.Context, arg WriteQuarantineActionParams) (QuarantineAction, error) { row := q.db.QueryRow(ctx, writeQuarantineAction, arg.TrackID, arg.TrackTitle, @@ -342,10 +338,9 @@ func (q *Queries) WriteQuarantineAction(ctx context.Context, arg WriteQuarantine arg.AlbumTitle, arg.Action, arg.AdminID, - arg.LidarrAlbumMbid, arg.AffectedUsers, ) - var i LidarrQuarantineAction + var i QuarantineAction err := row.Scan( &i.ID, &i.TrackID, @@ -354,7 +349,7 @@ func (q *Queries) WriteQuarantineAction(ctx context.Context, arg WriteQuarantine &i.AlbumTitle, &i.Action, &i.AdminID, - &i.LidarrAlbumMbid, + &i.AlbumMbid, &i.AffectedUsers, &i.CreatedAt, ) diff --git a/internal/db/dbq/reacquisition.sql.go b/internal/db/dbq/reacquisition.sql.go index 0a4e9da0..f82244f6 100644 --- a/internal/db/dbq/reacquisition.sql.go +++ b/internal/db/dbq/reacquisition.sql.go @@ -11,40 +11,18 @@ import ( "github.com/jackc/pgx/v5/pgtype" ) -const clearRecoveredReacquisitions = `-- name: ClearRecoveredReacquisitions :execrows -DELETE FROM missing_reacquisitions r - WHERE NOT EXISTS ( - SELECT 1 FROM tracks - WHERE tracks.album_id = r.album_id - AND tracks.missing_since IS NOT NULL - ) -` - -// Drops state for albums that no longer have any missing track — the files -// came back, or the scanner adopted them at a new path (#2528). Deleting -// rather than resetting counters means a future loss starts from a clean -// budget, which is right: it is a new problem, not a continuation. -func (q *Queries) ClearRecoveredReacquisitions(ctx context.Context) (int64, error) { - result, err := q.db.Exec(ctx, clearRecoveredReacquisitions) - if err != nil { - return 0, err - } - return result.RowsAffected(), nil -} - const countAlbumsMissingWithoutMbid = `-- name: CountAlbumsMissingWithoutMbid :one SELECT COUNT(DISTINCT albums.id)::bigint FROM albums JOIN artists ON artists.id = albums.artist_id JOIN tracks ON tracks.album_id = albums.id WHERE tracks.missing_since IS NOT NULL - AND (albums.mbid IS NULL OR artists.mbid IS NULL) + AND ((albums.mbid IS NULL AND albums.release_group_mbid IS NULL) OR artists.mbid IS NULL) ` -// Albums with missing files that can never be auto-requested because nothing -// identifies them to MusicBrainz. Surfaced on the admin card so the gap is -// visible: silently doing nothing for these would read as the feature being -// broken. +// Albums with missing files that can never be wanted automatically because +// nothing identifies them to MusicBrainz. Surfaced on the admin card so the +// gap is visible: silently doing nothing for these would read as broken. func (q *Queries) CountAlbumsMissingWithoutMbid(ctx context.Context) (int64, error) { row := q.db.QueryRow(ctx, countAlbumsMissingWithoutMbid) var column_1 int64 @@ -52,29 +30,56 @@ func (q *Queries) CountAlbumsMissingWithoutMbid(ctx context.Context) (int64, err return column_1, err } -const getReacquisitionForAlbums = `-- name: GetReacquisitionForAlbums :many -SELECT album_id, attempts, last_attempt_at, last_request_id, gave_up_at, created_at, updated_at FROM missing_reacquisitions WHERE album_id = ANY($1::uuid[]) +const getReacquisitionSettings = `-- name: GetReacquisitionSettings :one + +SELECT id, enabled, grace_hours, max_per_pass FROM reacquisition_settings WHERE id = true ` -// State for the admin missing-files surface, so each directory group can say -// whether a re-acquisition is in flight, waiting, or given up. -func (q *Queries) GetReacquisitionForAlbums(ctx context.Context, albumIds []pgtype.UUID) ([]MissingReacquisition, error) { - rows, err := q.db.Query(ctx, getReacquisitionForAlbums, albumIds) +// Re-acquisition of missing files. +// The unit is the album: an album whose files went missing becomes a want for +// its release group, and the search schedule looks for it. +func (q *Queries) GetReacquisitionSettings(ctx context.Context) (ReacquisitionSetting, error) { + row := q.db.QueryRow(ctx, getReacquisitionSettings) + var i ReacquisitionSetting + err := row.Scan( + &i.ID, + &i.Enabled, + &i.GraceHours, + &i.MaxPerPass, + ) + return i, err +} + +const getWantsForAlbums = `-- name: GetWantsForAlbums :many +SELECT a.id AS album_id, w.id AS want_id, w.wanted_at, w.fulfilled_at + FROM albums a + JOIN wants w ON w.album_id = a.id OR w.release_group_mbid = a.release_group_mbid + WHERE a.id = ANY($1::uuid[]) + ORDER BY w.wanted_at DESC +` + +type GetWantsForAlbumsRow struct { + AlbumID pgtype.UUID + WantID pgtype.UUID + WantedAt pgtype.Timestamptz + FulfilledAt pgtype.Timestamptz +} + +// What the wanted list says about each album on the missing-files page. +func (q *Queries) GetWantsForAlbums(ctx context.Context, albumIds []pgtype.UUID) ([]GetWantsForAlbumsRow, error) { + rows, err := q.db.Query(ctx, getWantsForAlbums, albumIds) if err != nil { return nil, err } defer rows.Close() - var items []MissingReacquisition + var items []GetWantsForAlbumsRow for rows.Next() { - var i MissingReacquisition + var i GetWantsForAlbumsRow if err := rows.Scan( &i.AlbumID, - &i.Attempts, - &i.LastAttemptAt, - &i.LastRequestID, - &i.GaveUpAt, - &i.CreatedAt, - &i.UpdatedAt, + &i.WantID, + &i.WantedAt, + &i.FulfilledAt, ); err != nil { return nil, err } @@ -86,67 +91,38 @@ func (q *Queries) GetReacquisitionForAlbums(ctx context.Context, albumIds []pgty return items, nil } -const getReacquisitionSettings = `-- name: GetReacquisitionSettings :one - -SELECT id, enabled, grace_hours, backoff_base_hours, backoff_max_hours, max_attempts, max_per_pass, auto_approve FROM reacquisition_settings WHERE id = true -` - -// Auto re-acquisition of missing files (milestone #290). The unit is the -// album: Lidarr acquires releases, and grouping collapses "40 missing files" -// into "3 albums to ask for". -func (q *Queries) GetReacquisitionSettings(ctx context.Context) (ReacquisitionSetting, error) { - row := q.db.QueryRow(ctx, getReacquisitionSettings) - var i ReacquisitionSetting - err := row.Scan( - &i.ID, - &i.Enabled, - &i.GraceHours, - &i.BackoffBaseHours, - &i.BackoffMaxHours, - &i.MaxAttempts, - &i.MaxPerPass, - &i.AutoApprove, - ) - return i, err -} - const listAlbumsDueReacquisition = `-- name: ListAlbumsDueReacquisition :many -SELECT albums.id AS album_id, - albums.title AS album_title, - albums.mbid AS album_mbid, +SELECT albums.id AS album_id, + albums.title AS album_title, + albums.mbid AS album_mbid, albums.release_group_mbid AS album_release_group_mbid, - artists.id AS artist_id, - artists.name AS artist_name, - artists.mbid AS artist_mbid, - COUNT(tracks.id)::bigint AS missing_track_count, - COALESCE(r.attempts, 0)::int AS attempts + artists.name AS artist_name, + artists.mbid AS artist_mbid, + COUNT(tracks.id)::bigint AS missing_track_count, + MAX(tracks.missing_since)::timestamptz AS last_missing_since FROM albums JOIN artists ON artists.id = albums.artist_id JOIN tracks ON tracks.album_id = albums.id - LEFT JOIN missing_reacquisitions r ON r.album_id = albums.id WHERE tracks.missing_since IS NOT NULL AND tracks.missing_since <= now() - make_interval(hours => $1::int) - AND albums.mbid IS NOT NULL + AND (albums.mbid IS NOT NULL OR albums.release_group_mbid IS NOT NULL) AND artists.mbid IS NOT NULL - AND (r.gave_up_at IS NULL) - AND ( - r.last_attempt_at IS NULL - OR r.last_attempt_at <= now() - make_interval(hours => LEAST( - ($2::int - * POWER(2, GREATEST(COALESCE(r.attempts, 0) - 1, 0)))::int, - $3::int)) - ) GROUP BY albums.id, albums.title, albums.mbid, albums.release_group_mbid, - artists.id, artists.name, artists.mbid, r.attempts, r.last_attempt_at - ORDER BY r.last_attempt_at NULLS FIRST, albums.sort_title - LIMIT $4 + artists.name, artists.mbid +HAVING EXISTS ( + SELECT 1 FROM wants w + WHERE w.album_id = albums.id OR w.release_group_mbid = albums.release_group_mbid) + AND NOT EXISTS ( + SELECT 1 FROM wants w + WHERE (w.album_id = albums.id OR w.release_group_mbid = albums.release_group_mbid) + AND (w.fulfilled_at IS NULL OR w.fulfilled_at > MAX(tracks.missing_since))) + ORDER BY MIN(tracks.missing_since), albums.sort_title + LIMIT $2 ` type ListAlbumsDueReacquisitionParams struct { - GraceHours int32 - BackoffBaseHours int32 - BackoffMaxHours int32 - PageLimit int32 + GraceHours int32 + PageLimit int32 } type ListAlbumsDueReacquisitionRow struct { @@ -154,37 +130,29 @@ type ListAlbumsDueReacquisitionRow struct { AlbumTitle string AlbumMbid *string AlbumReleaseGroupMbid *string - ArtistID pgtype.UUID ArtistName string ArtistMbid *string MissingTrackCount int64 - Attempts int32 + LastMissingSince pgtype.Timestamptz } // The sweeper's selection. An album qualifies when: // // - it still has at least one track whose file has been missing longer than // the grace window. Measured on missing_since, which reconcile never -// re-stamps (#2523), so it is a genuine "gone since" clock rather than -// "when we last noticed"; -// - MusicBrainz can name it. Both the album and its artist MBID are -// required — Create rejects an album-kind request without them, and there -// is nothing to ask Lidarr for anyway. Albums failing this are counted -// separately (CountAlbumsMissingWithoutMbid) rather than vanishing; -// - it has not spent its attempt budget (gave_up_at IS NULL); -// - its backoff has elapsed: base * 2^(attempts-1) hours since the last -// attempt, clamped to the configured maximum. First attempt (no row, or -// last_attempt_at NULL) is always due. -// -// Oldest attempt first, never-attempted first, so a large loss drains in a -// stable order across passes instead of re-picking the same head each time. +// re-stamps, so it is a genuine "gone since" clock; +// - MusicBrainz can name it: the artist has an MBID, and the album a +// release MBID or a release group. The release MBID is only ever used to +// find the group, and an album matched to a group may carry no release +// id. Albums failing this are counted separately +// (CountAlbumsMissingWithoutMbid); +// - it is on the wanted list: an album taken off the list, or +// dropped into the library by hand, is not fetched back; +// - no want already covers it: none open, and none held since its files +// went missing. A want held after the loss means the library has a full +// copy despite the missing rows, so asking again would loop. func (q *Queries) ListAlbumsDueReacquisition(ctx context.Context, arg ListAlbumsDueReacquisitionParams) ([]ListAlbumsDueReacquisitionRow, error) { - rows, err := q.db.Query(ctx, listAlbumsDueReacquisition, - arg.GraceHours, - arg.BackoffBaseHours, - arg.BackoffMaxHours, - arg.PageLimit, - ) + rows, err := q.db.Query(ctx, listAlbumsDueReacquisition, arg.GraceHours, arg.PageLimit) if err != nil { return nil, err } @@ -197,11 +165,10 @@ func (q *Queries) ListAlbumsDueReacquisition(ctx context.Context, arg ListAlbums &i.AlbumTitle, &i.AlbumMbid, &i.AlbumReleaseGroupMbid, - &i.ArtistID, &i.ArtistName, &i.ArtistMbid, &i.MissingTrackCount, - &i.Attempts, + &i.LastMissingSince, ); err != nil { return nil, err } @@ -213,101 +180,93 @@ func (q *Queries) ListAlbumsDueReacquisition(ctx context.Context, arg ListAlbums return items, nil } -const markReacquisitionGaveUp = `-- name: MarkReacquisitionGaveUp :exec -UPDATE missing_reacquisitions - SET gave_up_at = now(), - updated_at = now() - WHERE album_id = $1 - AND gave_up_at IS NULL -` - -// Stamped when the attempt budget is spent. Stored as a timestamp rather than -// inferred from `attempts >= max_attempts` so the verdict survives an operator -// later raising the maximum, and so the admin surface can say when. -func (q *Queries) MarkReacquisitionGaveUp(ctx context.Context, albumID pgtype.UUID) error { - _, err := q.db.Exec(ctx, markReacquisitionGaveUp, albumID) - return err -} - -const recordReacquisitionAttempt = `-- name: RecordReacquisitionAttempt :one -INSERT INTO missing_reacquisitions (album_id, attempts, last_attempt_at, last_request_id) -VALUES ($1, 1, now(), $2) -ON CONFLICT (album_id) DO UPDATE - SET attempts = missing_reacquisitions.attempts + 1, - last_attempt_at = now(), - last_request_id = COALESCE(EXCLUDED.last_request_id, - missing_reacquisitions.last_request_id), - updated_at = now() -RETURNING album_id, attempts, last_attempt_at, last_request_id, gave_up_at, created_at, updated_at -` - -type RecordReacquisitionAttemptParams struct { - AlbumID pgtype.UUID - LastRequestID pgtype.UUID -} - -// Bumps the attempt counter and stamps the clock the backoff measures from. -// Upsert because the first attempt has no row yet. -func (q *Queries) RecordReacquisitionAttempt(ctx context.Context, arg RecordReacquisitionAttemptParams) (MissingReacquisition, error) { - row := q.db.QueryRow(ctx, recordReacquisitionAttempt, arg.AlbumID, arg.LastRequestID) - var i MissingReacquisition - err := row.Scan( - &i.AlbumID, - &i.Attempts, - &i.LastAttemptAt, - &i.LastRequestID, - &i.GaveUpAt, - &i.CreatedAt, - &i.UpdatedAt, - ) - return i, err -} - const updateReacquisitionSettings = `-- name: UpdateReacquisitionSettings :one UPDATE reacquisition_settings - SET enabled = $1, - grace_hours = $2, - backoff_base_hours = $3, - backoff_max_hours = $4, - max_attempts = $5, - max_per_pass = $6, - auto_approve = $7 + SET enabled = $1, + grace_hours = $2, + max_per_pass = $3 WHERE id = true -RETURNING id, enabled, grace_hours, backoff_base_hours, backoff_max_hours, max_attempts, max_per_pass, auto_approve +RETURNING id, enabled, grace_hours, max_per_pass ` type UpdateReacquisitionSettingsParams struct { - Enabled bool - GraceHours int32 - BackoffBaseHours int32 - BackoffMaxHours int32 - MaxAttempts int32 - MaxPerPass int32 - AutoApprove bool + Enabled bool + GraceHours int32 + MaxPerPass int32 } // Whole-row write from the admin card; the CHECKs in migration 0056 are the // validation, so a bad value fails loudly rather than being clamped silently. func (q *Queries) UpdateReacquisitionSettings(ctx context.Context, arg UpdateReacquisitionSettingsParams) (ReacquisitionSetting, error) { - row := q.db.QueryRow(ctx, updateReacquisitionSettings, - arg.Enabled, - arg.GraceHours, - arg.BackoffBaseHours, - arg.BackoffMaxHours, - arg.MaxAttempts, - arg.MaxPerPass, - arg.AutoApprove, - ) + row := q.db.QueryRow(ctx, updateReacquisitionSettings, arg.Enabled, arg.GraceHours, arg.MaxPerPass) var i ReacquisitionSetting err := row.Scan( &i.ID, &i.Enabled, &i.GraceHours, - &i.BackoffBaseHours, - &i.BackoffMaxHours, - &i.MaxAttempts, &i.MaxPerPass, - &i.AutoApprove, + ) + return i, err +} + +const wantMissingAlbum = `-- name: WantMissingAlbum :one +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, album_id) +VALUES ($1, $2, $3, + $4, 'missing', $5) +ON CONFLICT (release_group_mbid) DO UPDATE + SET reason = 'missing', + album_id = EXCLUDED.album_id, + wanted_at = now(), + fulfilled_at = NULL, + updated_at = now() + WHERE wants.fulfilled_at IS NOT NULL +RETURNING id, release_group_mbid, artist_mbid, artist_name, title, primary_type, first_release_date, reason, album_id, created_by, wanted_at, fulfilled_at, created_at, updated_at, searched_at, search_outcome, search_considered, best_release, best_verdict, search_error, upgrade, held_tier, held_quality, upgrade_checked_at +` + +type WantMissingAlbumParams struct { + ReleaseGroupMbid string + ArtistMbid string + ArtistName string + Title string + AlbumID pgtype.UUID +} + +// Records an album with missing files as wanted. A held want for the same +// group is reopened; an open one is left as it is, and no row comes back. +func (q *Queries) WantMissingAlbum(ctx context.Context, arg WantMissingAlbumParams) (Want, error) { + row := q.db.QueryRow(ctx, wantMissingAlbum, + arg.ReleaseGroupMbid, + arg.ArtistMbid, + arg.ArtistName, + arg.Title, + arg.AlbumID, + ) + var i Want + err := row.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.Title, + &i.PrimaryType, + &i.FirstReleaseDate, + &i.Reason, + &i.AlbumID, + &i.CreatedBy, + &i.WantedAt, + &i.FulfilledAt, + &i.CreatedAt, + &i.UpdatedAt, + &i.SearchedAt, + &i.SearchOutcome, + &i.SearchConsidered, + &i.BestRelease, + &i.BestVerdict, + &i.SearchError, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.UpgradeCheckedAt, ) return i, err } diff --git a/internal/db/dbq/recommendation.sql.go b/internal/db/dbq/recommendation.sql.go index 442fafee..80c8aede 100644 --- a/internal/db/dbq/recommendation.sql.go +++ b/internal/db/dbq/recommendation.sql.go @@ -19,7 +19,7 @@ ORDER BY started_at DESC LIMIT 1 ` -// The device_class of the user's most recent play (#1551), used as the +// The device_class of the user's most recent play, used as the // "current device" for radio context conditioning. NULL when the latest play // predates device capture or came from a client that didn't send one. func (q *Queries) GetLatestPlayDeviceClassForUser(ctx context.Context, userID pgtype.UUID) (*string, error) { @@ -87,10 +87,10 @@ type ListArtistContextPlayCountsForUserRow struct { } // Per-artist completed-play counts split by whether each play falls in the -// CURRENT context cell, in the user's local timezone (#1531). The cell is -// daypart × weekday-type, optionally narrowed by device class (#1551): when -// $2 (the current device) is ”, the device dimension is ignored (identical to -// the time-only #1531 behaviour, used by the daily mixes which have no device); +// CURRENT context cell, in the user's local timezone. The cell is +// daypart × weekday-type, optionally narrowed by device class: when +// $2 (the current device) is ”, the device dimension is ignored (time only, +// for the daily mixes, which have no device); // when $2 is set (radio), a play only counts in the cell if its device_class // matches. Feeds the context-affinity scoring term. Skips excluded; a 365-day // window bounds cost. Daypart buckets: night [22,5) morning [5,12) @@ -154,7 +154,7 @@ type ListLastPlayedArtistsForUserRow struct { LastPlayedAt pgtype.Timestamptz } -// M6a: artists ranked by max(play_events.started_at) for the user, with +// Artists ranked by max(play_events.started_at) for the user, with // a derived cover_album_id via a representative-album lateral join (most // recent album that has cover_art_path set). album_count joined for the // ArtistRef wire shape. @@ -208,7 +208,7 @@ WITH plays AS ( WHERE user_id = $2 AND was_skipped = false GROUP BY track_id ) -SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, +SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at, albums.title AS album_title, artists.name AS artist_name FROM plays p @@ -216,9 +216,9 @@ JOIN tracks t ON t.id = p.track_id JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id WHERE t.artist_id = $1 - AND t.missing_since IS NULL -- #2523: never offer a file that is gone + AND t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = t.id ) ORDER BY p.cnt DESC, t.id @@ -275,6 +275,7 @@ func (q *Queries) ListMostPlayedTracksForArtist(ctx context.Context, arg ListMos &i.Track.SongKey, &i.Track.SourceVerdict, &i.Track.SourceVerdictAt, + &i.Track.EmptiedAt, &i.AlbumTitle, &i.ArtistName, ); err != nil { @@ -295,16 +296,16 @@ WITH plays AS ( WHERE user_id = $1 AND was_skipped = false GROUP BY track_id ) -SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, +SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at, albums.title AS album_title, artists.name AS artist_name FROM plays p JOIN tracks t ON t.id = p.track_id JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id -WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone +WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY p.cnt DESC, t.id @@ -322,7 +323,7 @@ type ListMostPlayedTracksForUserRow struct { ArtistName string } -// M6a: top-N tracks by completed-play count for the user. was_skipped +// Top-N tracks by completed-play count for the user. was_skipped // excludes skips so a user spamming next doesn't fabricate a top track. // Quarantined tracks (per-user soft-hide from M5b) are filtered out. // @@ -364,6 +365,7 @@ func (q *Queries) ListMostPlayedTracksForUser(ctx context.Context, arg ListMostP &i.Track.SongKey, &i.Track.SourceVerdict, &i.Track.SourceVerdictAt, + &i.Track.EmptiedAt, &i.AlbumTitle, &i.ArtistName, ); err != nil { @@ -703,7 +705,7 @@ func (q *Queries) ListRediscoverArtistsForUser(ctx context.Context, arg ListRedi const loadRadioCandidates = `-- name: LoadRadioCandidates :many SELECT - t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, + t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at, (l.user_id IS NOT NULL)::bool AS is_liked, pe.last_played_at::timestamptz AS last_played_at, pe.play_count, @@ -721,14 +723,14 @@ LEFT JOIN LATERAL ( WHERE user_id = $1 AND track_id = t.id ) pe ON true WHERE t.id <> $2 - AND t.missing_since IS NULL -- #2523: never offer a file that is gone + AND t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( SELECT 1 FROM play_events WHERE user_id = $1 AND track_id = t.id AND started_at > now() - $3 * interval '1 hour' ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ` @@ -789,6 +791,7 @@ func (q *Queries) LoadRadioCandidates(ctx context.Context, arg LoadRadioCandidat &i.Track.SongKey, &i.Track.SourceVerdict, &i.Track.SourceVerdictAt, + &i.Track.EmptiedAt, &i.IsLiked, &i.LastPlayedAt, &i.PlayCount, @@ -827,11 +830,11 @@ excluded_ids AS ( WHERE pe.user_id = $1 AND pe.started_at > now() - $3 * interval '1 hour' UNION ALL SELECT q.track_id AS id - FROM lidarr_quarantine q + FROM quarantine q WHERE q.user_id = $1 ), lb_similar AS ( - SELECT ts.track_b_id AS track_id, ts.score AS sim_score + SELECT ts.track_b_id AS track_id, 0.5 + 0.5 * ts.score AS sim_score FROM track_similarity ts WHERE ts.track_a_id = $2 AND ts.source = 'listenbrainz' @@ -840,11 +843,10 @@ lb_similar AS ( LIMIT $5 ), similar_artists AS ( - -- Round-robin across the similar artists (#5297): each artist's first + -- Round-robin across the similar artists: each artist's first -- track, best-scoring artist first, then each one's second, and so on. - -- Ordering by artist score alone let the closest similar artist's - -- catalogue fill the whole LIMIT; on the operator's library this arm held - -- exactly one artist for all 17 seeds measured (#3879). + -- Ordering by artist score alone would let the closest similar artist's + -- catalogue fill the whole LIMIT. SELECT track_id, sim_score FROM ( SELECT t.id AS track_id, asim.score * 0.5 AS sim_score, asim.score AS artist_score, @@ -861,8 +863,8 @@ similar_artists AS ( ), tag_overlap AS ( SELECT t.id AS track_id, - (count(DISTINCT trim(g))::float8 - / GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score + 0.5 * (count(DISTINCT trim(g))::float8 + / GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score FROM tracks t LEFT JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true WHERE trim(g_split.g) IN (SELECT tag FROM seed_tags) @@ -870,11 +872,15 @@ tag_overlap AS ( AND t.id <> $2 GROUP BY t.id HAVING count(DISTINCT trim(g_split.g)) > 0 - ORDER BY sim_score DESC + -- md5 breaks the ties. A seed with one tag gives every track of that + -- genre the same score, and with no tie-break the LIMIT would take + -- whichever rows Postgres reached first: the same slice for every seed of + -- the genre, and not one the order seed controls. + ORDER BY sim_score DESC, md5(t.id::text || $12::text) LIMIT $7 ), likes_overlap AS ( - SELECT gl.track_id, 0.6::float8 AS sim_score + SELECT gl.track_id, 0.3::float8 AS sim_score FROM general_likes gl WHERE gl.user_id = $1 AND gl.track_id NOT IN (SELECT id FROM excluded_ids) @@ -888,18 +894,27 @@ likes_overlap AS ( LIMIT $8 ), taste_overlap AS ( - SELECT t.id AS track_id, 0.0::float8 AS sim_score - FROM taste_profile_artists tpa - JOIN tracks t ON t.artist_id = tpa.artist_id - WHERE tpa.user_id = $1 - AND tpa.weight > 0 - AND t.id NOT IN (SELECT id FROM excluded_ids) - AND t.id <> $2 - ORDER BY tpa.weight DESC, t.id + -- Round-robin across the taste artists, as similar_artists is: each + -- artist's first track, strongest artist first, then each one's second. + -- Ordered by weight then track id, the strongest artist's catalogue would + -- fill the whole arm with the same lowest-id tracks every time. + SELECT track_id, sim_score + FROM ( + SELECT t.id AS track_id, 0.0::float8 AS sim_score, tpa.weight, + row_number() OVER (PARTITION BY tpa.artist_id + ORDER BY md5(t.id::text || $12::text)) AS artist_turn + FROM taste_profile_artists tpa + JOIN tracks t ON t.artist_id = tpa.artist_id AND t.missing_since IS NULL + WHERE tpa.user_id = $1 + AND tpa.weight > 0 + AND t.id NOT IN (SELECT id FROM excluded_ids) + AND t.id <> $2 + ) ranked + ORDER BY artist_turn, weight DESC, md5(track_id::text || $12::text) LIMIT $10 ), coplay_artists AS ( - -- Household co-play (#1533): tracks by artists co-played across the + -- Household co-play: tracks by artists co-played across the -- instance with the seed's artist (source='user_cooccurrence', built by -- the coplay worker). Mirrors similar_artists but from local co-occurrence -- instead of ListenBrainz; empty on single-user servers. Same 0.5 damp as @@ -937,7 +952,7 @@ random_fill AS ( LIMIT $9 ) SELECT - t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, + t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at, (l.user_id IS NOT NULL)::bool AS is_liked, pe.last_played_at::timestamptz AS last_played_at, pe.play_count, @@ -953,7 +968,7 @@ FROM ( UNION ALL SELECT track_id, sim_score FROM coplay_artists UNION ALL SELECT track_id, sim_score FROM random_fill ) u -JOIN tracks t ON t.id = u.track_id AND t.missing_since IS NULL -- #2523: never offer a file that is gone +JOIN tracks t ON t.id = u.track_id AND t.missing_since IS NULL -- never offer a file that is gone JOIN albums al ON al.id = t.album_id LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id LEFT JOIN LATERAL ( @@ -995,26 +1010,23 @@ type LoadRadioCandidatesV2Row struct { SimilarityScore interface{} } -// M4c: similarity-driven candidate pool. 6-way UNION: +// Similarity-driven candidate pool. 6-way UNION: // // $1 user_id, $2 seed_track_id, $3 recently_played_hours, // $4 exclude (uuid[]), $5 lb_similar K, $6 similar_artists K, // $7 tag_overlap K, $8 likes_overlap K, $9 random_fill K, -// $10 taste_overlap K (#796 phase 2b — tracks by the user's top +// $10 taste_overlap K (tracks by the user's top // positively-weighted taste-profile artists, so taste-relevant tracks // enter the pool even when the similarity/random arms miss them; scored // in Go via TasteMatch, so sim_score here is 0 pool-inclusion), -// $11 coplay_artists K (#1533 — tracks by artists co-played across the +// $11 coplay_artists K (tracks by artists co-played across the // instance with the seed's artist; source='user_cooccurrence'), // $12 order_seed (text) — see below. // -// $12 REPLACES `ORDER BY random()` IN FOUR ARMS (#3889). Those arms returned -// a stable set only while their LIMIT exceeded the rows eligible for them: at -// that point they returned all of them and the order stopped mattering, -// because the caller sorts by track id before scoring. Below that threshold -// they returned a random SUBSET, and two builds on the same day drew -// different ones — so "daily determinism" held by accident, and only for -// libraries smaller than the limits. +// $12 orders four arms by md5(id || seed), not random(). An arm whose LIMIT is +// below the rows eligible for it returns a SUBSET, and with random() two +// builds on the same day would draw different ones; the caller sorts by track +// id before scoring, so only the subset matters. // // md5(id || seed) keeps the intent — an arbitrary spread that changes when // the seed does — while making it reproducible for a given seed. The CALLER @@ -1022,6 +1034,20 @@ type LoadRadioCandidatesV2Row struct { // the determinism they promise; radio passes a fresh value per request and // keeps varying, which is what a radio should do. // Returns same shape as LoadRadioCandidates plus similarity_score column. +// +// sim_score BANDS keep the eligibility tiers in order. Similarity edges +// are stored relative to the seed's best match, 0 to 1, so the arms are placed +// in bands rather than compared raw: +// +// tier 1 lb_similar 0.5 + 0.5 × edge (0.5, 1.0] +// tier 2 similar_artists, coplay_artists 0.5 × edge [0, 0.5] +// tag_overlap 0.5 × share of the seed's tags +// likes_overlap 0.3, flat +// tier 3 taste_overlap, random_fill 0 +// +// Without the bands, the seed's weaker LB edges would fall below a shared +// genre tag and the flat likes score, and the one arm that measures the seed +// would lose to the ones that guess. func (q *Queries) LoadRadioCandidatesV2(ctx context.Context, arg LoadRadioCandidatesV2Params) ([]LoadRadioCandidatesV2Row, error) { rows, err := q.db.Query(ctx, loadRadioCandidatesV2, arg.UserID, @@ -1069,6 +1095,7 @@ func (q *Queries) LoadRadioCandidatesV2(ctx context.Context, arg LoadRadioCandid &i.Track.SongKey, &i.Track.SourceVerdict, &i.Track.SourceVerdictAt, + &i.Track.EmptiedAt, &i.IsLiked, &i.LastPlayedAt, &i.PlayCount, @@ -1088,9 +1115,9 @@ func (q *Queries) LoadRadioCandidatesV2(ctx context.Context, arg LoadRadioCandid const suggestArtistsForUser = `-- name: SuggestArtistsForUser :many WITH artist_plays AS ( - -- Completed plays only. The previous seed query counted every play_event, - -- so skipping an artist repeatedly INCREASED its signal and pushed more of - -- its neighbours at the user (issue #2367 mechanism 3). + -- Completed plays only: counting every play_event would make skipping an + -- artist repeatedly INCREASE its signal and push more of its neighbours + -- at the user. SELECT t.artist_id, count(*)::bigint AS play_count FROM play_events pe JOIN tracks t ON t.id = pe.track_id @@ -1142,12 +1169,12 @@ contributions AS ( JOIN seeds ON seeds.artist_id = u.seed_artist_id WHERE NOT EXISTS (SELECT 1 FROM artists WHERE mbid = u.candidate_mbid) AND NOT EXISTS ( - SELECT 1 FROM lidarr_requests r + SELECT 1 FROM requests r WHERE r.user_id = $1 - AND r.lidarr_artist_mbid = u.candidate_mbid + AND r.artist_mbid = u.candidate_mbid AND r.status NOT IN ('rejected', 'failed') ) - -- Snoozed by this user and not yet expired (#2374). Time-boxed and + -- Snoozed by this user and not yet expired. Time-boxed and -- per-user: the candidate returns on its own once snoozed_until -- passes, and stays visible to everyone else meanwhile. Deliberately -- filtered at the candidate stage, NOT folded into the score — a @@ -1192,7 +1219,7 @@ type SuggestArtistsForUserRow struct { // Per-user artist suggestions ranked by taste signal x similarity, projected // through artist_similarity_unmatched (out-of-library candidates only). // -// Seeds are TIERED (rule #131) so the surface never empties: +// Seeds are TIERED so the surface never empties: // // tier 1 - taste_profile_artists.weight: engagement-graded, time-decayed and // SIGNED by internal/taste, so an artist the user has drifted away @@ -1200,9 +1227,9 @@ type SuggestArtistsForUserRow struct { // tier 2 - likes + completed plays, used ONLY when the profile has no rows // (new account, or before the first daily recompute). // -// The signal is log-damped: contribution is signal x similarity, and the old -// undamped sum let one heavily-played artist's neighbours take every slot -- -// entrenching harder the MORE the user listened (issue #2367 mechanism 2). +// The signal is log-damped: contribution is signal x similarity, and an +// undamped sum would let one heavily-played artist's neighbours take every +// slot -- entrenching harder the MORE the user listened. // // Candidates already in the library, already requested and not terminal, or // snoozed by this user, are excluded. $1=user_id, $2=half_life_days (tier 2 diff --git a/internal/db/dbq/recommendation_metrics.sql.go b/internal/db/dbq/recommendation_metrics.sql.go index 1fe4528b..39527937 100644 --- a/internal/db/dbq/recommendation_metrics.sql.go +++ b/internal/db/dbq/recommendation_metrics.sql.go @@ -44,12 +44,12 @@ type RecommendationSourceMetricsForUserRow struct { // $1 user_id, $2 window_days. plays/skips are counts; avg_completion is the // mean completion ratio over the completion_n plays that recorded one. -// pick_kind splits For You plays into taste/fresh/unattributed (#1249); +// pick_kind splits For You plays into taste/fresh/unattributed; // it is NULL for every other source, so those still group to one row. // // completion_sqsum carries the sum of SQUARED completion ratios so the Go // handler can compute a variance — needed for the margin of error on a -// completion delta (#2495). It is the sum rather than `stddev_samp` on purpose: +// completion delta. It is the sum rather than `stddev_samp` on purpose: // raw source rows get merged into surface families in Go, and sums of squares // add across groups exactly, whereas standard deviations cannot be combined // without them. Variance = (sqsum - sum²/n) / (n-1), with sum = avg × n. @@ -112,17 +112,17 @@ type RecommendationWeeklyTrendsRow struct { TasteHits int64 } -// Recommendation observability (#796 phase 4). Per-source play outcomes so the +// Recommendation observability. Per-source play outcomes so the // operator can see whether each recommendation surface is landing and tune the // taste weights. Source is stamped on play_events when a play is launched from // a recommendation surface; NULL means the user picked the track manually — // those rows are INCLUDED here as the baseline control group the surfaces are -// judged against (milestone #127: delta-vs-baseline is what makes the numbers -// actionable). Raw source strings are bucketed into stable surface families in -// the Go handler; completion_n is carried so family merges can weight +// judged against (delta-vs-baseline is what makes the numbers actionable). +// Raw source strings are bucketed into stable surface families in the Go +// handler; completion_n is carried so family merges can weight // avg_completion correctly. // Weekly per-source outcome series for the tuning lab's trend view -// (#1251). Aggregated across ALL users: the tuning knobs are global, +// . Aggregated across ALL users: the tuning knobs are global, // so judging a knob turn needs global outcomes — rows carry rates // only, no track or user identity. NULL-source (manual) rows are // included as the baseline family. diff --git a/internal/db/dbq/recommendation_tuning.sql.go b/internal/db/dbq/recommendation_tuning.sql.go index 251eeed5..1e1f76c3 100644 --- a/internal/db/dbq/recommendation_tuning.sql.go +++ b/internal/db/dbq/recommendation_tuning.sql.go @@ -70,7 +70,7 @@ SELECT id, changed_at, scope, action, changes LIMIT $1 ` -// Newest first; consumed by the metrics trend view (#1251) to annotate +// Newest first; consumed by the metrics trend view to annotate // knob turns on the timeline. func (q *Queries) ListTuningAudit(ctx context.Context, limit int32) ([]RecommendationTuningAudit, error) { rows, err := q.db.Query(ctx, listTuningAudit, limit) @@ -279,8 +279,8 @@ type UpsertDiscoverTuningDefaultsParams struct { SnoozeDays float64 } -// Boot reconcile for the Discover scope (#2377). Never overwrites -// operator-tuned values, same contract as the other two. +// Boot reconcile for the Discover scope. Never overwrites +// admin-tuned values, same contract as the other two. func (q *Queries) UpsertDiscoverTuningDefaults(ctx context.Context, arg UpsertDiscoverTuningDefaultsParams) error { _, err := q.db.Exec(ctx, upsertDiscoverTuningDefaults, arg.TagOverlapWeight, arg.SnoozeDays) return err @@ -341,10 +341,10 @@ type UpsertWeightProfileDefaultsParams struct { ContextTimeWeight float64 } -// Recommendation tuning lab queries (#1250). Seeding happens via the +// Recommendation tuning lab queries. Seeding happens via the // recsettings boot reconcile; shipped defaults live in Go only. // Boot reconcile: insert the shipped defaults for a profile if the row -// doesn't exist yet. Never overwrites operator-tuned values. +// doesn't exist yet. Never overwrites admin-tuned values. func (q *Queries) UpsertWeightProfileDefaults(ctx context.Context, arg UpsertWeightProfileDefaultsParams) error { _, err := q.db.Exec(ctx, upsertWeightProfileDefaults, arg.Profile, diff --git a/internal/db/dbq/requests.sql.go b/internal/db/dbq/requests.sql.go new file mode 100644 index 00000000..c2e2b6d1 --- /dev/null +++ b/internal/db/dbq/requests.sql.go @@ -0,0 +1,528 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: requests.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const approveRequest = `-- name: ApproveRequest :one +UPDATE requests + SET status = 'approved', + decided_at = now(), + decided_by = $2, + updated_at = now() + WHERE id = $1 AND status = 'pending' + RETURNING id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at +` + +type ApproveRequestParams struct { + ID pgtype.UUID + DecidedBy pgtype.UUID +} + +func (q *Queries) ApproveRequest(ctx context.Context, arg ApproveRequestParams) (Request, error) { + row := q.db.QueryRow(ctx, approveRequest, arg.ID, arg.DecidedBy) + var i Request + err := row.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ) + return i, err +} + +const cancelRequest = `-- name: CancelRequest :one +UPDATE requests + SET status = 'rejected', + notes = 'cancelled by user', + decided_at = now(), + decided_by = $2, + updated_at = now() + WHERE id = $1 AND user_id = $2 AND status = 'pending' + RETURNING id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at +` + +type CancelRequestParams struct { + ID pgtype.UUID + DecidedBy pgtype.UUID +} + +// $2 serves as both the ownership guard (user_id = $2) and the +// decided_by audit field. The generated parameter struct names it +// DecidedBy — callers must pass the cancelling user's UUID, and +// a wrong UUID will silently no-op (zero rows updated) rather than +// erroring at the SQL layer. +func (q *Queries) CancelRequest(ctx context.Context, arg CancelRequestParams) (Request, error) { + row := q.db.QueryRow(ctx, cancelRequest, arg.ID, arg.DecidedBy) + var i Request + err := row.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ) + return i, err +} + +const completeRequest = `-- name: CompleteRequest :one +UPDATE requests + SET status = 'completed', + matched_track_id = $2, + matched_album_id = $3, + matched_artist_id = $4, + completed_at = now(), + updated_at = now() + WHERE id = $1 AND status = 'approved' + RETURNING id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at +` + +type CompleteRequestParams struct { + ID pgtype.UUID + MatchedTrackID pgtype.UUID + MatchedAlbumID pgtype.UUID + MatchedArtistID pgtype.UUID +} + +// Reconciler transitions an approved request to completed when its +// target track/album/artist has appeared in the library. Callers +// should set ONLY the matched_*_id corresponding to the request's +// kind and pass pgtype.UUID{} (Valid: false) for the others — +// per-kind one-of-three is the contract, NULL for the others +// preserves auditability if the matched row is later evicted. +func (q *Queries) CompleteRequest(ctx context.Context, arg CompleteRequestParams) (Request, error) { + row := q.db.QueryRow(ctx, completeRequest, + arg.ID, + arg.MatchedTrackID, + arg.MatchedAlbumID, + arg.MatchedArtistID, + ) + var i Request + err := row.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ) + return i, err +} + +const createRequest = `-- name: CreateRequest :one +INSERT INTO requests ( + user_id, kind, + artist_mbid, album_mbid, track_mbid, + artist_name, album_title, track_title +) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) +RETURNING id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at +` + +type CreateRequestParams struct { + UserID pgtype.UUID + Kind RequestKind + ArtistMbid string + AlbumMbid *string + TrackMbid *string + ArtistName string + AlbumTitle *string + TrackTitle *string +} + +func (q *Queries) CreateRequest(ctx context.Context, arg CreateRequestParams) (Request, error) { + row := q.db.QueryRow(ctx, createRequest, + arg.UserID, + arg.Kind, + arg.ArtistMbid, + arg.AlbumMbid, + arg.TrackMbid, + arg.ArtistName, + arg.AlbumTitle, + arg.TrackTitle, + ) + var i Request + err := row.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ) + return i, err +} + +const getNonTerminalRequestForMBID = `-- name: GetNonTerminalRequestForMBID :one +SELECT id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at FROM requests + WHERE status IN ('pending', 'approved', 'completed') + AND ((kind = 'artist' AND artist_mbid = $1) + OR (kind = 'album' AND album_mbid = $1) + OR (kind = 'track' AND track_mbid = $1)) + ORDER BY requested_at ASC + LIMIT 1 +` + +// Returns the oldest non-terminal (pending/approved/completed) request +// whose own-kind MBID matches @mbid, or pgx.ErrNoRows when none. Create() +// uses this to dedupe: a re-request for something already in flight +// returns the existing row instead of inserting a duplicate. +func (q *Queries) GetNonTerminalRequestForMBID(ctx context.Context, mbid string) (Request, error) { + row := q.db.QueryRow(ctx, getNonTerminalRequestForMBID, mbid) + var i Request + err := row.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ) + return i, err +} + +const getRequestByID = `-- name: GetRequestByID :one +SELECT id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at FROM requests WHERE id = $1 +` + +func (q *Queries) GetRequestByID(ctx context.Context, id pgtype.UUID) (Request, error) { + row := q.db.QueryRow(ctx, getRequestByID, id) + var i Request + err := row.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ) + return i, err +} + +const hasNonTerminalRequestForMBID = `-- name: HasNonTerminalRequestForMBID :one +SELECT (EXISTS ( + SELECT 1 FROM requests r + WHERE r.status IN ('pending', 'approved', 'completed') + AND ((r.kind = 'artist' AND r.artist_mbid = $1) + OR (r.kind = 'album' AND r.album_mbid = $1) + OR (r.kind = 'track' AND r.track_mbid = $1)) +) OR EXISTS ( + SELECT 1 FROM wants w WHERE w.release_group_mbid = $1 AND w.fulfilled_at IS NULL +) OR EXISTS ( + SELECT 1 FROM follows f WHERE f.artist_mbid = $1 +))::bool AS requested +` + +// Whether something is already on its way: a pending, approved or completed +// request at the given level, an open want for the release group, or a +// follow of the artist. Sets the `requested` flag on catalogue search. +// Terminal-status (rejected, failed) requests do not count. MBIDs are unique +// per entity type, so checking one id against every column cannot +// false-match. +func (q *Queries) HasNonTerminalRequestForMBID(ctx context.Context, mbid string) (bool, error) { + row := q.db.QueryRow(ctx, hasNonTerminalRequestForMBID, mbid) + var requested bool + err := row.Scan(&requested) + return requested, err +} + +const listApprovedRequestsForReconcile = `-- name: ListApprovedRequestsForReconcile :many +SELECT id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at FROM requests +WHERE status = 'approved' +ORDER BY decided_at ASC +LIMIT $1 +` + +func (q *Queries) ListApprovedRequestsForReconcile(ctx context.Context, limit int32) ([]Request, error) { + rows, err := q.db.Query(ctx, listApprovedRequestsForReconcile, limit) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Request + for rows.Next() { + var i Request + if err := rows.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listRequestsByStatus = `-- name: ListRequestsByStatus :many +SELECT id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at FROM requests +WHERE status = $1 +ORDER BY requested_at DESC +LIMIT $2 +` + +type ListRequestsByStatusParams struct { + Status RequestStatus + Limit int32 +} + +func (q *Queries) ListRequestsByStatus(ctx context.Context, arg ListRequestsByStatusParams) ([]Request, error) { + rows, err := q.db.Query(ctx, listRequestsByStatus, arg.Status, arg.Limit) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Request + for rows.Next() { + var i Request + if err := rows.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listRequestsForUser = `-- name: ListRequestsForUser :many +SELECT id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at FROM requests +WHERE user_id = $1 +ORDER BY requested_at DESC +LIMIT $2 +` + +type ListRequestsForUserParams struct { + UserID pgtype.UUID + Limit int32 +} + +func (q *Queries) ListRequestsForUser(ctx context.Context, arg ListRequestsForUserParams) ([]Request, error) { + rows, err := q.db.Query(ctx, listRequestsForUser, arg.UserID, arg.Limit) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Request + for rows.Next() { + var i Request + if err := rows.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const rejectRequest = `-- name: RejectRequest :one +UPDATE requests + SET status = 'rejected', + notes = $2, + decided_at = now(), + decided_by = $3, + updated_at = now() + WHERE id = $1 AND status = 'pending' + RETURNING id, user_id, status, kind, artist_mbid, album_mbid, track_mbid, artist_name, album_title, track_title, decided_at, decided_by, notes, completed_at, matched_track_id, matched_album_id, matched_artist_id, requested_at, updated_at +` + +type RejectRequestParams struct { + ID pgtype.UUID + Notes *string + DecidedBy pgtype.UUID +} + +func (q *Queries) RejectRequest(ctx context.Context, arg RejectRequestParams) (Request, error) { + row := q.db.QueryRow(ctx, rejectRequest, arg.ID, arg.Notes, arg.DecidedBy) + var i Request + err := row.Scan( + &i.ID, + &i.UserID, + &i.Status, + &i.Kind, + &i.ArtistMbid, + &i.AlbumMbid, + &i.TrackMbid, + &i.ArtistName, + &i.AlbumTitle, + &i.TrackTitle, + &i.DecidedAt, + &i.DecidedBy, + &i.Notes, + &i.CompletedAt, + &i.MatchedTrackID, + &i.MatchedAlbumID, + &i.MatchedArtistID, + &i.RequestedAt, + &i.UpdatedAt, + ) + return i, err +} + +const repointRequestsAlbumMbid = `-- name: RepointRequestsAlbumMbid :exec +UPDATE requests + SET album_mbid = $1, + updated_at = now() + WHERE album_mbid = $2 +` + +type RepointRequestsAlbumMbidParams struct { + NewMbid *string + OldMbid *string +} + +// A want moved from a release id to its group takes the requests +// that name it along, so the reconciler finds it. +func (q *Queries) RepointRequestsAlbumMbid(ctx context.Context, arg RepointRequestsAlbumMbidParams) error { + _, err := q.db.Exec(ctx, repointRequestsAlbumMbid, arg.NewMbid, arg.OldMbid) + return err +} diff --git a/internal/db/dbq/scan_runs.sql.go b/internal/db/dbq/scan_runs.sql.go index 94cb53bd..45727164 100644 --- a/internal/db/dbq/scan_runs.sql.go +++ b/internal/db/dbq/scan_runs.sql.go @@ -98,7 +98,7 @@ type InsertScanRunRow struct { StartedAt pgtype.Timestamptz } -// M7 #381: scan-run lifecycle queries. +// Scan-run lifecycle queries. // Creates a new scan_runs row with started_at=now(). Stages are filled in // as each completes via UpdateScanRunStage; finished_at is set by FinishScanRun. func (q *Queries) InsertScanRun(ctx context.Context) (InsertScanRunRow, error) { @@ -119,7 +119,7 @@ type UpdateScanRunArtistArtEnrichParams struct { ArtistArtEnrich []byte } -// M7 cover-sources: persists the artist-art enrich stage tally JSON. +// Persists the artist-art enrich stage tally JSON. func (q *Queries) UpdateScanRunArtistArtEnrich(ctx context.Context, arg UpdateScanRunArtistArtEnrichParams) error { _, err := q.db.Exec(ctx, updateScanRunArtistArtEnrich, arg.ID, arg.ArtistArtEnrich) return err diff --git a/internal/db/dbq/search.sql.go b/internal/db/dbq/search.sql.go new file mode 100644 index 00000000..3e5986e1 --- /dev/null +++ b/internal/db/dbq/search.sql.go @@ -0,0 +1,680 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: search.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const blockRelease = `-- name: BlockRelease :exec +INSERT INTO release_blocks (info_hash, name, reason, detail, want_id, blocked_by, expires_at) +VALUES ($1, $2, $3, $4, + $5, $6, $7) +ON CONFLICT (info_hash) DO UPDATE + SET reason = EXCLUDED.reason, detail = EXCLUDED.detail, blocked_at = now(), + expires_at = EXCLUDED.expires_at, blocked_by = EXCLUDED.blocked_by + WHERE release_blocks.expires_at IS NOT NULL OR EXCLUDED.expires_at IS NULL +` + +type BlockReleaseParams struct { + InfoHash string + Name string + Reason string + Detail string + WantID pgtype.UUID + BlockedBy pgtype.UUID + ExpiresAt pgtype.Timestamptz +} + +// A release set aside. expires_at NULL is for good. A block for good is never +// shortened by a timed one: only a timed block, or another for good, is +// replaced. +func (q *Queries) BlockRelease(ctx context.Context, arg BlockReleaseParams) error { + _, err := q.db.Exec(ctx, blockRelease, + arg.InfoHash, + arg.Name, + arg.Reason, + arg.Detail, + arg.WantID, + arg.BlockedBy, + arg.ExpiresAt, + ) + return err +} + +const deleteWantCandidates = `-- name: DeleteWantCandidates :exec +DELETE FROM want_candidates WHERE want_id = $1 +` + +func (q *Queries) DeleteWantCandidates(ctx context.Context, wantID pgtype.UUID) error { + _, err := q.db.Exec(ctx, deleteWantCandidates, wantID) + return err +} + +const editionTrackSpan = `-- name: EditionTrackSpan :one +SELECT COALESCE(MIN(track_count) FILTER (WHERE status = 'Official'), MIN(track_count), 0)::int AS min_tracks, + COALESCE(MAX(track_count), 0)::int AS max_tracks + FROM mb_releases + WHERE release_group_mbid = $1 AND track_count > 0 +` + +type EditionTrackSpanRow struct { + MinTracks int32 + MaxTracks int32 +} + +// The fewest tracks an official edition of the group has (any edition when +// none is official), and the most any has. Zero when none is catalogued. +func (q *Queries) EditionTrackSpan(ctx context.Context, releaseGroupMbid string) (EditionTrackSpanRow, error) { + row := q.db.QueryRow(ctx, editionTrackSpan, releaseGroupMbid) + var i EditionTrackSpanRow + err := row.Scan(&i.MinTracks, &i.MaxTracks) + return i, err +} + +const endGrab = `-- name: EndGrab :exec +UPDATE grabs + SET ended_at = now(), end_reason = $1, end_detail = $2 + WHERE id = $3 AND ended_at IS NULL +` + +type EndGrabParams struct { + EndReason string + EndDetail string + ID pgtype.UUID +} + +func (q *Queries) EndGrab(ctx context.Context, arg EndGrabParams) error { + _, err := q.db.Exec(ctx, endGrab, arg.EndReason, arg.EndDetail, arg.ID) + return err +} + +const getAcquisitionSettings = `-- name: GetAcquisitionSettings :one + +SELECT id, auto_grab, search_interval_hours, allowed_qualities, last_pass_at, last_pass, updated_at, import_root, seed_ratio, seed_days, stall_hours, library_listing_started_at, library_listed_at, library_unlisted, upgrade_until FROM acquisition_settings WHERE id = true +` + +// Searching for wants and grabbing releases. +func (q *Queries) GetAcquisitionSettings(ctx context.Context) (AcquisitionSetting, error) { + row := q.db.QueryRow(ctx, getAcquisitionSettings) + var i AcquisitionSetting + err := row.Scan( + &i.ID, + &i.AutoGrab, + &i.SearchIntervalHours, + &i.AllowedQualities, + &i.LastPassAt, + &i.LastPass, + &i.UpdatedAt, + &i.ImportRoot, + &i.SeedRatio, + &i.SeedDays, + &i.StallHours, + &i.LibraryListingStartedAt, + &i.LibraryListedAt, + &i.LibraryUnlisted, + &i.UpgradeUntil, + ) + return i, err +} + +const getLiveGrab = `-- name: GetLiveGrab :one +SELECT id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at FROM grabs WHERE want_id = $1 AND ended_at IS NULL +` + +func (q *Queries) GetLiveGrab(ctx context.Context, wantID pgtype.UUID) (Grab, error) { + row := q.db.QueryRow(ctx, getLiveGrab, wantID) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const getWantCandidate = `-- name: GetWantCandidate :one +SELECT want_id, key, name, info_hash, indexer, source, size_bytes, seeders, files_count, format, quality, year, verdict, reason, reasons, folded_from, rank, searched_at FROM want_candidates WHERE want_id = $1 AND key = $2 +` + +type GetWantCandidateParams struct { + WantID pgtype.UUID + Key string +} + +func (q *Queries) GetWantCandidate(ctx context.Context, arg GetWantCandidateParams) (WantCandidate, error) { + row := q.db.QueryRow(ctx, getWantCandidate, arg.WantID, arg.Key) + var i WantCandidate + err := row.Scan( + &i.WantID, + &i.Key, + &i.Name, + &i.InfoHash, + &i.Indexer, + &i.Source, + &i.SizeBytes, + &i.Seeders, + &i.FilesCount, + &i.Format, + &i.Quality, + &i.Year, + &i.Verdict, + &i.Reason, + &i.Reasons, + &i.FoldedFrom, + &i.Rank, + &i.SearchedAt, + ) + return i, err +} + +const getWantForSearch = `-- name: GetWantForSearch :one +SELECT w.id, w.release_group_mbid, w.artist_mbid, w.artist_name, w.title, w.reason, w.fulfilled_at, + w.upgrade, w.held_tier, w.held_quality, w.album_id, + COALESCE(NULLIF(w.primary_type, ''), g.primary_type, '')::text AS primary_type, + COALESCE(g.secondary_types, '{}')::text[] AS secondary_types, + COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, '')::text AS first_release_date + FROM wants w + LEFT JOIN mb_release_groups g ON g.mbid = w.release_group_mbid + WHERE w.id = $1 +` + +type GetWantForSearchRow struct { + ID pgtype.UUID + ReleaseGroupMbid string + ArtistMbid string + ArtistName string + Title string + Reason string + FulfilledAt pgtype.Timestamptz + Upgrade string + HeldTier string + HeldQuality string + AlbumID pgtype.UUID + PrimaryType string + SecondaryTypes []string + FirstReleaseDate string +} + +func (q *Queries) GetWantForSearch(ctx context.Context, id pgtype.UUID) (GetWantForSearchRow, error) { + row := q.db.QueryRow(ctx, getWantForSearch, id) + var i GetWantForSearchRow + err := row.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.Title, + &i.Reason, + &i.FulfilledAt, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.AlbumID, + &i.PrimaryType, + &i.SecondaryTypes, + &i.FirstReleaseDate, + ) + return i, err +} + +const insertGrab = `-- name: InsertGrab :one +INSERT INTO grabs (want_id, info_hash, name, key, indexer, source, client, save_path, quality, + origin, forced, grabbed_by) +VALUES ($1, $2, $3, $4, $5, + $6, $7, $8, $9, + $10, $11, $12) +RETURNING id, want_id, info_hash, name, key, indexer, source, client, save_path, quality, origin, forced, grabbed_by, grabbed_at, ended_at, end_reason, end_detail, progress, import_state, import_reason, import_detail, import_forced, import_attempts, import_updated_at, imported_at, imported_files, destination, seeding, ratio, seed_released_at +` + +type InsertGrabParams struct { + WantID pgtype.UUID + InfoHash string + Name string + Key string + Indexer string + Source string + Client string + SavePath string + Quality string + Origin string + Forced bool + GrabbedBy pgtype.UUID +} + +func (q *Queries) InsertGrab(ctx context.Context, arg InsertGrabParams) (Grab, error) { + row := q.db.QueryRow(ctx, insertGrab, + arg.WantID, + arg.InfoHash, + arg.Name, + arg.Key, + arg.Indexer, + arg.Source, + arg.Client, + arg.SavePath, + arg.Quality, + arg.Origin, + arg.Forced, + arg.GrabbedBy, + ) + var i Grab + err := row.Scan( + &i.ID, + &i.WantID, + &i.InfoHash, + &i.Name, + &i.Key, + &i.Indexer, + &i.Source, + &i.Client, + &i.SavePath, + &i.Quality, + &i.Origin, + &i.Forced, + &i.GrabbedBy, + &i.GrabbedAt, + &i.EndedAt, + &i.EndReason, + &i.EndDetail, + &i.Progress, + &i.ImportState, + &i.ImportReason, + &i.ImportDetail, + &i.ImportForced, + &i.ImportAttempts, + &i.ImportUpdatedAt, + &i.ImportedAt, + &i.ImportedFiles, + &i.Destination, + &i.Seeding, + &i.Ratio, + &i.SeedReleasedAt, + ) + return i, err +} + +const insertWantCandidate = `-- name: InsertWantCandidate :exec +INSERT INTO want_candidates (want_id, key, name, info_hash, indexer, source, size_bytes, seeders, + files_count, format, quality, year, verdict, reason, reasons, folded_from, rank) +VALUES ($1, $2, $3, $4, $5, + $6, $7, $8, $9, $10, + $11, $12, $13, $14, $15, + $16, $17) +ON CONFLICT (want_id, key) DO NOTHING +` + +type InsertWantCandidateParams struct { + WantID pgtype.UUID + Key string + Name string + InfoHash string + Indexer string + Source string + SizeBytes int64 + Seeders int32 + FilesCount int32 + Format string + Quality string + Year int32 + Verdict string + Reason string + Reasons []byte + FoldedFrom []string + Rank int32 +} + +func (q *Queries) InsertWantCandidate(ctx context.Context, arg InsertWantCandidateParams) error { + _, err := q.db.Exec(ctx, insertWantCandidate, + arg.WantID, + arg.Key, + arg.Name, + arg.InfoHash, + arg.Indexer, + arg.Source, + arg.SizeBytes, + arg.Seeders, + arg.FilesCount, + arg.Format, + arg.Quality, + arg.Year, + arg.Verdict, + arg.Reason, + arg.Reasons, + arg.FoldedFrom, + arg.Rank, + ) + return err +} + +const listReleaseBlocks = `-- name: ListReleaseBlocks :many +SELECT info_hash, name, reason, detail, want_id, blocked_by, blocked_at, expires_at FROM release_blocks + WHERE expires_at IS NULL OR expires_at > now() + ORDER BY blocked_at DESC +` + +// Every release set aside and still held back: a stalled one lifts by itself +// at expires_at. +func (q *Queries) ListReleaseBlocks(ctx context.Context) ([]ReleaseBlock, error) { + rows, err := q.db.Query(ctx, listReleaseBlocks) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ReleaseBlock + for rows.Next() { + var i ReleaseBlock + if err := rows.Scan( + &i.InfoHash, + &i.Name, + &i.Reason, + &i.Detail, + &i.WantID, + &i.BlockedBy, + &i.BlockedAt, + &i.ExpiresAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listWantCandidates = `-- name: ListWantCandidates :many +SELECT want_id, key, name, info_hash, indexer, source, size_bytes, seeders, files_count, format, quality, year, verdict, reason, reasons, folded_from, rank, searched_at FROM want_candidates WHERE want_id = $1 ORDER BY rank, key +` + +func (q *Queries) ListWantCandidates(ctx context.Context, wantID pgtype.UUID) ([]WantCandidate, error) { + rows, err := q.db.Query(ctx, listWantCandidates, wantID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []WantCandidate + for rows.Next() { + var i WantCandidate + if err := rows.Scan( + &i.WantID, + &i.Key, + &i.Name, + &i.InfoHash, + &i.Indexer, + &i.Source, + &i.SizeBytes, + &i.Seeders, + &i.FilesCount, + &i.Format, + &i.Quality, + &i.Year, + &i.Verdict, + &i.Reason, + &i.Reasons, + &i.FoldedFrom, + &i.Rank, + &i.SearchedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listWantsDueForSearch = `-- name: ListWantsDueForSearch :many +SELECT w.id, w.release_group_mbid, w.artist_name, w.title, w.reason, w.upgrade, w.held_tier, w.held_quality, + COALESCE(NULLIF(w.primary_type, ''), g.primary_type, '')::text AS primary_type, + COALESCE(g.secondary_types, '{}')::text[] AS secondary_types, + COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, '')::text AS first_release_date + FROM wants w + LEFT JOIN mb_release_groups g ON g.mbid = w.release_group_mbid + WHERE (w.fulfilled_at IS NULL OR w.upgrade <> '') + AND NOT EXISTS (SELECT 1 FROM grabs gr WHERE gr.want_id = w.id AND gr.ended_at IS NULL) + AND (w.searched_at IS NULL + OR w.searched_at < now() - make_interval(hours => CASE WHEN w.fulfilled_at IS NULL + THEN $1::int + ELSE GREATEST($1::int, 168) END)) + ORDER BY w.fulfilled_at IS NOT NULL, + w.searched_at IS NOT NULL, + CASE w.reason WHEN 'request' THEN 0 WHEN 'missing' THEN 1 ELSE 2 END, + w.searched_at NULLS FIRST, w.wanted_at, w.id + LIMIT $2 +` + +type ListWantsDueForSearchParams struct { + Hours int32 + MaxRows int32 +} + +type ListWantsDueForSearchRow struct { + ID pgtype.UUID + ReleaseGroupMbid string + ArtistName string + Title string + Reason string + Upgrade string + HeldTier string + HeldQuality string + PrimaryType string + SecondaryTypes []string + FirstReleaseDate string +} + +// Wants with no download running that were never searched or not since +// @hours ago: open wants first (never-searched, then requests, then missing +// files, then followed artists' releases), then held albums wanted for an +// upgrade, which are searched at most weekly. +func (q *Queries) ListWantsDueForSearch(ctx context.Context, arg ListWantsDueForSearchParams) ([]ListWantsDueForSearchRow, error) { + rows, err := q.db.Query(ctx, listWantsDueForSearch, arg.Hours, arg.MaxRows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListWantsDueForSearchRow + for rows.Next() { + var i ListWantsDueForSearchRow + if err := rows.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistName, + &i.Title, + &i.Reason, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.PrimaryType, + &i.SecondaryTypes, + &i.FirstReleaseDate, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const markCandidatesBlocked = `-- name: MarkCandidatesBlocked :exec +UPDATE want_candidates + SET verdict = 'blocked', reason = $1, + reasons = jsonb_build_array(jsonb_build_object('kind', 'hard', 'code', 'blocked', 'text', $1::text)) + WHERE info_hash = $2 +` + +type MarkCandidatesBlockedParams struct { + Reason string + InfoHash string +} + +func (q *Queries) MarkCandidatesBlocked(ctx context.Context, arg MarkCandidatesBlockedParams) error { + _, err := q.db.Exec(ctx, markCandidatesBlocked, arg.Reason, arg.InfoHash) + return err +} + +const markCandidatesUnblocked = `-- name: MarkCandidatesUnblocked :exec +UPDATE want_candidates + SET verdict = 'review', reason = $1, + reasons = jsonb_build_array(jsonb_build_object('kind', 'soft', 'code', 'unblocked', 'text', $1::text)) + WHERE info_hash = $2 AND verdict = 'blocked' +` + +type MarkCandidatesUnblockedParams struct { + Reason string + InfoHash string +} + +func (q *Queries) MarkCandidatesUnblocked(ctx context.Context, arg MarkCandidatesUnblockedParams) error { + _, err := q.db.Exec(ctx, markCandidatesUnblocked, arg.Reason, arg.InfoHash) + return err +} + +const recordSearchPass = `-- name: RecordSearchPass :exec +UPDATE acquisition_settings SET last_pass_at = now(), last_pass = $1 WHERE id = true +` + +func (q *Queries) RecordSearchPass(ctx context.Context, report []byte) error { + _, err := q.db.Exec(ctx, recordSearchPass, report) + return err +} + +const recordWantSearch = `-- name: RecordWantSearch :exec +UPDATE wants + SET searched_at = now(), + search_outcome = $1, + search_considered = $2, + best_release = $3, + best_verdict = $4, + search_error = $5, + updated_at = now() + WHERE id = $6 +` + +type RecordWantSearchParams struct { + Outcome string + Considered int32 + BestRelease string + BestVerdict string + SearchError string + ID pgtype.UUID +} + +func (q *Queries) RecordWantSearch(ctx context.Context, arg RecordWantSearchParams) error { + _, err := q.db.Exec(ctx, recordWantSearch, + arg.Outcome, + arg.Considered, + arg.BestRelease, + arg.BestVerdict, + arg.SearchError, + arg.ID, + ) + return err +} + +const unblockRelease = `-- name: UnblockRelease :execrows +DELETE FROM release_blocks WHERE info_hash = $1 +` + +func (q *Queries) UnblockRelease(ctx context.Context, infoHash string) (int64, error) { + result, err := q.db.Exec(ctx, unblockRelease, infoHash) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const updateAcquisitionSettings = `-- name: UpdateAcquisitionSettings :one +UPDATE acquisition_settings + SET auto_grab = $1, + search_interval_hours = $2, + allowed_qualities = $3, + import_root = $4, + seed_ratio = $5, + seed_days = $6, + stall_hours = $7, + upgrade_until = $8, + updated_at = now() + WHERE id = true +RETURNING id, auto_grab, search_interval_hours, allowed_qualities, last_pass_at, last_pass, updated_at, import_root, seed_ratio, seed_days, stall_hours, library_listing_started_at, library_listed_at, library_unlisted, upgrade_until +` + +type UpdateAcquisitionSettingsParams struct { + AutoGrab bool + SearchIntervalHours int32 + AllowedQualities []string + ImportRoot string + SeedRatio float32 + SeedDays int32 + StallHours int32 + UpgradeUntil string +} + +func (q *Queries) UpdateAcquisitionSettings(ctx context.Context, arg UpdateAcquisitionSettingsParams) (AcquisitionSetting, error) { + row := q.db.QueryRow(ctx, updateAcquisitionSettings, + arg.AutoGrab, + arg.SearchIntervalHours, + arg.AllowedQualities, + arg.ImportRoot, + arg.SeedRatio, + arg.SeedDays, + arg.StallHours, + arg.UpgradeUntil, + ) + var i AcquisitionSetting + err := row.Scan( + &i.ID, + &i.AutoGrab, + &i.SearchIntervalHours, + &i.AllowedQualities, + &i.LastPassAt, + &i.LastPass, + &i.UpdatedAt, + &i.ImportRoot, + &i.SeedRatio, + &i.SeedDays, + &i.StallHours, + &i.LibraryListingStartedAt, + &i.LibraryListedAt, + &i.LibraryUnlisted, + &i.UpgradeUntil, + ) + return i, err +} diff --git a/internal/db/dbq/sessions.sql.go b/internal/db/dbq/sessions.sql.go index 307af3e7..0fcdfd83 100644 --- a/internal/db/dbq/sessions.sql.go +++ b/internal/db/dbq/sessions.sql.go @@ -57,7 +57,8 @@ type DeleteSessionForUserParams struct { UserID pgtype.UUID } -// Scoped by user_id, not just id (rule #47). Keyed on the id alone, any +// Scoped by user_id, not just id: household data is per-user. Keyed on the +// id alone, any // household member could revoke another member's session by guessing a uuid. // execrows lets the handler answer 404 rather than a false 204 when the row // isn't theirs. diff --git a/internal/db/dbq/similarity.sql.go b/internal/db/dbq/similarity.sql.go index 480c64cd..d1f69b8a 100644 --- a/internal/db/dbq/similarity.sql.go +++ b/internal/db/dbq/similarity.sql.go @@ -134,8 +134,8 @@ type ListPlayedTracksNeedingSimilarityRow struct { // ListenBrainz has never answered for, or not in 30 days. Never-fetched first, // then the oldest answer. Freshness is read from track_similarity_fetches, not // from the edges, so a seed whose answer matched nothing in the library still -// counts as fetched (#5296); reading the edges is what let 25 such seeds hold -// the head of the queue forever. +// counts as fetched; reading the edges would let such seeds hold the head of +// the queue forever. func (q *Queries) ListPlayedTracksNeedingSimilarity(ctx context.Context, limit int32) ([]ListPlayedTracksNeedingSimilarityRow, error) { rows, err := q.db.Query(ctx, listPlayedTracksNeedingSimilarity, limit) if err != nil { @@ -260,10 +260,19 @@ func (q *Queries) RecordTrackSimilarityFetch(ctx context.Context, arg RecordTrac } const resolveListenBrainzTrackEdges = `-- name: ResolveListenBrainzTrackEdges :exec -WITH want AS ( +WITH best AS ( + SELECT seed_track_id, max(score) AS score + FROM listenbrainz_similar_recordings + WHERE $1::uuid[] IS NULL + OR seed_track_id = ANY($1::uuid[]) + GROUP BY seed_track_id +), +want AS ( SELECT DISTINCT ON (c.seed_track_id, c.recording_mbid) - c.seed_track_id AS track_a_id, t.id AS track_b_id, c.score + c.seed_track_id AS track_a_id, t.id AS track_b_id, + CASE WHEN b.score > 0 THEN c.score / b.score ELSE 0 END AS score FROM listenbrainz_similar_recordings c + JOIN best b ON b.seed_track_id = c.seed_track_id JOIN tracks t ON t.mbid = c.recording_mbid AND t.missing_since IS NULL AND t.id <> c.seed_track_id @@ -299,6 +308,13 @@ WHERE ts.source = 'listenbrainz' // pool would offer the same song twice. No cap per seed: LB's answer is already // at most 50. // +// The score is RELATIVE: each edge's LB score over the seed's best score in its +// whole cached answer, so 1.0 is the seed's closest recording and the rest fall +// below it. LB's raw scores are counts, into the thousands, and the scorer +// reads similarity as 0 to 1, so a raw edge would outweigh likes, skips and +// recency together. The cache keeps the raw +// answer; only the edge is scaled. +// // Only edges of seeds the cache owns are removed, those with a fetch row. // Edges written before the cache existed, or copied onto a survivor by a fold // (merge.sql), stay until their seed is fetched. diff --git a/internal/db/dbq/songs.sql.go b/internal/db/dbq/songs.sql.go index 0820427d..a441f9c6 100644 --- a/internal/db/dbq/songs.sql.go +++ b/internal/db/dbq/songs.sql.go @@ -33,7 +33,7 @@ type LinkTracksAsSongRow struct { Moved int64 } -// Song links (M498 #5438): copies of one song on different releases share a +// Song links: copies of one song on different releases share a // song_key (migration 0076). // Links the tracks into one song, joining any songs they already belong to. // The song keeps the lowest key among them, so linking the same set again @@ -57,7 +57,7 @@ type ListTrackSongKeysRow struct { } // What the mix writer needs to place a track: its song, and whether it is a -// video rip held back from the mixes (#5439). +// video rip held back from the mixes. func (q *Queries) ListTrackSongKeys(ctx context.Context, ids []pgtype.UUID) ([]ListTrackSongKeysRow, error) { rows, err := q.db.Query(ctx, listTrackSongKeys, ids) if err != nil { @@ -85,7 +85,8 @@ UPDATE tracks s WHERE s.id = $1::uuid AND l.id = $2::uuid AND s.song_id IS NULL - AND l.song_key <> l.id + AND EXISTS (SELECT 1 FROM tracks o + WHERE o.song_key = l.song_key AND o.id <> l.id AND o.id <> s.id) ` type MergeInheritSongLinkParams struct { @@ -95,7 +96,8 @@ type MergeInheritSongLinkParams struct { // A merge keeps the removed copy's song link: when the copy being removed was // linked to copies on other releases and the survivor was not, the survivor -// joins that song. +// joins that song. The copy whose id names the song has no song_id of its +// own, so "linked" is another copy sharing its key, not a song_id. func (q *Queries) MergeInheritSongLink(ctx context.Context, arg MergeInheritSongLinkParams) error { _, err := q.db.Exec(ctx, mergeInheritSongLink, arg.SurvivorID, arg.LoserID) return err diff --git a/internal/db/dbq/suggestion_snoozes.sql.go b/internal/db/dbq/suggestion_snoozes.sql.go index f1642a34..5da3eb77 100644 --- a/internal/db/dbq/suggestion_snoozes.sql.go +++ b/internal/db/dbq/suggestion_snoozes.sql.go @@ -85,7 +85,7 @@ type SnoozeSuggestionParams struct { Column4 float64 } -// Time-boxed "not right now" on a Discover artist suggestion (#2374). +// Time-boxed "not right now" on a Discover artist suggestion. // // See 0049_suggestion_snoozes.up.sql for why this is a snooze and not a // dismissal: it records no verdict on the music, expires on its own, and diff --git a/internal/db/dbq/system_mixes.sql.go b/internal/db/dbq/system_mixes.sql.go index d8046f1b..32387753 100644 --- a/internal/db/dbq/system_mixes.sql.go +++ b/internal/db/dbq/system_mixes.sql.go @@ -39,11 +39,11 @@ SELECT t.id, t.album_id, t.artist_id JOIN affinity_artists aa ON aa.artist_id = t.artist_id LEFT JOIN play_counts pc ON pc.track_id = t.id LEFT JOIN skip_counts sc ON sc.track_id = t.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND COALESCE(pc.c, 0) <= 2 AND COALESCE(sc.c, 0) < 2 AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY md5(t.id::text || $2::text) @@ -62,18 +62,18 @@ type ListDeepCutsTracksRow struct { } // Candidate queries for the discovery system-playlist mixes -// (#419-423). Each returns (id, album_id, artist_id) rows; the Go +// . Each returns (id, album_id, artist_id) rows; the Go // producer caps + truncates and converts to rankedCandidate. Daily- // deterministic md5(id||dateStr) ordering (where used) keeps a mix // stable within a day while rotating across days; shuffle-on-play // gives per-play variety on top. -// #419 Deep Cuts: low-play tracks (<=2 plays) from artists the user +// Deep Cuts: low-play tracks (<=2 plays) from artists the user // has liked OR played heavily (>=5 non-skip plays across the artist). -// Tracks the user has skipped twice or more don't qualify (#1257): -// eligibility used to count only unskipped plays, so the most actively -// rejected tracks read as "barely heard" and kept being re-offered. -// Threshold 2 so a single accidental skip doesn't banish a track; -// passive-signal only (no dislike UI, rule #101). +// Tracks the user has skipped twice or more don't qualify: counting only +// unskipped plays would make the most actively rejected tracks read as +// "barely heard" and keep re-offering them. Threshold 2 so a single +// accidental skip doesn't banish a track; passive signal only (there is no +// dislike button). // $1 user_id, $2 date string. func (q *Queries) ListDeepCutsTracks(ctx context.Context, arg ListDeepCutsTracksParams) ([]ListDeepCutsTracksRow, error) { rows, err := q.db.Query(ctx, listDeepCutsTracks, arg.UserID, arg.Column2) @@ -125,11 +125,11 @@ SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier FROM tracks t JOIN albums al ON al.id = t.album_id JOIN albums_tiered alt ON alt.album_id = al.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND alt.tier IS NOT NULL AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY @@ -153,11 +153,10 @@ type ListFirstListensTracksRow struct { Tier int32 } -// #423 First Listens: songs the user has never heard or even -// attempted. "Attempted" is track-level with a >=30s listen threshold -// (#1268, operator decision 2026-07-03) — the old version disqualified -// a whole album on ANY play_event, so a 2-second accidental brush -// banished it permanently. +// First Listens: songs the user has never heard or even +// attempted. "Attempted" is track-level with a >=30s listen threshold: +// disqualifying a whole album on ANY play_event would let a 2-second +// accidental brush banish it permanently. // // tier 1 the exact desire: albums with ZERO attempted tracks // tier 2 step back: barely-attempted albums (<=25% of tracks @@ -227,9 +226,9 @@ albums_tiered AS ( SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier FROM tracks t JOIN albums_tiered alt ON alt.album_id = t.album_id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY alt.tier, alt.created_at DESC, t.album_id, @@ -244,14 +243,13 @@ type ListNewForYouTracksRow struct { Tier int32 } -// #421 New for you, tiered per project rule #131 (#1267). The old -// single hard rule (added <30d AND direct affinity) had two failure -// modes with one root: no notion of consumption — the album you -// devoured in week one crowded the mix for three more weeks, and -// after a quiet month nothing qualified at all. +// New for you, tiered so the mix degrades rather than vanishes. A +// single hard rule (added <30d AND direct affinity) has no notion of +// consumption: the album you devoured in week one would crowd the mix +// for three more weeks, and after a quiet month nothing would qualify. // // "Consumed" = any track attempted for >=30s (the First Listens -// threshold, #1268): once you've meaningfully engaged, the album is +// threshold): once you've meaningfully engaged, the album is // no longer news. Applies to every tier — played albums leave the // mix the next build. // @@ -306,9 +304,9 @@ WITH windowed AS ( SELECT t.id, t.album_id, t.artist_id FROM tracks t JOIN windowed w ON w.track_id = t.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY w.c DESC, md5(t.id::text || $2::text) @@ -326,18 +324,18 @@ type ListOnThisDayTracksRow struct { ArtistID pgtype.UUID } -// #422 On This Day: tracks the user played around this calendar date +// On This Day: tracks the user played around this calendar date // (±10 day-of-year) in the past, excluding the very recent (>30 days // ago) so it's nostalgic, not just "last week". Weighted by how much -// they were played in those windows. Floor relaxed from 60→30 days -// and window ±7→±10 so it surfaces on a months-old library instead -// of needing a full year of history; still skips cleanly (no rows → -// no playlist) when there's no qualifying history yet. -// Day-of-year distance is circular (#1256): plain ABS made Dec 28 vs -// Jan 3 read as 359 days apart, silently gutting the window for ~3 -// weeks around every New Year — precisely when holiday nostalgia is -// the point. LEAST(d, 365-d) wraps the boundary; leap-year drift of -// ±1 is absorbed by the ±10 window. The build date ($2, already here +// they were played in those windows. The 30-day floor and ±10 window +// let it surface on a months-old library rather than needing a full +// year of history; it still skips cleanly (no rows → no playlist) +// when there's no qualifying history yet. +// Day-of-year distance is circular: plain ABS would read Dec 28 vs +// Jan 3 as 359 days apart, gutting the window for ~3 weeks around +// every New Year — precisely when holiday nostalgia is the point. +// LEAST(d, 365-d) wraps the boundary; leap-year drift of ±1 is absorbed +// by the ±10 window. The build date ($2, already here // for the md5 rotation) anchors "today" instead of now() so the // window is testable and consistent with the mix's daily determinism. // $1 user_id, $2 date string. @@ -371,18 +369,18 @@ WITH stats AS ( SELECT t.id, t.album_id, t.artist_id FROM tracks t JOIN stats s ON s.track_id = t.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND s.c >= 3 AND s.last_at <= now() - interval '30 days' AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) - -- Declined-resurfacing cooldown (#1258): a skip on a + -- Declined-resurfacing cooldown: a skip on a -- rediscover-sourced play is the user explicitly saying "I've moved -- on" to the exact invitation this mix extends — such tracks sit -- out ~90 days instead of re-qualifying the next day. Passive - -- signal only (rule #101). + -- signal only. AND NOT EXISTS ( SELECT 1 FROM play_events rp WHERE rp.user_id = $1 @@ -403,11 +401,10 @@ type ListRediscoverTracksRow struct { ArtistID pgtype.UUID } -// #420 Rediscover: tracks the user played a lot but has drifted away -// from. One blended pool (issue #1246: the old two-tier UNION was -// all-or-nothing — a single ">=6 months cold" row suppressed the whole -// ">=30 days" tier, which is how a one-song playlist shipped; the 6mo -// tier was also a strict subset of the 30d tier). Eligibility is >=3 +// Rediscover: tracks the user played a lot but has drifted away +// from. One blended pool, not a two-tier UNION: the ">=6 months cold" tier is +// a strict subset of the ">=30 days" one, and a UNION that let one row of it +// suppress the wider tier would ship a one-song playlist. Eligibility is >=3 // non-skip plays and >=30 days cold; the bar sits at 3 rather than 5 // so a weeks-old library still fields a pool — on young histories the // >=5-play tracks are precisely the ones still in rotation. Ordering diff --git a/internal/db/dbq/system_playlist_rotation.sql.go b/internal/db/dbq/system_playlist_rotation.sql.go index 6aeff689..f78c26e6 100644 --- a/internal/db/dbq/system_playlist_rotation.sql.go +++ b/internal/db/dbq/system_playlist_rotation.sql.go @@ -34,7 +34,7 @@ type AppendRotationPlayedParams struct { TrackID pgtype.UUID } -// Rotation state for system-playlist sample-history dedup (#415). +// Rotation state for system-playlist sample-history dedup. // One row per (user, playlist_kind). played_track_ids accumulates // tracks heard from that playlist in the current rotation; shuffle // prefers ids NOT in this set, then resets when the snapshot is diff --git a/internal/db/dbq/system_playlists.sql.go b/internal/db/dbq/system_playlists.sql.go index d8fa1ae4..0a892e48 100644 --- a/internal/db/dbq/system_playlists.sql.go +++ b/internal/db/dbq/system_playlists.sql.go @@ -198,14 +198,14 @@ SELECT u.id FROM users u ) ` -// Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a +// Track picks here join `tracks ... AND t.missing_since IS NULL`: a // seed or For-You candidate has to be something that can actually play. Note // this only affects newly GENERATED playlists — already-stored system // playlists keep their rows until the next daily rebuild, which is why the // shared ListPlaylistTracks read path is deliberately left unfiltered (it // also serves user-curated playlists, where hiding a track the user added // themselves would be wrong). -// M7 #352 slice 2: system-generated playlist queries. +// System-generated playlist queries. // Active = had a play in the last 7 days. The cron iterates this list. func (q *Queries) ListActiveUsersForSystemPlaylists(ctx context.Context) ([]pgtype.UUID, error) { rows, err := q.db.Query(ctx, listActiveUsersForSystemPlaylists) @@ -368,10 +368,10 @@ type PickSeedArtistsRow struct { } // Top-12 most-engaged distinct artist candidates, tiered so the -// "Songs like X" mixes never silently vanish (#1255): the old hard -// 7-day window emptied the seed pool after a quiet week, and the -// daily atomic-replace build then deleted every existing mix until -// the user played something again. Same fallback shape as +// "Songs like X" mixes never silently vanish: a hard 7-day window +// would empty the seed pool after a quiet week, and the daily +// atomic-replace build would then delete every existing mix until the +// user played something again. Same fallback shape as // PickTopPlayedTracksForUser (For You's seeds): // // tier 0 engagement in the last 7 days @@ -380,12 +380,12 @@ type PickSeedArtistsRow struct { // tier 3 (only if 0-2 empty) liked artists with no play history // // All returned rows share one tier; produceSeedMixes maps it onto the -// rule-#131 pick-kind ladder and stamps the built tracks, so metrics +// tiered pick-kind ladder and stamps the built tracks, so metrics // can compare mixes seeded from fresh vs stale engagement. // Score = unskipped-play count + 5 if user has liked the artist. The // Go-side picker (pickSeedArtistsForDay) shuffles the 12 // daily-deterministically and takes songsLikeSeedCount (6) so the mix -// set both rotates day-to-day and fills the dedicated Home row (#1491). +// set both rotates day-to-day and fills the dedicated Home row. func (q *Queries) PickSeedArtists(ctx context.Context, userID pgtype.UUID) ([]PickSeedArtistsRow, error) { rows, err := q.db.Query(ctx, pickSeedArtists, userID) if err != nil { @@ -452,7 +452,7 @@ SELECT COALESCE( FROM tracks t JOIN albums a ON a.id = t.album_id WHERE t.artist_id = $2 - AND t.missing_since IS NULL -- #2701: the play branch filters; this one must too + AND t.missing_since IS NULL -- the play branch filters; this one must too ORDER BY a.release_date DESC NULLS LAST, t.disc_number NULLS LAST, t.track_number NULLS LAST, @@ -530,7 +530,7 @@ SELECT id // library with sparse history. // A likes-only read still joins tracks: a like outlives its file, and a // seed whose file is gone points For You at something the user can't hear -// (#2701). The other two tiers get the same filter from their play_events +// . The other two tiers get the same filter from their play_events // join. func (q *Queries) PickTopPlayedTracksForUser(ctx context.Context, userID pgtype.UUID) ([]pgtype.UUID, error) { rows, err := q.db.Query(ctx, pickTopPlayedTracksForUser, userID) diff --git a/internal/db/dbq/tag_settings.sql.go b/internal/db/dbq/tag_settings.sql.go index 61ee4524..0e2e5c6e 100644 --- a/internal/db/dbq/tag_settings.sql.go +++ b/internal/db/dbq/tag_settings.sql.go @@ -74,7 +74,7 @@ SELECT provider_id, enabled, api_key, display_order, created_at, updated_at ORDER BY display_order, provider_id ` -// Tag-enrichment provider settings queries (milestone #160, #1490 Step 2). +// Tag-enrichment provider settings queries. // The tags.SettingsService consumes these — parallel to the cover-art // SettingsService (coverart_settings.sql). Keeping the two independent // means a new tag source is added without touching the art settings. @@ -150,7 +150,7 @@ type UpsertTagProviderSettingsParams struct { // INSERT a default row for a newly-registered provider, or refresh an // existing row's display_order on boot. Does NOT overwrite enabled / -// api_key on conflict — the operator's settings persist across restarts. +// api_key on conflict — an admin's settings persist across restarts. func (q *Queries) UpsertTagProviderSettings(ctx context.Context, arg UpsertTagProviderSettingsParams) error { _, err := q.db.Exec(ctx, upsertTagProviderSettings, arg.ProviderID, arg.Enabled, arg.DisplayOrder) return err diff --git a/internal/db/dbq/taste_profile.sql.go b/internal/db/dbq/taste_profile.sql.go index 473af444..c7aa5c7f 100644 --- a/internal/db/dbq/taste_profile.sql.go +++ b/internal/db/dbq/taste_profile.sql.go @@ -153,7 +153,7 @@ type ListLikedTrackTasteInputsForUserRow struct { // (track_id, artist_id, genre, release_date) for each track the user has // explicitly liked. Feeds the track-like bonus into the liked track's artist, -// tags, and era (#1530); track_id keys the enriched track_tags lookup (#1490). +// tags, and era; track_id keys the enriched track_tags lookup. func (q *Queries) ListLikedTrackTasteInputsForUser(ctx context.Context, userID pgtype.UUID) ([]ListLikedTrackTasteInputsForUserRow, error) { rows, err := q.db.Query(ctx, listLikedTrackTasteInputsForUser, userID) if err != nil { @@ -197,7 +197,7 @@ JOIN albums a ON a.id = t.album_id WHERE pe.user_id = $1 AND pe.started_at > now() - ($2::float8 * INTERVAL '1 day') AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ` @@ -216,14 +216,14 @@ type ListPlayEngagementInputsForUserRow struct { AgeDays float64 } -// Taste profile (#796 phase 1). The daily build computes signed artist/tag +// Taste profile. The daily build computes signed artist/tag // weights in Go (graded engagement + decay + like bonus) and atomic-replaces -// them here. Read queries serve "top taste" lookups (used in phase 2). +// them here. Read queries serve the scorer's "top taste" lookups. // One row per play in the decay-relevant window. completion is the play's // completion ratio (precomputed column when present, else duration_played / // track duration, clamped to [0,1]); age_days drives the time-decay. Genre // is split into tags in Go; release_date derives the decade for the era -// facet (#1530). Quarantined tracks are excluded. +// facet. Quarantined tracks are excluded. func (q *Queries) ListPlayEngagementInputsForUser(ctx context.Context, arg ListPlayEngagementInputsForUserParams) ([]ListPlayEngagementInputsForUserRow, error) { rows, err := q.db.Query(ctx, listPlayEngagementInputsForUser, arg.UserID, arg.Column2) if err != nil { @@ -269,7 +269,7 @@ type ListTasteProfileArtistsForUserRow struct { Weight float64 } -// Top-weighted taste artists (phase-2 consumption + tests). +// Top-weighted taste artists, for the scorer and tests. func (q *Queries) ListTasteProfileArtistsForUser(ctx context.Context, arg ListTasteProfileArtistsForUserParams) ([]ListTasteProfileArtistsForUserRow, error) { rows, err := q.db.Query(ctx, listTasteProfileArtistsForUser, arg.UserID, arg.Limit) if err != nil { @@ -308,7 +308,7 @@ type ListTasteProfileErasForUserRow struct { Weight float64 } -// Top-weighted taste eras (#1530); consumed by the scorer's era term. +// Top-weighted taste eras; consumed by the scorer's era term. func (q *Queries) ListTasteProfileErasForUser(ctx context.Context, arg ListTasteProfileErasForUserParams) ([]ListTasteProfileErasForUserRow, error) { rows, err := q.db.Query(ctx, listTasteProfileErasForUser, arg.UserID, arg.Limit) if err != nil { @@ -347,7 +347,7 @@ type ListTasteProfileMoodsForUserRow struct { Weight float64 } -// Top-weighted taste moods (#1534); consumed by the scorer's mood term. +// Top-weighted taste moods; consumed by the scorer's mood term. func (q *Queries) ListTasteProfileMoodsForUser(ctx context.Context, arg ListTasteProfileMoodsForUserParams) ([]ListTasteProfileMoodsForUserRow, error) { rows, err := q.db.Query(ctx, listTasteProfileMoodsForUser, arg.UserID, arg.Limit) if err != nil { diff --git a/internal/db/dbq/track_tags.sql.go b/internal/db/dbq/track_tags.sql.go index 70ddcec4..3e00caae 100644 --- a/internal/db/dbq/track_tags.sql.go +++ b/internal/db/dbq/track_tags.sql.go @@ -121,7 +121,7 @@ type ListTrackTagsForTracksRow struct { Tag string } -// Enriched tags for a set of candidate tracks (#1534), so the scorer can +// Enriched tags for a set of candidate tracks, so the scorer can // derive each candidate's mood buckets at scoring time. One row per (track, // tag); the caller filters to mood words via the internal/mood vocabulary. func (q *Queries) ListTrackTagsForTracks(ctx context.Context, dollar_1 []pgtype.UUID) ([]ListTrackTagsForTracksRow, error) { @@ -168,7 +168,7 @@ type ListTracksMissingTagsRow struct { ArtistMbid *string } -// Track-tag enrichment queries (milestone #160, #1490). The tag enricher +// Track-tag enrichment queries. The tag enricher // drains ListTracksMissingTags, fetches from the provider chain, and // writes the merged top-K via DeleteTrackTags + InsertTrackTag, then // stamps SetTrackTagSource. Mirrors the coverart artist-enricher shape. @@ -176,7 +176,7 @@ type ListTracksMissingTagsRow struct { // or previously settled 'none' under an older provider version. Returns // the fields the provider chain needs — recording MBID (nullable) for // keyed lookups, title + artist name for name-based fallback, and the -// artist MBID (nullable) for MusicBrainz's artist-tag fallback (#1519). +// artist MBID (nullable) for MusicBrainz's artist-tag fallback. // $1 = current tag_sources_version, $2 = limit. func (q *Queries) ListTracksMissingTags(ctx context.Context, arg ListTracksMissingTagsParams) ([]ListTracksMissingTagsRow, error) { rows, err := q.db.Query(ctx, listTracksMissingTags, arg.TagSourcesVersion, arg.Limit) diff --git a/internal/db/dbq/tracks.sql.go b/internal/db/dbq/tracks.sql.go index 6a9de912..5f136d09 100644 --- a/internal/db/dbq/tracks.sql.go +++ b/internal/db/dbq/tracks.sql.go @@ -46,8 +46,8 @@ UPDATE tracks AND NOT regexp_replace(file_path, '^.*/', '') ~* $1::text ` -// A held-back track whose name no longer carries a marker (renamed, retagged -// by Lidarr) is released again. +// A held-back track whose name no longer carries a marker (renamed or +// retagged) is released again. func (q *Queries) ClearStaleSuspectSourceTracks(ctx context.Context, pattern string) (int64, error) { result, err := q.db.Exec(ctx, clearStaleSuspectSourceTracks, pattern) if err != nil { @@ -56,6 +56,22 @@ func (q *Queries) ClearStaleSuspectSourceTracks(ctx context.Context, pattern str return result.RowsAffected(), nil } +const clearTrackMbids = `-- name: ClearTrackMbids :exec +UPDATE tracks + SET mbid = NULL, mbid_source = NULL + WHERE id = ANY($1::uuid[]) +` + +// Forgets the recording MBID of rows whose file was another song's (the +// duplicate resolver's wrong-slot removal). The MBID came from that file, so it +// names the song kept elsewhere on the album, and missing-pair folding would +// fold the emptied slot into it, so it would never be wanted back. +// mbid_source goes with it (tracks_mbid_source_pair). +func (q *Queries) ClearTrackMbids(ctx context.Context, ids []pgtype.UUID) error { + _, err := q.db.Exec(ctx, clearTrackMbids, ids) + return err +} + const clearTracksMissing = `-- name: ClearTracksMissing :execrows UPDATE tracks SET missing_since = NULL @@ -130,7 +146,7 @@ const countTracksMatching = `-- name: CountTracksMatching :one SELECT COUNT(*) FROM tracks WHERE title ILIKE '%' || $1::text || '%' AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = tracks.id ) ` @@ -161,9 +177,9 @@ type DeleteTrackRow struct { Mbid *string } -// M7 #372: hard delete with FK cascade. The CASCADE on track_id from -// play_events / general_likes_tracks / lidarr_quarantine / -// lidarr_quarantine_actions handles their cleanup. RETURNING gives us +// Hard delete with FK cascade. The CASCADE on track_id from +// play_events / general_likes_tracks / quarantine / +// quarantine_actions handles their cleanup. RETURNING gives us // album_id + artist_id for the album-empty / artist-empty cascade // checks the service does next. func (q *Queries) DeleteTrack(ctx context.Context, id pgtype.UUID) (DeleteTrackRow, error) { @@ -193,11 +209,11 @@ type FindMissingTrackByAudioHashRow struct { FilePath string } -// Move detection fallback for files with no MBID (#2528, #3914). The audio stream +// Move detection fallback for files with no MBID. The audio stream // hash identifies the encoded audio itself, so it survives a rename, a move and a -// retag — anything short of a re-encode. It replaced (file_size, duration_ms), -// which could pair two unrelated files that happened to share a byte count and a -// duration, and missed a file retagged in place, whose size changes. +// retag — anything short of a re-encode. (file_size, duration_ms) would pair two +// unrelated files that share a byte count and a duration, and miss a file +// retagged in place, whose size changes. // // Same missing-only constraint and same LIMIT 2 rationale as the MBID variant. func (q *Queries) FindMissingTrackByAudioHash(ctx context.Context, audioStreamSha256 []byte) ([]FindMissingTrackByAudioHashRow, error) { @@ -233,7 +249,7 @@ type FindMissingTrackByMbidRow struct { FilePath string } -// Move detection, strongest signal (#2528). A file that turned up at a new path +// Move detection, strongest signal. A file that turned up at a new path // carrying a recording MBID we already have on a MISSING row is that recording, // moved — not a new track. // @@ -271,8 +287,8 @@ UPDATE tracks AND regexp_replace(file_path, '^.*/', '') ~* $1::text ` -// Holds back present tracks whose basename carries a video-rip marker (M498 -// #5439), the same pattern the report lists. A track the operator called fine +// Holds back present tracks whose basename carries a video-rip marker, the +// same pattern the report lists. A track an admin called fine // keeps that verdict. func (q *Queries) FlagSuspectSourceTracks(ctx context.Context, pattern string) (int64, error) { result, err := q.db.Exec(ctx, flagSuspectSourceTracks, pattern) @@ -283,7 +299,7 @@ func (q *Queries) FlagSuspectSourceTracks(ctx context.Context, pattern string) ( } const getTrackByID = `-- name: GetTrackByID :one -SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at FROM tracks WHERE id = $1 +SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at, emptied_at FROM tracks WHERE id = $1 ` func (q *Queries) GetTrackByID(ctx context.Context, id pgtype.UUID) (Track, error) { @@ -314,12 +330,13 @@ func (q *Queries) GetTrackByID(ctx context.Context, id pgtype.UUID) (Track, erro &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ) return i, err } const getTrackByPath = `-- name: GetTrackByPath :one -SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at FROM tracks WHERE file_path = $1 +SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at, emptied_at FROM tracks WHERE file_path = $1 ` func (q *Queries) GetTrackByPath(ctx context.Context, filePath string) (Track, error) { @@ -350,16 +367,17 @@ func (q *Queries) GetTrackByPath(ctx context.Context, filePath string) (Track, e &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ) return i, err } const getTracksByIDs = `-- name: GetTracksByIDs :many -SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at FROM tracks WHERE id = ANY($1::uuid[]) +SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at, emptied_at FROM tracks WHERE id = ANY($1::uuid[]) ` // Batched lookup used by /api/library/sync to hydrate upsert payloads -// (#357). Mirror of GetArtistsByIDs. +// . Mirror of GetArtistsByIDs. func (q *Queries) GetTracksByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([]Track, error) { rows, err := q.db.Query(ctx, getTracksByIDs, dollar_1) if err != nil { @@ -394,6 +412,7 @@ func (q *Queries) GetTracksByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([ &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ); err != nil { return nil, err } @@ -406,7 +425,7 @@ func (q *Queries) GetTracksByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([ } const listArtistTracksForUser = `-- name: ListArtistTracksForUser :many -SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, +SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at, albums.title AS album_title, artists.name AS artist_name FROM tracks t @@ -414,7 +433,7 @@ JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id WHERE t.artist_id = $1 AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = t.id ) ORDER BY albums.release_date NULLS LAST, albums.sort_title, @@ -432,8 +451,8 @@ type ListArtistTracksForUserRow struct { ArtistName string } -// M6a: every track for the artist across their albums, with album_title -// and artist_name joined. Honors per-user lidarr_quarantine. Used by +// Every track for the artist across their albums, with album_title +// and artist_name joined. Honors per-user quarantine. Used by // /api/artists/{id}/tracks for the artist-card play affordance, which // shuffles client-side. Ordering matches album/track natural order so // the shuffle has a deterministic input. @@ -471,6 +490,7 @@ func (q *Queries) ListArtistTracksForUser(ctx context.Context, arg ListArtistTra &i.Track.SongKey, &i.Track.SourceVerdict, &i.Track.SourceVerdictAt, + &i.Track.EmptiedAt, &i.AlbumTitle, &i.ArtistName, ); err != nil { @@ -525,19 +545,18 @@ type ListMissingTracksRow struct { LastPlayedAt pgtype.Timestamptz } -// The admin review surface for files the scan could not find (#2527). +// The admin review surface for files the scan could not find. // // Ordered by directory, then by the file's own position within its album, -// because the unit an operator actually reasons about is a FOLDER: the case -// this was built for was three whole albums that had been reorganised, and a -// flat list ordered by timestamp presents that as forty unrelated decisions. +// because the unit an admin actually reasons about is a FOLDER: a reorganised +// album, listed flat by timestamp, reads as dozens of unrelated decisions. // Grouping happens in the handler; the ordering here is what makes a group // contiguous, so a page boundary splits a directory at worst. // // last_played_at is a correlated MAX rather than a join so a track with no // plays stays in the result with NULL. It is here because "gone six months, // never played" and "gone yesterday, played 200 times" deserve opposite -// reactions, and the operator can't tell them apart from a path. +// reactions, and an admin can't tell them apart from a path. func (q *Queries) ListMissingTracks(ctx context.Context, arg ListMissingTracksParams) ([]ListMissingTracksRow, error) { rows, err := q.db.Query(ctx, listMissingTracks, arg.PageOffset, arg.PageLimit) if err != nil { @@ -571,14 +590,14 @@ func (q *Queries) ListMissingTracks(ctx context.Context, arg ListMissingTracksPa } const listRandomTracksForUser = `-- name: ListRandomTracksForUser :many -SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, +SELECT t.id, t.title, t.album_id, t.artist_id, t.track_number, t.disc_number, t.duration_ms, t.file_path, t.file_size, t.file_format, t.bitrate, t.mbid, t.genre, t.added_at, t.updated_at, t.tag_source, t.tag_sources_version, t.tag_read_version, t.missing_since, t.mbid_source, t.song_id, t.song_key, t.source_verdict, t.source_verdict_at, t.emptied_at, albums.title AS album_title, artists.name AS artist_name FROM tracks t JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id WHERE NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY random() @@ -596,7 +615,7 @@ type ListRandomTracksForUserRow struct { ArtistName string } -// #427 S4: backing query for GET /api/library/shuffle — the online +// Backing query for GET /api/library/shuffle — the online // source for "Shuffle all". N random tracks across the whole // library, per-user-quarantine filtered. $1 user_id, $2 limit. func (q *Queries) ListRandomTracksForUser(ctx context.Context, arg ListRandomTracksForUserParams) ([]ListRandomTracksForUserRow, error) { @@ -633,6 +652,7 @@ func (q *Queries) ListRandomTracksForUser(ctx context.Context, arg ListRandomTra &i.Track.SongKey, &i.Track.SourceVerdict, &i.Track.SourceVerdictAt, + &i.Track.EmptiedAt, &i.AlbumTitle, &i.ArtistName, ); err != nil { @@ -646,6 +666,43 @@ func (q *Queries) ListRandomTracksForUser(ctx context.Context, arg ListRandomTra return items, nil } +const listStillMissingTrackLabels = `-- name: ListStillMissingTrackLabels :many +SELECT ar.name AS artist_name, t.title + FROM tracks t + JOIN artists ar ON ar.id = t.artist_id + WHERE t.id = ANY($1::uuid[]) + AND t.missing_since IS NOT NULL + ORDER BY ar.name, t.title +` + +type ListStillMissingTrackLabelsRow struct { + ArtistName string + Title string +} + +// Which of the tracks a scan marked missing still are once it has adopted +// moves and folded replacements: the ones worth telling an admin +// about, by name. A quality upgrade folds into its replacement and drops out. +func (q *Queries) ListStillMissingTrackLabels(ctx context.Context, ids []pgtype.UUID) ([]ListStillMissingTrackLabelsRow, error) { + rows, err := q.db.Query(ctx, listStillMissingTrackLabels, ids) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListStillMissingTrackLabelsRow + for rows.Next() { + var i ListStillMissingTrackLabelsRow + if err := rows.Scan(&i.ArtistName, &i.Title); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + const listSuspectSourceTracks = `-- name: ListSuspectSourceTracks :many SELECT t.id, t.title, @@ -659,10 +716,18 @@ SELECT t.id, artists.id AS artist_id, artists.name AS artist_name, t.source_verdict, - t.source_verdict_at + t.source_verdict_at, + -- The album's place on the wanted list: wanted for a clean copy, and + -- when that was last searched for. + COALESCE(w.upgrade, '')::text AS want_upgrade, + w.searched_at AS searched_at FROM tracks t JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id + LEFT JOIN LATERAL ( + SELECT x.upgrade, x.searched_at FROM wants x + WHERE x.album_id = albums.id AND x.fulfilled_at IS NOT NULL LIMIT 1 + ) w ON true WHERE t.missing_since IS NULL AND regexp_replace(t.file_path, '^.*/', '') ~* $1::text ORDER BY directory, t.disc_number NULLS FIRST, t.track_number NULLS FIRST, t.title @@ -689,10 +754,12 @@ type ListSuspectSourceTracksRow struct { ArtistName string SourceVerdict *string SourceVerdictAt pgtype.Timestamptz + WantUpgrade string + SearchedAt pgtype.Timestamptz } // The admin report of present tracks whose filename looks like a video rip -// (#5410): "(Official Video)", "[Audio]", "Visualizer" and the like. +// : "(Official Video)", "[Audio]", "Visualizer" and the like. // // pattern is built in Go (internal/api/admin_suspect_sources.go) from the // same marker list that labels each row, so the filter and the labels cannot @@ -724,6 +791,8 @@ func (q *Queries) ListSuspectSourceTracks(ctx context.Context, arg ListSuspectSo &i.ArtistName, &i.SourceVerdict, &i.SourceVerdictAt, + &i.WantUpgrade, + &i.SearchedAt, ); err != nil { return nil, err } @@ -746,7 +815,7 @@ type ListTrackPathsForReconcileRow struct { } // Every row's path + current missing mark, for the scanner's reconcile pass -// (#2523). Deliberately unfiltered and unpaged: reconcile has to compare the +// . Deliberately unfiltered and unpaged: reconcile has to compare the // WHOLE table against what the walk saw, and a filtered subset would let rows // outside it drift forever. Three narrow columns keep it cheap even on a // library of a few hundred thousand tracks. @@ -771,10 +840,10 @@ func (q *Queries) ListTrackPathsForReconcile(ctx context.Context) ([]ListTrackPa } const listTracksByAlbum = `-- name: ListTracksByAlbum :many -SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at FROM tracks +SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at, emptied_at FROM tracks WHERE album_id = $1 AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = tracks.id ) ORDER BY disc_number NULLS LAST, track_number NULLS LAST @@ -822,6 +891,7 @@ func (q *Queries) ListTracksByAlbum(ctx context.Context, arg ListTracksByAlbumPa &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ); err != nil { return nil, err } @@ -850,7 +920,7 @@ type ListTracksMissingMbidWithPathRow struct { // Track recording-MBID backfill: tracks with NULL mbid that still have // a file to re-read. $1 caps the batch (mirrors the album backfill). // Missing tracks are skipped: there is no file, and opening it anyway -// logged a warning per track on every scan (#5139). +// would log a warning per track on every scan. func (q *Queries) ListTracksMissingMbidWithPath(ctx context.Context, limit int32) ([]ListTracksMissingMbidWithPathRow, error) { rows, err := q.db.Query(ctx, listTracksMissingMbidWithPath, limit) if err != nil { @@ -871,6 +941,21 @@ func (q *Queries) ListTracksMissingMbidWithPath(ctx context.Context, limit int32 return items, nil } +const markTracksEmptied = `-- name: MarkTracksEmptied :exec +UPDATE tracks + SET emptied_at = missing_since + WHERE id = ANY($1::uuid[]) + AND missing_since IS NOT NULL +` + +// Stamps rows the duplicate resolver emptied, whose song the album holds no +// other copy of, with the time they went missing (migration 0095): missing-pair +// folding then waits for a copy fetched again. +func (q *Queries) MarkTracksEmptied(ctx context.Context, ids []pgtype.UUID) error { + _, err := q.db.Exec(ctx, markTracksEmptied, ids) + return err +} + const markTracksMissing = `-- name: MarkTracksMissing :execrows UPDATE tracks SET missing_since = now() @@ -895,10 +980,10 @@ func (q *Queries) MarkTracksMissing(ctx context.Context, ids []pgtype.UUID) (int } const searchTracks = `-- name: SearchTracks :many -SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at FROM tracks +SELECT id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at, emptied_at FROM tracks WHERE title ILIKE '%' || $1::text || '%' AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = tracks.id ) ORDER BY title @@ -953,6 +1038,7 @@ func (q *Queries) SearchTracks(ctx context.Context, arg SearchTracksParams) ([]T &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ); err != nil { return nil, err } @@ -994,7 +1080,7 @@ type SetTrackSourceVerdictParams struct { ID pgtype.UUID } -// The operator's call on a flagged track: 'fine' lets it back into radio and +// An admin's call on a flagged track: 'fine' lets it back into radio and // the mixes for good; 'suspect' holds it back again. func (q *Queries) SetTrackSourceVerdict(ctx context.Context, arg SetTrackSourceVerdictParams) (int64, error) { result, err := q.db.Exec(ctx, setTrackSourceVerdict, arg.Verdict, arg.ID) @@ -1028,10 +1114,10 @@ ON CONFLICT (file_path) DO UPDATE SET WHEN tracks.mbid_source = 'acoustid' THEN 'acoustid' END, genre = EXCLUDED.genre, -- Stamped on update too, so a tag-repair pass marks rows as done and the - -- next scan can short-circuit them again (#2499). + -- next scan can short-circuit them again. tag_read_version = EXCLUDED.tag_read_version, updated_at = now() -RETURNING id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at +RETURNING id, title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, added_at, updated_at, tag_source, tag_sources_version, tag_read_version, missing_since, mbid_source, song_id, song_key, source_verdict, source_verdict_at, emptied_at ` type UpsertTrackParams struct { @@ -1051,8 +1137,8 @@ type UpsertTrackParams struct { } // file_path is the canonical identity for library scan; mbid is secondary. -// mbid is the file's tag. A looked-up id (mbid_source 'acoustid', M401) -// survives a re-read that finds no tag id, and a tag id replaces it (D4). +// mbid is the file's tag. A looked-up id (mbid_source 'acoustid') +// survives a re-read that finds no tag id, and a tag id replaces it. func (q *Queries) UpsertTrack(ctx context.Context, arg UpsertTrackParams) (Track, error) { row := q.db.QueryRow(ctx, upsertTrack, arg.Title, @@ -1095,6 +1181,7 @@ func (q *Queries) UpsertTrack(ctx context.Context, arg UpsertTrackParams) (Track &i.SongKey, &i.SourceVerdict, &i.SourceVerdictAt, + &i.EmptiedAt, ) return i, err } diff --git a/internal/db/dbq/users.sql.go b/internal/db/dbq/users.sql.go index 0c1173c5..d21696fa 100644 --- a/internal/db/dbq/users.sql.go +++ b/internal/db/dbq/users.sql.go @@ -205,7 +205,7 @@ DELETE FROM users WHERE id = $1 // Hard delete with cascade. The schema's ON DELETE CASCADE foreign keys // handle plays/likes/sessions/etc. Caller must enforce the last-admin // guard at the HTTP layer (count admins, refuse if target is the only -// admin). The lidarr_quarantine, general_likes*, play_events, +// admin). The quarantine, general_likes*, play_events, // sessions, and other user-FK tables all have ON DELETE CASCADE so // this is a clean drop. func (q *Queries) DeleteUser(ctx context.Context, id pgtype.UUID) error { @@ -239,43 +239,6 @@ func (q *Queries) GetListenBrainzConfig(ctx context.Context, id pgtype.UUID) (Ge return i, err } -const getOldestAdmin = `-- name: GetOldestAdmin :one -SELECT id, username, password_hash, is_admin, created_at, subsonic_password, listenbrainz_token, listenbrainz_enabled, display_name, auto_approve_requests, email, timezone, timezone_updated_at, debug_mode_enabled, api_token_hash FROM users - WHERE is_admin = true - ORDER BY created_at, id - LIMIT 1 -` - -// The account a system-initiated action is attributed to (milestone #290). -// lidarr_requests.user_id is NOT NULL and a re-acquisition has no requesting -// human, so the row is owned by the longest-standing admin: it keeps the -// request auditable and puts it in the same admin queue as everything else, -// without inventing a synthetic principal the rest of the schema would have -// to understand. Ordered by id as a tiebreak so the choice is stable across -// calls rather than depending on scan order. -func (q *Queries) GetOldestAdmin(ctx context.Context) (User, error) { - row := q.db.QueryRow(ctx, getOldestAdmin) - var i User - err := row.Scan( - &i.ID, - &i.Username, - &i.PasswordHash, - &i.IsAdmin, - &i.CreatedAt, - &i.SubsonicPassword, - &i.ListenbrainzToken, - &i.ListenbrainzEnabled, - &i.DisplayName, - &i.AutoApproveRequests, - &i.Email, - &i.Timezone, - &i.TimezoneUpdatedAt, - &i.DebugModeEnabled, - &i.ApiTokenHash, - ) - return i, err -} - const getUserByAPITokenHash = `-- name: GetUserByAPITokenHash :one SELECT id, username, password_hash, is_admin, created_at, subsonic_password, listenbrainz_token, listenbrainz_enabled, display_name, auto_approve_requests, email, timezone, timezone_updated_at, debug_mode_enabled, api_token_hash FROM users WHERE api_token_hash = $1 ` @@ -425,6 +388,37 @@ func (q *Queries) ListActiveUsersWithTimezones(ctx context.Context) ([]ListActiv return items, nil } +const listUsernamesByIDs = `-- name: ListUsernamesByIDs :many +SELECT id, username FROM users WHERE id = ANY($1::uuid[]) +` + +type ListUsernamesByIDsRow struct { + ID pgtype.UUID + Username string +} + +// Names a list's people in one round trip, e.g. who made each request on +// /admin/requests. +func (q *Queries) ListUsernamesByIDs(ctx context.Context, dollar_1 []pgtype.UUID) ([]ListUsernamesByIDsRow, error) { + rows, err := q.db.Query(ctx, listUsernamesByIDs, dollar_1) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListUsernamesByIDsRow + for rows.Next() { + var i ListUsernamesByIDsRow + if err := rows.Scan(&i.ID, &i.Username); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + const listUsers = `-- name: ListUsers :many SELECT id, username, display_name, is_admin, auto_approve_requests, debug_mode_enabled, created_at @@ -590,7 +584,7 @@ type SetSubsonicPasswordParams struct { // Stores (or clears with NULL) the per-user Subsonic legacy credential used // for t/s and p auth on /rest/*. Must be plaintext; see migration 0003. Only -// ever a server-generated value, never the login password (#5026). +// ever a server-generated value, never the login password. func (q *Queries) SetSubsonicPassword(ctx context.Context, arg SetSubsonicPasswordParams) error { _, err := q.db.Exec(ctx, setSubsonicPassword, arg.ID, arg.SubsonicPassword) return err diff --git a/internal/db/dbq/wanted.sql.go b/internal/db/dbq/wanted.sql.go new file mode 100644 index 00000000..efe87083 --- /dev/null +++ b/internal/db/dbq/wanted.sql.go @@ -0,0 +1,1697 @@ +// Code generated by sqlc. DO NOT EDIT. +// versions: +// sqlc v1.31.1 +// source: wanted.sql + +package dbq + +import ( + "context" + + "github.com/jackc/pgx/v5/pgtype" +) + +const clearDefaultReleaseTypeProfile = `-- name: ClearDefaultReleaseTypeProfile :exec +UPDATE release_type_profiles SET is_default = false, updated_at = now() WHERE is_default +` + +func (q *Queries) ClearDefaultReleaseTypeProfile(ctx context.Context) error { + _, err := q.db.Exec(ctx, clearDefaultReleaseTypeProfile) + return err +} + +const clearHeldUpgrades = `-- name: ClearHeldUpgrades :execrows +UPDATE wants SET upgrade = '', held_tier = '', held_quality = '', upgrade_checked_at = now() + WHERE fulfilled_at IS NOT NULL AND upgrade <> '' + AND NOT EXISTS (SELECT 1 FROM tracks t + WHERE t.album_id = wants.album_id AND t.missing_since IS NULL AND t.duration_ms > 0) +` + +// A held want whose album has no track left to judge is not wanted again. +func (q *Queries) ClearHeldUpgrades(ctx context.Context) (int64, error) { + result, err := q.db.Exec(ctx, clearHeldUpgrades) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const countAlbumsToMatch = `-- name: CountAlbumsToMatch :one +SELECT COUNT(*) + FROM albums a + WHERE (a.created_at <= $1::timestamptz + OR EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id AND t.added_at <= $1::timestamptz)) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) + AND NOT EXISTS (SELECT 1 FROM library_album_matches m + WHERE m.album_id = a.id + AND (m.outcome IN ('listed', 'dismissed') + OR (m.matcher >= $2::int + AND NOT EXISTS (SELECT 1 FROM tracks t + WHERE t.album_id = a.id AND t.updated_at > m.decided_at)))) +` + +type CountAlbumsToMatchParams struct { + Cutoff pgtype.Timestamptz + Matcher int32 +} + +// Whether a finished listing has albums to go back to: those +// ListAlbumsToMatch would hand it. +func (q *Queries) CountAlbumsToMatch(ctx context.Context, arg CountAlbumsToMatchParams) (int64, error) { + row := q.db.QueryRow(ctx, countAlbumsToMatch, arg.Cutoff, arg.Matcher) + var count int64 + err := row.Scan(&count) + return count, err +} + +const countWants = `-- name: CountWants :one +SELECT COUNT(*) FILTER (WHERE fulfilled_at IS NULL AND reason <> 'missing')::bigint AS wanted, + COUNT(*) FILTER (WHERE fulfilled_at IS NULL AND reason = 'missing')::bigint AS missing, + COUNT(*) FILTER (WHERE fulfilled_at IS NOT NULL)::bigint AS held, + -- Held albums wanted for a better copy. + COUNT(*) FILTER (WHERE fulfilled_at IS NOT NULL AND upgrade <> '')::bigint AS upgrades, + COUNT(DISTINCT artist_mbid)::bigint AS artists, + -- Library albums waiting to be matched by hand. + (SELECT COUNT(*) FROM library_album_matches m + WHERE m.outcome = 'unmatched' + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id))::bigint AS unmatched, + (SELECT COUNT(*) FROM follows)::bigint AS following, + -- The Wanted view's other tabs: downloads running, those + -- waiting on a person, and releases set aside. + (SELECT COUNT(*) FROM grabs WHERE ended_at IS NULL)::bigint AS activity, + (SELECT COUNT(*) FROM grabs + WHERE ended_at IS NULL AND import_state IN ('review', 'failed'))::bigint AS attention, + (SELECT COUNT(*) FROM release_blocks b + WHERE b.expires_at IS NULL OR b.expires_at > now())::bigint AS blocked + FROM wants +` + +type CountWantsRow struct { + Wanted int64 + Missing int64 + Held int64 + Upgrades int64 + Artists int64 + Unmatched int64 + Following int64 + Activity int64 + Attention int64 + Blocked int64 +} + +func (q *Queries) CountWants(ctx context.Context) (CountWantsRow, error) { + row := q.db.QueryRow(ctx, countWants) + var i CountWantsRow + err := row.Scan( + &i.Wanted, + &i.Missing, + &i.Held, + &i.Upgrades, + &i.Artists, + &i.Unmatched, + &i.Following, + &i.Activity, + &i.Attention, + &i.Blocked, + ) + return i, err +} + +const createReleaseTypeProfile = `-- name: CreateReleaseTypeProfile :one +INSERT INTO release_type_profiles (name, primary_types, secondary_types) +VALUES ($1, $2, $3) +RETURNING id, name, primary_types, secondary_types, is_default, created_at, updated_at +` + +type CreateReleaseTypeProfileParams struct { + Name string + PrimaryTypes []string + SecondaryTypes []string +} + +func (q *Queries) CreateReleaseTypeProfile(ctx context.Context, arg CreateReleaseTypeProfileParams) (ReleaseTypeProfile, error) { + row := q.db.QueryRow(ctx, createReleaseTypeProfile, arg.Name, arg.PrimaryTypes, arg.SecondaryTypes) + var i ReleaseTypeProfile + err := row.Scan( + &i.ID, + &i.Name, + &i.PrimaryTypes, + &i.SecondaryTypes, + &i.IsDefault, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const deleteFollow = `-- name: DeleteFollow :execrows +DELETE FROM follows WHERE artist_mbid = $1 +` + +func (q *Queries) DeleteFollow(ctx context.Context, artistMbid string) (int64, error) { + result, err := q.db.Exec(ctx, deleteFollow, artistMbid) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const deleteReleaseTypeProfile = `-- name: DeleteReleaseTypeProfile :execrows +DELETE FROM release_type_profiles WHERE id = $1 AND NOT is_default +` + +// The default stays; one in use is refused by the follows foreign key. +func (q *Queries) DeleteReleaseTypeProfile(ctx context.Context, id pgtype.UUID) (int64, error) { + result, err := q.db.Exec(ctx, deleteReleaseTypeProfile, id) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const deleteWant = `-- name: DeleteWant :execrows +DELETE FROM wants WHERE id = $1 +` + +func (q *Queries) DeleteWant(ctx context.Context, id pgtype.UUID) (int64, error) { + result, err := q.db.Exec(ctx, deleteWant, id) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const finishLibraryListing = `-- name: FinishLibraryListing :one +UPDATE acquisition_settings + SET library_listed_at = COALESCE(library_listed_at, now()), + library_unlisted = (SELECT COUNT(*) FROM library_album_matches m + WHERE m.outcome = 'unmatched' + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id)) + WHERE id = true +RETURNING library_listed_at, library_unlisted +` + +type FinishLibraryListingRow struct { + LibraryListedAt pgtype.Timestamptz + LibraryUnlisted int32 +} + +// The listing is done, or done again after a newer matcher's retries: +// count the albums no match was found for. The first finish is kept. +func (q *Queries) FinishLibraryListing(ctx context.Context) (FinishLibraryListingRow, error) { + row := q.db.QueryRow(ctx, finishLibraryListing) + var i FinishLibraryListingRow + err := row.Scan(&i.LibraryListedAt, &i.LibraryUnlisted) + return i, err +} + +const fulfillWant = `-- name: FulfillWant :exec +UPDATE wants + SET fulfilled_at = now(), + album_id = $1, + updated_at = now() + WHERE id = $2 AND fulfilled_at IS NULL +` + +type FulfillWantParams struct { + AlbumID pgtype.UUID + ID pgtype.UUID +} + +func (q *Queries) FulfillWant(ctx context.Context, arg FulfillWantParams) error { + _, err := q.db.Exec(ctx, fulfillWant, arg.AlbumID, arg.ID) + return err +} + +const getAlbumToMatch = `-- name: GetAlbumToMatch :one +SELECT a.id, a.title, a.mbid, a.release_group_mbid, ar.name AS artist_name, ar.mbid AS artist_mbid + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + JOIN library_album_matches m ON m.album_id = a.id AND m.outcome IN ('unmatched', 'dismissed') + WHERE a.id = $1 +` + +type GetAlbumToMatchRow struct { + ID pgtype.UUID + Title string + Mbid *string + ReleaseGroupMbid *string + ArtistName string + ArtistMbid *string +} + +// One album a person is matching by hand, as the matcher reads albums. +func (q *Queries) GetAlbumToMatch(ctx context.Context, id pgtype.UUID) (GetAlbumToMatchRow, error) { + row := q.db.QueryRow(ctx, getAlbumToMatch, id) + var i GetAlbumToMatchRow + err := row.Scan( + &i.ID, + &i.Title, + &i.Mbid, + &i.ReleaseGroupMbid, + &i.ArtistName, + &i.ArtistMbid, + ) + return i, err +} + +const getDefaultReleaseTypeProfile = `-- name: GetDefaultReleaseTypeProfile :one +SELECT id, name, primary_types, secondary_types, is_default, created_at, updated_at FROM release_type_profiles WHERE is_default +` + +func (q *Queries) GetDefaultReleaseTypeProfile(ctx context.Context) (ReleaseTypeProfile, error) { + row := q.db.QueryRow(ctx, getDefaultReleaseTypeProfile) + var i ReleaseTypeProfile + err := row.Scan( + &i.ID, + &i.Name, + &i.PrimaryTypes, + &i.SecondaryTypes, + &i.IsDefault, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const getFollow = `-- name: GetFollow :one +SELECT f.artist_mbid, f.artist_name, f.profile_id, f.followed_by, f.followed_at, f.baseline_at, f.want_existing, f.checked_at, f.updated_at, p.name AS profile_name + FROM follows f + JOIN release_type_profiles p ON p.id = f.profile_id + WHERE f.artist_mbid = $1 +` + +type GetFollowRow struct { + ArtistMbid string + ArtistName string + ProfileID pgtype.UUID + FollowedBy pgtype.UUID + FollowedAt pgtype.Timestamptz + BaselineAt pgtype.Timestamptz + WantExisting bool + CheckedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz + ProfileName string +} + +func (q *Queries) GetFollow(ctx context.Context, artistMbid string) (GetFollowRow, error) { + row := q.db.QueryRow(ctx, getFollow, artistMbid) + var i GetFollowRow + err := row.Scan( + &i.ArtistMbid, + &i.ArtistName, + &i.ProfileID, + &i.FollowedBy, + &i.FollowedAt, + &i.BaselineAt, + &i.WantExisting, + &i.CheckedAt, + &i.UpdatedAt, + &i.ProfileName, + ) + return i, err +} + +const getLibraryListing = `-- name: GetLibraryListing :one + +SELECT library_listing_started_at, library_listed_at, library_unlisted + FROM acquisition_settings WHERE id = true +` + +type GetLibraryListingRow struct { + LibraryListingStartedAt pgtype.Timestamptz + LibraryListedAt pgtype.Timestamptz + LibraryUnlisted int32 +} + +// The one-time listing of the library. --------------------------- +func (q *Queries) GetLibraryListing(ctx context.Context) (GetLibraryListingRow, error) { + row := q.db.QueryRow(ctx, getLibraryListing) + var i GetLibraryListingRow + err := row.Scan(&i.LibraryListingStartedAt, &i.LibraryListedAt, &i.LibraryUnlisted) + return i, err +} + +const getReleaseTypeProfile = `-- name: GetReleaseTypeProfile :one +SELECT id, name, primary_types, secondary_types, is_default, created_at, updated_at FROM release_type_profiles WHERE id = $1 +` + +func (q *Queries) GetReleaseTypeProfile(ctx context.Context, id pgtype.UUID) (ReleaseTypeProfile, error) { + row := q.db.QueryRow(ctx, getReleaseTypeProfile, id) + var i ReleaseTypeProfile + err := row.Scan( + &i.ID, + &i.Name, + &i.PrimaryTypes, + &i.SecondaryTypes, + &i.IsDefault, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const getWantAlbum = `-- name: GetWantAlbum :one +SELECT id, release_group_mbid, album_id FROM wants WHERE id = $1 +` + +type GetWantAlbumRow struct { + ID pgtype.UUID + ReleaseGroupMbid string + AlbumID pgtype.UUID +} + +// What the Wanted view's track list needs of a want. +func (q *Queries) GetWantAlbum(ctx context.Context, id pgtype.UUID) (GetWantAlbumRow, error) { + row := q.db.QueryRow(ctx, getWantAlbum, id) + var i GetWantAlbumRow + err := row.Scan(&i.ID, &i.ReleaseGroupMbid, &i.AlbumID) + return i, err +} + +const getWantByReleaseGroup = `-- name: GetWantByReleaseGroup :one +SELECT id, release_group_mbid, artist_mbid, artist_name, title, primary_type, first_release_date, reason, album_id, created_by, wanted_at, fulfilled_at, created_at, updated_at, searched_at, search_outcome, search_considered, best_release, best_verdict, search_error, upgrade, held_tier, held_quality, upgrade_checked_at FROM wants WHERE release_group_mbid = $1 +` + +func (q *Queries) GetWantByReleaseGroup(ctx context.Context, releaseGroupMbid string) (Want, error) { + row := q.db.QueryRow(ctx, getWantByReleaseGroup, releaseGroupMbid) + var i Want + err := row.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.Title, + &i.PrimaryType, + &i.FirstReleaseDate, + &i.Reason, + &i.AlbumID, + &i.CreatedBy, + &i.WantedAt, + &i.FulfilledAt, + &i.CreatedAt, + &i.UpdatedAt, + &i.SearchedAt, + &i.SearchOutcome, + &i.SearchConsidered, + &i.BestRelease, + &i.BestVerdict, + &i.SearchError, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.UpgradeCheckedAt, + ) + return i, err +} + +const insertLibraryWant = `-- name: InsertLibraryWant :execrows +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, primary_type, first_release_date, + reason, album_id, wanted_at, fulfilled_at) +VALUES ($1, $2, $3, $4, + $5, $6, 'library', $7::uuid, now(), + COALESCE((SELECT MIN(t.missing_since) FROM tracks t WHERE t.album_id = $7::uuid) + - interval '1 second', now())) +ON CONFLICT (release_group_mbid) DO NOTHING +` + +type InsertLibraryWantParams struct { + ReleaseGroupMbid string + ArtistMbid string + ArtistName string + Title string + PrimaryType string + FirstReleaseDate string + AlbumID pgtype.UUID +} + +// Puts one matched album on the list as held, from just before its first +// lost track if it has one. A group already on the list is left as it is. +func (q *Queries) InsertLibraryWant(ctx context.Context, arg InsertLibraryWantParams) (int64, error) { + result, err := q.db.Exec(ctx, insertLibraryWant, + arg.ReleaseGroupMbid, + arg.ArtistMbid, + arg.ArtistName, + arg.Title, + arg.PrimaryType, + arg.FirstReleaseDate, + arg.AlbumID, + ) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const libraryHoldsReleaseGroup = `-- name: LibraryHoldsReleaseGroup :one +SELECT EXISTS ( + SELECT 1 FROM albums a + WHERE (a.release_group_mbid = $1::text + OR a.mbid = $1 + OR a.mbid IN (SELECT r.mbid FROM mb_releases r WHERE r.release_group_mbid = $1) + OR (lower(a.title) = lower($2::text) + AND a.artist_id IN (SELECT ar.id FROM artists ar WHERE ar.mbid = $3::text))) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id AND t.missing_since IS NULL) +) +` + +type LibraryHoldsReleaseGroupParams struct { + GroupMbid string + Title string + ArtistMbid string +} + +// Whether the library already has the group: an album tagged with it or +// with one of its releases, or failing that an album of the artist's with +// the same title. Only albums with a track on disk count. +func (q *Queries) LibraryHoldsReleaseGroup(ctx context.Context, arg LibraryHoldsReleaseGroupParams) (bool, error) { + row := q.db.QueryRow(ctx, libraryHoldsReleaseGroup, arg.GroupMbid, arg.Title, arg.ArtistMbid) + var exists bool + err := row.Scan(&exists) + return exists, err +} + +const listAlbumTracksToMatch = `-- name: ListAlbumTracksToMatch :many +SELECT title, disc_number, track_number, duration_ms, mbid + FROM tracks + WHERE album_id = $1 + ORDER BY disc_number NULLS FIRST, track_number NULLS LAST, title +` + +type ListAlbumTracksToMatchRow struct { + Title string + DiscNumber *int32 + TrackNumber *int32 + DurationMs int32 + Mbid *string +} + +// What an album's tracks say about themselves, to check a match against. +func (q *Queries) ListAlbumTracksToMatch(ctx context.Context, albumID pgtype.UUID) ([]ListAlbumTracksToMatchRow, error) { + rows, err := q.db.Query(ctx, listAlbumTracksToMatch, albumID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListAlbumTracksToMatchRow + for rows.Next() { + var i ListAlbumTracksToMatchRow + if err := rows.Scan( + &i.Title, + &i.DiscNumber, + &i.TrackNumber, + &i.DurationMs, + &i.Mbid, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listAlbumsToMatch = `-- name: ListAlbumsToMatch :many +SELECT a.id, a.title, a.mbid, a.release_group_mbid, ar.name AS artist_name, ar.mbid AS artist_mbid + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + WHERE (a.created_at <= $1::timestamptz + OR EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id AND t.added_at <= $1::timestamptz)) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) + AND NOT EXISTS (SELECT 1 FROM library_album_matches m + WHERE m.album_id = a.id + AND (m.outcome IN ('listed', 'dismissed') + OR (m.matcher >= $2::int + AND NOT EXISTS (SELECT 1 FROM tracks t + WHERE t.album_id = a.id AND t.updated_at > m.decided_at)))) + ORDER BY a.created_at, a.id + LIMIT $3 +` + +type ListAlbumsToMatchParams struct { + Cutoff pgtype.Timestamptz + Matcher int32 + MaxRows int32 +} + +type ListAlbumsToMatchRow struct { + ID pgtype.UUID + Title string + Mbid *string + ReleaseGroupMbid *string + ArtistName string + ArtistMbid *string +} + +// Library albums the listing has not decided, or that an older matcher +// refused, or whose tracks changed since they were refused: each is matched +// to MusicBrainz. A library album is one from before the cutoff, or +// one whose tracks were: a re-tag files a track under a new album row but +// keeps the track. An album row with no tracks is not an album in +// the library; one a person left off stays off. +func (q *Queries) ListAlbumsToMatch(ctx context.Context, arg ListAlbumsToMatchParams) ([]ListAlbumsToMatchRow, error) { + rows, err := q.db.Query(ctx, listAlbumsToMatch, arg.Cutoff, arg.Matcher, arg.MaxRows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListAlbumsToMatchRow + for rows.Next() { + var i ListAlbumsToMatchRow + if err := rows.Scan( + &i.ID, + &i.Title, + &i.Mbid, + &i.ReleaseGroupMbid, + &i.ArtistName, + &i.ArtistMbid, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listFollowGroups = `-- name: ListFollowGroups :many +SELECT g.mbid, g.title, g.primary_type, g.secondary_types, g.first_release_date, + EXISTS (SELECT 1 FROM follow_seen_groups s + WHERE s.artist_mbid = $1 AND s.release_group_mbid = g.mbid) AS seen + FROM mb_artist_release_groups ag + JOIN mb_release_groups g ON g.mbid = ag.release_group_mbid + WHERE ag.artist_mbid = $1 + ORDER BY g.first_release_date, g.title +` + +type ListFollowGroupsRow struct { + Mbid string + Title string + PrimaryType string + SecondaryTypes []string + FirstReleaseDate string + Seen bool +} + +// An artist's release groups from the catalogue, each with whether the +// follow has judged it before. +func (q *Queries) ListFollowGroups(ctx context.Context, artistMbid string) ([]ListFollowGroupsRow, error) { + rows, err := q.db.Query(ctx, listFollowGroups, artistMbid) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListFollowGroupsRow + for rows.Next() { + var i ListFollowGroupsRow + if err := rows.Scan( + &i.Mbid, + &i.Title, + &i.PrimaryType, + &i.SecondaryTypes, + &i.FirstReleaseDate, + &i.Seen, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listFollows = `-- name: ListFollows :many +SELECT f.artist_mbid, f.artist_name, f.profile_id, f.followed_at, f.baseline_at, f.checked_at, + p.name AS profile_name, + (SELECT a.id FROM artists a WHERE a.mbid = f.artist_mbid) AS artist_id, + (SELECT COUNT(*) FROM wants w + WHERE w.artist_mbid = f.artist_mbid AND w.fulfilled_at IS NULL)::bigint AS open_wants + FROM follows f + JOIN release_type_profiles p ON p.id = f.profile_id + ORDER BY lower(f.artist_name), f.artist_mbid +` + +type ListFollowsRow struct { + ArtistMbid string + ArtistName string + ProfileID pgtype.UUID + FollowedAt pgtype.Timestamptz + BaselineAt pgtype.Timestamptz + CheckedAt pgtype.Timestamptz + ProfileName string + ArtistID pgtype.UUID + OpenWants int64 +} + +func (q *Queries) ListFollows(ctx context.Context) ([]ListFollowsRow, error) { + rows, err := q.db.Query(ctx, listFollows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListFollowsRow + for rows.Next() { + var i ListFollowsRow + if err := rows.Scan( + &i.ArtistMbid, + &i.ArtistName, + &i.ProfileID, + &i.FollowedAt, + &i.BaselineAt, + &i.CheckedAt, + &i.ProfileName, + &i.ArtistID, + &i.OpenWants, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listFollowsDue = `-- name: ListFollowsDue :many +SELECT f.artist_mbid, f.artist_name, f.profile_id, f.followed_by, f.followed_at, f.baseline_at, f.want_existing, f.checked_at, f.updated_at + FROM follows f + WHERE f.baseline_at IS NULL OR f.checked_at IS NULL OR f.checked_at < $1 + ORDER BY f.baseline_at NULLS FIRST, f.checked_at NULLS FIRST, f.artist_mbid + LIMIT $2 +` + +type ListFollowsDueParams struct { + Before pgtype.Timestamptz + MaxRows int32 +} + +// Follows whose discography has not been applied since @before, never-read +// first, so a new follow's baseline is not stuck behind daily re-checks. +func (q *Queries) ListFollowsDue(ctx context.Context, arg ListFollowsDueParams) ([]Follow, error) { + rows, err := q.db.Query(ctx, listFollowsDue, arg.Before, arg.MaxRows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Follow + for rows.Next() { + var i Follow + if err := rows.Scan( + &i.ArtistMbid, + &i.ArtistName, + &i.ProfileID, + &i.FollowedBy, + &i.FollowedAt, + &i.BaselineAt, + &i.WantExisting, + &i.CheckedAt, + &i.UpdatedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listHeldWantFormats = `-- name: ListHeldWantFormats :many + +SELECT w.id AS want_id, w.upgrade, w.held_tier, w.held_quality, + lower(t.file_format)::text AS file_format, + SUM(t.file_size)::bigint AS bytes, + SUM(t.duration_ms)::bigint AS duration_ms, + COUNT(*)::int AS tracks, + COUNT(*) FILTER (WHERE t.source_verdict = 'suspect')::int AS rips + FROM wants w + JOIN tracks t ON t.album_id = w.album_id + WHERE w.fulfilled_at IS NOT NULL + AND t.missing_since IS NULL AND t.duration_ms > 0 + GROUP BY w.id, lower(t.file_format) + ORDER BY w.id +` + +type ListHeldWantFormatsRow struct { + WantID pgtype.UUID + Upgrade string + HeldTier string + HeldQuality string + FileFormat string + Bytes int64 + DurationMs int64 + Tracks int32 + Rips int32 +} + +// Upgrades. ------------------------------------------------------- +// What each held album's copy is made of, for judging whether it is worth +// upgrading: its present tracks grouped by format, with their bytes and +// length summed, and how many are held back as video rips. A held want whose +// album has no track with a length yet has no rows. +func (q *Queries) ListHeldWantFormats(ctx context.Context) ([]ListHeldWantFormatsRow, error) { + rows, err := q.db.Query(ctx, listHeldWantFormats) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListHeldWantFormatsRow + for rows.Next() { + var i ListHeldWantFormatsRow + if err := rows.Scan( + &i.WantID, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.FileFormat, + &i.Bytes, + &i.DurationMs, + &i.Tracks, + &i.Rips, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listLibraryAlbumTracks = `-- name: ListLibraryAlbumTracks :many +SELECT title, disc_number, track_number, duration_ms, (missing_since IS NOT NULL)::bool AS missing + FROM tracks + WHERE album_id = $1 + ORDER BY disc_number NULLS FIRST, track_number NULLS LAST, title +` + +type ListLibraryAlbumTracksRow struct { + Title string + DiscNumber *int32 + TrackNumber *int32 + DurationMs int32 + Missing bool +} + +// A held or missing album's own tracks, for the Wanted view's track list; +// missing says its file is gone. +func (q *Queries) ListLibraryAlbumTracks(ctx context.Context, albumID pgtype.UUID) ([]ListLibraryAlbumTracksRow, error) { + rows, err := q.db.Query(ctx, listLibraryAlbumTracks, albumID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListLibraryAlbumTracksRow + for rows.Next() { + var i ListLibraryAlbumTracksRow + if err := rows.Scan( + &i.Title, + &i.DiscNumber, + &i.TrackNumber, + &i.DurationMs, + &i.Missing, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listLibraryAlbums = `-- name: ListLibraryAlbums :execrows +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, album_id, wanted_at, fulfilled_at) +SELECT DISTINCT ON (a.release_group_mbid) + a.release_group_mbid, ar.mbid, ar.name, a.title, 'library', a.id, now(), + COALESCE((SELECT MIN(t.missing_since) FROM tracks t WHERE t.album_id = a.id) - interval '1 second', now()) + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + WHERE a.release_group_mbid IS NOT NULL AND a.release_group_mbid <> '' + AND ar.mbid IS NOT NULL AND ar.mbid <> '' + AND a.created_at <= $1::timestamptz + AND ($2::uuid IS NULL OR a.id = $2::uuid) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) + ORDER BY a.release_group_mbid, + (SELECT COUNT(*) FROM tracks t WHERE t.album_id = a.id AND t.missing_since IS NULL) DESC, a.id +ON CONFLICT (release_group_mbid) DO NOTHING +` + +type ListLibraryAlbumsParams struct { + Cutoff pgtype.Timestamptz + AlbumID pgtype.UUID +} + +// Puts library albums on the list as held: each release group once, by the +// album holding the most tracks. Only albums the library had at the cutoff, +// and with album_id given only that album. A group already on the list is +// left as it is. An album with tracks already missing is held from just +// before the first loss, so re-acquisition still fetches it back. +func (q *Queries) ListLibraryAlbums(ctx context.Context, arg ListLibraryAlbumsParams) (int64, error) { + result, err := q.db.Exec(ctx, listLibraryAlbums, arg.Cutoff, arg.AlbumID) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const listOpenWants = `-- name: ListOpenWants :many +SELECT id, release_group_mbid, artist_mbid, artist_name, title, primary_type, first_release_date, reason, album_id, created_by, wanted_at, fulfilled_at, created_at, updated_at, searched_at, search_outcome, search_considered, best_release, best_verdict, search_error, upgrade, held_tier, held_quality, upgrade_checked_at FROM wants + WHERE fulfilled_at IS NULL + ORDER BY wanted_at, id + LIMIT $1 +` + +func (q *Queries) ListOpenWants(ctx context.Context, limit int32) ([]Want, error) { + rows, err := q.db.Query(ctx, listOpenWants, limit) + if err != nil { + return nil, err + } + defer rows.Close() + var items []Want + for rows.Next() { + var i Want + if err := rows.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.Title, + &i.PrimaryType, + &i.FirstReleaseDate, + &i.Reason, + &i.AlbumID, + &i.CreatedBy, + &i.WantedAt, + &i.FulfilledAt, + &i.CreatedAt, + &i.UpdatedAt, + &i.SearchedAt, + &i.SearchOutcome, + &i.SearchConsidered, + &i.BestRelease, + &i.BestVerdict, + &i.SearchError, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.UpgradeCheckedAt, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listReleaseTypeProfiles = `-- name: ListReleaseTypeProfiles :many + +SELECT p.id, p.name, p.primary_types, p.secondary_types, p.is_default, p.created_at, p.updated_at, + (SELECT COUNT(*) FROM follows f WHERE f.profile_id = p.id)::bigint AS follow_count + FROM release_type_profiles p + ORDER BY p.is_default DESC, p.name +` + +type ListReleaseTypeProfilesRow struct { + ID pgtype.UUID + Name string + PrimaryTypes []string + SecondaryTypes []string + IsDefault bool + CreatedAt pgtype.Timestamptz + UpdatedAt pgtype.Timestamptz + FollowCount int64 +} + +// The wanted list: release-type profiles, follows and wants. +func (q *Queries) ListReleaseTypeProfiles(ctx context.Context) ([]ListReleaseTypeProfilesRow, error) { + rows, err := q.db.Query(ctx, listReleaseTypeProfiles) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListReleaseTypeProfilesRow + for rows.Next() { + var i ListReleaseTypeProfilesRow + if err := rows.Scan( + &i.ID, + &i.Name, + &i.PrimaryTypes, + &i.SecondaryTypes, + &i.IsDefault, + &i.CreatedAt, + &i.UpdatedAt, + &i.FollowCount, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listUnmatchedLibraryAlbums = `-- name: ListUnmatchedLibraryAlbums :many +SELECT m.album_id, a.title, ar.name AS artist_name, m.detail, m.outcome, m.decided_at, + (SELECT COUNT(*) FROM tracks t WHERE t.album_id = a.id AND t.missing_since IS NULL)::int AS tracks + FROM library_album_matches m + JOIN albums a ON a.id = m.album_id + JOIN artists ar ON ar.id = a.artist_id + WHERE m.outcome IN ('unmatched', 'dismissed') + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id) + ORDER BY m.outcome DESC, lower(ar.name), lower(a.title) + LIMIT $1 +` + +type ListUnmatchedLibraryAlbumsRow struct { + AlbumID pgtype.UUID + Title string + ArtistName string + Detail string + Outcome string + DecidedAt pgtype.Timestamptz + Tracks int32 +} + +// The Collection's Unmatched section: the albums the listing could +// not match, and below them the ones a person left off. An album whose +// tracks have all moved to another row is gone, not unmatched. +func (q *Queries) ListUnmatchedLibraryAlbums(ctx context.Context, maxRows int32) ([]ListUnmatchedLibraryAlbumsRow, error) { + rows, err := q.db.Query(ctx, listUnmatchedLibraryAlbums, maxRows) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListUnmatchedLibraryAlbumsRow + for rows.Next() { + var i ListUnmatchedLibraryAlbumsRow + if err := rows.Scan( + &i.AlbumID, + &i.Title, + &i.ArtistName, + &i.Detail, + &i.Outcome, + &i.DecidedAt, + &i.Tracks, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listWantArtists = `-- name: ListWantArtists :many +WITH a AS ( + SELECT w.artist_mbid, + (array_agg(w.artist_name ORDER BY w.wanted_at DESC))[1] AS artist_name, + COUNT(*) FILTER (WHERE w.fulfilled_at IS NOT NULL)::int AS held, + COUNT(*) FILTER (WHERE w.fulfilled_at IS NULL AND w.reason <> 'missing')::int AS wanted, + COUNT(*) FILTER (WHERE w.fulfilled_at IS NULL AND w.reason = 'missing')::int AS missing, + COUNT(*)::int AS albums, + bool_or($1::text <> '' AND strpos(lower(w.title), lower($1::text)) > 0) AS title_match + FROM wants w + WHERE $4::text = '' OR w.artist_mbid = $4::text + GROUP BY w.artist_mbid +) +SELECT a.artist_mbid, a.artist_name::text AS artist_name, a.held, a.wanted, a.missing, a.albums, + COALESCE(NULLIF(la.sort_name, ''), a.artist_name)::text AS sort_name, + la.id AS library_artist_id, + (f.artist_mbid IS NOT NULL)::bool AS following, + (SELECT COUNT(*) FROM grabs g JOIN wants x ON x.id = g.want_id + WHERE x.artist_mbid = a.artist_mbid AND g.ended_at IS NULL)::int AS downloading, + COUNT(*) OVER () AS total + FROM a + LEFT JOIN LATERAL (SELECT ar.id, ar.sort_name FROM artists ar WHERE ar.mbid = a.artist_mbid LIMIT 1) la ON true + LEFT JOIN follows f ON f.artist_mbid = a.artist_mbid + WHERE $1::text = '' + OR strpos(lower(a.artist_name), lower($1::text)) > 0 + OR a.title_match + ORDER BY lower(COALESCE(NULLIF(la.sort_name, ''), a.artist_name)), a.artist_mbid + LIMIT $3 OFFSET $2 +` + +type ListWantArtistsParams struct { + Q string + Skip int32 + MaxRows int32 + ArtistMbid string +} + +type ListWantArtistsRow struct { + ArtistMbid string + ArtistName string + Held int32 + Wanted int32 + Missing int32 + Albums int32 + SortName string + LibraryArtistID pgtype.UUID + Following bool + Downloading int32 + Total int64 +} + +// The Wanted view's main page: every artist on the list, A–Z like +// the library, with how many of their albums are held of how many are on +// the list. q matches the artist's name or any of their albums' titles; +// artist_mbid narrows to one artist, for their page's header. +func (q *Queries) ListWantArtists(ctx context.Context, arg ListWantArtistsParams) ([]ListWantArtistsRow, error) { + rows, err := q.db.Query(ctx, listWantArtists, + arg.Q, + arg.Skip, + arg.MaxRows, + arg.ArtistMbid, + ) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListWantArtistsRow + for rows.Next() { + var i ListWantArtistsRow + if err := rows.Scan( + &i.ArtistMbid, + &i.ArtistName, + &i.Held, + &i.Wanted, + &i.Missing, + &i.Albums, + &i.SortName, + &i.LibraryArtistID, + &i.Following, + &i.Downloading, + &i.Total, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const listWants = `-- name: ListWants :many +SELECT w.id, w.release_group_mbid, w.artist_mbid, w.artist_name, w.title, w.reason, + w.album_id, w.wanted_at, w.fulfilled_at, + -- Why a held album is wanted again, and the copy it has. + w.upgrade, w.held_quality, + -- A request names an album before the catalogue has read it: the + -- catalogue's type and date fill in once it has. + COALESCE(NULLIF(w.primary_type, ''), g.primary_type, '')::text AS primary_type, + COALESCE(g.secondary_types, '{}')::text[] AS secondary_types, + COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, '')::text AS first_release_date, + u.username AS requested_by, + la.id AS library_artist_id, + COUNT(*) OVER () AS total, + -- The search schedule's last look, and the download running. + w.searched_at, w.search_outcome, w.search_considered, w.best_release, w.best_verdict, w.search_error, + -- Its latest download, running or ended, and what the import made of + -- it. + gr.id AS grab_id, gr.name AS grab_name, gr.grabbed_at, gr.origin AS grab_origin, + gr.quality AS grab_quality, gr.ended_at AS grab_ended_at, gr.end_reason AS grab_end_reason, + gr.progress AS grab_progress, gr.import_state AS grab_import_state, + gr.import_reason AS grab_import_reason, gr.end_detail AS grab_end_detail, gr.imported_at AS grab_imported_at, + gr.imported_files AS grab_imported_files + FROM wants w + LEFT JOIN users u ON u.id = w.created_by + LEFT JOIN mb_release_groups g ON g.mbid = w.release_group_mbid + LEFT JOIN LATERAL (SELECT ar.id, ar.sort_name FROM artists ar WHERE ar.mbid = w.artist_mbid LIMIT 1) la ON true + LEFT JOIN grabs gr ON gr.id = (SELECT x.id FROM grabs x WHERE x.want_id = w.id + ORDER BY x.ended_at IS NULL DESC, x.grabbed_at DESC LIMIT 1) + WHERE CASE $1::text + WHEN 'all' THEN true + WHEN 'held' THEN w.fulfilled_at IS NOT NULL + WHEN 'upgrade' THEN w.fulfilled_at IS NOT NULL AND w.upgrade <> '' + WHEN 'missing' THEN w.fulfilled_at IS NULL AND w.reason = 'missing' + ELSE w.fulfilled_at IS NULL AND w.reason <> 'missing' + END + AND ($2::text = '' OR w.artist_mbid = $2::text) + -- want_ids narrows to given wants: the Missing page's folders. + AND (COALESCE(cardinality($3::uuid[]), 0) = 0 OR w.id = ANY($3::uuid[])) + AND ($4::text = '' + OR strpos(lower(w.artist_name), lower($4::text)) > 0 + OR strpos(lower(w.title), lower($4::text)) > 0) + ORDER BY lower(COALESCE(NULLIF(la.sort_name, ''), w.artist_name)), w.artist_mbid, + NULLIF(COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, ''), '') NULLS LAST, + lower(w.title), w.id + LIMIT $6 OFFSET $5 +` + +type ListWantsParams struct { + State string + ArtistMbid string + WantIds []pgtype.UUID + Q string + Skip int32 + MaxRows int32 +} + +type ListWantsRow struct { + ID pgtype.UUID + ReleaseGroupMbid string + ArtistMbid string + ArtistName string + Title string + Reason string + AlbumID pgtype.UUID + WantedAt pgtype.Timestamptz + FulfilledAt pgtype.Timestamptz + Upgrade string + HeldQuality string + PrimaryType string + SecondaryTypes []string + FirstReleaseDate string + RequestedBy *string + LibraryArtistID pgtype.UUID + Total int64 + SearchedAt pgtype.Timestamptz + SearchOutcome string + SearchConsidered int32 + BestRelease string + BestVerdict string + SearchError string + GrabID pgtype.UUID + GrabName *string + GrabbedAt pgtype.Timestamptz + GrabOrigin *string + GrabQuality *string + GrabEndedAt pgtype.Timestamptz + GrabEndReason *string + GrabProgress *float32 + GrabImportState *string + GrabImportReason *string + GrabEndDetail *string + GrabImportedAt pgtype.Timestamptz + GrabImportedFiles *int32 +} + +// The Wanted page: open wants (all but missing ones, or only missing ones), +// the held ones, or all of them; artist_mbid narrows to one artist's page, +// want_ids to given wants. Ordered like the library: artists A–Z by the +// library's sort name when it holds the artist, each artist's albums by +// release date, so the page groups them as they arrive. q narrows to an +// artist or album title; total counts every match, past the page. +func (q *Queries) ListWants(ctx context.Context, arg ListWantsParams) ([]ListWantsRow, error) { + rows, err := q.db.Query(ctx, listWants, + arg.State, + arg.ArtistMbid, + arg.WantIds, + arg.Q, + arg.Skip, + arg.MaxRows, + ) + if err != nil { + return nil, err + } + defer rows.Close() + var items []ListWantsRow + for rows.Next() { + var i ListWantsRow + if err := rows.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.Title, + &i.Reason, + &i.AlbumID, + &i.WantedAt, + &i.FulfilledAt, + &i.Upgrade, + &i.HeldQuality, + &i.PrimaryType, + &i.SecondaryTypes, + &i.FirstReleaseDate, + &i.RequestedBy, + &i.LibraryArtistID, + &i.Total, + &i.SearchedAt, + &i.SearchOutcome, + &i.SearchConsidered, + &i.BestRelease, + &i.BestVerdict, + &i.SearchError, + &i.GrabID, + &i.GrabName, + &i.GrabbedAt, + &i.GrabOrigin, + &i.GrabQuality, + &i.GrabEndedAt, + &i.GrabEndReason, + &i.GrabProgress, + &i.GrabImportState, + &i.GrabImportReason, + &i.GrabEndDetail, + &i.GrabImportedAt, + &i.GrabImportedFiles, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const markFollowChecked = `-- name: MarkFollowChecked :exec +UPDATE follows + SET checked_at = now(), + baseline_at = COALESCE(baseline_at, now()), + want_existing = false, + updated_at = now() + WHERE artist_mbid = $1 +` + +func (q *Queries) MarkFollowChecked(ctx context.Context, artistMbid string) error { + _, err := q.db.Exec(ctx, markFollowChecked, artistMbid) + return err +} + +const markGroupSeen = `-- name: MarkGroupSeen :exec +INSERT INTO follow_seen_groups (artist_mbid, release_group_mbid) +VALUES ($1, $2) +ON CONFLICT DO NOTHING +` + +type MarkGroupSeenParams struct { + ArtistMbid string + ReleaseGroupMbid string +} + +func (q *Queries) MarkGroupSeen(ctx context.Context, arg MarkGroupSeenParams) error { + _, err := q.db.Exec(ctx, markGroupSeen, arg.ArtistMbid, arg.ReleaseGroupMbid) + return err +} + +const recordAlbumMatch = `-- name: RecordAlbumMatch :exec +INSERT INTO library_album_matches (album_id, outcome, method, release_group_mbid, detail, matcher) +VALUES ($1, $2, $3, $4, $5, + $6) +ON CONFLICT (album_id) DO UPDATE + SET outcome = EXCLUDED.outcome, method = EXCLUDED.method, release_group_mbid = EXCLUDED.release_group_mbid, + detail = EXCLUDED.detail, matcher = EXCLUDED.matcher, decided_at = now() +` + +type RecordAlbumMatchParams struct { + AlbumID pgtype.UUID + Outcome string + Method string + ReleaseGroupMbid string + Detail string + Matcher int32 +} + +func (q *Queries) RecordAlbumMatch(ctx context.Context, arg RecordAlbumMatchParams) error { + _, err := q.db.Exec(ctx, recordAlbumMatch, + arg.AlbumID, + arg.Outcome, + arg.Method, + arg.ReleaseGroupMbid, + arg.Detail, + arg.Matcher, + ) + return err +} + +const recordListedAlbums = `-- name: RecordListedAlbums :exec +INSERT INTO library_album_matches (album_id, outcome, method, release_group_mbid) +SELECT a.id, 'listed', 'tags', a.release_group_mbid + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + WHERE a.release_group_mbid IS NOT NULL AND a.release_group_mbid <> '' + AND ar.mbid IS NOT NULL AND ar.mbid <> '' + AND a.created_at <= $1::timestamptz + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) +ON CONFLICT (album_id) DO NOTHING +` + +// The albums ListLibraryAlbums takes are decided as listed by their tags, +// so the matching that follows leaves them alone. +func (q *Queries) RecordListedAlbums(ctx context.Context, cutoff pgtype.Timestamptz) error { + _, err := q.db.Exec(ctx, recordListedAlbums, cutoff) + return err +} + +const recountLibraryUnlisted = `-- name: RecountLibraryUnlisted :exec +UPDATE acquisition_settings + SET library_unlisted = (SELECT COUNT(*) FROM library_album_matches m + WHERE m.outcome = 'unmatched' + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id)) + WHERE id = true +` + +// After a person matches or leaves off an album, the count the catalogue +// shows. +func (q *Queries) RecountLibraryUnlisted(ctx context.Context) error { + _, err := q.db.Exec(ctx, recountLibraryUnlisted) + return err +} + +const repointWant = `-- name: RepointWant :exec +UPDATE wants SET release_group_mbid = $2, updated_at = now() WHERE id = $1 +` + +type RepointWantParams struct { + ID pgtype.UUID + ReleaseGroupMbid string +} + +// A want recorded under a release id is moved to its group. +func (q *Queries) RepointWant(ctx context.Context, arg RepointWantParams) error { + _, err := q.db.Exec(ctx, repointWant, arg.ID, arg.ReleaseGroupMbid) + return err +} + +const setAlbumMatchOutcome = `-- name: SetAlbumMatchOutcome :execrows +UPDATE library_album_matches SET outcome = $1, decided_at = now() + WHERE album_id = $2 AND outcome = $3 +` + +type SetAlbumMatchOutcomeParams struct { + Outcome string + AlbumID pgtype.UUID + Was string +} + +// Leave an unmatched album off the list, or put a left-off one back. +func (q *Queries) SetAlbumMatchOutcome(ctx context.Context, arg SetAlbumMatchOutcomeParams) (int64, error) { + result, err := q.db.Exec(ctx, setAlbumMatchOutcome, arg.Outcome, arg.AlbumID, arg.Was) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const setAlbumReleaseGroup = `-- name: SetAlbumReleaseGroup :exec +UPDATE albums SET release_group_mbid = $1, updated_at = now() + WHERE id = $2 AND (release_group_mbid IS NULL OR release_group_mbid = '') +` + +type SetAlbumReleaseGroupParams struct { + ReleaseGroupMbid *string + ID pgtype.UUID +} + +// Names a matched album's release group, filling only what nothing named. +func (q *Queries) SetAlbumReleaseGroup(ctx context.Context, arg SetAlbumReleaseGroupParams) error { + _, err := q.db.Exec(ctx, setAlbumReleaseGroup, arg.ReleaseGroupMbid, arg.ID) + return err +} + +const setDefaultReleaseTypeProfile = `-- name: SetDefaultReleaseTypeProfile :execrows +UPDATE release_type_profiles SET is_default = true, updated_at = now() WHERE id = $1 +` + +func (q *Queries) SetDefaultReleaseTypeProfile(ctx context.Context, id pgtype.UUID) (int64, error) { + result, err := q.db.Exec(ctx, setDefaultReleaseTypeProfile, id) + if err != nil { + return 0, err + } + return result.RowsAffected(), nil +} + +const setWantUpgrades = `-- name: SetWantUpgrades :exec +UPDATE wants w + SET upgrade = v.upgrade, + held_tier = v.held_tier, + held_quality = v.held_quality, + upgrade_checked_at = now(), + updated_at = CASE WHEN w.upgrade <> v.upgrade THEN now() ELSE w.updated_at END + FROM (SELECT unnest($1::uuid[]) AS id, + unnest($2::text[]) AS upgrade, + unnest($3::text[]) AS held_tier, + unnest($4::text[]) AS held_quality) v + WHERE w.id = v.id +` + +type SetWantUpgradesParams struct { + Ids []pgtype.UUID + Upgrades []string + HeldTiers []string + HeldQualities []string +} + +// Writes the upgrade check's verdicts, one row per want. +func (q *Queries) SetWantUpgrades(ctx context.Context, arg SetWantUpgradesParams) error { + _, err := q.db.Exec(ctx, setWantUpgrades, + arg.Ids, + arg.Upgrades, + arg.HeldTiers, + arg.HeldQualities, + ) + return err +} + +const startLibraryListing = `-- name: StartLibraryListing :one +UPDATE acquisition_settings + SET library_listing_started_at = COALESCE(library_listing_started_at, now()) + WHERE id = true +RETURNING library_listing_started_at +` + +// Marks the listing begun, once; its time is the cutoff for which albums +// count as the library's. +func (q *Queries) StartLibraryListing(ctx context.Context) (pgtype.Timestamptz, error) { + row := q.db.QueryRow(ctx, startLibraryListing) + var library_listing_started_at pgtype.Timestamptz + err := row.Scan(&library_listing_started_at) + return library_listing_started_at, err +} + +const updateReleaseTypeProfile = `-- name: UpdateReleaseTypeProfile :one +UPDATE release_type_profiles + SET name = $2, + primary_types = $3, + secondary_types = $4, + updated_at = now() + WHERE id = $1 +RETURNING id, name, primary_types, secondary_types, is_default, created_at, updated_at +` + +type UpdateReleaseTypeProfileParams struct { + ID pgtype.UUID + Name string + PrimaryTypes []string + SecondaryTypes []string +} + +func (q *Queries) UpdateReleaseTypeProfile(ctx context.Context, arg UpdateReleaseTypeProfileParams) (ReleaseTypeProfile, error) { + row := q.db.QueryRow(ctx, updateReleaseTypeProfile, + arg.ID, + arg.Name, + arg.PrimaryTypes, + arg.SecondaryTypes, + ) + var i ReleaseTypeProfile + err := row.Scan( + &i.ID, + &i.Name, + &i.PrimaryTypes, + &i.SecondaryTypes, + &i.IsDefault, + &i.CreatedAt, + &i.UpdatedAt, + ) + return i, err +} + +const upsertFollow = `-- name: UpsertFollow :one +INSERT INTO follows (artist_mbid, artist_name, profile_id, followed_by, want_existing) +VALUES ($1, $2, $3, + $4, $5) +ON CONFLICT (artist_mbid) DO UPDATE + SET artist_name = EXCLUDED.artist_name, + profile_id = EXCLUDED.profile_id, + want_existing = follows.want_existing OR EXCLUDED.want_existing, + baseline_at = CASE WHEN EXCLUDED.want_existing THEN NULL ELSE follows.baseline_at END, + updated_at = now() +RETURNING artist_mbid, artist_name, profile_id, followed_by, followed_at, baseline_at, want_existing, checked_at, updated_at +` + +type UpsertFollowParams struct { + ArtistMbid string + ArtistName string + ProfileID pgtype.UUID + FollowedBy pgtype.UUID + WantExisting bool +} + +// Following an artist already followed changes its profile. want_existing +// only ever turns on here; the worker turns it off once applied, and turning +// it on again re-reads the baseline so the existing releases are judged. +func (q *Queries) UpsertFollow(ctx context.Context, arg UpsertFollowParams) (Follow, error) { + row := q.db.QueryRow(ctx, upsertFollow, + arg.ArtistMbid, + arg.ArtistName, + arg.ProfileID, + arg.FollowedBy, + arg.WantExisting, + ) + var i Follow + err := row.Scan( + &i.ArtistMbid, + &i.ArtistName, + &i.ProfileID, + &i.FollowedBy, + &i.FollowedAt, + &i.BaselineAt, + &i.WantExisting, + &i.CheckedAt, + &i.UpdatedAt, + ) + return i, err +} + +const wantHoldings = `-- name: WantHoldings :many +SELECT a.id AS album_id, + COUNT(*) FILTER (WHERE t.missing_since IS NULL)::int AS present, + COALESCE(bool_or(t.missing_since IS NULL AND t.added_at > $1::timestamptz), false)::bool AS arrived_since, + COALESCE(bool_or(t.missing_since IS NOT NULL AND t.missing_since < $1::timestamptz), false)::bool AS missing_before, + -- Missing tracks no track on disk stands in for, by position or + -- title. A missing row whose song is back at a new path is + -- covered; a lost bonus track is not. A track with no disc number is + -- on disc 1, so a missing track 6 with no disc beside a present 1/6 + -- does not keep the album wanted. + COUNT(*) FILTER (WHERE t.missing_since IS NOT NULL AND NOT EXISTS ( + SELECT 1 FROM tracks p + WHERE p.album_id = a.id AND p.missing_since IS NULL + AND ((p.track_number = t.track_number AND COALESCE(p.disc_number, 1) = COALESCE(t.disc_number, 1)) + OR lower(p.title) = lower(t.title))))::int AS uncovered_missing + FROM albums a + JOIN tracks t ON t.album_id = a.id + WHERE a.release_group_mbid = $2::text + OR a.mbid = $2 + OR a.id = $3 + OR a.mbid IN (SELECT r.mbid FROM mb_releases r WHERE r.release_group_mbid = $2) + GROUP BY a.id + ORDER BY present DESC, a.id +` + +type WantHoldingsParams struct { + Since pgtype.Timestamptz + GroupMbid string + AlbumID pgtype.UUID +} + +type WantHoldingsRow struct { + AlbumID pgtype.UUID + Present int32 + ArrivedSince bool + MissingBefore bool + UncoveredMissing int32 +} + +// Each library album that may answer a want, with what it has on disk: +// tracks present, whether one arrived since the want, and whether a track +// missing before the want is still missing. +func (q *Queries) WantHoldings(ctx context.Context, arg WantHoldingsParams) ([]WantHoldingsRow, error) { + rows, err := q.db.Query(ctx, wantHoldings, arg.Since, arg.GroupMbid, arg.AlbumID) + if err != nil { + return nil, err + } + defer rows.Close() + var items []WantHoldingsRow + for rows.Next() { + var i WantHoldingsRow + if err := rows.Scan( + &i.AlbumID, + &i.Present, + &i.ArrivedSince, + &i.MissingBefore, + &i.UncoveredMissing, + ); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + +const wantReleaseGroup = `-- name: WantReleaseGroup :one +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, primary_type, + first_release_date, reason, created_by) +VALUES ($1, $2, $3, $4, + $5, $6, $7, $8) +ON CONFLICT (release_group_mbid) DO UPDATE SET updated_at = wants.updated_at +RETURNING id, release_group_mbid, artist_mbid, artist_name, title, primary_type, first_release_date, reason, album_id, created_by, wanted_at, fulfilled_at, created_at, updated_at, searched_at, search_outcome, search_considered, best_release, best_verdict, search_error, upgrade, held_tier, held_quality, upgrade_checked_at +` + +type WantReleaseGroupParams struct { + ReleaseGroupMbid string + ArtistMbid string + ArtistName string + Title string + PrimaryType string + FirstReleaseDate string + Reason string + CreatedBy pgtype.UUID +} + +// Wants a release group, or returns the want already recorded for it. A +// held want is not reopened here: the library has it. +func (q *Queries) WantReleaseGroup(ctx context.Context, arg WantReleaseGroupParams) (Want, error) { + row := q.db.QueryRow(ctx, wantReleaseGroup, + arg.ReleaseGroupMbid, + arg.ArtistMbid, + arg.ArtistName, + arg.Title, + arg.PrimaryType, + arg.FirstReleaseDate, + arg.Reason, + arg.CreatedBy, + ) + var i Want + err := row.Scan( + &i.ID, + &i.ReleaseGroupMbid, + &i.ArtistMbid, + &i.ArtistName, + &i.Title, + &i.PrimaryType, + &i.FirstReleaseDate, + &i.Reason, + &i.AlbumID, + &i.CreatedBy, + &i.WantedAt, + &i.FulfilledAt, + &i.CreatedAt, + &i.UpdatedAt, + &i.SearchedAt, + &i.SearchOutcome, + &i.SearchConsidered, + &i.BestRelease, + &i.BestVerdict, + &i.SearchError, + &i.Upgrade, + &i.HeldTier, + &i.HeldQuality, + &i.UpgradeCheckedAt, + ) + return i, err +} diff --git a/internal/db/dbq/you_might_like.sql.go b/internal/db/dbq/you_might_like.sql.go index 066e01f6..a389b7e2 100644 --- a/internal/db/dbq/you_might_like.sql.go +++ b/internal/db/dbq/you_might_like.sql.go @@ -26,7 +26,7 @@ type CountListeningSignalForUserRow struct { DistinctArtists int64 } -// "You might like" Home rows (#790). The daily build computes ranked +// "You might like" Home rows. The daily build computes ranked // album/artist IDs in Go (similarity roll-up + cold-start gate) and // atomic-replaces them here; /api/home reads them back hydrated. // Cold-start gate. Returns the breadth of the user's real listening: diff --git a/internal/db/migrations/0002_core_library.up.sql b/internal/db/migrations/0002_core_library.up.sql index f8e0cba8..80cc3fe0 100644 --- a/internal/db/migrations/0002_core_library.up.sql +++ b/internal/db/migrations/0002_core_library.up.sql @@ -1,5 +1,5 @@ --- Core library schema per spec §5. Dual-like, session, scrobble, and --- similarity tables land with their respective feature milestones. +-- Core library schema: artists, albums, tracks and users. Likes, sessions, +-- scrobbling and similarity get tables of their own in later migrations. CREATE EXTENSION IF NOT EXISTS "pgcrypto"; diff --git a/internal/db/migrations/0004_sessions.up.sql b/internal/db/migrations/0004_sessions.up.sql index ebffcece..cac6544a 100644 --- a/internal/db/migrations/0004_sessions.up.sql +++ b/internal/db/migrations/0004_sessions.up.sql @@ -1,10 +1,9 @@ -- Session tokens authenticate /api/* requests. We store sha256(token) so a -- read-only DB leak doesn't grant active sessions; the raw token lives only --- in the client's cookie (web) or bearer header (Flutter). +-- in the client's cookie (web) or bearer header (native clients). -- --- last_seen_at enables an "active sessions" UI later (not wired in this plan) --- without schema churn. user_agent is captured at issue time for the same --- reason — free metadata now, no migration later. +-- last_seen_at and user_agent (captured at issue time) are for an "active +-- sessions" list. CREATE TABLE sessions ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), diff --git a/internal/db/migrations/0005_events.up.sql b/internal/db/migrations/0005_events.up.sql index 8b17dd2f..bcfc20b0 100644 --- a/internal/db/migrations/0005_events.up.sql +++ b/internal/db/migrations/0005_events.up.sql @@ -1,11 +1,10 @@ --- M2 listening telemetry: play_sessions, play_events, skip_events. --- Note the table is named play_sessions (not sessions — that table already +-- Listening telemetry: play_sessions, play_events, skip_events. +-- The table is named play_sessions (not sessions — that table already -- exists from migration 0004 and holds HTTP auth sessions). Listening --- sessions are an entirely separate concept driven by spec §6. +-- sessions are an entirely separate concept. -- --- session_vector_at_play and contextual_likes ship as nullable now even --- though M2 writes NULL; M3 will populate them without a follow-up --- migration. +-- session_vector_at_play is nullable: plays are recorded before the +-- recommendation engine fills it. CREATE TABLE play_sessions ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), diff --git a/internal/db/migrations/0006_likes.up.sql b/internal/db/migrations/0006_likes.up.sql index fdbf33c8..9f34df31 100644 --- a/internal/db/migrations/0006_likes.up.sql +++ b/internal/db/migrations/0006_likes.up.sql @@ -1,6 +1,5 @@ --- M2 likes: three tables, one per entity type. Schema follows the spec §5 --- pattern (general_likes was originally track-only); this slice extends it --- to albums and artists for full Subsonic star/unstar support. +-- Likes: three tables, one per entity type. general_likes was track-only; +-- albums and artists get the same shape for Subsonic star/unstar. CREATE TABLE general_likes ( user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, diff --git a/internal/db/migrations/0007_contextual_likes.up.sql b/internal/db/migrations/0007_contextual_likes.up.sql index 93882b5d..69f14ae9 100644 --- a/internal/db/migrations/0007_contextual_likes.up.sql +++ b/internal/db/migrations/0007_contextual_likes.up.sql @@ -1,16 +1,12 @@ --- contextual_likes captures a per-session-context snapshot at the time of --- each like. The recommendation engine in M3 sub-plan #3 uses these rows --- to compute contextual_match_score per (current session, candidate track). +-- contextual_likes captures a per-session-context snapshot at the time of each +-- like. The recommendation engine uses these rows to compute +-- contextual_match_score per (current session, candidate track). -- -- Multiple rows per (user, track) are allowed and expected — each row -- represents a like in a specific session context. Soft-deleted rows -- remain in the table (deleted_at populated) but are filtered out of the -- engine's queries via the partial index. Re-liking a previously-unliked -- track adds a NEW row (does not undelete the old one). --- --- Note: the M2 events migration (0005) commented that contextual_likes --- "ships nullable now" but the actual CREATE TABLE was missed. This slice --- ships the table for the first time. CREATE TABLE contextual_likes ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), @@ -27,6 +23,6 @@ CREATE INDEX contextual_likes_active_idx ON contextual_likes (user_id, track_id) WHERE deleted_at IS NULL; --- Vector similarity queries from M3 sub-plan #3 will use this. +-- For the engine's vector similarity queries. CREATE INDEX contextual_likes_vector_idx ON contextual_likes USING gin (session_vector); diff --git a/internal/db/migrations/0008_scrobble.up.sql b/internal/db/migrations/0008_scrobble.up.sql index 536d268f..32f3e15a 100644 --- a/internal/db/migrations/0008_scrobble.up.sql +++ b/internal/db/migrations/0008_scrobble.up.sql @@ -1,8 +1,8 @@ --- M4a: outbound ListenBrainz scrobble worker. +-- Outbound ListenBrainz scrobble worker. -- Per-user LB config (plaintext token; users rotate via /settings if leaked). -- scrobble_queue is a work list, not a log: successfully-sent rows are --- DELETEd by the worker (the canonical record stays in play_events.scrobbled_at --- via M2's column). Only `pending` and `failed` states exist. +-- DELETEd by the worker (the canonical record stays in +-- play_events.scrobbled_at). Only `pending` and `failed` states exist. ALTER TABLE users ADD COLUMN listenbrainz_token TEXT NULL, diff --git a/internal/db/migrations/0009_similarity.up.sql b/internal/db/migrations/0009_similarity.up.sql index 8110a2bd..382631c2 100644 --- a/internal/db/migrations/0009_similarity.up.sql +++ b/internal/db/migrations/0009_similarity.up.sql @@ -1,4 +1,4 @@ --- M4b: ListenBrainz similarity ingest. Two symmetric tables — tracks and +-- ListenBrainz similarity ingest. Two symmetric tables — tracks and -- artists — both keyed by (a_id, b_id, source) so the schema reserves room -- for additional similarity sources (musicbrainz_tag, user_cooccurrence) -- alongside listenbrainz. M4b only writes 'listenbrainz' rows. diff --git a/internal/db/migrations/0010_lidarr.up.sql b/internal/db/migrations/0010_lidarr.up.sql index 8efb7b03..b2586ed1 100644 --- a/internal/db/migrations/0010_lidarr.up.sql +++ b/internal/db/migrations/0010_lidarr.up.sql @@ -1,6 +1,6 @@ --- M5a: Lidarr integration foundation. Two tables: +-- Lidarr integration. Two tables: -- --- lidarr_config — singleton (CHECK id=1) holding the operator's Lidarr +-- lidarr_config — singleton (CHECK id=1) holding the Lidarr -- connection. enabled=false is the unconfigured state. -- -- lidarr_requests — per-request lifecycle row created by users at diff --git a/internal/db/migrations/0011_lidarr_quarantine.up.sql b/internal/db/migrations/0011_lidarr_quarantine.up.sql index 8ff73259..1d2443c9 100644 --- a/internal/db/migrations/0011_lidarr_quarantine.up.sql +++ b/internal/db/migrations/0011_lidarr_quarantine.up.sql @@ -1,4 +1,4 @@ --- M5b: per-user track quarantines + admin action audit log. +-- Per-user track quarantines + admin action audit log. -- -- lidarr_quarantine — one row per (user, track) complaint. PK matches -- the general_likes pattern. Re-flagging the same track upserts. Deleted diff --git a/internal/db/migrations/0012_artist_similarity_unmatched.up.sql b/internal/db/migrations/0012_artist_similarity_unmatched.up.sql index 0d7cf1cb..b89b7bb1 100644 --- a/internal/db/migrations/0012_artist_similarity_unmatched.up.sql +++ b/internal/db/migrations/0012_artist_similarity_unmatched.up.sql @@ -1,4 +1,4 @@ --- M5c: persist unmatched-similar-artist MBIDs that the M4b worker would +-- Persist unmatched-similar-artist MBIDs that the M4b worker would -- otherwise discard. Mirrors artist_similarity shape: same composite PK -- with source, same (seed_id, score DESC) index, same source enum check. -- The candidate side is text + name (no FK) — that's the whole point. diff --git a/internal/db/migrations/0013_lidarr_metadata_profile.up.sql b/internal/db/migrations/0013_lidarr_metadata_profile.up.sql index 97ac5913..627d46f9 100644 --- a/internal/db/migrations/0013_lidarr_metadata_profile.up.sql +++ b/internal/db/migrations/0013_lidarr_metadata_profile.up.sql @@ -1,9 +1,8 @@ --- M6a follow-up: persist the operator's metadata-profile choice alongside --- quality profile + root folder. Lidarr POST /api/v1/artist requires --- metadataProfileId; without storing it we'd refetch and pick first on --- every Approve, denying the operator a choice. Nullable: existing rows --- (and operators upgrading from before this migration) get NULL and the --- service falls back to "fetch list and pick first" until they save. +-- Persist the chosen Lidarr metadata profile alongside quality profile + +-- root folder. Lidarr POST /api/v1/artist requires metadataProfileId; +-- without storing it every Approve would refetch and pick the first. +-- Nullable: existing rows get NULL and the service falls back to "fetch +-- list and pick first" until an admin saves. ALTER TABLE lidarr_config ADD COLUMN default_metadata_profile_id integer; diff --git a/internal/db/migrations/0014_playlists.up.sql b/internal/db/migrations/0014_playlists.up.sql index 21642891..6cf9d825 100644 --- a/internal/db/migrations/0014_playlists.up.sql +++ b/internal/db/migrations/0014_playlists.up.sql @@ -1,5 +1,4 @@ --- M7 #352 slice 1: playlists CRUD foundation. --- See docs/superpowers/specs/2026-05-03-m7-playlists-crud-design.md. +-- User playlists. CREATE TABLE playlists ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), diff --git a/internal/db/migrations/0015_system_playlists.up.sql b/internal/db/migrations/0015_system_playlists.up.sql index 32eb016f..7e138ec5 100644 --- a/internal/db/migrations/0015_system_playlists.up.sql +++ b/internal/db/migrations/0015_system_playlists.up.sql @@ -1,7 +1,7 @@ --- M7 #352 slice 2: system-generated playlists. +-- System-generated playlists. -- Adds kind/system_variant/seed_artist_id columns to playlists and a --- per-user run-tracking table. Backfills existing slice-1 playlists --- to kind='user'. +-- per-user run-tracking table. Backfills existing playlists to +-- kind='user'. -- Discriminator on existing playlists table. ALTER TABLE playlists diff --git a/internal/db/migrations/0016_album_cover_source.up.sql b/internal/db/migrations/0016_album_cover_source.up.sql index ff2f718f..b198c989 100644 --- a/internal/db/migrations/0016_album_cover_source.up.sql +++ b/internal/db/migrations/0016_album_cover_source.up.sql @@ -1,5 +1,5 @@ --- M7 #353: album cover origin attribution. --- 'embedded' — extracted from audio file (reserved for future slice). +-- Album cover origin attribution. +-- 'embedded' — extracted from audio file (reserved). -- 'sidecar' — found in the album directory (cover.jpg/folder.jpg/etc). -- 'mbcaa' — fetched from MusicBrainz Cover Art Archive. -- 'none' — tried, no cover available. Skip auto-retry; only manual. diff --git a/internal/db/migrations/0017_scan_runs.up.sql b/internal/db/migrations/0017_scan_runs.up.sql index e17f4048..ed3a528b 100644 --- a/internal/db/migrations/0017_scan_runs.up.sql +++ b/internal/db/migrations/0017_scan_runs.up.sql @@ -1,4 +1,4 @@ --- M7 #381: track library-scan / MBID-backfill / cover-enrich runs so the +-- Track library-scan / MBID-backfill / cover-enrich runs so the -- admin overview can show last-run results + manual trigger state. CREATE TABLE scan_runs ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), diff --git a/internal/db/migrations/0018_artist_art_and_provider_settings.up.sql b/internal/db/migrations/0018_artist_art_and_provider_settings.up.sql index 37dd3203..b1ba95ba 100644 --- a/internal/db/migrations/0018_artist_art_and_provider_settings.up.sql +++ b/internal/db/migrations/0018_artist_art_and_provider_settings.up.sql @@ -1,6 +1,5 @@ --- M7 cover-sources: schema for additional cover-art providers (TheAudioDB --- as the v1 implementation), artist art (thumb + fanart), and the --- recheck-on-source-change trigger. +-- Schema for additional cover-art providers (TheAudioDB first), artist art +-- (thumb + fanart), and the recheck-on-source-change trigger. -- (0) Extend the existing albums.cover_art_source CHECK to accept the -- new 'theaudiodb' value. The constraint was created in migration 0016 diff --git a/internal/db/migrations/0019_scan_schedule.up.sql b/internal/db/migrations/0019_scan_schedule.up.sql index 5d2b2e58..0d872edb 100644 --- a/internal/db/migrations/0019_scan_schedule.up.sql +++ b/internal/db/migrations/0019_scan_schedule.up.sql @@ -1,4 +1,4 @@ --- M7 recurring-scans: singleton table holding the operator's schedule +-- Recurring scans: singleton table holding the scan schedule -- config. mode is the discriminator: 'off' / 'interval' / 'daily' / 'weekly'. -- Per-mode fields are populated only when their mode is selected; others -- stay NULL. weekly_day uses ISO 8601: 1=Monday … 7=Sunday. diff --git a/internal/db/migrations/0020_relax_art_source_checks.up.sql b/internal/db/migrations/0020_relax_art_source_checks.up.sql index 272dfcc5..b663e06e 100644 --- a/internal/db/migrations/0020_relax_art_source_checks.up.sql +++ b/internal/db/migrations/0020_relax_art_source_checks.up.sql @@ -1,5 +1,5 @@ -- Relax cover-art and artist-art source CHECK constraints so the new --- Deezer and Last.fm providers (next slice) can persist their IDs as +-- Deezer and Last.fm providers can persist their IDs as -- accepted source values. Also add the -- last_registered_providers_hash column to cover_art_sources_meta for -- boot-time auto-bump detection (when the registered-provider set diff --git a/internal/db/migrations/0022_user_management.up.sql b/internal/db/migrations/0022_user_management.up.sql index f371d5b4..434a951d 100644 --- a/internal/db/migrations/0022_user_management.up.sql +++ b/internal/db/migrations/0022_user_management.up.sql @@ -1,14 +1,14 @@ --- User management foundation (U1 of #376): +-- User management: -- -- - users.display_name: optional human-readable name distinct from username. -- - user_invites: admin-generated tokens; user redeems during signup. -- - registration_settings (singleton): controls whether registration is -- 'invite_only' (default after first admin) or 'open' (admin can --- re-enable). The handler in U1-T2 also has a special-case for the +-- re-enable). The registration handler also has a special-case for the -- "no users yet" state where it ignores invites and lets the first -- register as admin. -- - audit_log: append-only record of admin-driven user mutations. --- No retention policy in v1. +-- No retention policy. ALTER TABLE users ADD COLUMN IF NOT EXISTS display_name text; diff --git a/internal/db/migrations/0023_user_auto_approve.up.sql b/internal/db/migrations/0023_user_auto_approve.up.sql index 9dadaae3..76f0b333 100644 --- a/internal/db/migrations/0023_user_auto_approve.up.sql +++ b/internal/db/migrations/0023_user_auto_approve.up.sql @@ -1,7 +1,5 @@ --- Adds users.auto_approve_requests for the per-user auto-approve toggle --- (#355 sub-feature surface). The handler that honors this flag in --- the request submission flow is sibling work (U2.5 follow-up); this --- migration just lands the schema + the column default. +-- Adds users.auto_approve_requests for the per-user auto-approve toggle: +-- the request submission flow approves that user's requests at once. -- -- Default false: existing users keep manual approval, opt-in via the -- admin user-management UI. diff --git a/internal/db/migrations/0025_library_changes.up.sql b/internal/db/migrations/0025_library_changes.up.sql index 770c2c8a..f4252cc4 100644 --- a/internal/db/migrations/0025_library_changes.up.sql +++ b/internal/db/migrations/0025_library_changes.up.sql @@ -1,13 +1,10 @@ -- Append-only change log for library entities. Every mutation on -- artists/albums/tracks/likes/playlists/playlist_tracks writes a row --- in the same transaction as the mutation itself. Powers the Flutter --- delta-sync endpoint (#357) — clients pass back the last-seen `id` --- as a cursor. +-- in the same transaction as the mutation itself. Powers the delta-sync +-- endpoint — clients pass back the last-seen `id` as a cursor. -- --- Retention: not capped here. If the table grows unbounded a follow-up --- adds a periodic compactor (collapse contiguous upserts on the same --- entity, drop pre-delete history). Cursor reset path (410) covers the --- compactor's edge. +-- Retention: not capped here. A cursor older than what the table keeps gets +-- the reset path (410). CREATE TABLE library_changes ( id BIGSERIAL PRIMARY KEY, diff --git a/internal/db/migrations/0026_users_timezone.up.sql b/internal/db/migrations/0026_users_timezone.up.sql index 52edc73d..ec14c39f 100644 --- a/internal/db/migrations/0026_users_timezone.up.sql +++ b/internal/db/migrations/0026_users_timezone.up.sql @@ -1,5 +1,5 @@ -- 0026_users_timezone.up.sql — per-user timezone for the system-playlist --- scheduler (#392 Half B). Default 'UTC' so existing rows + brand-new +-- scheduler. Default 'UTC' so existing rows + brand-new -- users have a safe value before their first client check-in via -- PUT /api/me/timezone. The scheduler reads this column to decide -- which IANA zone to anchor each user's daily 03:00 build to. diff --git a/internal/db/migrations/0027_system_playlist_rotation.up.sql b/internal/db/migrations/0027_system_playlist_rotation.up.sql index 3314a4ea..5c604801 100644 --- a/internal/db/migrations/0027_system_playlist_rotation.up.sql +++ b/internal/db/migrations/0027_system_playlist_rotation.up.sql @@ -1,5 +1,5 @@ -- 0027_system_playlist_rotation.up.sql — sample-history dedup for --- system playlists (Fable #415). Two parts: +-- system playlists. Two parts: -- -- 1. play_events.source: which system playlist a play originated -- from ('for_you' | 'discover' | future kinds), or NULL for any diff --git a/internal/db/migrations/0028_discovery_mix_variants.up.sql b/internal/db/migrations/0028_discovery_mix_variants.up.sql index b50d92fb..e2eee325 100644 --- a/internal/db/migrations/0028_discovery_mix_variants.up.sql +++ b/internal/db/migrations/0028_discovery_mix_variants.up.sql @@ -1,11 +1,9 @@ --- #411 R3 bugfix: the five discovery mixes (deep_cuts, rediscover, --- new_for_you, on_this_day, first_listens) were added as producers + --- registry entries but their system_variant values were never added --- to the two playlists CHECK constraints. Result: insertSystemPlaylist --- for any new mix fails with playlists_kind_variant_consistent --- (SQLSTATE 23514), which — because BuildSystemPlaylists is one --- all-or-nothing transaction — aborts the WHOLE build, so For-You / --- Discover refresh 500s and the lazy daily build fails too. +-- Before: the five discovery mixes (deep_cuts, rediscover, new_for_you, +-- on_this_day, first_listens) had producers and registry entries but no +-- system_variant values in the two playlists CHECK constraints, so inserting +-- any of them failed playlists_kind_variant_consistent (SQLSTATE 23514) — +-- and because BuildSystemPlaylists is one all-or-nothing transaction, that +-- aborts the WHOLE build. After: the constraints accept them. -- -- All five are seedless (like for_you/discover): seed_artist_id IS -- NULL. Drop + re-add both constraints with the expanded set, same diff --git a/internal/db/migrations/0030_relax_art_source_check_to_nonempty.up.sql b/internal/db/migrations/0030_relax_art_source_check_to_nonempty.up.sql index 35ca702c..7b6dd3fd 100644 --- a/internal/db/migrations/0030_relax_art_source_check_to_nonempty.up.sql +++ b/internal/db/migrations/0030_relax_art_source_check_to_nonempty.up.sql @@ -2,9 +2,8 @@ -- provider's registry ID. coverart.Register makes the provider set -- extensible, so a fixed IN-list CHECK (0016 → 0018 → 0020) required a -- schema migration for every new provider and rejected any --- out-of-list value (e.g. test stub providers). Same brittleness --- class as the discovery-mix CHECK incident (#433): a fixed-value --- CHECK fighting an extensible value set. Relax to "NULL or any +-- out-of-list value (e.g. test stub providers): a fixed-value CHECK +-- fighting an extensible value set. Relax to "NULL or any -- non-empty string" — the registry, not the schema, is the source of -- truth for valid provider IDs. diff --git a/internal/db/migrations/0031_lidarr_add_confirmed_at.up.sql b/internal/db/migrations/0031_lidarr_add_confirmed_at.up.sql index 37b05a72..b5468d98 100644 --- a/internal/db/migrations/0031_lidarr_add_confirmed_at.up.sql +++ b/internal/db/migrations/0031_lidarr_add_confirmed_at.up.sql @@ -2,7 +2,7 @@ -- for an approved request. NULL = not yet sent/confirmed; the -- reconciler idempotently (re)sends the add until this is set, then -- watches for the import. Decouples the durable admin approval from the --- best-effort Lidarr call so a transient Lidarr outage no longer forces --- a manual re-approve. +-- best-effort Lidarr call, so a transient Lidarr outage does not force a +-- manual re-approve. ALTER TABLE lidarr_requests ADD COLUMN lidarr_add_confirmed_at timestamptz; diff --git a/internal/db/migrations/0032_playback_errors.up.sql b/internal/db/migrations/0032_playback_errors.up.sql index 5f4b8d7e..12de900c 100644 --- a/internal/db/migrations/0032_playback_errors.up.sql +++ b/internal/db/migrations/0032_playback_errors.up.sql @@ -1,6 +1,6 @@ -- Client-reported playback errors. Populated by clients when a track -- fails to play (zero duration, decode error, etc.); surfaced in the --- admin /admin/playback-errors inbox so the operator can hide / delete +-- admin /admin/playback-errors inbox so an admin can hide / delete -- / re-request the offending track. -- -- resolved_at + resolved_by + resolution are NULL until an admin acts @@ -10,8 +10,8 @@ -- unresolved-queue lookup fast even as resolved history accumulates. -- -- CHECK constraints on kind/resolution gate the enum values so client --- typos can't pollute the data (per the standing rule about --- enum-CHECK whitelists needing migrations). +-- typos can't pollute the data; a new value means swapping the constraint +-- in the same migration as the code that writes it. CREATE TABLE playback_errors ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), diff --git a/internal/db/migrations/0033_drop_scan_schedule.up.sql b/internal/db/migrations/0033_drop_scan_schedule.up.sql index ffaecb1e..346ed9ca 100644 --- a/internal/db/migrations/0033_drop_scan_schedule.up.sql +++ b/internal/db/migrations/0033_drop_scan_schedule.up.sql @@ -1,4 +1,4 @@ -- Retire the configurable scan scheduler. New-file pickup is now handled by -- the fsnotify watcher + a hardcoded safety-net delta walk (cmd/minstrel), --- so the operator-configurable schedule (and its admin UI) is gone. +-- so the configurable schedule (and its admin UI) is gone. DROP TABLE IF EXISTS scan_schedule; diff --git a/internal/db/migrations/0034_you_might_like.up.sql b/internal/db/migrations/0034_you_might_like.up.sql index dc9cf162..6be652a2 100644 --- a/internal/db/migrations/0034_you_might_like.up.sql +++ b/internal/db/migrations/0034_you_might_like.up.sql @@ -1,4 +1,4 @@ --- 0034_you_might_like.up.sql — "You might like" Home rows (#790). +-- 0034_you_might_like.up.sql — "You might like" Home rows. -- -- Per-user ranked lists of IN-LIBRARY albums/artists the listener -- doesn't actively spin but is predicted to enjoy, derived from the diff --git a/internal/db/migrations/0035_taste_profile.up.sql b/internal/db/migrations/0035_taste_profile.up.sql index 595375a3..47084fa5 100644 --- a/internal/db/migrations/0035_taste_profile.up.sql +++ b/internal/db/migrations/0035_taste_profile.up.sql @@ -1,11 +1,11 @@ --- 0035_taste_profile.up.sql — per-user taste profile (#796 phase 1). +-- 0035_taste_profile.up.sql — per-user taste profile. -- -- A persistent, decaying model of each user's taste, rebuilt daily from -- graded play engagement (completion + skip position) plus explicit -- likes. Two facet tables: weighted artists and weighted genre tags. -- Weight is a signed float — positive = drawn to, negative = (passively) --- avoided; magnitude reflects decayed engagement. Phase 1 only BUILDS --- these; consumption (candidate sourcing + scoring) lands in phase 2. +-- avoided; magnitude reflects decayed engagement. Candidate sourcing and +-- scoring read them. -- -- CASCADE on user/artist deletes keeps them clean. Indexed by -- (user_id, weight DESC) to serve "top taste artists/tags" reads cheaply. diff --git a/internal/db/migrations/0036_diagnostics.up.sql b/internal/db/migrations/0036_diagnostics.up.sql index 37876893..afcab288 100644 --- a/internal/db/migrations/0036_diagnostics.up.sql +++ b/internal/db/migrations/0036_diagnostics.up.sql @@ -1,4 +1,4 @@ --- Device diagnostics / debug-reporting subsystem (M9). +-- Device diagnostics / debug-reporting subsystem. -- -- When an account has users.debug_mode_enabled = true, that account's -- client(s) stream a timeseries of device state here: connectivity + @@ -8,12 +8,12 @@ -- /admin/diagnostics view filters by account + device + time window and -- exports the slice as JSON for offline analysis. -- --- `kind` is a COARSE category gated by a CHECK whitelist; the finer --- event discriminator + all event-specific fields live in the `payload` --- jsonb. This keeps the enum small and stable so new event variants --- don't require a migration (per the enum-CHECK-whitelist rule, only a --- new *category* would). If you add a category here, mirror it in the --- handler whitelist in internal/api/diagnostics.go. +-- `kind` is a COARSE category gated by a CHECK whitelist; the finer event +-- discriminator + all event-specific fields live in the `payload` jsonb. This +-- keeps the enum small and stable so new event variants don't require a +-- migration (only a new *category* would: a new CHECK value means swapping the +-- constraint in the same migration). If you add a category here, mirror it in +-- the handler whitelist in internal/api/diagnostics.go. -- -- Two timestamps on purpose: occurred_at is the client device clock -- (when the event happened on the phone — may be skewed/wrong, e.g. in diff --git a/internal/db/migrations/0037_diagnostics_playback_kind.up.sql b/internal/db/migrations/0037_diagnostics_playback_kind.up.sql index 722d1dd6..290ee688 100644 --- a/internal/db/migrations/0037_diagnostics_playback_kind.up.sql +++ b/internal/db/migrations/0037_diagnostics_playback_kind.up.sql @@ -1,14 +1,14 @@ --- Add 'playback' to the diagnostic_events kind whitelist (M9 follow-up). +-- Add 'playback' to the diagnostic_events kind whitelist. -- -- `route` and `player_state` events fire for EVERY output route (phone -- speaker, Bluetooth, UPnP/Sonos), so bucketing them under 'upnp_sync' --- was misleading — an operator on Bluetooth earbuds saw "upnp_sync" +-- was misleading — a listener on Bluetooth earbuds produced "upnp_sync" -- rows. 'upnp_sync' now means genuinely UPnP/Sonos-specific signal -- (drops, and future resync/desync); general playback + route telemetry -- moves to the new 'playback' kind. -- --- Per the enum-CHECK-whitelist rule, the new value lands by dropping and --- re-adding the constraint in the same change. Existing rows tagged +-- The new value lands by dropping and re-adding the constraint in the same +-- migration. Existing rows tagged -- 'upnp_sync' stay as-is (debug telemetry on a 30-day retention). ALTER TABLE diagnostic_events DROP CONSTRAINT diagnostic_events_kind_check; diff --git a/internal/db/migrations/0038_for_you_pick_kind.up.sql b/internal/db/migrations/0038_for_you_pick_kind.up.sql index fdf86a39..29b582f6 100644 --- a/internal/db/migrations/0038_for_you_pick_kind.up.sql +++ b/internal/db/migrations/0038_for_you_pick_kind.up.sql @@ -1,9 +1,9 @@ --- For You exploration attribution (milestone #127 step 2, Scribe #1249). +-- For You exploration attribution. -- -- For You is deliberately composed of two populations: a head of -- top-scored similarity matches ("taste" picks) and a tail sampled from -- deeper in the candidate ranking ("fresh" picks — the freshness --- injection). Until now the metrics could only judge For You as one +-- injection). Before this the metrics could only judge For You as one -- blob, so a high skip rate couldn't be read as "the taste engine is -- missing" vs "the freshness tax is too high". -- diff --git a/internal/db/migrations/0039_pick_kind_vocabulary.up.sql b/internal/db/migrations/0039_pick_kind_vocabulary.up.sql index 36f619d2..c1502d73 100644 --- a/internal/db/migrations/0039_pick_kind_vocabulary.up.sql +++ b/internal/db/migrations/0039_pick_kind_vocabulary.up.sql @@ -1,6 +1,6 @@ --- Provenance as standard (milestone #127, Scribe #1270). +-- Provenance as standard. -- --- #1249 introduced pick_kind as a For You one-off ('taste'/'fresh'). +-- 0038 introduced pick_kind as a For You one-off ('taste'/'fresh'). -- The mechanism — stamp WHY a track is in the snapshot at build time, -- freeze it onto the play at ingestion, break it down in metrics — is -- now the standard for every system mix, so the CHECK vocabulary @@ -11,9 +11,9 @@ -- Discover's three candidate buckets — makes -- the 40/30/30 allocation measurable instead -- of a guess --- tier1 / tier2 / tier3 the tiered-eligibility ladder (project rule --- #131): tier 1 pins the mix's exact desire, --- each higher tier steps back a little. Tier +-- tier1 / tier2 / tier3 the tiered-eligibility ladder: tier 1 pins the +-- mix's exact desire, each higher tier steps +-- back a little. Tier -- provenance is how we measure what the -- step-back trades away. -- diff --git a/internal/db/migrations/0040_recommendation_tuning.up.sql b/internal/db/migrations/0040_recommendation_tuning.up.sql index 7aba7c30..3d3705d0 100644 --- a/internal/db/migrations/0040_recommendation_tuning.up.sql +++ b/internal/db/migrations/0040_recommendation_tuning.up.sql @@ -1,11 +1,10 @@ --- Recommendation tuning lab (milestone #127, Scribe #1250). +-- Recommendation tuning lab. -- -- The scoring weights move out of YAML (radio profile) and out of the -- systemMixWeights hard-code (daily_mix profile) into DB-backed --- settings with live effect — the defaults-discovery lab (decision --- #1247): the operator turns knobs here to FIND good values, which --- then get baked into shipped defaults; end users and other operators --- should never need this card. +-- settings with live effect — a lab for FINDING good values, which then +-- get baked into shipped defaults; nobody should need this card to get +-- good recommendations. -- -- Rows are seeded by the recsettings service's boot reconcile (the -- coverart SettingsService pattern), not by this migration, so shipped @@ -39,7 +38,7 @@ CREATE TABLE taste_tuning ( ); -- Every knob turn writes one audit row; the metrics trend view --- (#1251) annotates these on its timeline so cause→effect is visible +-- annotates these on its timeline so cause→effect is visible -- after a change. changes is a jsonb array of {field, old, new}. CREATE TABLE recommendation_tuning_audit ( id bigserial PRIMARY KEY, diff --git a/internal/db/migrations/0041_discover_taste_unheard.up.sql b/internal/db/migrations/0041_discover_taste_unheard.up.sql index c2cee8bd..4f8934d1 100644 --- a/internal/db/migrations/0041_discover_taste_unheard.up.sql +++ b/internal/db/migrations/0041_discover_taste_unheard.up.sql @@ -1,16 +1,13 @@ --- Discover taste-targeted novelty bucket (milestone #127, Scribe #1252/#1488). +-- Discover taste-targeted novelty bucket. -- --- The 2-week metrics review found Discover beating the manual baseline on --- skip rate — which for a discovery surface means it is playing it safe: --- on a single-user server it is effectively dormant + crude-random, and --- the per-user taste profile (n_tags, #796) went unused. This adds a +-- Without it, Discover on a single-user server is effectively dormant + +-- crude-random, and the per-user taste profile's tags go unused. This adds a -- fourth Discover candidate bucket that ranks unheard tracks by the -- user's taste-profile tag weights ("novelty that fits your vibe"), so -- its picks need a new pick_kind provenance value. -- -- Postgres CHECK whitelists can't be altered in place: DROP + re-ADD with --- the expanded list, in the same migration (family rule — new CHECK-enum --- value needs a same-change migration). +-- the expanded list, in the same migration. ALTER TABLE playlist_tracks DROP CONSTRAINT playlist_tracks_pick_kind_check; ALTER TABLE playlist_tracks diff --git a/internal/db/migrations/0042_track_tags.up.sql b/internal/db/migrations/0042_track_tags.up.sql index 212da925..8c49156d 100644 --- a/internal/db/migrations/0042_track_tags.up.sql +++ b/internal/db/migrations/0042_track_tags.up.sql @@ -1,4 +1,4 @@ --- Track-level folksonomy tags for taste enrichment (milestone #160, #1490). +-- Track-level folksonomy tags for taste enrichment. -- -- A global (not per-user) cache of style/mood tags fetched from -- MusicBrainz (keyless) + Last.fm (when a key is configured), keyed by diff --git a/internal/db/migrations/0043_tag_provider_settings.up.sql b/internal/db/migrations/0043_tag_provider_settings.up.sql index a9438bf1..f7a87cd8 100644 --- a/internal/db/migrations/0043_tag_provider_settings.up.sql +++ b/internal/db/migrations/0043_tag_provider_settings.up.sql @@ -1,4 +1,4 @@ --- Tag-enrichment provider settings (milestone #160, #1490 Step 2). +-- Tag-enrichment provider settings. -- -- Mirrors the cover-art provider-settings substrate (migration 0018) so -- adding a new tag source later is "register a provider + it auto-upserts @@ -28,8 +28,9 @@ CREATE TABLE tag_sources_meta ( INSERT INTO tag_sources_meta (id, current_version) VALUES (true, 1); -- Seed the v1 providers. MusicBrainz is the keyless always-on default; --- Last.fm ships disabled and is opt-in once the operator supplies a key --- (Step 4 Settings UI), per the no-coercive-settings rule (#26). The +-- Last.fm ships disabled and is opt-in once an admin supplies a key in +-- Settings: a feature that needs a third-party account is never on by +-- default. The -- SettingsService also upserts these at boot, so the seed is belt-and- -- suspenders / documents the intended defaults. INSERT INTO tag_provider_settings (provider_id, enabled, display_order) VALUES diff --git a/internal/db/migrations/0044_taste_tuning_enriched_tag_scale.up.sql b/internal/db/migrations/0044_taste_tuning_enriched_tag_scale.up.sql index 4b605c32..c02818b4 100644 --- a/internal/db/migrations/0044_taste_tuning_enriched_tag_scale.up.sql +++ b/internal/db/migrations/0044_taste_tuning_enriched_tag_scale.up.sql @@ -1,6 +1,6 @@ --- Expose EnrichedTagScale in the tuning lab (#1520). Adds the enriched-tag --- weight knob to the taste_tuning singleton so operators can dial how much --- folksonomy tags (#1490) count vs raw ID3 genre. DEFAULT 0.5 backfills the +-- Expose EnrichedTagScale in the tuning lab. Adds the enriched-tag +-- weight knob to the taste_tuning singleton so an admin can dial how much +-- folksonomy tags count vs raw ID3 genre. DEFAULT 0.5 backfills the -- existing row and matches taste.DefaultConfig().EnrichedTagScale. ALTER TABLE taste_tuning ADD COLUMN enriched_tag_scale double precision NOT NULL DEFAULT 0.5; diff --git a/internal/db/migrations/0045_taste_profile_eras.up.sql b/internal/db/migrations/0045_taste_profile_eras.up.sql index 54acd866..b4cdbfe1 100644 --- a/internal/db/migrations/0045_taste_profile_eras.up.sql +++ b/internal/db/migrations/0045_taste_profile_eras.up.sql @@ -1,5 +1,5 @@ --- 0045_taste_profile_eras.up.sql — era/decade taste facet (#1530, --- milestone #160 Opt 2). A third taste facet alongside artists + tags: +-- 0045_taste_profile_eras.up.sql — era/decade taste facet. A third taste +-- facet alongside artists + tags: -- signed weights over decade buckets ("1990s") derived from -- albums.release_date, rebuilt daily by internal/taste next to the -- artist/tag facets. Weight is a signed float — positive = drawn to @@ -18,8 +18,8 @@ CREATE TABLE taste_profile_eras ( CREATE INDEX taste_profile_eras_user_weight_idx ON taste_profile_eras (user_id, weight DESC); --- Era-facet build knob, tunable in the lab (mirrors enriched_tag_scale, --- #1520): scales how strongly a decade-play imprints on the profile. +-- Era-facet build knob, tunable in the lab (mirrors enriched_tag_scale): +-- scales how strongly a decade-play imprints on the profile. -- DEFAULT 0.5 backfills the existing singleton and matches -- taste.DefaultConfig().EraScale; 0 disables the era facet entirely. ALTER TABLE taste_tuning diff --git a/internal/db/migrations/0046_context_time_weight.up.sql b/internal/db/migrations/0046_context_time_weight.up.sql index 6b7b7388..f592279a 100644 --- a/internal/db/migrations/0046_context_time_weight.up.sql +++ b/internal/db/migrations/0046_context_time_weight.up.sql @@ -1,5 +1,5 @@ --- 0046_context_time_weight.up.sql — time-of-day/weekday context conditioning --- (#1531, milestone #160 Opt 3). Adds a per-profile scoring weight for the new +-- 0046_context_time_weight.up.sql — time-of-day/weekday context conditioning. +-- Adds a per-profile scoring weight for the new -- context-affinity term: how strongly a candidate is boosted when its artist's -- play history concentrates in the current daypart × weekday-type cell (in the -- user's local timezone). Mirrors the other ScoringWeights columns. @@ -7,6 +7,6 @@ -- DEFAULT 1.0 backfills both existing profile rows to a modest on-value; the -- Go ShippedRadioWeights/ShippedDailyMixWeights carry the same 1.0 so fresh -- installs seed identically. Reconcile only seeds MISSING rows (ON CONFLICT DO --- NOTHING), so existing rows rely on this DEFAULT until an operator resets. +-- NOTHING), so existing rows rely on this DEFAULT until an admin resets. ALTER TABLE recommendation_weight_profiles ADD COLUMN context_time_weight double precision NOT NULL DEFAULT 1.0; diff --git a/internal/db/migrations/0047_taste_profile_moods.up.sql b/internal/db/migrations/0047_taste_profile_moods.up.sql index 3d16c380..5fd5ffbb 100644 --- a/internal/db/migrations/0047_taste_profile_moods.up.sql +++ b/internal/db/migrations/0047_taste_profile_moods.up.sql @@ -1,11 +1,11 @@ --- 0047_taste_profile_moods.up.sql — mood taste facet (#1534, milestone #160 --- Opt 2b). A fourth taste facet alongside artists + tags + eras: signed weights --- over canonical mood buckets (melancholic / energetic / chill / …) derived --- from a track's enriched folksonomy tags (#1490) via the internal/mood --- vocabulary, rebuilt daily. Weight is a signed float — positive = drawn to --- that mood, negative = passively avoided. Coverage is partial (grows with tag --- enrichment), so it's a supplement, not a foundation. Mirrors --- taste_profile_tags: CASCADE on user delete, indexed by (user_id, weight DESC). +-- 0047_taste_profile_moods.up.sql — mood taste facet. A fourth taste facet +-- alongside artists + tags + eras: signed weights over canonical mood buckets +-- (melancholic / energetic / chill / …) derived from a track's enriched +-- folksonomy tags via the internal/mood vocabulary, rebuilt daily. Weight is a +-- signed float — positive = drawn to that mood, negative = passively avoided. +-- Coverage is partial (grows with tag enrichment), so it's a supplement, not a +-- foundation. Mirrors taste_profile_tags: CASCADE on user delete, indexed by +-- (user_id, weight DESC). CREATE TABLE taste_profile_moods ( user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE, mood text NOT NULL, diff --git a/internal/db/migrations/0048_play_events_device_class.up.sql b/internal/db/migrations/0048_play_events_device_class.up.sql index d8072d25..6177f1a7 100644 --- a/internal/db/migrations/0048_play_events_device_class.up.sql +++ b/internal/db/migrations/0048_play_events_device_class.up.sql @@ -1,9 +1,9 @@ --- 0048_play_events_device_class.up.sql — device-class context conditioning --- (#1551, milestone #160 Opt 3b). Records which class of device a play came --- from (mobile / web / …), sent by the client on play-start and normalized --- server-side. Extends the #1531 time-of-day/weekday affinity cell to --- (daypart × weekday × device) for the radio path. Nullable + no CHECK: values --- are normalized in Go (avoids a CHECK-migration each time a client class is --- added, rule #36), and historical rows stay NULL → excluded from the device --- dimension (a graceful cold-start; the facet ramps as new plays land). +-- 0048_play_events_device_class.up.sql — device-class context conditioning. +-- Records which class of device a play came from (mobile / web / …), sent by +-- the client on play-start and normalized server-side. Extends the +-- time-of-day/weekday affinity cell (0046) to (daypart × weekday × device) for +-- the radio path. Nullable + no CHECK: values are normalized in Go, so a new +-- client class needs no migration, and historical rows stay NULL → excluded +-- from the device dimension (a graceful cold-start; the facet ramps as new +-- plays land). ALTER TABLE play_events ADD COLUMN device_class text; diff --git a/internal/db/migrations/0049_suggestion_snoozes.up.sql b/internal/db/migrations/0049_suggestion_snoozes.up.sql index 02b0907f..9b55ef27 100644 --- a/internal/db/migrations/0049_suggestion_snoozes.up.sql +++ b/internal/db/migrations/0049_suggestion_snoozes.up.sql @@ -1,14 +1,15 @@ -- 0049_suggestion_snoozes.up.sql — time-boxed "not right now" on a Discover --- artist suggestion (#2374, milestone #268 slice 3). +-- artist suggestion. -- --- This is NOT a dislike. Rule #101 forbids a "Not for me" / thumbs-down / --- exclusion UI, and a snooze deliberately isn't one: it records no verdict on +-- This is NOT a dislike. Minstrel has no "Not for me" / thumbs-down / +-- exclusion control — taste is learned from listening — and a snooze +-- deliberately isn't one: it records no verdict on -- the music, expires on its own, and MUST NEVER feed the taste profile. It is -- acquisition triage — "I don't want to request this right now" — so the same -- candidate is free to return once snoozed_until passes. Anything that reads -- this table as negative preference signal is a bug. -- --- Per-user (rule #47), never global: one household member parking a +-- Per-user, never global: one household member parking a -- suggestion must not remove it from anyone else's deck. -- -- candidate_mbid is text with NO foreign key, on purpose. Suggestions come diff --git a/internal/db/migrations/0050_candidate_artist_tags.up.sql b/internal/db/migrations/0050_candidate_artist_tags.up.sql index 660a3da7..72896af5 100644 --- a/internal/db/migrations/0050_candidate_artist_tags.up.sql +++ b/internal/db/migrations/0050_candidate_artist_tags.up.sql @@ -1,18 +1,17 @@ -- 0050_candidate_artist_tags.up.sql — folksonomy tags for OUT-OF-LIBRARY --- artists (#2376, milestone #268 slice 5). +-- artists. -- -- track_tags (0042) cannot hold these: it is FK'd to tracks(id), and a -- Discover candidate has no local row by definition. So this is a parallel -- cache keyed by the candidate's MusicBrainz MBID — the only stable identity -- available for an artist we don't have. -- --- Purpose is slice 6: rank suggestions by overlap between these tags and the +-- Purpose: rank Discover suggestions by overlap between these tags and the -- user's taste_profile_tags, turning "neighbour of an artist you play" into -- "matches the sound you like". -- -- GLOBAL, not per-user: a candidate's tags are a property of the artist, not --- of anyone's taste. Nothing here is user-scoped, so rule #47 has nothing to --- scope — the per-user part lives entirely in slice 6's ranking. +-- of anyone's taste. The per-user part lives entirely in the ranking. -- -- weight is a normalized folksonomy strength in [0,1], same scale as -- track_tags, so the two can be compared without a conversion step. diff --git a/internal/db/migrations/0051_discover_tuning.up.sql b/internal/db/migrations/0051_discover_tuning.up.sql index 54bc2be0..e0cdacfe 100644 --- a/internal/db/migrations/0051_discover_tuning.up.sql +++ b/internal/db/migrations/0051_discover_tuning.up.sql @@ -1,14 +1,14 @@ -- 0051_discover_tuning.up.sql — tunable knobs for the Discover request --- surface (#2377, milestone #268 slice 6). +-- surface. -- -- A FOURTH tuning scope alongside radio / daily_mix / taste. Its own scope -- rather than extra columns on taste_tuning, for a reason that matters: -- snooze_days lives here, and a snooze must never be read as taste signal --- (#2374). Filing it under 'taste' would put it one careless join away from --- exactly the leak that design forbids. +-- (0049). Filing it under 'taste' would put it one careless join away from +-- exactly that leak. -- --- Per rule #25 these are DB-backed and editable in the admin UI with no --- restart — the shipped values below are defaults, not settings. +-- These are DB-backed and editable in the admin UI with no restart — the +-- shipped values below are defaults, not settings. CREATE TABLE discover_tuning ( singleton boolean PRIMARY KEY DEFAULT true CONSTRAINT discover_tuning_singleton_check CHECK (singleton), @@ -17,10 +17,10 @@ CREATE TABLE discover_tuning ( -- [0,1]. So 0 disables the feature outright and leaves pure similarity -- ranking, 1.0 lets a perfectly-matching candidate double its score, and -- a candidate with no cached tags is unchanged rather than penalised - -- (rule #131 — tag coverage is permanently partial, see #2376). + -- (tag coverage is permanently partial, so a missing tag is no signal). tag_overlap_weight double precision NOT NULL, - -- Default snooze duration in days. Was a Go constant in - -- internal/api/suggestions.go; moved here per rule #25. + -- Default snooze duration in days. Before this a Go constant in + -- internal/api/suggestions.go; now an admin setting. snooze_days double precision NOT NULL, updated_at timestamptz NOT NULL DEFAULT now() ); diff --git a/internal/db/migrations/0052_session_ips.up.sql b/internal/db/migrations/0052_session_ips.up.sql index 1da0bf48..b461d925 100644 --- a/internal/db/migrations/0052_session_ips.up.sql +++ b/internal/db/migrations/0052_session_ips.up.sql @@ -1,4 +1,4 @@ --- Session provenance for the active-sessions surface (#370). +-- Session provenance for the active-sessions surface. -- -- TWO addresses, not one, and the pair is the point: a session created at -- home and now being used from somewhere else is the shape of a stolen diff --git a/internal/db/migrations/0053_network_settings.up.sql b/internal/db/migrations/0053_network_settings.up.sql index d53649ff..a995af37 100644 --- a/internal/db/migrations/0053_network_settings.up.sql +++ b/internal/db/migrations/0053_network_settings.up.sql @@ -1,4 +1,4 @@ --- Trusted reverse-proxy depth for client-IP extraction (#2453). +-- Trusted reverse-proxy depth for client-IP extraction. -- -- X-Forwarded-For grows left-to-right: each proxy APPENDS the peer it -- received the request from. For client -> CDN -> own-proxy -> Minstrel the @@ -11,14 +11,15 @@ -- -- Default 1: a publicly reachable Minstrel needs a TLS terminator in front of -- it, and recording that terminator's own address for every session makes the --- active-sessions surface (#370) useless — created_ip and last_ip would both +-- active-sessions surface useless — created_ip and last_ip would both -- be the proxy, so the "address changed" signal could never fire. -- -- The cost, stated on the admin card rather than buried: hops >= 1 DECLARES -- that a proxy exists. If one doesn't, a client can forge X-Forwarded-For and -- choose what its own session row shows, which defeats exactly the compromise --- detection #370 exists for. That is inherent to the trusted-hop model, which --- is why 0 is a first-class setting and not a hidden escape hatch. +-- detection the active-sessions surface exists for. That is inherent to the +-- trusted-hop model, which is why 0 is a first-class setting and not a hidden +-- escape hatch. -- -- Upper bound 10 guards a typo turning into "trust the whole header"; no real -- deployment chains ten proxies. diff --git a/internal/db/migrations/0054_track_tag_read_version.up.sql b/internal/db/migrations/0054_track_tag_read_version.up.sql index a66580b8..e9b0f3a8 100644 --- a/internal/db/migrations/0054_track_tag_read_version.up.sql +++ b/internal/db/migrations/0054_track_tag_read_version.up.sql @@ -1,11 +1,11 @@ -- Records which version of the scanner's tag-extraction logic last wrote a --- track's tag-derived columns (#2499). +-- track's tag-derived columns. -- -- DEFAULT 0 is the point of this migration: every existing row lands below the -- scanner's current library.tagReadVersion, so the next scan re-reads its tags -- instead of short-circuiting on the mtime check. That repairs genre values the -- old reader welded together ("Alternative Rock" + "Rock" -> "Alternative --- RockRock") without asking the operator to wipe and rebuild the library. +-- RockRock") without anyone wiping and rebuilding the library. -- -- Bump library.tagReadVersion in Go — not this default — whenever a tag -- extraction fix needs to reach already-indexed files. That makes tag repairs a diff --git a/internal/db/migrations/0055_track_missing_since.up.sql b/internal/db/migrations/0055_track_missing_since.up.sql index 11952ade..64cdea80 100644 --- a/internal/db/migrations/0055_track_missing_since.up.sql +++ b/internal/db/migrations/0055_track_missing_since.up.sql @@ -1,4 +1,4 @@ --- Marks a track whose file the scanner could no longer find (#2523). +-- Marks a track whose file the scanner could no longer find. -- -- NULL means present. A timestamp means the file was absent as of that scan, -- and is the point from which "how long has this been gone" is measured — which diff --git a/internal/db/migrations/0056_missing_reacquisition.up.sql b/internal/db/migrations/0056_missing_reacquisition.up.sql index a4e73b4b..175b576f 100644 --- a/internal/db/migrations/0056_missing_reacquisition.up.sql +++ b/internal/db/migrations/0056_missing_reacquisition.up.sql @@ -1,14 +1,12 @@ --- Auto re-acquisition of missing files via Lidarr (#2527 slice 3, milestone --- #290). A file going missing has been a dead end until now: reconcile marks --- it (#2523), every selection path skips it, the admin surface lists it, and --- there it sits. +-- Auto re-acquisition of missing files via Lidarr. Before this a missing +-- file was a dead end: reconcile marks it, every selection path skips it, +-- the admin surface lists it, and there it sits. -- -- The unit here is the ALBUM, not the track, and that is the design decision -- carrying most of the safety. Lidarr acquires releases; there is no -- meaningful "fetch me one track" operation, and a track-kind request needs a -- recording MBID plenty of files simply don't have. Grouping by album means --- the case that produced #2523 -- three reorganised albums, ~40 missing files --- -- becomes three requests instead of forty. +-- a reorganised album with dozens of missing files becomes one request. CREATE TABLE missing_reacquisitions ( album_id uuid PRIMARY KEY REFERENCES albums (id) ON DELETE CASCADE, @@ -36,12 +34,13 @@ CREATE INDEX missing_reacquisitions_due_idx WHERE gave_up_at IS NULL; -- Settings, singleton in the style of network_settings (0053). Every value an --- operator might want to tune lives here rather than in YAML (rule #25). +-- admin might want to tune lives here rather than in YAML, changed without a +-- restart. CREATE TABLE reacquisition_settings ( id boolean PRIMARY KEY DEFAULT true, - -- Master switch. Default true: the operator asked for this to happen by - -- itself, and a feature that ships switched off is a feature nobody finds. + -- Master switch. Default true: this is meant to happen by itself, and a + -- feature that ships switched off is a feature nobody finds. enabled boolean NOT NULL DEFAULT true, -- How long a file must have been missing before the FIRST attempt. This @@ -49,7 +48,7 @@ CREATE TABLE reacquisition_settings ( -- transiently -- an unmounted volume, a container that started before its -- media mount attached, a NAS mid-reboot -- and every one of those -- resolves itself well inside a day at no cost. tracks.missing_since is - -- never re-stamped (#2523), so it is a true "gone since" clock to measure + -- never re-stamped, so it is a true "gone since" clock to measure -- against. grace_hours int NOT NULL DEFAULT 24, diff --git a/internal/db/migrations/0057_songs_like_tuning.up.sql b/internal/db/migrations/0057_songs_like_tuning.up.sql index cf8d6c1e..c8071e66 100644 --- a/internal/db/migrations/0057_songs_like_tuning.up.sql +++ b/internal/db/migrations/0057_songs_like_tuning.up.sql @@ -1,13 +1,9 @@ --- 0057_songs_like_tuning.up.sql — a THIRD weight profile, for Songs-like --- (Scribe #3881, milestone #398). +-- 0057_songs_like_tuning.up.sql — a THIRD weight profile, for Songs-like. -- --- Songs-like shared the `daily_mix` profile with For-You, and that is the bug. --- The two surfaces want opposite things: For-You is a broad "what will they --- enjoy today", Songs-like answers "what sounds like THIS", and under one set --- of weights the broad answer wins. Operator, 2026-09-10: "I'm expecting to --- get a consistent sound and style from the experience... I was getting a --- seeming wide variety of music from each one when I was hoping to stay in a --- certain neighborhood." +-- Before: Songs-like shared the `daily_mix` profile with For-You. The two +-- surfaces want opposite things: For-You is a broad "what will they enjoy +-- today", Songs-like answers "what sounds like THIS" and should stay in one +-- neighbourhood of sound, and under one set of weights the broad answer wins. -- -- Under the shared daily_mix weights, an UNRELATED track the user had liked -- and not played recently scored 1.0 + 2.0 + 1.0 = 4.0 before taste, while a diff --git a/internal/db/migrations/0058_track_fingerprints.up.sql b/internal/db/migrations/0058_track_fingerprints.up.sql index aa0f8c9c..a42e6151 100644 --- a/internal/db/migrations/0058_track_fingerprints.up.sql +++ b/internal/db/migrations/0058_track_fingerprints.up.sql @@ -1,5 +1,4 @@ --- 0058_track_fingerprints.up.sql — an acoustic identity per track (Scribe --- milestone #400: #3905, #3906). +-- 0058_track_fingerprints.up.sql — an acoustic identity per track. -- -- A table of its own rather than columns on tracks, for the hot path's sake: -- tracks is read with SELECT * by eight queries, among them ListTracksByAlbum, @@ -16,9 +15,9 @@ -- A failure that says nothing about the file — a timeout, a cancelled scan, a -- missing binary — writes no row at all, so the backfill tries again. CREATE TABLE track_fingerprints ( - -- CASCADE is right here, unlike for the likes and play history M400's - -- merge has to carry across: a fingerprint describes one file's bytes and - -- means nothing once that file's row is gone. + -- CASCADE is right here, unlike for the likes and play history the + -- duplicate merge has to carry across: a fingerprint describes one file's + -- bytes and means nothing once that file's row is gone. track_id uuid PRIMARY KEY REFERENCES tracks (id) ON DELETE CASCADE, -- SHA-256 of the ENCODED audio packets (ffmpeg -c:a copy -f hash), not of -- decoded samples. internal/library/fingerprint.go says why. diff --git a/internal/db/migrations/0059_duplicate_groups.up.sql b/internal/db/migrations/0059_duplicate_groups.up.sql index 17a112b5..67073cd6 100644 --- a/internal/db/migrations/0059_duplicate_groups.up.sql +++ b/internal/db/migrations/0059_duplicate_groups.up.sql @@ -1,9 +1,9 @@ -- 0059_duplicate_groups.up.sql — proposed duplicates and the sweeps that find --- them (Scribe milestone #400: #3910). +-- them. -- --- The sweep compares fingerprints (track_fingerprints, 0058) and proposes groups --- of tracks that hold one recording. Nothing here merges anything: a group is a --- proposal the operator reviews, and the merge (#3911) is a separate act. +-- The sweep compares fingerprints (track_fingerprints, 0058) and proposes +-- groups of tracks that hold one recording. Nothing here merges anything: a +-- group is a proposal an admin reviews, and the merge is a separate act. -- One row per sweep. Lets the report tell "the sweep has never run" apart from -- "it ran and found nothing", and gives the in-flight guard something to check, @@ -49,6 +49,6 @@ CREATE TABLE duplicate_group_members ( CREATE INDEX duplicate_group_members_track_idx ON duplicate_group_members (track_id); -- The sweep streams candidates in (duration_ms, id) order, keyset-paged, so it --- only ever holds a few seconds' worth of durations in memory. Without this each --- page would sort the whole library again. +-- only ever holds a few seconds' worth of durations in memory. Without this +-- each page would sort the whole library again. CREATE INDEX tracks_duration_id_idx ON tracks (duration_ms, id); diff --git a/internal/db/migrations/0060_play_events_track_index.up.sql b/internal/db/migrations/0060_play_events_track_index.up.sql index 35036982..2c19bd36 100644 --- a/internal/db/migrations/0060_play_events_track_index.up.sql +++ b/internal/db/migrations/0060_play_events_track_index.up.sql @@ -1,8 +1,8 @@ --- 0060_play_events_track_index.up.sql — play_events by track (Scribe #3912, #3911). +-- 0060_play_events_track_index.up.sql — play_events by track. -- -- play_events is indexed by (user_id, started_at) and (user_id, track_id), both -- led by user. Nothing reached it by track alone until the duplicates report, -- which shows each copy's play count — a scan of the whole table per copy — and --- the merge (#3911), which repoints a duplicate's play history onto the copy +-- the duplicate merge, which repoints a duplicate's play history onto the copy -- being kept. Both ask "every play of this track", whoever played it. CREATE INDEX play_events_track_idx ON play_events (track_id); diff --git a/internal/db/migrations/0061_fingerprint_settings.up.sql b/internal/db/migrations/0061_fingerprint_settings.up.sql index 021f85af..89911d69 100644 --- a/internal/db/migrations/0061_fingerprint_settings.up.sql +++ b/internal/db/migrations/0061_fingerprint_settings.up.sql @@ -1,20 +1,18 @@ --- 0061_fingerprint_settings.up.sql — fingerprinting's knobs, in admin Settings --- (Scribe #3913, milestone #400). Rule 25: anything an operator might tune is a --- database row, changed without a restart. Singleton in the style of --- reacquisition_settings (0056). +-- 0061_fingerprint_settings.up.sql — fingerprinting's knobs, in admin +-- Settings: anything an admin might tune is a database row, changed without a +-- restart. Singleton in the style of reacquisition_settings (0056). CREATE TABLE fingerprint_settings ( id boolean PRIMARY KEY DEFAULT true, - -- Fingerprinting new files, the backfill, and the duplicate sweep. Off stops - -- the decode work entirely — the reason to turn it off is a slow NAS, and - -- that is the operator's call. On by default: a library that cannot tell its - -- duplicates apart is what milestone #400 exists to end. + -- Fingerprinting new files, the backfill, and the duplicate sweep. Off + -- stops the decode work entirely — the reason to turn it off is a slow NAS. + -- On by default: without it the library cannot tell its duplicates apart. enabled boolean NOT NULL DEFAULT true, -- Seconds of audio fpcalc fingerprints. Chromaprints taken at different -- lengths cannot be compared, which is why track_fingerprints records the - -- length each row was taken at (below): change this and every chromaprint is - -- re-derived, and until then only rows at the new length are compared. + -- length each row was taken at (below): change this and every chromaprint + -- is re-derived, and until then only rows at the new length are compared. chromaprint_length_sec integer NOT NULL DEFAULT 120, -- The most disagreement two aligned fingerprints may show and still be diff --git a/internal/db/migrations/0062_network_public_url.up.sql b/internal/db/migrations/0062_network_public_url.up.sql index 0d1f6dac..4eb7dee8 100644 --- a/internal/db/migrations/0062_network_public_url.up.sql +++ b/internal/db/migrations/0062_network_public_url.up.sql @@ -1,8 +1,8 @@ -- The address users reach Minstrel at, e.g. https://music.example.com. -- --- Password-reset links used to be built from the request's Host header, --- which the requester controls: a forgot-password call with a forged Host +-- Before: password-reset links were built from the request's Host header, +-- which the requester controls, so a forgot-password call with a forged Host -- would email the victim a real reset token on a link to the attacker's --- server. Links are now built only from this operator-set value, and none --- are sent while it is empty (M462 #4981). +-- server. After: links are built only from this admin-set value, and none +-- are sent while it is empty. ALTER TABLE network_settings ADD COLUMN public_url text NOT NULL DEFAULT ''; diff --git a/internal/db/migrations/0063_api_token_hash.up.sql b/internal/db/migrations/0063_api_token_hash.up.sql index fd161a19..646d6b18 100644 --- a/internal/db/migrations/0063_api_token_hash.up.sql +++ b/internal/db/migrations/0063_api_token_hash.up.sql @@ -2,7 +2,7 @@ -- same way session tokens are. A leaked database row or backup no longer -- hands out working keys. Existing keys are hashed in place, so every -- Subsonic client keeps working; the key itself can no longer be shown --- again, only replaced (M462 #4983). +-- again, only replaced. ALTER TABLE users ADD COLUMN api_token_hash text; UPDATE users SET api_token_hash = encode(sha256(convert_to(api_token, 'UTF8')), 'hex'); ALTER TABLE users ALTER COLUMN api_token_hash SET NOT NULL; diff --git a/internal/db/migrations/0064_clear_subsonic_password.up.sql b/internal/db/migrations/0064_clear_subsonic_password.up.sql index ab83c2b4..bcb885bc 100644 --- a/internal/db/migrations/0064_clear_subsonic_password.up.sql +++ b/internal/db/migrations/0064_clear_subsonic_password.up.sql @@ -1,9 +1,9 @@ --- `minstrel admin reset-password` used to copy the new login password into +-- Before: `minstrel admin reset-password` copied the new login password into -- subsonic_password, so every account recovered through the CLI had its login -- password stored in plain text, and a later password change in Settings left --- that copy behind (M462 #5026). The CLI no longer writes this column; a --- Subsonic password is now generated separately in Settings and is never the --- login password. Clearing every value here removes the copies already made. +-- that copy behind. After: the CLI does not write this column; a Subsonic +-- password is generated separately in Settings and is never the login +-- password. Clearing every value here removes the copies already made. -- -- Accounts whose Subsonic client signs in with t/s stop working until the user -- generates a Subsonic password (or switches the client to an API key). diff --git a/internal/db/migrations/0065_track_loudness.up.sql b/internal/db/migrations/0065_track_loudness.up.sql index ae039dc6..17b46ace 100644 --- a/internal/db/migrations/0065_track_loudness.up.sql +++ b/internal/db/migrations/0065_track_loudness.up.sql @@ -1,5 +1,5 @@ -- 0065_track_loudness.up.sql — measured loudness per track, for loudness --- normalization (Scribe milestone #464, #4995). +-- normalization. -- -- Measured with ffmpeg's EBU R128 filter rather than read from ReplayGain tags: -- tags in the wild are written against four different reference levels, and @@ -32,9 +32,9 @@ CREATE TABLE track_loudness ( loudness_range_lu real, -- How many 400 ms gating blocks fell in each 0.1 LU bin. Bin i holds blocks -- measuring -70.0 + (block_hist_start + i) / 10 LUFS; the array is trimmed - -- to the first and last non-empty bins. Album loudness is the gated loudness - -- of every block on the album, so it is computed from these exactly, with no - -- second decode (#4996). + -- to the first and last non-empty bins. Album loudness is the gated + -- loudness of every block on the album, so it is computed from these + -- exactly, with no second decode. block_hist_start smallint, block_hist integer[], unreadable boolean NOT NULL DEFAULT false, diff --git a/internal/db/migrations/0066_album_loudness.up.sql b/internal/db/migrations/0066_album_loudness.up.sql index 9243dcb9..0e04820f 100644 --- a/internal/db/migrations/0066_album_loudness.up.sql +++ b/internal/db/migrations/0066_album_loudness.up.sql @@ -1,5 +1,4 @@ --- 0066_album_loudness.up.sql — album loudness for album-mode normalization --- (Scribe milestone #464, #4996). +-- 0066_album_loudness.up.sql — album loudness for album-mode normalization. -- -- An album's loudness is the gated loudness of every 400 ms block on the album, -- not an average of its tracks' values: a quiet interlude and a loud single diff --git a/internal/db/migrations/0067_user_normalization_prefs.up.sql b/internal/db/migrations/0067_user_normalization_prefs.up.sql index 30163933..4f767df7 100644 --- a/internal/db/migrations/0067_user_normalization_prefs.up.sql +++ b/internal/db/migrations/0067_user_normalization_prefs.up.sql @@ -1,4 +1,4 @@ --- Each user's loudness-normalization preference (M464 #4998). It is kept on +-- Each user's loudness-normalization preference. It is kept on -- the server, not per device, so the web player, the Android app and the -- leveled stream for UPnP/Sonos all apply the same one. -- diff --git a/internal/db/migrations/0068_leveled_stream_cache.up.sql b/internal/db/migrations/0068_leveled_stream_cache.up.sql index 941bd128..84293517 100644 --- a/internal/db/migrations/0068_leveled_stream_cache.up.sql +++ b/internal/db/migrations/0068_leveled_stream_cache.up.sql @@ -1,11 +1,10 @@ -- 0068_leveled_stream_cache.up.sql — the size cap of the leveled-stream render --- cache (Scribe milestone #464, #5001). +-- cache. -- -- Sonos and other UPnP renderers fetch a track's URL themselves, so the phone -- cannot level what they play. The server renders a gain-applied FLAC for them -- instead and keeps it on disk, oldest-used first out once the cache is over --- this size. An operator setting (rule 25), on the loudness settings row it --- belongs with. +-- this size. An admin setting, on the loudness settings row it belongs with. ALTER TABLE loudness_settings ADD COLUMN leveled_cache_mb integer NOT NULL DEFAULT 2048, ADD CONSTRAINT loudness_settings_leveled_cache_range diff --git a/internal/db/migrations/0069_acoustid_lookup.up.sql b/internal/db/migrations/0069_acoustid_lookup.up.sql index 7610c8f4..ecd56c99 100644 --- a/internal/db/migrations/0069_acoustid_lookup.up.sql +++ b/internal/db/migrations/0069_acoustid_lookup.up.sql @@ -1,11 +1,11 @@ -- 0069_acoustid_lookup.up.sql — recording MBIDs looked up through AcoustID --- for tracks whose tags carry none (Scribe milestone #401). +-- for tracks whose tags carry none. -- -- A track with no recording MBID is invisible to the ListenBrainz similarity -- arm in both directions, so this fills the ids the tags leave empty. --- Where tracks.mbid came from. The file's own tag always outranks a lookup --- (M401 D4): the scan's upsert keeps a looked-up id while the tag carries +-- Where tracks.mbid came from. The file's own tag always outranks a lookup: +-- the scan's upsert keeps a looked-up id while the tag carries -- none, and replaces it the moment the tag carries one. ALTER TABLE tracks ADD COLUMN mbid_source text; -- Every id stored so far was read from a tag (the scan or the tag backfill), @@ -25,7 +25,7 @@ ALTER TABLE tracks -- after disambiguating by title and length; recording_mbid is it -- ambiguous several recordings cleared it and nothing told them apart. -- Nothing is written: a wrong MBID would feed similarity the --- wrong neighbours, which is worse than none (M401 D5) +-- wrong neighbours, which is worse than none -- no_match AcoustID linked nothing at or above the threshold -- failed fpcalc rejected the file, or AcoustID rejected its print -- Every state is settled until the file changes. A failure that says nothing @@ -50,12 +50,12 @@ CREATE TABLE track_acoustid_lookups ( ); CREATE INDEX track_acoustid_lookups_state ON track_acoustid_lookups (state); --- AcoustID's knobs, in admin Settings (rule 25). Singleton in the style of +-- AcoustID's knobs, in admin Settings. Singleton in the style of -- loudness_settings (0065). CREATE TABLE acoustid_settings ( id boolean PRIMARY KEY DEFAULT true, - -- Off by default (rule 26): AcoustID needs a key the operator registers, - -- and the lookup sends fingerprints to a third party. + -- Off by default: AcoustID needs a key an admin registers, and the lookup + -- sends fingerprints to a third party. enabled boolean NOT NULL DEFAULT false, api_key text, -- The score a recording needs before it is considered at all. AcoustID diff --git a/internal/db/migrations/0070_album_release_group_mbid.up.sql b/internal/db/migrations/0070_album_release_group_mbid.up.sql index 08a85f2e..e6068030 100644 --- a/internal/db/migrations/0070_album_release_group_mbid.up.sql +++ b/internal/db/migrations/0070_album_release_group_mbid.up.sql @@ -1,4 +1,4 @@ --- MusicBrainz release-group id for an album (#5241). albums.mbid holds the +-- MusicBrainz release-group id for an album. albums.mbid holds the -- *release* id (Picard's musicbrainz_albumid: one edition), but Lidarr names -- albums by release group, so re-acquisition and request completion need -- this one. Not unique: several releases of one group can each be an album diff --git a/internal/db/migrations/0071_reopen_false_reacquisition_completions.up.sql b/internal/db/migrations/0071_reopen_false_reacquisition_completions.up.sql index dd240a0b..e7e18c49 100644 --- a/internal/db/migrations/0071_reopen_false_reacquisition_completions.up.sql +++ b/internal/db/migrations/0071_reopen_false_reacquisition_completions.up.sql @@ -1,10 +1,10 @@ --- #5263: the reconciler completed album and track requests as soon as the album --- had any track on disk. Re-acquisition targets albums with only SOME tracks --- missing, so those requests completed the moment Lidarr accepted the add, --- with nothing downloaded. +-- Before: the reconciler completed album and track requests as soon as the +-- album had any track on disk. Re-acquisition targets albums with only SOME +-- tracks missing, so those requests completed the moment Lidarr accepted the +-- add, with nothing downloaded. -- -- Reopen every completed album/track request whose matched album does not meet --- the corrected test (see albumForRequest in internal/lidarrrequests): a track +-- the corrected test: a track -- that arrived after the request, or no track that was already missing at the -- request still missing. The reconciler then judges them again; the Lidarr add -- stays confirmed, so nothing is re-sent. Requests that genuinely completed diff --git a/internal/db/migrations/0072_similarity_cache.up.sql b/internal/db/migrations/0072_similarity_cache.up.sql index c23ca242..f11866f0 100644 --- a/internal/db/migrations/0072_similarity_cache.up.sql +++ b/internal/db/migrations/0072_similarity_cache.up.sql @@ -1,7 +1,7 @@ --- #5296: keep ListenBrainz's whole similar-recordings answer, and record every +-- Keep ListenBrainz's whole similar-recordings answer, and record every -- fetch, so the library match is made locally and can be made again. -- --- Until now the worker kept only the recordings already in the library (at most +-- Before: the worker kept only the recordings already in the library (at most -- 20 of LB's 50) and threw the rest away. Two things followed: -- -- * A recording that reached the library later — a Lidarr import, or an MBID @@ -9,8 +9,8 @@ -- fetched again. -- * A seed with no in-library match wrote nothing at all, and freshness was -- read from track_similarity, so that seed was never fresh. With the queue --- ordered by id, 25 such seeds held its head and were re-asked every hour --- while 2,449 others waited (#3879). +-- ordered by id, such seeds held its head and were re-asked every hour +-- while the rest waited. -- -- The listenbrainz rows in track_similarity are now derived from this cache by -- ResolveListenBrainzTrackEdges. diff --git a/internal/db/migrations/0073_user_notifications.up.sql b/internal/db/migrations/0073_user_notifications.up.sql index 945bb0d1..ea6fdf9e 100644 --- a/internal/db/migrations/0073_user_notifications.up.sql +++ b/internal/db/migrations/0073_user_notifications.up.sql @@ -1,11 +1,11 @@ --- M489: a persistent, per-user notifications inbox (#726). +-- A persistent, per-user notifications inbox. -- -- The event bus is fire-and-forget: a client that is not connected when a -- request completes, or when the scanner marks tracks missing, never hears of -- it. A row here is the durable record; the bus only nudges open clients to -- come and read it. -- --- `kind` is CHECK-gated (rule 36). A new kind adds its value here, in the +-- `kind` is CHECK-gated. A new kind adds its value here, in the -- same migration as the code that produces it. The list is mirrored in -- internal/notifications/kinds.go, and both CHECKs below must agree with it. diff --git a/internal/db/migrations/0074_notification_email.up.sql b/internal/db/migrations/0074_notification_email.up.sql index f1129e66..917fd02a 100644 --- a/internal/db/migrations/0074_notification_email.up.sql +++ b/internal/db/migrations/0074_notification_email.up.sql @@ -1,11 +1,11 @@ --- M489 #5346: notifications by email, grouped. Nothing is emailed per event. +-- Notifications by email, grouped. Nothing is emailed per event. -- -- New music (request_completed) goes out as at most one summary a day, at a set -- hour in each user's own timezone. Everything else is batched: a batch opens -- at the first un-emailed item and one email goes out a window later, holding -- whatever accumulated. internal/notifications/digest.go is the one sender. --- The two knobs, in admin Settings (rule 25). Singleton in the style of +-- The two knobs, in admin Settings. Singleton in the style of -- fingerprint_settings (0061). CREATE TABLE notification_email_settings ( id boolean PRIMARY KEY DEFAULT true, diff --git a/internal/db/migrations/0075_duplicate_resolve.up.sql b/internal/db/migrations/0075_duplicate_resolve.up.sql index 29431611..95b958da 100644 --- a/internal/db/migrations/0075_duplicate_resolve.up.sql +++ b/internal/db/migrations/0075_duplicate_resolve.up.sql @@ -1,30 +1,30 @@ --- 0075_duplicate_resolve.up.sql — duplicates resolve themselves (Scribe --- milestone #498: #5435, #5436, #5437). +-- 0075_duplicate_resolve.up.sql — duplicates resolve themselves. -- --- The sweep (0059) proposes groups and, until now, every proposal waited for the --- operator. The resolver classifies each pending group and acts on the classes --- that are safe. What makes deletion safe is Lidarr's view of the file: Lidarr --- maps one file to each track of the release it monitors, so deleting a mapped --- file opens a hole it fills by downloading again. Only a copy Lidarr does not --- map may go. +-- Before: the sweep (0059) proposed groups and every proposal waited for an +-- admin. After: the resolver classifies each pending group and acts on the +-- classes that are safe. What makes deletion safe is Lidarr's view of the file: +-- Lidarr maps one file to each track of the release it monitors, so deleting a +-- mapped file opens a hole it fills by downloading again. Only a copy Lidarr +-- does not map may go. -- What kind of duplicate a group is. NULL until the resolver has classified it. -- same_release every copy is on one album -- cross_release the same song on different releases (a single and its album) -- mismatch identical audio under different titles on different albums: -- a file imported as the wrong song --- review anything else: the operator decides --- Rule 36: a new value swaps this constraint in the same migration. +-- review anything else: an admin decides +-- A new value swaps this constraint in the same migration. ALTER TABLE duplicate_groups ADD COLUMN class text; ALTER TABLE duplicate_groups ADD CONSTRAINT duplicate_groups_class_check CHECK (class IN ('same_release', 'cross_release', 'mismatch', 'review')); --- Why the resolver left a group for the operator, in words the report shows --- ("Lidarr tracks every copy as its own track"). NULL when it had nothing to say. +-- Why the resolver left a group for an admin, in words the report shows +-- ("Lidarr tracks every copy as its own track"). NULL when it had nothing to +-- say. ALTER TABLE duplicate_groups ADD COLUMN resolve_note text; -- True when the resolver merged the group itself, so the report can tell its --- merges from the operator's. +-- merges from an admin's. ALTER TABLE duplicate_groups ADD COLUMN resolved_automatically boolean NOT NULL DEFAULT false; -- Lidarr's view of each copy, as of the resolver's last pass: @@ -36,12 +36,12 @@ ALTER TABLE duplicate_group_members ADD COLUMN lidarr_state text; ALTER TABLE duplicate_group_members ADD CONSTRAINT duplicate_group_members_lidarr_state_check CHECK (lidarr_state IN ('tracked', 'unmapped')); --- Rule 25: the operator can turn the resolver off. On by default: the operator --- asked for duplicates to resolve themselves (2026-10-09). +-- An admin can turn the resolver off. On by default: duplicates are meant to +-- resolve themselves. ALTER TABLE fingerprint_settings ADD COLUMN auto_resolve boolean NOT NULL DEFAULT true; --- The resolver tells admins what it did (rule 36: the kind CHECKs of 0073 are --- swapped to carry the new value). Postgres named both from their columns. +-- The resolver tells admins what it did (the kind CHECKs of 0073 are swapped +-- to carry the new value). Postgres named both from their columns. ALTER TABLE user_notifications DROP CONSTRAINT user_notifications_kind_check; ALTER TABLE user_notifications ADD CONSTRAINT user_notifications_kind_check CHECK (kind IN ('request_approved', 'request_rejected', 'request_completed', diff --git a/internal/db/migrations/0076_track_song_link.up.sql b/internal/db/migrations/0076_track_song_link.up.sql index b314512c..94789e09 100644 --- a/internal/db/migrations/0076_track_song_link.up.sql +++ b/internal/db/migrations/0076_track_song_link.up.sql @@ -1,5 +1,5 @@ -- 0076_track_song_link.up.sql — the same song on several releases counts as --- one song (Scribe milestone #498, #5438; operator decision D-b). +-- one song. -- -- A single and the album it came from each hold the song, and Lidarr keeps -- both: each fulfils its own release. Minstrel keeps both files and links diff --git a/internal/db/migrations/0077_track_source_verdict.up.sql b/internal/db/migrations/0077_track_source_verdict.up.sql index c1361d5f..529b5d3e 100644 --- a/internal/db/migrations/0077_track_source_verdict.up.sql +++ b/internal/db/migrations/0077_track_source_verdict.up.sql @@ -1,13 +1,12 @@ -- 0077_track_source_verdict.up.sql — video rips are held back from mixes and --- radio until the operator says otherwise (Scribe milestone #498, #5439; --- operator decision D-d). +-- radio until an admin says otherwise. -- --- A file whose name carries a video-rip marker ("(Official Video)", "[HD]", --- #5410) holds the audio of an upload, which can carry intros, skits or a +-- A file whose name carries a video-rip marker ("(Official Video)", "[HD]") +-- holds the audio of an upload, which can carry intros, skits or a -- different take. The duplicate resolver sets 'suspect' on such a present -- track each pass, and clears it when the name no longer matches. 'suspect' -- keeps the track out of radio and the system mixes; it still plays when --- chosen. 'fine' is the operator's "this one is fine" and is never +-- chosen. 'fine' is an admin's "this one is fine" and is never -- overwritten by the resolver. ALTER TABLE tracks ADD COLUMN source_verdict text CONSTRAINT tracks_source_verdict_check CHECK (source_verdict IN ('suspect', 'fine')); diff --git a/internal/db/migrations/0078_drop_duplicate_lidarr_state.down.sql b/internal/db/migrations/0078_drop_duplicate_lidarr_state.down.sql new file mode 100644 index 00000000..b325605d --- /dev/null +++ b/internal/db/migrations/0078_drop_duplicate_lidarr_state.down.sql @@ -0,0 +1,3 @@ +ALTER TABLE duplicate_group_members ADD COLUMN lidarr_state text; +ALTER TABLE duplicate_group_members ADD CONSTRAINT duplicate_group_members_lidarr_state_check + CHECK (lidarr_state IN ('tracked', 'unmapped')); diff --git a/internal/db/migrations/0078_drop_duplicate_lidarr_state.up.sql b/internal/db/migrations/0078_drop_duplicate_lidarr_state.up.sql new file mode 100644 index 00000000..943423a1 --- /dev/null +++ b/internal/db/migrations/0078_drop_duplicate_lidarr_state.up.sql @@ -0,0 +1,9 @@ +-- 0078_drop_duplicate_lidarr_state.up.sql — the duplicate resolver works +-- without Lidarr. +-- +-- 0075 recorded Lidarr's view of each duplicate copy (tracked or unmapped) +-- because a copy Lidarr mapped could not be removed without Lidarr downloading +-- it again. Lidarr is gone: Minstrel is the only program writing the library, +-- so no copy is anyone else's to keep, and the column has nothing to say. +ALTER TABLE duplicate_group_members DROP CONSTRAINT IF EXISTS duplicate_group_members_lidarr_state_check; +ALTER TABLE duplicate_group_members DROP COLUMN IF EXISTS lidarr_state; diff --git a/internal/db/migrations/0079_acquisition_services.down.sql b/internal/db/migrations/0079_acquisition_services.down.sql new file mode 100644 index 00000000..49d5d65c --- /dev/null +++ b/internal/db/migrations/0079_acquisition_services.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS acquisition_services; diff --git a/internal/db/migrations/0079_acquisition_services.up.sql b/internal/db/migrations/0079_acquisition_services.up.sql new file mode 100644 index 00000000..b4487dac --- /dev/null +++ b/internal/db/migrations/0079_acquisition_services.up.sql @@ -0,0 +1,61 @@ +-- 0079_acquisition_services.up.sql — the indexers and download clients the +-- admin adds in Admin → Integrations. +-- +-- Minstrel replaces Lidarr, so it searches and downloads itself. Nothing is +-- seeded and nothing is copied from Lidarr: a connection is a row an admin +-- adds, from a list of what Minstrel can speak: +-- +-- kind which list the row appears in: search (Indexers) or download +-- (Download clients). +-- protocol which client is built: prowlarr (its native /api/v1/search, +-- because Prowlarr has no aggregate Torznab feed), torznab (any +-- single feed), qbittorrent (the WebUI v2), exchange. +-- +-- FabledExchange is ONE row under Download clients with a switch per job: +-- downloading, searching its DHT index, and describing a torrent's files and +-- swarm before a grab. The serves_* switches mean something on an exchange +-- row only; every other row does the one job its kind names. +-- +-- Secrets (api_key, password) are write-only: no query the API answers with +-- returns them, only whether one is set. +CREATE TABLE acquisition_services ( + -- code is the stable identity: the protocol for the first of its kind, + -- then protocol-2, protocol-3. Legible in a log, never typed. + code text PRIMARY KEY, + kind text NOT NULL + CONSTRAINT acquisition_services_kind_check CHECK (kind IN ('search', 'download')), + protocol text NOT NULL + CONSTRAINT acquisition_services_protocol_check + CHECK (protocol IN ('prowlarr', 'torznab', 'qbittorrent', 'exchange')), + label text NOT NULL, + -- A new row is off: it has no address yet. + enabled boolean NOT NULL DEFAULT false, + -- The address as an admin's browser opens it. For a Torznab feed it + -- is the whole feed URL up to and including /api. + base_url text NOT NULL DEFAULT '', + api_key text NOT NULL DEFAULT '', + -- qBittorrent only. Both may be empty: a WebUI with auth bypassed for + -- local addresses takes neither. + username text NOT NULL DEFAULT '', + password text NOT NULL DEFAULT '', + -- Download clients: everything Minstrel adds is filed under it. + category text NOT NULL DEFAULT '', + timeout_ms integer NOT NULL DEFAULT 30000 + CONSTRAINT acquisition_services_timeout_check CHECK (timeout_ms BETWEEN 1000 AND 300000), + serves_search boolean NOT NULL DEFAULT true, + serves_downloads boolean NOT NULL DEFAULT true, + serves_enrichment boolean NOT NULL DEFAULT true, + -- One downloads folder, two names: where Minstrel has it mounted, and + -- where the client sees it (empty = the same path, nothing translated). + local_path text NOT NULL DEFAULT '/downloads' + CONSTRAINT acquisition_services_local_path_check CHECK (local_path LIKE '/%'), + remote_path text NOT NULL DEFAULT '' + CONSTRAINT acquisition_services_remote_path_check CHECK (remote_path = '' OR remote_path LIKE '/%'), + -- The most searches the schedule sends this source an hour; 0 = unpaced. + -- Prowlarr's indexers answer 429 when asked too often; an Exchange + -- searches its own database and starts unpaced. + searches_per_hour integer NOT NULL DEFAULT 120 + CONSTRAINT acquisition_services_searches_per_hour_check CHECK (searches_per_hour BETWEEN 0 AND 3600), + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); diff --git a/internal/db/migrations/0080_release_catalogue.down.sql b/internal/db/migrations/0080_release_catalogue.down.sql new file mode 100644 index 00000000..bb9e6ac3 --- /dev/null +++ b/internal/db/migrations/0080_release_catalogue.down.sql @@ -0,0 +1,5 @@ +DROP TABLE IF EXISTS mb_release_tracks; +DROP TABLE IF EXISTS mb_releases; +DROP TABLE IF EXISTS mb_artist_release_groups; +DROP TABLE IF EXISTS mb_release_groups; +DROP TABLE IF EXISTS mb_artists; diff --git a/internal/db/migrations/0080_release_catalogue.up.sql b/internal/db/migrations/0080_release_catalogue.up.sql new file mode 100644 index 00000000..88e3086c --- /dev/null +++ b/internal/db/migrations/0080_release_catalogue.up.sql @@ -0,0 +1,86 @@ +-- 0080_release_catalogue.up.sql — Minstrel's own copy of what artists have +-- released, read from MusicBrainz. +-- +-- With Lidarr gone, the wanted list, search and import need to know an +-- artist's discography and each release's track list. MusicBrainz allows one +-- request a second, so what has been read is kept here and refreshed on a +-- schedule rather than asked again for every page. +-- +-- mb_artists artists the catalogue has been asked about. +-- mb_release_groups what a person calls an album (Lidarr's albums +-- were release groups too), with MusicBrainz's +-- primary and secondary types. +-- mb_artist_release_groups who is credited on each group: a collaboration +-- belongs to every credited artist's discography. +-- mb_releases the editions of a group, read when something +-- needs a track list. +-- mb_release_tracks each release's track list. +-- +-- Dates stay as MusicBrainz's partial-date text ("1997", "1997-05", +-- "1997-05-21"): most early releases have only a year. + +CREATE TABLE mb_artists ( + mbid text PRIMARY KEY, + name text NOT NULL, + sort_name text NOT NULL DEFAULT '', + disambiguation text NOT NULL DEFAULT '', + artist_type text NOT NULL DEFAULT '', + country text NOT NULL DEFAULT '', + -- When the discography was last read in full; NULL means never. + discography_refreshed_at timestamptz, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX mb_artists_refresh_idx ON mb_artists (discography_refreshed_at NULLS FIRST); + +CREATE TABLE mb_release_groups ( + mbid text PRIMARY KEY, + title text NOT NULL, + primary_type text NOT NULL DEFAULT '', + secondary_types text[] NOT NULL DEFAULT '{}', + first_release_date text NOT NULL DEFAULT '', + artist_credit text NOT NULL DEFAULT '', + -- When the releases and their track lists were last read; NULL means never. + releases_refreshed_at timestamptz, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE mb_artist_release_groups ( + artist_mbid text NOT NULL REFERENCES mb_artists (mbid) ON DELETE CASCADE, + release_group_mbid text NOT NULL REFERENCES mb_release_groups (mbid) ON DELETE CASCADE, + PRIMARY KEY (artist_mbid, release_group_mbid) +); + +CREATE INDEX mb_artist_release_groups_group_idx ON mb_artist_release_groups (release_group_mbid); + +CREATE TABLE mb_releases ( + mbid text PRIMARY KEY, + release_group_mbid text NOT NULL REFERENCES mb_release_groups (mbid) ON DELETE CASCADE, + title text NOT NULL, + status text NOT NULL DEFAULT '', + release_date text NOT NULL DEFAULT '', + country text NOT NULL DEFAULT '', + disambiguation text NOT NULL DEFAULT '', + formats text NOT NULL DEFAULT '', + track_count integer NOT NULL DEFAULT 0, + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX mb_releases_group_idx ON mb_releases (release_group_mbid); + +CREATE TABLE mb_release_tracks ( + release_mbid text NOT NULL REFERENCES mb_releases (mbid) ON DELETE CASCADE, + medium_position integer NOT NULL, + position integer NOT NULL, + medium_format text NOT NULL DEFAULT '', + number text NOT NULL DEFAULT '', + title text NOT NULL, + length_ms integer NOT NULL DEFAULT 0, + recording_mbid text NOT NULL DEFAULT '', + PRIMARY KEY (release_mbid, medium_position, position) +); + +CREATE INDEX mb_release_tracks_recording_idx ON mb_release_tracks (recording_mbid) + WHERE recording_mbid <> ''; diff --git a/internal/db/migrations/0081_wanted.down.sql b/internal/db/migrations/0081_wanted.down.sql new file mode 100644 index 00000000..4087ae4d --- /dev/null +++ b/internal/db/migrations/0081_wanted.down.sql @@ -0,0 +1,38 @@ +ALTER TABLE reacquisition_settings + ADD COLUMN backoff_base_hours int NOT NULL DEFAULT 6, + ADD COLUMN backoff_max_hours int NOT NULL DEFAULT 168, + ADD COLUMN max_attempts int NOT NULL DEFAULT 3, + ADD COLUMN auto_approve boolean NOT NULL DEFAULT true; + +ALTER INDEX requests_user_id_idx RENAME TO lidarr_requests_user_id_idx; +ALTER INDEX requests_status_idx RENAME TO lidarr_requests_status_idx; +ALTER INDEX requests_artist_mbid_idx RENAME TO lidarr_requests_artist_mbid_idx; +ALTER INDEX requests_album_mbid_idx RENAME TO lidarr_requests_album_mbid_idx; +ALTER TABLE requests ADD COLUMN lidarr_add_confirmed_at timestamptz; +ALTER TABLE requests ADD COLUMN root_folder_path text; +ALTER TABLE requests ADD COLUMN quality_profile_id int; +ALTER TABLE requests RENAME COLUMN track_mbid TO lidarr_track_mbid; +ALTER TABLE requests RENAME COLUMN album_mbid TO lidarr_album_mbid; +ALTER TABLE requests RENAME COLUMN artist_mbid TO lidarr_artist_mbid; +ALTER TYPE request_kind RENAME TO lidarr_request_kind; +ALTER TYPE request_status RENAME TO lidarr_request_status; +ALTER TABLE requests RENAME TO lidarr_requests; + +CREATE TABLE missing_reacquisitions ( + album_id uuid PRIMARY KEY REFERENCES albums (id) ON DELETE CASCADE, + attempts int NOT NULL DEFAULT 0, + last_attempt_at timestamptz, + last_request_id uuid REFERENCES lidarr_requests (id) ON DELETE SET NULL, + gave_up_at timestamptz, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT missing_reacquisitions_attempts_nonneg CHECK (attempts >= 0) +); +CREATE INDEX missing_reacquisitions_due_idx + ON missing_reacquisitions (last_attempt_at NULLS FIRST) + WHERE gave_up_at IS NULL; + +DROP TABLE wants; +DROP TABLE follow_seen_groups; +DROP TABLE follows; +DROP TABLE release_type_profiles; diff --git a/internal/db/migrations/0081_wanted.up.sql b/internal/db/migrations/0081_wanted.up.sql new file mode 100644 index 00000000..d358ab3d --- /dev/null +++ b/internal/db/migrations/0081_wanted.up.sql @@ -0,0 +1,165 @@ +-- 0081_wanted.up.sql — the wanted list. +-- +-- With Lidarr gone, Minstrel keeps its own record of what it is looking for: +-- +-- release_type_profiles which kinds of release a follow wants: primary +-- types (Album, EP, Single…) and the secondary types +-- allowed (Live, Compilation, Remix…). An empty +-- secondary list means studio releases only, which +-- is how live recordings are left out. +-- follows artists whose new releases are wanted, each on a +-- profile. +-- follow_seen_groups release groups a follow has already judged, so +-- each is judged once. +-- wants release groups wanted: a request, a followed +-- artist's new release, or an album whose files went +-- missing. A want is held once the library has a +-- full copy (internal/wanted). +-- +-- lidarr_requests becomes requests with no Lidarr columns: approving a request +-- now records a want or a follow instead of sending an add to Lidarr. The +-- missing-file sweeper's attempt bookkeeping goes too; retrying belongs to the +-- search schedule, which works per want. + +CREATE TABLE release_type_profiles ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + name text NOT NULL UNIQUE, + primary_types text[] NOT NULL, + secondary_types text[] NOT NULL DEFAULT '{}', + is_default boolean NOT NULL DEFAULT false, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT release_type_profiles_primary_nonempty CHECK (cardinality(primary_types) > 0) +); + +-- At most one default; the service keeps exactly one by moving it. +CREATE UNIQUE INDEX release_type_profiles_one_default + ON release_type_profiles (is_default) WHERE is_default; + +-- Lidarr's "Standard" metadata profile, which followed artists used: albums, +-- EPs and singles, studio releases only. +INSERT INTO release_type_profiles (name, primary_types, secondary_types, is_default) +VALUES ('Standard', ARRAY['Album', 'EP', 'Single'], '{}', true); + +CREATE TABLE follows ( + artist_mbid text PRIMARY KEY, + artist_name text NOT NULL, + -- RESTRICT: a profile in use cannot be deleted out from under a follow. + profile_id uuid NOT NULL REFERENCES release_type_profiles (id) ON DELETE RESTRICT, + followed_by uuid REFERENCES users (id) ON DELETE SET NULL, + followed_at timestamptz NOT NULL DEFAULT now(), + -- When the first discography read marked everything then listed as seen. + -- NULL until then: a follow wants only what appears after it. + baseline_at timestamptz, + -- Whether the baseline also wants what the artist had already released + -- that the profile passes and the library lacks: an approved artist + -- request, or an admin ticking "also want…". Cleared once applied. + want_existing boolean NOT NULL DEFAULT false, + -- The discography read this follow last applied. + checked_at timestamptz, + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX follows_profile_idx ON follows (profile_id); + +CREATE TABLE follow_seen_groups ( + artist_mbid text NOT NULL REFERENCES follows (artist_mbid) ON DELETE CASCADE, + release_group_mbid text NOT NULL, + seen_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (artist_mbid, release_group_mbid) +); + +CREATE TABLE wants ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + release_group_mbid text NOT NULL UNIQUE, + artist_mbid text NOT NULL, + artist_name text NOT NULL, + title text NOT NULL, + primary_type text NOT NULL DEFAULT '', + first_release_date text NOT NULL DEFAULT '', + reason text NOT NULL + CONSTRAINT wants_reason_check CHECK (reason IN ('request', 'follow', 'missing')), + -- The library album this want is about (a missing want), or the one that + -- answered it. + album_id uuid REFERENCES albums (id) ON DELETE SET NULL, + created_by uuid REFERENCES users (id) ON DELETE SET NULL, + -- When it was last wanted: set again when a held want is reopened, so + -- "arrived since it was wanted" measures from the reopening. + wanted_at timestamptz NOT NULL DEFAULT now(), + fulfilled_at timestamptz, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX wants_open_idx ON wants (wanted_at) WHERE fulfilled_at IS NULL; +CREATE INDEX wants_artist_idx ON wants (artist_mbid); + +-- Requests lose Lidarr. +DROP TABLE missing_reacquisitions; + +ALTER TABLE lidarr_requests RENAME TO requests; +ALTER TYPE lidarr_request_status RENAME TO request_status; +ALTER TYPE lidarr_request_kind RENAME TO request_kind; +ALTER TABLE requests RENAME COLUMN lidarr_artist_mbid TO artist_mbid; +ALTER TABLE requests RENAME COLUMN lidarr_album_mbid TO album_mbid; +ALTER TABLE requests RENAME COLUMN lidarr_track_mbid TO track_mbid; +ALTER TABLE requests DROP COLUMN quality_profile_id; +ALTER TABLE requests DROP COLUMN root_folder_path; +ALTER TABLE requests DROP COLUMN lidarr_add_confirmed_at; +ALTER INDEX lidarr_requests_user_id_idx RENAME TO requests_user_id_idx; +ALTER INDEX lidarr_requests_status_idx RENAME TO requests_status_idx; +ALTER INDEX lidarr_requests_artist_mbid_idx RENAME TO requests_artist_mbid_idx; +ALTER INDEX lidarr_requests_album_mbid_idx RENAME TO requests_album_mbid_idx; + +-- Older requests may name albums by release id. A want is keyed by +-- release group, so repoint every request the library can translate. +UPDATE requests r + SET album_mbid = a.release_group_mbid, + updated_at = now() + FROM albums a + WHERE a.mbid = r.album_mbid + AND a.release_group_mbid IS NOT NULL + AND a.release_group_mbid <> r.album_mbid; + +-- Approved album and track requests still waiting become wants, judged by +-- the same test as any other (internal/wanted). +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, created_by, wanted_at) +SELECT DISTINCT ON (r.album_mbid) + r.album_mbid, r.artist_mbid, r.artist_name, COALESCE(r.album_title, ''), 'request', r.user_id, r.requested_at + FROM requests r + WHERE r.status = 'approved' + AND r.kind IN ('album', 'track') + AND r.album_mbid IS NOT NULL + ORDER BY r.album_mbid, r.requested_at +ON CONFLICT (release_group_mbid) DO NOTHING; + +-- The library's album artists start out followed on the default profile: the +-- artists already in the library are the wanted artists. Various Artists and +-- MusicBrainz's bracketed placeholders ([unknown], [anonymous]…) are not +-- artists anyone follows. +INSERT INTO follows (artist_mbid, artist_name, profile_id) +SELECT ar.mbid, ar.name, p.id + FROM artists ar + CROSS JOIN (SELECT id FROM release_type_profiles WHERE is_default) p + WHERE ar.mbid IS NOT NULL AND ar.mbid <> '' + AND ar.mbid <> '89ad4ac3-39f7-470e-963a-56509c546377' + AND ar.name NOT LIKE '[%]' + AND EXISTS (SELECT 1 FROM albums al WHERE al.artist_id = ar.id) +ON CONFLICT (artist_mbid) DO NOTHING; + +-- An approved artist request was an artist added to Lidarr: it is followed. +INSERT INTO follows (artist_mbid, artist_name, profile_id, followed_by) +SELECT DISTINCT ON (r.artist_mbid) r.artist_mbid, r.artist_name, p.id, r.decided_by + FROM requests r + CROSS JOIN (SELECT id FROM release_type_profiles WHERE is_default) p + WHERE r.status = 'approved' AND r.kind = 'artist' + ORDER BY r.artist_mbid, r.requested_at +ON CONFLICT (artist_mbid) DO NOTHING; + +-- The sweeper now only records a want; how often to look again is the +-- search schedule's business. +ALTER TABLE reacquisition_settings + DROP COLUMN backoff_base_hours, + DROP COLUMN backoff_max_hours, + DROP COLUMN max_attempts, + DROP COLUMN auto_approve; diff --git a/internal/db/migrations/0082_search_and_grab.down.sql b/internal/db/migrations/0082_search_and_grab.down.sql new file mode 100644 index 00000000..630d58e6 --- /dev/null +++ b/internal/db/migrations/0082_search_and_grab.down.sql @@ -0,0 +1,12 @@ +DROP TABLE release_blocks; +DROP TABLE grabs; +DROP TABLE want_candidates; +DROP INDEX wants_search_due_idx; +ALTER TABLE wants + DROP COLUMN searched_at, + DROP COLUMN search_outcome, + DROP COLUMN search_considered, + DROP COLUMN best_release, + DROP COLUMN best_verdict, + DROP COLUMN search_error; +DROP TABLE acquisition_settings; diff --git a/internal/db/migrations/0082_search_and_grab.up.sql b/internal/db/migrations/0082_search_and_grab.up.sql new file mode 100644 index 00000000..962586a1 --- /dev/null +++ b/internal/db/migrations/0082_search_and_grab.up.sql @@ -0,0 +1,101 @@ +-- 0082_search_and_grab.up.sql — searching for wants and grabbing releases. +-- +-- acquisition_settings one row: whether the schedule takes the best release +-- by itself, how often a want is searched again, which +-- quality tiers are taken, and the last pass's report. +-- wants (columns) when each want was last searched and what came of it. +-- want_candidates every release the last search found for a want, with +-- its verdict and the reasons behind it. +-- grabs downloads started for wants; at most one live per want. +-- release_blocks releases never to take again, by infohash. + +CREATE TABLE acquisition_settings ( + id boolean PRIMARY KEY DEFAULT true CONSTRAINT acquisition_settings_one_row CHECK (id), + auto_grab boolean NOT NULL DEFAULT true, + -- 0 switches the schedule off. + search_interval_hours integer NOT NULL DEFAULT 24 + CONSTRAINT acquisition_settings_interval_check CHECK (search_interval_hours BETWEEN 0 AND 720), + -- As Lidarr's "Any" profile: trash lossy to lossless; unknown off. + allowed_qualities text[] NOT NULL DEFAULT ARRAY['lossless_hires', 'lossless', 'lossy_high', + 'lossy_standard', 'lossy_low', 'lossy_poor', 'lossy_trash'], + last_pass_at timestamptz, + last_pass jsonb NOT NULL DEFAULT '{}', + updated_at timestamptz NOT NULL DEFAULT now() +); + +INSERT INTO acquisition_settings DEFAULT VALUES; + +ALTER TABLE wants + ADD COLUMN searched_at timestamptz, + ADD COLUMN search_outcome text NOT NULL DEFAULT '' + CONSTRAINT wants_search_outcome_check CHECK (search_outcome IN ('', 'grabbed', 'waiting', 'nothing', 'error')), + ADD COLUMN search_considered integer NOT NULL DEFAULT 0, + ADD COLUMN best_release text NOT NULL DEFAULT '', + ADD COLUMN best_verdict text NOT NULL DEFAULT '', + ADD COLUMN search_error text NOT NULL DEFAULT ''; + +CREATE INDEX wants_search_due_idx ON wants (searched_at NULLS FIRST) WHERE fulfilled_at IS NULL; + +CREATE TABLE want_candidates ( + want_id uuid NOT NULL REFERENCES wants (id) ON DELETE CASCADE, + -- The infohash, or the name when the source gave none. + key text NOT NULL, + name text NOT NULL, + info_hash text NOT NULL DEFAULT '', + indexer text NOT NULL DEFAULT '', + source text NOT NULL DEFAULT '', + size_bytes bigint NOT NULL DEFAULT 0, + seeders integer NOT NULL DEFAULT 0, + files_count integer NOT NULL DEFAULT 0, + format text NOT NULL DEFAULT '', + quality text NOT NULL DEFAULT '', + year integer NOT NULL DEFAULT 0, + verdict text NOT NULL + CONSTRAINT want_candidates_verdict_check CHECK (verdict IN ('grab', 'review', 'refused', 'rejected', 'blocked')), + reason text NOT NULL DEFAULT '', + reasons jsonb NOT NULL DEFAULT '[]', + folded_from text[] NOT NULL DEFAULT '{}', + -- Position in the search's order, best first. + rank integer NOT NULL DEFAULT 0, + searched_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (want_id, key) +); + +CREATE INDEX want_candidates_info_hash_idx ON want_candidates (info_hash) WHERE info_hash <> ''; + +CREATE TABLE grabs ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + want_id uuid REFERENCES wants (id) ON DELETE SET NULL, + info_hash text NOT NULL DEFAULT '', + name text NOT NULL, + key text NOT NULL, + indexer text NOT NULL DEFAULT '', + source text NOT NULL DEFAULT '', + -- The acquisition_services row of the download client it went to. + client text NOT NULL, + save_path text NOT NULL, + quality text NOT NULL DEFAULT '', + origin text NOT NULL CONSTRAINT grabs_origin_check CHECK (origin IN ('schedule', 'operator')), + forced boolean NOT NULL DEFAULT false, + grabbed_by uuid REFERENCES users (id) ON DELETE SET NULL, + grabbed_at timestamptz NOT NULL DEFAULT now(), + ended_at timestamptz, + end_reason text NOT NULL DEFAULT '' + CONSTRAINT grabs_end_reason_check CHECK (end_reason IN ('', 'superseded')), + end_detail text NOT NULL DEFAULT '' +); + +-- One live grab per want; the service decides which, this is the backstop. +CREATE UNIQUE INDEX grabs_one_live_per_want ON grabs (want_id) WHERE ended_at IS NULL AND want_id IS NOT NULL; +CREATE INDEX grabs_info_hash_idx ON grabs (info_hash) WHERE info_hash <> ''; + +CREATE TABLE release_blocks ( + info_hash text PRIMARY KEY, + name text NOT NULL DEFAULT '', + -- operator: "never this one"; files: its .torrent held the wrong thing. + reason text NOT NULL CONSTRAINT release_blocks_reason_check CHECK (reason IN ('operator', 'files')), + detail text NOT NULL DEFAULT '', + want_id uuid REFERENCES wants (id) ON DELETE SET NULL, + blocked_by uuid REFERENCES users (id) ON DELETE SET NULL, + blocked_at timestamptz NOT NULL DEFAULT now() +); diff --git a/internal/db/migrations/0083_import.down.sql b/internal/db/migrations/0083_import.down.sql new file mode 100644 index 00000000..a7677dcb --- /dev/null +++ b/internal/db/migrations/0083_import.down.sql @@ -0,0 +1,24 @@ +ALTER TABLE acquisition_settings DROP COLUMN import_root; + +DELETE FROM release_blocks WHERE reason = 'import'; +ALTER TABLE release_blocks DROP CONSTRAINT release_blocks_reason_check; +ALTER TABLE release_blocks ADD CONSTRAINT release_blocks_reason_check + CHECK (reason IN ('operator', 'files')); + +DROP INDEX grabs_live_idx; + +UPDATE grabs SET end_reason = 'superseded' WHERE end_reason IN ('imported', 'refused', 'held'); +ALTER TABLE grabs DROP CONSTRAINT grabs_end_reason_check; +ALTER TABLE grabs ADD CONSTRAINT grabs_end_reason_check CHECK (end_reason IN ('', 'superseded')); + +ALTER TABLE grabs + DROP COLUMN progress, + DROP COLUMN import_state, + DROP COLUMN import_reason, + DROP COLUMN import_detail, + DROP COLUMN import_forced, + DROP COLUMN import_attempts, + DROP COLUMN import_updated_at, + DROP COLUMN imported_at, + DROP COLUMN imported_files, + DROP COLUMN destination; diff --git a/internal/db/migrations/0083_import.up.sql b/internal/db/migrations/0083_import.up.sql new file mode 100644 index 00000000..0cf4d5bb --- /dev/null +++ b/internal/db/migrations/0083_import.up.sql @@ -0,0 +1,37 @@ +-- Import with verification: a finished download is checked +-- against the album it was grabbed for before anything reaches the library. +-- +-- grabs (columns) how far the download has got, and what the import +-- made of it: the verdict, its reason and evidence, +-- where the files went. +-- grabs.end_reason a grab also ends by being imported, refused, or +-- found to be an album already held. +-- release_blocks.reason import: the files were not the album. +-- acquisition_settings which library folder new music is filed into +-- ('' = the first scan path). +ALTER TABLE grabs + ADD COLUMN progress real NOT NULL DEFAULT 0, + ADD COLUMN import_state text NOT NULL DEFAULT '' + CONSTRAINT grabs_import_state_check CHECK (import_state IN + ('', 'downloading', 'waiting', 'importing', 'imported', 'refused', 'review', 'held', 'failed')), + ADD COLUMN import_reason text NOT NULL DEFAULT '', + ADD COLUMN import_detail jsonb NOT NULL DEFAULT '{}', + -- A person said "import anyway" to a download the check held for them. + ADD COLUMN import_forced boolean NOT NULL DEFAULT false, + ADD COLUMN import_attempts integer NOT NULL DEFAULT 0, + ADD COLUMN import_updated_at timestamptz, + ADD COLUMN imported_at timestamptz, + ADD COLUMN imported_files integer NOT NULL DEFAULT 0, + ADD COLUMN destination text NOT NULL DEFAULT ''; + +ALTER TABLE grabs DROP CONSTRAINT grabs_end_reason_check; +ALTER TABLE grabs ADD CONSTRAINT grabs_end_reason_check + CHECK (end_reason IN ('', 'superseded', 'imported', 'refused', 'held')); + +CREATE INDEX grabs_live_idx ON grabs (grabbed_at) WHERE ended_at IS NULL; + +ALTER TABLE release_blocks DROP CONSTRAINT release_blocks_reason_check; +ALTER TABLE release_blocks ADD CONSTRAINT release_blocks_reason_check + CHECK (reason IN ('operator', 'files', 'import')); + +ALTER TABLE acquisition_settings ADD COLUMN import_root text NOT NULL DEFAULT ''; diff --git a/internal/db/migrations/0084_download_lifecycle.down.sql b/internal/db/migrations/0084_download_lifecycle.down.sql new file mode 100644 index 00000000..9e00a4d5 --- /dev/null +++ b/internal/db/migrations/0084_download_lifecycle.down.sql @@ -0,0 +1,23 @@ +ALTER TABLE acquisition_settings + DROP COLUMN seed_ratio, + DROP COLUMN seed_days, + DROP COLUMN stall_hours; + +ALTER TABLE release_blocks DROP COLUMN expires_at; +DELETE FROM release_blocks WHERE reason = 'stalled'; +ALTER TABLE release_blocks DROP CONSTRAINT release_blocks_reason_check; +ALTER TABLE release_blocks ADD CONSTRAINT release_blocks_reason_check + CHECK (reason IN ('operator', 'files', 'import')); + +DROP INDEX grabs_seeding_idx; +ALTER TABLE grabs + DROP COLUMN seeding, + DROP COLUMN ratio, + DROP COLUMN seed_released_at; + +UPDATE grabs SET end_reason = 'superseded' WHERE end_reason IN ('stalled', 'failed', 'abandoned', 'gone'); +ALTER TABLE grabs DROP CONSTRAINT grabs_end_reason_check; +ALTER TABLE grabs ADD CONSTRAINT grabs_end_reason_check + CHECK (end_reason IN ('', 'superseded', 'imported', 'refused', 'held')); + +DROP TABLE download_interventions; diff --git a/internal/db/migrations/0084_download_lifecycle.up.sql b/internal/db/migrations/0084_download_lifecycle.up.sql new file mode 100644 index 00000000..eedda40a --- /dev/null +++ b/internal/db/migrations/0084_download_lifecycle.up.sql @@ -0,0 +1,89 @@ +-- Download lifecycle: a download that stalls or fails is given up and its album +-- searched again; an imported download seeds, then is removed. +-- +-- download_interventions what was seen wrong with a download or a client, +-- what Minstrel did about it, and whether it +-- cleared. One open row per (client, download, +-- condition), updated in place. +-- grabs (columns) a grab also ends by being given up (stalled, +-- failed), set aside by a person, or leaving the +-- client; an imported one records its seeding. +-- release_blocks a release given up on is held back for a while, +-- not for ever: expires_at. +-- acquisition_settings how long to seed, and how long a download may sit +-- with nobody sharing it. +CREATE TABLE download_interventions ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + -- The acquisition_services code of the download client. + client text NOT NULL, + -- The torrent, or '' for a condition of the client itself. + info_hash text NOT NULL DEFAULT '', + grab_id uuid REFERENCES grabs (id) ON DELETE SET NULL, + name text NOT NULL DEFAULT '', + -- no_peers nobody is sharing an unfinished download. + -- no_metadata still fetching the torrent's file list. + -- client_error the client gave up on it, in its own words. + -- client_unreachable the client does not answer, or its engine is down. + -- gone the client no longer lists a running download. + -- not_removed given up on, and the client would not remove it. + trigger text NOT NULL CONSTRAINT download_interventions_trigger_check CHECK (trigger IN ( + 'no_peers', 'no_metadata', 'client_error', 'client_unreachable', 'gone', 'not_removed')), + -- escalated seen and shown; nothing done yet. + -- retried asked the trackers again and re-read the files. + -- given_up the download was removed and its album searched again. + -- removed a download left behind was removed. + action text NOT NULL DEFAULT 'escalated' + CONSTRAINT download_interventions_action_check CHECK (action IN ('escalated', 'retried', 'given_up', 'removed')), + outcome text NOT NULL DEFAULT 'open' + CONSTRAINT download_interventions_outcome_check CHECK (outcome IN ('open', 'resolved')), + detail text NOT NULL DEFAULT '', + -- How many passes have seen it, and how many times Minstrel acted on it. + -- The second is the retry ladder's rung, kept here so a restart does not + -- hand a stuck download fresh retries. + observations integer NOT NULL DEFAULT 1, + remediations integer NOT NULL DEFAULT 0, + first_seen timestamptz NOT NULL DEFAULT now(), + last_seen timestamptz NOT NULL DEFAULT now(), + settled_at timestamptz, + CONSTRAINT download_interventions_settled_together CHECK ((outcome = 'open') = (settled_at IS NULL)) +); + +-- One open row per condition: a pass that sees it again lands on the same row. +CREATE UNIQUE INDEX download_interventions_one_open + ON download_interventions (client, info_hash, trigger) WHERE outcome = 'open'; +CREATE INDEX download_interventions_grab_idx ON download_interventions (grab_id) WHERE grab_id IS NOT NULL; + +ALTER TABLE grabs DROP CONSTRAINT grabs_end_reason_check; +ALTER TABLE grabs ADD CONSTRAINT grabs_end_reason_check + CHECK (end_reason IN ('', 'superseded', 'imported', 'refused', 'held', 'stalled', 'failed', 'abandoned', 'gone')); + +-- An imported download goes on sharing (the files were copied) until its +-- ratio reaches the setting or its time is up; seed_released_at is when it +-- was removed, or found gone. +ALTER TABLE grabs + ADD COLUMN seeding boolean NOT NULL DEFAULT false, + ADD COLUMN ratio real NOT NULL DEFAULT 0, + ADD COLUMN seed_released_at timestamptz; + +CREATE INDEX grabs_seeding_idx ON grabs (imported_at) WHERE seeding AND seed_released_at IS NULL; + +-- stalled: given up on because it would not finish; held back until +-- expires_at. NULL is for good. +ALTER TABLE release_blocks DROP CONSTRAINT release_blocks_reason_check; +ALTER TABLE release_blocks ADD CONSTRAINT release_blocks_reason_check + CHECK (reason IN ('operator', 'files', 'import', 'stalled')); +ALTER TABLE release_blocks ADD COLUMN expires_at timestamptz; + +-- FL's rulings: seed to ratio 2.0 or for 14 days at most; a download nobody +-- has shared for 6 hours is retried, then given up. +-- seed_ratio 0 the download is removed once imported. +-- seed_days 0 no time limit: only the ratio ends it. +-- stall_hours 0 never give up on a download for want of peers (a failed +-- one is still given up). +ALTER TABLE acquisition_settings + ADD COLUMN seed_ratio real NOT NULL DEFAULT 2.0 + CONSTRAINT acquisition_settings_seed_ratio_check CHECK (seed_ratio BETWEEN 0 AND 100), + ADD COLUMN seed_days integer NOT NULL DEFAULT 14 + CONSTRAINT acquisition_settings_seed_days_check CHECK (seed_days BETWEEN 0 AND 365), + ADD COLUMN stall_hours integer NOT NULL DEFAULT 6 + CONSTRAINT acquisition_settings_stall_hours_check CHECK (stall_hours BETWEEN 0 AND 720); diff --git a/internal/db/migrations/0085_library_wants.down.sql b/internal/db/migrations/0085_library_wants.down.sql new file mode 100644 index 00000000..3fe34736 --- /dev/null +++ b/internal/db/migrations/0085_library_wants.down.sql @@ -0,0 +1,8 @@ +ALTER TABLE acquisition_settings + DROP COLUMN library_listing_started_at, + DROP COLUMN library_listed_at, + DROP COLUMN library_unlisted; + +DELETE FROM wants WHERE reason = 'library'; +ALTER TABLE wants DROP CONSTRAINT wants_reason_check; +ALTER TABLE wants ADD CONSTRAINT wants_reason_check CHECK (reason IN ('request', 'follow', 'missing')); diff --git a/internal/db/migrations/0085_library_wants.up.sql b/internal/db/migrations/0085_library_wants.up.sql new file mode 100644 index 00000000..9f7b87ce --- /dev/null +++ b/internal/db/migrations/0085_library_wants.up.sql @@ -0,0 +1,23 @@ +-- The library's albums go on the wanted list as held, once. After that the list +-- is the truth: an album taken off it stays off, and one dropped into the +-- library folder by hand is not added. Re-acquisition fetches back only albums +-- on the list. +-- +-- wants.reason library: on the list because the library +-- held it when the list began. +-- acquisition_settings when the one-time listing began (its cutoff: +-- albums added later are not listed), when it +-- finished, and how many albums it could not +-- list because nothing names them to +-- MusicBrainz. +-- +-- The listing itself runs in the wanted worker after boot, not here: it may +-- have to ask MusicBrainz for each album's release group. +ALTER TABLE wants DROP CONSTRAINT wants_reason_check; +ALTER TABLE wants ADD CONSTRAINT wants_reason_check + CHECK (reason IN ('request', 'follow', 'missing', 'library')); + +ALTER TABLE acquisition_settings + ADD COLUMN library_listing_started_at timestamptz, + ADD COLUMN library_listed_at timestamptz, + ADD COLUMN library_unlisted integer NOT NULL DEFAULT 0; diff --git a/internal/db/migrations/0086_library_album_matches.down.sql b/internal/db/migrations/0086_library_album_matches.down.sql new file mode 100644 index 00000000..b61ff4fa --- /dev/null +++ b/internal/db/migrations/0086_library_album_matches.down.sql @@ -0,0 +1 @@ +DROP TABLE library_album_matches; diff --git a/internal/db/migrations/0086_library_album_matches.up.sql b/internal/db/migrations/0086_library_album_matches.up.sql new file mode 100644 index 00000000..fd7986da --- /dev/null +++ b/internal/db/migrations/0086_library_album_matches.up.sql @@ -0,0 +1,45 @@ +-- Every library album is matched to MusicBrainz before it goes on the list, so +-- its metadata comes from MusicBrainz. The listing in 0085 took only albums +-- whose tags already named their release group and artist; the rest are now +-- matched by the release their tags name, the artist's discography, or a +-- search, and checked against their own tracks. +-- +-- library_album_matches one row per album the listing has decided: +-- listed (with how it was matched) or unmatched +-- (with why). It is the record that the listing +-- saw the album, so an album taken off the list +-- is never listed again. +CREATE TABLE library_album_matches ( + album_id uuid PRIMARY KEY REFERENCES albums (id) ON DELETE CASCADE, + outcome text NOT NULL + CONSTRAINT library_album_matches_outcome_check CHECK (outcome IN ('listed', 'unmatched')), + -- tags its tags named the release group. + -- release its tags named a release; MusicBrainz named the group. + -- discography its title is one of the artist's release groups. + -- search MusicBrainz's search found it by artist and title. + method text NOT NULL DEFAULT '' + CONSTRAINT library_album_matches_method_check CHECK (method IN ('', 'tags', 'release', 'discography', 'search')), + release_group_mbid text NOT NULL DEFAULT '', + detail text NOT NULL DEFAULT '', + decided_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX library_album_matches_unmatched_idx ON library_album_matches (decided_at) WHERE outcome = 'unmatched'; + +-- The albums 0085's listing already took, by its own test, are decided: +-- one taken off the list since stays off. +INSERT INTO library_album_matches (album_id, outcome, method, release_group_mbid) +SELECT a.id, 'listed', 'tags', a.release_group_mbid + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + JOIN acquisition_settings s ON s.id = true + WHERE s.library_listing_started_at IS NOT NULL + AND a.created_at <= s.library_listing_started_at + AND a.release_group_mbid IS NOT NULL AND a.release_group_mbid <> '' + AND ar.mbid IS NOT NULL AND ar.mbid <> '' + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id); + +-- The rest are matched by the wanted worker after boot: the listing resumes. +UPDATE acquisition_settings + SET library_listed_at = NULL, library_unlisted = 0 + WHERE library_listing_started_at IS NOT NULL; diff --git a/internal/db/migrations/0087_library_rematch.down.sql b/internal/db/migrations/0087_library_rematch.down.sql new file mode 100644 index 00000000..66b5e525 --- /dev/null +++ b/internal/db/migrations/0087_library_rematch.down.sql @@ -0,0 +1,2 @@ +-- Nothing to undo: the refused albums are matched again either way. +SELECT 1; diff --git a/internal/db/migrations/0087_library_rematch.up.sql b/internal/db/migrations/0087_library_rematch.up.sql new file mode 100644 index 00000000..cc278ed9 --- /dev/null +++ b/internal/db/migrations/0087_library_rematch.up.sql @@ -0,0 +1,9 @@ +-- The first matching (0086) required every track's length to fit as well +-- as its title, and refused singles and near-complete albums the library +-- plainly holds. Matching now takes the titles alone under the album's own +-- name: the albums it refused are matched again. +DELETE FROM library_album_matches WHERE outcome = 'unmatched'; + +UPDATE acquisition_settings + SET library_listed_at = NULL, library_unlisted = 0 + WHERE library_listing_started_at IS NOT NULL; diff --git a/internal/db/migrations/0088_library_matcher_version.down.sql b/internal/db/migrations/0088_library_matcher_version.down.sql new file mode 100644 index 00000000..1e22094a --- /dev/null +++ b/internal/db/migrations/0088_library_matcher_version.down.sql @@ -0,0 +1 @@ +ALTER TABLE library_album_matches DROP COLUMN matcher; diff --git a/internal/db/migrations/0088_library_matcher_version.up.sql b/internal/db/migrations/0088_library_matcher_version.up.sql new file mode 100644 index 00000000..568921f5 --- /dev/null +++ b/internal/db/migrations/0088_library_matcher_version.up.sql @@ -0,0 +1,4 @@ +-- Which version of the matching decided an album. A refusal by an +-- older matcher is tried again by a newer one, so improving the matching +-- reaches the albums it refused without a migration each time. +ALTER TABLE library_album_matches ADD COLUMN matcher integer NOT NULL DEFAULT 0; diff --git a/internal/db/migrations/0089_library_match_by_hand.down.sql b/internal/db/migrations/0089_library_match_by_hand.down.sql new file mode 100644 index 00000000..678f4eab --- /dev/null +++ b/internal/db/migrations/0089_library_match_by_hand.down.sql @@ -0,0 +1,8 @@ +UPDATE library_album_matches SET outcome = 'unmatched' WHERE outcome = 'dismissed'; +UPDATE library_album_matches SET method = 'search' WHERE method = 'manual'; +ALTER TABLE library_album_matches DROP CONSTRAINT library_album_matches_outcome_check; +ALTER TABLE library_album_matches ADD CONSTRAINT library_album_matches_outcome_check + CHECK (outcome IN ('listed', 'unmatched')); +ALTER TABLE library_album_matches DROP CONSTRAINT library_album_matches_method_check; +ALTER TABLE library_album_matches ADD CONSTRAINT library_album_matches_method_check + CHECK (method IN ('', 'tags', 'release', 'discography', 'search')); diff --git a/internal/db/migrations/0089_library_match_by_hand.up.sql b/internal/db/migrations/0089_library_match_by_hand.up.sql new file mode 100644 index 00000000..8d8e47ca --- /dev/null +++ b/internal/db/migrations/0089_library_match_by_hand.up.sql @@ -0,0 +1,10 @@ +-- The Collection's Unmatched section: a person matches an album the +-- listing could not, by a MusicBrainz link or a search (method 'manual'), +-- or leaves it off the list for good (outcome 'dismissed': a local band's +-- album MusicBrainz will never have, a mix that is no album). +ALTER TABLE library_album_matches DROP CONSTRAINT library_album_matches_outcome_check; +ALTER TABLE library_album_matches ADD CONSTRAINT library_album_matches_outcome_check + CHECK (outcome IN ('listed', 'unmatched', 'dismissed')); +ALTER TABLE library_album_matches DROP CONSTRAINT library_album_matches_method_check; +ALTER TABLE library_album_matches ADD CONSTRAINT library_album_matches_method_check + CHECK (method IN ('', 'tags', 'release', 'discography', 'search', 'manual')); diff --git a/internal/db/migrations/0090_upgrades.down.sql b/internal/db/migrations/0090_upgrades.down.sql new file mode 100644 index 00000000..8986bbc1 --- /dev/null +++ b/internal/db/migrations/0090_upgrades.down.sql @@ -0,0 +1,7 @@ +DROP INDEX IF EXISTS wants_upgrade_idx; +ALTER TABLE wants + DROP COLUMN upgrade_checked_at, + DROP COLUMN held_quality, + DROP COLUMN held_tier, + DROP COLUMN upgrade; +ALTER TABLE acquisition_settings DROP COLUMN upgrade_until; diff --git a/internal/db/migrations/0090_upgrades.up.sql b/internal/db/migrations/0090_upgrades.up.sql new file mode 100644 index 00000000..c64734d4 --- /dev/null +++ b/internal/db/migrations/0090_upgrades.up.sql @@ -0,0 +1,29 @@ +-- Upgrades: a held album whose copy is below the quality an admin settles +-- for, or that holds a video rip, stays wanted for a better +-- copy. The import replaces the old tracks with the new ones. +-- +-- acquisition_settings.upgrade_until the tier a held album is upgraded to; +-- '' switches quality upgrades off. +-- Lidarr's "Any" profile upgraded up to +-- High Quality Lossy. +-- wants.upgrade why a held album is wanted again: +-- quality (below upgrade_until) or rip +-- (a track is a video rip); '' not. +-- wants.held_tier, held_quality the library copy's tier, estimated +-- from format and size per second, and +-- the same in words. +-- wants.upgrade_checked_at when the wanted worker last judged it. +-- want_candidates.verdict unchanged: a release no better than +-- the copy held is "refused". +ALTER TABLE acquisition_settings + ADD COLUMN upgrade_until text NOT NULL DEFAULT 'lossy_high'; + +ALTER TABLE wants + ADD COLUMN upgrade text NOT NULL DEFAULT '' + CONSTRAINT wants_upgrade_check CHECK (upgrade IN ('', 'quality', 'rip')), + ADD COLUMN held_tier text NOT NULL DEFAULT '', + ADD COLUMN held_quality text NOT NULL DEFAULT '', + ADD COLUMN upgrade_checked_at timestamptz; + +CREATE INDEX wants_upgrade_idx ON wants (searched_at NULLS FIRST) + WHERE fulfilled_at IS NOT NULL AND upgrade <> ''; diff --git a/internal/db/migrations/0091_remove_lidarr.down.sql b/internal/db/migrations/0091_remove_lidarr.down.sql new file mode 100644 index 00000000..dceb707f --- /dev/null +++ b/internal/db/migrations/0091_remove_lidarr.down.sql @@ -0,0 +1,29 @@ +ALTER TABLE quarantine_actions RENAME CONSTRAINT quarantine_actions_admin_id_fkey TO lidarr_quarantine_actions_admin_id_fkey; +ALTER TABLE quarantine RENAME CONSTRAINT quarantine_track_id_fkey TO lidarr_quarantine_track_id_fkey; +ALTER TABLE quarantine RENAME CONSTRAINT quarantine_user_id_fkey TO lidarr_quarantine_user_id_fkey; + +ALTER INDEX quarantine_actions_created_idx RENAME TO lidarr_quarantine_actions_created_idx; +ALTER INDEX quarantine_actions_track_idx RENAME TO lidarr_quarantine_actions_track_idx; +ALTER INDEX quarantine_actions_pkey RENAME TO lidarr_quarantine_actions_pkey; +ALTER TABLE quarantine_actions RENAME COLUMN album_mbid TO lidarr_album_mbid; +ALTER TYPE quarantine_action_kind RENAME VALUE 'deleted_album' TO 'deleted_via_lidarr'; +ALTER TYPE quarantine_action_kind RENAME TO lidarr_quarantine_action_kind; +ALTER TABLE quarantine_actions RENAME TO lidarr_quarantine_actions; + +ALTER INDEX quarantine_pkey RENAME TO lidarr_quarantine_pkey; +ALTER INDEX quarantine_track_idx RENAME TO lidarr_quarantine_track_idx; +ALTER TYPE quarantine_reason RENAME TO lidarr_quarantine_reason; +ALTER TABLE quarantine RENAME TO lidarr_quarantine; + +CREATE TABLE lidarr_config ( + id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1), + enabled boolean NOT NULL DEFAULT false, + base_url text, + api_key text, + default_quality_profile_id int, + default_root_folder_path text, + default_metadata_profile_id integer, + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now() +); +INSERT INTO lidarr_config (id, enabled) VALUES (1, false); diff --git a/internal/db/migrations/0091_remove_lidarr.up.sql b/internal/db/migrations/0091_remove_lidarr.up.sql new file mode 100644 index 00000000..b8e64723 --- /dev/null +++ b/internal/db/migrations/0091_remove_lidarr.up.sql @@ -0,0 +1,29 @@ +-- Minstrel does not talk to Lidarr: it finds, downloads, +-- checks and files music itself. +-- +-- lidarr_config the connection's settings: dropped. +-- lidarr_quarantine(_actions) the per-user track reports and the admin +-- actions on them, which never needed Lidarr: +-- renamed to quarantine(_actions), with their +-- types and indexes. +-- 'deleted_via_lidarr' the history of an album removed through +-- Lidarr reads as what it did: 'deleted_album'. +-- lidarr_album_mbid the album id that removal recorded: album_mbid. +DROP TABLE lidarr_config; + +ALTER TABLE lidarr_quarantine RENAME TO quarantine; +ALTER TYPE lidarr_quarantine_reason RENAME TO quarantine_reason; +ALTER INDEX lidarr_quarantine_track_idx RENAME TO quarantine_track_idx; +ALTER INDEX lidarr_quarantine_pkey RENAME TO quarantine_pkey; + +ALTER TABLE lidarr_quarantine_actions RENAME TO quarantine_actions; +ALTER TYPE lidarr_quarantine_action_kind RENAME TO quarantine_action_kind; +ALTER TYPE quarantine_action_kind RENAME VALUE 'deleted_via_lidarr' TO 'deleted_album'; +ALTER TABLE quarantine_actions RENAME COLUMN lidarr_album_mbid TO album_mbid; +ALTER INDEX lidarr_quarantine_actions_pkey RENAME TO quarantine_actions_pkey; +ALTER INDEX lidarr_quarantine_actions_track_idx RENAME TO quarantine_actions_track_idx; +ALTER INDEX lidarr_quarantine_actions_created_idx RENAME TO quarantine_actions_created_idx; + +ALTER TABLE quarantine RENAME CONSTRAINT lidarr_quarantine_user_id_fkey TO quarantine_user_id_fkey; +ALTER TABLE quarantine RENAME CONSTRAINT lidarr_quarantine_track_id_fkey TO quarantine_track_id_fkey; +ALTER TABLE quarantine_actions RENAME CONSTRAINT lidarr_quarantine_actions_admin_id_fkey TO quarantine_actions_admin_id_fkey; diff --git a/internal/db/migrations/0092_play_events_started_idx.down.sql b/internal/db/migrations/0092_play_events_started_idx.down.sql new file mode 100644 index 00000000..62eafa51 --- /dev/null +++ b/internal/db/migrations/0092_play_events_started_idx.down.sql @@ -0,0 +1 @@ +DROP INDEX IF EXISTS play_events_started_idx; diff --git a/internal/db/migrations/0092_play_events_started_idx.up.sql b/internal/db/migrations/0092_play_events_started_idx.up.sql new file mode 100644 index 00000000..d5404011 --- /dev/null +++ b/internal/db/migrations/0092_play_events_started_idx.up.sql @@ -0,0 +1,4 @@ +-- The admin Overview reads plays across every user by time: the ones +-- in progress and the last day's count. The only index on play_events leads +-- with user_id, so without this every Overview load scanned the table. +CREATE INDEX play_events_started_idx ON play_events (started_at); diff --git a/internal/db/migrations/0093_library_settings.down.sql b/internal/db/migrations/0093_library_settings.down.sql new file mode 100644 index 00000000..793e3ffb --- /dev/null +++ b/internal/db/migrations/0093_library_settings.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS library_settings; diff --git a/internal/db/migrations/0093_library_settings.up.sql b/internal/db/migrations/0093_library_settings.up.sql new file mode 100644 index 00000000..4bf3473d --- /dev/null +++ b/internal/db/migrations/0093_library_settings.up.sql @@ -0,0 +1,11 @@ +-- The library's folders, set in Settings. Until someone saves them +-- there, scan_paths is NULL and MINSTREL_LIBRARY_SCAN_PATHS names the folders, +-- as it always has; from the first save the row is what the server reads. +CREATE TABLE library_settings ( + id boolean PRIMARY KEY DEFAULT true, + scan_paths text[], + updated_at timestamptz NOT NULL DEFAULT now(), + + CONSTRAINT library_settings_singleton CHECK (id = true) +); +INSERT INTO library_settings (id) VALUES (true) ON CONFLICT (id) DO NOTHING; diff --git a/internal/db/migrations/0094_duplicate_distinct.down.sql b/internal/db/migrations/0094_duplicate_distinct.down.sql new file mode 100644 index 00000000..e5e44233 --- /dev/null +++ b/internal/db/migrations/0094_duplicate_distinct.down.sql @@ -0,0 +1,4 @@ +UPDATE duplicate_groups SET class = 'review' WHERE class = 'distinct'; +ALTER TABLE duplicate_groups DROP CONSTRAINT duplicate_groups_class_check; +ALTER TABLE duplicate_groups ADD CONSTRAINT duplicate_groups_class_check + CHECK (class IN ('same_release', 'cross_release', 'mismatch', 'review')); diff --git a/internal/db/migrations/0094_duplicate_distinct.up.sql b/internal/db/migrations/0094_duplicate_distinct.up.sql new file mode 100644 index 00000000..856757e5 --- /dev/null +++ b/internal/db/migrations/0094_duplicate_distinct.up.sql @@ -0,0 +1,11 @@ +-- 0094_duplicate_distinct.up.sql — duplicates settle themselves, without +-- waiting on an admin. +-- +-- distinct: one album holds two versions of a song that sound alike ("Lovesick" +-- and "Lovesick (Instrumental)"). They are not duplicates, and the resolver +-- dismisses them itself. mismatch now also covers one album holding the same +-- audio in several track slots, the resolver's wrong-file case. +-- The new value swaps the constraint in the same migration. +ALTER TABLE duplicate_groups DROP CONSTRAINT duplicate_groups_class_check; +ALTER TABLE duplicate_groups ADD CONSTRAINT duplicate_groups_class_check + CHECK (class IN ('same_release', 'cross_release', 'mismatch', 'distinct', 'review')); diff --git a/internal/db/migrations/0095_tracks_emptied_at.down.sql b/internal/db/migrations/0095_tracks_emptied_at.down.sql new file mode 100644 index 00000000..f3feee97 --- /dev/null +++ b/internal/db/migrations/0095_tracks_emptied_at.down.sql @@ -0,0 +1 @@ +ALTER TABLE tracks DROP COLUMN emptied_at; diff --git a/internal/db/migrations/0095_tracks_emptied_at.up.sql b/internal/db/migrations/0095_tracks_emptied_at.up.sql new file mode 100644 index 00000000..3ab7883a --- /dev/null +++ b/internal/db/migrations/0095_tracks_emptied_at.up.sql @@ -0,0 +1,13 @@ +-- 0095_tracks_emptied_at.up.sql — a slot the duplicate resolver emptied +-- waits for a copy fetched again. +-- +-- When the resolver removes a file filed under another song's title and the +-- album holds no copy of that song as long as the song is, the row is marked +-- missing and stamped here with the same time. Missing-pair folding then pairs +-- it by title only with a track added after that time, a copy fetched again, +-- and not with one already on the album, which under that title holds +-- another song. +-- +-- The stamp counts only while it equals missing_since: a row whose file came +-- back and later went missing again is an ordinary missing row. +ALTER TABLE tracks ADD COLUMN emptied_at timestamptz; diff --git a/internal/db/migrations/0096_similarity_relative_scores.down.sql b/internal/db/migrations/0096_similarity_relative_scores.down.sql new file mode 100644 index 00000000..7c4e468f --- /dev/null +++ b/internal/db/migrations/0096_similarity_relative_scores.down.sql @@ -0,0 +1,3 @@ +-- The raw scores are not kept for artists; the worker rewrites every seed with +-- relative scores on its next fetch either way. Nothing to undo. +SELECT 1; diff --git a/internal/db/migrations/0096_similarity_relative_scores.up.sql b/internal/db/migrations/0096_similarity_relative_scores.up.sql new file mode 100644 index 00000000..111e110e --- /dev/null +++ b/internal/db/migrations/0096_similarity_relative_scores.up.sql @@ -0,0 +1,51 @@ +-- Before: ListenBrainz similarity scores were stored raw. They are counts, into +-- the thousands, and every reader treats similarity as 0 to 1. After: the +-- stored rows are rescaled relative to each seed's best score, as the worker +-- now writes them. +-- +-- A seed's best is taken over everything stored for it: the cached LB answer +-- and its edges for tracks, matched and unmatched rows for artists. A seed +-- whose best is already at most 1 is left alone, so a second run changes +-- nothing. + +WITH best AS ( + SELECT seed_id, max(score) AS score + FROM ( + SELECT track_a_id AS seed_id, score FROM track_similarity WHERE source = 'listenbrainz' + UNION ALL + SELECT seed_track_id, score FROM listenbrainz_similar_recordings + ) s + GROUP BY seed_id +) +UPDATE track_similarity ts + SET score = ts.score / b.score + FROM best b + WHERE ts.source = 'listenbrainz' + AND ts.track_a_id = b.seed_id + AND b.score > 1; + +-- One statement, so both updates divide by the best as it stood before either. +WITH best AS ( + SELECT seed_id, max(score) AS score + FROM ( + SELECT artist_a_id AS seed_id, score FROM artist_similarity WHERE source = 'listenbrainz' + UNION ALL + SELECT seed_artist_id, score FROM artist_similarity_unmatched WHERE source = 'listenbrainz' + ) s + GROUP BY seed_id +), +matched AS ( + UPDATE artist_similarity a + SET score = a.score / b.score + FROM best b + WHERE a.source = 'listenbrainz' + AND a.artist_a_id = b.seed_id + AND b.score > 1 + RETURNING 1 +) +UPDATE artist_similarity_unmatched u + SET score = u.score / b.score + FROM best b + WHERE u.source = 'listenbrainz' + AND u.seed_artist_id = b.seed_id + AND b.score > 1; diff --git a/internal/db/migrations/0097_requeue_short_track_prints.down.sql b/internal/db/migrations/0097_requeue_short_track_prints.down.sql new file mode 100644 index 00000000..f1513b02 --- /dev/null +++ b/internal/db/migrations/0097_requeue_short_track_prints.down.sql @@ -0,0 +1,3 @@ +-- The cleared rows were failures; the workers write them again, or better. +-- Nothing to undo. +SELECT 1; diff --git a/internal/db/migrations/0097_requeue_short_track_prints.up.sql b/internal/db/migrations/0097_requeue_short_track_prints.up.sql new file mode 100644 index 00000000..47288f43 --- /dev/null +++ b/internal/db/migrations/0097_requeue_short_track_prints.up.sql @@ -0,0 +1,20 @@ +-- The image's fpcalc exits 3 ("Error decoding audio frame (End of file)") on +-- every file shorter than the span it is asked for, so every track under two +-- minutes failed both prints. Before: the failure was stored as a verdict, and +-- a verdict is not retried until the file changes. After: fpcalcReadToEnd keeps +-- those prints; clear the failures so the workers take them again. + +-- The AcoustID lookup's queue takes a track with no lookup row. +DELETE FROM track_acoustid_lookups + WHERE state = 'failed' + AND detail LIKE '%Error decoding audio frame (End of file)%'; + +-- The fingerprint backfill's queue takes a track with no fingerprint row. A +-- NULL chromaprint is always a verdict (an inconclusive attempt stores +-- nothing), and on a track no longer than the span it was taken at it is this +-- failure. A track that is damaged fails again and is stamped again. +DELETE FROM track_fingerprints f + USING tracks t + WHERE f.track_id = t.id + AND f.chromaprint IS NULL + AND t.duration_ms <= f.chromaprint_length_sec * 1000; diff --git a/internal/db/migrations/0098_release_blocks_gone.down.sql b/internal/db/migrations/0098_release_blocks_gone.down.sql new file mode 100644 index 00000000..97d45aea --- /dev/null +++ b/internal/db/migrations/0098_release_blocks_gone.down.sql @@ -0,0 +1,4 @@ +DELETE FROM release_blocks WHERE reason = 'gone'; +ALTER TABLE release_blocks DROP CONSTRAINT release_blocks_reason_check; +ALTER TABLE release_blocks ADD CONSTRAINT release_blocks_reason_check + CHECK (reason IN ('operator', 'files', 'import', 'stalled')); diff --git a/internal/db/migrations/0098_release_blocks_gone.up.sql b/internal/db/migrations/0098_release_blocks_gone.up.sql new file mode 100644 index 00000000..1b2a2cfe --- /dev/null +++ b/internal/db/migrations/0098_release_blocks_gone.up.sql @@ -0,0 +1,6 @@ +-- gone: a release its download client would not let Minstrel follow: +-- held outside Minstrel's category, or leaving the client twice. Held back +-- until expires_at, as stalled is. +ALTER TABLE release_blocks DROP CONSTRAINT release_blocks_reason_check; +ALTER TABLE release_blocks ADD CONSTRAINT release_blocks_reason_check + CHECK (reason IN ('operator', 'files', 'import', 'stalled', 'gone')); diff --git a/internal/db/queries/acoustid.sql b/internal/db/queries/acoustid.sql index 7a4aa1e1..cb677a6c 100644 --- a/internal/db/queries/acoustid.sql +++ b/internal/db/queries/acoustid.sql @@ -1,4 +1,4 @@ --- AcoustID recording-id lookup (M401). Migration 0069 says what each +-- AcoustID recording-id lookup. Migration 0069 says what each -- lookup state means. -- name: ListTracksNeedingAcoustIDLookup :many @@ -29,7 +29,7 @@ ON CONFLICT (track_id) DO UPDATE SET looked_up_at = now(); -- name: SetTrackMbidFromAcoustID :execrows --- Never over a tag id (D4): the guard is here, not only in the worker, so a +-- Never over a tag id: the guard is here, not only in the worker, so a -- tag the scan read between the worker's list and this write still wins. -- updated_at is left alone: it gates the scan's mtime skip, and moving it -- past a file's mtime would hide a change made just before the lookup. diff --git a/internal/db/queries/acquisition.sql b/internal/db/queries/acquisition.sql new file mode 100644 index 00000000..82e11b15 --- /dev/null +++ b/internal/db/queries/acquisition.sql @@ -0,0 +1,76 @@ +-- Indexers and download clients. + +-- name: ListAcquisitionServices :many +-- Returns no secret, only whether one is set: this feeds the settings page. +SELECT code, kind, protocol, label, enabled, base_url, + (api_key <> '')::boolean AS has_api_key, + (password <> '')::boolean AS has_password, + username, category, timeout_ms, + serves_search, serves_downloads, serves_enrichment, + local_path, remote_path, searches_per_hour, + created_at, updated_at +FROM acquisition_services +ORDER BY kind, label, code; + +-- name: GetAcquisitionService :one +-- The full row, credentials included. For building a client, never for a +-- response. +SELECT * FROM acquisition_services WHERE code = sqlc.arg(code); + +-- name: ListUsableAcquisitionServices :many +-- Every enabled, addressed service doing one job. An exchange row does a job +-- when its switch for it is on; any other row does the job its kind names. +-- Ordered by code so a fan-out is deterministic. +SELECT * FROM acquisition_services +WHERE enabled AND base_url <> '' + AND CASE WHEN protocol = 'exchange' THEN + CASE sqlc.arg(job)::text + WHEN 'search' THEN serves_search + WHEN 'download' THEN serves_downloads + WHEN 'enrichment' THEN serves_enrichment + ELSE false + END + ELSE kind = sqlc.arg(job)::text + END +ORDER BY code; + +-- name: CreateAcquisitionService :one +-- ON CONFLICT so the caller can try the next code when this one is taken. +INSERT INTO acquisition_services + (code, kind, protocol, label, category, timeout_ms, searches_per_hour) +VALUES + (sqlc.arg(code), sqlc.arg(kind), sqlc.arg(protocol), sqlc.arg(label), + sqlc.arg(category), sqlc.arg(timeout_ms), sqlc.arg(searches_per_hour)) +ON CONFLICT (code) DO NOTHING +RETURNING *; + +-- name: UpdateAcquisitionService :one +-- Everything on the settings form except the credentials, which have their +-- own write-only calls. +UPDATE acquisition_services SET + label = sqlc.arg(label), + enabled = sqlc.arg(enabled), + base_url = sqlc.arg(base_url), + username = sqlc.arg(username), + category = sqlc.arg(category), + timeout_ms = sqlc.arg(timeout_ms), + serves_search = sqlc.arg(serves_search), + serves_downloads = sqlc.arg(serves_downloads), + serves_enrichment = sqlc.arg(serves_enrichment), + local_path = sqlc.arg(local_path), + remote_path = sqlc.arg(remote_path), + searches_per_hour = sqlc.arg(searches_per_hour), + updated_at = now() +WHERE code = sqlc.arg(code) +RETURNING *; + +-- name: DeleteAcquisitionService :execrows +DELETE FROM acquisition_services WHERE code = sqlc.arg(code); + +-- name: SetAcquisitionServiceAPIKey :execrows +UPDATE acquisition_services SET api_key = sqlc.arg(api_key), updated_at = now() +WHERE code = sqlc.arg(code); + +-- name: SetAcquisitionServicePassword :execrows +UPDATE acquisition_services SET password = sqlc.arg(password), updated_at = now() +WHERE code = sqlc.arg(code); diff --git a/internal/db/queries/admin_status.sql b/internal/db/queries/admin_status.sql new file mode 100644 index 00000000..e36b2a37 --- /dev/null +++ b/internal/db/queries/admin_status.sql @@ -0,0 +1,54 @@ +-- The admin Overview's status sections: who is listening, what is +-- downloading, what the library holds. Each is one cheap read. + +-- name: ListListeningNow :many +-- Plays in progress: open play events still inside their track's length, +-- with a few minutes' grace for a pause. A client that never says a play +-- ended would otherwise read as listening until the 24-hour sweep closes it. +SELECT pe.started_at, pe.client_id, pe.device_class, + u.username, + t.id AS track_id, t.title AS track_title, t.duration_ms, + ar.name AS artist_name, + al.id AS album_id, al.title AS album_title + FROM play_events pe + JOIN users u ON u.id = pe.user_id + JOIN tracks t ON t.id = pe.track_id + JOIN albums al ON al.id = t.album_id + JOIN artists ar ON ar.id = t.artist_id + WHERE pe.ended_at IS NULL + AND pe.started_at > now() - INTERVAL '24 hours' + AND pe.started_at + make_interval(secs => t.duration_ms / 1000.0) + INTERVAL '5 minutes' > now() + ORDER BY pe.started_at DESC + LIMIT 20; + +-- name: PlaysLastDay :one +-- The last day's listening, for the line under Listening now. +SELECT COUNT(*)::bigint AS plays, + COUNT(DISTINCT user_id)::bigint AS listeners, + MAX(started_at)::timestamptz AS last_at + FROM play_events + WHERE started_at > now() - INTERVAL '24 hours'; + +-- name: DownloadsStatus :one +-- Downloads at a glance: running, waiting on a person (and since when), and +-- the last day's imports and give-ups. +SELECT COUNT(*) FILTER (WHERE ended_at IS NULL AND import_state IN ('', 'downloading', 'waiting', 'importing'))::bigint AS running, + COUNT(*) FILTER (WHERE ended_at IS NULL AND import_state IN ('review', 'failed'))::bigint AS needs_you, + MIN(COALESCE(import_updated_at, grabbed_at)) FILTER (WHERE ended_at IS NULL AND import_state IN ('review', 'failed'))::timestamptz AS needs_you_since, + COUNT(*) FILTER (WHERE imported_at > now() - INTERVAL '24 hours')::bigint AS imported_day, + COUNT(*) FILTER (WHERE ended_at > now() - INTERVAL '24 hours' + AND end_reason IN ('stalled', 'failed', 'gone', 'abandoned'))::bigint AS given_up_day + FROM grabs + WHERE ended_at IS NULL + OR ended_at > now() - INTERVAL '24 hours' + OR imported_at > now() - INTERVAL '24 hours'; + +-- name: LibraryStatus :one +-- What the library holds on disk: tracks whose files are present, the +-- albums and artists they make, and their size. +SELECT COUNT(*)::bigint AS tracks, + COUNT(DISTINCT album_id)::bigint AS albums, + COUNT(DISTINCT artist_id)::bigint AS artists, + COALESCE(SUM(file_size), 0)::bigint AS bytes + FROM tracks + WHERE missing_since IS NULL; diff --git a/internal/db/queries/admin_summary.sql b/internal/db/queries/admin_summary.sql new file mode 100644 index 00000000..9d4167f7 --- /dev/null +++ b/internal/db/queries/admin_summary.sql @@ -0,0 +1,32 @@ +-- name: AdminSummary :one +-- What the admin rail and the Overview's verdict read: each queue +-- that waits on a person, with how many and since when. One round trip, so +-- the rail costs one cheap read on every admin page. The "since" is the +-- oldest item still waiting: how long the queue has been left, which is what +-- tells an admin a count of 2 is fine today and wrong after a month. +SELECT + (SELECT count(*) FROM requests WHERE status = 'pending')::bigint AS requests, + (SELECT min(requested_at) FROM requests WHERE status = 'pending')::timestamptz AS requests_since, + -- A broken track is one report however it was found: a listener flagged + -- it, or it failed to play. Counted by track, as the Reports + -- page lists them. + (SELECT count(*) FROM ( + SELECT track_id FROM quarantine + UNION + SELECT track_id FROM playback_errors WHERE resolved_at IS NULL) r)::bigint AS reports, + (SELECT min(at) FROM ( + SELECT created_at AS at FROM quarantine + UNION ALL + SELECT occurred_at FROM playback_errors WHERE resolved_at IS NULL) r)::timestamptz AS reports_since, + -- Duplicates needing review: a cross-release group is kept and linked by + -- the resolver and waits on nobody (the report's review view). + (SELECT count(*) FROM duplicate_groups g + WHERE g.status = 'pending' AND g.class IS DISTINCT FROM 'cross_release' + AND (SELECT count(*) FROM duplicate_group_members m WHERE m.group_id = g.id) >= 2)::bigint AS duplicates, + (SELECT min(g.detected_at) FROM duplicate_groups g + WHERE g.status = 'pending' AND g.class IS DISTINCT FROM 'cross_release' + AND (SELECT count(*) FROM duplicate_group_members m WHERE m.group_id = g.id) >= 2)::timestamptz AS duplicates_since, + (SELECT count(*) FROM tracks WHERE source_verdict = 'suspect' AND missing_since IS NULL)::bigint AS suspect, + (SELECT min(source_verdict_at) FROM tracks WHERE source_verdict = 'suspect' AND missing_since IS NULL)::timestamptz AS suspect_since, + (SELECT count(*) FROM tracks WHERE missing_since IS NOT NULL)::bigint AS missing, + (SELECT min(missing_since) FROM tracks WHERE missing_since IS NOT NULL)::timestamptz AS missing_since; diff --git a/internal/db/queries/albums.sql b/internal/db/queries/albums.sql index 4ce5452f..b74cb1fb 100644 --- a/internal/db/queries/albums.sql +++ b/internal/db/queries/albums.sql @@ -36,8 +36,23 @@ WHERE albums.artist_id = $1 ORDER BY release_date NULLS LAST, sort_title; -- name: GetAlbumByArtistAndTitle :one --- Scanner uses this for the no-mbid dedupe path: resolve-or-create. -SELECT * FROM albums WHERE artist_id = $1 AND title = $2 LIMIT 1; +-- Scanner uses this for the no-mbid dedupe path: resolve-or-create. An album +-- of the same name in another release group is a different album (one +-- artist's self-titled albums), so a row whose +-- group is known and differs is not returned. A row of the file's own group +-- comes first, then one with no group yet, then the one holding the most +-- tracks from the file's folder: a file tagged with no group (a live bonus +-- track beside its album) belongs with its folder, not the oldest +-- row, and a few strays already misfiled do not outvote the folder. +SELECT albums.* FROM albums + WHERE albums.artist_id = sqlc.arg(artist_id) AND albums.title = sqlc.arg(title) + AND (sqlc.narg(release_group_mbid)::text IS NULL OR albums.release_group_mbid IS NULL + OR albums.release_group_mbid = sqlc.narg(release_group_mbid)) + ORDER BY (albums.release_group_mbid IS NOT DISTINCT FROM sqlc.narg(release_group_mbid)) DESC, + (SELECT count(*) FROM tracks t + WHERE t.album_id = albums.id AND starts_with(t.file_path, sqlc.arg(dir)::text || '/')) DESC, + albums.id + LIMIT 1; -- name: ListAlbumsByArtist :many SELECT * FROM albums WHERE artist_id = $1 ORDER BY release_date NULLS LAST, sort_title; @@ -64,12 +79,10 @@ SELECT * FROM albums ORDER BY random() LIMIT $1; -- Album "belongs to" a genre if any of its tracks carry that genre. -- Serves Subsonic getAlbumList?type=byGenre. -- --- Splits tracks.genre on [;,] as of #367. It previously compared the whole --- column verbatim, so a track tagged "Rock;Pop" was unreachable from EITHER --- "Rock" or "Pop" — a Subsonic client asking for a genre silently missed --- every multi-genre track. This also aligns the endpoint with --- recommendation.sql / discover.sql, which have always split, and with the --- genre browse index that #367 adds. +-- Splits tracks.genre on [;,], so a track tagged "Rock;Pop" is reachable from +-- both "Rock" and "Pop"; comparing the whole column would hide every +-- multi-genre track from a Subsonic client asking for a genre. The same split +-- as recommendation.sql, discover.sql and the genre browse index. -- -- EXISTS rather than JOIN + DISTINCT ON: the lateral split emits one row per -- (track, genre-fragment), so a join would multiply rows per album and lean @@ -86,6 +99,56 @@ WHERE EXISTS ( ORDER BY albums.sort_title, albums.id LIMIT sqlc.arg(lim) OFFSET sqlc.arg(off); +-- name: ListAlbumsRecentlyPlayedByUser :many +-- The user's albums, most recently played first: an album's time is its +-- latest play of any of its tracks, skips included, since a skip is still +-- the user reaching for that album. Plays of tracks whose file is gone do +-- not count. Serves Subsonic getAlbumList2?type=recent. +-- +-- Aggregates play_events per track first (play_events_user_track_idx), then +-- folds tracks into albums, so the join runs over distinct tracks rather +-- than every play. +WITH plays AS ( + SELECT track_id, max(started_at) AS last_played + FROM play_events + WHERE user_id = sqlc.arg(user_id) + GROUP BY track_id +), per_album AS ( + SELECT t.album_id, max(p.last_played) AS last_played + FROM plays p + JOIN tracks t ON t.id = p.track_id + WHERE t.missing_since IS NULL + GROUP BY t.album_id +) +SELECT albums.* +FROM per_album pa +JOIN albums ON albums.id = pa.album_id +ORDER BY pa.last_played DESC, albums.id +LIMIT sqlc.arg(lim) OFFSET sqlc.arg(off); + +-- name: ListAlbumsFrequentlyPlayedByUser :many +-- The user's albums, most played first: an album's count is the user's +-- completed plays across its tracks. Skips do not count, so skipping +-- through an album does not make it a favourite. Plays of tracks whose file +-- is gone do not count. Serves Subsonic getAlbumList2?type=frequent. +WITH plays AS ( + SELECT track_id, count(*) AS cnt + FROM play_events + WHERE user_id = sqlc.arg(user_id) AND was_skipped = false + GROUP BY track_id +), per_album AS ( + SELECT t.album_id, sum(p.cnt) AS cnt + FROM plays p + JOIN tracks t ON t.id = p.track_id + WHERE t.missing_since IS NULL + GROUP BY t.album_id +) +SELECT albums.* +FROM per_album pa +JOIN albums ON albums.id = pa.album_id +ORDER BY pa.cnt DESC, albums.id +LIMIT sqlc.arg(lim) OFFSET sqlc.arg(off); + -- name: SearchAlbums :many SELECT * FROM albums WHERE title ILIKE '%' || $1 || '%' @@ -96,7 +159,7 @@ LIMIT $2 OFFSET $3; SELECT COUNT(*) FROM albums WHERE title ILIKE '%' || $1::text || '%'; -- name: ListRecentlyAddedAlbumsWithArtist :many --- M6a: recently-added albums joined with artist_name + artist_id for the +-- Recently-added albums joined with artist_name + artist_id for the -- home-page section. created_at is the row-insert timestamp from the -- scanner — the closest proxy to "added to library". Stable secondary -- ordering by id keeps pagination tie-break deterministic. @@ -107,7 +170,7 @@ ORDER BY albums.created_at DESC, albums.id LIMIT $1; -- name: ListAlbumsAlphaWithArtist :many --- M6a: alpha-sorted album list joined with artist_name. Used by +-- Alpha-sorted album list joined with artist_name. Used by -- /api/library/albums for the wrapping-grid page. Stable id-tiebreak. SELECT sqlc.embed(albums), artists.name AS artist_name FROM albums @@ -116,7 +179,7 @@ ORDER BY albums.sort_title, albums.id LIMIT $1 OFFSET $2; -- name: CountAlbums :one --- M6a: total album count for the /api/library/albums envelope. +-- Total album count for the /api/library/albums envelope. SELECT COUNT(*) FROM albums; -- name: CountAlbumsByArtist :one @@ -125,7 +188,7 @@ SELECT COUNT(*) FROM albums; SELECT COUNT(*) FROM albums WHERE artist_id = $1; -- name: DeleteAlbumIfEmpty :one --- M7 #372: deletes the album row only when it has no remaining tracks. +-- Deletes the album row only when it has no remaining tracks. -- Used after a track delete to tidy up orphaned albums. RETURNING gives -- us the artist_id so the service can chain into the artist-empty check. -- Returns no rows when the album still has tracks; the service treats @@ -136,7 +199,7 @@ WHERE a.id = $1 RETURNING a.id, a.artist_id; -- name: SetAlbumMbidIfNull :exec --- M7 #379: heal MBID on existing album rows during rescan. Idempotent — +-- Heal MBID on existing album rows during rescan. Idempotent — -- only updates when the existing mbid is NULL. UPDATE albums SET mbid = $2, @@ -144,7 +207,7 @@ UPDATE albums WHERE id = $1 AND mbid IS NULL; -- name: SetAlbumReleaseGroupMbidIfNull :exec --- #5241: heal the release-group id on an existing album row, from a rescan's +-- Heal the release-group id on an existing album row, from a rescan's -- tags or from the re-acquisition sweeper's MusicBrainz lookup. Only fills a -- NULL, so a tag never gets overwritten by a later guess or vice versa. UPDATE albums @@ -153,7 +216,7 @@ UPDATE albums WHERE id = $1 AND release_group_mbid IS NULL; -- name: GetReleaseGroupForReleaseMbid :one --- #5241: the release group the library already knows for a release id, so a +-- The release group the library already knows for a release id, so a -- request carrying a release id can be repaired without asking MusicBrainz. SELECT release_group_mbid::text FROM albums @@ -161,7 +224,7 @@ SELECT release_group_mbid::text LIMIT 1; -- name: SetReleaseGroupForReleaseMbidIfNull :exec --- #5241: cache a MusicBrainz-resolved release group onto the album row that +-- Cache a MusicBrainz-resolved release group onto the album row that -- holds the release, filling only a NULL. UPDATE albums SET release_group_mbid = sqlc.arg(release_group_mbid)::text, @@ -173,7 +236,7 @@ UPDATE albums -- one of its tracks' file_path so the worker can re-read tags. LIMIT -- supplied by caller for batching/progress purposes. The track read is -- one still on disk; an album with none left is skipped until a scan --- finds its files again (#5139). +-- finds its files again. SELECT a.id AS album_id, a.artist_id AS artist_id, a.title AS title, @@ -192,7 +255,7 @@ SELECT a.id AS album_id, LIMIT $1; -- name: GetAlbumCoverageRollup :one --- M7 coverage-gauge: library-wide cover-art coverage rollup for the +-- Library-wide cover-art coverage rollup for the -- admin gauge. Single sequential scan with FILTER aggregates — -- sub-millisecond on realistic libraries. pending_no_mbid is a SUBSET -- of pending; the UI surfaces it separately so operators can see how @@ -202,12 +265,9 @@ SELECT a.id AS album_id, -- Invariant: with_art + pending + settled = total. (pending_no_mbid is -- not part of the sum — it's a subset of pending, surfaced separately.) -- --- The IN list below must stay in sync with the cover_art_source CHECK --- constraint — currently relaxed by migration 0020 to include 'deezer' --- and 'lastfm', and migration 0030 further relaxed it to any non-empty --- string. If a new source value is added without updating this query, --- with_art will silently undercount. Drift #556 caught the deezer + --- lastfm omission introduced by migration 0020. +-- The IN list below must name every source the enrichers write (the +-- cover_art_source CHECK accepts any non-empty string since migration +-- 0030). A source missing here makes with_art silently undercount. SELECT COUNT(*) AS total, COUNT(*) FILTER (WHERE cover_art_source IN @@ -218,7 +278,7 @@ SELECT FROM albums; -- name: SetAlbumCoverWithVersion :exec --- M7 cover-sources: records a successful or settled enrichment with the +-- Records a successful or settled enrichment with the -- current sources_version stamp. cover_art_path is NULL when source is -- 'none' (no file written). Atomic; called per album per pass. UPDATE albums @@ -231,5 +291,5 @@ UPDATE albums -- name: GetAlbumsByIDs :many -- Batched lookup used by /api/library/sync to hydrate upsert payloads --- (#357). Mirror of GetArtistsByIDs. +--. Mirror of GetArtistsByIDs. SELECT * FROM albums WHERE id = ANY($1::uuid[]); diff --git a/internal/db/queries/artists.sql b/internal/db/queries/artists.sql index 62535726..dbb8048f 100644 --- a/internal/db/queries/artists.sql +++ b/internal/db/queries/artists.sql @@ -46,7 +46,7 @@ SELECT COUNT(*) FROM artists WHERE name ILIKE '%' || $1::text || '%'; SELECT * FROM artists WHERE id = ANY($1::uuid[]); -- name: ListArtistsAlphaWithCovers :many --- M6a: alpha-sorted artist list with derived cover_album_id (most-recent +-- Alpha-sorted artist list with derived cover_album_id (most-recent -- album whose cover_art_path is set). Replaces ListArtistsAlpha for the -- /api/artists?sort=alpha branch — the new column is appended, so the -- alpha branch is purely additive on the wire. album_count is computed @@ -68,7 +68,7 @@ ORDER BY artists.sort_name, artists.name, artists.id LIMIT $1 OFFSET $2; -- name: DeleteArtistIfEmpty :one --- M7 #372: deletes the artist row only when it has no remaining albums +-- Deletes the artist row only when it has no remaining albums -- AND no remaining tracks (defensive — tracks can in principle exist -- without an album, though the scanner doesn't write that shape today). DELETE FROM artists a @@ -78,7 +78,7 @@ WHERE a.id = $1 RETURNING a.id; -- name: SetArtistMbidIfNull :exec --- M7 #379: heal MBID on existing artist rows during rescan. Idempotent — +-- Heal MBID on existing artist rows during rescan. Idempotent — -- only updates when the existing mbid is NULL, so concurrent scans don't -- overwrite an existing value with a stale one. UPDATE artists @@ -87,7 +87,7 @@ UPDATE artists WHERE id = $1 AND mbid IS NULL; -- name: ListArtistsMissingArt :many --- M7 cover-sources: returns artist rows the artist-art enricher should +-- Returns artist rows the artist-art enricher should -- attempt. Eligible: artist_art_source IS NULL OR (= 'none' AND stale -- artist_art_sources_version). MBID-less artists are returned but the -- enricher skips them at the row.mbid == nil guard inside the chain. @@ -99,7 +99,7 @@ SELECT a.id LIMIT $2; -- name: SetArtistArtWithVersion :exec --- M7 cover-sources: records a successful or settled artist-art +-- Records a successful or settled artist-art -- enrichment with the current sources_version stamp. thumb_path / -- fanart_path are NULL on a 'none' result, or independently NULL if -- the provider returned only one of the two image types (e.g. @@ -113,7 +113,7 @@ UPDATE artists WHERE id = $1; -- name: ClearArtistArtNone :exec --- M7 cover-sources: parallel of ClearAlbumCoverNone. Used by the +-- Parallel of ClearAlbumCoverNone. Used by the -- eligibility loop to flip stale 'none' rows back to NULL. UPDATE artists SET artist_art_source = NULL, diff --git a/internal/db/queries/audit_log.sql b/internal/db/queries/audit_log.sql index 153a5cb6..ea699eb1 100644 --- a/internal/db/queries/audit_log.sql +++ b/internal/db/queries/audit_log.sql @@ -15,8 +15,8 @@ SELECT count(*)::bigint FROM audit_log -- name: ListAuditLogByActions :many -- One page of the given actions, newest first: the duplicates report's --- "resolved automatically" activity (M498) reads the resolver's merges and --- Lidarr release changes from here. +-- "resolved automatically" activity reads the resolver's merges from +-- here. SELECT * FROM audit_log WHERE action = ANY(sqlc.arg(actions)::text[]) AND (NOT sqlc.arg(system_only)::boolean OR actor_id IS NULL) diff --git a/internal/db/queries/browse.sql b/internal/db/queries/browse.sql index 4b3d9ced..4a200fc4 100644 --- a/internal/db/queries/browse.sql +++ b/internal/db/queries/browse.sql @@ -1,29 +1,23 @@ --- Every query in this file filters `tracks.missing_since IS NULL` (#2523). --- A row whose file has vanished keeps its genre forever — the scanner walks the --- filesystem, so it never revisits a path that no longer exists — which is how --- pre-#2499 welded genres survived a full re-scan and kept showing in the index. --- Browsing is a way of finding something to play, so a track that cannot play +-- Every query in this file filters `tracks.missing_since IS NULL`. A row whose +-- file has vanished keeps its genre forever — the scanner walks the +-- filesystem, so it never revisits a path that no longer exists — and +-- browsing is a way of finding something to play, so a track that cannot play -- should not shape it. -- --- The year axis filters too (#2702). It used to join albums only, on the --- reasoning that an album is a real release even when some of its tracks are --- gone — true, but it left the two browse axes disagreeing: the genre queries --- above filter tracks.missing_since, so an album whose every file had vanished --- was already absent from genre while still listed under its year, where --- opening it found nothing playable. +-- The year axis filters too, so the two axes agree: an album whose every file +-- has vanished is absent from both, rather than absent from its genre and +-- still listed under its year, where opening it would find nothing playable. -- --- Hiding is the answer rather than showing-and-marking because browsing is how --- you go looking for something to play, and the operator's rule for that case --- is to take it out of view; the admin missing-files surface is where absence --- is reported. It also means changing the axis that was inconsistent rather --- than the one that was already right. +-- Hiding rather than showing-and-marking, because browsing is how you go +-- looking for something to play; the admin missing-files surface is where +-- absence is reported. -- -- The predicate is "has at least one playable track", so it also excludes an -- album with no tracks at all. That is the same answer for the same reason: -- nothing to play, nothing to browse to. -- name: ListGenresWithCount :many --- Genre browse index (#367). +-- Genre browse index. -- -- Genres live inline on tracks.genre as a delimited string, so this splits on -- the same [;,] pattern already used by recommendation.sql and discover.sql — @@ -34,14 +28,14 @@ -- trim() but deliberately NO lower(): trimming repairs an artifact of OUR -- splitting ("Rock; Pop" yields " Pop", and showing that as a distinct genre -- would be a bug), whereas case is what the tag actually says. Raw ID3 is --- exposed as-is for v1, so "Rock" and "rock" appear as separate rows. +-- exposed as-is, so "Rock" and "rock" appear as separate rows. -- -- COUNT(DISTINCT) because a sloppy tag like "Rock;Rock" would otherwise -- inflate its own row. -- -- Ordered by count first: raw ID3 data has a long tail of one-off junk tags, --- so alphabetical would bury the handful of genres an operator actually has a --- library's worth of. Name breaks ties for a stable order. +-- so alphabetical would bury the handful of genres a library actually has +-- a lot of. Name breaks ties for a stable order. SELECT trim(g.genre) AS genre, COUNT(DISTINCT tracks.id)::bigint AS track_count FROM tracks JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true @@ -87,7 +81,7 @@ WHERE EXISTS ( ); -- name: ListAlbumYearsWithCount :many --- Year browse index (#367). Only albums with a release_date appear — an +-- Year browse index. Only albums with a release_date appear — an -- album with no date isn't "year unknown" as a browsable bucket, it's absent -- from this axis, and the UI says so rather than inventing a 0 row. -- Newest first: recent releases are the likelier browse target. @@ -132,8 +126,7 @@ WHERE release_date IS NOT NULL -- name: ListGenresForAlbum :many -- Distinct genres carried by an album's tracks, for the album detail page's -- quick-jump chips. Split and trimmed identically to ListGenresWithCount, so a --- chip always leads to a page that actually contains this album — the two --- diverging is exactly the bug #367 had to fix in ListAlbumsByGenre. +-- chip always leads to a page that actually contains this album. SELECT DISTINCT trim(g.genre) AS genre FROM tracks JOIN LATERAL regexp_split_to_table(coalesce(tracks.genre, ''), '[;,]') AS g(genre) ON true diff --git a/internal/db/queries/candidate_artist_tags.sql b/internal/db/queries/candidate_artist_tags.sql index 4403689f..4bc94881 100644 --- a/internal/db/queries/candidate_artist_tags.sql +++ b/internal/db/queries/candidate_artist_tags.sql @@ -1,4 +1,4 @@ --- Folksonomy tags for out-of-library Discover candidates (#2376). Parallel to +-- Folksonomy tags for out-of-library Discover candidates. Parallel to -- track_tags.sql, but keyed by MBID because the artist has no local row. See -- 0050_candidate_artist_tags.up.sql for why the bookkeeping is its own table. @@ -63,7 +63,7 @@ ON CONFLICT (candidate_mbid) DO UPDATE updated_at = now(); -- name: ListCandidateArtistTagsForMbids :many --- Cached tags for a set of candidates, for slice 6's taste-overlap ranking. +-- Cached tags for a set of candidates, for the taste-overlap ranking. -- One row per (candidate, tag). SELECT candidate_mbid, tag, weight FROM candidate_artist_tags diff --git a/internal/db/queries/catalogue.sql b/internal/db/queries/catalogue.sql new file mode 100644 index 00000000..6d8d5309 --- /dev/null +++ b/internal/db/queries/catalogue.sql @@ -0,0 +1,101 @@ +-- The release catalogue read from MusicBrainz. + +-- name: UpsertMBArtist :exec +INSERT INTO mb_artists (mbid, name, sort_name, disambiguation, artist_type, country) +VALUES (sqlc.arg(mbid), sqlc.arg(name), sqlc.arg(sort_name), sqlc.arg(disambiguation), + sqlc.arg(artist_type), sqlc.arg(country)) +ON CONFLICT (mbid) DO UPDATE SET + name = EXCLUDED.name, + sort_name = EXCLUDED.sort_name, + disambiguation = EXCLUDED.disambiguation, + artist_type = EXCLUDED.artist_type, + country = EXCLUDED.country, + updated_at = now(); + +-- name: GetMBArtist :one +SELECT * FROM mb_artists WHERE mbid = sqlc.arg(mbid); + +-- name: MarkMBArtistDiscographyRefreshed :exec +UPDATE mb_artists SET discography_refreshed_at = now() WHERE mbid = sqlc.arg(mbid); + +-- name: ListMBArtistsDueForRefresh :many +-- Artists whose discography has never been read, or was read before the +-- cutoff: never-read first, then oldest first. +SELECT mbid FROM mb_artists +WHERE discography_refreshed_at IS NULL OR discography_refreshed_at < sqlc.arg(before) +ORDER BY discography_refreshed_at NULLS FIRST, mbid +LIMIT sqlc.arg(max_rows); + +-- name: UpsertMBReleaseGroup :exec +-- Leaves releases_refreshed_at alone: re-reading a discography says nothing +-- about whether the group's track lists are current. +INSERT INTO mb_release_groups (mbid, title, primary_type, secondary_types, first_release_date, artist_credit) +VALUES (sqlc.arg(mbid), sqlc.arg(title), sqlc.arg(primary_type), sqlc.arg(secondary_types)::text[], + sqlc.arg(first_release_date), sqlc.arg(artist_credit)) +ON CONFLICT (mbid) DO UPDATE SET + title = EXCLUDED.title, + primary_type = EXCLUDED.primary_type, + secondary_types = EXCLUDED.secondary_types, + first_release_date = EXCLUDED.first_release_date, + artist_credit = EXCLUDED.artist_credit, + updated_at = now(); + +-- name: GetMBReleaseGroup :one +SELECT * FROM mb_release_groups WHERE mbid = sqlc.arg(mbid); + +-- name: DeleteMBArtistReleaseGroups :exec +DELETE FROM mb_artist_release_groups WHERE artist_mbid = sqlc.arg(artist_mbid); + +-- name: AddMBArtistReleaseGroup :exec +INSERT INTO mb_artist_release_groups (artist_mbid, release_group_mbid) +VALUES (sqlc.arg(artist_mbid), sqlc.arg(release_group_mbid)) +ON CONFLICT DO NOTHING; + +-- name: ListMBArtistReleaseGroups :many +-- An artist's discography, oldest first; undated groups last. +SELECT g.* FROM mb_release_groups g +JOIN mb_artist_release_groups ag ON ag.release_group_mbid = g.mbid +WHERE ag.artist_mbid = sqlc.arg(artist_mbid) +ORDER BY g.first_release_date = '', g.first_release_date, g.title, g.mbid; + +-- name: MarkMBReleaseGroupReleasesRefreshed :exec +UPDATE mb_release_groups SET releases_refreshed_at = now() WHERE mbid = sqlc.arg(mbid); + +-- name: DeleteMBReleasesForGroup :exec +-- Tracks go with their releases (ON DELETE CASCADE). +DELETE FROM mb_releases WHERE release_group_mbid = sqlc.arg(release_group_mbid); + +-- name: InsertMBRelease :exec +INSERT INTO mb_releases (mbid, release_group_mbid, title, status, release_date, country, + disambiguation, formats, track_count) +VALUES (sqlc.arg(mbid), sqlc.arg(release_group_mbid), sqlc.arg(title), sqlc.arg(status), + sqlc.arg(release_date), sqlc.arg(country), sqlc.arg(disambiguation), sqlc.arg(formats), + sqlc.arg(track_count)) +ON CONFLICT (mbid) DO UPDATE SET + release_group_mbid = EXCLUDED.release_group_mbid, + title = EXCLUDED.title, + status = EXCLUDED.status, + release_date = EXCLUDED.release_date, + country = EXCLUDED.country, + disambiguation = EXCLUDED.disambiguation, + formats = EXCLUDED.formats, + track_count = EXCLUDED.track_count; + +-- name: InsertMBReleaseTrack :exec +INSERT INTO mb_release_tracks (release_mbid, medium_position, position, medium_format, number, + title, length_ms, recording_mbid) +VALUES (sqlc.arg(release_mbid), sqlc.arg(medium_position), sqlc.arg(position), + sqlc.arg(medium_format), sqlc.arg(number), sqlc.arg(title), sqlc.arg(length_ms), + sqlc.arg(recording_mbid)); + +-- name: ListMBReleasesForGroup :many +-- Official editions first, then by date; undated last. +SELECT * FROM mb_releases +WHERE release_group_mbid = sqlc.arg(release_group_mbid) +ORDER BY status <> 'Official', release_date = '', release_date, title, mbid; + +-- name: ListMBReleaseTracksForGroup :many +SELECT t.* FROM mb_release_tracks t +JOIN mb_releases r ON r.mbid = t.release_mbid +WHERE r.release_group_mbid = sqlc.arg(release_group_mbid) +ORDER BY t.release_mbid, t.medium_position, t.position; diff --git a/internal/db/queries/coplay.sql b/internal/db/queries/coplay.sql index d2e07c0a..a4714fb9 100644 --- a/internal/db/queries/coplay.sql +++ b/internal/db/queries/coplay.sql @@ -1,4 +1,4 @@ --- Household co-play similarity (#1533, milestone #160 Opt 5). Minstrel is a +-- Household co-play similarity. Minstrel is a -- single shared-library, multi-user server (no per-user library ACL), so the -- "household" is the whole instance's user set. These queries recompute -- artist–artist co-occurrence edges from play_events and store them in diff --git a/internal/db/queries/coverart_settings.sql b/internal/db/queries/coverart_settings.sql index aa3b898a..3695a885 100644 --- a/internal/db/queries/coverart_settings.sql +++ b/internal/db/queries/coverart_settings.sql @@ -1,4 +1,4 @@ --- M7 cover-sources: queries for the cover_art_provider_settings table +-- Queries for the cover_art_provider_settings table -- and the cover_art_sources_meta singleton. The SettingsService -- consumes these. @@ -13,7 +13,7 @@ SELECT provider_id, enabled, api_key, display_order, created_at, updated_at -- name: UpsertProviderSettings :exec -- INSERT a default row for a newly-registered provider, or update an -- existing row's display_order on boot reconciliation. Does NOT --- overwrite enabled / api_key on conflict — the operator's settings +-- overwrite enabled / api_key on conflict — an admin's settings -- persist across restarts. INSERT INTO cover_art_provider_settings (provider_id, enabled, display_order) VALUES ($1, $2, $3) diff --git a/internal/db/queries/covers.sql b/internal/db/queries/covers.sql index bec85f77..ecad7846 100644 --- a/internal/db/queries/covers.sql +++ b/internal/db/queries/covers.sql @@ -1,7 +1,7 @@ --- M7 #353: album cover enrichment queries. +-- Album cover enrichment queries. -- name: ListAlbumsMissingCover :many --- M7 cover-sources: returns rows the enricher should attempt. Eligible: +-- Returns rows the enricher should attempt. Eligible: -- cover_art_source IS NULL (never tried), OR cover_art_source = 'none' -- AND cover_art_sources_version != current (settled under an older -- source set; recheck against the current chain). LIMIT supplied by @@ -53,7 +53,7 @@ UPDATE albums WHERE id = $1; -- name: ClearAlbumCoverNone :exec --- M7 #380: when MBID backfill heals an album, clear its 'none' cover_art_source +-- When MBID backfill heals an album, clear its 'none' cover_art_source -- back to NULL so the enricher's next batch retries the MBCAA fetch. -- Idempotent — only updates rows currently stamped 'none'. UPDATE albums diff --git a/internal/db/queries/discover.sql b/internal/db/queries/discover.sql index 2c297610..d44049ae 100644 --- a/internal/db/queries/discover.sql +++ b/internal/db/queries/discover.sql @@ -29,7 +29,7 @@ dormant_artists AS ( SELECT t.id, t.album_id, t.artist_id FROM tracks t JOIN dormant_artists da ON da.id = t.artist_id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( SELECT 1 FROM play_events pe WHERE pe.user_id = $1 @@ -41,7 +41,7 @@ SELECT t.id, t.album_id, t.artist_id WHERE gl.user_id = $1 AND gl.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY md5(t.id::text || $2::text) @@ -56,12 +56,12 @@ SELECT t.id, t.album_id, t.artist_id -- -- GROUP BY (not SELECT DISTINCT) so the md5 ORDER BY expression need -- not appear in the select list — Postgres rejects DISTINCT + ORDER BY --- by-expression at plan time (SQLSTATE 42P10), which previously caused --- the entire Discover build to fail silently. +-- by-expression at plan time (SQLSTATE 42P10), which would fail the +-- entire Discover build. SELECT t.id, t.album_id, t.artist_id FROM general_likes gl JOIN tracks t ON t.id = gl.track_id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND gl.user_id != $1 AND NOT EXISTS ( SELECT 1 FROM play_events pe @@ -74,7 +74,7 @@ SELECT t.id, t.album_id, t.artist_id WHERE ml.user_id = $1 AND ml.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) GROUP BY t.id, t.album_id, t.artist_id @@ -88,7 +88,7 @@ SELECT t.id, t.album_id, t.artist_id -- $1 = user_id, $2 = date string for md5 ordering. SELECT t.id, t.album_id, t.artist_id FROM tracks t - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( SELECT 1 FROM play_events pe WHERE pe.user_id = $1 @@ -100,7 +100,7 @@ SELECT t.id, t.album_id, t.artist_id WHERE gl.user_id = $1 AND gl.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY md5(t.id::text || $2::text) @@ -108,7 +108,7 @@ SELECT t.id, t.album_id, t.artist_id -- name: ListTasteUnheardTracksForDiscover :many -- Taste-targeted novelty: unheard tracks whose genres overlap the user's --- taste-profile tags (taste_profile_tags, #796), ranked by summed tag +-- taste-profile tags (taste_profile_tags), ranked by summed tag -- weight — "new to you, but your vibe" rather than the crude random arm. -- Genres live inline on tracks.genre as a delimited string, split the -- same way the radio tag_overlap arm does (regexp_split_to_table on @@ -116,13 +116,13 @@ SELECT t.id, t.album_id, t.artist_id -- when the user has no taste tags yet (cold start), so the caller -- redistributes its slots to the other buckets. Stamped 'taste_unheard'. -- --- The per-album and per-artist caps apply BEFORE the LIMIT (#5356). Summed --- weight rewards a track for carrying many of the user's tags, so a few --- artists whose every track is tagged with the whole profile take every row --- of a plain LIMIT; the caller's caps then left 6 of 120 on the deploy, and --- the best-performing arm handed its slots to the others. Ranking within --- album, then within artist over what the album cap kept, is the same walk --- capByAlbumAndArtist makes, so the caller's caps keep everything here. +-- The per-album and per-artist caps apply BEFORE the LIMIT. Summed weight +-- rewards a track for carrying many of the user's tags, so a few artists +-- whose every track is tagged with the whole profile would take every row of +-- a plain LIMIT, the caller's caps would then leave a handful, and the arm +-- would hand its slots to the others. Ranking within album, then within +-- artist over what the album cap kept, is the same walk capByAlbumAndArtist +-- makes, so the caller's caps keep everything here. WITH scored AS ( SELECT t.id, t.album_id, t.artist_id, SUM(nt.weight) AS weight, @@ -130,7 +130,7 @@ WITH scored AS ( FROM tracks t JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true JOIN taste_profile_tags nt ON nt.user_id = sqlc.arg(user_id) AND trim(g_split.g) = nt.tag - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND nt.weight > 0 AND trim(g_split.g) <> '' AND NOT EXISTS ( @@ -144,7 +144,7 @@ WITH scored AS ( WHERE gl.user_id = sqlc.arg(user_id) AND gl.track_id = t.id ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = sqlc.arg(user_id) AND q.track_id = t.id ) GROUP BY t.id, t.album_id, t.artist_id diff --git a/internal/db/queries/duplicates.sql b/internal/db/queries/duplicates.sql index 7ca041be..168e6aed 100644 --- a/internal/db/queries/duplicates.sql +++ b/internal/db/queries/duplicates.sql @@ -55,14 +55,14 @@ SELECT t.id, t.duration_ms, f.audio_stream_sha256, f.chromaprint AND f.chromaprint IS NOT NULL -- Only chromaprints taken at the current length: prints at two lengths are not -- comparable, and after a length change the backfill is still re-deriving the - -- rest (#3913). + -- rest. AND f.chromaprint_length_sec = sqlc.arg(chromaprint_length_sec) AND (t.duration_ms, t.id) > (sqlc.arg(after_duration_ms)::integer, sqlc.arg(after_id)::uuid) ORDER BY t.duration_ms, t.id LIMIT sqlc.arg(page_limit); -- name: ListDismissedDuplicateMemberSets :many --- What the operator has already said are not duplicates. A new proposal whose +-- What an admin has already said are not duplicates. A new proposal whose -- every member sat together in one of these is not proposed again. SELECT g.id, array_agg(m.track_id ORDER BY m.track_id)::uuid[] AS track_ids FROM duplicate_groups g @@ -113,8 +113,8 @@ SELECT count(*)::bigint AND (SELECT count(*) FROM duplicate_group_members m WHERE m.group_id = g.id) >= 2; -- name: CountPendingDuplicateGroupsInView :one --- CountPendingDuplicateGroups narrowed to one view of the report (M498): --- review what the operator still has to decide: every class but +-- CountPendingDuplicateGroups narrowed to one view of the report: +-- review what an admin still has to decide: every class but -- cross_release, and groups not yet classified -- cross_release the same song on different releases, which is kept, not merged SELECT count(*)::bigint @@ -130,9 +130,8 @@ SELECT count(*)::bigint -- One page of proposals in one view (see CountPendingDuplicateGroupsInView), -- newest first, flattened to one row per member so the handler folds them -- without a query per group. What each copy carries — likes and plays from --- every user — is here because it is what the operator weighs when deciding --- which copy to keep; Lidarr's view of it, because a copy Lidarr tracks cannot --- be removed without Lidarr downloading it again. +-- every user — is here because it is what an admin weighs when deciding +-- which copy to keep. WITH page AS ( SELECT g.id, g.tier, g.worst_bit_error_rate, g.detected_at, g.class, g.resolve_note FROM duplicate_groups g @@ -151,7 +150,6 @@ SELECT p.id AS group_id, p.detected_at, p.class, p.resolve_note, - m.lidarr_state, t.disc_number, t.track_number, (t.mbid IS NOT NULL)::boolean AS has_mbid, @@ -162,6 +160,7 @@ SELECT p.id AS group_id, AND o.track_number = t.track_number)::boolean AS position_clash, t.id AS track_id, t.title, + artists.id AS artist_id, artists.name AS artist_name, albums.id AS album_id, albums.title AS album_title, @@ -186,49 +185,55 @@ UPDATE duplicate_groups SET status = 'dismissed', resolved_at = now() WHERE id = sqlc.arg(id) AND status = 'pending'; --- name: CountDuplicateGroupsDetectedSince :one --- Pending proposals first made at or after `since`: what a sweep that started --- then found for the first time. A refreshed proposal keeps its detected_at, --- so a sweep that only re-finds known groups counts none (M489: admins are --- told about new duplicates, not reminded of the same ones every sweep). -SELECT count(*)::bigint - FROM duplicate_groups - WHERE status = 'pending' - AND detected_at >= sqlc.arg(since); - -- name: ListDuplicateGroupsForResolve :many --- Every pending group's present copies, for the resolver (M498 #5435): what it +-- Every pending group's present copies, for the resolver: what it -- needs to classify the group and to choose the copy to keep. One row per copy, -- grouped by group. position_clash is true when another present track on the -- same album claims the same disc and track number: tags that do not fit the --- album, which the survivor rule weighs against a copy. +-- album, which the survivor rule weighs against a copy. source_verdict says +-- whether the copy is a video rip ('suspect') or one an admin kept ('fine'). +-- prior_class is the class the last pass gave the group, NULL for a group no +-- pass has seen. release_group_mbid is the album's MusicBrainz group, from its +-- tags or the library listing's match; '' when neither names one. +-- acoustid_state and acoustid_recording are what AcoustID said of the file, +-- when it was looked up: which song a recording cut to several slots is. +-- group_id narrows it to one group, for an admin settling it; NULL reads them +-- all. SELECT g.id AS group_id, g.tier, + g.worst_bit_error_rate, + g.class AS prior_class, t.id AS track_id, t.album_id, t.title, + t.source_verdict, artists.name AS artist_name, t.file_path, t.file_format, t.file_size, + t.duration_ms, t.added_at, t.disc_number, t.track_number, (t.mbid IS NOT NULL)::boolean AS has_mbid, - albums.release_group_mbid, - albums.title AS album_title, EXISTS (SELECT 1 FROM tracks o WHERE o.album_id = t.album_id AND o.id <> t.id AND o.missing_since IS NULL AND t.track_number IS NOT NULL AND o.disc_number IS NOT DISTINCT FROM t.disc_number - AND o.track_number = t.track_number)::boolean AS position_clash + AND o.track_number = t.track_number)::boolean AS position_clash, + COALESCE(NULLIF(al.release_group_mbid, ''), lam.release_group_mbid, '')::text AS release_group_mbid, + l.state AS acoustid_state, + l.recording_mbid AS acoustid_recording FROM duplicate_groups g JOIN duplicate_group_members m ON m.group_id = g.id JOIN tracks t ON t.id = m.track_id - JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id + JOIN albums al ON al.id = t.album_id + LEFT JOIN library_album_matches lam ON lam.album_id = t.album_id AND lam.outcome = 'listed' + LEFT JOIN track_acoustid_lookups l ON l.track_id = t.id WHERE g.status = 'pending' AND t.missing_since IS NULL + AND (sqlc.narg(group_id)::uuid IS NULL OR g.id = sqlc.narg(group_id)::uuid) ORDER BY g.id, t.id; -- name: SetDuplicateGroupClasses :exec @@ -244,22 +249,12 @@ UPDATE duplicate_groups g AND g.status = 'pending' AND (g.class IS DISTINCT FROM v.class OR g.resolve_note IS DISTINCT FROM NULLIF(v.note, '')); --- name: SetDuplicateMemberLidarrStates :exec --- Lidarr's view of each copy as of this pass; an empty state clears it (Lidarr --- not consulted). Unchanged rows are left alone. -UPDATE duplicate_group_members m - SET lidarr_state = NULLIF(v.state, '') - FROM (SELECT unnest(sqlc.arg(group_ids)::uuid[]) AS group_id, - unnest(sqlc.arg(track_ids)::uuid[]) AS track_id, - unnest(sqlc.arg(states)::text[]) AS state) v - WHERE m.group_id = v.group_id - AND m.track_id = v.track_id - AND m.lidarr_state IS DISTINCT FROM NULLIF(v.state, ''); +-- name: SettleDuplicateGroupAutomatically :execrows +-- The resolver closing a group itself: 'dismissed' for versions that are not +-- duplicates, 'merged' when it removed the wrong copies. Only a pending group. +UPDATE duplicate_groups + SET status = sqlc.arg(status), resolved_at = now(), resolved_automatically = true + WHERE id = sqlc.arg(id) AND status = 'pending'; -- name: MarkDuplicateGroupResolvedAutomatically :exec UPDATE duplicate_groups SET resolved_automatically = true WHERE id = sqlc.arg(id); - --- name: ListAlbumPresentTrackTitles :many --- The titles an album has on disk, for choosing the Lidarr release that covers --- them best (M498 #5437). -SELECT title FROM tracks WHERE album_id = sqlc.arg(album_id) AND missing_since IS NULL; diff --git a/internal/db/queries/events.sql b/internal/db/queries/events.sql index 2be5df2f..5a245ff5 100644 --- a/internal/db/queries/events.sql +++ b/internal/db/queries/events.sql @@ -26,7 +26,7 @@ LIMIT 1; -- name: InsertPlayEvent :one -- pick_kind is non-NULL only for system-playlist plays whose track was -- found (with a stamped kind) in the user's live snapshot for that --- variant at ingestion time (#1249, generalized in #1270). +-- variant at ingestion time. INSERT INTO play_events ( user_id, track_id, session_id, started_at, client_id, source, pick_kind, device_class @@ -38,7 +38,7 @@ RETURNING *; -- Looks a track up in the user's CURRENT snapshot of the given system -- variant and returns its pick_kind. Used at play-ingestion time to -- freeze provenance onto the play_event — snapshots rebuild daily, so --- attribution can't be reconstructed at read time (#1249/#1270). No +-- attribution can't be reconstructed at read time. No -- row = track not in today's snapshot (caller stores NULL); a row with -- NULL pick_kind = the variant doesn't stamp (yet). songs_like_artist -- is non-singleton (up to 3 mixes/user); a track in two of them takes diff --git a/internal/db/queries/fingerprints.sql b/internal/db/queries/fingerprints.sql index ae68f233..38e77634 100644 --- a/internal/db/queries/fingerprints.sql +++ b/internal/db/queries/fingerprints.sql @@ -1,6 +1,6 @@ -- name: UpsertTrackFingerprint :exec -- Written whenever a track's fingerprint is derived: by the scan when a file is --- new or its bytes changed, and by the backfill (#3908) for rows derived by an +-- new or its bytes changed, and by the backfill for rows derived by an -- older method. Replaces the row wholesale — a fingerprint of the old bytes has -- no standing once the file has changed. INSERT INTO track_fingerprints ( @@ -23,7 +23,7 @@ ON CONFLICT (track_id) DO UPDATE SET DELETE FROM track_fingerprints WHERE track_id = $1; -- name: ListTracksNeedingFingerprint :many --- The backfill's work queue (#3908): tracks with no fingerprint, or one derived +-- The backfill's work queue: tracks with no fingerprint, or one derived -- by an older method. Keyset-paged on id so a pass visits each track at most -- once. That cursor is load-bearing: an inconclusive attempt writes no row, so -- without it a file that keeps timing out would be listed again straight away @@ -34,7 +34,7 @@ SELECT t.id, t.file_path LEFT JOIN track_fingerprints f ON f.track_id = t.id WHERE t.missing_since IS NULL -- A row taken at another length is as stale as one from an older method: - -- chromaprints at two lengths cannot be compared (#3913). + -- chromaprints at two lengths cannot be compared. AND (f.track_id IS NULL OR f.fingerprint_version < sqlc.arg(current_version) OR f.chromaprint_length_sec <> sqlc.arg(chromaprint_length_sec)) diff --git a/internal/db/queries/gc.sql b/internal/db/queries/gc.sql index 6934ca41..e5d50620 100644 --- a/internal/db/queries/gc.sql +++ b/internal/db/queries/gc.sql @@ -1,11 +1,10 @@ --- Background garbage-collector / lifecycle queries. All five address --- drift findings from the 2026-06-02 audit (Scribe parent #552): --- #565 #566 #567 #574 #575. Sweeper runs on a long tick (default 1h) +-- Background garbage-collector / lifecycle queries: rows nothing else closes +-- or removes. Sweeper runs on a long tick (default 1h) -- so per-query cost is amortised; each is idempotent (re-running on -- already-closed/-deleted rows is a no-op). -- name: GcCloseStalePlayEvents :execrows --- #566: play_events rows opened more than 24h ago that never got a +-- Play_events rows opened more than 24h ago that never got a -- play_ended. Client crashed mid-track, network dropped, etc. We -- synthesize ended_at = started_at + duration_played_ms when present, -- otherwise leave duration_played_ms null and stamp ended_at = now() @@ -21,10 +20,9 @@ UPDATE play_events AND started_at < now() - INTERVAL '24 hours'; -- name: GcClosePlaySessionsWithNoRecentEvents :execrows --- #565: play_sessions.ended_at was added but never populated by any --- writer. Close sessions whose last_event_at is older than 6h — --- treating that as "user moved on" the same way audio_service does --- after grace periods. Sessions with NO events (track_count = 0) +-- play_sessions.ended_at has no other writer. Close sessions whose +-- last_event_at is older than 6h — treating that as "user moved on" the +-- same way audio_service does after grace periods. Sessions with NO events (track_count = 0) -- older than 1h are also closed (stale handshakes from clients that -- never recorded a play). ended_at is set to last_event_at so the -- session's duration reads naturally. @@ -38,7 +36,7 @@ UPDATE play_sessions ); -- name: GcExpireScrobbleQueueFailedRows :execrows --- #567: scrobble_queue rows that have been in status='failed' for +-- Scrobble_queue rows that have been in status='failed' for -- more than 14 days. The worker stops retrying after maxAttempts; -- failed rows accumulate forever otherwise. CASCADE from play_events -- already drops the row when the underlying event is deleted, so this @@ -48,7 +46,7 @@ DELETE FROM scrobble_queue AND enqueued_at < now() - INTERVAL '14 days'; -- name: GcResetStuckSystemPlaylistRuns :execrows --- #574: system_playlist_runs.in_flight = true can wedge on a +-- System_playlist_runs.in_flight = true can wedge on a -- goroutine panic between SET in_flight=true and SET in_flight=false. -- The duplicate-prevention check refuses to start a fresh regen while -- in_flight, so a stuck row blocks all future regens for that user. @@ -61,7 +59,7 @@ UPDATE system_playlist_runs AND last_run_at < now() - INTERVAL '10 minutes'; -- name: GcDeleteExpiredPasswordResets :execrows --- #575: password_resets accumulates expired + used rows forever. The +-- Password_resets accumulates expired + used rows forever. The -- validation path already rejects them; this just keeps the table -- from growing unbounded. Used rows are kept for 7 days for audit; -- unused expired rows go immediately. diff --git a/internal/db/queries/history.sql b/internal/db/queries/history.sql index dd6176e4..ab39bba5 100644 --- a/internal/db/queries/history.sql +++ b/internal/db/queries/history.sql @@ -1,5 +1,5 @@ -- name: ListUserHistory :many --- M7 #365: track-level listening history. Returns one row per +-- Track-level listening history. Returns one row per -- play_events entry, newest first, joined with tracks/albums/artists -- so the SPA can render a TrackRef without follow-up fetches. Filters -- out skipped events and tracks the user has quarantined (per-user @@ -16,7 +16,7 @@ SELECT pe.id AS event_id, WHERE pe.user_id = $1 AND pe.was_skipped = false AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY pe.started_at DESC diff --git a/internal/db/queries/imports.sql b/internal/db/queries/imports.sql new file mode 100644 index 00000000..e373303c --- /dev/null +++ b/internal/db/queries/imports.sql @@ -0,0 +1,176 @@ +-- Import with verification. + +-- name: ListLiveGrabs :many +-- Every download still running or waiting on its import, oldest first. +SELECT g.*, w.release_group_mbid, w.artist_mbid, w.artist_name, w.title AS want_title, + w.reason AS want_reason, w.album_id AS want_album_id + FROM grabs g + LEFT JOIN wants w ON w.id = g.want_id + WHERE g.ended_at IS NULL + ORDER BY g.grabbed_at, g.id; + +-- name: GetGrab :one +SELECT * FROM grabs WHERE id = $1; + +-- name: SetGrabProgress :exec +-- What the download client says of a download not yet finished, or finished +-- and waiting for what the import needs. Leaves a claimed, decided or failed +-- import alone. +UPDATE grabs + SET progress = sqlc.arg(progress), + import_state = sqlc.arg(import_state), + import_reason = sqlc.arg(import_reason), + import_updated_at = now() + WHERE id = sqlc.arg(id) AND ended_at IS NULL + AND import_state IN ('', 'downloading', 'waiting'); + +-- name: ClaimGrabImport :one +-- Takes a finished download for import. The update is the lock: a second +-- pass finds nothing to claim. +UPDATE grabs + SET import_state = 'importing', import_attempts = import_attempts + 1, + import_updated_at = now(), progress = 1 + WHERE id = $1 AND ended_at IS NULL AND import_state IN ('', 'downloading', 'waiting') +RETURNING *; + +-- name: ReclaimStaleImports :execrows +-- An import cut short (a restart mid-copy) is tried again. +UPDATE grabs + SET import_state = 'waiting', + import_reason = 'The last import was cut short, so it is being tried again.' + WHERE ended_at IS NULL AND import_state = 'importing' + AND import_updated_at < now() - make_interval(mins => sqlc.arg(minutes)::int); + +-- name: RecordImport :one +-- The import's outcome. An end reason ends the grab: imported, refused or +-- held. Without one the grab stays live: review and failed wait for a person. +UPDATE grabs + SET import_state = sqlc.arg(import_state), + import_reason = sqlc.arg(import_reason), + import_detail = sqlc.arg(import_detail), + import_updated_at = now(), + import_forced = false, + imported_at = CASE WHEN sqlc.arg(import_state)::text = 'imported' THEN now() ELSE imported_at END, + -- The files were copied, so an imported download goes on sharing + -- until the lifecycle removes it. + seeding = sqlc.arg(import_state)::text = 'imported', + imported_files = sqlc.arg(imported_files), + destination = sqlc.arg(destination), + ended_at = CASE WHEN sqlc.arg(end_reason)::text <> '' THEN now() ELSE ended_at END, + end_reason = sqlc.arg(end_reason), + end_detail = CASE WHEN sqlc.arg(end_reason)::text <> '' THEN sqlc.arg(import_reason)::text ELSE end_detail END + WHERE id = sqlc.arg(id) +RETURNING *; + +-- name: ForceGrabImport :one +-- "Import anyway": the next pass imports the download whatever the check said. +UPDATE grabs + SET import_state = '', import_forced = true, import_reason = '', import_updated_at = now() + WHERE id = $1 AND ended_at IS NULL AND import_state IN ('review', 'failed') +RETURNING *; + +-- name: RetryGrabImport :one +UPDATE grabs + SET import_state = '', import_reason = '', import_updated_at = now() + WHERE id = $1 AND ended_at IS NULL AND import_state = 'failed' +RETURNING *; + +-- name: RehomeGrab :one +-- A download that turned out to be another wanted album is that want's grab, +-- when the want has none running. +UPDATE grabs g + SET want_id = sqlc.arg(want_id) + WHERE g.id = sqlc.arg(id) AND g.ended_at IS NULL + AND NOT EXISTS (SELECT 1 FROM grabs o WHERE o.want_id = sqlc.arg(want_id) AND o.ended_at IS NULL) +RETURNING *; + +-- name: OpenWantsForGroups :many +-- Open wants for these release groups with no download running. +SELECT w.* FROM wants w + WHERE w.fulfilled_at IS NULL AND w.release_group_mbid = ANY(sqlc.arg(groups)::text[]) + AND NOT EXISTS (SELECT 1 FROM grabs g WHERE g.want_id = w.id AND g.ended_at IS NULL); + +-- name: MarkWantSearched :exec +-- Imported: nothing to search for until the wanted pass settles the want. +UPDATE wants SET searched_at = now() WHERE id = $1; + +-- name: SearchWantAgain :exec +-- Refused: search again on the next pass. +UPDATE wants SET searched_at = NULL, search_outcome = '' WHERE id = $1; + +-- name: HeldRecordings :many +-- Which of these recordings the library holds a file for. +SELECT DISTINCT mbid::text AS mbid FROM tracks + WHERE mbid = ANY(sqlc.arg(recordings)::text[]) AND missing_since IS NULL; + +-- name: HeldPositions :many +-- The disc and track positions the library holds for an album of this group. +SELECT DISTINCT COALESCE(t.disc_number, 1)::int AS disc, t.track_number::int AS track + FROM tracks t + JOIN albums a ON a.id = t.album_id + WHERE a.release_group_mbid = sqlc.arg(release_group_mbid) + AND t.missing_since IS NULL AND t.track_number IS NOT NULL; + +-- name: ListGrabs :many +-- The Wanted view's Activity tab: downloads waiting on a person first, then +-- the rest still running, then the ended ones newest first. Each carries +-- what is wrong with it now, if anything. +SELECT g.id, g.want_id, g.name, g.indexer, g.client, g.quality, g.origin, g.forced, + g.grabbed_at, g.ended_at, g.end_reason, g.end_detail, g.progress, g.import_state, + g.import_reason, g.import_detail, g.import_updated_at, g.imported_at, g.imported_files, + g.destination, g.seeding, g.ratio, g.seed_released_at, + w.title AS want_title, w.artist_name AS want_artist, + di.trigger AS condition, di.detail AS condition_detail, di.remediations AS condition_retries, + di.first_seen AS condition_since + FROM grabs g + LEFT JOIN wants w ON w.id = g.want_id + LEFT JOIN download_interventions di + ON di.id = (SELECT x.id FROM download_interventions x + WHERE x.grab_id = g.id AND x.outcome = 'open' + ORDER BY x.last_seen DESC LIMIT 1) + ORDER BY (g.ended_at IS NULL AND g.import_state IN ('review', 'failed')) DESC, + g.ended_at IS NULL DESC, g.grabbed_at DESC, g.id + LIMIT sqlc.arg(max_rows) OFFSET sqlc.arg(skip); + +-- name: ListReleaseBlocksPage :many +SELECT b.info_hash, b.name, b.reason, b.detail, b.blocked_at, b.expires_at, w.title AS want_title, + w.artist_name AS want_artist + FROM release_blocks b + LEFT JOIN wants w ON w.id = b.want_id + WHERE b.expires_at IS NULL OR b.expires_at > now() + ORDER BY b.blocked_at DESC, b.info_hash + LIMIT sqlc.arg(max_rows) OFFSET sqlc.arg(skip); + +-- name: ListUpgradeTracks :many +-- The tracks an upgrade may replace: the present tracks of the album +-- the want holds. +SELECT id, file_path, mbid, COALESCE(disc_number, 1)::int AS disc, track_number, + file_format, file_size, duration_ms, source_verdict + FROM tracks + WHERE album_id = $1 AND missing_since IS NULL; + +-- name: FinishUpgrade :exec +-- An upgrade was imported: the want holds the album the new tracks are in, +-- and is judged again by the next upgrade check. +UPDATE wants + SET album_id = COALESCE(sqlc.narg(album_id), album_id), + upgrade = '', + upgrade_checked_at = NULL, + updated_at = now() + WHERE id = sqlc.arg(id); + +-- name: ClearTrackSourceVerdict :exec +-- A track whose file was replaced in place is no longer the rip it was. +UPDATE tracks SET source_verdict = NULL, source_verdict_at = NULL + WHERE id = $1 AND source_verdict = 'suspect'; + +-- name: ListImportedGrabsHolding :many +-- The downloads Minstrel imported into the folder holding path: the releases a +-- file there came from, to block when the files proved not to be the album +--. A file imported before Minstrel downloaded anything has none. +SELECT id, info_hash, name, want_id + FROM grabs + WHERE import_state = 'imported' + AND info_hash <> '' + AND destination <> '' + AND starts_with(sqlc.arg(path)::text, rtrim(destination, '/') || '/'); diff --git a/internal/db/queries/library_changes.sql b/internal/db/queries/library_changes.sql index 86a17848..0437b36a 100644 --- a/internal/db/queries/library_changes.sql +++ b/internal/db/queries/library_changes.sql @@ -3,11 +3,10 @@ INSERT INTO library_changes (entity_type, entity_id, op) VALUES ($1, $2, $3); -- name: InsertLibraryChanges :exec --- Batch form, for a mutation that touches many rows at once (#2704: the scan's --- reconcile pass can mark up to a quarter of a library missing in one go). --- Every caller before this one changed a single entity, so per-row inserts --- were the right shape; a loop here would be thousands of round-trips inside --- a scan that is already the slow path. +-- Batch form, for a mutation that touches many rows at once (the scan's +-- reconcile pass can mark up to a quarter of a library missing in one go). A +-- loop of single inserts would be thousands of round-trips inside a scan that +-- is already the slow path. INSERT INTO library_changes (entity_type, entity_id, op) SELECT sqlc.arg(entity_type)::text, unnest(sqlc.arg(entity_ids)::text[]), diff --git a/internal/db/queries/library_settings.sql b/internal/db/queries/library_settings.sql new file mode 100644 index 00000000..99d0fd37 --- /dev/null +++ b/internal/db/queries/library_settings.sql @@ -0,0 +1,45 @@ +-- name: GetLibrarySettings :one +SELECT * FROM library_settings WHERE id = true; + +-- name: UpdateLibraryScanPaths :one +UPDATE library_settings + SET scan_paths = sqlc.arg(scan_paths)::text[], + updated_at = now() + WHERE id = true +RETURNING *; + +-- A library folder that moved: every path under old_root is rewritten +-- to the same place under new_root, so the tracks keep their ids and with them +-- their plays, likes and playlists. updated_at is left alone on purpose: the +-- scan compares it with each file's mtime, and the files have not changed. + +-- name: RerootTrackPaths :execrows +UPDATE tracks + SET file_path = sqlc.arg(new_root)::text || substr(file_path, length(sqlc.arg(old_root)::text) + 1) + WHERE starts_with(file_path, sqlc.arg(old_root)::text || '/'); + +-- name: RerootAlbumCoverPaths :execrows +UPDATE albums + SET cover_art_path = sqlc.arg(new_root)::text || substr(cover_art_path, length(sqlc.arg(old_root)::text) + 1) + WHERE starts_with(cover_art_path, sqlc.arg(old_root)::text || '/'); + +-- name: RerootGrabDestinations :execrows +UPDATE grabs + SET destination = sqlc.arg(new_root)::text || substr(destination, length(sqlc.arg(old_root)::text) + 1) + WHERE destination = sqlc.arg(old_root)::text + OR starts_with(destination, sqlc.arg(old_root)::text || '/'); + +-- name: RerootImportRoot :execrows +UPDATE acquisition_settings + SET import_root = sqlc.arg(new_root)::text + WHERE import_root = sqlc.arg(old_root)::text; + +-- name: SampleTrackPathsUnder :many +-- A few present tracks under a folder, to check a move before rewriting it. +SELECT file_path FROM tracks + WHERE starts_with(file_path, sqlc.arg(root)::text || '/') AND missing_since IS NULL + ORDER BY random() + LIMIT sqlc.arg(n)::int; + +-- name: CountTracksUnder :one +SELECT count(*) FROM tracks WHERE starts_with(file_path, sqlc.arg(root)::text || '/'); diff --git a/internal/db/queries/lidarr_config.sql b/internal/db/queries/lidarr_config.sql deleted file mode 100644 index 5b31c633..00000000 --- a/internal/db/queries/lidarr_config.sql +++ /dev/null @@ -1,20 +0,0 @@ --- name: GetLidarrConfig :one -SELECT id, enabled, base_url, api_key, default_quality_profile_id, - default_metadata_profile_id, default_root_folder_path, - created_at, updated_at -FROM lidarr_config -WHERE id = 1; - --- name: UpdateLidarrConfig :one -UPDATE lidarr_config - SET enabled = $1, - base_url = $2, - api_key = $3, - default_quality_profile_id = $4, - default_metadata_profile_id = $5, - default_root_folder_path = $6, - updated_at = now() - WHERE id = 1 - RETURNING id, enabled, base_url, api_key, default_quality_profile_id, - default_metadata_profile_id, default_root_folder_path, - created_at, updated_at; diff --git a/internal/db/queries/lidarr_requests.sql b/internal/db/queries/lidarr_requests.sql deleted file mode 100644 index ec96568c..00000000 --- a/internal/db/queries/lidarr_requests.sql +++ /dev/null @@ -1,130 +0,0 @@ --- name: CreateLidarrRequest :one -INSERT INTO lidarr_requests ( - user_id, kind, - lidarr_artist_mbid, lidarr_album_mbid, lidarr_track_mbid, - artist_name, album_title, track_title -) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) -RETURNING *; - --- name: GetLidarrRequestByID :one -SELECT * FROM lidarr_requests WHERE id = $1; - --- name: ListLidarrRequestsForUser :many -SELECT * FROM lidarr_requests -WHERE user_id = $1 -ORDER BY requested_at DESC -LIMIT $2; - --- name: ListLidarrRequestsByStatus :many -SELECT * FROM lidarr_requests -WHERE status = $1 -ORDER BY requested_at DESC -LIMIT $2; - --- name: ListApprovedLidarrRequestsForReconcile :many -SELECT * FROM lidarr_requests -WHERE status = 'approved' -ORDER BY decided_at ASC -LIMIT $1; - --- name: ApproveLidarrRequest :one -UPDATE lidarr_requests - SET status = 'approved', - quality_profile_id = $2, - root_folder_path = $3, - decided_at = now(), - decided_by = $4, - updated_at = now() - WHERE id = $1 AND status = 'pending' - RETURNING *; - --- name: RejectLidarrRequest :one -UPDATE lidarr_requests - SET status = 'rejected', - notes = $2, - decided_at = now(), - decided_by = $3, - updated_at = now() - WHERE id = $1 AND status = 'pending' - RETURNING *; - --- name: CancelLidarrRequest :one --- $2 serves as both the ownership guard (user_id = $2) and the --- decided_by audit field. The generated parameter struct names it --- DecidedBy — callers must pass the cancelling user's UUID, and --- a wrong UUID will silently no-op (zero rows updated) rather than --- erroring at the SQL layer. -UPDATE lidarr_requests - SET status = 'rejected', - notes = 'cancelled by user', - decided_at = now(), - decided_by = $2, - updated_at = now() - WHERE id = $1 AND user_id = $2 AND status = 'pending' - RETURNING *; - --- name: CompleteLidarrRequest :one --- Reconciler transitions an approved request to completed when its --- target track/album/artist has appeared in the library. Callers --- should set ONLY the matched_*_id corresponding to the request's --- kind and pass pgtype.UUID{} (Valid: false) for the others — --- per-kind one-of-three is the contract, NULL for the others --- preserves auditability if the matched row is later evicted. -UPDATE lidarr_requests - SET status = 'completed', - matched_track_id = $2, - matched_album_id = $3, - matched_artist_id = $4, - completed_at = now(), - updated_at = now() - WHERE id = $1 AND status = 'approved' - RETURNING *; - --- name: HasNonTerminalRequestForMBID :one --- Returns true if any user has a pending/approved/completed request --- whose MBID matches at the given level. Used to set the `requested` --- flag on /api/lidarr/search responses. Terminal-status (rejected, --- failed) rows do not count. The same MBID parameter is checked --- against artist/album/track columns based on kind. -SELECT EXISTS ( - SELECT 1 FROM lidarr_requests - WHERE status IN ('pending', 'approved', 'completed') - AND ((kind = 'artist' AND lidarr_artist_mbid = @mbid) - OR (kind = 'album' AND lidarr_album_mbid = @mbid) - OR (kind = 'track' AND lidarr_track_mbid = @mbid)) -); - --- name: GetNonTerminalRequestForMBID :one --- Returns the oldest non-terminal (pending/approved/completed) request --- whose own-kind MBID matches @mbid, or pgx.ErrNoRows when none. Create() --- uses this to dedupe: a re-request for something already in flight --- returns the existing row instead of inserting a duplicate. MBIDs are --- globally unique per entity type, so the cross-column OR cannot --- false-match (an artist MBID never collides with an album/track MBID). -SELECT * FROM lidarr_requests - WHERE status IN ('pending', 'approved', 'completed') - AND ((kind = 'artist' AND lidarr_artist_mbid = @mbid) - OR (kind = 'album' AND lidarr_album_mbid = @mbid) - OR (kind = 'track' AND lidarr_track_mbid = @mbid)) - ORDER BY requested_at ASC - LIMIT 1; - --- name: MarkLidarrRequestAddConfirmed :exec --- Idempotently records that Lidarr accepted the add for an approved --- request. The lidarr_add_confirmed_at IS NULL guard makes a redundant --- reconciler retry a no-op rather than bumping updated_at every tick. -UPDATE lidarr_requests - SET lidarr_add_confirmed_at = now(), - updated_at = now() - WHERE id = $1 - AND status = 'approved' - AND lidarr_add_confirmed_at IS NULL; - --- name: SetLidarrRequestAlbumMbid :exec --- #5241: repoint a request at the album's release group. Requests made before --- the release-group read named albums by MusicBrainz release id, which Lidarr --- does not index; the reconciler rewrites them once it has resolved the group. -UPDATE lidarr_requests - SET lidarr_album_mbid = $2, - updated_at = now() - WHERE id = $1; diff --git a/internal/db/queries/lifecycle.sql b/internal/db/queries/lifecycle.sql new file mode 100644 index 00000000..8b5b21c0 --- /dev/null +++ b/internal/db/queries/lifecycle.sql @@ -0,0 +1,102 @@ +-- The download lifecycle: the intervention ledger, giving a +-- download up, and seeding after import. + +-- name: RaiseIntervention :one +-- A condition seen, or seen again: a second pass lands on the open row, so +-- a download stuck for a week is one row that says how long. The name only +-- improves, and the detail is the latest word. +INSERT INTO download_interventions (client, info_hash, grab_id, name, trigger, detail) +VALUES (sqlc.arg(client), sqlc.arg(info_hash), sqlc.narg(grab_id), sqlc.arg(name), + sqlc.arg(trigger), sqlc.arg(detail)) +ON CONFLICT (client, info_hash, trigger) WHERE outcome = 'open' DO UPDATE SET + observations = download_interventions.observations + 1, + last_seen = now(), + detail = EXCLUDED.detail, + grab_id = COALESCE(EXCLUDED.grab_id, download_interventions.grab_id), + name = CASE WHEN EXCLUDED.name <> '' THEN EXCLUDED.name ELSE download_interventions.name END +RETURNING *; + +-- name: SettleIntervention :execrows +-- The condition cleared. +UPDATE download_interventions + SET outcome = 'resolved', settled_at = now(), last_seen = now() + WHERE client = sqlc.arg(client) AND info_hash = sqlc.arg(info_hash) + AND trigger = sqlc.arg(trigger) AND outcome = 'open'; + +-- name: SettleInterventionsFor :execrows +-- Everything open about one download is over: it is gone from the client. +UPDATE download_interventions + SET outcome = 'resolved', settled_at = now(), last_seen = now() + WHERE client = sqlc.arg(client) AND info_hash = sqlc.arg(info_hash) AND outcome = 'open'; + +-- name: SettleEndedGrabs :execrows +-- What was wrong with a download whose grab has ended (imported, refused, +-- given up) is over, except a download left in the client to remove. +UPDATE download_interventions di + SET outcome = 'resolved', settled_at = now(), last_seen = now() + FROM grabs g + WHERE g.id = di.grab_id AND g.ended_at IS NOT NULL + AND di.outcome = 'open' AND di.trigger <> 'not_removed'; + +-- name: CountRemediation :one +-- Minstrel acted on an open condition. A retry climbs the ladder; giving up +-- or removing ends the condition, since what it was about is gone. Written +-- before the act, so a retry that happened is never uncounted. +UPDATE download_interventions + SET remediations = remediations + 1, + action = sqlc.arg(action), + last_seen = now(), + outcome = CASE WHEN sqlc.arg(action)::text IN ('given_up', 'removed') THEN 'resolved' ELSE outcome END, + settled_at = CASE WHEN sqlc.arg(action)::text IN ('given_up', 'removed') THEN now() ELSE settled_at END + WHERE client = sqlc.arg(client) AND info_hash = sqlc.arg(info_hash) + AND trigger = sqlc.arg(trigger) AND outcome = 'open' +RETURNING remediations; + +-- name: ListLeftovers :many +-- Downloads given up on that the client would not remove; each pass asks +-- again. +SELECT * FROM download_interventions + WHERE trigger = 'not_removed' AND outcome = 'open' AND info_hash <> '' + ORDER BY first_seen; + +-- name: ListClientsDown :many +-- Download clients not answering now, for the Activity tab. +SELECT client, detail, first_seen FROM download_interventions + WHERE trigger = 'client_unreachable' AND outcome = 'open' + ORDER BY client; + +-- name: LiveGrabOnHash :one +SELECT EXISTS (SELECT 1 FROM grabs WHERE info_hash = sqlc.arg(info_hash) AND ended_at IS NULL)::bool; + +-- name: GiveUpGrab :one +-- Ends a running grab, given up on. Only from the states named: the update +-- is the lock against an import that claimed the download meanwhile. +UPDATE grabs + SET ended_at = now(), end_reason = sqlc.arg(end_reason), end_detail = sqlc.arg(end_detail), + import_updated_at = now() + WHERE id = sqlc.arg(id) AND ended_at IS NULL AND import_state = ANY(sqlc.arg(states)::text[]) +RETURNING *; + +-- name: CountGoneGrabs :one +-- How many of a release's grabs ended gone since a time: a release that keeps +-- leaving its client is held back rather than taken on every pass. +SELECT count(*) FROM grabs + WHERE info_hash = sqlc.arg(info_hash) AND end_reason = 'gone' AND ended_at > sqlc.arg(since); + +-- name: ListSeedingGrabs :many +-- Imported downloads still sharing, oldest import first. +SELECT id, client, info_hash, name, imported_at FROM grabs + WHERE seeding AND seed_released_at IS NULL + ORDER BY imported_at, id; + +-- name: SetGrabRatio :exec +UPDATE grabs SET ratio = sqlc.arg(ratio) WHERE id = sqlc.arg(id); + +-- name: ReleaseSeed :exec +-- The seeding download was removed, or found gone. +UPDATE grabs SET seed_released_at = now() WHERE id = $1 AND seed_released_at IS NULL; + +-- name: DeleteExpiredBlocks :execrows +-- A release given up on is held back for a while; after that the search may +-- take it again. +DELETE FROM release_blocks WHERE expires_at IS NOT NULL AND expires_at <= now(); diff --git a/internal/db/queries/likes.sql b/internal/db/queries/likes.sql index 350b7f59..11289c29 100644 --- a/internal/db/queries/likes.sql +++ b/internal/db/queries/likes.sql @@ -1,5 +1,5 @@ -- name: LikeTrack :many --- A like is on the song (M498 #5438): every copy of it sharing the track's +-- A like is on the song: every copy of it sharing the track's -- song_key is liked. Returns the tracks newly liked, for the sync log; empty -- when the song was already liked. INSERT INTO general_likes (user_id, track_id) diff --git a/internal/db/queries/loudness.sql b/internal/db/queries/loudness.sql index 90a92c2d..39087d72 100644 --- a/internal/db/queries/loudness.sql +++ b/internal/db/queries/loudness.sql @@ -1,5 +1,5 @@ -- name: UpsertTrackLoudness :exec --- Written when the backfill measures a track (#4995). Replaces the row +-- Written when the backfill measures a track. Replaces the row -- wholesale: a measurement of the old bytes has no standing once the file has -- changed. INSERT INTO track_loudness ( @@ -78,7 +78,7 @@ UPDATE loudness_settings RETURNING *; -- name: ListAlbumsNeedingLoudness :many --- The album pass's work queue (#4996): albums whose present tracks or their +-- The album pass's work queue: albums whose present tracks or their -- measurements have changed since album loudness was last computed, or that -- never had it. The digest is over every present track's id and the -- measurement it holds at the current version ('-' for none), so a track @@ -154,7 +154,7 @@ SELECT integrated_lufs, true_peak_dbtp FROM album_loudness WHERE album_id = $1; -- name: GetReplayGainByTrackIDs :many -- Track and album loudness for a set of tracks, for every surface that hands --- gains to a client (#4997). A measurement from an older analysis version is +-- gains to a client. A measurement from an older analysis version is -- still delivered: it is a better gain than none until the backfill redoes it. SELECT t.id, tl.integrated_lufs AS track_lufs, diff --git a/internal/db/queries/merge.sql b/internal/db/queries/merge.sql index 024e5d58..4ea5001d 100644 --- a/internal/db/queries/merge.sql +++ b/internal/db/queries/merge.sql @@ -1,4 +1,4 @@ --- Duplicate merge (Scribe #3911). Every statement here runs inside the one +-- Duplicate merge. Every statement here runs inside the one -- transaction library.MergeDuplicateGroup opens, after the removed copy's file -- is already gone. The loser's own track row is deleted last with DeleteTrack; -- what these do is move everything it carries onto the survivor first, so that @@ -13,11 +13,9 @@ SELECT id, tier, status FOR UPDATE; -- name: ListDuplicateGroupMergeMembers :many -SELECT t.id, t.file_path, t.file_format, t.file_size, t.added_at, t.album_id, - t.mbid, albums.mbid AS album_mbid +SELECT t.id, t.file_path, t.file_format, t.file_size, t.added_at, t.album_id FROM duplicate_group_members m - JOIN tracks t ON t.id = m.track_id - JOIN albums ON albums.id = t.album_id + JOIN tracks t ON t.id = m.track_id WHERE m.group_id = sqlc.arg(group_id) ORDER BY t.id; @@ -35,8 +33,8 @@ UPDATE contextual_likes SET track_id = sqlc.arg(survivor_id)::uuid WHERE track_i -- name: MergeRepointPlaybackErrors :execrows UPDATE playback_errors SET track_id = sqlc.arg(survivor_id)::uuid WHERE track_id = sqlc.arg(loser_id)::uuid; --- name: MergeRepointLidarrRequests :execrows -UPDATE lidarr_requests SET matched_track_id = sqlc.arg(survivor_id)::uuid +-- name: MergeRepointRequests :execrows +UPDATE requests SET matched_track_id = sqlc.arg(survivor_id)::uuid WHERE matched_track_id = sqlc.arg(loser_id)::uuid; -- name: MergeRepointPlaylistTracks :many diff --git a/internal/db/queries/missing_pairs.sql b/internal/db/queries/missing_pairs.sql index 322a5dfb..f7eec982 100644 --- a/internal/db/queries/missing_pairs.sql +++ b/internal/db/queries/missing_pairs.sql @@ -1,30 +1,34 @@ --- Missing-pair pass (M485). A track marked missing whose replacement is already --- on disk under another row — most often a Lidarr quality upgrade, mp3 → flac at --- a new path — is folded into that replacement, so its history moves across and +-- Missing-pair pass. A track marked missing whose replacement is already +-- on disk under another row — most often a quality upgrade, mp3 → flac at a new +-- path — is folded into that replacement, so its history moves across and -- re-acquisition stops chasing music the library already has. -- --- Move adoption (#2528) cannot catch these: it links a new file to a missing row +-- Move adoption cannot catch these: it links a new file to a missing row -- at the moment the file is first inserted, and the replacement arrives as a -- re-encode with no recording MBID, so it has nothing to match on. AcoustID may -- give it an MBID later; by then both rows exist and only a fold can join them. -- name: ListMissingTrackPairs :many -- Every candidate (missing, present) pair, with how many candidates each side --- has. Two rules, measured on the operator's library on 2026-10-07 (#5274): +-- has. Two rules: -- -- mbid the same recording MBID, within the album group (the album row, or --- albums sharing a release group). 142 of 359 pairs. --- title the same album row and the same title, ignoring case. 213 more. The +-- albums sharing a release group). +-- title the same album row and the same title, ignoring case. The -- replacements are often not the original file — another edition, -- re-encoded, untagged — so the metadata is what is left to go on. -- --- Deliberately NOT gated on duration or track position. On the 142 pairs known --- to be one recording, 18% differed by over 2s (old mp3 durations are poor --- estimates), and the poorly tagged set is exactly where numbering is broken. +-- Deliberately NOT gated on duration or track position. Pairs known to be one +-- recording often differ by over 2s (old mp3 durations are poor estimates), +-- and the poorly tagged set is exactly where numbering is broken. -- -- A pair whose MBIDs are both set and differ is not a title match: the tags say -- they are different recordings. The caller folds a pair only when each side has -- exactly one candidate; anything else is ambiguous and left alone. +-- +-- A slot the duplicate resolver emptied (emptied_at, migration 0095) pairs by +-- title only with a track added since: the album held no copy of the song as +-- long as it is, so one already there under the title is another song. WITH grp AS ( SELECT id AS album_id, COALESCE(release_group_mbid, id::text) AS g FROM albums @@ -47,6 +51,7 @@ cand AS ( AND lower(p.title) = lower(m.title) WHERE m.missing_since IS NOT NULL AND (m.mbid IS NULL OR p.mbid IS NULL OR m.mbid = p.mbid) + AND (m.emptied_at IS DISTINCT FROM m.missing_since OR p.added_at > m.emptied_at) ), pairs AS ( -- A pair both rules found is one candidate, credited to the stronger rule. diff --git a/internal/db/queries/notifications.sql b/internal/db/queries/notifications.sql index fa82ca38..34e15991 100644 --- a/internal/db/queries/notifications.sql +++ b/internal/db/queries/notifications.sql @@ -1,4 +1,4 @@ --- M489 notifications inbox (#726). Rows are written only through +-- Notifications inbox. Rows are written only through -- internal/notifications.Notifier, which decides recipients and honours each -- recipient's inbox preference before it reaches these. @@ -13,30 +13,29 @@ RETURNING id; -- name: UpsertCoalescedNotification :one -- One unread row per (user, coalesce_key). A new event while that row is --- unread updates it in place and moves it back to the top of the inbox. +-- unread updates it in place: its `count` adds to the row's, and its `items` +-- (the names of what changed) go in front of the row's, keeping the newest +-- ten. Every coalesced kind's event is "N more happened", never a restated +-- total, so the row always says what is new since it was last read. -- --- sum_count: the payload's `count` adds to the unread row's count rather than --- replacing it. For kinds whose event is "N more happened" (tracks marked --- missing). Kinds whose event states the whole current total (pending --- duplicate groups) pass false and the payload simply replaces. --- --- emailed_at is cleared so the newer state goes out in the next batch; an --- emailed-but-unread row that keeps growing is news the user hasn't seen. --- Unless email_wanted is false, as in InsertNotification. +-- created_at and emailed_at are left alone: an update changes what the +-- notice says, not when it arrived. Moving created_at would make the phone +-- announce it again and clearing emailed_at would send it again, so a notice +-- that keeps growing would alert on every update. INSERT INTO user_notifications (user_id, kind, payload, coalesce_key, emailed_at) VALUES (sqlc.arg(user_id), sqlc.arg(kind), sqlc.arg(payload), sqlc.arg(coalesce_key), CASE WHEN sqlc.arg(email_wanted)::boolean THEN NULL ELSE now() END) ON CONFLICT (user_id, coalesce_key) WHERE read_at IS NULL AND coalesce_key IS NOT NULL DO UPDATE SET - payload = CASE - WHEN sqlc.arg(sum_count)::boolean THEN - EXCLUDED.payload || jsonb_build_object('count', - COALESCE((user_notifications.payload->>'count')::bigint, 0) - + COALESCE((EXCLUDED.payload->>'count')::bigint, 0)) - ELSE EXCLUDED.payload - END, - created_at = now(), - emailed_at = EXCLUDED.emailed_at + payload = EXCLUDED.payload || jsonb_build_object( + 'count', COALESCE((user_notifications.payload->>'count')::bigint, 0) + + COALESCE((EXCLUDED.payload->>'count')::bigint, 0), + 'items', (SELECT COALESCE(jsonb_agg(e.item ORDER BY e.n), '[]'::jsonb) + FROM jsonb_array_elements( + COALESCE(EXCLUDED.payload->'items', '[]'::jsonb) + || COALESCE(user_notifications.payload->'items', '[]'::jsonb) + ) WITH ORDINALITY AS e(item, n) + WHERE e.n <= 10)) RETURNING id; -- name: ListNotifications :many @@ -65,8 +64,9 @@ UPDATE user_notifications -- name: MarkAllNotificationsRead :execrows -- up_to, when set, limits it to what existed when the user asked: a "mark -- all read" queued offline and replayed later must not mark notices that --- arrived in between, which the user never saw. A coalesced row updated --- since then carries a newer created_at, so it stays unread too. +-- arrived in between, which the user never saw. A coalesced row that grew +-- since then is marked read with it: it arrived before they asked, and its +-- growth is a quiet update to a notice they had seen. UPDATE user_notifications SET read_at = now() WHERE user_id = sqlc.arg(user_id) @@ -104,7 +104,7 @@ ON CONFLICT (user_id, kind) DO UPDATE SET email = EXCLUDED.email, updated_at = now(); --- Email digest (#5346) ------------------------------------------------------ +-- Email digest ------------------------------------------------------ -- name: ListEmailPendingNotifications :many -- Every unread, un-emailed row of a user who has an address, oldest first. diff --git a/internal/db/queries/playback_errors.sql b/internal/db/queries/playback_errors.sql index 823a76ee..c89da41f 100644 --- a/internal/db/queries/playback_errors.sql +++ b/internal/db/queries/playback_errors.sql @@ -61,6 +61,3 @@ UPDATE playback_errors resolved_by = $2, resolution = $3 WHERE track_id = $1 AND resolved_at IS NULL; - --- name: CountUnresolvedPlaybackErrors :one -SELECT count(*)::bigint FROM playback_errors WHERE resolved_at IS NULL; diff --git a/internal/db/queries/playlists.sql b/internal/db/queries/playlists.sql index fa414aaa..92585184 100644 --- a/internal/db/queries/playlists.sql +++ b/internal/db/queries/playlists.sql @@ -55,7 +55,7 @@ RETURNING id, cover_path; -- (track was removed from the library). The denormalized snapshot fields -- on playlist_tracks remain authoritative for title/artist/album text. -- --- Deliberately NOT filtered on missing_since (#2527), unlike every browse / +-- Deliberately NOT filtered on missing_since, unlike every browse / -- discover / mix query. A playlist entry is something the user put here on -- purpose, so a missing file stays in the list and renders as a dead row -- rather than silently vanishing; missing_since rides along so the service @@ -76,7 +76,7 @@ ORDER BY pt.position; -- Inserts at the next available position. Snapshot fields are copied -- from the tracks/albums/artists join at insert time. tracks.duration_ms -- is converted to seconds for the snapshot. pick_kind is NULL for manual --- playlists; the For You builder stamps 'taste'/'fresh' (#1249). +-- playlists; the For You builder stamps 'taste'/'fresh'. INSERT INTO playlist_tracks (playlist_id, position, track_id, title, artist_name, album_title, duration_sec, pick_kind) SELECT sqlc.arg(playlist_id)::uuid, @@ -120,5 +120,5 @@ LIMIT $2; -- name: GetPlaylistsByIDs :many -- Batched lookup used by /api/library/sync to hydrate upsert payloads --- (#357). Mirror of GetArtistsByIDs. +--. Mirror of GetArtistsByIDs. SELECT * FROM playlists WHERE id = ANY($1::uuid[]); diff --git a/internal/db/queries/lidarr_quarantine.sql b/internal/db/queries/quarantine.sql similarity index 84% rename from internal/db/queries/lidarr_quarantine.sql rename to internal/db/queries/quarantine.sql index 8c43fc9c..85cb8c92 100644 --- a/internal/db/queries/lidarr_quarantine.sql +++ b/internal/db/queries/quarantine.sql @@ -1,7 +1,7 @@ -- name: UpsertQuarantine :one -- Insert a new quarantine row, or update reason/notes if the user has -- already flagged this track. -INSERT INTO lidarr_quarantine (user_id, track_id, reason, notes) +INSERT INTO quarantine (user_id, track_id, reason, notes) VALUES ($1, $2, $3, $4) ON CONFLICT (user_id, track_id) DO UPDATE SET reason = EXCLUDED.reason, @@ -12,7 +12,7 @@ RETURNING user_id, track_id, reason, notes, created_at; -- name: DeleteQuarantine :one -- Removes the caller's row. Returns the deleted row so the handler can -- distinguish "no row existed" (zero rows -> ErrNoRows) from success. -DELETE FROM lidarr_quarantine +DELETE FROM quarantine WHERE user_id = $1 AND track_id = $2 RETURNING user_id, track_id, reason, notes, created_at; @@ -31,7 +31,7 @@ SELECT al.cover_art_path AS album_cover_art_path, ar.id AS artist_id, ar.name AS artist_name -FROM lidarr_quarantine q +FROM quarantine q JOIN tracks t ON t.id = q.track_id JOIN albums al ON al.id = t.album_id JOIN artists ar ON ar.id = t.artist_id @@ -48,14 +48,13 @@ SELECT ar.name AS artist_name, al.title AS album_title, al.id AS album_id, - al.mbid AS lidarr_album_mbid, count(q.user_id)::int AS report_count, max(q.created_at)::timestamptz AS latest_at -FROM lidarr_quarantine q +FROM quarantine q JOIN tracks t ON t.id = q.track_id JOIN albums al ON al.id = t.album_id JOIN artists ar ON ar.id = t.artist_id -GROUP BY t.id, ar.name, al.title, al.id, al.mbid +GROUP BY t.id, ar.name, al.title, al.id ORDER BY max(q.created_at) DESC; -- name: ListQuarantineReportsForTrack :many @@ -67,7 +66,7 @@ SELECT q.reason, q.notes, q.created_at -FROM lidarr_quarantine q +FROM quarantine q JOIN users u ON u.id = q.user_id WHERE q.track_id = $1 ORDER BY q.created_at DESC; @@ -76,21 +75,21 @@ ORDER BY q.created_at DESC; -- Clears all per-user rows for a given track. Used by Resolve and the -- two delete actions. Caller writes the audit row separately before -- this fires (so we can capture the affected_users count). -DELETE FROM lidarr_quarantine WHERE track_id = $1; +DELETE FROM quarantine WHERE track_id = $1; -- name: CountQuarantineForTrack :one -- Reads affected_users for the audit row before the delete fires. -SELECT count(*)::int FROM lidarr_quarantine WHERE track_id = $1; +SELECT count(*)::int FROM quarantine WHERE track_id = $1; -- name: WriteQuarantineAction :one -- Audit row for an admin destructive action. -INSERT INTO lidarr_quarantine_actions ( +INSERT INTO quarantine_actions ( track_id, track_title, artist_name, album_title, - action, admin_id, lidarr_album_mbid, affected_users -) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) + action, admin_id, affected_users +) VALUES ($1, $2, $3, $4, $5, $6, $7) RETURNING *; -- name: ListQuarantineActions :many -SELECT * FROM lidarr_quarantine_actions +SELECT * FROM quarantine_actions ORDER BY created_at DESC LIMIT $1; diff --git a/internal/db/queries/reacquisition.sql b/internal/db/queries/reacquisition.sql index 85f3de86..be665a19 100644 --- a/internal/db/queries/reacquisition.sql +++ b/internal/db/queries/reacquisition.sql @@ -1,6 +1,6 @@ --- Auto re-acquisition of missing files (milestone #290). The unit is the --- album: Lidarr acquires releases, and grouping collapses "40 missing files" --- into "3 albums to ask for". +-- Re-acquisition of missing files. +-- The unit is the album: an album whose files went missing becomes a want for +-- its release group, and the search schedule looks for it. -- name: GetReacquisitionSettings :one SELECT * FROM reacquisition_settings WHERE id = true; @@ -9,13 +9,9 @@ SELECT * FROM reacquisition_settings WHERE id = true; -- Whole-row write from the admin card; the CHECKs in migration 0056 are the -- validation, so a bad value fails loudly rather than being clamped silently. UPDATE reacquisition_settings - SET enabled = sqlc.arg(enabled), - grace_hours = sqlc.arg(grace_hours), - backoff_base_hours = sqlc.arg(backoff_base_hours), - backoff_max_hours = sqlc.arg(backoff_max_hours), - max_attempts = sqlc.arg(max_attempts), - max_per_pass = sqlc.arg(max_per_pass), - auto_approve = sqlc.arg(auto_approve) + SET enabled = sqlc.arg(enabled), + grace_hours = sqlc.arg(grace_hours), + max_per_pass = sqlc.arg(max_per_pass) WHERE id = true RETURNING *; @@ -24,97 +20,74 @@ RETURNING *; -- -- * it still has at least one track whose file has been missing longer than -- the grace window. Measured on missing_since, which reconcile never --- re-stamps (#2523), so it is a genuine "gone since" clock rather than --- "when we last noticed"; --- * MusicBrainz can name it. Both the album and its artist MBID are --- required — Create rejects an album-kind request without them, and there --- is nothing to ask Lidarr for anyway. Albums failing this are counted --- separately (CountAlbumsMissingWithoutMbid) rather than vanishing; --- * it has not spent its attempt budget (gave_up_at IS NULL); --- * its backoff has elapsed: base * 2^(attempts-1) hours since the last --- attempt, clamped to the configured maximum. First attempt (no row, or --- last_attempt_at NULL) is always due. --- --- Oldest attempt first, never-attempted first, so a large loss drains in a --- stable order across passes instead of re-picking the same head each time. -SELECT albums.id AS album_id, - albums.title AS album_title, - albums.mbid AS album_mbid, +-- re-stamps, so it is a genuine "gone since" clock; +-- * MusicBrainz can name it: the artist has an MBID, and the album a +-- release MBID or a release group. The release MBID is only ever used to +-- find the group, and an album matched to a group may carry no release +-- id. Albums failing this are counted separately +-- (CountAlbumsMissingWithoutMbid); +-- * it is on the wanted list: an album taken off the list, or +-- dropped into the library by hand, is not fetched back; +-- * no want already covers it: none open, and none held since its files +-- went missing. A want held after the loss means the library has a full +-- copy despite the missing rows, so asking again would loop. +SELECT albums.id AS album_id, + albums.title AS album_title, + albums.mbid AS album_mbid, albums.release_group_mbid AS album_release_group_mbid, - artists.id AS artist_id, - artists.name AS artist_name, - artists.mbid AS artist_mbid, - COUNT(tracks.id)::bigint AS missing_track_count, - COALESCE(r.attempts, 0)::int AS attempts + artists.name AS artist_name, + artists.mbid AS artist_mbid, + COUNT(tracks.id)::bigint AS missing_track_count, + MAX(tracks.missing_since)::timestamptz AS last_missing_since FROM albums JOIN artists ON artists.id = albums.artist_id JOIN tracks ON tracks.album_id = albums.id - LEFT JOIN missing_reacquisitions r ON r.album_id = albums.id WHERE tracks.missing_since IS NOT NULL AND tracks.missing_since <= now() - make_interval(hours => sqlc.arg(grace_hours)::int) - AND albums.mbid IS NOT NULL + AND (albums.mbid IS NOT NULL OR albums.release_group_mbid IS NOT NULL) AND artists.mbid IS NOT NULL - AND (r.gave_up_at IS NULL) - AND ( - r.last_attempt_at IS NULL - OR r.last_attempt_at <= now() - make_interval(hours => LEAST( - (sqlc.arg(backoff_base_hours)::int - * POWER(2, GREATEST(COALESCE(r.attempts, 0) - 1, 0)))::int, - sqlc.arg(backoff_max_hours)::int)) - ) GROUP BY albums.id, albums.title, albums.mbid, albums.release_group_mbid, - artists.id, artists.name, artists.mbid, r.attempts, r.last_attempt_at - ORDER BY r.last_attempt_at NULLS FIRST, albums.sort_title + artists.name, artists.mbid +HAVING EXISTS ( + SELECT 1 FROM wants w + WHERE w.album_id = albums.id OR w.release_group_mbid = albums.release_group_mbid) + AND NOT EXISTS ( + SELECT 1 FROM wants w + WHERE (w.album_id = albums.id OR w.release_group_mbid = albums.release_group_mbid) + AND (w.fulfilled_at IS NULL OR w.fulfilled_at > MAX(tracks.missing_since))) + ORDER BY MIN(tracks.missing_since), albums.sort_title LIMIT sqlc.arg(page_limit); -- name: CountAlbumsMissingWithoutMbid :one --- Albums with missing files that can never be auto-requested because nothing --- identifies them to MusicBrainz. Surfaced on the admin card so the gap is --- visible: silently doing nothing for these would read as the feature being --- broken. +-- Albums with missing files that can never be wanted automatically because +-- nothing identifies them to MusicBrainz. Surfaced on the admin card so the +-- gap is visible: silently doing nothing for these would read as broken. SELECT COUNT(DISTINCT albums.id)::bigint FROM albums JOIN artists ON artists.id = albums.artist_id JOIN tracks ON tracks.album_id = albums.id WHERE tracks.missing_since IS NOT NULL - AND (albums.mbid IS NULL OR artists.mbid IS NULL); + AND ((albums.mbid IS NULL AND albums.release_group_mbid IS NULL) OR artists.mbid IS NULL); --- name: RecordReacquisitionAttempt :one --- Bumps the attempt counter and stamps the clock the backoff measures from. --- Upsert because the first attempt has no row yet. -INSERT INTO missing_reacquisitions (album_id, attempts, last_attempt_at, last_request_id) -VALUES (sqlc.arg(album_id), 1, now(), sqlc.narg(last_request_id)) -ON CONFLICT (album_id) DO UPDATE - SET attempts = missing_reacquisitions.attempts + 1, - last_attempt_at = now(), - last_request_id = COALESCE(EXCLUDED.last_request_id, - missing_reacquisitions.last_request_id), - updated_at = now() +-- name: WantMissingAlbum :one +-- Records an album with missing files as wanted. A held want for the same +-- group is reopened; an open one is left as it is, and no row comes back. +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, album_id) +VALUES (sqlc.arg(release_group_mbid), sqlc.arg(artist_mbid), sqlc.arg(artist_name), + sqlc.arg(title), 'missing', sqlc.arg(album_id)) +ON CONFLICT (release_group_mbid) DO UPDATE + SET reason = 'missing', + album_id = EXCLUDED.album_id, + wanted_at = now(), + fulfilled_at = NULL, + updated_at = now() + WHERE wants.fulfilled_at IS NOT NULL RETURNING *; --- name: MarkReacquisitionGaveUp :exec --- Stamped when the attempt budget is spent. Stored as a timestamp rather than --- inferred from `attempts >= max_attempts` so the verdict survives an operator --- later raising the maximum, and so the admin surface can say when. -UPDATE missing_reacquisitions - SET gave_up_at = now(), - updated_at = now() - WHERE album_id = sqlc.arg(album_id) - AND gave_up_at IS NULL; - --- name: ClearRecoveredReacquisitions :execrows --- Drops state for albums that no longer have any missing track — the files --- came back, or the scanner adopted them at a new path (#2528). Deleting --- rather than resetting counters means a future loss starts from a clean --- budget, which is right: it is a new problem, not a continuation. -DELETE FROM missing_reacquisitions r - WHERE NOT EXISTS ( - SELECT 1 FROM tracks - WHERE tracks.album_id = r.album_id - AND tracks.missing_since IS NOT NULL - ); - --- name: GetReacquisitionForAlbums :many --- State for the admin missing-files surface, so each directory group can say --- whether a re-acquisition is in flight, waiting, or given up. -SELECT * FROM missing_reacquisitions WHERE album_id = ANY(sqlc.arg(album_ids)::uuid[]); +-- name: GetWantsForAlbums :many +-- What the wanted list says about each album on the missing-files page. +SELECT a.id AS album_id, w.id AS want_id, w.wanted_at, w.fulfilled_at + FROM albums a + JOIN wants w ON w.album_id = a.id OR w.release_group_mbid = a.release_group_mbid + WHERE a.id = ANY(sqlc.arg(album_ids)::uuid[]) + ORDER BY w.wanted_at DESC; diff --git a/internal/db/queries/recommendation.sql b/internal/db/queries/recommendation.sql index cb164e5f..d4c3c805 100644 --- a/internal/db/queries/recommendation.sql +++ b/internal/db/queries/recommendation.sql @@ -24,37 +24,34 @@ LEFT JOIN LATERAL ( WHERE user_id = $1 AND track_id = t.id ) pe ON true WHERE t.id <> $2 - AND t.missing_since IS NULL -- #2523: never offer a file that is gone + AND t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( SELECT 1 FROM play_events WHERE user_id = $1 AND track_id = t.id AND started_at > now() - $3 * interval '1 hour' ) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ); -- name: LoadRadioCandidatesV2 :many --- M4c: similarity-driven candidate pool. 6-way UNION: +-- Similarity-driven candidate pool. 6-way UNION: -- $1 user_id, $2 seed_track_id, $3 recently_played_hours, -- $4 exclude (uuid[]), $5 lb_similar K, $6 similar_artists K, -- $7 tag_overlap K, $8 likes_overlap K, $9 random_fill K, --- $10 taste_overlap K (#796 phase 2b — tracks by the user's top +-- $10 taste_overlap K (tracks by the user's top -- positively-weighted taste-profile artists, so taste-relevant tracks -- enter the pool even when the similarity/random arms miss them; scored -- in Go via TasteMatch, so sim_score here is 0 pool-inclusion), --- $11 coplay_artists K (#1533 — tracks by artists co-played across the +-- $11 coplay_artists K (tracks by artists co-played across the -- instance with the seed's artist; source='user_cooccurrence'), -- $12 order_seed (text) — see below. -- --- $12 REPLACES `ORDER BY random()` IN FOUR ARMS (#3889). Those arms returned --- a stable set only while their LIMIT exceeded the rows eligible for them: at --- that point they returned all of them and the order stopped mattering, --- because the caller sorts by track id before scoring. Below that threshold --- they returned a random SUBSET, and two builds on the same day drew --- different ones — so "daily determinism" held by accident, and only for --- libraries smaller than the limits. +-- $12 orders four arms by md5(id || seed), not random(). An arm whose LIMIT is +-- below the rows eligible for it returns a SUBSET, and with random() two +-- builds on the same day would draw different ones; the caller sorts by track +-- id before scoring, so only the subset matters. -- -- md5(id || seed) keeps the intent — an arbitrary spread that changes when -- the seed does — while making it reproducible for a given seed. The CALLER @@ -62,6 +59,18 @@ WHERE t.id <> $2 -- the determinism they promise; radio passes a fresh value per request and -- keeps varying, which is what a radio should do. -- Returns same shape as LoadRadioCandidates plus similarity_score column. +-- +-- sim_score BANDS keep the eligibility tiers in order. Similarity edges +-- are stored relative to the seed's best match, 0 to 1, so the arms are placed +-- in bands rather than compared raw: +-- tier 1 lb_similar 0.5 + 0.5 × edge (0.5, 1.0] +-- tier 2 similar_artists, coplay_artists 0.5 × edge [0, 0.5] +-- tag_overlap 0.5 × share of the seed's tags +-- likes_overlap 0.3, flat +-- tier 3 taste_overlap, random_fill 0 +-- Without the bands, the seed's weaker LB edges would fall below a shared +-- genre tag and the flat likes score, and the one arm that measures the seed +-- would lose to the ones that guess. WITH seed_artist AS ( @@ -83,11 +92,11 @@ excluded_ids AS ( WHERE pe.user_id = $1 AND pe.started_at > now() - $3 * interval '1 hour' UNION ALL SELECT q.track_id AS id - FROM lidarr_quarantine q + FROM quarantine q WHERE q.user_id = $1 ), lb_similar AS ( - SELECT ts.track_b_id AS track_id, ts.score AS sim_score + SELECT ts.track_b_id AS track_id, 0.5 + 0.5 * ts.score AS sim_score FROM track_similarity ts WHERE ts.track_a_id = $2 AND ts.source = 'listenbrainz' @@ -96,11 +105,10 @@ lb_similar AS ( LIMIT $5 ), similar_artists AS ( - -- Round-robin across the similar artists (#5297): each artist's first + -- Round-robin across the similar artists: each artist's first -- track, best-scoring artist first, then each one's second, and so on. - -- Ordering by artist score alone let the closest similar artist's - -- catalogue fill the whole LIMIT; on the operator's library this arm held - -- exactly one artist for all 17 seeds measured (#3879). + -- Ordering by artist score alone would let the closest similar artist's + -- catalogue fill the whole LIMIT. SELECT track_id, sim_score FROM ( SELECT t.id AS track_id, asim.score * 0.5 AS sim_score, asim.score AS artist_score, @@ -117,8 +125,8 @@ similar_artists AS ( ), tag_overlap AS ( SELECT t.id AS track_id, - (count(DISTINCT trim(g))::float8 - / GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score + 0.5 * (count(DISTINCT trim(g))::float8 + / GREATEST((SELECT count(*) FROM seed_tags), 1)) AS sim_score FROM tracks t LEFT JOIN LATERAL regexp_split_to_table(coalesce(t.genre, ''), '[;,]') AS g_split(g) ON true WHERE trim(g_split.g) IN (SELECT tag FROM seed_tags) @@ -126,11 +134,15 @@ tag_overlap AS ( AND t.id <> $2 GROUP BY t.id HAVING count(DISTINCT trim(g_split.g)) > 0 - ORDER BY sim_score DESC + -- md5 breaks the ties. A seed with one tag gives every track of that + -- genre the same score, and with no tie-break the LIMIT would take + -- whichever rows Postgres reached first: the same slice for every seed of + -- the genre, and not one the order seed controls. + ORDER BY sim_score DESC, md5(t.id::text || $12::text) LIMIT $7 ), likes_overlap AS ( - SELECT gl.track_id, 0.6::float8 AS sim_score + SELECT gl.track_id, 0.3::float8 AS sim_score FROM general_likes gl WHERE gl.user_id = $1 AND gl.track_id NOT IN (SELECT id FROM excluded_ids) @@ -144,18 +156,27 @@ likes_overlap AS ( LIMIT $8 ), taste_overlap AS ( - SELECT t.id AS track_id, 0.0::float8 AS sim_score - FROM taste_profile_artists tpa - JOIN tracks t ON t.artist_id = tpa.artist_id - WHERE tpa.user_id = $1 - AND tpa.weight > 0 - AND t.id NOT IN (SELECT id FROM excluded_ids) - AND t.id <> $2 - ORDER BY tpa.weight DESC, t.id + -- Round-robin across the taste artists, as similar_artists is: each + -- artist's first track, strongest artist first, then each one's second. + -- Ordered by weight then track id, the strongest artist's catalogue would + -- fill the whole arm with the same lowest-id tracks every time. + SELECT track_id, sim_score + FROM ( + SELECT t.id AS track_id, 0.0::float8 AS sim_score, tpa.weight, + row_number() OVER (PARTITION BY tpa.artist_id + ORDER BY md5(t.id::text || $12::text)) AS artist_turn + FROM taste_profile_artists tpa + JOIN tracks t ON t.artist_id = tpa.artist_id AND t.missing_since IS NULL + WHERE tpa.user_id = $1 + AND tpa.weight > 0 + AND t.id NOT IN (SELECT id FROM excluded_ids) + AND t.id <> $2 + ) ranked + ORDER BY artist_turn, weight DESC, md5(track_id::text || $12::text) LIMIT $10 ), coplay_artists AS ( - -- Household co-play (#1533): tracks by artists co-played across the + -- Household co-play: tracks by artists co-played across the -- instance with the seed's artist (source='user_cooccurrence', built by -- the coplay worker). Mirrors similar_artists but from local co-occurrence -- instead of ListenBrainz; empty on single-user servers. Same 0.5 damp as @@ -209,7 +230,7 @@ FROM ( UNION ALL SELECT track_id, sim_score FROM coplay_artists UNION ALL SELECT track_id, sim_score FROM random_fill ) u -JOIN tracks t ON t.id = u.track_id AND t.missing_since IS NULL -- #2523: never offer a file that is gone +JOIN tracks t ON t.id = u.track_id AND t.missing_since IS NULL -- never offer a file that is gone JOIN albums al ON al.id = t.album_id LEFT JOIN general_likes l ON l.user_id = $1 AND l.track_id = t.id LEFT JOIN LATERAL ( @@ -227,10 +248,10 @@ GROUP BY t.id, t.title, t.album_id, t.artist_id, t.duration_ms, t.file_path, -- name: ListArtistContextPlayCountsForUser :many -- Per-artist completed-play counts split by whether each play falls in the --- CURRENT context cell, in the user's local timezone (#1531). The cell is --- daypart × weekday-type, optionally narrowed by device class (#1551): when --- $2 (the current device) is '', the device dimension is ignored (identical to --- the time-only #1531 behaviour, used by the daily mixes which have no device); +-- CURRENT context cell, in the user's local timezone. The cell is +-- daypart × weekday-type, optionally narrowed by device class: when +-- $2 (the current device) is '', the device dimension is ignored (time only, +-- for the daily mixes, which have no device); -- when $2 is set (radio), a play only counts in the cell if its device_class -- matches. Feeds the context-affinity scoring term. Skips excluded; a 365-day -- window bounds cost. Daypart buckets: night [22,5) morning [5,12) @@ -280,7 +301,7 @@ FROM plays p GROUP BY p.artist_id; -- name: GetLatestPlayDeviceClassForUser :one --- The device_class of the user's most recent play (#1551), used as the +-- The device_class of the user's most recent play, used as the -- "current device" for radio context conditioning. NULL when the latest play -- predates device capture or came from a client that didn't send one. SELECT device_class @@ -293,24 +314,24 @@ LIMIT 1; -- Per-user artist suggestions ranked by taste signal x similarity, projected -- through artist_similarity_unmatched (out-of-library candidates only). -- --- Seeds are TIERED (rule #131) so the surface never empties: +-- Seeds are TIERED so the surface never empties: -- tier 1 - taste_profile_artists.weight: engagement-graded, time-decayed and -- SIGNED by internal/taste, so an artist the user has drifted away -- from stops contributing instead of accumulating forever. -- tier 2 - likes + completed plays, used ONLY when the profile has no rows -- (new account, or before the first daily recompute). -- --- The signal is log-damped: contribution is signal x similarity, and the old --- undamped sum let one heavily-played artist's neighbours take every slot -- --- entrenching harder the MORE the user listened (issue #2367 mechanism 2). +-- The signal is log-damped: contribution is signal x similarity, and an +-- undamped sum would let one heavily-played artist's neighbours take every +-- slot -- entrenching harder the MORE the user listened. -- -- Candidates already in the library, already requested and not terminal, or -- snoozed by this user, are excluded. $1=user_id, $2=half_life_days (tier 2 -- decay), $3=limit. WITH artist_plays AS ( - -- Completed plays only. The previous seed query counted every play_event, - -- so skipping an artist repeatedly INCREASED its signal and pushed more of - -- its neighbours at the user (issue #2367 mechanism 3). + -- Completed plays only: counting every play_event would make skipping an + -- artist repeatedly INCREASE its signal and push more of its neighbours + -- at the user. SELECT t.artist_id, count(*)::bigint AS play_count FROM play_events pe JOIN tracks t ON t.id = pe.track_id @@ -362,12 +383,12 @@ contributions AS ( JOIN seeds ON seeds.artist_id = u.seed_artist_id WHERE NOT EXISTS (SELECT 1 FROM artists WHERE mbid = u.candidate_mbid) AND NOT EXISTS ( - SELECT 1 FROM lidarr_requests r + SELECT 1 FROM requests r WHERE r.user_id = $1 - AND r.lidarr_artist_mbid = u.candidate_mbid + AND r.artist_mbid = u.candidate_mbid AND r.status NOT IN ('rejected', 'failed') ) - -- Snoozed by this user and not yet expired (#2374). Time-boxed and + -- Snoozed by this user and not yet expired. Time-boxed and -- per-user: the candidate returns on its own once snoozed_until -- passes, and stays visible to everyone else meanwhile. Deliberately -- filtered at the candidate stage, NOT folded into the score — a @@ -393,7 +414,7 @@ ORDER BY total_score DESC LIMIT $3; -- name: ListMostPlayedTracksForUser :many --- M6a: top-N tracks by completed-play count for the user. was_skipped +-- Top-N tracks by completed-play count for the user. was_skipped -- excludes skips so a user spamming next doesn't fabricate a top track. -- Quarantined tracks (per-user soft-hide from M5b) are filtered out. -- @@ -414,9 +435,9 @@ FROM plays p JOIN tracks t ON t.id = p.track_id JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id -WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone +WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY p.cnt DESC, t.id @@ -441,16 +462,16 @@ JOIN tracks t ON t.id = p.track_id JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id WHERE t.artist_id = sqlc.arg(artist_id) - AND t.missing_since IS NULL -- #2523: never offer a file that is gone + AND t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = sqlc.arg(user_id) AND q.track_id = t.id ) ORDER BY p.cnt DESC, t.id LIMIT sqlc.arg(result_limit); -- name: ListLastPlayedArtistsForUser :many --- M6a: artists ranked by max(play_events.started_at) for the user, with +-- Artists ranked by max(play_events.started_at) for the user, with -- a derived cover_album_id via a representative-album lateral join (most -- recent album that has cover_art_path set). album_count joined for the -- ArtistRef wire shape. diff --git a/internal/db/queries/recommendation_metrics.sql b/internal/db/queries/recommendation_metrics.sql index 75c70a5f..f7bf5c59 100644 --- a/internal/db/queries/recommendation_metrics.sql +++ b/internal/db/queries/recommendation_metrics.sql @@ -1,16 +1,16 @@ --- Recommendation observability (#796 phase 4). Per-source play outcomes so the +-- Recommendation observability. Per-source play outcomes so the -- operator can see whether each recommendation surface is landing and tune the -- taste weights. Source is stamped on play_events when a play is launched from -- a recommendation surface; NULL means the user picked the track manually — -- those rows are INCLUDED here as the baseline control group the surfaces are --- judged against (milestone #127: delta-vs-baseline is what makes the numbers --- actionable). Raw source strings are bucketed into stable surface families in --- the Go handler; completion_n is carried so family merges can weight +-- judged against (delta-vs-baseline is what makes the numbers actionable). +-- Raw source strings are bucketed into stable surface families in the Go +-- handler; completion_n is carried so family merges can weight -- avg_completion correctly. -- name: RecommendationWeeklyTrends :many -- Weekly per-source outcome series for the tuning lab's trend view --- (#1251). Aggregated across ALL users: the tuning knobs are global, +--. Aggregated across ALL users: the tuning knobs are global, -- so judging a knob turn needs global outcomes — rows carry rates -- only, no track or user identity. NULL-source (manual) rows are -- included as the baseline family. @@ -42,12 +42,12 @@ ORDER BY 1, 2; -- name: RecommendationSourceMetricsForUser :many -- $1 user_id, $2 window_days. plays/skips are counts; avg_completion is the -- mean completion ratio over the completion_n plays that recorded one. --- pick_kind splits For You plays into taste/fresh/unattributed (#1249); +-- pick_kind splits For You plays into taste/fresh/unattributed; -- it is NULL for every other source, so those still group to one row. -- -- completion_sqsum carries the sum of SQUARED completion ratios so the Go -- handler can compute a variance — needed for the margin of error on a --- completion delta (#2495). It is the sum rather than `stddev_samp` on purpose: +-- completion delta. It is the sum rather than `stddev_samp` on purpose: -- raw source rows get merged into surface families in Go, and sums of squares -- add across groups exactly, whereas standard deviations cannot be combined -- without them. Variance = (sqsum - sum²/n) / (n-1), with sum = avg × n. diff --git a/internal/db/queries/recommendation_tuning.sql b/internal/db/queries/recommendation_tuning.sql index 92ae3bce..49c627ed 100644 --- a/internal/db/queries/recommendation_tuning.sql +++ b/internal/db/queries/recommendation_tuning.sql @@ -1,9 +1,9 @@ --- Recommendation tuning lab queries (#1250). Seeding happens via the +-- Recommendation tuning lab queries. Seeding happens via the -- recsettings boot reconcile; shipped defaults live in Go only. -- name: UpsertWeightProfileDefaults :exec -- Boot reconcile: insert the shipped defaults for a profile if the row --- doesn't exist yet. Never overwrites operator-tuned values. +-- doesn't exist yet. Never overwrites admin-tuned values. INSERT INTO recommendation_weight_profiles ( profile, base_weight, like_boost, recency_weight, skip_penalty, jitter_magnitude, context_weight, similarity_weight, taste_weight, @@ -54,8 +54,8 @@ UPDATE taste_tuning RETURNING *; -- name: UpsertDiscoverTuningDefaults :exec --- Boot reconcile for the Discover scope (#2377). Never overwrites --- operator-tuned values, same contract as the other two. +-- Boot reconcile for the Discover scope. Never overwrites +-- admin-tuned values, same contract as the other two. INSERT INTO discover_tuning (singleton, tag_overlap_weight, snooze_days) VALUES (true, $1, $2) ON CONFLICT (singleton) DO NOTHING; @@ -77,7 +77,7 @@ INSERT INTO recommendation_tuning_audit (scope, action, changes) VALUES ($1, $2, $3); -- name: ListTuningAudit :many --- Newest first; consumed by the metrics trend view (#1251) to annotate +-- Newest first; consumed by the metrics trend view to annotate -- knob turns on the timeline. SELECT id, changed_at, scope, action, changes FROM recommendation_tuning_audit diff --git a/internal/db/queries/requests.sql b/internal/db/queries/requests.sql new file mode 100644 index 00000000..7eff0d01 --- /dev/null +++ b/internal/db/queries/requests.sql @@ -0,0 +1,119 @@ +-- name: CreateRequest :one +INSERT INTO requests ( + user_id, kind, + artist_mbid, album_mbid, track_mbid, + artist_name, album_title, track_title +) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) +RETURNING *; + +-- name: GetRequestByID :one +SELECT * FROM requests WHERE id = $1; + +-- name: ListRequestsForUser :many +SELECT * FROM requests +WHERE user_id = $1 +ORDER BY requested_at DESC +LIMIT $2; + +-- name: ListRequestsByStatus :many +SELECT * FROM requests +WHERE status = $1 +ORDER BY requested_at DESC +LIMIT $2; + +-- name: ListApprovedRequestsForReconcile :many +SELECT * FROM requests +WHERE status = 'approved' +ORDER BY decided_at ASC +LIMIT $1; + +-- name: ApproveRequest :one +UPDATE requests + SET status = 'approved', + decided_at = now(), + decided_by = $2, + updated_at = now() + WHERE id = $1 AND status = 'pending' + RETURNING *; + +-- name: RejectRequest :one +UPDATE requests + SET status = 'rejected', + notes = $2, + decided_at = now(), + decided_by = $3, + updated_at = now() + WHERE id = $1 AND status = 'pending' + RETURNING *; + +-- name: CancelRequest :one +-- $2 serves as both the ownership guard (user_id = $2) and the +-- decided_by audit field. The generated parameter struct names it +-- DecidedBy — callers must pass the cancelling user's UUID, and +-- a wrong UUID will silently no-op (zero rows updated) rather than +-- erroring at the SQL layer. +UPDATE requests + SET status = 'rejected', + notes = 'cancelled by user', + decided_at = now(), + decided_by = $2, + updated_at = now() + WHERE id = $1 AND user_id = $2 AND status = 'pending' + RETURNING *; + +-- name: CompleteRequest :one +-- Reconciler transitions an approved request to completed when its +-- target track/album/artist has appeared in the library. Callers +-- should set ONLY the matched_*_id corresponding to the request's +-- kind and pass pgtype.UUID{} (Valid: false) for the others — +-- per-kind one-of-three is the contract, NULL for the others +-- preserves auditability if the matched row is later evicted. +UPDATE requests + SET status = 'completed', + matched_track_id = $2, + matched_album_id = $3, + matched_artist_id = $4, + completed_at = now(), + updated_at = now() + WHERE id = $1 AND status = 'approved' + RETURNING *; + +-- name: HasNonTerminalRequestForMBID :one +-- Whether something is already on its way: a pending, approved or completed +-- request at the given level, an open want for the release group, or a +-- follow of the artist. Sets the `requested` flag on catalogue search. +-- Terminal-status (rejected, failed) requests do not count. MBIDs are unique +-- per entity type, so checking one id against every column cannot +-- false-match. +SELECT (EXISTS ( + SELECT 1 FROM requests r + WHERE r.status IN ('pending', 'approved', 'completed') + AND ((r.kind = 'artist' AND r.artist_mbid = @mbid) + OR (r.kind = 'album' AND r.album_mbid = @mbid) + OR (r.kind = 'track' AND r.track_mbid = @mbid)) +) OR EXISTS ( + SELECT 1 FROM wants w WHERE w.release_group_mbid = @mbid AND w.fulfilled_at IS NULL +) OR EXISTS ( + SELECT 1 FROM follows f WHERE f.artist_mbid = @mbid +))::bool AS requested; + +-- name: GetNonTerminalRequestForMBID :one +-- Returns the oldest non-terminal (pending/approved/completed) request +-- whose own-kind MBID matches @mbid, or pgx.ErrNoRows when none. Create() +-- uses this to dedupe: a re-request for something already in flight +-- returns the existing row instead of inserting a duplicate. +SELECT * FROM requests + WHERE status IN ('pending', 'approved', 'completed') + AND ((kind = 'artist' AND artist_mbid = @mbid) + OR (kind = 'album' AND album_mbid = @mbid) + OR (kind = 'track' AND track_mbid = @mbid)) + ORDER BY requested_at ASC + LIMIT 1; + +-- name: RepointRequestsAlbumMbid :exec +-- A want moved from a release id to its group takes the requests +-- that name it along, so the reconciler finds it. +UPDATE requests + SET album_mbid = sqlc.arg(new_mbid), + updated_at = now() + WHERE album_mbid = sqlc.arg(old_mbid); diff --git a/internal/db/queries/scan_runs.sql b/internal/db/queries/scan_runs.sql index c2a005df..03d1856d 100644 --- a/internal/db/queries/scan_runs.sql +++ b/internal/db/queries/scan_runs.sql @@ -1,4 +1,4 @@ --- M7 #381: scan-run lifecycle queries. +-- Scan-run lifecycle queries. -- name: InsertScanRun :one -- Creates a new scan_runs row with started_at=now(). Stages are filled in @@ -33,7 +33,7 @@ SELECT id, started_at, finished_at, library, mbid_backfill, cover_enrich, artist LIMIT 1; -- name: UpdateScanRunArtistArtEnrich :exec --- M7 cover-sources: persists the artist-art enrich stage tally JSON. +-- Persists the artist-art enrich stage tally JSON. UPDATE scan_runs SET artist_art_enrich = $2 WHERE id = $1; diff --git a/internal/db/queries/search.sql b/internal/db/queries/search.sql new file mode 100644 index 00000000..01c17fa3 --- /dev/null +++ b/internal/db/queries/search.sql @@ -0,0 +1,141 @@ +-- Searching for wants and grabbing releases. + +-- name: GetAcquisitionSettings :one +SELECT * FROM acquisition_settings WHERE id = true; + +-- name: UpdateAcquisitionSettings :one +UPDATE acquisition_settings + SET auto_grab = sqlc.arg(auto_grab), + search_interval_hours = sqlc.arg(search_interval_hours), + allowed_qualities = sqlc.arg(allowed_qualities), + import_root = sqlc.arg(import_root), + seed_ratio = sqlc.arg(seed_ratio), + seed_days = sqlc.arg(seed_days), + stall_hours = sqlc.arg(stall_hours), + upgrade_until = sqlc.arg(upgrade_until), + updated_at = now() + WHERE id = true +RETURNING *; + +-- name: RecordSearchPass :exec +UPDATE acquisition_settings SET last_pass_at = now(), last_pass = sqlc.arg(report) WHERE id = true; + +-- name: ListWantsDueForSearch :many +-- Wants with no download running that were never searched or not since +-- @hours ago: open wants first (never-searched, then requests, then missing +-- files, then followed artists' releases), then held albums wanted for an +-- upgrade, which are searched at most weekly. +SELECT w.id, w.release_group_mbid, w.artist_name, w.title, w.reason, w.upgrade, w.held_tier, w.held_quality, + COALESCE(NULLIF(w.primary_type, ''), g.primary_type, '')::text AS primary_type, + COALESCE(g.secondary_types, '{}')::text[] AS secondary_types, + COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, '')::text AS first_release_date + FROM wants w + LEFT JOIN mb_release_groups g ON g.mbid = w.release_group_mbid + WHERE (w.fulfilled_at IS NULL OR w.upgrade <> '') + AND NOT EXISTS (SELECT 1 FROM grabs gr WHERE gr.want_id = w.id AND gr.ended_at IS NULL) + AND (w.searched_at IS NULL + OR w.searched_at < now() - make_interval(hours => CASE WHEN w.fulfilled_at IS NULL + THEN sqlc.arg(hours)::int + ELSE GREATEST(sqlc.arg(hours)::int, 168) END)) + ORDER BY w.fulfilled_at IS NOT NULL, + w.searched_at IS NOT NULL, + CASE w.reason WHEN 'request' THEN 0 WHEN 'missing' THEN 1 ELSE 2 END, + w.searched_at NULLS FIRST, w.wanted_at, w.id + LIMIT sqlc.arg(max_rows); + +-- name: GetWantForSearch :one +SELECT w.id, w.release_group_mbid, w.artist_mbid, w.artist_name, w.title, w.reason, w.fulfilled_at, + w.upgrade, w.held_tier, w.held_quality, w.album_id, + COALESCE(NULLIF(w.primary_type, ''), g.primary_type, '')::text AS primary_type, + COALESCE(g.secondary_types, '{}')::text[] AS secondary_types, + COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, '')::text AS first_release_date + FROM wants w + LEFT JOIN mb_release_groups g ON g.mbid = w.release_group_mbid + WHERE w.id = $1; + +-- name: EditionTrackSpan :one +-- The fewest tracks an official edition of the group has (any edition when +-- none is official), and the most any has. Zero when none is catalogued. +SELECT COALESCE(MIN(track_count) FILTER (WHERE status = 'Official'), MIN(track_count), 0)::int AS min_tracks, + COALESCE(MAX(track_count), 0)::int AS max_tracks + FROM mb_releases + WHERE release_group_mbid = $1 AND track_count > 0; + +-- name: RecordWantSearch :exec +UPDATE wants + SET searched_at = now(), + search_outcome = sqlc.arg(outcome), + search_considered = sqlc.arg(considered), + best_release = sqlc.arg(best_release), + best_verdict = sqlc.arg(best_verdict), + search_error = sqlc.arg(search_error), + updated_at = now() + WHERE id = sqlc.arg(id); + +-- name: DeleteWantCandidates :exec +DELETE FROM want_candidates WHERE want_id = $1; + +-- name: InsertWantCandidate :exec +INSERT INTO want_candidates (want_id, key, name, info_hash, indexer, source, size_bytes, seeders, + files_count, format, quality, year, verdict, reason, reasons, folded_from, rank) +VALUES (sqlc.arg(want_id), sqlc.arg(key), sqlc.arg(name), sqlc.arg(info_hash), sqlc.arg(indexer), + sqlc.arg(source), sqlc.arg(size_bytes), sqlc.arg(seeders), sqlc.arg(files_count), sqlc.arg(format), + sqlc.arg(quality), sqlc.arg(year), sqlc.arg(verdict), sqlc.arg(reason), sqlc.arg(reasons), + sqlc.arg(folded_from), sqlc.arg(rank)) +ON CONFLICT (want_id, key) DO NOTHING; + +-- name: ListWantCandidates :many +SELECT * FROM want_candidates WHERE want_id = $1 ORDER BY rank, key; + +-- name: GetWantCandidate :one +SELECT * FROM want_candidates WHERE want_id = $1 AND key = $2; + +-- name: MarkCandidatesBlocked :exec +UPDATE want_candidates + SET verdict = 'blocked', reason = sqlc.arg(reason), + reasons = jsonb_build_array(jsonb_build_object('kind', 'hard', 'code', 'blocked', 'text', sqlc.arg(reason)::text)) + WHERE info_hash = sqlc.arg(info_hash); + +-- name: MarkCandidatesUnblocked :exec +UPDATE want_candidates + SET verdict = 'review', reason = sqlc.arg(reason), + reasons = jsonb_build_array(jsonb_build_object('kind', 'soft', 'code', 'unblocked', 'text', sqlc.arg(reason)::text)) + WHERE info_hash = sqlc.arg(info_hash) AND verdict = 'blocked'; + +-- name: GetLiveGrab :one +SELECT * FROM grabs WHERE want_id = $1 AND ended_at IS NULL; + +-- name: InsertGrab :one +INSERT INTO grabs (want_id, info_hash, name, key, indexer, source, client, save_path, quality, + origin, forced, grabbed_by) +VALUES (sqlc.narg(want_id), sqlc.arg(info_hash), sqlc.arg(name), sqlc.arg(key), sqlc.arg(indexer), + sqlc.arg(source), sqlc.arg(client), sqlc.arg(save_path), sqlc.arg(quality), + sqlc.arg(origin), sqlc.arg(forced), sqlc.narg(grabbed_by)) +RETURNING *; + +-- name: EndGrab :exec +UPDATE grabs + SET ended_at = now(), end_reason = sqlc.arg(end_reason), end_detail = sqlc.arg(end_detail) + WHERE id = sqlc.arg(id) AND ended_at IS NULL; + +-- name: ListReleaseBlocks :many +-- Every release set aside and still held back: a stalled one lifts by itself +-- at expires_at. +SELECT * FROM release_blocks + WHERE expires_at IS NULL OR expires_at > now() + ORDER BY blocked_at DESC; + +-- name: BlockRelease :exec +-- A release set aside. expires_at NULL is for good. A block for good is never +-- shortened by a timed one: only a timed block, or another for good, is +-- replaced. +INSERT INTO release_blocks (info_hash, name, reason, detail, want_id, blocked_by, expires_at) +VALUES (sqlc.arg(info_hash), sqlc.arg(name), sqlc.arg(reason), sqlc.arg(detail), + sqlc.narg(want_id), sqlc.narg(blocked_by), sqlc.narg(expires_at)) +ON CONFLICT (info_hash) DO UPDATE + SET reason = EXCLUDED.reason, detail = EXCLUDED.detail, blocked_at = now(), + expires_at = EXCLUDED.expires_at, blocked_by = EXCLUDED.blocked_by + WHERE release_blocks.expires_at IS NOT NULL OR EXCLUDED.expires_at IS NULL; + +-- name: UnblockRelease :execrows +DELETE FROM release_blocks WHERE info_hash = $1; diff --git a/internal/db/queries/sessions.sql b/internal/db/queries/sessions.sql index a17245fd..6db52936 100644 --- a/internal/db/queries/sessions.sql +++ b/internal/db/queries/sessions.sql @@ -41,7 +41,8 @@ DELETE FROM sessions WHERE id = $1; DELETE FROM sessions WHERE token_hash = $1; -- name: DeleteSessionForUser :execrows --- Scoped by user_id, not just id (rule #47). Keyed on the id alone, any +-- Scoped by user_id, not just id: household data is per-user. Keyed on the +-- id alone, any -- household member could revoke another member's session by guessing a uuid. -- execrows lets the handler answer 404 rather than a false 204 when the row -- isn't theirs. diff --git a/internal/db/queries/similarity.sql b/internal/db/queries/similarity.sql index 70949a35..b4beef0d 100644 --- a/internal/db/queries/similarity.sql +++ b/internal/db/queries/similarity.sql @@ -3,8 +3,8 @@ -- ListenBrainz has never answered for, or not in 30 days. Never-fetched first, -- then the oldest answer. Freshness is read from track_similarity_fetches, not -- from the edges, so a seed whose answer matched nothing in the library still --- counts as fetched (#5296); reading the edges is what let 25 such seeds hold --- the head of the queue forever. +-- counts as fetched; reading the edges would let such seeds hold the head of +-- the queue forever. SELECT t.id, t.mbid FROM tracks t LEFT JOIN track_similarity_fetches f ON f.track_id = t.id @@ -68,13 +68,29 @@ ON CONFLICT (artist_id) DO UPDATE SET fetched_at = now(), returned = EXCLUDED.re -- pool would offer the same song twice. No cap per seed: LB's answer is already -- at most 50. -- +-- The score is RELATIVE: each edge's LB score over the seed's best score in its +-- whole cached answer, so 1.0 is the seed's closest recording and the rest fall +-- below it. LB's raw scores are counts, into the thousands, and the scorer +-- reads similarity as 0 to 1, so a raw edge would outweigh likes, skips and +-- recency together. The cache keeps the raw +-- answer; only the edge is scaled. +-- -- Only edges of seeds the cache owns are removed, those with a fetch row. -- Edges written before the cache existed, or copied onto a survivor by a fold -- (merge.sql), stay until their seed is fetched. -WITH want AS ( +WITH best AS ( + SELECT seed_track_id, max(score) AS score + FROM listenbrainz_similar_recordings + WHERE sqlc.narg(seed_ids)::uuid[] IS NULL + OR seed_track_id = ANY(sqlc.narg(seed_ids)::uuid[]) + GROUP BY seed_track_id +), +want AS ( SELECT DISTINCT ON (c.seed_track_id, c.recording_mbid) - c.seed_track_id AS track_a_id, t.id AS track_b_id, c.score + c.seed_track_id AS track_a_id, t.id AS track_b_id, + CASE WHEN b.score > 0 THEN c.score / b.score ELSE 0 END AS score FROM listenbrainz_similar_recordings c + JOIN best b ON b.seed_track_id = c.seed_track_id JOIN tracks t ON t.mbid = c.recording_mbid AND t.missing_since IS NULL AND t.id <> c.seed_track_id diff --git a/internal/db/queries/songs.sql b/internal/db/queries/songs.sql index bc3e200a..5095c918 100644 --- a/internal/db/queries/songs.sql +++ b/internal/db/queries/songs.sql @@ -1,4 +1,4 @@ --- Song links (M498 #5438): copies of one song on different releases share a +-- Song links: copies of one song on different releases share a -- song_key (migration 0076). -- name: LinkTracksAsSong :one @@ -41,18 +41,20 @@ UPDATE tracks SET song_id = NULL -- name: ListTrackSongKeys :many -- What the mix writer needs to place a track: its song, and whether it is a --- video rip held back from the mixes (#5439). +-- video rip held back from the mixes. SELECT id, song_key, (source_verdict IS NOT DISTINCT FROM 'suspect')::boolean AS held_back FROM tracks WHERE id = ANY(sqlc.arg(ids)::uuid[]); -- name: MergeInheritSongLink :exec -- A merge keeps the removed copy's song link: when the copy being removed was -- linked to copies on other releases and the survivor was not, the survivor --- joins that song. +-- joins that song. The copy whose id names the song has no song_id of its +-- own, so "linked" is another copy sharing its key, not a song_id. UPDATE tracks s SET song_id = l.song_key FROM tracks l WHERE s.id = sqlc.arg(survivor_id)::uuid AND l.id = sqlc.arg(loser_id)::uuid AND s.song_id IS NULL - AND l.song_key <> l.id; + AND EXISTS (SELECT 1 FROM tracks o + WHERE o.song_key = l.song_key AND o.id <> l.id AND o.id <> s.id); diff --git a/internal/db/queries/suggestion_snoozes.sql b/internal/db/queries/suggestion_snoozes.sql index 952b1ef7..1bd47d0e 100644 --- a/internal/db/queries/suggestion_snoozes.sql +++ b/internal/db/queries/suggestion_snoozes.sql @@ -1,4 +1,4 @@ --- Time-boxed "not right now" on a Discover artist suggestion (#2374). +-- Time-boxed "not right now" on a Discover artist suggestion. -- -- See 0049_suggestion_snoozes.up.sql for why this is a snooze and not a -- dismissal: it records no verdict on the music, expires on its own, and diff --git a/internal/db/queries/system_mixes.sql b/internal/db/queries/system_mixes.sql index 6469ae06..e83b2da3 100644 --- a/internal/db/queries/system_mixes.sql +++ b/internal/db/queries/system_mixes.sql @@ -1,18 +1,18 @@ -- Candidate queries for the discovery system-playlist mixes --- (#419-423). Each returns (id, album_id, artist_id) rows; the Go +--. Each returns (id, album_id, artist_id) rows; the Go -- producer caps + truncates and converts to rankedCandidate. Daily- -- deterministic md5(id||dateStr) ordering (where used) keeps a mix -- stable within a day while rotating across days; shuffle-on-play -- gives per-play variety on top. -- name: ListDeepCutsTracks :many --- #419 Deep Cuts: low-play tracks (<=2 plays) from artists the user +-- Deep Cuts: low-play tracks (<=2 plays) from artists the user -- has liked OR played heavily (>=5 non-skip plays across the artist). --- Tracks the user has skipped twice or more don't qualify (#1257): --- eligibility used to count only unskipped plays, so the most actively --- rejected tracks read as "barely heard" and kept being re-offered. --- Threshold 2 so a single accidental skip doesn't banish a track; --- passive-signal only (no dislike UI, rule #101). +-- Tracks the user has skipped twice or more don't qualify: counting only +-- unskipped plays would make the most actively rejected tracks read as +-- "barely heard" and keep re-offering them. Threshold 2 so a single +-- accidental skip doesn't banish a track; passive signal only (there is no +-- dislike button). -- $1 user_id, $2 date string. WITH affinity_artists AS ( SELECT artist_id FROM general_likes_artists WHERE user_id = $1 @@ -40,22 +40,21 @@ SELECT t.id, t.album_id, t.artist_id JOIN affinity_artists aa ON aa.artist_id = t.artist_id LEFT JOIN play_counts pc ON pc.track_id = t.id LEFT JOIN skip_counts sc ON sc.track_id = t.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND COALESCE(pc.c, 0) <= 2 AND COALESCE(sc.c, 0) < 2 AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY md5(t.id::text || $2::text) LIMIT 200; -- name: ListRediscoverTracks :many --- #420 Rediscover: tracks the user played a lot but has drifted away --- from. One blended pool (issue #1246: the old two-tier UNION was --- all-or-nothing — a single ">=6 months cold" row suppressed the whole --- ">=30 days" tier, which is how a one-song playlist shipped; the 6mo --- tier was also a strict subset of the 30d tier). Eligibility is >=3 +-- Rediscover: tracks the user played a lot but has drifted away +-- from. One blended pool, not a two-tier UNION: the ">=6 months cold" tier is +-- a strict subset of the ">=30 days" one, and a UNION that let one row of it +-- suppress the wider tier would ship a one-song playlist. Eligibility is >=3 -- non-skip plays and >=30 days cold; the bar sits at 3 rather than 5 -- so a weeks-old library still fields a pool — on young histories the -- >=5-play tracks are precisely the ones still in rotation. Ordering @@ -71,18 +70,18 @@ WITH stats AS ( SELECT t.id, t.album_id, t.artist_id FROM tracks t JOIN stats s ON s.track_id = t.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND s.c >= 3 AND s.last_at <= now() - interval '30 days' AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) - -- Declined-resurfacing cooldown (#1258): a skip on a + -- Declined-resurfacing cooldown: a skip on a -- rediscover-sourced play is the user explicitly saying "I've moved -- on" to the exact invitation this mix extends — such tracks sit -- out ~90 days instead of re-qualifying the next day. Passive - -- signal only (rule #101). + -- signal only. AND NOT EXISTS ( SELECT 1 FROM play_events rp WHERE rp.user_id = $1 @@ -97,14 +96,13 @@ SELECT t.id, t.album_id, t.artist_id LIMIT 200; -- name: ListNewForYouTracks :many --- #421 New for you, tiered per project rule #131 (#1267). The old --- single hard rule (added <30d AND direct affinity) had two failure --- modes with one root: no notion of consumption — the album you --- devoured in week one crowded the mix for three more weeks, and --- after a quiet month nothing qualified at all. +-- New for you, tiered so the mix degrades rather than vanishes. A +-- single hard rule (added <30d AND direct affinity) has no notion of +-- consumption: the album you devoured in week one would crowd the mix +-- for three more weeks, and after a quiet month nothing would qualify. -- -- "Consumed" = any track attempted for >=30s (the First Listens --- threshold, #1268): once you've meaningfully engaged, the album is +-- threshold): once you've meaningfully engaged, the album is -- no longer news. Applies to every tier — played albums leave the -- mix the next build. -- @@ -151,9 +149,9 @@ albums_tiered AS ( SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier FROM tracks t JOIN albums_tiered alt ON alt.album_id = t.album_id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY alt.tier, alt.created_at DESC, t.album_id, @@ -161,18 +159,18 @@ SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier LIMIT 300; -- name: ListOnThisDayTracks :many --- #422 On This Day: tracks the user played around this calendar date +-- On This Day: tracks the user played around this calendar date -- (±10 day-of-year) in the past, excluding the very recent (>30 days -- ago) so it's nostalgic, not just "last week". Weighted by how much --- they were played in those windows. Floor relaxed from 60→30 days --- and window ±7→±10 so it surfaces on a months-old library instead --- of needing a full year of history; still skips cleanly (no rows → --- no playlist) when there's no qualifying history yet. --- Day-of-year distance is circular (#1256): plain ABS made Dec 28 vs --- Jan 3 read as 359 days apart, silently gutting the window for ~3 --- weeks around every New Year — precisely when holiday nostalgia is --- the point. LEAST(d, 365-d) wraps the boundary; leap-year drift of --- ±1 is absorbed by the ±10 window. The build date ($2, already here +-- they were played in those windows. The 30-day floor and ±10 window +-- let it surface on a months-old library rather than needing a full +-- year of history; it still skips cleanly (no rows → no playlist) +-- when there's no qualifying history yet. +-- Day-of-year distance is circular: plain ABS would read Dec 28 vs +-- Jan 3 as 359 days apart, gutting the window for ~3 weeks around +-- every New Year — precisely when holiday nostalgia is the point. +-- LEAST(d, 365-d) wraps the boundary; leap-year drift of ±1 is absorbed +-- by the ±10 window. The build date ($2, already here -- for the md5 rotation) anchors "today" instead of now() so the -- window is testable and consistent with the mix's daily determinism. -- $1 user_id, $2 date string. @@ -190,20 +188,19 @@ WITH windowed AS ( SELECT t.id, t.album_id, t.artist_id FROM tracks t JOIN windowed w ON w.track_id = t.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY w.c DESC, md5(t.id::text || $2::text) LIMIT 200; -- name: ListFirstListensTracks :many --- #423 First Listens: songs the user has never heard or even --- attempted. "Attempted" is track-level with a >=30s listen threshold --- (#1268, operator decision 2026-07-03) — the old version disqualified --- a whole album on ANY play_event, so a 2-second accidental brush --- banished it permanently. +-- First Listens: songs the user has never heard or even +-- attempted. "Attempted" is track-level with a >=30s listen threshold: +-- disqualifying a whole album on ANY play_event would let a 2-second +-- accidental brush banish it permanently. -- -- tier 1 the exact desire: albums with ZERO attempted tracks -- tier 2 step back: barely-attempted albums (<=25% of tracks @@ -244,11 +241,11 @@ SELECT t.id, t.album_id, t.artist_id, alt.tier::int AS tier FROM tracks t JOIN albums al ON al.id = t.album_id JOIN albums_tiered alt ON alt.album_id = al.id - WHERE t.missing_since IS NULL -- #2523: never offer a file that is gone + WHERE t.missing_since IS NULL -- never offer a file that is gone AND alt.tier IS NOT NULL AND NOT EXISTS (SELECT 1 FROM attempted a WHERE a.track_id = t.id) AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY diff --git a/internal/db/queries/system_playlist_rotation.sql b/internal/db/queries/system_playlist_rotation.sql index 3f95050e..d58a792a 100644 --- a/internal/db/queries/system_playlist_rotation.sql +++ b/internal/db/queries/system_playlist_rotation.sql @@ -1,4 +1,4 @@ --- Rotation state for system-playlist sample-history dedup (#415). +-- Rotation state for system-playlist sample-history dedup. -- One row per (user, playlist_kind). played_track_ids accumulates -- tracks heard from that playlist in the current rotation; shuffle -- prefers ids NOT in this set, then resets when the snapshot is diff --git a/internal/db/queries/system_playlists.sql b/internal/db/queries/system_playlists.sql index 7c97a7a1..f5f76d0f 100644 --- a/internal/db/queries/system_playlists.sql +++ b/internal/db/queries/system_playlists.sql @@ -1,4 +1,4 @@ --- Track picks here join `tracks ... AND t.missing_since IS NULL` (#2523): a +-- Track picks here join `tracks ... AND t.missing_since IS NULL`: a -- seed or For-You candidate has to be something that can actually play. Note -- this only affects newly GENERATED playlists — already-stored system -- playlists keep their rows until the next daily rebuild, which is why the @@ -6,7 +6,7 @@ -- also serves user-curated playlists, where hiding a track the user added -- themselves would be wrong). --- M7 #352 slice 2: system-generated playlist queries. +-- System-generated playlist queries. -- name: ListActiveUsersForSystemPlaylists :many -- Active = had a play in the last 7 days. The cron iterates this list. @@ -56,22 +56,22 @@ UPDATE system_playlist_runs SET in_flight = false WHERE in_flight = true; -- name: PickSeedArtists :many -- Top-12 most-engaged distinct artist candidates, tiered so the --- "Songs like X" mixes never silently vanish (#1255): the old hard --- 7-day window emptied the seed pool after a quiet week, and the --- daily atomic-replace build then deleted every existing mix until --- the user played something again. Same fallback shape as +-- "Songs like X" mixes never silently vanish: a hard 7-day window +-- would empty the seed pool after a quiet week, and the daily +-- atomic-replace build would then delete every existing mix until the +-- user played something again. Same fallback shape as -- PickTopPlayedTracksForUser (For You's seeds): -- tier 0 engagement in the last 7 days -- tier 1 (only if tier 0 empty) last 30 days -- tier 2 (only if tiers 0+1 empty) all-time -- tier 3 (only if 0-2 empty) liked artists with no play history -- All returned rows share one tier; produceSeedMixes maps it onto the --- rule-#131 pick-kind ladder and stamps the built tracks, so metrics +-- tiered pick-kind ladder and stamps the built tracks, so metrics -- can compare mixes seeded from fresh vs stale engagement. -- Score = unskipped-play count + 5 if user has liked the artist. The -- Go-side picker (pickSeedArtistsForDay) shuffles the 12 -- daily-deterministically and takes songsLikeSeedCount (6) so the mix --- set both rotates day-to-day and fills the dedicated Home row (#1491). +-- set both rotates day-to-day and fills the dedicated Home row. WITH liked AS ( SELECT gla.artist_id FROM general_likes_artists gla WHERE gla.user_id = $1 ), @@ -163,7 +163,7 @@ alltime AS ( ), -- A likes-only read still joins tracks: a like outlives its file, and a -- seed whose file is gone points For You at something the user can't hear --- (#2701). The other two tiers get the same filter from their play_events +--. The other two tiers get the same filter from their play_events -- join. liked AS ( SELECT gl.track_id AS id, 0::bigint AS c, 2 AS tier @@ -206,7 +206,7 @@ SELECT COALESCE( FROM tracks t JOIN albums a ON a.id = t.album_id WHERE t.artist_id = $2 - AND t.missing_since IS NULL -- #2701: the play branch filters; this one must too + AND t.missing_since IS NULL -- the play branch filters; this one must too ORDER BY a.release_date DESC NULLS LAST, t.disc_number NULLS LAST, t.track_number NULLS LAST, diff --git a/internal/db/queries/tag_settings.sql b/internal/db/queries/tag_settings.sql index 03748656..f48664e3 100644 --- a/internal/db/queries/tag_settings.sql +++ b/internal/db/queries/tag_settings.sql @@ -1,4 +1,4 @@ --- Tag-enrichment provider settings queries (milestone #160, #1490 Step 2). +-- Tag-enrichment provider settings queries. -- The tags.SettingsService consumes these — parallel to the cover-art -- SettingsService (coverart_settings.sql). Keeping the two independent -- means a new tag source is added without touching the art settings. @@ -12,7 +12,7 @@ SELECT provider_id, enabled, api_key, display_order, created_at, updated_at -- name: UpsertTagProviderSettings :exec -- INSERT a default row for a newly-registered provider, or refresh an -- existing row's display_order on boot. Does NOT overwrite enabled / --- api_key on conflict — the operator's settings persist across restarts. +-- api_key on conflict — an admin's settings persist across restarts. INSERT INTO tag_provider_settings (provider_id, enabled, display_order) VALUES ($1, $2, $3) ON CONFLICT (provider_id) DO UPDATE diff --git a/internal/db/queries/taste_profile.sql b/internal/db/queries/taste_profile.sql index d9ddb987..d04da13f 100644 --- a/internal/db/queries/taste_profile.sql +++ b/internal/db/queries/taste_profile.sql @@ -1,13 +1,13 @@ --- Taste profile (#796 phase 1). The daily build computes signed artist/tag +-- Taste profile. The daily build computes signed artist/tag -- weights in Go (graded engagement + decay + like bonus) and atomic-replaces --- them here. Read queries serve "top taste" lookups (used in phase 2). +-- them here. Read queries serve the scorer's "top taste" lookups. -- name: ListPlayEngagementInputsForUser :many -- One row per play in the decay-relevant window. completion is the play's -- completion ratio (precomputed column when present, else duration_played / -- track duration, clamped to [0,1]); age_days drives the time-decay. Genre -- is split into tags in Go; release_date derives the decade for the era --- facet (#1530). Quarantined tracks are excluded. +-- facet. Quarantined tracks are excluded. SELECT t.id AS track_id, t.artist_id, @@ -24,14 +24,14 @@ JOIN albums a ON a.id = t.album_id WHERE pe.user_id = $1 AND pe.started_at > now() - ($2::float8 * INTERVAL '1 day') AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ); -- name: ListLikedTrackTasteInputsForUser :many -- (track_id, artist_id, genre, release_date) for each track the user has -- explicitly liked. Feeds the track-like bonus into the liked track's artist, --- tags, and era (#1530); track_id keys the enriched track_tags lookup (#1490). +-- tags, and era; track_id keys the enriched track_tags lookup. SELECT t.id AS track_id, t.artist_id, t.genre, a.release_date FROM general_likes gl JOIN tracks t ON t.id = gl.track_id @@ -57,7 +57,7 @@ INSERT INTO taste_profile_tags (user_id, tag, weight) VALUES ($1, $2, $3); -- name: ListTasteProfileArtistsForUser :many --- Top-weighted taste artists (phase-2 consumption + tests). +-- Top-weighted taste artists, for the scorer and tests. SELECT artist_id, weight FROM taste_profile_artists WHERE user_id = $1 @@ -79,7 +79,7 @@ INSERT INTO taste_profile_eras (user_id, era, weight) VALUES ($1, $2, $3); -- name: ListTasteProfileErasForUser :many --- Top-weighted taste eras (#1530); consumed by the scorer's era term. +-- Top-weighted taste eras; consumed by the scorer's era term. SELECT era, weight FROM taste_profile_eras WHERE user_id = $1 @@ -94,7 +94,7 @@ INSERT INTO taste_profile_moods (user_id, mood, weight) VALUES ($1, $2, $3); -- name: ListTasteProfileMoodsForUser :many --- Top-weighted taste moods (#1534); consumed by the scorer's mood term. +-- Top-weighted taste moods; consumed by the scorer's mood term. SELECT mood, weight FROM taste_profile_moods WHERE user_id = $1 diff --git a/internal/db/queries/track_tags.sql b/internal/db/queries/track_tags.sql index 9449265a..44c003c5 100644 --- a/internal/db/queries/track_tags.sql +++ b/internal/db/queries/track_tags.sql @@ -1,4 +1,4 @@ --- Track-tag enrichment queries (milestone #160, #1490). The tag enricher +-- Track-tag enrichment queries. The tag enricher -- drains ListTracksMissingTags, fetches from the provider chain, and -- writes the merged top-K via DeleteTrackTags + InsertTrackTag, then -- stamps SetTrackTagSource. Mirrors the coverart artist-enricher shape. @@ -8,7 +8,7 @@ -- or previously settled 'none' under an older provider version. Returns -- the fields the provider chain needs — recording MBID (nullable) for -- keyed lookups, title + artist name for name-based fallback, and the --- artist MBID (nullable) for MusicBrainz's artist-tag fallback (#1519). +-- artist MBID (nullable) for MusicBrainz's artist-tag fallback. -- $1 = current tag_sources_version, $2 = limit. SELECT t.id, t.mbid, t.title, a.name AS artist_name, a.mbid AS artist_mbid FROM tracks t @@ -58,7 +58,7 @@ SELECT tt.track_id, tt.tag, tt.weight WHERE gl.user_id = $1; -- name: ListTrackTagsForTracks :many --- Enriched tags for a set of candidate tracks (#1534), so the scorer can +-- Enriched tags for a set of candidate tracks, so the scorer can -- derive each candidate's mood buckets at scoring time. One row per (track, -- tag); the caller filters to mood words via the internal/mood vocabulary. SELECT tt.track_id, tt.tag diff --git a/internal/db/queries/tracks.sql b/internal/db/queries/tracks.sql index b8e7d9d3..6be1133e 100644 --- a/internal/db/queries/tracks.sql +++ b/internal/db/queries/tracks.sql @@ -1,7 +1,7 @@ -- name: UpsertTrack :one -- file_path is the canonical identity for library scan; mbid is secondary. --- mbid is the file's tag. A looked-up id (mbid_source 'acoustid', M401) --- survives a re-read that finds no tag id, and a tag id replaces it (D4). +-- mbid is the file's tag. A looked-up id (mbid_source 'acoustid') +-- survives a re-read that finds no tag id, and a tag id replaces it. INSERT INTO tracks ( title, album_id, artist_id, track_number, disc_number, duration_ms, file_path, file_size, file_format, bitrate, mbid, genre, @@ -25,7 +25,7 @@ ON CONFLICT (file_path) DO UPDATE SET WHEN tracks.mbid_source = 'acoustid' THEN 'acoustid' END, genre = EXCLUDED.genre, -- Stamped on update too, so a tag-repair pass marks rows as done and the - -- next scan can short-circuit them again (#2499). + -- next scan can short-circuit them again. tag_read_version = EXCLUDED.tag_read_version, updated_at = now() RETURNING *; @@ -34,7 +34,7 @@ RETURNING *; -- Track recording-MBID backfill: tracks with NULL mbid that still have -- a file to re-read. $1 caps the batch (mirrors the album backfill). -- Missing tracks are skipped: there is no file, and opening it anyway --- logged a warning per track on every scan (#5139). +-- would log a warning per track on every scan. SELECT id, file_path FROM tracks WHERE mbid IS NULL @@ -62,7 +62,7 @@ SELECT * FROM tracks WHERE file_path = $1; SELECT * FROM tracks WHERE album_id = $1 AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = tracks.id ) ORDER BY disc_number NULLS LAST, track_number NULLS LAST; @@ -76,7 +76,7 @@ SELECT count(*) FROM tracks WHERE album_id = $1; SELECT * FROM tracks WHERE title ILIKE '%' || $1::text || '%' AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = tracks.id ) ORDER BY title @@ -87,13 +87,13 @@ LIMIT $3 OFFSET $4; SELECT COUNT(*) FROM tracks WHERE title ILIKE '%' || $1::text || '%' AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = tracks.id ); -- name: ListArtistTracksForUser :many --- M6a: every track for the artist across their albums, with album_title --- and artist_name joined. Honors per-user lidarr_quarantine. Used by +-- Every track for the artist across their albums, with album_title +-- and artist_name joined. Honors per-user quarantine. Used by -- /api/artists/{id}/tracks for the artist-card play affordance, which -- shuffles client-side. Ordering matches album/track natural order so -- the shuffle has a deterministic input. @@ -105,14 +105,14 @@ JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id WHERE t.artist_id = $1 AND NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $2 AND q.track_id = t.id ) ORDER BY albums.release_date NULLS LAST, albums.sort_title, t.disc_number NULLS FIRST, t.track_number NULLS FIRST, t.id; -- name: ListRandomTracksForUser :many --- #427 S4: backing query for GET /api/library/shuffle — the online +-- Backing query for GET /api/library/shuffle — the online -- source for "Shuffle all". N random tracks across the whole -- library, per-user-quarantine filtered. $1 user_id, $2 limit. SELECT sqlc.embed(t), @@ -122,7 +122,7 @@ FROM tracks t JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id WHERE NOT EXISTS ( - SELECT 1 FROM lidarr_quarantine q + SELECT 1 FROM quarantine q WHERE q.user_id = $1 AND q.track_id = t.id ) ORDER BY random() @@ -135,9 +135,9 @@ LIMIT $2; SELECT COUNT(*) FROM tracks WHERE artist_id = $1; -- name: DeleteTrack :one --- M7 #372: hard delete with FK cascade. The CASCADE on track_id from --- play_events / general_likes_tracks / lidarr_quarantine / --- lidarr_quarantine_actions handles their cleanup. RETURNING gives us +-- Hard delete with FK cascade. The CASCADE on track_id from +-- play_events / general_likes_tracks / quarantine / +-- quarantine_actions handles their cleanup. RETURNING gives us -- album_id + artist_id for the album-empty / artist-empty cascade -- checks the service does next. DELETE FROM tracks WHERE id = $1 @@ -145,11 +145,11 @@ RETURNING id, album_id, artist_id, file_path, mbid; -- name: GetTracksByIDs :many -- Batched lookup used by /api/library/sync to hydrate upsert payloads --- (#357). Mirror of GetArtistsByIDs. +--. Mirror of GetArtistsByIDs. SELECT * FROM tracks WHERE id = ANY($1::uuid[]); -- name: FindMissingTrackByMbid :many --- Move detection, strongest signal (#2528). A file that turned up at a new path +-- Move detection, strongest signal. A file that turned up at a new path -- carrying a recording MBID we already have on a MISSING row is that recording, -- moved — not a new track. -- @@ -166,11 +166,11 @@ SELECT id, file_path FROM tracks LIMIT 2; -- name: FindMissingTrackByAudioHash :many --- Move detection fallback for files with no MBID (#2528, #3914). The audio stream +-- Move detection fallback for files with no MBID. The audio stream -- hash identifies the encoded audio itself, so it survives a rename, a move and a --- retag — anything short of a re-encode. It replaced (file_size, duration_ms), --- which could pair two unrelated files that happened to share a byte count and a --- duration, and missed a file retagged in place, whose size changes. +-- retag — anything short of a re-encode. (file_size, duration_ms) would pair two +-- unrelated files that share a byte count and a duration, and miss a file +-- retagged in place, whose size changes. -- -- Same missing-only constraint and same LIMIT 2 rationale as the MBID variant. SELECT t.id, t.file_path @@ -196,7 +196,7 @@ UPDATE tracks -- name: ListTrackPathsForReconcile :many -- Every row's path + current missing mark, for the scanner's reconcile pass --- (#2523). Deliberately unfiltered and unpaged: reconcile has to compare the +--. Deliberately unfiltered and unpaged: reconcile has to compare the -- WHOLE table against what the walk saw, and a filtered subset would let rows -- outside it drift forever. Three narrow columns keep it cheap even on a -- library of a few hundred thousand tracks. @@ -216,6 +216,25 @@ UPDATE tracks WHERE id = ANY(sqlc.arg(ids)::uuid[]) AND missing_since IS NULL; +-- name: ClearTrackMbids :exec +-- Forgets the recording MBID of rows whose file was another song's (the +-- duplicate resolver's wrong-slot removal). The MBID came from that file, so it +-- names the song kept elsewhere on the album, and missing-pair folding would +-- fold the emptied slot into it, so it would never be wanted back. +-- mbid_source goes with it (tracks_mbid_source_pair). +UPDATE tracks + SET mbid = NULL, mbid_source = NULL + WHERE id = ANY(sqlc.arg(ids)::uuid[]); + +-- name: MarkTracksEmptied :exec +-- Stamps rows the duplicate resolver emptied, whose song the album holds no +-- other copy of, with the time they went missing (migration 0095): missing-pair +-- folding then waits for a copy fetched again. +UPDATE tracks + SET emptied_at = missing_since + WHERE id = ANY(sqlc.arg(ids)::uuid[]) + AND missing_since IS NOT NULL; + -- name: ClearTracksMissing :execrows -- Clears the mark on rows whose file is back. Runs independently of the mtime -- skip check, so a file that reappears unchanged is un-marked even though the @@ -226,19 +245,18 @@ UPDATE tracks AND missing_since IS NOT NULL; -- name: ListMissingTracks :many --- The admin review surface for files the scan could not find (#2527). +-- The admin review surface for files the scan could not find. -- -- Ordered by directory, then by the file's own position within its album, --- because the unit an operator actually reasons about is a FOLDER: the case --- this was built for was three whole albums that had been reorganised, and a --- flat list ordered by timestamp presents that as forty unrelated decisions. +-- because the unit an admin actually reasons about is a FOLDER: a reorganised +-- album, listed flat by timestamp, reads as dozens of unrelated decisions. -- Grouping happens in the handler; the ordering here is what makes a group -- contiguous, so a page boundary splits a directory at worst. -- -- last_played_at is a correlated MAX rather than a join so a track with no -- plays stays in the result with NULL. It is here because "gone six months, -- never played" and "gone yesterday, played 200 times" deserve opposite --- reactions, and the operator can't tell them apart from a path. +-- reactions, and an admin can't tell them apart from a path. SELECT t.id, t.title, t.file_path, @@ -266,7 +284,7 @@ SELECT COUNT(*) FROM tracks WHERE missing_since IS NOT NULL; -- name: ListSuspectSourceTracks :many -- The admin report of present tracks whose filename looks like a video rip --- (#5410): "(Official Video)", "[Audio]", "Visualizer" and the like. +--: "(Official Video)", "[Audio]", "Visualizer" and the like. -- -- pattern is built in Go (internal/api/admin_suspect_sources.go) from the -- same marker list that labels each row, so the filter and the labels cannot @@ -287,10 +305,18 @@ SELECT t.id, artists.id AS artist_id, artists.name AS artist_name, t.source_verdict, - t.source_verdict_at + t.source_verdict_at, + -- The album's place on the wanted list: wanted for a clean copy, and + -- when that was last searched for. + COALESCE(w.upgrade, '')::text AS want_upgrade, + w.searched_at AS searched_at FROM tracks t JOIN albums ON albums.id = t.album_id JOIN artists ON artists.id = t.artist_id + LEFT JOIN LATERAL ( + SELECT x.upgrade, x.searched_at FROM wants x + WHERE x.album_id = albums.id AND x.fulfilled_at IS NOT NULL LIMIT 1 + ) w ON true WHERE t.missing_since IS NULL AND regexp_replace(t.file_path, '^.*/', '') ~* sqlc.arg(pattern)::text ORDER BY directory, t.disc_number NULLS FIRST, t.track_number NULLS FIRST, t.title @@ -303,8 +329,8 @@ SELECT COUNT(*) FROM tracks t AND regexp_replace(t.file_path, '^.*/', '') ~* sqlc.arg(pattern)::text; -- name: FlagSuspectSourceTracks :execrows --- Holds back present tracks whose basename carries a video-rip marker (M498 --- #5439), the same pattern the report lists. A track the operator called fine +-- Holds back present tracks whose basename carries a video-rip marker, the +-- same pattern the report lists. A track an admin called fine -- keeps that verdict. UPDATE tracks SET source_verdict = 'suspect', source_verdict_at = now() @@ -313,17 +339,28 @@ UPDATE tracks AND regexp_replace(file_path, '^.*/', '') ~* sqlc.arg(pattern)::text; -- name: ClearStaleSuspectSourceTracks :execrows --- A held-back track whose name no longer carries a marker (renamed, retagged --- by Lidarr) is released again. +-- A held-back track whose name no longer carries a marker (renamed or +-- retagged) is released again. UPDATE tracks SET source_verdict = NULL, source_verdict_at = NULL WHERE source_verdict = 'suspect' AND NOT regexp_replace(file_path, '^.*/', '') ~* sqlc.arg(pattern)::text; -- name: SetTrackSourceVerdict :execrows --- The operator's call on a flagged track: 'fine' lets it back into radio and +-- An admin's call on a flagged track: 'fine' lets it back into radio and -- the mixes for good; 'suspect' holds it back again. UPDATE tracks SET source_verdict = sqlc.arg(verdict)::text, source_verdict_at = now() WHERE id = sqlc.arg(id) AND source_verdict IS NOT NULL; + +-- name: ListStillMissingTrackLabels :many +-- Which of the tracks a scan marked missing still are once it has adopted +-- moves and folded replacements: the ones worth telling an admin +-- about, by name. A quality upgrade folds into its replacement and drops out. +SELECT ar.name AS artist_name, t.title + FROM tracks t + JOIN artists ar ON ar.id = t.artist_id + WHERE t.id = ANY(sqlc.arg(ids)::uuid[]) + AND t.missing_since IS NOT NULL + ORDER BY ar.name, t.title; diff --git a/internal/db/queries/users.sql b/internal/db/queries/users.sql index 22f0ef88..71e6400a 100644 --- a/internal/db/queries/users.sql +++ b/internal/db/queries/users.sql @@ -37,12 +37,17 @@ SELECT count(*) FROM users; -- name: SetSubsonicPassword :exec -- Stores (or clears with NULL) the per-user Subsonic legacy credential used -- for t/s and p auth on /rest/*. Must be plaintext; see migration 0003. Only --- ever a server-generated value, never the login password (#5026). +-- ever a server-generated value, never the login password. UPDATE users SET subsonic_password = $2 WHERE id = $1; -- name: GetUserByID :one SELECT * FROM users WHERE id = $1; +-- name: ListUsernamesByIDs :many +-- Names a list's people in one round trip, e.g. who made each request on +-- /admin/requests. +SELECT id, username FROM users WHERE id = ANY($1::uuid[]); + -- name: SetListenBrainzToken :exec UPDATE users SET listenbrainz_token = $2, @@ -96,7 +101,7 @@ RETURNING *; -- Hard delete with cascade. The schema's ON DELETE CASCADE foreign keys -- handle plays/likes/sessions/etc. Caller must enforce the last-admin -- guard at the HTTP layer (count admins, refuse if target is the only --- admin). The lidarr_quarantine, general_likes*, play_events, +-- admin). The quarantine, general_likes*, play_events, -- sessions, and other user-FK tables all have ON DELETE CASCADE so -- this is a clean drop. DELETE FROM users WHERE id = $1; @@ -174,16 +179,3 @@ SELECT u.id, u.timezone FROM users u WHERE pe.user_id = u.id AND pe.started_at > now() - INTERVAL '7 days' ); - --- name: GetOldestAdmin :one --- The account a system-initiated action is attributed to (milestone #290). --- lidarr_requests.user_id is NOT NULL and a re-acquisition has no requesting --- human, so the row is owned by the longest-standing admin: it keeps the --- request auditable and puts it in the same admin queue as everything else, --- without inventing a synthetic principal the rest of the schema would have --- to understand. Ordered by id as a tiebreak so the choice is stable across --- calls rather than depending on scan order. -SELECT * FROM users - WHERE is_admin = true - ORDER BY created_at, id - LIMIT 1; diff --git a/internal/db/queries/wanted.sql b/internal/db/queries/wanted.sql new file mode 100644 index 00000000..11b49a98 --- /dev/null +++ b/internal/db/queries/wanted.sql @@ -0,0 +1,501 @@ +-- The wanted list: release-type profiles, follows and wants. + +-- name: ListReleaseTypeProfiles :many +SELECT p.*, + (SELECT COUNT(*) FROM follows f WHERE f.profile_id = p.id)::bigint AS follow_count + FROM release_type_profiles p + ORDER BY p.is_default DESC, p.name; + +-- name: GetReleaseTypeProfile :one +SELECT * FROM release_type_profiles WHERE id = $1; + +-- name: GetDefaultReleaseTypeProfile :one +SELECT * FROM release_type_profiles WHERE is_default; + +-- name: CreateReleaseTypeProfile :one +INSERT INTO release_type_profiles (name, primary_types, secondary_types) +VALUES ($1, $2, $3) +RETURNING *; + +-- name: UpdateReleaseTypeProfile :one +UPDATE release_type_profiles + SET name = $2, + primary_types = $3, + secondary_types = $4, + updated_at = now() + WHERE id = $1 +RETURNING *; + +-- name: ClearDefaultReleaseTypeProfile :exec +UPDATE release_type_profiles SET is_default = false, updated_at = now() WHERE is_default; + +-- name: SetDefaultReleaseTypeProfile :execrows +UPDATE release_type_profiles SET is_default = true, updated_at = now() WHERE id = $1; + +-- name: DeleteReleaseTypeProfile :execrows +-- The default stays; one in use is refused by the follows foreign key. +DELETE FROM release_type_profiles WHERE id = $1 AND NOT is_default; + +-- name: UpsertFollow :one +-- Following an artist already followed changes its profile. want_existing +-- only ever turns on here; the worker turns it off once applied, and turning +-- it on again re-reads the baseline so the existing releases are judged. +INSERT INTO follows (artist_mbid, artist_name, profile_id, followed_by, want_existing) +VALUES (sqlc.arg(artist_mbid), sqlc.arg(artist_name), sqlc.arg(profile_id), + sqlc.narg(followed_by), sqlc.arg(want_existing)) +ON CONFLICT (artist_mbid) DO UPDATE + SET artist_name = EXCLUDED.artist_name, + profile_id = EXCLUDED.profile_id, + want_existing = follows.want_existing OR EXCLUDED.want_existing, + baseline_at = CASE WHEN EXCLUDED.want_existing THEN NULL ELSE follows.baseline_at END, + updated_at = now() +RETURNING *; + +-- name: GetFollow :one +SELECT f.*, p.name AS profile_name + FROM follows f + JOIN release_type_profiles p ON p.id = f.profile_id + WHERE f.artist_mbid = $1; + +-- name: DeleteFollow :execrows +DELETE FROM follows WHERE artist_mbid = $1; + +-- name: ListFollows :many +SELECT f.artist_mbid, f.artist_name, f.profile_id, f.followed_at, f.baseline_at, f.checked_at, + p.name AS profile_name, + (SELECT a.id FROM artists a WHERE a.mbid = f.artist_mbid) AS artist_id, + (SELECT COUNT(*) FROM wants w + WHERE w.artist_mbid = f.artist_mbid AND w.fulfilled_at IS NULL)::bigint AS open_wants + FROM follows f + JOIN release_type_profiles p ON p.id = f.profile_id + ORDER BY lower(f.artist_name), f.artist_mbid; + +-- name: ListFollowsDue :many +-- Follows whose discography has not been applied since @before, never-read +-- first, so a new follow's baseline is not stuck behind daily re-checks. +SELECT f.* + FROM follows f + WHERE f.baseline_at IS NULL OR f.checked_at IS NULL OR f.checked_at < sqlc.arg(before) + ORDER BY f.baseline_at NULLS FIRST, f.checked_at NULLS FIRST, f.artist_mbid + LIMIT sqlc.arg(max_rows); + +-- name: MarkFollowChecked :exec +UPDATE follows + SET checked_at = now(), + baseline_at = COALESCE(baseline_at, now()), + want_existing = false, + updated_at = now() + WHERE artist_mbid = $1; + +-- name: ListFollowGroups :many +-- An artist's release groups from the catalogue, each with whether the +-- follow has judged it before. +SELECT g.mbid, g.title, g.primary_type, g.secondary_types, g.first_release_date, + EXISTS (SELECT 1 FROM follow_seen_groups s + WHERE s.artist_mbid = sqlc.arg(artist_mbid) AND s.release_group_mbid = g.mbid) AS seen + FROM mb_artist_release_groups ag + JOIN mb_release_groups g ON g.mbid = ag.release_group_mbid + WHERE ag.artist_mbid = sqlc.arg(artist_mbid) + ORDER BY g.first_release_date, g.title; + +-- name: MarkGroupSeen :exec +INSERT INTO follow_seen_groups (artist_mbid, release_group_mbid) +VALUES ($1, $2) +ON CONFLICT DO NOTHING; + +-- name: LibraryHoldsReleaseGroup :one +-- Whether the library already has the group: an album tagged with it or +-- with one of its releases, or failing that an album of the artist's with +-- the same title. Only albums with a track on disk count. +SELECT EXISTS ( + SELECT 1 FROM albums a + WHERE (a.release_group_mbid = sqlc.arg(group_mbid)::text + OR a.mbid = sqlc.arg(group_mbid) + OR a.mbid IN (SELECT r.mbid FROM mb_releases r WHERE r.release_group_mbid = sqlc.arg(group_mbid)) + OR (lower(a.title) = lower(sqlc.arg(title)::text) + AND a.artist_id IN (SELECT ar.id FROM artists ar WHERE ar.mbid = sqlc.arg(artist_mbid)::text))) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id AND t.missing_since IS NULL) +); + +-- name: WantReleaseGroup :one +-- Wants a release group, or returns the want already recorded for it. A +-- held want is not reopened here: the library has it. +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, primary_type, + first_release_date, reason, created_by) +VALUES (sqlc.arg(release_group_mbid), sqlc.arg(artist_mbid), sqlc.arg(artist_name), sqlc.arg(title), + sqlc.arg(primary_type), sqlc.arg(first_release_date), sqlc.arg(reason), sqlc.narg(created_by)) +ON CONFLICT (release_group_mbid) DO UPDATE SET updated_at = wants.updated_at +RETURNING *; + +-- name: GetWantByReleaseGroup :one +SELECT * FROM wants WHERE release_group_mbid = $1; + +-- name: ListOpenWants :many +SELECT * FROM wants + WHERE fulfilled_at IS NULL + ORDER BY wanted_at, id + LIMIT $1; + +-- name: WantHoldings :many +-- Each library album that may answer a want, with what it has on disk: +-- tracks present, whether one arrived since the want, and whether a track +-- missing before the want is still missing. +SELECT a.id AS album_id, + COUNT(*) FILTER (WHERE t.missing_since IS NULL)::int AS present, + COALESCE(bool_or(t.missing_since IS NULL AND t.added_at > sqlc.arg(since)::timestamptz), false)::bool AS arrived_since, + COALESCE(bool_or(t.missing_since IS NOT NULL AND t.missing_since < sqlc.arg(since)::timestamptz), false)::bool AS missing_before, + -- Missing tracks no track on disk stands in for, by position or + -- title. A missing row whose song is back at a new path is + -- covered; a lost bonus track is not. A track with no disc number is + -- on disc 1, so a missing track 6 with no disc beside a present 1/6 + -- does not keep the album wanted. + COUNT(*) FILTER (WHERE t.missing_since IS NOT NULL AND NOT EXISTS ( + SELECT 1 FROM tracks p + WHERE p.album_id = a.id AND p.missing_since IS NULL + AND ((p.track_number = t.track_number AND COALESCE(p.disc_number, 1) = COALESCE(t.disc_number, 1)) + OR lower(p.title) = lower(t.title))))::int AS uncovered_missing + FROM albums a + JOIN tracks t ON t.album_id = a.id + WHERE a.release_group_mbid = sqlc.arg(group_mbid)::text + OR a.mbid = sqlc.arg(group_mbid) + OR a.id = sqlc.narg(album_id) + OR a.mbid IN (SELECT r.mbid FROM mb_releases r WHERE r.release_group_mbid = sqlc.arg(group_mbid)) + GROUP BY a.id + ORDER BY present DESC, a.id; + +-- name: FulfillWant :exec +UPDATE wants + SET fulfilled_at = now(), + album_id = sqlc.arg(album_id), + updated_at = now() + WHERE id = sqlc.arg(id) AND fulfilled_at IS NULL; + +-- name: RepointWant :exec +-- A want recorded under a release id is moved to its group. +UPDATE wants SET release_group_mbid = $2, updated_at = now() WHERE id = $1; + +-- name: DeleteWant :execrows +DELETE FROM wants WHERE id = $1; + +-- name: ListWants :many +-- The Wanted page: open wants (all but missing ones, or only missing ones), +-- the held ones, or all of them; artist_mbid narrows to one artist's page, +-- want_ids to given wants. Ordered like the library: artists A–Z by the +-- library's sort name when it holds the artist, each artist's albums by +-- release date, so the page groups them as they arrive. q narrows to an +-- artist or album title; total counts every match, past the page. +SELECT w.id, w.release_group_mbid, w.artist_mbid, w.artist_name, w.title, w.reason, + w.album_id, w.wanted_at, w.fulfilled_at, + -- Why a held album is wanted again, and the copy it has. + w.upgrade, w.held_quality, + -- A request names an album before the catalogue has read it: the + -- catalogue's type and date fill in once it has. + COALESCE(NULLIF(w.primary_type, ''), g.primary_type, '')::text AS primary_type, + COALESCE(g.secondary_types, '{}')::text[] AS secondary_types, + COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, '')::text AS first_release_date, + u.username AS requested_by, + la.id AS library_artist_id, + COUNT(*) OVER () AS total, + -- The search schedule's last look, and the download running. + w.searched_at, w.search_outcome, w.search_considered, w.best_release, w.best_verdict, w.search_error, + -- Its latest download, running or ended, and what the import made of + -- it. + gr.id AS grab_id, gr.name AS grab_name, gr.grabbed_at, gr.origin AS grab_origin, + gr.quality AS grab_quality, gr.ended_at AS grab_ended_at, gr.end_reason AS grab_end_reason, + gr.progress AS grab_progress, gr.import_state AS grab_import_state, + gr.import_reason AS grab_import_reason, gr.end_detail AS grab_end_detail, gr.imported_at AS grab_imported_at, + gr.imported_files AS grab_imported_files + FROM wants w + LEFT JOIN users u ON u.id = w.created_by + LEFT JOIN mb_release_groups g ON g.mbid = w.release_group_mbid + LEFT JOIN LATERAL (SELECT ar.id, ar.sort_name FROM artists ar WHERE ar.mbid = w.artist_mbid LIMIT 1) la ON true + LEFT JOIN grabs gr ON gr.id = (SELECT x.id FROM grabs x WHERE x.want_id = w.id + ORDER BY x.ended_at IS NULL DESC, x.grabbed_at DESC LIMIT 1) + WHERE CASE sqlc.arg(state)::text + WHEN 'all' THEN true + WHEN 'held' THEN w.fulfilled_at IS NOT NULL + WHEN 'upgrade' THEN w.fulfilled_at IS NOT NULL AND w.upgrade <> '' + WHEN 'missing' THEN w.fulfilled_at IS NULL AND w.reason = 'missing' + ELSE w.fulfilled_at IS NULL AND w.reason <> 'missing' + END + AND (sqlc.arg(artist_mbid)::text = '' OR w.artist_mbid = sqlc.arg(artist_mbid)::text) + -- want_ids narrows to given wants: the Missing page's folders. + AND (COALESCE(cardinality(sqlc.arg(want_ids)::uuid[]), 0) = 0 OR w.id = ANY(sqlc.arg(want_ids)::uuid[])) + AND (sqlc.arg(q)::text = '' + OR strpos(lower(w.artist_name), lower(sqlc.arg(q)::text)) > 0 + OR strpos(lower(w.title), lower(sqlc.arg(q)::text)) > 0) + ORDER BY lower(COALESCE(NULLIF(la.sort_name, ''), w.artist_name)), w.artist_mbid, + NULLIF(COALESCE(NULLIF(w.first_release_date, ''), g.first_release_date, ''), '') NULLS LAST, + lower(w.title), w.id + LIMIT sqlc.arg(max_rows) OFFSET sqlc.arg(skip); + +-- name: ListWantArtists :many +-- The Wanted view's main page: every artist on the list, A–Z like +-- the library, with how many of their albums are held of how many are on +-- the list. q matches the artist's name or any of their albums' titles; +-- artist_mbid narrows to one artist, for their page's header. +WITH a AS ( + SELECT w.artist_mbid, + (array_agg(w.artist_name ORDER BY w.wanted_at DESC))[1] AS artist_name, + COUNT(*) FILTER (WHERE w.fulfilled_at IS NOT NULL)::int AS held, + COUNT(*) FILTER (WHERE w.fulfilled_at IS NULL AND w.reason <> 'missing')::int AS wanted, + COUNT(*) FILTER (WHERE w.fulfilled_at IS NULL AND w.reason = 'missing')::int AS missing, + COUNT(*)::int AS albums, + bool_or(sqlc.arg(q)::text <> '' AND strpos(lower(w.title), lower(sqlc.arg(q)::text)) > 0) AS title_match + FROM wants w + WHERE sqlc.arg(artist_mbid)::text = '' OR w.artist_mbid = sqlc.arg(artist_mbid)::text + GROUP BY w.artist_mbid +) +SELECT a.artist_mbid, a.artist_name::text AS artist_name, a.held, a.wanted, a.missing, a.albums, + COALESCE(NULLIF(la.sort_name, ''), a.artist_name)::text AS sort_name, + la.id AS library_artist_id, + (f.artist_mbid IS NOT NULL)::bool AS following, + (SELECT COUNT(*) FROM grabs g JOIN wants x ON x.id = g.want_id + WHERE x.artist_mbid = a.artist_mbid AND g.ended_at IS NULL)::int AS downloading, + COUNT(*) OVER () AS total + FROM a + LEFT JOIN LATERAL (SELECT ar.id, ar.sort_name FROM artists ar WHERE ar.mbid = a.artist_mbid LIMIT 1) la ON true + LEFT JOIN follows f ON f.artist_mbid = a.artist_mbid + WHERE sqlc.arg(q)::text = '' + OR strpos(lower(a.artist_name), lower(sqlc.arg(q)::text)) > 0 + OR a.title_match + ORDER BY lower(COALESCE(NULLIF(la.sort_name, ''), a.artist_name)), a.artist_mbid + LIMIT sqlc.arg(max_rows) OFFSET sqlc.arg(skip); + +-- name: GetWantAlbum :one +-- What the Wanted view's track list needs of a want. +SELECT id, release_group_mbid, album_id FROM wants WHERE id = $1; + +-- name: ListLibraryAlbumTracks :many +-- A held or missing album's own tracks, for the Wanted view's track list; +-- missing says its file is gone. +SELECT title, disc_number, track_number, duration_ms, (missing_since IS NOT NULL)::bool AS missing + FROM tracks + WHERE album_id = $1 + ORDER BY disc_number NULLS FIRST, track_number NULLS LAST, title; + +-- name: CountWants :one +SELECT COUNT(*) FILTER (WHERE fulfilled_at IS NULL AND reason <> 'missing')::bigint AS wanted, + COUNT(*) FILTER (WHERE fulfilled_at IS NULL AND reason = 'missing')::bigint AS missing, + COUNT(*) FILTER (WHERE fulfilled_at IS NOT NULL)::bigint AS held, + -- Held albums wanted for a better copy. + COUNT(*) FILTER (WHERE fulfilled_at IS NOT NULL AND upgrade <> '')::bigint AS upgrades, + COUNT(DISTINCT artist_mbid)::bigint AS artists, + -- Library albums waiting to be matched by hand. + (SELECT COUNT(*) FROM library_album_matches m + WHERE m.outcome = 'unmatched' + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id))::bigint AS unmatched, + (SELECT COUNT(*) FROM follows)::bigint AS following, + -- The Wanted view's other tabs: downloads running, those + -- waiting on a person, and releases set aside. + (SELECT COUNT(*) FROM grabs WHERE ended_at IS NULL)::bigint AS activity, + (SELECT COUNT(*) FROM grabs + WHERE ended_at IS NULL AND import_state IN ('review', 'failed'))::bigint AS attention, + (SELECT COUNT(*) FROM release_blocks b + WHERE b.expires_at IS NULL OR b.expires_at > now())::bigint AS blocked + FROM wants; + +-- The one-time listing of the library. --------------------------- + +-- name: GetLibraryListing :one +SELECT library_listing_started_at, library_listed_at, library_unlisted + FROM acquisition_settings WHERE id = true; + +-- name: StartLibraryListing :one +-- Marks the listing begun, once; its time is the cutoff for which albums +-- count as the library's. +UPDATE acquisition_settings + SET library_listing_started_at = COALESCE(library_listing_started_at, now()) + WHERE id = true +RETURNING library_listing_started_at; + +-- name: ListLibraryAlbums :execrows +-- Puts library albums on the list as held: each release group once, by the +-- album holding the most tracks. Only albums the library had at the cutoff, +-- and with album_id given only that album. A group already on the list is +-- left as it is. An album with tracks already missing is held from just +-- before the first loss, so re-acquisition still fetches it back. +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, album_id, wanted_at, fulfilled_at) +SELECT DISTINCT ON (a.release_group_mbid) + a.release_group_mbid, ar.mbid, ar.name, a.title, 'library', a.id, now(), + COALESCE((SELECT MIN(t.missing_since) FROM tracks t WHERE t.album_id = a.id) - interval '1 second', now()) + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + WHERE a.release_group_mbid IS NOT NULL AND a.release_group_mbid <> '' + AND ar.mbid IS NOT NULL AND ar.mbid <> '' + AND a.created_at <= sqlc.arg(cutoff)::timestamptz + AND (sqlc.narg(album_id)::uuid IS NULL OR a.id = sqlc.narg(album_id)::uuid) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) + ORDER BY a.release_group_mbid, + (SELECT COUNT(*) FROM tracks t WHERE t.album_id = a.id AND t.missing_since IS NULL) DESC, a.id +ON CONFLICT (release_group_mbid) DO NOTHING; + +-- name: RecordListedAlbums :exec +-- The albums ListLibraryAlbums takes are decided as listed by their tags, +-- so the matching that follows leaves them alone. +INSERT INTO library_album_matches (album_id, outcome, method, release_group_mbid) +SELECT a.id, 'listed', 'tags', a.release_group_mbid + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + WHERE a.release_group_mbid IS NOT NULL AND a.release_group_mbid <> '' + AND ar.mbid IS NOT NULL AND ar.mbid <> '' + AND a.created_at <= sqlc.arg(cutoff)::timestamptz + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) +ON CONFLICT (album_id) DO NOTHING; + +-- name: ListAlbumsToMatch :many +-- Library albums the listing has not decided, or that an older matcher +-- refused, or whose tracks changed since they were refused: each is matched +-- to MusicBrainz. A library album is one from before the cutoff, or +-- one whose tracks were: a re-tag files a track under a new album row but +-- keeps the track. An album row with no tracks is not an album in +-- the library; one a person left off stays off. +SELECT a.id, a.title, a.mbid, a.release_group_mbid, ar.name AS artist_name, ar.mbid AS artist_mbid + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + WHERE (a.created_at <= sqlc.arg(cutoff)::timestamptz + OR EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id AND t.added_at <= sqlc.arg(cutoff)::timestamptz)) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) + AND NOT EXISTS (SELECT 1 FROM library_album_matches m + WHERE m.album_id = a.id + AND (m.outcome IN ('listed', 'dismissed') + OR (m.matcher >= sqlc.arg(matcher)::int + AND NOT EXISTS (SELECT 1 FROM tracks t + WHERE t.album_id = a.id AND t.updated_at > m.decided_at)))) + ORDER BY a.created_at, a.id + LIMIT sqlc.arg(max_rows); + +-- name: CountAlbumsToMatch :one +-- Whether a finished listing has albums to go back to: those +-- ListAlbumsToMatch would hand it. +SELECT COUNT(*) + FROM albums a + WHERE (a.created_at <= sqlc.arg(cutoff)::timestamptz + OR EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id AND t.added_at <= sqlc.arg(cutoff)::timestamptz)) + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id) + AND NOT EXISTS (SELECT 1 FROM library_album_matches m + WHERE m.album_id = a.id + AND (m.outcome IN ('listed', 'dismissed') + OR (m.matcher >= sqlc.arg(matcher)::int + AND NOT EXISTS (SELECT 1 FROM tracks t + WHERE t.album_id = a.id AND t.updated_at > m.decided_at)))); + +-- name: ListAlbumTracksToMatch :many +-- What an album's tracks say about themselves, to check a match against. +SELECT title, disc_number, track_number, duration_ms, mbid + FROM tracks + WHERE album_id = $1 + ORDER BY disc_number NULLS FIRST, track_number NULLS LAST, title; + +-- name: SetAlbumReleaseGroup :exec +-- Names a matched album's release group, filling only what nothing named. +UPDATE albums SET release_group_mbid = sqlc.arg(release_group_mbid), updated_at = now() + WHERE id = sqlc.arg(id) AND (release_group_mbid IS NULL OR release_group_mbid = ''); + +-- name: InsertLibraryWant :execrows +-- Puts one matched album on the list as held, from just before its first +-- lost track if it has one. A group already on the list is left as it is. +INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, primary_type, first_release_date, + reason, album_id, wanted_at, fulfilled_at) +VALUES (sqlc.arg(release_group_mbid), sqlc.arg(artist_mbid), sqlc.arg(artist_name), sqlc.arg(title), + sqlc.arg(primary_type), sqlc.arg(first_release_date), 'library', sqlc.arg(album_id)::uuid, now(), + COALESCE((SELECT MIN(t.missing_since) FROM tracks t WHERE t.album_id = sqlc.arg(album_id)::uuid) + - interval '1 second', now())) +ON CONFLICT (release_group_mbid) DO NOTHING; + +-- name: RecordAlbumMatch :exec +INSERT INTO library_album_matches (album_id, outcome, method, release_group_mbid, detail, matcher) +VALUES (sqlc.arg(album_id), sqlc.arg(outcome), sqlc.arg(method), sqlc.arg(release_group_mbid), sqlc.arg(detail), + sqlc.arg(matcher)) +ON CONFLICT (album_id) DO UPDATE + SET outcome = EXCLUDED.outcome, method = EXCLUDED.method, release_group_mbid = EXCLUDED.release_group_mbid, + detail = EXCLUDED.detail, matcher = EXCLUDED.matcher, decided_at = now(); + +-- name: FinishLibraryListing :one +-- The listing is done, or done again after a newer matcher's retries: +-- count the albums no match was found for. The first finish is kept. +UPDATE acquisition_settings + SET library_listed_at = COALESCE(library_listed_at, now()), + library_unlisted = (SELECT COUNT(*) FROM library_album_matches m + WHERE m.outcome = 'unmatched' + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id)) + WHERE id = true +RETURNING library_listed_at, library_unlisted; + +-- name: ListUnmatchedLibraryAlbums :many +-- The Collection's Unmatched section: the albums the listing could +-- not match, and below them the ones a person left off. An album whose +-- tracks have all moved to another row is gone, not unmatched. +SELECT m.album_id, a.title, ar.name AS artist_name, m.detail, m.outcome, m.decided_at, + (SELECT COUNT(*) FROM tracks t WHERE t.album_id = a.id AND t.missing_since IS NULL)::int AS tracks + FROM library_album_matches m + JOIN albums a ON a.id = m.album_id + JOIN artists ar ON ar.id = a.artist_id + WHERE m.outcome IN ('unmatched', 'dismissed') + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id) + ORDER BY m.outcome DESC, lower(ar.name), lower(a.title) + LIMIT sqlc.arg(max_rows); + +-- name: RecountLibraryUnlisted :exec +-- After a person matches or leaves off an album, the count the catalogue +-- shows. +UPDATE acquisition_settings + SET library_unlisted = (SELECT COUNT(*) FROM library_album_matches m + WHERE m.outcome = 'unmatched' + AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = m.album_id)) + WHERE id = true; + +-- name: GetAlbumToMatch :one +-- One album a person is matching by hand, as the matcher reads albums. +SELECT a.id, a.title, a.mbid, a.release_group_mbid, ar.name AS artist_name, ar.mbid AS artist_mbid + FROM albums a + JOIN artists ar ON ar.id = a.artist_id + JOIN library_album_matches m ON m.album_id = a.id AND m.outcome IN ('unmatched', 'dismissed') + WHERE a.id = $1; + +-- name: SetAlbumMatchOutcome :execrows +-- Leave an unmatched album off the list, or put a left-off one back. +UPDATE library_album_matches SET outcome = sqlc.arg(outcome), decided_at = now() + WHERE album_id = sqlc.arg(album_id) AND outcome = sqlc.arg(was); + + +-- Upgrades. ------------------------------------------------------- + +-- name: ListHeldWantFormats :many +-- What each held album's copy is made of, for judging whether it is worth +-- upgrading: its present tracks grouped by format, with their bytes and +-- length summed, and how many are held back as video rips. A held want whose +-- album has no track with a length yet has no rows. +SELECT w.id AS want_id, w.upgrade, w.held_tier, w.held_quality, + lower(t.file_format)::text AS file_format, + SUM(t.file_size)::bigint AS bytes, + SUM(t.duration_ms)::bigint AS duration_ms, + COUNT(*)::int AS tracks, + COUNT(*) FILTER (WHERE t.source_verdict = 'suspect')::int AS rips + FROM wants w + JOIN tracks t ON t.album_id = w.album_id + WHERE w.fulfilled_at IS NOT NULL + AND t.missing_since IS NULL AND t.duration_ms > 0 + GROUP BY w.id, lower(t.file_format) + ORDER BY w.id; + +-- name: SetWantUpgrades :exec +-- Writes the upgrade check's verdicts, one row per want. +UPDATE wants w + SET upgrade = v.upgrade, + held_tier = v.held_tier, + held_quality = v.held_quality, + upgrade_checked_at = now(), + updated_at = CASE WHEN w.upgrade <> v.upgrade THEN now() ELSE w.updated_at END + FROM (SELECT unnest(sqlc.arg(ids)::uuid[]) AS id, + unnest(sqlc.arg(upgrades)::text[]) AS upgrade, + unnest(sqlc.arg(held_tiers)::text[]) AS held_tier, + unnest(sqlc.arg(held_qualities)::text[]) AS held_quality) v + WHERE w.id = v.id; + +-- name: ClearHeldUpgrades :execrows +-- A held want whose album has no track left to judge is not wanted again. +UPDATE wants SET upgrade = '', held_tier = '', held_quality = '', upgrade_checked_at = now() + WHERE fulfilled_at IS NOT NULL AND upgrade <> '' + AND NOT EXISTS (SELECT 1 FROM tracks t + WHERE t.album_id = wants.album_id AND t.missing_since IS NULL AND t.duration_ms > 0); diff --git a/internal/db/queries/you_might_like.sql b/internal/db/queries/you_might_like.sql index 5b4697ab..62c2022d 100644 --- a/internal/db/queries/you_might_like.sql +++ b/internal/db/queries/you_might_like.sql @@ -1,4 +1,4 @@ --- "You might like" Home rows (#790). The daily build computes ranked +-- "You might like" Home rows. The daily build computes ranked -- album/artist IDs in Go (similarity roll-up + cold-start gate) and -- atomic-replaces them here; /api/home reads them back hydrated. diff --git a/internal/dbtest/reset.go b/internal/dbtest/reset.go index 2d80157a..de71db8d 100644 --- a/internal/dbtest/reset.go +++ b/internal/dbtest/reset.go @@ -1,10 +1,9 @@ // Package dbtest provides shared helpers for integration tests that need -// a clean Postgres state without disturbing the operator's admin user. +// a clean Postgres state without disturbing the developer's admin user. // -// Background: when integration tests run against the same Postgres -// instance an operator uses for local dev (a common dev-laptop setup), -// a TRUNCATE that includes the users table wipes the operator's admin -// login between every test run. To avoid that, tests should: +// When integration tests run against the same Postgres a developer uses +// for local dev, a TRUNCATE that includes the users table wipes their admin +// login on every test run. To avoid that, tests should: // // 1. Call ResetDB instead of issuing TRUNCATE statements directly. It // truncates every data table EXCEPT users, then deletes only those @@ -40,8 +39,8 @@ const TestUserPrefix = "test-" var dataTables = []string{ "artist_similarity", "track_similarity", - "artist_similarity_unmatched", // M5c - "listenbrainz_similar_recordings", // #5296 + "artist_similarity_unmatched", + "listenbrainz_similar_recordings", "track_similarity_fetches", "artist_similarity_fetches", "scrobble_queue", @@ -53,15 +52,15 @@ var dataTables = []string{ "skip_events", "play_sessions", "sessions", - "lidarr_quarantine_actions", - "lidarr_quarantine", - // #2374. Keyed by (user_id, candidate_mbid) with no FK to artists — + "quarantine_actions", + "quarantine", + // Keyed by (user_id, candidate_mbid) with no FK to artists — // candidates are out-of-library — so the CASCADE from artists/users // does NOT reach it for a leftover row whose user survived. Truncate // explicitly or a stale snooze silently hides a candidate from the // next test's suggestion assertions. "suggestion_snoozes", - // #2376. Same reasoning: keyed by candidate MBID with no FK anywhere, + // Same reasoning: keyed by candidate MBID with no FK anywhere, // so nothing cascades to them. A leftover tag row would make a // candidate look enriched to the next test, and a leftover state row // would make it look already-settled and thus ineligible. @@ -69,7 +68,7 @@ var dataTables = []string{ "candidate_artist_tag_state", "playlist_tracks", "playlists", - "library_changes", // M7 #357 — must reset to keep cursor isolated per test + "library_changes", // reset so each test's sync cursor starts clean // SettingsService.reconcile() idempotently re-UpsertProviderSettings // for every registered provider at boot, so truncating this is the // correct per-test reset (clears test-modified enabled/api_key rows). @@ -77,31 +76,46 @@ var dataTables = []string{ // (never recreates the singleton, seeded once by 0018); ResetDB // resets its counter via UPDATE below instead. "cover_art_provider_settings", - // Same reasoning for the tag-enrichment settings (#1490): truncate the + // Same reasoning for the tag-enrichment settings: truncate the // per-provider rows (reconcile re-seeds registered ones), and reset the // tag_sources_meta counter via UPDATE below rather than truncating it. "tag_provider_settings", - // recsettings.New reconciles shipped defaults on every construction - // (#1250), so truncating gives each test pristine tuning values. + // recsettings.New reconciles shipped defaults on every construction, + // so truncating gives each test pristine tuning values. "recommendation_weight_profiles", "taste_tuning", - // #2377. Same reasoning as taste_tuning above: recsettings.New re-seeds + // Same reasoning as taste_tuning above: recsettings.New re-seeds // shipped defaults on every construction, so truncating gives each test // pristine Discover knobs rather than whatever a previous test tuned. "discover_tuning", "recommendation_tuning_audit", - "duplicate_group_members", // M400 + "duplicate_group_members", "duplicate_groups", "duplicate_sweeps", - "track_fingerprints", // M400 - "track_loudness", // M464 - "album_loudness", // M464 - "user_normalization_prefs", // M464 - // M489. Both cascade from users, but the operator's own admin row is + "track_fingerprints", + "track_loudness", + "album_loudness", + "user_normalization_prefs", + // Both cascade from users, but the developer's own admin row is // never deleted, so rows a test wrote for it would otherwise survive. "user_notifications", "user_notification_prefs", - "user_notification_email_state", // #5346 + "user_notification_email_state", + "acquisition_services", + "mb_release_tracks", + "mb_releases", + "mb_artist_release_groups", + "mb_release_groups", + "mb_artists", + "requests", + "download_interventions", + "library_album_matches", + "release_blocks", + "grabs", + "want_candidates", + "wants", + "follow_seen_groups", + "follows", "tracks", "albums", "artists", @@ -109,7 +123,7 @@ var dataTables = []string{ // ResetDB clears all data tables and removes any user whose username // begins with TestUserPrefix. Users without that prefix (notably the -// operator's admin row) are left alone. Calls t.Fatalf on error. +// developer's admin row) are left alone. Calls t.Fatalf on error. func ResetDB(t *testing.T, pool *pgxpool.Pool) { t.Helper() ctx := context.Background() @@ -133,7 +147,7 @@ func ResetDB(t *testing.T, pool *pgxpool.Pool) { ); err != nil { t.Fatalf("dbtest.ResetDB reset cover-art version: %v", err) } - // Same for the tag-sources counter (#1490) — clears cross-test version + // Same for the tag-sources counter — clears cross-test version // accumulation that would spuriously report version_bumped on a // key-only change. if _, err := pool.Exec(ctx, @@ -141,7 +155,7 @@ func ResetDB(t *testing.T, pool *pgxpool.Pool) { ); err != nil { t.Fatalf("dbtest.ResetDB reset tag-sources version: %v", err) } - // Fingerprinting settings (M400 #3913), a singleton like the counters above. + // Fingerprinting settings, a singleton like the counters above. // Every column goes back to its migration default rather than to literals // written here, so a test can pin the Go defaults to the migration's. if _, err := pool.Exec(ctx, ` @@ -152,14 +166,37 @@ func ResetDB(t *testing.T, pool *pgxpool.Pool) { ); err != nil { t.Fatalf("dbtest.ResetDB reset fingerprint settings: %v", err) } - // Loudness analysis settings (M464 #4995), reset the same way. + // Loudness analysis settings, reset the same way. if _, err := pool.Exec(ctx, ` UPDATE loudness_settings SET enabled = DEFAULT, backfill_concurrency = DEFAULT, updated_at = DEFAULT`, ); err != nil { t.Fatalf("dbtest.ResetDB reset loudness settings: %v", err) } - // Notification email settings (M489 #5346), reset the same way. + // Release-type profiles: migration 0081 seeds "Standard" as + // the default. Tests add and edit profiles, so put back exactly the + // seeded row rather than truncating it away. + if _, err := pool.Exec(ctx, ` + DELETE FROM release_type_profiles WHERE name <> 'Standard'; + UPDATE release_type_profiles + SET primary_types = ARRAY['Album', 'EP', 'Single'], secondary_types = '{}', is_default = true + WHERE name = 'Standard'`, + ); err != nil { + t.Fatalf("dbtest.ResetDB reset release-type profiles: %v", err) + } + // Search and grab settings, a singleton reset the same way. + // The library listing reads as done, so a test's albums are not + // put on the wanted list behind its back; its own tests clear it. + if _, err := pool.Exec(ctx, ` + UPDATE acquisition_settings + SET auto_grab = DEFAULT, search_interval_hours = DEFAULT, allowed_qualities = DEFAULT, import_root = DEFAULT, + seed_ratio = DEFAULT, seed_days = DEFAULT, stall_hours = DEFAULT, upgrade_until = DEFAULT, + library_listing_started_at = now(), library_listed_at = now(), library_unlisted = 0, + last_pass_at = NULL, last_pass = DEFAULT, updated_at = DEFAULT`, + ); err != nil { + t.Fatalf("dbtest.ResetDB reset acquisition settings: %v", err) + } + // Notification email settings, reset the same way. if _, err := pool.Exec(ctx, ` UPDATE notification_email_settings SET summary_hour = DEFAULT, batch_window_minutes = DEFAULT, updated_at = DEFAULT`, diff --git a/internal/downloader/downloader_test.go b/internal/downloader/downloader_test.go new file mode 100644 index 00000000..5af62b57 --- /dev/null +++ b/internal/downloader/downloader_test.go @@ -0,0 +1,156 @@ +package downloader + +import ( + "context" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" +) + +func TestPathMap(t *testing.T) { + m := PathMap{Local: "/downloads", Remote: "/data/torrents/"} + for in, want := range map[string]string{ + "/downloads": "/data/torrents", + "/downloads/abc/x.mp3": "/data/torrents/abc/x.mp3", + "/downloads2/abc": "/downloads2/abc", + "/music/a": "/music/a", + "": "", + } { + if got := m.ToRemote(in); got != want { + t.Errorf("ToRemote(%q) = %q, want %q", in, got, want) + } + } + if got := m.ToLocal("/data/torrents/abc"); got != "/downloads/abc" { + t.Errorf("ToLocal = %q", got) + } + if (PathMap{Local: "/downloads", Remote: "/downloads/"}).Active() { + t.Error("the same folder under one name is not a mapping") + } +} + +// fakeQbit is a qBittorrent WebUI that wants a login, and that expires the +// session once to make the client log in again. +func fakeQbit(t *testing.T, expireOnce bool) (*httptest.Server, *atomic.Int32) { + t.Helper() + var logins atomic.Int32 + var expired atomic.Bool + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Referer") == "" { + w.WriteHeader(http.StatusForbidden) + return + } + switch r.URL.Path { + case "/api/v2/auth/login": + _ = r.ParseForm() + if r.Form.Get("password") != "pw" { + _, _ = w.Write([]byte("Fails.")) + return + } + logins.Add(1) + http.SetCookie(w, &http.Cookie{Name: "SID", Value: "s", Path: "/"}) + _, _ = w.Write([]byte("Ok.")) + case "/api/v2/torrents/info": + if _, err := r.Cookie("SID"); err != nil { + w.WriteHeader(http.StatusForbidden) + return + } + if expireOnce && !expired.Swap(true) { + w.WriteHeader(http.StatusForbidden) + return + } + if r.URL.Query().Get("category") != "minstrel" { + t.Errorf("category = %q", r.URL.Query().Get("category")) + } + _, _ = w.Write([]byte(`[{"hash": "ABC", "name": "x", "save_path": "/data/torrents/abc", "progress": 1}]`)) + case "/api/v2/torrents/add": + body, _ := io.ReadAll(r.Body) + if !strings.Contains(string(body), "/data/torrents/new") { + t.Errorf("add body has no translated save path: %s", body) + } + _, _ = w.Write([]byte("Ok.")) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + return srv, &logins +} + +func TestQBittorrent_LogsInAndLogsInAgainWhenTheSessionExpires(t *testing.T) { + srv, logins := fakeQbit(t, true) + d := Mapped(&Client{Config: Config{BaseURL: srv.URL, Username: "u", Password: "pw", Category: "minstrel"}}, + PathMap{Local: "/downloads", Remote: "/data/torrents"}) + ts, err := d.Torrents(context.Background()) + if err != nil { + t.Fatalf("torrents: %v", err) + } + if logins.Load() != 2 { + t.Errorf("logins = %d, want 2 (the second after the 403)", logins.Load()) + } + if len(ts) != 1 || ts[0].Hash != "abc" || ts[0].SavePath != "/downloads/abc" || !ts[0].Done() { + t.Errorf("torrents = %+v, want one, hash lowercased, path in Minstrel's terms", ts) + } + if err := d.AddTorrent(context.Background(), Add{Magnet: "magnet:?xt=urn:btih:abc", SavePath: "/downloads/new"}); err != nil { + t.Errorf("add: %v", err) + } +} + +func TestQBittorrent_AWrongPasswordIsAuth(t *testing.T) { + srv, _ := fakeQbit(t, false) + c := &Client{Config: Config{BaseURL: srv.URL, Password: "nope"}} + if _, err := c.Torrents(context.Background()); !errors.Is(err, ErrAuth) { + t.Fatalf("err = %v, want ErrAuth", err) + } +} + +func TestExchange_MapsDownloadsAndRefusesWhileRestoring(t *testing.T) { + var restoring, sentKey atomic.Bool + restoring.Store(true) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "" { + sentKey.Store(true) + } + pending := "0" + if restoring.Load() { + pending = "3" + } + _, _ = io.WriteString(w, `{"enabled": true, "running": false, "held_reason": "disk full", "restore_pending": `+pending+`, "downloads": [ + {"infohash": "ABC", "name": "Album", "save_path": "/dl", "state": "seeding", "progress": 1, "length": 10, "bytes": 10, "category": "minstrel"}, + {"infohash": "DEF", "name": "Other", "save_path": "/dl", "state": "fast", "category": "someone-else"}, + {"infohash": "123", "name": "Broken", "save_path": "/dl", "failed": true, "fail_reason": "tracker gone", "category": "minstrel"} + ]}`) + })) + defer srv.Close() + + c := &Exchange{Config: Config{BaseURL: srv.URL, Category: "minstrel"}} + if _, err := c.Torrents(context.Background()); !errors.Is(err, ErrRestoring) { + t.Fatalf("while restoring: err = %v, want ErrRestoring", err) + } + if sentKey.Load() { + t.Error("an Authorization header was sent with no key set") + } + restoring.Store(false) + ts, err := c.Torrents(context.Background()) + if err != nil { + t.Fatalf("torrents: %v", err) + } + if len(ts) != 2 { + t.Fatalf("torrents = %+v, want the two in Minstrel's category", ts) + } + if ts[0].Hash != "abc" || ts[0].State != "uploading" || ts[0].ContentPath != "/dl/Album" || !strings.Contains(ts[0].EngineDown, "disk full") { + t.Errorf("first = %+v", ts[0]) + } + if !ts[1].Failed() || ts[1].FailReason != "tracker gone" { + t.Errorf("second = %+v, want failed with Exchange's reason", ts[1]) + } +} + +func TestFor_RefusesAnUnknownProtocol(t *testing.T) { + if _, err := For("transmission", Config{}); err == nil { + t.Fatal("want an error for a client Minstrel cannot drive") + } +} diff --git a/internal/downloader/driver.go b/internal/downloader/driver.go new file mode 100644 index 00000000..2b1d8d99 --- /dev/null +++ b/internal/downloader/driver.go @@ -0,0 +1,197 @@ +// Package downloader drives the download clients Minstrel hands a grab to: +// qBittorrent's WebUI v2, and FabledExchange's native /api/downloads. +// +// Both answer through one Driver, and Torrent is qBittorrent's shape: the +// Exchange driver maps its own downloads onto it, so the grab, the importer +// and the stall watchdog read one kind of record whichever client holds the +// torrent. +package downloader + +import ( + "context" + "errors" + "fmt" + "time" +) + +// Protocols a download client row may name. +const ( + KindQBittorrent = "qbittorrent" + KindExchange = "exchange" +) + +// ErrUnconfigured is a client with no address. +var ErrUnconfigured = errors.New("download client is not configured") + +// ErrUnreachable is a call that could not be made, or a refusal that is not +// about credentials. +var ErrUnreachable = errors.New("download client unreachable") + +// ErrAuth is credentials the client refused. +var ErrAuth = errors.New("download client refused the credentials") + +// ErrRestoring is an Exchange still bringing its downloads back after a +// restart: its list is incomplete, so nothing may be concluded from it. +var ErrRestoring = errors.New("download client is still restoring its downloads after a restart") + +// ErrDuplicate is a torrent the client already holds. +var ErrDuplicate = errors.New("download client already holds this torrent") + +// ErrUnsupported is a call this client has no equivalent for. +var ErrUnsupported = errors.New("download client cannot do that") + +// UserAgent names Minstrel in the client's logs. +const UserAgent = "Minstrel (self-hosted music server; download client)" + +// DefaultTimeout is what an unset Timeout means: a download client answers +// from its own state. +const DefaultTimeout = 15 * time.Second + +// Config is one download client. +type Config struct { + BaseURL string + // Username and Password are qBittorrent's. Both may be empty for a WebUI + // that bypasses auth for local addresses. + Username string + Password string + // APIKey is Exchange's, sent as a bearer only when set. + APIKey string + // Category files everything Minstrel adds, so its torrents can be told + // from anything else in the client. + Category string + Timeout time.Duration +} + +// Driver is what Minstrel needs from a download client. +type Driver interface { + AddTorrent(ctx context.Context, a Add) error + // Torrents lists this client's torrents in Config.Category, or only the + // given hashes. + Torrents(ctx context.Context, hashes ...string) ([]Torrent, error) + // Connected says whether the client can reach any peer at all. + Connected(ctx context.Context) (bool, error) + Stop(ctx context.Context, hashes ...string) error + Start(ctx context.Context, hashes ...string) error + Reannounce(ctx context.Context, hashes ...string) error + Recheck(ctx context.Context, hashes ...string) error + Delete(ctx context.Context, deleteFiles bool, hashes ...string) error +} + +// FileSkipper sets which files of a torrent are downloaded. +type FileSkipper interface { + SkipFiles(ctx context.Context, hash string, indices []int) error + KeepFiles(ctx context.Context, hash string, indices []int) error +} + +// File is one file inside a torrent. +type File struct { + Index int + Path string + Size int64 +} + +// FileLister lists a torrent's files. +type FileLister interface { + Files(ctx context.Context, hash string) ([]File, error) +} + +// FileState is one file's progress. +type FileState struct { + Path string + Size int64 + Done int64 + Skipped bool +} + +// Unfinished says whether the file is wanted and not yet whole. +func (f FileState) Unfinished() bool { return !f.Skipped && f.Done < f.Size } + +// FileProgress reports per-file progress. +type FileProgress interface { + FileStates(ctx context.Context, hash string) ([]FileState, error) +} + +var ( + _ Driver = (*Client)(nil) + _ Driver = (*Exchange)(nil) + _ FileSkipper = (*Client)(nil) + _ FileSkipper = (*Exchange)(nil) + _ FileSkipper = (*mapped)(nil) + _ FileLister = (*Client)(nil) + _ FileLister = (*Exchange)(nil) + _ FileLister = (*mapped)(nil) + _ FileProgress = (*Client)(nil) + _ FileProgress = (*Exchange)(nil) + _ FileProgress = (*mapped)(nil) +) + +// For builds the driver a protocol names. +func For(protocol string, cfg Config) (Driver, error) { + switch protocol { + case KindQBittorrent: + return &Client{Config: cfg}, nil + case KindExchange: + return &Exchange{Config: cfg}, nil + } + return nil, fmt.Errorf("download client %q is not one Minstrel can drive", protocol) +} + +// Torrent is one torrent in qBittorrent's JSON shape. +type Torrent struct { + Hash string `json:"hash"` + Name string `json:"name"` + Size int64 `json:"size"` + Progress float64 `json:"progress"` + State string `json:"state"` + Category string `json:"category"` + Tags string `json:"tags"` + SavePath string `json:"save_path"` + ContentPath string `json:"content_path"` + AddedOn int64 `json:"added_on"` + NumSeeds int `json:"num_seeds"` + NumLeechs int `json:"num_leechs"` + AmountLeft int64 `json:"amount_left"` + Ratio float64 `json:"ratio"` + DLSpeed int64 `json:"dlspeed"` + ETA int64 `json:"eta"` + // EngineDown is why the client's whole engine is not moving anything, in + // words; empty when it is running. Exchange says so; qBittorrent does not. + EngineDown string `json:"-"` + // FailReason is the client's own reason for a failed torrent. + FailReason string `json:"-"` +} + +// Done says the torrent is whole. +func (t Torrent) Done() bool { return t.Progress >= 1 } + +// Failed says the client gave up on it. +func (t Torrent) Failed() bool { return t.State == "error" || t.State == "missingFiles" } + +// Fetching says it is still fetching metadata. +func (t Torrent) Fetching() bool { return t.State == "metaDL" || t.State == "forcedMetaDL" } + +// Alone says nobody is sharing an unfinished torrent. +func (t Torrent) Alone() bool { return !t.Done() && t.NumSeeds == 0 && t.NumLeechs == 0 } + +// Added is when the client took it. +func (t Torrent) Added() time.Time { + if t.AddedOn <= 0 { + return time.Time{} + } + return time.Unix(t.AddedOn, 0) +} + +// Add is one torrent to hand a client: a magnet or the .torrent bytes. +type Add struct { + Magnet string + Torrent []byte + // SavePath is where it lands, as Minstrel sees it; Mapped translates it. + // Required: a download with no save path lands outside the folder + // Minstrel imports from. + SavePath string + Tags []string + Paused bool + // StopAtMetadata asks qBittorrent to stop once it has the file list, so + // unwanted files can be skipped before any data moves. + StopAtMetadata bool +} diff --git a/internal/downloader/exchange.go b/internal/downloader/exchange.go new file mode 100644 index 00000000..e1b04fd0 --- /dev/null +++ b/internal/downloader/exchange.go @@ -0,0 +1,340 @@ +package downloader + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + "path" + "strings" + "time" +) + +// Exchange drives FabledExchange's native /api/downloads: not its qBittorrent +// facade, so its own states, failure reasons and engine status come through. +type Exchange struct { + Config Config + // HTTP is injectable for tests. Nil takes a plain client. + HTTP *http.Client +} + +type exchangeItem struct { + InfoHash string `json:"infohash"` + Name string `json:"name"` + Bytes int64 `json:"bytes"` + Length int64 `json:"length"` + Progress float64 `json:"progress"` + State string `json:"state"` + Peers int `json:"peers"` + Category string `json:"category"` + Tags []string `json:"tags"` + SavePath string `json:"save_path"` + AddedAt time.Time `json:"added_at"` + Failed bool `json:"failed"` + FailReason string `json:"fail_reason"` + Ratio float64 `json:"ratio"` +} + +type exchangeSnapshot struct { + Enabled bool `json:"enabled"` + Running bool `json:"running"` + HeldReason string `json:"held_reason"` + InboundConns int64 `json:"inbound_conns"` + RestorePending int `json:"restore_pending"` + Downloads []exchangeItem `json:"downloads"` +} + +func (c *Exchange) AddTorrent(ctx context.Context, a Add) error { + if a.Magnet == "" && len(a.Torrent) == 0 { + return fmt.Errorf("add: neither a magnet nor torrent bytes") + } + if a.SavePath == "" { + return fmt.Errorf("add: no save path, which would land the download outside the folder Minstrel imports from") + } + if len(a.Torrent) > 0 { + q := url.Values{"save_path": {a.SavePath}} + if c.Config.Category != "" { + q.Set("category", c.Config.Category) + } + for _, t := range a.Tags { + q.Add("tag", t) + } + if a.Paused { + q.Set("paused", "true") + } + _, err := c.call(ctx, http.MethodPost, "/api/downloads/torrent?"+q.Encode(), "application/x-bittorrent", a.Torrent) + return err + } + body, err := json.Marshal(map[string]any{ + "magnet": a.Magnet, "category": c.Config.Category, "save_path": a.SavePath, + "tags": a.Tags, "paused": a.Paused, + }) + if err != nil { + return fmt.Errorf("add: %w", err) + } + _, err = c.call(ctx, http.MethodPost, "/api/downloads", "application/json", body) + return err +} + +// Torrents maps Exchange's downloads onto qBittorrent's shape. While Exchange +// is still restoring after a restart its list is incomplete, so it answers +// ErrRestoring rather than a list that looks like downloads vanished. +func (c *Exchange) Torrents(ctx context.Context, hashes ...string) ([]Torrent, error) { + snap, err := c.snapshot(ctx) + if err != nil { + return nil, err + } + if snap.RestorePending > 0 { + return nil, fmt.Errorf("%w (%d still to come back)", ErrRestoring, snap.RestorePending) + } + want := map[string]bool{} + for _, h := range hashes { + want[strings.ToLower(strings.TrimSpace(h))] = true + } + down := "" + switch { + case snap.Running: + case !snap.Enabled: + down = "the download engine is switched off in FabledExchange" + case snap.HeldReason != "": + down = "the download engine is held (" + snap.HeldReason + ") and resumes by itself" + default: + down = "the download engine is enabled but not running; FabledExchange is retrying it" + } + out := make([]Torrent, 0, len(snap.Downloads)) + for _, it := range snap.Downloads { + if c.Config.Category != "" && it.Category != c.Config.Category { + continue + } + t := asTorrent(it) + t.EngineDown = down + if len(want) > 0 && !want[t.Hash] { + continue + } + out = append(out, t) + } + return out, nil +} + +func asTorrent(it exchangeItem) Torrent { + content := it.SavePath + if it.Name != "" { + content = path.Join(it.SavePath, it.Name) + } + left := max(it.Length-it.Bytes, 0) + state := exchangeState(it.State, it.Progress) + if it.Failed { + state = "error" + } + var added int64 + if !it.AddedAt.IsZero() { + added = it.AddedAt.Unix() + } + return Torrent{ + Hash: strings.ToLower(it.InfoHash), Name: it.Name, Size: it.Length, + Progress: it.Progress, State: state, Category: it.Category, + Tags: strings.Join(it.Tags, ","), SavePath: it.SavePath, ContentPath: content, + AddedOn: added, NumLeechs: it.Peers, AmountLeft: left, + FailReason: it.FailReason, Ratio: it.Ratio, + } +} + +// exchangeState names Exchange's state in qBittorrent's vocabulary. +func exchangeState(tier string, progress float64) string { + switch tier { + case "fast", "slow": + return "downloading" + case "metadata": + return "metaDL" + case "stalled": + return "stalledDL" + case "queued": + return "queuedDL" + case "seeding": + return "uploading" + case "relocating": + return "moving" + case "paused": + if progress >= 1 { + return "pausedUP" + } + return "pausedDL" + } + return "unknown" +} + +// Connected is a running engine that has had an inbound connection: the +// nearest Exchange comes to qBittorrent's "connected". +func (c *Exchange) Connected(ctx context.Context) (bool, error) { + snap, err := c.snapshot(ctx) + if err != nil { + return false, err + } + return snap.Running && snap.InboundConns > 0, nil +} + +func (c *Exchange) Stop(ctx context.Context, hashes ...string) error { + return c.each(ctx, hashes, func(h string) error { + _, err := c.call(ctx, http.MethodPost, "/api/downloads/"+url.PathEscape(h)+"/pause", "", nil) + return err + }) +} + +func (c *Exchange) Start(ctx context.Context, hashes ...string) error { + return c.each(ctx, hashes, func(h string) error { + _, err := c.call(ctx, http.MethodPost, "/api/downloads/"+url.PathEscape(h)+"/resume", "", nil) + return err + }) +} + +func (c *Exchange) SkipFiles(ctx context.Context, hash string, indices []int) error { + return c.filePriority(ctx, hash, indices, "skip") +} + +func (c *Exchange) KeepFiles(ctx context.Context, hash string, indices []int) error { + return c.filePriority(ctx, hash, indices, "normal") +} + +func (c *Exchange) filePriority(ctx context.Context, hash string, indices []int, priority string) error { + if len(indices) == 0 { + return nil + } + body, err := json.Marshal(map[string]any{"indices": indices, "priority": priority}) + if err != nil { + return err + } + _, err = c.call(ctx, http.MethodPatch, "/api/downloads/"+url.PathEscape(hash)+"/files", "application/json", body) + return err +} + +func (c *Exchange) Files(ctx context.Context, hash string) ([]File, error) { + raw, err := c.files(ctx, hash) + if err != nil { + return nil, err + } + out := make([]File, len(raw)) + for i, f := range raw { + out[i] = File{Index: f.Index, Path: f.Path, Size: f.Length} + } + return out, nil +} + +func (c *Exchange) FileStates(ctx context.Context, hash string) ([]FileState, error) { + raw, err := c.files(ctx, hash) + if err != nil { + return nil, err + } + out := make([]FileState, len(raw)) + for i, f := range raw { + if f.Bytes == nil { + return nil, fmt.Errorf("%w: FabledExchange does not say how much of each file it has", ErrUnsupported) + } + out[i] = FileState{Path: f.Path, Size: f.Length, Done: *f.Bytes, Skipped: f.Priority == "skip"} + } + return out, nil +} + +type exchangeFile struct { + Index int `json:"index"` + Path string `json:"path"` + Length int64 `json:"length"` + Bytes *int64 `json:"bytes"` + Priority string `json:"priority"` +} + +func (c *Exchange) files(ctx context.Context, hash string) ([]exchangeFile, error) { + body, err := c.call(ctx, http.MethodGet, "/api/downloads/"+url.PathEscape(hash), "", nil) + if err != nil { + return nil, err + } + var detail struct { + Files []exchangeFile `json:"files"` + } + if err := json.Unmarshal(body, &detail); err != nil { + return nil, fmt.Errorf("%w: parse download detail: %v", ErrUnreachable, err) + } + return detail.Files, nil +} + +func (c *Exchange) Reannounce(context.Context, ...string) error { + return fmt.Errorf("%w: FabledExchange schedules its own announces", ErrUnsupported) +} + +func (c *Exchange) Recheck(context.Context, ...string) error { + return fmt.Errorf("%w: FabledExchange has no recheck", ErrUnsupported) +} + +func (c *Exchange) Delete(ctx context.Context, deleteFiles bool, hashes ...string) error { + return c.each(ctx, hashes, func(h string) error { + p := "/api/downloads/" + url.PathEscape(h) + if deleteFiles { + p += "?delete_files=true" + } + _, err := c.call(ctx, http.MethodDelete, p, "", nil) + return err + }) +} + +func (c *Exchange) each(ctx context.Context, hashes []string, do func(string) error) error { + for _, h := range hashes { + if ctx.Err() != nil { + return ctx.Err() + } + if err := do(strings.ToLower(strings.TrimSpace(h))); err != nil { + return err + } + } + return nil +} + +func (c *Exchange) snapshot(ctx context.Context) (exchangeSnapshot, error) { + var snap exchangeSnapshot + body, err := c.call(ctx, http.MethodGet, "/api/downloads", "", nil) + if err != nil { + return snap, err + } + if err := json.Unmarshal(body, &snap); err != nil { + return snap, fmt.Errorf("%w: parse download list: %v", ErrUnreachable, err) + } + return snap, nil +} + +func (c *Exchange) call(ctx context.Context, method, p, contentType string, body []byte) ([]byte, error) { + if c.Config.BaseURL == "" { + return nil, ErrUnconfigured + } + httpc := c.HTTP + if httpc == nil { + httpc = &http.Client{} + } + status, out, err := send(ctx, httpc, c.Config.Timeout, method, + strings.TrimRight(c.Config.BaseURL, "/")+p, contentType, body, func(h http.Header) { + h.Set("Accept", "application/json") + if c.Config.APIKey != "" { + h.Set("Authorization", "Bearer "+c.Config.APIKey) + } + }) + if err != nil { + return nil, err + } + if status >= 200 && status < 300 { + return out, nil + } + var e struct { + Error string `json:"error"` + Reason string `json:"reason"` + } + _ = json.Unmarshal(out, &e) + said := strings.TrimSpace(e.Error) + if said == "" { + said = fmt.Sprintf("HTTP %d", status) + } + switch { + case status == http.StatusUnauthorized || status == http.StatusForbidden: + return nil, fmt.Errorf("%w: FabledExchange refused the API key (%s)", ErrAuth, said) + case status == http.StatusConflict && e.Reason == "duplicate": + return nil, fmt.Errorf("%w: %s", ErrDuplicate, said) + } + return nil, fmt.Errorf("%w: %s", ErrUnreachable, said) +} diff --git a/internal/downloader/pathmap.go b/internal/downloader/pathmap.go new file mode 100644 index 00000000..be43772e --- /dev/null +++ b/internal/downloader/pathmap.go @@ -0,0 +1,107 @@ +package downloader + +import ( + "context" + "path" + "strings" +) + +// PathMap is one downloads folder under two names: where Minstrel has it +// mounted (Local) and where the client sees it (Remote). The two usually run +// in different containers. Every path crossing between them swaps one prefix +// for the other. +type PathMap struct { + Local string + Remote string +} + +// Active says the two names differ, so there is something to translate. +func (m PathMap) Active() bool { + return m.Remote != "" && m.Local != "" && path.Clean(m.Remote) != path.Clean(m.Local) +} + +// ToRemote is a Minstrel path as the client names it. +func (m PathMap) ToRemote(p string) string { return swap(p, m.Local, m.Remote) } + +// ToLocal is a client's path as Minstrel names it. +func (m PathMap) ToLocal(p string) string { return swap(p, m.Remote, m.Local) } + +// swap replaces the prefix from with to on segment boundaries +// (/downloads2 is not inside /downloads). A path outside from is unchanged. +func swap(p, from, to string) string { + if p == "" || from == "" || to == "" { + return p + } + from, to = path.Clean(from), path.Clean(to) + clean := path.Clean(p) + if clean == from { + return to + } + if from == "/" { + return path.Join(to, clean) + } + if rest, ok := strings.CutPrefix(clean, from+"/"); ok { + return path.Join(to, rest) + } + return p +} + +// Mapped wraps d so every path it is given or answers with is translated. +// File indices and in-torrent paths need no translation. +func Mapped(d Driver, m PathMap) Driver { + if !m.Active() { + return d + } + return &mapped{Driver: d, m: m} +} + +type mapped struct { + Driver + m PathMap +} + +func (c *mapped) AddTorrent(ctx context.Context, a Add) error { + a.SavePath = c.m.ToRemote(a.SavePath) + return c.Driver.AddTorrent(ctx, a) +} + +func (c *mapped) Torrents(ctx context.Context, hashes ...string) ([]Torrent, error) { + ts, err := c.Driver.Torrents(ctx, hashes...) + for i := range ts { + ts[i].SavePath = c.m.ToLocal(ts[i].SavePath) + ts[i].ContentPath = c.m.ToLocal(ts[i].ContentPath) + } + return ts, err +} + +func (c *mapped) SkipFiles(ctx context.Context, hash string, indices []int) error { + fs, ok := c.Driver.(FileSkipper) + if !ok { + return ErrUnsupported + } + return fs.SkipFiles(ctx, hash, indices) +} + +func (c *mapped) KeepFiles(ctx context.Context, hash string, indices []int) error { + fs, ok := c.Driver.(FileSkipper) + if !ok { + return ErrUnsupported + } + return fs.KeepFiles(ctx, hash, indices) +} + +func (c *mapped) Files(ctx context.Context, hash string) ([]File, error) { + fl, ok := c.Driver.(FileLister) + if !ok { + return nil, ErrUnsupported + } + return fl.Files(ctx, hash) +} + +func (c *mapped) FileStates(ctx context.Context, hash string) ([]FileState, error) { + fp, ok := c.Driver.(FileProgress) + if !ok { + return nil, ErrUnsupported + } + return fp.FileStates(ctx, hash) +} diff --git a/internal/downloader/qbittorrent.go b/internal/downloader/qbittorrent.go new file mode 100644 index 00000000..73d1eb6b --- /dev/null +++ b/internal/downloader/qbittorrent.go @@ -0,0 +1,367 @@ +package downloader + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "mime/multipart" + "net/http" + "net/http/cookiejar" + "net/url" + "strconv" + "strings" + "sync" + "time" +) + +// Client drives one qBittorrent through its WebUI API v2. +type Client struct { + Config Config + + mu sync.Mutex + // http carries the SID cookie once logged in; nil until then. + http *http.Client +} + +// AddTorrent hands qBittorrent a magnet or .torrent bytes, filed under the +// configured category with the save path already set. +func (c *Client) AddTorrent(ctx context.Context, a Add) error { + if a.Magnet == "" && len(a.Torrent) == 0 { + return fmt.Errorf("add: neither a magnet nor torrent bytes") + } + if a.SavePath == "" { + return fmt.Errorf("add: no save path, which would land the download outside the folder Minstrel imports from") + } + body := &bytes.Buffer{} + form := multipart.NewWriter(body) + fields := map[string]string{ + "savepath": a.SavePath, + "category": c.Config.Category, + // Both, because qBittorrent 5 renamed paused to stopped and an older + // one ignores the field it does not know. + "paused": strconv.FormatBool(a.Paused), + "stopped": strconv.FormatBool(a.Paused), + } + if len(a.Tags) > 0 { + fields["tags"] = strings.Join(a.Tags, ",") + } + if a.StopAtMetadata && !a.Paused { + fields["stopCondition"] = "MetadataReceived" + } + if a.Magnet != "" { + fields["urls"] = a.Magnet + } + for k, v := range fields { + if v == "" { + continue + } + if err := form.WriteField(k, v); err != nil { + return fmt.Errorf("add: build form: %w", err) + } + } + if len(a.Torrent) > 0 { + part, err := form.CreateFormFile("torrents", "release.torrent") + if err != nil { + return fmt.Errorf("add: build form: %w", err) + } + if _, err := part.Write(a.Torrent); err != nil { + return fmt.Errorf("add: build form: %w", err) + } + } + if err := form.Close(); err != nil { + return fmt.Errorf("add: build form: %w", err) + } + resp, err := c.post(ctx, "/api/v2/torrents/add", form.FormDataContentType(), body.Bytes()) + if err != nil { + return err + } + // qBittorrent refuses with HTTP 200 and the body "Fails.". + if strings.HasPrefix(strings.TrimSpace(string(resp)), "Fails") { + return fmt.Errorf("%w: the client refused the torrent", ErrUnreachable) + } + return nil +} + +// Torrents lists the configured category's torrents, or the given hashes. +func (c *Client) Torrents(ctx context.Context, hashes ...string) ([]Torrent, error) { + params := url.Values{} + if c.Config.Category != "" { + params.Set("category", c.Config.Category) + } + if len(hashes) > 0 { + params.Set("hashes", strings.Join(hashes, "|")) + } + body, err := c.get(ctx, "/api/v2/torrents/info", params) + if err != nil { + return nil, err + } + var out []Torrent + if err := json.Unmarshal(body, &out); err != nil { + return nil, fmt.Errorf("%w: parse torrent list: %v", ErrUnreachable, err) + } + for i := range out { + out[i].Hash = strings.ToLower(out[i].Hash) + } + return out, nil +} + +// Connected reads qBittorrent's own connection status. +func (c *Client) Connected(ctx context.Context) (bool, error) { + body, err := c.get(ctx, "/api/v2/transfer/info", nil) + if err != nil { + return false, err + } + var info struct { + ConnectionStatus string `json:"connection_status"` + } + if err := json.Unmarshal(body, &info); err != nil { + return false, fmt.Errorf("%w: parse transfer info: %v", ErrUnreachable, err) + } + return info.ConnectionStatus == "connected", nil +} + +// Stop and Start try qBittorrent 5's names first and fall back to 4's. +func (c *Client) Stop(ctx context.Context, hashes ...string) error { + return c.command(ctx, hashes, "/api/v2/torrents/stop", "/api/v2/torrents/pause") +} + +func (c *Client) Start(ctx context.Context, hashes ...string) error { + return c.command(ctx, hashes, "/api/v2/torrents/start", "/api/v2/torrents/resume") +} + +func (c *Client) Reannounce(ctx context.Context, hashes ...string) error { + return c.command(ctx, hashes, "/api/v2/torrents/reannounce", "/api/v2/torrents/reannounce") +} + +func (c *Client) Recheck(ctx context.Context, hashes ...string) error { + return c.command(ctx, hashes, "/api/v2/torrents/recheck", "/api/v2/torrents/recheck") +} + +func (c *Client) Delete(ctx context.Context, deleteFiles bool, hashes ...string) error { + if len(hashes) == 0 { + return nil + } + params := url.Values{ + "hashes": {strings.Join(hashes, "|")}, + "deleteFiles": {strconv.FormatBool(deleteFiles)}, + } + _, err := c.post(ctx, "/api/v2/torrents/delete", "application/x-www-form-urlencoded", []byte(params.Encode())) + return err +} + +func (c *Client) SkipFiles(ctx context.Context, hash string, indices []int) error { + return c.filePriority(ctx, hash, indices, "0") +} + +func (c *Client) KeepFiles(ctx context.Context, hash string, indices []int) error { + return c.filePriority(ctx, hash, indices, "1") +} + +func (c *Client) filePriority(ctx context.Context, hash string, indices []int, priority string) error { + if len(indices) == 0 { + return nil + } + ids := make([]string, len(indices)) + for i, n := range indices { + ids[i] = strconv.Itoa(n) + } + params := url.Values{"hash": {hash}, "id": {strings.Join(ids, "|")}, "priority": {priority}} + _, err := c.post(ctx, "/api/v2/torrents/filePrio", "application/x-www-form-urlencoded", []byte(params.Encode())) + return err +} + +func (c *Client) Files(ctx context.Context, hash string) ([]File, error) { + raw, err := c.files(ctx, hash) + if err != nil { + return nil, err + } + out := make([]File, len(raw)) + for i, f := range raw { + out[i] = File{Index: f.Index, Path: f.Name, Size: f.Size} + } + return out, nil +} + +func (c *Client) FileStates(ctx context.Context, hash string) ([]FileState, error) { + raw, err := c.files(ctx, hash) + if err != nil { + return nil, err + } + out := make([]FileState, len(raw)) + for i, f := range raw { + done := f.Size + if f.Progress < 1 { + // Progress is a float; never round an unfinished file up to whole. + done = min(int64(f.Progress*float64(f.Size)), f.Size-1) + } + out[i] = FileState{Path: f.Name, Size: f.Size, Done: max(done, 0), Skipped: f.Priority == 0} + } + return out, nil +} + +type qbitFile struct { + Index int `json:"index"` + Name string `json:"name"` + Size int64 `json:"size"` + Progress float64 `json:"progress"` + Priority int `json:"priority"` +} + +func (c *Client) files(ctx context.Context, hash string) ([]qbitFile, error) { + body, err := c.get(ctx, "/api/v2/torrents/files", url.Values{"hash": {hash}}) + if err != nil { + return nil, err + } + var raw []qbitFile + if err := json.Unmarshal(body, &raw); err != nil { + return nil, fmt.Errorf("%w: parse file list: %v", ErrUnreachable, err) + } + return raw, nil +} + +func (c *Client) command(ctx context.Context, hashes []string, primary, fallback string) error { + if len(hashes) == 0 { + return nil + } + params := []byte(url.Values{"hashes": {strings.Join(hashes, "|")}}.Encode()) + _, err := c.post(ctx, primary, "application/x-www-form-urlencoded", params) + var status statusError + if errors.As(err, &status) && (status.code == http.StatusNotFound || status.code == http.StatusMethodNotAllowed) { + _, err = c.post(ctx, fallback, "application/x-www-form-urlencoded", params) + } + return err +} + +type statusError struct { + code int + what string +} + +func (e statusError) Error() string { return fmt.Sprintf("%s: HTTP %d", e.what, e.code) } + +func (e statusError) Unwrap() error { return ErrUnreachable } + +func (c *Client) get(ctx context.Context, path string, params url.Values) ([]byte, error) { + endpoint := path + if len(params) > 0 { + endpoint += "?" + params.Encode() + } + return c.do(ctx, http.MethodGet, endpoint, "", nil) +} + +func (c *Client) post(ctx context.Context, path, contentType string, body []byte) ([]byte, error) { + return c.do(ctx, http.MethodPost, path, contentType, body) +} + +// do logs in when there is no session, and once more on a 403, which is how +// qBittorrent says a session expired. +func (c *Client) do(ctx context.Context, method, path, contentType string, body []byte) ([]byte, error) { + if c.Config.BaseURL == "" { + return nil, ErrUnconfigured + } + if err := c.ensureSession(ctx, false); err != nil { + return nil, err + } + out, err := c.call(ctx, method, path, contentType, body) + var status statusError + if errors.As(err, &status) && status.code == http.StatusForbidden { + if err := c.ensureSession(ctx, true); err != nil { + return nil, err + } + return c.call(ctx, method, path, contentType, body) + } + return out, err +} + +func (c *Client) ensureSession(ctx context.Context, force bool) error { + c.mu.Lock() + defer c.mu.Unlock() + if c.http != nil && !force { + return nil + } + jar, err := cookiejar.New(nil) + if err != nil { + return fmt.Errorf("cookie jar: %w", err) + } + httpc := &http.Client{Jar: jar} + params := url.Values{"username": {c.Config.Username}, "password": {c.Config.Password}} + out, err := c.callWith(ctx, httpc, http.MethodPost, "/api/v2/auth/login", + "application/x-www-form-urlencoded", []byte(params.Encode())) + if err != nil { + var status statusError + if errors.As(err, &status) && status.code == http.StatusForbidden { + return fmt.Errorf("%w: too many failed attempts, or this address is banned", ErrAuth) + } + return err + } + if strings.HasPrefix(strings.TrimSpace(string(out)), "Fails") { + return fmt.Errorf("%w: wrong username or password", ErrAuth) + } + c.http = httpc + return nil +} + +func (c *Client) call(ctx context.Context, method, path, contentType string, body []byte) ([]byte, error) { + c.mu.Lock() + httpc := c.http + c.mu.Unlock() + if httpc == nil { + httpc = &http.Client{} + } + return c.callWith(ctx, httpc, method, path, contentType, body) +} + +func (c *Client) callWith(ctx context.Context, httpc *http.Client, method, path, contentType string, body []byte) ([]byte, error) { + base := strings.TrimRight(c.Config.BaseURL, "/") + status, out, err := send(ctx, httpc, c.Config.Timeout, method, base+path, contentType, body, func(h http.Header) { + // qBittorrent's CSRF check refuses a request without a matching + // Referer. + h.Set("Referer", base) + }) + if err != nil { + return nil, err + } + if status < 200 || status >= 300 { + return nil, statusError{code: status, what: path} + } + return out, nil +} + +// send makes one request to a download client under its timeout, lets the +// caller add headers, and reads at most 16 MB. It returns the status for the +// caller to read, because each client's refusals are its own. +func send(ctx context.Context, httpc *http.Client, timeout time.Duration, method, target, contentType string, body []byte, set func(http.Header)) (int, []byte, error) { + if timeout <= 0 { + timeout = DefaultTimeout + } + ctx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + var reader io.Reader + if body != nil { + reader = bytes.NewReader(body) + } + req, err := http.NewRequestWithContext(ctx, method, target, reader) + if err != nil { + return 0, nil, fmt.Errorf("%w: build request: %v", ErrUnreachable, err) + } + req.Header.Set("User-Agent", UserAgent) + if contentType != "" { + req.Header.Set("Content-Type", contentType) + } + if set != nil { + set(req.Header) + } + resp, err := httpc.Do(req) + if err != nil { + return 0, nil, fmt.Errorf("%w: %v", ErrUnreachable, err) + } + defer func() { _ = resp.Body.Close() }() + out, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20)) + if err != nil { + return 0, nil, fmt.Errorf("%w: read body: %v", ErrUnreachable, err) + } + return resp.StatusCode, out, nil +} diff --git a/internal/eventbus/eventbus.go b/internal/eventbus/eventbus.go index 1f3504e6..d9b42eda 100644 --- a/internal/eventbus/eventbus.go +++ b/internal/eventbus/eventbus.go @@ -1,5 +1,6 @@ // Package eventbus is an in-process publish/subscribe bus for real-time -// events the server emits to subscribed clients (today: Flutter via SSE). +// events the server emits to subscribed clients (the web and Android apps, +// over SSE). // // Writers (likes handler, requests reconciler, scanner, etc.) call Publish // to announce a state change. Each connected SSE client holds a @@ -13,9 +14,9 @@ // - No persistence. A client that connects after an event was emitted // will not see that event. SSE clients are expected to fetch a fresh // snapshot via the normal /api/* endpoints on (re)connect. -// - No cross-process broadcast. v1 runs as a single Go binary; a future -// multi-instance deployment will need a real broker (NATS / Redis -// pub-sub / etc.) but that's out of scope here. +// - No cross-process broadcast. Minstrel runs as a single Go binary; a +// multi-instance deployment would need a real broker (NATS / Redis +// pub-sub / etc.). package eventbus import ( @@ -38,6 +39,7 @@ type Event struct { type Bus struct { mu sync.RWMutex subscribers map[chan Event]struct{} + closed bool } // New returns a Bus with no subscribers. @@ -74,6 +76,13 @@ func (b *Bus) Subscribe(buffer int) (<-chan Event, func()) { } ch := make(chan Event, buffer) b.mu.Lock() + if b.closed { + // A subscriber arriving during shutdown ends at once, like the + // ones Close already ended. + b.mu.Unlock() + close(ch) + return ch, func() {} + } b.subscribers[ch] = struct{}{} b.mu.Unlock() unsub := func() { @@ -87,6 +96,21 @@ func (b *Bus) Subscribe(buffer int) (<-chan Event, func()) { return ch, unsub } +// Close ends every subscription by closing its channel, and any later +// Subscribe gets a channel that is already closed. The server calls it as +// shutdown begins: an SSE stream otherwise runs until its client leaves, +// and holds the graceful shutdown open for its whole deadline. Publish +// after Close is a no-op. Safe to call more than once. +func (b *Bus) Close() { + b.mu.Lock() + defer b.mu.Unlock() + b.closed = true + for ch := range b.subscribers { + delete(b.subscribers, ch) + close(ch) + } +} + // SubscriberCount returns the current number of active subscribers. For // tests + observability; not load-bearing in hot paths. func (b *Bus) SubscriberCount() int { diff --git a/internal/eventbus/eventbus_test.go b/internal/eventbus/eventbus_test.go index 65898b77..ea8d4ef7 100644 --- a/internal/eventbus/eventbus_test.go +++ b/internal/eventbus/eventbus_test.go @@ -148,3 +148,44 @@ func TestPublish_ConcurrentWritersAndSubscribers(t *testing.T) { } subWG.Wait() } + +// Close is what lets the server shut down with SSE clients connected: each +// stream returns when its channel closes. +func TestClose_EndsEverySubscription(t *testing.T) { + t.Parallel() + b := New() + ch1, unsub1 := b.Subscribe(4) + ch2, unsub2 := b.Subscribe(4) + + b.Close() + + for i, ch := range []<-chan Event{ch1, ch2} { + if _, ok := <-ch; ok { + t.Fatalf("subscriber %d: channel still open after Close", i) + } + } + if got := b.SubscriberCount(); got != 0 { + t.Fatalf("SubscriberCount=%d after Close, want 0", got) + } + // The handlers' deferred unsubscribes still run; they must not + // double-close. + unsub1() + unsub2() + b.Close() +} + +func TestSubscribe_AfterCloseIsAlreadyClosed(t *testing.T) { + t.Parallel() + b := New() + b.Close() + + ch, unsub := b.Subscribe(4) + defer unsub() + if _, ok := <-ch; ok { + t.Fatal("a subscription made after Close should already be closed") + } + if got := b.SubscriberCount(); got != 0 { + t.Fatalf("SubscriberCount=%d, want 0", got) + } + b.Publish(Event{Kind: "after.close"}) // must not panic +} diff --git a/internal/exchange/exchange.go b/internal/exchange/exchange.go new file mode 100644 index 00000000..987c90bc --- /dev/null +++ b/internal/exchange/exchange.go @@ -0,0 +1,181 @@ +// Package exchange talks to FabledExchange's native API for the two jobs that +// are not downloading: searching its DHT index (/api/search) and describing +// one torrent (/api/torrents/{infohash}). Downloading is internal/downloader's. +// +// The native search rather than Exchange's Torznab feed, because it reports +// whether a swarm was measured and how many files a torrent holds, which the +// feed cannot say. +package exchange + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "strings" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// ErrUnconfigured is an Exchange with no address. +var ErrUnconfigured = errors.New("exchange is not configured") + +// ErrUnreachable is a call that could not be made or was not understood. +var ErrUnreachable = errors.New("exchange unreachable") + +// ErrAuth is an API key Exchange refused, or a secured Exchange asked with +// none. +var ErrAuth = errors.New("exchange refused the API key") + +// ErrUnknown is a torrent Exchange has not crawled: a JSON 404, which is an +// answer and not a fault. +var ErrUnknown = errors.New("exchange has not crawled this torrent") + +// UserAgent names Minstrel in Exchange's logs. +const UserAgent = "Minstrel (self-hosted music server; release search)" + +// DefaultTimeout is what an unset Timeout means. Exchange answers from its +// own database. +const DefaultTimeout = 20 * time.Second + +// Config is one Exchange. +type Config struct { + BaseURL string + // APIKey is optional and sent only when set: an empty bearer makes a + // secured Exchange answer 401. + APIKey string + Timeout time.Duration +} + +// Client asks one Exchange. +type Client struct { + Config Config + // HTTP is injectable for tests. Nil takes http.DefaultClient. + HTTP *http.Client + // Code is the acquisition_services row, stamped onto search results. + Code string + Logger *slog.Logger +} + +func (c *Client) log() *slog.Logger { + if c == nil || c.Logger == nil { + return slog.Default() + } + return c.Logger +} + +// Detail is what Exchange knows about one torrent. +type Detail struct { + InfoHash string `json:"infohash"` + Name string `json:"name"` + Size int64 `json:"size"` + FilesCount int `json:"files_count"` + Seeders int `json:"seeders"` + SwarmKnown bool `json:"swarm_known"` + Leechers int `json:"leechers"` + LeechersKnown bool `json:"leechers_known"` + ContentType string `json:"content_type"` + Files []DetailFile `json:"files"` +} + +// DetailFile is one file inside a torrent. +type DetailFile struct { + Index int `json:"index"` + Path string `json:"path"` + Size int64 `json:"size"` +} + +// Torrent asks Exchange about one v1 infohash. +func (c *Client) Torrent(ctx context.Context, infohash string) (Detail, error) { + if strings.TrimSpace(c.Config.BaseURL) == "" { + return Detail{}, ErrUnconfigured + } + hash := indexer.NormalizeHash(infohash) + if len(hash) != 40 || !indexer.IsInfoHash(hash) { + return Detail{}, fmt.Errorf("%w: %q is not a 40-character infohash", ErrUnreachable, infohash) + } + body, err := c.get(ctx, "/api/torrents/"+hash) + if err != nil { + return Detail{}, err + } + var out Detail + if err := json.Unmarshal(body, &out); err != nil { + return Detail{}, fmt.Errorf("%w: parse torrent detail: %v", ErrUnreachable, err) + } + if out.InfoHash == "" { + return Detail{}, ErrUnknown + } + return out, nil +} + +func (c *Client) get(ctx context.Context, path string) ([]byte, error) { + timeout := c.Config.Timeout + if timeout <= 0 { + timeout = DefaultTimeout + } + ctx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + + target := strings.TrimRight(c.Config.BaseURL, "/") + path + req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) + } + req.Header.Set("User-Agent", UserAgent) + req.Header.Set("Accept", "application/json") + if key := strings.TrimSpace(c.Config.APIKey); key != "" { + req.Header.Set("Authorization", "Bearer "+key) + } + client := c.HTTP + if client == nil { + client = http.DefaultClient + } + res, err := client.Do(req) + if err != nil { + return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) + } + defer func() { _ = res.Body.Close() }() + body, err := io.ReadAll(io.LimitReader(res.Body, 1<<20)) + if err != nil { + return nil, fmt.Errorf("%w: read answer: %v", ErrUnreachable, err) + } + switch { + case res.StatusCode == http.StatusNotFound: + // A 404 that is not JSON came from something else at this address, + // which must not look like a healthy Exchange that lacks a torrent. + if !looksLikeJSON(body) { + return nil, fmt.Errorf("%w: something at this address answered 404, "+ + "but it was not FabledExchange", ErrUnreachable) + } + return nil, ErrUnknown + case res.StatusCode == http.StatusUnauthorized || res.StatusCode == http.StatusForbidden: + return nil, fmt.Errorf("%w: %s", ErrAuth, + message(body, "set this FabledExchange's API key in Minstrel's settings")) + case res.StatusCode == http.StatusServiceUnavailable: + return nil, fmt.Errorf("%w: %s", ErrUnreachable, message(body, "no index is configured")) + case res.StatusCode < 200 || res.StatusCode >= 300: + return nil, fmt.Errorf("%w: %s", ErrUnreachable, + message(body, fmt.Sprintf("returned %d", res.StatusCode))) + } + return body, nil +} + +// message is Exchange's own {"error": "..."} text, or fallback. +func message(body []byte, fallback string) string { + var env struct { + Error string `json:"error"` + } + if err := json.Unmarshal(body, &env); err == nil && strings.TrimSpace(env.Error) != "" { + return strings.TrimSpace(env.Error) + } + return fallback +} + +func looksLikeJSON(body []byte) bool { + t := strings.TrimSpace(string(body)) + return strings.HasPrefix(t, "{") || strings.HasPrefix(t, "[") +} diff --git a/internal/exchange/exchange_test.go b/internal/exchange/exchange_test.go new file mode 100644 index 00000000..bfd5aa92 --- /dev/null +++ b/internal/exchange/exchange_test.go @@ -0,0 +1,108 @@ +package exchange + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +func TestTorrent_SendsTheKeyOnlyWhenSet(t *testing.T) { + auth := make(chan string, 2) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + auth <- r.Header.Get("Authorization") + _, _ = w.Write([]byte(`{"infohash": "` + strings.Repeat("a", 40) + `", "files_count": 12}`)) + })) + defer srv.Close() + + for _, key := range []string{"", "k"} { + c := &Client{Config: Config{BaseURL: srv.URL, APIKey: key}} + d, err := c.Torrent(context.Background(), strings.Repeat("A", 40)) + if err != nil || d.FilesCount != 12 { + t.Fatalf("key %q: detail %+v err %v", key, d, err) + } + } + if without, with := <-auth, <-auth; without != "" || with != "Bearer k" { + t.Errorf("Authorization = %q then %q, want none without a key and a bearer with one", without, with) + } +} + +func TestTorrent_TellsAnUncrawledTorrentFromAnotherService(t *testing.T) { + for name, tc := range map[string]struct { + status int + body string + want error + }{ + "Exchange has not crawled it": {http.StatusNotFound, `{"error": "unknown torrent"}`, ErrUnknown}, + "not FabledExchange": {http.StatusNotFound, `404`, ErrUnreachable}, + "a refused key": {http.StatusUnauthorized, `{"error": "bad key"}`, ErrAuth}, + } { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(tc.status) + _, _ = w.Write([]byte(tc.body)) + })) + _, err := (&Client{Config: Config{BaseURL: srv.URL}}).Torrent(context.Background(), strings.Repeat("0", 40)) + srv.Close() + if !errors.Is(err, tc.want) { + t.Errorf("%s: err = %v, want %v", name, err, tc.want) + } + } +} + +func TestSearch_PagesAndKeepsMeasuredSwarmsOnly(t *testing.T) { + queries := make(chan string, 4) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + queries <- r.URL.RawQuery + offset := r.URL.Query().Get("offset") + var rows []string + if offset == "0" { + for i := 0; i < SearchLimit; i++ { + rows = append(rows, fmt.Sprintf(`{"infohash": "%040x", "name": "r%d", "content_type": "music", "seeders": 5, "swarm_known": %t}`, i, i, i == 0)) + } + } else { + // A repeat of page one's first hash, which must not be listed twice. + rows = append(rows, fmt.Sprintf(`{"infohash": "%040x", "name": "again"}`, 0)) + } + _, _ = fmt.Fprintf(w, `{"total": %d, "results": [%s]}`, SearchLimit+1, strings.Join(rows, ",")) + })) + defer srv.Close() + + c := &Client{Code: "exchange", Config: Config{BaseURL: srv.URL}} + results, err := c.Search(context.Background(), indexer.Query{Text: "x", Categories: indexer.MusicCategories}) + if err != nil { + t.Fatalf("search: %v", err) + } + close(queries) + var asked []string + for q := range queries { + asked = append(asked, q) + } + if len(asked) != 2 || !strings.Contains(asked[0], "type=music") { + t.Errorf("queries = %v, want two pages asking for music", asked) + } + if len(results) != SearchLimit { + t.Errorf("results = %d, want %d (the repeat dropped)", len(results), SearchLimit) + } + if r := results[0]; !r.SwarmKnown || r.Seeders != 5 || r.Category != 3000 || r.Source != "exchange" || !strings.HasPrefix(r.Magnet, "magnet:") { + t.Errorf("first = %+v", r) + } + if r := results[1]; r.SwarmKnown || r.Seeders != 0 { + t.Errorf("an unmeasured swarm reported seeders: %+v", r) + } +} + +func TestSearch_ARefusedKeyReadsAsAuth(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusForbidden) + })) + defer srv.Close() + _, err := (&Client{Config: Config{BaseURL: srv.URL}}).Search(context.Background(), indexer.Query{Text: "x"}) + if !errors.Is(err, indexer.ErrAuth) { + t.Fatalf("err = %v, want indexer.ErrAuth", err) + } +} diff --git a/internal/exchange/search.go b/internal/exchange/search.go new file mode 100644 index 00000000..d2fc8724 --- /dev/null +++ b/internal/exchange/search.go @@ -0,0 +1,169 @@ +package exchange + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/url" + "strings" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// SearchLimit is one page of /api/search; SearchDepth is the most read across +// pages. A query matching more than SearchDepth is too broad to judge anyway. +const ( + SearchLimit = 100 + SearchDepth = 300 +) + +type searchResult struct { + InfoHash string `json:"infohash"` + Name string `json:"name"` + Size int64 `json:"size"` + FilesCount int `json:"files_count"` + ContentType string `json:"content_type"` + Seeders int `json:"seeders"` + SwarmKnown bool `json:"swarm_known"` + Leechers int `json:"leechers"` + LeechersKnown bool `json:"leechers_known"` +} + +type searchPage struct { + Total int `json:"total"` + Results []searchResult `json:"results"` +} + +// Search asks Exchange's DHT index. Every result is a magnet, so there is no +// .torrent to fetch through it. A later page that fails keeps the earlier +// ones; a first page that fails is the search failing. +func (c *Client) Search(ctx context.Context, q indexer.Query) ([]indexer.Result, error) { + if strings.TrimSpace(c.Config.BaseURL) == "" { + return nil, indexer.ErrUnconfigured + } + params := url.Values{"q": {q.Text}, "limit": {fmt.Sprint(SearchLimit)}} + if t := contentType(q.Categories); t != "" { + params.Set("type", t) + } + var ( + out []indexer.Result + seen = map[string]bool{} + ) + for offset := 0; offset < SearchDepth; offset += SearchLimit { + params.Set("offset", fmt.Sprint(offset)) + page, err := c.searchPage(ctx, params) + if err != nil { + if offset == 0 { + return nil, err + } + c.log().Warn("exchange: a later page of a search could not be read; keeping the pages before it", + "q", q.Text, "offset", offset, "err", err) + break + } + for _, r := range page.Results { + res, ok := c.result(r) + if !ok || seen[res.InfoHash] { + continue + } + seen[res.InfoHash] = true + out = append(out, res) + } + if len(page.Results) < SearchLimit || offset+SearchLimit >= page.Total { + break + } + } + return out, nil +} + +func (c *Client) searchPage(ctx context.Context, params url.Values) (searchPage, error) { + var page searchPage + body, err := c.get(ctx, "/api/search?"+params.Encode()) + if err != nil { + return page, searchError(err) + } + if err := json.Unmarshal(body, &page); err != nil { + return page, fmt.Errorf("%w: parse search results: %v", indexer.ErrUnreachable, err) + } + return page, nil +} + +func (c *Client) result(r searchResult) (indexer.Result, bool) { + hash := indexer.NormalizeHash(r.InfoHash) + if hash == "" || strings.TrimSpace(r.Name) == "" { + return indexer.Result{}, false + } + res := indexer.Result{ + Title: r.Name, + InfoHash: hash, + Magnet: magnet(hash, r.Name), + SizeBytes: r.Size, + Category: category(r.ContentType), + Source: c.Code, + FilesCount: r.FilesCount, + SwarmKnown: r.SwarmKnown, + } + // Seeders only when measured: an unmeasured count is not health. + if r.SwarmKnown { + res.Seeders, res.Peers = r.Seeders, r.Seeders + if r.LeechersKnown { + res.Peers = r.Seeders + r.Leechers + } + } + return res, true +} + +// searchError puts Exchange's errors in the vocabulary every search source +// shares, so a refused key reads the same whichever source refused it. +func searchError(err error) error { + switch { + case errors.Is(err, ErrAuth): + return fmt.Errorf("%w: %v", indexer.ErrAuth, err) + case errors.Is(err, ErrUnknown): + return fmt.Errorf("%w: this FabledExchange has no search API", indexer.ErrUnreachable) + } + return fmt.Errorf("%w: %v", indexer.ErrUnreachable, err) +} + +// contentType narrows Exchange's search to music when the query asks only +// for music categories (3000s, not 3030 audiobooks). +func contentType(cats []int) string { + music := false + for _, c := range cats { + if c < 3000 || c >= 4000 || c == 3030 { + return "" + } + music = true + } + if music { + return "music" + } + return "" +} + +func category(contentType string) int { + switch contentType { + case "music": + return 3000 + case "audiobook": + return 3030 + case "ebook": + return 7020 + case "comic": + return 7030 + case "movie": + return 2000 + case "tv": + return 5000 + } + return 8000 +} + +// magnet is a trackerless magnet for a hash Exchange found on the DHT. +func magnet(hash, name string) string { + m := "magnet:?xt=urn:btih:" + hash + if name != "" { + m += "&dn=" + url.QueryEscape(name) + } + return m +} diff --git a/internal/gc/worker.go b/internal/gc/worker.go index e0d31068..c88ad965 100644 --- a/internal/gc/worker.go +++ b/internal/gc/worker.go @@ -1,8 +1,6 @@ -// Package gc runs periodic garbage-collection / lifecycle sweeps -// against tables that have NO writer-side close path or NO retention -// policy. Each sweep addresses a drift finding from the 2026-06-02 -// audit (Scribe parent #552) and is idempotent — re-running it on -// already-clean rows is a no-op. +// Package gc runs periodic garbage-collection / lifecycle sweeps against +// tables that have no writer-side close path or no retention policy. Each +// sweep is idempotent: re-running it on already-clean rows is a no-op. // // One Worker handles all sweeps so a single long-tick goroutine // amortises the per-tick fixed cost. Each individual sweep is small @@ -11,16 +9,16 @@ // visible in the application log without an explicit metrics layer. // // Sweeps: -// - GcCloseStalePlayEvents (#566) -// - GcClosePlaySessionsWithNoRecentEvents (#565) -// - GcExpireScrobbleQueueFailedRows (#567) -// - GcResetStuckSystemPlaylistRuns (#574) -// - GcDeleteExpiredPasswordResets (#575) -// - GcDeleteExpiredSessions (M462 #4978 — idle 30d / absolute 1y) -// - GcPruneDiagnostics (M9 — diagnostics 30d retention) -// - GcDeleteExpiredSuggestionSnoozes (#2374 — snoozes expire, then go) -// - GcDeleteOrphanedCandidateArtistTags(+State) (#2376 — the similarity -// feed churns, so cached candidate tags outlive their candidates) +// - GcCloseStalePlayEvents +// - GcClosePlaySessionsWithNoRecentEvents +// - GcExpireScrobbleQueueFailedRows +// - GcResetStuckSystemPlaylistRuns +// - GcDeleteExpiredPasswordResets +// - GcDeleteExpiredSessions (idle 30d / absolute 1y) +// - GcPruneDiagnostics (30d retention) +// - GcDeleteExpiredSuggestionSnoozes (snoozes expire, then go) +// - GcDeleteOrphanedCandidateArtistTags(+State) (the similarity feed +// churns, so cached candidate tags outlive their candidates) package gc import ( diff --git a/internal/importer/importer.go b/internal/importer/importer.go new file mode 100644 index 00000000..024c96c4 --- /dev/null +++ b/internal/importer/importer.go @@ -0,0 +1,610 @@ +package importer + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log/slog" + "os" + "path/filepath" + "slices" + "strconv" + "sync" + "time" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// Import states, as grabs.import_state holds them. +const ( + StateDownloading = "downloading" + StateWaiting = "waiting" + StateImporting = "importing" + StateImported = "imported" + StateRefused = "refused" + StateReview = "review" + StateHeld = "held" + StateFailed = "failed" +) + +const ( + // interval: a finished download is noticed within a minute. One + // Torrents call per download client, so the client barely notices. + interval = time.Minute + // passTimeout bounds a pass: fingerprinting a long album against + // AcoustID's pace takes a minute or two, so half an hour is a hang. + passTimeout = 30 * time.Minute + // staleMinutes: an import still "importing" after this was cut short + // (a restart mid-copy) and is tried again. + staleMinutes = 30 +) + +// Scanner reads new files into the library. +type Scanner interface { + ScanFiles(ctx context.Context, paths []string) ([]pgtype.UUID, error) +} + +// Catalogue lists a release group's editions. +type Catalogue interface { + Releases(ctx context.Context, groupMBID string) (catalogue.Editions, error) +} + +// Importer watches every live grab's download and imports it once finished. +type Importer struct { + pool *pgxpool.Pool + Scanner Scanner + Catalogue Catalogue + Printer Printer + // Roots are the library's folders (the scan paths), read on every use + // because they change in Settings. New music is filed under the + // one the settings name, or the first. + Roots func() []string + // DataDir is where cached art lives, for tidying an artist an upgrade + // leaves with nothing. Empty skips it. + DataDir string + // ClientFor builds the download client a grab went to. Defaults to + // acquire.DownloadClientFor. + ClientFor func(ctx context.Context, code string) (downloader.Driver, error) + logger *slog.Logger + wake chan struct{} + mu sync.Mutex // one pass at a time +} + +// New returns an Importer. printer may be nil: the check then rests on +// lengths and tags. +func New(pool *pgxpool.Pool, scanner Scanner, cat Catalogue, printer Printer, roots func() []string, logger *slog.Logger) *Importer { + if logger == nil { + logger = slog.Default() + } + im := &Importer{pool: pool, Scanner: scanner, Catalogue: cat, Printer: printer, Roots: roots, + logger: logger, wake: make(chan struct{}, 1)} + im.ClientFor = func(ctx context.Context, code string) (downloader.Driver, error) { + return acquire.DownloadClientFor(ctx, pool, code) + } + return im +} + +// Wake asks for a pass now: after a grab, or a person's decision. +func (im *Importer) Wake() { + select { + case im.wake <- struct{}{}: + default: + } +} + +// Run passes every minute, and when woken, until ctx ends. +func (im *Importer) Run(ctx context.Context) { + t := time.NewTicker(interval) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + case <-im.wake: + } + rep := im.Pass(ctx) + if rep.Imported+rep.Refused+rep.Review+rep.Held+rep.Failed > 0 || rep.Error != "" { + im.logger.Info("import: pass", "imported", rep.Imported, "refused", rep.Refused, + "review", rep.Review, "held", rep.Held, "failed", rep.Failed, + "downloading", rep.Downloading, "error", rep.Error) + } + } +} + +// Report is one pass. +type Report struct { + Downloading, Imported, Refused, Review, Held, Failed, Waiting int + Error string +} + +// Pass looks at every live grab once. +func (im *Importer) Pass(ctx context.Context) Report { + im.mu.Lock() + defer im.mu.Unlock() + ctx, cancel := context.WithTimeout(ctx, passTimeout) + defer cancel() + var rep Report + q := dbq.New(im.pool) + if _, err := q.ReclaimStaleImports(ctx, staleMinutes); err != nil { + rep.Error = err.Error() + return rep + } + grabs, err := q.ListLiveGrabs(ctx) + if err != nil { + rep.Error = err.Error() + return rep + } + byClient := map[string][]dbq.ListLiveGrabsRow{} + var order []string + for _, g := range grabs { + if _, ok := byClient[g.Client]; !ok { + order = append(order, g.Client) + } + byClient[g.Client] = append(byClient[g.Client], g) + } + for _, code := range order { + im.passClient(ctx, q, code, byClient[code], &rep) + } + return rep +} + +func (im *Importer) passClient(ctx context.Context, q *dbq.Queries, code string, gs []dbq.ListLiveGrabsRow, rep *Report) { + wait := func(g dbq.ListLiveGrabsRow, why string) { + rep.Waiting++ + _ = q.SetGrabProgress(ctx, dbq.SetGrabProgressParams{ + ID: g.ID, Progress: g.Progress, ImportState: StateWaiting, ImportReason: why}) + } + d, err := im.ClientFor(ctx, code) + if err != nil { + for _, g := range gs { + wait(g, fmt.Sprintf("Can't reach its download client: %v.", err)) + } + return + } + hashes := make([]string, 0, len(gs)) + for _, g := range gs { + hashes = append(hashes, g.InfoHash) + } + ts, err := d.Torrents(ctx, hashes...) + if err != nil { + for _, g := range gs { + wait(g, fmt.Sprintf("Can't reach its download client %s: %s", code, indexer.FirstLine(err.Error()))) + } + return + } + byHash := map[string]downloader.Torrent{} + for _, t := range ts { + byHash[indexer.NormalizeHash(t.Hash)] = t + } + for _, g := range gs { + if ctx.Err() != nil { + return + } + switch g.ImportState { + case StateReview, StateFailed, StateImporting: + continue + } + t, ok := byHash[indexer.NormalizeHash(g.InfoHash)] + if !ok { + continue // gone from the client: the lifecycle settles it + } + if !g.WantID.Valid { + wait(g, "Its album is no longer wanted.") + continue + } + if !finished(ctx, d, t) { + rep.Downloading++ + _ = q.SetGrabProgress(ctx, dbq.SetGrabProgressParams{ + ID: g.ID, Progress: float32(min(t.Progress, 0.999)), ImportState: StateDownloading}) + continue + } + claimed, err := q.ClaimGrabImport(ctx, g.ID) + if err != nil { + continue // claimed by a pass racing this one, or ended + } + switch out := im.importOne(ctx, q, claimed, d, t); out { + case StateImported: + rep.Imported++ + case StateRefused: + rep.Refused++ + case StateReview: + rep.Review++ + case StateHeld: + rep.Held++ + case StateFailed: + rep.Failed++ + case StateWaiting: + rep.Waiting++ + } + } +} + +// upStates are qBittorrent's states for a torrent that has every byte and +// is seeding, paused or queued to seed. Exchange's states arrive in the same +// vocabulary. +var upStates = map[string]bool{ + "uploading": true, "stalledUP": true, "pausedUP": true, "stoppedUP": true, + "queuedUP": true, "forcedUP": true, "checkingUP": true, +} + +// finished: whole, in a seeding state, and no wanted file short of bytes. +func finished(ctx context.Context, d downloader.Driver, t downloader.Torrent) bool { + if !t.Done() || !upStates[t.State] { + return false + } + if fp, ok := d.(downloader.FileProgress); ok { + if states, err := fp.FileStates(ctx, t.Hash); err == nil { + for _, f := range states { + if f.Unfinished() { + return false + } + } + } + } + return true +} + +// Detail is what grabs.import_detail keeps: the verdict and what it rested +// on, the edition chosen, and what happened to the files. +type Detail struct { + Verdict Verdict `json:"verdict"` + Edition string `json:"edition,omitempty"` + EditionMBID string `json:"edition_mbid,omitempty"` + Files []string `json:"files,omitempty"` + Placed int `json:"placed,omitempty"` + Skipped int `json:"skipped,omitempty"` + Untagged int `json:"untagged,omitempty"` + // RehomedFrom: the album it was grabbed for, when it turned out to be + // another wanted album. + RehomedFrom string `json:"rehomed_from,omitempty"` +} + +// importOne checks and files one finished download and records the outcome. +func (im *Importer) importOne(ctx context.Context, q *dbq.Queries, g dbq.Grab, d downloader.Driver, t downloader.Torrent) string { + record := func(state, reason, end string, det Detail, files int, dest string) string { + body, _ := json.Marshal(det) + // The outcome is written even when the pass's context is ending: + // a claimed grab left "importing" waits half an hour to be retried. + wctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 15*time.Second) + defer cancel() + if _, err := q.RecordImport(wctx, dbq.RecordImportParams{ + ID: g.ID, ImportState: state, ImportReason: reason, ImportDetail: body, + ImportedFiles: int32(files), Destination: dest, EndReason: end, + }); err != nil { + im.logger.Warn("import: record outcome", "grab", g.Name, "state", state, "err", err) + } + im.logger.Info("import: "+state, "grab", g.Name, "reason", reason) + return state + } + + w, err := im.want(ctx, q, g.WantID) + if err != nil { + return record(StateWaiting, "Its album could not be read: "+err.Error(), "", Detail{}, 0, "") + } + editions, err := im.editions(ctx, w.GroupMBID) + if err != nil { + return record(StateWaiting, "MusicBrainz could not be reached to check it against the album; trying again shortly.", + "", Detail{}, 0, "") + } + src := t.ContentPath + if src == "" { + src = filepath.Join(g.SavePath, t.Name) + } + files, printed, err := inspect(ctx, src, im.Printer) + if err != nil { + return record(StateFailed, fmt.Sprintf("Couldn't read the download at %s: %v", src, err), "", Detail{}, 0, "") + } + v := Verify(w, editions, files, printed, g.ImportForced) + det := Detail{Verdict: v} + for _, f := range files { + det.Files = append(det.Files, f.Rel) + } + if v.Edition >= 0 { + det.Edition, det.EditionMBID = describe(editions[v.Edition]), editions[v.Edition].MBID + } + + switch v.Outcome { + case Review: + return record(StateReview, v.Reason, "", det, 0, "") + case Refuse: + if v.Code == CodeWrongAlbum { + if out, ok := im.rehome(ctx, q, g, d, w, files, printed, record); ok { + return out + } + } + im.refuse(ctx, q, g, d, w, v.Reason, pgtype.UUID{}) + return record(StateRefused, v.Reason, StateRefused, det, 0, "") + } + return im.place(ctx, q, g, d, w, editions, files, v, det, record) +} + +type recordFunc func(state, reason, end string, det Detail, files int, dest string) string + +// place files an accepted download and records it imported, or held when +// the library already had every track. +func (im *Importer) place(ctx context.Context, q *dbq.Queries, g dbq.Grab, d downloader.Driver, w Want, + editions []Edition, files []File, v Verdict, det Detail, record recordFunc) string { + root, err := im.root(ctx, q) + if err != nil { + return record(StateFailed, "Not imported: "+err.Error()+".", "", det, 0, "") + } + var e *Edition + if v.Edition >= 0 { + e = &editions[v.Edition] + } + plan := Plan(root, w, e, files, v.Pairs) + if w.Upgrade != "" { + return im.placeUpgrade(ctx, q, g, d, w, e, files, plan, v, det, record) + } + heldRec, heldPos, err := im.held(ctx, q, w, plan) + if err != nil { + return record(StateWaiting, "Couldn't read what the library holds: "+err.Error(), "", det, 0, "") + } + + var placed []string + for _, pl := range plan { + if (pl.Recording != "" && heldRec[pl.Recording]) || heldPos[[2]int{pl.Disc, pl.Track}] { + det.Skipped++ + continue + } + tags := Tags{Title: pl.Title, Artist: w.Artist, Album: w.Title, AlbumArtist: w.Artist, Year: w.Year, + Disc: pl.Disc, Discs: pl.Discs, Track: pl.Track, Tracks: pl.Tracks, + GroupMBID: w.GroupMBID, RecordingMBID: pl.Recording, ArtistMBID: w.ArtistMBID} + if e != nil { + tags.ReleaseMBID = e.MBID + } + tagged, err := Place(ctx, files[pl.File].Path, pl.Dest, tags) + if errors.Is(err, ErrOccupied) { + det.Skipped++ + continue + } + if err != nil { + for _, p := range placed { + _ = os.Remove(p) + } + return record(StateFailed, fmt.Sprintf("Copying into the library failed, and nothing was left behind: %v", err), + "", det, 0, "") + } + placed = append(placed, pl.Dest) + if !tagged { + det.Untagged++ + } + } + det.Placed = len(placed) + if len(placed) == 0 { + im.removeFromClient(ctx, d, g) + return record(StateHeld, fmt.Sprintf("The library already holds every track of %s, so nothing was imported.", w.Title), + StateHeld, det, 0, "") + } + + if _, err := im.Scanner.ScanFiles(ctx, placed); err != nil { + // The watcher and the next scan find the files anyway. + im.logger.Warn("import: scan the new files", "grab", g.Name, "err", err) + } + if err := q.MarkWantSearched(ctx, g.WantID); err != nil { + im.logger.Warn("import: mark the want searched", "grab", g.Name, "err", err) + } + dest := filepath.Dir(placed[0]) + if e != nil && e.Media() > 1 { + dest = filepath.Dir(dest) + } + reason := fmt.Sprintf("Imported %s into %s. %s", tracks(len(placed)), dest, v.Reason) + if det.Skipped > 0 { + reason += fmt.Sprintf(" %s already in the library left as they were.", tracks(det.Skipped)) + } + if det.Untagged > 0 { + reason += fmt.Sprintf(" %s copied with their own tags: ffmpeg could not rewrite them.", tracks(det.Untagged)) + } + return record(StateImported, reason, StateImported, det, len(placed), dest) +} + +// rehome imports a download that turned out to be another wanted album, +// named by its tags, for that album, when its own grab is not running. +func (im *Importer) rehome(ctx context.Context, q *dbq.Queries, g dbq.Grab, d downloader.Driver, + from Want, files []File, printed bool, record recordFunc) (string, bool) { + group := majority(files, func(f File) string { return f.GroupMBID }) + if group == "" { + return "", false + } + others, err := q.OpenWantsForGroups(ctx, []string{group}) + if err != nil || len(others) == 0 { + return "", false + } + other := others[0] + w, err := im.want(ctx, q, other.ID) + if err != nil { + return "", false + } + editions, err := im.editions(ctx, w.GroupMBID) + if err != nil { + return "", false + } + v := Verify(w, editions, files, printed, false) + if v.Outcome != Accept { + return "", false + } + moved, err := q.RehomeGrab(ctx, dbq.RehomeGrabParams{ID: g.ID, WantID: other.ID}) + if err != nil { + return "", false + } + if err := q.SearchWantAgain(ctx, g.WantID); err != nil { + im.logger.Warn("import: search the first album again", "want", from.Title, "err", err) + } + det := Detail{Verdict: v, RehomedFrom: from.Title} + for _, f := range files { + det.Files = append(det.Files, f.Rel) + } + if v.Edition >= 0 { + det.Edition, det.EditionMBID = describe(editions[v.Edition]), editions[v.Edition].MBID + } + v.Reason = fmt.Sprintf("Grabbed for %s, it is %s, which is wanted too. %s", from.Title, w.Title, v.Reason) + det.Verdict = v + return im.place(ctx, q, moved, d, w, editions, files, v, det, record), true +} + +// refuse blocks the release, asks for the album to be searched again and +// removes the download with its files: it is not the album. +func (im *Importer) refuse(ctx context.Context, q *dbq.Queries, g dbq.Grab, d downloader.Driver, w Want, why string, by pgtype.UUID) { + if err := acquire.NewMemory(im.pool).Block(ctx, dbq.BlockReleaseParams{ + InfoHash: g.InfoHash, Name: g.Name, Reason: acquire.BlockImport, Detail: why, WantID: g.WantID, BlockedBy: by, + }); err != nil { + im.logger.Warn("import: block the release", "grab", g.Name, "err", err) + } + if err := q.SearchWantAgain(ctx, g.WantID); err != nil { + im.logger.Warn("import: search again", "want", w.Title, "err", err) + } + im.removeFromClient(ctx, d, g) +} + +func (im *Importer) removeFromClient(ctx context.Context, d downloader.Driver, g dbq.Grab) { + if d == nil || g.InfoHash == "" { + return + } + if err := d.Delete(ctx, true, g.InfoHash); err != nil { + im.logger.Warn("import: remove the download", "grab", g.Name, "err", err) + } +} + +// Refuse is a person's "refuse" on a download the check held for them, or +// one that failed: it is refused as the check would have. +func (im *Importer) Refuse(ctx context.Context, grabID, by pgtype.UUID) (dbq.Grab, error) { + q := dbq.New(im.pool) + g, err := q.GetGrab(ctx, grabID) + if err != nil { + return dbq.Grab{}, err + } + if g.EndedAt.Valid || (g.ImportState != StateReview && g.ImportState != StateFailed) { + return dbq.Grab{}, ErrNotDecidable + } + var w Want + if g.WantID.Valid { + w, _ = im.want(ctx, q, g.WantID) + } + d, err := im.ClientFor(ctx, g.Client) + if err != nil { + d = nil + } + why := "A person refused it: it is not the album." + im.refuse(ctx, q, g, d, w, why, by) + out, err := q.RecordImport(ctx, dbq.RecordImportParams{ + ID: g.ID, ImportState: StateRefused, ImportReason: why, ImportDetail: g.ImportDetail, + EndReason: StateRefused, + }) + im.Wake() + return out, err +} + +// ErrNotDecidable: the grab is not waiting on a person. +var ErrNotDecidable = errors.New("importer: that download is not waiting on a decision") + +func (im *Importer) want(ctx context.Context, q *dbq.Queries, id pgtype.UUID) (Want, error) { + row, err := q.GetWantForSearch(ctx, id) + if err != nil { + return Want{}, err + } + w := Want{ + GroupMBID: row.ReleaseGroupMbid, ArtistMBID: row.ArtistMbid, Artist: row.ArtistName, Title: row.Title, + Year: yearOf(row.FirstReleaseDate), PrimaryType: row.PrimaryType, SecondaryTypes: row.SecondaryTypes, + } + if row.FulfilledAt.Valid && row.Upgrade != "" && row.AlbumID.Valid { + w.Upgrade, w.AlbumID = row.Upgrade, row.AlbumID + } + return w, nil +} + +func (im *Importer) editions(ctx context.Context, group string) ([]Edition, error) { + if im.Catalogue == nil { + return nil, errors.New("no catalogue") + } + eds, err := im.Catalogue.Releases(ctx, group) + if err != nil { + return nil, err + } + out := make([]Edition, 0, len(eds.Editions)) + for _, e := range eds.Editions { + ed := Edition{MBID: e.Release.Mbid, Title: e.Release.Title, Status: e.Release.Status, + Date: e.Release.ReleaseDate, Country: e.Release.Country} + for _, t := range e.Tracks { + ed.Tracks = append(ed.Tracks, Track{Medium: int(t.MediumPosition), Format: t.MediumFormat, + Position: int(t.Position), Title: t.Title, RecordingMBID: t.RecordingMbid, LengthMs: int(t.LengthMs)}) + } + slices.SortStableFunc(ed.Tracks, func(a, b Track) int { + if a.Medium != b.Medium { + return a.Medium - b.Medium + } + return a.Position - b.Position + }) + out = append(out, ed) + } + return out, nil +} + +// root is the library folder new music is filed into. +func (im *Importer) root(ctx context.Context, q *dbq.Queries) (string, error) { + roots := im.Roots() + if len(roots) == 0 { + return "", errors.New("there is no library folder to file new music into; add one in Settings → Library") + } + s, err := q.GetAcquisitionSettings(ctx) + if err == nil && s.ImportRoot != "" && slices.Contains(roots, s.ImportRoot) { + return s.ImportRoot, nil + } + return roots[0], nil +} + +// held reads which of the planned tracks the library already has: by +// recording, and by disc and track of an album of the same group. +func (im *Importer) held(ctx context.Context, q *dbq.Queries, w Want, plan []Placement) (map[string]bool, map[[2]int]bool, error) { + var recs []string + for _, pl := range plan { + if pl.Recording != "" { + recs = append(recs, pl.Recording) + } + } + rec := map[string]bool{} + if len(recs) > 0 { + got, err := q.HeldRecordings(ctx, recs) + if err != nil { + return nil, nil, err + } + for _, r := range got { + rec[r] = true + } + } + pos := map[[2]int]bool{} + got, err := q.HeldPositions(ctx, &w.GroupMBID) + if err != nil { + return nil, nil, err + } + for _, p := range got { + pos[[2]int{int(p.Disc), int(p.Track)}] = true + } + return rec, pos, nil +} + +func yearOf(date string) int { + if len(date) < 4 { + return 0 + } + y, err := strconv.Atoi(date[:4]) + if err != nil { + return 0 + } + return y +} + +func tracks(n int) string { + if n == 1 { + return "1 track" + } + return strconv.Itoa(n) + " tracks" +} diff --git a/internal/importer/importer_test.go b/internal/importer/importer_test.go new file mode 100644 index 00000000..dd442f1e --- /dev/null +++ b/internal/importer/importer_test.go @@ -0,0 +1,268 @@ +package importer + +import ( + "context" + "io" + "log/slog" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" +) + +func newPool(t *testing.T) *pgxpool.Pool { + t.Helper() + if testing.Short() { + t.Skip("skipping integration test in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(context.Background(), dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + dbtest.ResetDB(t, pool) + return pool +} + +const hash = "0123456789abcdef0123456789abcdef01234567" + +// fakeClient is a download client holding one finished torrent. +type fakeClient struct { + downloader.Driver + content string + deleted []string +} + +func (c *fakeClient) Torrents(_ context.Context, _ ...string) ([]downloader.Torrent, error) { + return []downloader.Torrent{{Hash: hash, Name: "Daft Punk - RAM [FLAC]", Progress: 1, State: "uploading", + ContentPath: c.content}}, nil +} + +func (c *fakeClient) Delete(_ context.Context, _ bool, hashes ...string) error { + c.deleted = append(c.deleted, hashes...) + return nil +} + +// fakeCatalogue knows RAM as one three-track edition. +type fakeCatalogue struct{} + +func (fakeCatalogue) Releases(_ context.Context, _ string) (catalogue.Editions, error) { + return catalogue.Editions{Editions: []catalogue.Edition{{ + Release: dbq.MbRelease{Mbid: "rel-ram", Title: "Random Access Memories", Status: "Official", ReleaseDate: "2013-05-17"}, + Tracks: []dbq.MbReleaseTrack{ + {MediumPosition: 1, MediumFormat: "CD", Position: 1, Title: "Give Life Back to Music", RecordingMbid: "rec-01"}, + {MediumPosition: 1, MediumFormat: "CD", Position: 2, Title: "The Game of Love", RecordingMbid: "rec-02"}, + {MediumPosition: 1, MediumFormat: "CD", Position: 3, Title: "Giorgio by Moroder", RecordingMbid: "rec-03"}, + }, + }}}, nil +} + +// fakePrinter names each file's recording by its file name: 01.flac is +// rec-01, unless wrong is set. +type fakePrinter struct{ wrong bool } + +func (p fakePrinter) Recordings(_ context.Context, path string) ([]string, error) { + if p.wrong { + return []string{"rec-other"}, nil + } + return []string{"rec-" + strings.TrimSuffix(filepath.Base(path), ".flac")}, nil +} + +type fakeScanner struct{ scanned []string } + +func (s *fakeScanner) ScanFiles(_ context.Context, paths []string) ([]pgtype.UUID, error) { + s.scanned = append(s.scanned, paths...) + return nil, nil +} + +type stack struct { + pool *pgxpool.Pool + q *dbq.Queries + im *Importer + client *fakeClient + scanner *fakeScanner + library string + want dbq.Want + grab dbq.Grab +} + +// newStack wants RAM, has a finished download of it waiting in a fake +// client, and an importer filing into a temp library. +func newStack(t *testing.T, printer Printer) *stack { + t.Helper() + ctx := context.Background() + pool := newPool(t) + q := dbq.New(pool) + content := t.TempDir() + for _, n := range []string{"01", "02", "03"} { + if err := os.WriteFile(filepath.Join(content, n+".flac"), []byte("not really flac "+n), 0o644); err != nil { + t.Fatal(err) + } + } + w, err := wanted.WantWith(ctx, q, wanted.WantParams{GroupMBID: "rg-ram", ArtistMBID: "mb-dp", ArtistName: "Daft Punk", + Title: "Random Access Memories", PrimaryType: "Album", FirstReleaseDate: "2013-05-17", Reason: wanted.ReasonRequest}) + if err != nil { + t.Fatal(err) + } + g, err := acquire.NewMemory(pool).Grabbed(ctx, dbq.InsertGrabParams{WantID: w.ID, InfoHash: hash, + Name: "Daft Punk - RAM [FLAC]", Key: hash, Client: "qbittorrent", SavePath: content, Origin: acquire.OriginSchedule}, + pgtype.UUID{}, "") + if err != nil { + t.Fatal(err) + } + s := &stack{pool: pool, q: q, client: &fakeClient{content: content}, scanner: &fakeScanner{}, + library: t.TempDir(), want: w, grab: g} + s.im = New(pool, s.scanner, fakeCatalogue{}, printer, func() []string { return []string{s.library} }, slog.New(slog.NewTextHandler(io.Discard, nil))) + s.im.ClientFor = func(context.Context, string) (downloader.Driver, error) { return s.client, nil } + return s +} + +func (s *stack) grabNow(t *testing.T) dbq.Grab { + t.Helper() + g, err := s.q.GetGrab(context.Background(), s.grab.ID) + if err != nil { + t.Fatal(err) + } + return g +} + +func TestImport_AFinishedDownloadOfTheAlbumIsFiled_Integration(t *testing.T) { + s := newStack(t, fakePrinter{}) + rep := s.im.Pass(context.Background()) + if rep.Imported != 1 { + t.Fatalf("pass = %+v", rep) + } + g := s.grabNow(t) + if g.ImportState != StateImported || g.EndReason != StateImported || !g.EndedAt.Valid || g.ImportedFiles != 3 { + t.Fatalf("grab = %s %s ended=%v files=%d: %s", g.ImportState, g.EndReason, g.EndedAt.Valid, g.ImportedFiles, g.ImportReason) + } + dir := filepath.Join(s.library, "Daft Punk", "Random Access Memories (2013)") + if g.Destination != dir { + t.Errorf("destination %q", g.Destination) + } + got, err := os.ReadFile(filepath.Join(dir, "Daft Punk - Random Access Memories - 02 - The Game of Love.flac")) + if err != nil || string(got) != "not really flac 02" { + t.Errorf("filed copy: %q %v", got, err) + } + if _, err := os.Stat(filepath.Join(s.client.content, "02.flac")); err != nil { + t.Errorf("the download's own file went: %v", err) + } + if len(s.scanner.scanned) != 3 || len(s.client.deleted) != 0 { + t.Errorf("scanned %v, deleted %v", s.scanner.scanned, s.client.deleted) + } + w, _ := s.q.GetWantByReleaseGroup(context.Background(), "rg-ram") + if !w.SearchedAt.Valid { + t.Error("an imported want is due for search again") + } + if s.im.Pass(context.Background()).Imported != 0 { + t.Error("imported twice") + } +} + +func TestImport_TheWrongAlbumIsRefusedBlockedAndSearchedAgain_Integration(t *testing.T) { + s := newStack(t, fakePrinter{wrong: true}) + ctx := context.Background() + if err := s.q.MarkWantSearched(ctx, s.want.ID); err != nil { + t.Fatal(err) + } + if rep := s.im.Pass(ctx); rep.Refused != 1 { + t.Fatalf("pass = %+v", rep) + } + g := s.grabNow(t) + if g.ImportState != StateRefused || g.EndReason != StateRefused || !strings.Contains(g.ImportReason, "recordings") { + t.Fatalf("grab = %s %s: %s", g.ImportState, g.EndReason, g.ImportReason) + } + blocks, err := s.q.ListReleaseBlocks(ctx) + if err != nil || len(blocks) != 1 || blocks[0].InfoHash != hash || blocks[0].Reason != acquire.BlockImport { + t.Errorf("blocks = %+v %v", blocks, err) + } + if len(s.client.deleted) != 1 { + t.Errorf("the refused download was not removed: %v", s.client.deleted) + } + w, _ := s.q.GetWantByReleaseGroup(ctx, "rg-ram") + if w.SearchedAt.Valid { + t.Error("a refused want is not due for search") + } + entries, _ := os.ReadDir(s.library) + if len(entries) != 0 { + t.Errorf("the library got %d entries", len(entries)) + } +} + +func TestImport_ADoubtfulDownloadWaitsForAPersonThenImportsAnyway_Integration(t *testing.T) { + s := newStack(t, nil) // no AcoustID; untagged files: nothing but lengths + ctx := context.Background() + if rep := s.im.Pass(ctx); rep.Review != 1 { + t.Fatalf("pass = %+v", rep) + } + g := s.grabNow(t) + if g.ImportState != StateReview || g.EndedAt.Valid || !strings.Contains(g.ImportReason, "A person decides") { + t.Fatalf("grab = %s ended=%v: %s", g.ImportState, g.EndedAt.Valid, g.ImportReason) + } + if rep := s.im.Pass(ctx); rep.Review+rep.Imported != 0 { + t.Errorf("a held download was looked at again: %+v", rep) + } + if _, err := s.q.ForceGrabImport(ctx, g.ID); err != nil { + t.Fatal(err) + } + if rep := s.im.Pass(ctx); rep.Imported != 1 { + t.Fatalf("after import anyway: %+v", rep) + } + if g := s.grabNow(t); g.ImportState != StateImported || !strings.Contains(g.ImportReason, "anyway") { + t.Errorf("grab = %s: %s", g.ImportState, g.ImportReason) + } +} + +func TestImport_APersonRefusesADoubtfulDownload_Integration(t *testing.T) { + s := newStack(t, nil) + ctx := context.Background() + s.im.Pass(ctx) + g, err := s.im.Refuse(ctx, s.grab.ID, pgtype.UUID{}) + if err != nil { + t.Fatal(err) + } + if g.ImportState != StateRefused || !g.EndedAt.Valid || len(s.client.deleted) != 1 { + t.Errorf("grab = %s ended=%v deleted=%v", g.ImportState, g.EndedAt.Valid, s.client.deleted) + } + if _, err := s.im.Refuse(ctx, s.grab.ID, pgtype.UUID{}); err == nil { + t.Error("refused twice") + } +} + +func TestImport_AnUnfinishedDownloadIsOnlyWatched_Integration(t *testing.T) { + s := newStack(t, fakePrinter{}) + s.im.ClientFor = func(context.Context, string) (downloader.Driver, error) { return &halfClient{s.client}, nil } + if rep := s.im.Pass(context.Background()); rep.Downloading != 1 { + t.Fatalf("pass = %+v", rep) + } + g := s.grabNow(t) + if g.ImportState != StateDownloading || g.Progress < 0.49 || g.Progress > 0.51 { + t.Errorf("grab = %s %.2f", g.ImportState, g.Progress) + } +} + +type halfClient struct{ *fakeClient } + +func (c *halfClient) Torrents(ctx context.Context, h ...string) ([]downloader.Torrent, error) { + ts, _ := c.fakeClient.Torrents(ctx, h...) + ts[0].Progress, ts[0].State = 0.5, "downloading" + return ts, nil +} diff --git a/internal/importer/inspect.go b/internal/importer/inspect.go new file mode 100644 index 00000000..e727b4cb --- /dev/null +++ b/internal/importer/inspect.go @@ -0,0 +1,176 @@ +package importer + +import ( + "context" + "errors" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/dhowden/tag" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acoustid" + "git.fabledsword.com/bvandeusen/minstrel/internal/library" +) + +// Printer says which MusicBrainz recordings a file's audio is. ErrNoPrints: +// it cannot say for any file (AcoustID is not set up, or not answering), so +// the check rests on lengths and tags. +type Printer interface { + Recordings(ctx context.Context, path string) ([]string, error) +} + +// ErrNoPrints: fingerprints are not available for this download. +var ErrNoPrints = errors.New("importer: fingerprints are not available") + +// AcoustIDPrinter fingerprints with fpcalc and asks AcoustID, under the +// AcoustID settings (key, minimum score). Each call reads the settings, so +// a change takes effect on the next file. +type AcoustIDPrinter struct { + Settings interface { + Get() library.AcoustIDSettings + } + Client *acoustid.Client +} + +// Recordings is every recording AcoustID gives at or above the minimum score: +// one audio can be several recordings (the album track and the single). +func (p *AcoustIDPrinter) Recordings(ctx context.Context, path string) ([]string, error) { + s := p.Settings.Get() + if !s.Ready() || p.Client == nil { + return nil, ErrNoPrints + } + fp, dur, err := library.LookupFingerprint(ctx, path) + if err != nil { + if library.IsInconclusive(err) { + return nil, ErrNoPrints + } + return nil, nil // fpcalc rejected the file: no evidence either way + } + cands, err := p.Client.Lookup(ctx, s.APIKey, fp, dur) + switch { + case errors.Is(err, acoustid.ErrInvalidKey), errors.Is(err, acoustid.ErrUnavailable): + return nil, ErrNoPrints + case err != nil: + return nil, nil + } + var out []string + for _, c := range cands { + if c.Score >= float64(s.MinScore) && c.ID != "" { + out = append(out, c.ID) + } + } + return out, nil +} + +// formatRank picks which format's files are the album when a download holds +// more than one (a FLAC folder and an MP3 folder): the most files, then the +// better format. +var formatRank = map[string]int{".flac": 6, ".wav": 5, ".m4a": 4, ".opus": 3, ".ogg": 2, ".mp3": 1, ".aac": 0} + +// audioFiles lists the audio files under root (a folder, or one file), in +// the one format that is the album. Dot-named files and folders are skipped: +// they are partial downloads, OS litter or Minstrel's own temp files. +func audioFiles(root string) ([]string, error) { + info, err := os.Stat(root) + if err != nil { + return nil, err + } + if !info.IsDir() { + if library.IsAudioFile(root) { + return []string{root}, nil + } + return nil, nil + } + byExt := map[string][]string{} + err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if strings.HasPrefix(d.Name(), ".") && path != root { + if d.IsDir() { + return filepath.SkipDir + } + return nil + } + if !d.IsDir() && library.IsAudioFile(path) { + ext := strings.ToLower(filepath.Ext(path)) + byExt[ext] = append(byExt[ext], path) + } + return nil + }) + if err != nil { + return nil, err + } + best := "" + for ext, paths := range byExt { + if best == "" || len(paths) > len(byExt[best]) || + (len(paths) == len(byExt[best]) && formatRank[ext] > formatRank[best]) { + best = ext + } + } + out := byExt[best] + sort.Strings(out) + return out, nil +} + +// readFile reads what one file's tags and length say. A file whose tags +// cannot be read still counts, by its length and name. +func readFile(ctx context.Context, root, path string) File { + f := File{Path: path, Rel: rel(root, path), Video: len(library.SourceMarkersFor(path)) > 0} + if ms, err := library.ProbeDurationMs(ctx, path); err == nil { + f.LengthMs = int(ms) + } + h, err := os.Open(path) + if err != nil { + return f + } + defer func() { _ = h.Close() }() + m, err := tag.ReadFrom(h) + if err != nil { + return f + } + f.Title, f.Artist, f.AlbumArtist, f.Album = m.Title(), m.Artist(), m.AlbumArtist(), m.Album() + f.Track, f.Disc = library.ReadTrackAndDisc(m) + ids := library.ReadTagMBIDs(m) + f.GroupMBID, f.ReleaseMBID, f.RecordingMBID = ids.ReleaseGroup, ids.Release, ids.Recording + return f +} + +func rel(root, path string) string { + if r, err := filepath.Rel(root, path); err == nil && r != "." { + return r + } + return filepath.Base(path) +} + +// inspect reads every audio file under root and asks the printer about each. +// fingerprinted reports whether the printer answered at all. +func inspect(ctx context.Context, root string, p Printer) (files []File, fingerprinted bool, err error) { + paths, err := audioFiles(root) + if err != nil { + return nil, false, err + } + fingerprinted = p != nil + for _, path := range paths { + if ctx.Err() != nil { + return nil, false, ctx.Err() + } + f := readFile(ctx, root, path) + if fingerprinted && len(paths) <= maxFiles { + recs, err := p.Recordings(ctx, path) + if errors.Is(err, ErrNoPrints) { + fingerprinted = false + for i := range files { + files[i].Recordings = nil + } + } else { + f.Recordings = recs + } + } + files = append(files, f) + } + return files, fingerprinted, nil +} diff --git a/internal/importer/name.go b/internal/importer/name.go new file mode 100644 index 00000000..b6a15a5b --- /dev/null +++ b/internal/importer/name.go @@ -0,0 +1,109 @@ +package importer + +import ( + "fmt" + "path/filepath" + "strings" + "unicode/utf8" +) + +// Placement is where one file goes and the names written into its tags. +type Placement struct { + File int // index into the files + Dest string // absolute + Title string + Disc int + Discs int + Track int + Tracks int // on its disc + // Recording is the MusicBrainz recording the file was matched to. + Recording string +} + +// Plan names every file's destination under root the way the library is +// conventionally named, so new arrivals sit beside what is there: +// +// {Artist}/{Album} ({Year})/{Artist} - {Album} - {track:00} - {Title}.ext +// +// with a "{Medium Format} {medium:00}" folder between when the edition has +// more than one medium. A file the check paired to a track takes the track's +// name and place; one it did not (an "import anyway" with more files than +// tracks) keeps its own tags. +func Plan(root string, w Want, e *Edition, files []File, pairs []Pair) []Placement { + album := w.Title + if w.Year > 0 { + album = fmt.Sprintf("%s (%d)", w.Title, w.Year) + } + dir := filepath.Join(root, Segment(w.Artist), Segment(album)) + media := 1 + if e != nil { + media = max(e.Media(), 1) + } + perDisc := map[int]int{} + if e != nil { + for _, t := range e.Tracks { + perDisc[t.Medium]++ + } + } + paired := map[int]Pair{} + for _, p := range pairs { + paired[p.File] = p + } + + out := make([]Placement, 0, len(files)) + for i, f := range files { + pl := Placement{File: i, Title: f.Title, Disc: max(f.Disc, 1), Track: f.Track, Discs: media} + format := "" + if p, ok := paired[i]; ok && e != nil { + t := e.Tracks[p.Track] + pl.Title, pl.Disc, pl.Track, pl.Recording, format = t.Title, max(t.Medium, 1), t.Position, t.RecordingMBID, t.Format + pl.Tracks = perDisc[t.Medium] + } + if pl.Title == "" { + pl.Title = strings.TrimSuffix(filepath.Base(f.Path), filepath.Ext(f.Path)) + } + d := dir + if media > 1 { + if format == "" { + format = "Disc" + } + d = filepath.Join(dir, Segment(fmt.Sprintf("%s %02d", format, pl.Disc))) + } + name := fmt.Sprintf("%s - %s - %02d - %s", w.Artist, w.Title, pl.Track, pl.Title) + pl.Dest = filepath.Join(d, Segment(name)+strings.ToLower(filepath.Ext(f.Path))) + out = append(out, pl) + } + return out +} + +// segmentMax keeps a folder or file name inside the 255 bytes most +// filesystems allow, with room for an extension and a temp prefix. +const segmentMax = 200 + +var illegal = strings.NewReplacer( + `\`, "+", "/", "+", "<", "", ">", "", "?", "!", "*", "-", "|", "", `"`, "", + // The "smart" colon: "Title: Subtitle" reads "Title - Subtitle". + ": ", " - ", ":", "-", +) + +// Segment makes a name safe as one path segment, the way the *arr programs +// clean file names: their illegal-character mapping, no leading or trailing +// spaces or dots, at most segmentMax bytes, never "" or "..". +func Segment(s string) string { + s = illegal.Replace(s) + s = strings.Map(func(r rune) rune { + if r < 0x20 || r == 0x7f { + return -1 + } + return r + }, s) + s = strings.Trim(s, " .") + for len(s) > segmentMax { + _, size := utf8.DecodeLastRuneInString(s) + s = strings.TrimRight(s[:len(s)-size], " .") + } + if s == "" { + return "_" + } + return s +} diff --git a/internal/importer/place.go b/internal/importer/place.go new file mode 100644 index 00000000..090091ea --- /dev/null +++ b/internal/importer/place.go @@ -0,0 +1,210 @@ +package importer + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "time" +) + +// ErrOccupied: the destination already has a file. Nothing is overwritten. +var ErrOccupied = errors.New("importer: destination already holds a file") + +// tagTimeout bounds one ffmpeg run. A remux copies the audio untouched, so a +// whole album track takes well under a second on local disk; two minutes is +// for a slow network mount, not for ffmpeg. +const tagTimeout = 2 * time.Minute + +// Tags are the names written into an imported file. Empty values are left +// out; the file's own tags stay for everything not named here. +type Tags struct { + Title, Artist, Album, AlbumArtist string + Year int + Disc, Discs, Track, Tracks int + // MusicBrainz ids, written the way Picard writes them. + ReleaseMBID, GroupMBID, RecordingMBID, ArtistMBID string +} + +// args turns the tags into ffmpeg -metadata pairs for a file of this +// extension. Vorbis comments (FLAC, Ogg, Opus) take Picard's key names; +// ID3 takes Picard's TXXX descriptions, which ffmpeg writes for any key it +// has no frame for. MP4's freeform atoms ffmpeg cannot write, so an M4A +// gets the plain names only. +// +// A Vorbis track or disc number is the number alone, its count in +// TRACKTOTAL and DISCTOTAL, as Picard writes them: dhowden/tag reads a +// TRACKNUMBER of "3/20" as 0, which would lose the album's order. ID3 and MP4 +// take "3/20", which their readers parse. +func (t Tags) args(ext string) []string { + vorbis := ext == ".flac" || ext == ".ogg" || ext == ".opus" + track, disc := ofN(t.Track, t.Tracks), ofN(t.Disc, t.Discs) + if vorbis { + track, disc = ofN(t.Track, 0), ofN(t.Disc, 0) + } + kv := [][2]string{ + {"title", t.Title}, {"artist", t.Artist}, {"album", t.Album}, {"album_artist", t.AlbumArtist}, + {"track", track}, {"disc", disc}, + } + if t.Year > 0 { + kv = append(kv, [2]string{"date", strconv.Itoa(t.Year)}) + } + switch ext { + case ".flac", ".ogg", ".opus": + if t.Track > 0 && t.Tracks > 0 { + kv = append(kv, [2]string{"TRACKTOTAL", strconv.Itoa(t.Tracks)}) + } + if t.Disc > 0 && t.Discs > 0 { + kv = append(kv, [2]string{"DISCTOTAL", strconv.Itoa(t.Discs)}) + } + kv = append(kv, + [2]string{"MUSICBRAINZ_ALBUMID", t.ReleaseMBID}, + [2]string{"MUSICBRAINZ_RELEASEGROUPID", t.GroupMBID}, + [2]string{"MUSICBRAINZ_TRACKID", t.RecordingMBID}, + [2]string{"MUSICBRAINZ_ARTISTID", t.ArtistMBID}, + [2]string{"MUSICBRAINZ_ALBUMARTISTID", t.ArtistMBID}) + case ".mp3": + kv = append(kv, + [2]string{"MusicBrainz Album Id", t.ReleaseMBID}, + [2]string{"MusicBrainz Release Group Id", t.GroupMBID}, + [2]string{"MusicBrainz Track Id", t.RecordingMBID}, + [2]string{"MusicBrainz Artist Id", t.ArtistMBID}, + [2]string{"MusicBrainz Album Artist Id", t.ArtistMBID}) + } + var out []string + for _, p := range kv { + if p[1] != "" { + out = append(out, "-metadata", p[0]+"="+p[1]) + } + } + return out +} + +func ofN(n, of int) string { + switch { + case n <= 0: + return "" + case of > 0: + return fmt.Sprintf("%d/%d", n, of) + } + return strconv.Itoa(n) +} + +// taggable: the formats ffmpeg can rewrite tags in by remuxing. +var taggable = map[string]bool{".flac": true, ".ogg": true, ".opus": true, ".mp3": true, ".m4a": true} + +// Place copies src to dst with tags written, leaving src as it was (the +// download keeps seeding). The copy is written to a dot-named temp file +// beside dst, which the scanner skips, and renamed into place, so the +// library never sees half a file. Tags are written by an ffmpeg remux that +// copies the audio untouched; when ffmpeg cannot (it is missing, or the +// container will not take the file's streams) the file is copied as it is +// and tagged reports false. An existing dst is never overwritten. +func Place(ctx context.Context, src, dst string, t Tags) (tagged bool, err error) { + return placeFile(ctx, src, dst, t, false) +} + +// Overwrite is Place onto a file an upgrade replaces: the new copy +// takes the old one's path in one rename, so the library never sees the +// path empty, and the track row at that path, history and all, carries on +// with the new file. +func Overwrite(ctx context.Context, src, dst string, t Tags) (tagged bool, err error) { + return placeFile(ctx, src, dst, t, true) +} + +func placeFile(ctx context.Context, src, dst string, t Tags, overwrite bool) (tagged bool, err error) { + if _, err := os.Lstat(dst); err == nil && !overwrite { + return false, fmt.Errorf("%w: %s", ErrOccupied, dst) + } + dir := filepath.Dir(dst) + if err := os.MkdirAll(dir, 0o755); err != nil { + return false, fmt.Errorf("make %s: %w", dir, err) + } + ext := strings.ToLower(filepath.Ext(dst)) + tmp := filepath.Join(dir, ".minstrel-import-"+randomHex()+ext) + defer func() { _ = os.Remove(tmp) }() // gone already once renamed + + if taggable[ext] { + tagged = remux(ctx, src, tmp, t.args(ext)) == nil + } + if !tagged { + if err := copyFile(src, tmp); err != nil { + return false, err + } + } + if err := os.Chmod(tmp, 0o644); err != nil { + return false, fmt.Errorf("chmod %s: %w", tmp, err) + } + if _, err := os.Lstat(dst); err == nil && !overwrite { + return false, fmt.Errorf("%w: %s", ErrOccupied, dst) + } + if err := os.Rename(tmp, dst); err != nil { + return false, fmt.Errorf("move into place %s: %w", dst, err) + } + return tagged, nil +} + +// remux writes src to dst through ffmpeg with every stream copied as it is +// (the cover art too) and the tags set. Its output is checked against the +// source's size: a remux shifts container bytes, never the audio, so a file +// much smaller than its source lost something on the way. +func remux(ctx context.Context, src, dst string, meta []string) error { + ctx, cancel := context.WithTimeout(ctx, tagTimeout) + defer cancel() + args := append([]string{"-nostdin", "-v", "error", "-y", "-i", src, + "-map", "0", "-c", "copy", "-map_metadata", "0"}, meta...) + args = append(args, dst) + cmd := exec.CommandContext(ctx, "ffmpeg", args...) + if out, err := cmd.CombinedOutput(); err != nil { + _ = os.Remove(dst) + return fmt.Errorf("ffmpeg: %w: %s", err, strings.TrimSpace(string(out))) + } + si, err := os.Stat(src) + if err != nil { + return err + } + di, err := os.Stat(dst) + if err != nil { + return err + } + if di.Size() < si.Size()/2 { + _ = os.Remove(dst) + return fmt.Errorf("ffmpeg wrote %d bytes from %d", di.Size(), si.Size()) + } + return nil +} + +// copyFile copies src to dst byte for byte and syncs it. +func copyFile(src, dst string) error { + in, err := os.Open(src) + if err != nil { + return err + } + defer func() { _ = in.Close() }() + out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + _ = out.Close() + return fmt.Errorf("copy %s: %w", src, err) + } + if err := out.Sync(); err != nil { + _ = out.Close() + return err + } + return out.Close() +} + +func randomHex() string { + b := make([]byte, 6) + _, _ = rand.Read(b) + return hex.EncodeToString(b) +} diff --git a/internal/importer/upgrade.go b/internal/importer/upgrade.go new file mode 100644 index 00000000..d40f9c08 --- /dev/null +++ b/internal/importer/upgrade.go @@ -0,0 +1,304 @@ +package importer + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/library" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// Importing an upgrade: a better copy of an album the library +// holds. Each file is matched to the track it would replace, by recording +// and then by disc and track. It replaces that track when the track is a +// video rip or the new file is a better tier; otherwise the old track stays. +// A file with no track to replace fills a gap, as any import does. +// +// The new copy is filed before the old one goes. On the old file's own path +// it takes the path in one rename and the row carries on; at a new path it +// is scanned in as a track of its own and the old track is folded into it +// (library.ReplaceTrack), so likes, plays and playlist entries carry over. + +// OldTrack is a library track an upgrade may replace. +type OldTrack struct { + ID pgtype.UUID + Path string + Recording string + Disc, Track int + Format string + Bytes int64 + Ms int64 + Rip bool +} + +// Tier is the old track's tier, estimated as the upgrade check estimates +// an album's; ok is false for a format the ladder cannot place. +func (o OldTrack) Tier() (release.Tier, bool) { + return release.TierOfFile(o.Format, release.FileKbps(o.Bytes, o.Ms)) +} + +// What an upgrade does with one placement. +const ( + stepPlace = "place" // no track to replace: fills a gap + stepReplace = "replace" // replaces Old from a new path + stepOverwrite = "overwrite" // replaces Old on its own path + stepKeep = "keep" // Old is as good: left as it is +) + +type upgradeStep struct { + Placement int + Action string + Old int // index into olds, or -1 +} + +// planUpgrade matches each placement to the track it would replace and +// decides what to do with it. newTiers is each placement's tier. +func planUpgrade(plan []Placement, newTiers []release.Tier, olds []OldTrack) []upgradeStep { + used := make([]bool, len(olds)) + match := make([]int, len(plan)) + for i := range match { + match[i] = -1 + } + for i, pl := range plan { + if pl.Recording == "" { + continue + } + for j, o := range olds { + if !used[j] && o.Recording == pl.Recording { + match[i], used[j] = j, true + break + } + } + } + for i, pl := range plan { + if match[i] >= 0 || pl.Track <= 0 { + continue + } + for j, o := range olds { + if !used[j] && o.Track == pl.Track && max(o.Disc, 1) == max(pl.Disc, 1) { + match[i], used[j] = j, true + break + } + } + } + steps := make([]upgradeStep, len(plan)) + for i, pl := range plan { + j := match[i] + steps[i] = upgradeStep{Placement: i, Old: j, Action: stepPlace} + if j < 0 { + continue + } + o := olds[j] + oldTier, _ := o.Tier() + better := release.Rank(newTiers[i]) > release.Rank(oldTier) + switch { + case !o.Rip && !better: + steps[i].Action = stepKeep + case filepath.Clean(pl.Dest) == filepath.Clean(o.Path): + steps[i].Action = stepOverwrite + default: + steps[i].Action = stepReplace + } + } + return steps +} + +// fileTier estimates a downloaded file's tier from its size and length, the +// way the library's are estimated, so the two compare like for like. A file +// that cannot be measured takes the release's tier. +func fileTier(f File, fallback release.Tier) release.Tier { + if info, err := os.Stat(f.Path); err == nil && f.LengthMs > 0 { + if t, ok := release.TierOfFile(filepath.Ext(f.Path), release.FileKbps(info.Size(), int64(f.LengthMs))); ok { + return t + } + } + return fallback +} + +func oldTracks(rows []dbq.ListUpgradeTracksRow) []OldTrack { + out := make([]OldTrack, 0, len(rows)) + for _, r := range rows { + o := OldTrack{ID: r.ID, Path: r.FilePath, Disc: int(r.Disc), Format: r.FileFormat, + Bytes: r.FileSize, Ms: int64(r.DurationMs), Rip: r.SourceVerdict != nil && *r.SourceVerdict == "suspect"} + if r.Mbid != nil { + o.Recording = *r.Mbid + } + if r.TrackNumber != nil { + o.Track = int(*r.TrackNumber) + } + out = append(out, o) + } + return out +} + +// placeUpgrade files an accepted upgrade, replacing the tracks it betters. +func (im *Importer) placeUpgrade(ctx context.Context, q *dbq.Queries, g dbq.Grab, d downloader.Driver, w Want, + e *Edition, files []File, plan []Placement, v Verdict, det Detail, record recordFunc) string { + rows, err := q.ListUpgradeTracks(ctx, w.AlbumID) + if err != nil { + return record(StateWaiting, "Couldn't read the album's tracks: "+err.Error(), "", det, 0, "") + } + olds := oldTracks(rows) + grabTier := release.Tier(g.Quality) + newTiers := make([]release.Tier, len(plan)) + for i, pl := range plan { + newTiers[i] = fileTier(files[pl.File], grabTier) + } + steps := planUpgrade(plan, newTiers, olds) + heldRec, heldPos, err := im.held(ctx, q, w, plan) + if err != nil { + return record(StateWaiting, "Couldn't read what the library holds: "+err.Error(), "", det, 0, "") + } + + type swap struct { + old OldTrack + dest string + } + var ( + placed []string // every file written, for the scan + swaps []swap + inPlace []OldTrack + failed []string + ) + for _, st := range steps { + pl := plan[st.Placement] + if st.Action == stepKeep { + det.Skipped++ + continue + } + if st.Action == stepPlace && ((pl.Recording != "" && heldRec[pl.Recording]) || heldPos[[2]int{pl.Disc, pl.Track}]) { + det.Skipped++ + continue + } + tags := Tags{Title: pl.Title, Artist: w.Artist, Album: w.Title, AlbumArtist: w.Artist, Year: w.Year, + Disc: pl.Disc, Discs: pl.Discs, Track: pl.Track, Tracks: pl.Tracks, + GroupMBID: w.GroupMBID, RecordingMBID: pl.Recording, ArtistMBID: w.ArtistMBID} + if e != nil { + tags.ReleaseMBID = e.MBID + } + src := files[pl.File].Path + var tagged bool + if st.Action == stepOverwrite { + tagged, err = Overwrite(ctx, src, pl.Dest, tags) + } else { + tagged, err = Place(ctx, src, pl.Dest, tags) + } + switch { + case errors.Is(err, ErrOccupied): + det.Skipped++ + continue + case err != nil: + // Each replacement stands alone: the old track stays where a + // copy failed, and the rest go ahead. + failed = append(failed, pl.Title) + im.logger.Warn("import: copy an upgrade into the library", "grab", g.Name, "dest", pl.Dest, "err", err) + continue + } + placed = append(placed, pl.Dest) + if !tagged { + det.Untagged++ + } + switch st.Action { + case stepOverwrite: + inPlace = append(inPlace, olds[st.Old]) + case stepReplace: + swaps = append(swaps, swap{old: olds[st.Old], dest: pl.Dest}) + } + } + det.Placed = len(placed) + + if len(placed) == 0 { + why := fmt.Sprintf("Nothing in it is better than the copy of %s you have, so nothing was imported.", w.Title) + if len(failed) > 0 { + return record(StateFailed, fmt.Sprintf("Copying into the library failed for %s.", strings.Join(failed, ", ")), + "", det, 0, "") + } + // Set aside, so the weekly search doesn't take it again. + if err := acquire.NewMemory(im.pool).Block(ctx, dbq.BlockReleaseParams{ + InfoHash: g.InfoHash, Name: g.Name, Reason: acquire.BlockImport, Detail: why, WantID: g.WantID, + }); err != nil { + im.logger.Warn("import: set the release aside", "grab", g.Name, "err", err) + } + if err := q.MarkWantSearched(ctx, g.WantID); err != nil { + im.logger.Warn("import: mark the want searched", "grab", g.Name, "err", err) + } + im.removeFromClient(ctx, d, g) + return record(StateHeld, why, StateHeld, det, 0, "") + } + + if _, err := im.Scanner.ScanFiles(ctx, placed); err != nil { + im.logger.Warn("import: scan the new files", "grab", g.Name, "err", err) + } + for _, o := range inPlace { + if o.Rip { + if err := q.ClearTrackSourceVerdict(ctx, o.ID); err != nil { + im.logger.Warn("import: release a replaced rip", "path", o.Path, "err", err) + } + } + } + replaced, kept := len(inPlace), 0 + for _, s := range swaps { + nt, err := q.GetTrackByPath(ctx, s.dest) + if err != nil { + kept++ + im.logger.Warn("import: find the new track", "path", s.dest, "err", err) + continue + } + moved, err := library.ReplaceTrack(ctx, im.pool, im.logger, im.DataDir, s.old.ID, nt.ID) + if err != nil { + // The old file stays; the duplicate sweep finds the two copies. + kept++ + im.logger.Warn("import: replace the old track", "old", s.old.Path, "new", s.dest, "err", err) + continue + } + replaced++ + im.logger.Info("import: replaced a track with a better copy", "old", s.old.Path, "new", s.dest, + "play_events", moved.PlayEvents, "likes", moved.Likes, "playlist_entries", moved.PlaylistEntries) + } + + // The want holds the album the new tracks are in, once the old album row + // has none left. + var album pgtype.UUID + if left, err := q.ListUpgradeTracks(ctx, w.AlbumID); err == nil && len(left) == 0 { + if nt, err := q.GetTrackByPath(ctx, placed[0]); err == nil { + album = nt.AlbumID + } + } + if err := q.FinishUpgrade(ctx, dbq.FinishUpgradeParams{ID: g.WantID, AlbumID: album}); err != nil { + im.logger.Warn("import: finish the upgrade", "grab", g.Name, "err", err) + } + if err := q.MarkWantSearched(ctx, g.WantID); err != nil { + im.logger.Warn("import: mark the want searched", "grab", g.Name, "err", err) + } + + dest := filepath.Dir(placed[0]) + if e != nil && e.Media() > 1 { + dest = filepath.Dir(dest) + } + reason := fmt.Sprintf("Replaced %s of %s with a better copy.", tracks(replaced), w.Title) + if added := len(placed) - replaced - kept; added > 0 { + reason += fmt.Sprintf(" Added %s it was missing.", tracks(added)) + } + if det.Skipped > 0 { + reason += fmt.Sprintf(" %s already as good left as they were.", tracks(det.Skipped)) + } + if kept > 0 { + reason += fmt.Sprintf(" %s couldn't be removed, so both copies are in the library.", tracks(kept)) + } + if len(failed) > 0 { + reason += fmt.Sprintf(" Copying failed for %s; those stay as they were.", strings.Join(failed, ", ")) + } + if det.Untagged > 0 { + reason += fmt.Sprintf(" %s copied with their own tags: ffmpeg could not rewrite them.", tracks(det.Untagged)) + } + return record(StateImported, reason+" "+v.Reason, StateImported, det, len(placed), dest) +} diff --git a/internal/importer/upgrade_test.go b/internal/importer/upgrade_test.go new file mode 100644 index 00000000..a415fd45 --- /dev/null +++ b/internal/importer/upgrade_test.go @@ -0,0 +1,53 @@ +package importer + +import ( + "testing" + + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +func TestPlanUpgrade_ReplacesOnlyWhatItBetters(t *testing.T) { + plan := []Placement{ + {File: 0, Dest: "/music/A/B (2001)/A - B - 01 - One.flac", Disc: 1, Track: 1, Recording: "rec-1"}, + {File: 1, Dest: "/music/A/B (2001)/A - B - 02 - Two.mp3", Disc: 1, Track: 2}, + {File: 2, Dest: "/music/A/B (2001)/A - B - 03 - Three.flac", Disc: 1, Track: 3}, + {File: 3, Dest: "/music/A/B (2001)/A - B - 04 - Four.flac", Disc: 1, Track: 4}, + {File: 4, Dest: "/music/A/B (2001)/A - B - 05 - Five.flac", Disc: 1, Track: 5}, + } + newTiers := []release.Tier{release.TierLossless, release.TierLossyHigh, release.TierLossless, + release.TierLossless, release.TierLossyLow} + olds := []OldTrack{ + // Matched by recording though its number is wrong: a 192 MP3. + {Path: "/music/A/B/01 One.mp3", Recording: "rec-1", Disc: 1, Track: 9, Format: "mp3", Bytes: 4_800_000, Ms: 200_000}, + // Same path as the new file, a 320 MP3: the new 320 is no better. + {Path: "/music/A/B (2001)/A - B - 02 - Two.mp3", Disc: 1, Track: 2, Format: "mp3", Bytes: 8_000_000, Ms: 200_000}, + // Already lossless: kept. + {Path: "/music/A/B/03 Three.flac", Disc: 1, Track: 3, Format: "flac", Bytes: 22_000_000, Ms: 200_000}, + // A video rip, even though lossless: replaced. + {Path: "/music/A/B/04 Four (Official Video).flac", Disc: 0, Track: 4, Format: "flac", Bytes: 22_000_000, Ms: 200_000, Rip: true}, + } + steps := planUpgrade(plan, newTiers, olds) + want := []struct { + action string + old int + }{ + {stepReplace, 0}, {stepKeep, 1}, {stepKeep, 2}, {stepReplace, 3}, {stepPlace, -1}, + } + for i, w := range want { + if steps[i].Action != w.action || steps[i].Old != w.old { + t.Errorf("placement %d: %s old %d, want %s old %d", i, steps[i].Action, steps[i].Old, w.action, w.old) + } + } +} + +func TestPlanUpgrade_OverwritesAWorseFileOnItsOwnPath(t *testing.T) { + dest := "/music/A/B (2001)/A - B - 01 - One.mp3" + steps := planUpgrade( + []Placement{{File: 0, Dest: dest, Disc: 1, Track: 1}}, + []release.Tier{release.TierLossyHigh}, + []OldTrack{{Path: dest, Disc: 1, Track: 1, Format: "mp3", Bytes: 3_200_000, Ms: 200_000}}, // 128 + ) + if steps[0].Action != stepOverwrite || steps[0].Old != 0 { + t.Errorf("got %+v", steps[0]) + } +} diff --git a/internal/importer/verify.go b/internal/importer/verify.go new file mode 100644 index 00000000..9181d46c --- /dev/null +++ b/internal/importer/verify.go @@ -0,0 +1,437 @@ +// Package importer checks a finished download against the album it was +// grabbed for and files it into the library. Nothing reaches +// the library before the check says the files are the album: a wrong album, +// another version of it, a video rip or an incomplete copy is refused, its +// release is blocked and the album is searched for again. +// +// The files are matched to one of the album's MusicBrainz editions by +// AcoustID fingerprints, track lengths, titles and the tags they carry. +package importer + +import ( + "fmt" + "slices" + "sort" + "strings" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// Outcome is what the check decided. +type Outcome string + +const ( + Accept Outcome = "accept" + // Review: the files may be the album and the evidence cannot say. A + // person decides. + Review Outcome = "review" + Refuse Outcome = "refuse" +) + +// Codes say why, beside the sentence in Verdict.Reason. +const ( + CodeMatched = "matched" + CodeForced = "forced" + CodeNoAudio = "no_audio" + CodeVideo = "video" + CodeWrongAlbum = "wrong_album" + CodeOtherVersion = release.VetoVersion + CodeWrongTitle = release.VetoTitle + CodeIncomplete = "incomplete" + CodeNoEdition = "no_edition" + CodeWeak = "weak" + CodeTooMany = "too_many" +) + +// maxFiles is the most audio files one album is checked for. More is a box +// set or a discography, which is a person's call, not a fingerprint queue. +const maxFiles = 60 + +// File is what one audio file in a download says about itself. +type File struct { + Path string // absolute, as Minstrel sees it + Rel string // under the download, for the person reading the verdict + // Disc and Track are the tags' positions; 0 when untagged. + Disc, Track int + Title string + Artist string + AlbumArtist string + Album string + // MusicBrainz ids from the tags, when a tagger wrote them. + GroupMBID string + ReleaseMBID string + RecordingMBID string + LengthMs int + // Recordings are the MusicBrainz recordings AcoustID says the audio is, + // at or above the configured minimum score. + Recordings []string + // Video: the file name says it is a music video, a lyric video or the + // like (library.SourceMarkersFor). + Video bool +} + +// Track is one track of an edition. +type Track struct { + Medium int + Format string // the medium's: CD, Digital Media, Vinyl + Position int + Title string + RecordingMBID string + LengthMs int +} + +// Edition is one MusicBrainz release of the wanted release group, its +// tracks in medium and position order. +type Edition struct { + MBID string + Title string + Status string + Date string + Country string + Tracks []Track +} + +// Media is how many media the edition has. +func (e Edition) Media() int { + n := 0 + for _, t := range e.Tracks { + n = max(n, t.Medium) + } + return n +} + +// Want is the album the download was grabbed for. +type Want struct { + GroupMBID string + ArtistMBID string + Artist string + Title string + Year int + PrimaryType string + SecondaryTypes []string + // Upgrade: a held album wanted for a better copy, quality or + // rip; AlbumID is the library album it holds, whose tracks the import + // replaces. + Upgrade string + AlbumID pgtype.UUID +} + +// Pair is a file matched to an edition's track, and what agreed. +type Pair struct { + File int `json:"file"` + Track int `json:"track"` + Recording bool `json:"recording"` + Length bool `json:"length"` + Title bool `json:"title"` +} + +// Evidence is what the check counted, kept with the verdict so a person can +// see what it rested on. +type Evidence struct { + Files int `json:"files"` + Tracks int `json:"tracks"` + // Identified: files with a recording from AcoustID or their tags. + Identified int `json:"identified"` + Recordings int `json:"recordings"` + Lengths int `json:"lengths"` + Titles int `json:"titles"` + // GroupTag: "same" when the tags name this release group, "other" when + // another, "" when they name none. + GroupTag string `json:"group_tag,omitempty"` + AlbumTag string `json:"album_tag,omitempty"` + // Fingerprinted: AcoustID was asked. False when it is not set up or + // could not be reached; the check then rests on lengths and tags. + Fingerprinted bool `json:"fingerprinted"` +} + +// Verdict is the check's answer. +type Verdict struct { + Outcome Outcome `json:"outcome"` + Code string `json:"code"` + Reason string `json:"reason"` + // Edition indexes the editions passed in; -1 when none was chosen. + Edition int `json:"-"` + Pairs []Pair `json:"pairs,omitempty"` + Evidence Evidence `json:"evidence"` +} + +// Verify decides whether files are the wanted album and, if so, which +// edition and which track each file is. force is a person's "import anyway": +// the best edition is taken whatever the evidence, but a download with no +// audio is still refused. +func Verify(w Want, editions []Edition, files []File, fingerprinted, force bool) Verdict { + v := Verdict{Edition: -1, Evidence: Evidence{Files: len(files), Fingerprinted: fingerprinted}} + if len(files) == 0 { + return v.set(Refuse, CodeNoAudio, "The download holds no audio files.") + } + if !force && len(files) > maxFiles { + return v.set(Review, CodeTooMany, fmt.Sprintf( + "The download holds %d audio files, more than one album; a person decides.", len(files))) + } + if video := countIf(files, func(f File) bool { return f.Video }); !force && video*2 > len(files) { + return v.set(Refuse, CodeVideo, fmt.Sprintf( + "%d of its %d audio files are named as music or lyric videos: the sound of a video, not the album.", video, len(files))) + } + + group := majority(files, func(f File) string { return f.GroupMBID }) + switch { + case group == "": + case strings.EqualFold(group, w.GroupMBID): + v.Evidence.GroupTag = "same" + default: + v.Evidence.GroupTag = "other" + } + album := majority(files, func(f File) string { return f.Album }) + v.Evidence.AlbumTag = album + var tagVeto release.Match + albumAgrees := false + if album != "" { + // The want's own artist goes in front: the tags' artist spelling is + // not what this asks. Only the album title and its version words. + m := release.Judge(release.Want{Artist: w.Artist, Title: w.Title, PrimaryType: w.PrimaryType, + SecondaryTypes: w.SecondaryTypes}, release.Parse(w.Artist+" - "+album)) + switch m.Veto { + case release.VetoVersion, release.VetoTitle: + tagVeto = m + case "": + albumAgrees = true + } + } + if !force && v.Evidence.GroupTag == "other" { + return v.set(Refuse, CodeWrongAlbum, fmt.Sprintf( + "Its tags name a different MusicBrainz release group (%s), not %s.", group, w.Title)) + } + + if len(editions) == 0 { + if force { + v.Code, v.Outcome, v.Reason = CodeForced, Accept, "Imported anyway at a person's word; MusicBrainz lists no edition to match it to." + return v + } + return v.set(Review, CodeNoEdition, fmt.Sprintf( + "MusicBrainz lists no edition of %s to check the files against; a person decides.", w.Title)) + } + + best, bestScore := -1, -1 + shortest, longest := len(editions[0].Tracks), 0 + for i, e := range editions { + shortest, longest = min(shortest, len(e.Tracks)), max(longest, len(e.Tracks)) + if len(e.Tracks) != len(files) && !force { + continue + } + pairs := pairUp(files, e.Tracks) + if s := score(pairs); s > bestScore || (s == bestScore && better(e, editions[best])) { + best, bestScore = i, s + } + } + if best < 0 { + if len(files) < shortest { + return v.set(Refuse, CodeIncomplete, fmt.Sprintf( + "It holds %d audio files; the shortest edition of %s has %d tracks.", len(files), w.Title, shortest)) + } + return v.set(Review, CodeNoEdition, fmt.Sprintf( + "It holds %d audio files; no edition of %s has %d tracks (they have %s). A person decides.", + len(files), w.Title, len(files), span(shortest, longest))) + } + + e := editions[best] + v.Edition = best + v.Pairs = pairUp(files, e.Tracks) + n := len(e.Tracks) + ev := &v.Evidence + ev.Tracks = n + for _, p := range v.Pairs { + f := files[p.File] + if f.RecordingMBID != "" || len(f.Recordings) > 0 { + ev.Identified++ + } + ev.Recordings += b2i(p.Recording) + ev.Lengths += b2i(p.Length) + ev.Titles += b2i(p.Title) + } + if force { + v.Outcome, v.Code = Accept, CodeForced + v.Reason = fmt.Sprintf("Imported anyway at a person's word, as %s. %s", describe(e), ev.sentence()) + return v + } + + half := (n + 1) / 2 + strong := ev.GroupTag == "same" || (ev.Identified >= half && ev.Recordings*10 >= ev.Identified*7) + switch { + case ev.Identified >= half && ev.Recordings*10 < ev.Identified*3 && ev.GroupTag != "same": + return v.set(Refuse, CodeWrongAlbum, fmt.Sprintf( + "Only %d of the %d tracks AcoustID knows are recordings on %s.", ev.Recordings, ev.Identified, w.Title)) + case tagVeto.Veto != "" && !strong: + return v.set(Refuse, tagVeto.Veto, fmt.Sprintf("Its tags call it %q. %s", album, tagVeto.Reason)) + case !strong && ev.Identified == 0 && ev.Lengths*2 < n: + return v.set(Refuse, CodeWrongAlbum, fmt.Sprintf( + "Only %d of its %d tracks are the length of %s's, and nothing else says it is the album.", ev.Lengths, n, w.Title)) + case strong && ev.Lengths*10 >= n*6, + !strong && ev.Lengths*10 >= n*8 && (ev.Titles*10 >= n*6 || albumAgrees): + v.Outcome, v.Code = Accept, CodeMatched + v.Reason = fmt.Sprintf("It is %s. %s", describe(e), ev.sentence()) + return v + } + return v.set(Review, CodeWeak, fmt.Sprintf( + "It may be %s, but the evidence is thin: %s A person decides.", describe(e), ev.sentence())) +} + +func (v Verdict) set(o Outcome, code, reason string) Verdict { + v.Outcome, v.Code, v.Reason = o, code, reason + return v +} + +// sentence says what agreed, for the verdict's reason. +func (ev Evidence) sentence() string { + var parts []string + if ev.GroupTag == "same" { + parts = append(parts, "its tags name this album") + } + if ev.Identified > 0 { + parts = append(parts, fmt.Sprintf("%d of %d identified tracks are its recordings", ev.Recordings, ev.Identified)) + } + parts = append(parts, fmt.Sprintf("%d of %d lengths fit", ev.Lengths, ev.Tracks), + fmt.Sprintf("%d of %d titles agree", ev.Titles, ev.Tracks)) + s := strings.Join(parts, ", ") + s = strings.ToUpper(s[:1]) + s[1:] + "." + if !ev.Fingerprinted { + s += " AcoustID was not asked (not set up or unreachable), so lengths and tags decided." + } + return s +} + +// describe names an edition for a person: "the 2013 CD release (13 tracks)". +func describe(e Edition) string { + var b strings.Builder + b.WriteString("the ") + if len(e.Date) >= 4 { + b.WriteString(e.Date[:4] + " ") + } + if f := e.format(); f != "" { + b.WriteString(f + " ") + } + fmt.Fprintf(&b, "edition (%d tracks)", len(e.Tracks)) + return b.String() +} + +func (e Edition) format() string { + if len(e.Tracks) == 0 { + return "" + } + f := e.Tracks[0].Format + for _, t := range e.Tracks { + if t.Format != f { + return "" + } + } + return f +} + +// pairUp matches files to tracks in order: by the tags' disc and track +// numbers when every file has them and no two share a place, by file path +// otherwise. With a person's "import anyway" the counts may differ; pairs +// run to the shorter. +func pairUp(files []File, tracks []Track) []Pair { + order := make([]int, len(files)) + for i := range order { + order[i] = i + } + if numbered(files) { + sort.SliceStable(order, func(a, b int) bool { + fa, fb := files[order[a]], files[order[b]] + if da, db := max(fa.Disc, 1), max(fb.Disc, 1); da != db { + return da < db + } + return fa.Track < fb.Track + }) + } else { + sort.SliceStable(order, func(a, b int) bool { return files[order[a]].Rel < files[order[b]].Rel }) + } + n := min(len(files), len(tracks)) + pairs := make([]Pair, n) + for i := range n { + f, t := files[order[i]], tracks[i] + pairs[i] = Pair{ + File: order[i], + Track: i, + Recording: t.RecordingMBID != "" && (f.RecordingMBID == t.RecordingMBID || slices.Contains(f.Recordings, t.RecordingMBID)), + Length: release.LengthFits(f.LengthMs, t.LengthMs), + Title: release.TitleAgrees(f.Title, t.Title), + } + } + return pairs +} + +// numbered: every file carries a track number and no two share a disc and +// track. +func numbered(files []File) bool { + seen := map[[2]int]bool{} + for _, f := range files { + k := [2]int{max(f.Disc, 1), f.Track} + if f.Track <= 0 || seen[k] { + return false + } + seen[k] = true + } + return true +} + +// score weighs recordings above lengths above titles: a fingerprint is the +// audio itself, a length is the audio's shape, a title is only a name. +func score(pairs []Pair) int { + s := 0 + for _, p := range pairs { + s += 3*b2i(p.Recording) + 2*b2i(p.Length) + b2i(p.Title) + } + return s +} + +// better breaks a tie between equally good editions: an official one, then +// the earlier. +func better(a, b Edition) bool { + if (a.Status == "Official") != (b.Status == "Official") { + return a.Status == "Official" + } + return a.Date != "" && (b.Date == "" || a.Date < b.Date) +} + +func majority(files []File, get func(File) string) string { + counts := map[string]int{} + best, n := "", 0 + for _, f := range files { + v := strings.TrimSpace(get(f)) + if v == "" { + continue + } + counts[strings.ToLower(v)]++ + if c := counts[strings.ToLower(v)]; c > n { + best, n = v, c + } + } + return best +} + +func countIf(files []File, f func(File) bool) int { + n := 0 + for _, x := range files { + if f(x) { + n++ + } + } + return n +} + +func span(a, b int) string { + if a == b { + return fmt.Sprint(a) + } + return fmt.Sprintf("%d to %d", a, b) +} + +func b2i(b bool) int { + if b { + return 1 + } + return 0 +} diff --git a/internal/importer/verify_test.go b/internal/importer/verify_test.go new file mode 100644 index 00000000..e3208fb7 --- /dev/null +++ b/internal/importer/verify_test.go @@ -0,0 +1,239 @@ +package importer + +import ( + "fmt" + "path/filepath" + "strings" + "testing" +) + +var ram = Want{GroupMBID: "rg-ram", ArtistMBID: "mb-dp", Artist: "Daft Punk", Title: "Random Access Memories", + Year: 2013, PrimaryType: "Album"} + +var ramTitles = []string{"Give Life Back to Music", "The Game of Love", "Giorgio by Moroder", "Within", + "Instant Crush", "Lose Yourself to Dance", "Touch", "Get Lucky", "Beyond", "Motherboard", + "Fragments of Time", "Doin' It Right", "Contact"} + +// edition is RAM's 13-track CD, recordings rec-01.., lengths 4:00 + n seconds. +func edition(mbid string, n int) Edition { + e := Edition{MBID: mbid, Title: "Random Access Memories", Status: "Official", Date: "2013-05-17"} + for i := range n { + e.Tracks = append(e.Tracks, Track{Medium: 1, Format: "CD", Position: i + 1, Title: ramTitles[i%len(ramTitles)], + RecordingMBID: fmt.Sprintf("rec-%02d", i+1), LengthMs: 240000 + i*1000}) + } + return e +} + +// download is n files lined up with the edition: tagged positions, lengths +// within a second, AcoustID naming each file's recording. +func download(n int) []File { + var fs []File + for i := range n { + fs = append(fs, File{Path: fmt.Sprintf("/dl/%02d.flac", i+1), Rel: fmt.Sprintf("%02d.flac", i+1), + Disc: 1, Track: i + 1, Title: ramTitles[i%len(ramTitles)], Album: "Random Access Memories", + LengthMs: 240000 + i*1000 + 400, Recordings: []string{fmt.Sprintf("rec-%02d", i+1)}}) + } + return fs +} + +func TestVerify_TheAlbumAndWhatIsNot(t *testing.T) { + eds := []Edition{edition("rel-cd", 13), edition("rel-deluxe", 15)} + cases := []struct { + name string + files func() []File + printed bool + out Outcome + code string + edition string + }{ + {"the album, fingerprinted", func() []File { return download(13) }, true, Accept, CodeMatched, "rel-cd"}, + {"the deluxe by count", func() []File { return download(15) }, true, Accept, CodeMatched, "rel-deluxe"}, + {"no audio", func() []File { return nil }, true, Refuse, CodeNoAudio, ""}, + {"music videos", func() []File { + fs := download(13) + for i := range fs { + fs[i].Video = i < 10 + } + return fs + }, true, Refuse, CodeVideo, ""}, + {"tags name another group", func() []File { + fs := download(13) + for i := range fs { + fs[i].GroupMBID = "rg-homework" + } + return fs + }, true, Refuse, CodeWrongAlbum, ""}, + {"fingerprints say other recordings", func() []File { + fs := download(13) + for i := range fs { + fs[i].Recordings = []string{"rec-other"} + } + return fs + }, true, Refuse, CodeWrongAlbum, "rel-cd"}, + {"the drumless edition: right lengths, album tag says otherwise", func() []File { + fs := download(13) + for i := range fs { + fs[i].Recordings, fs[i].Album = nil, "Random Access Memories (Drumless Edition)" + } + return fs + }, false, Refuse, CodeOtherVersion, "rel-cd"}, + {"fewer files than any edition", func() []File { return download(9) }, true, Refuse, CodeIncomplete, ""}, + {"no edition this long", func() []File { return download(14) }, true, Review, CodeNoEdition, ""}, + {"no fingerprints, lengths and titles agree", func() []File { + fs := download(13) + for i := range fs { + fs[i].Recordings = nil + } + return fs + }, false, Accept, CodeMatched, "rel-cd"}, + {"no fingerprints, no titles, nothing to go on but lengths", func() []File { + fs := download(13) + for i := range fs { + fs[i].Recordings, fs[i].Title, fs[i].Album = nil, "", "" + } + return fs + }, false, Review, CodeWeak, "rel-cd"}, + {"no fingerprints, lengths all wrong", func() []File { + fs := download(13) + for i := range fs { + fs[i].Recordings, fs[i].LengthMs = nil, 600000 + } + return fs + }, false, Refuse, CodeWrongAlbum, "rel-cd"}, + {"tags name the group: strong even unprinted", func() []File { + fs := download(13) + for i := range fs { + fs[i].Recordings, fs[i].Title, fs[i].GroupMBID = nil, "", "rg-ram" + } + return fs + }, false, Accept, CodeMatched, "rel-cd"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + v := Verify(ram, eds, c.files(), c.printed, false) + if v.Outcome != c.out || v.Code != c.code { + t.Fatalf("got %s/%s (%s), want %s/%s", v.Outcome, v.Code, v.Reason, c.out, c.code) + } + got := "" + if v.Edition >= 0 { + got = eds[v.Edition].MBID + } + if got != c.edition { + t.Errorf("edition %q, want %q", got, c.edition) + } + if v.Reason == "" { + t.Error("no reason given") + } + }) + } +} + +func TestVerify_UnnumberedFilesPairByPath(t *testing.T) { + fs := download(13) + for i := range fs { + fs[i].Track, fs[i].Disc = 0, 0 + } + fs[0], fs[5] = fs[5], fs[0] // listed out of order; their names sort right + v := Verify(ram, []Edition{edition("rel-cd", 13)}, fs, true, false) + if v.Outcome != Accept || v.Evidence.Recordings != 13 { + t.Fatalf("got %+v", v) + } +} + +func TestVerify_ImportAnywayTakesTheBestEdition(t *testing.T) { + v := Verify(ram, []Edition{edition("rel-cd", 13)}, download(14), true, true) + if v.Outcome != Accept || v.Code != CodeForced || v.Edition != 0 || len(v.Pairs) != 13 { + t.Fatalf("got %+v", v) + } + if v := Verify(ram, nil, nil, true, true); v.Outcome != Refuse { + t.Errorf("forced with no audio: %+v", v) + } +} + +func TestVerify_WithoutAcoustIDTheReasonSaysSo(t *testing.T) { + fs := download(13) + for i := range fs { + fs[i].Recordings = nil + } + v := Verify(ram, []Edition{edition("rel-cd", 13)}, fs, false, false) + if !strings.Contains(v.Reason, "AcoustID was not asked") { + t.Errorf("reason %q", v.Reason) + } +} + +func TestPlan_LibraryNaming(t *testing.T) { + e := edition("rel-cd", 2) + e.Tracks[1].Title = "Within: Part 2/3?" + pl := Plan("/music", ram, &e, download(2), Verify(ram, []Edition{e}, download(2), true, false).Pairs) + want := []string{ + "/music/Daft Punk/Random Access Memories (2013)/Daft Punk - Random Access Memories - 01 - Give Life Back to Music.flac", + "/music/Daft Punk/Random Access Memories (2013)/Daft Punk - Random Access Memories - 02 - Within - Part 2+3!.flac", + } + for i, p := range pl { + if p.Dest != want[i] { + t.Errorf("dest %d = %q, want %q", i, p.Dest, want[i]) + } + } + if pl[0].Recording != "rec-01" || pl[0].Tracks != 2 { + t.Errorf("placement %+v", pl[0]) + } +} + +func TestPlan_MultiDiscGetsAMediumFolder(t *testing.T) { + e := edition("rel-2cd", 4) + e.Tracks[2].Medium, e.Tracks[2].Position = 2, 1 + e.Tracks[3].Medium, e.Tracks[3].Position = 2, 2 + fs := download(4) + fs[2].Disc, fs[2].Track, fs[3].Disc, fs[3].Track = 2, 1, 2, 2 + pl := Plan("/music", ram, &e, fs, Verify(ram, []Edition{e}, fs, true, false).Pairs) + if got := filepath.Dir(pl[3].Dest); got != "/music/Daft Punk/Random Access Memories (2013)/CD 02" { + t.Errorf("disc folder %q", got) + } + if !strings.HasSuffix(pl[3].Dest, " - 02 - Get Lucky.flac") && !strings.Contains(pl[3].Dest, " - 02 - ") { + t.Errorf("dest %q", pl[3].Dest) + } +} + +func TestSegment(t *testing.T) { + for in, want := range map[string]string{ + "AC/DC": "AC+DC", `What?`: "What!", "Title: Sub": "Title - Sub", "a:b": "a-b", " ..x.. ": "x", + "": "_", "..": "_", `"Quoted" *y* |z|`: "Quoted x -y- z", + } { + if got := Segment(in); got != want { + t.Errorf("Segment(%q) = %q, want %q", in, got, want) + } + } + if long := Segment(strings.Repeat("é", 200)); len(long) > segmentMax { + t.Errorf("long segment is %d bytes", len(long)) + } +} + +func TestTagsArgs(t *testing.T) { + tg := Tags{Title: "Get Lucky", Artist: "Daft Punk", Track: 8, Tracks: 13, Disc: 1, Discs: 1, Year: 2013, + RecordingMBID: "rec-08", GroupMBID: "rg-ram"} + flac := strings.Join(tg.args(".flac"), " ") + for _, s := range []string{ + "title=Get Lucky", "track=8 ", "TRACKTOTAL=13", "disc=1 ", "DISCTOTAL=1", "date=2013", + "MUSICBRAINZ_TRACKID=rec-08", "MUSICBRAINZ_RELEASEGROUPID=rg-ram", + } { + if !strings.Contains(flac, s) { + t.Errorf("flac args lack %q: %s", s, flac) + } + } + // dhowden/tag reads a Vorbis TRACKNUMBER of "8/13" as 0. + if strings.Contains(flac, "/13") { + t.Errorf("flac track number carries its count: %s", flac) + } + if mp3 := strings.Join(tg.args(".mp3"), " "); !strings.Contains(mp3, "track=8/13") { + t.Errorf("mp3 track should be 8/13, which ID3 readers parse: %s", mp3) + } + if strings.Contains(flac, "MUSICBRAINZ_ALBUMID") { + t.Errorf("an empty release id was written: %s", flac) + } + if mp3 := strings.Join(tg.args(".mp3"), " "); !strings.Contains(mp3, "MusicBrainz Track Id=rec-08") { + t.Errorf("mp3 args %s", mp3) + } + if m4a := strings.Join(tg.args(".m4a"), " "); strings.Contains(m4a, "MusicBrainz") { + t.Errorf("m4a args %s", m4a) + } +} diff --git a/internal/indexer/indexer.go b/internal/indexer/indexer.go new file mode 100644 index 00000000..d6a8af17 --- /dev/null +++ b/internal/indexer/indexer.go @@ -0,0 +1,156 @@ +// Package indexer is what every search source answers with, and the Torznab +// client. +// +// A client here holds ONE endpoint and knows nothing about the others. The +// fan-out across sources, the merge and what a source being down means belong +// to internal/acquire, because they are decisions about a search rather than +// about one source. +// +// Prowlarr is not served from here: it has no aggregate Torznab feed (`all` +// 404s like any id it does not have), so asking it everything at once means +// its native API, which internal/prowlarr speaks. +package indexer + +import ( + "errors" + "fmt" + "io" + "net/http" + "strings" + "time" +) + +// ErrUnconfigured is a source with no address (or, for Prowlarr, no key): not +// a failure, an instance that does not search there. +var ErrUnconfigured = errors.New("indexer is not configured") + +// ErrUnreachable is a search that could not be made: no network, a timeout, a +// malformed answer, a status outside 2xx. +var ErrUnreachable = errors.New("indexer unreachable") + +// ErrAuth is a key the source refused. Separate from ErrUnreachable because +// it needs a different sentence and a different action: a refused key is a +// field in Settings, an outage is something to wait out. +var ErrAuth = errors.New("the API key was refused") + +// UserAgent names Minstrel in Prowlarr's and each indexer's logs. +const UserAgent = "Minstrel (self-hosted music server; release search)" + +// MusicCategories are the Newznab categories a music search asks for: 3000 +// Audio, with its MP3 (3010) and Lossless (3040) children alongside, because +// an indexer that files music under a child answers only for the child. +// 3030 (Audiobook) is deliberately absent. +var MusicCategories = []int{3000, 3010, 3040} + +// Query is one search: the words, and which categories to ask in. Empty +// Categories asks every category. +type Query struct { + Text string + Categories []int +} + +// Result is one search result, as the source described it. +type Result struct { + Title string + // InfoHash is the torrent's identity, lowercased; empty when the source + // gave none. + InfoHash string + // Magnet is a magnet URI when the source published one. + Magnet string + // DownloadURL is a .torrent URL when it did not. It must be fetched + // through the source that offered it (it holds the tracker credentials), + // never handed to the download client. + DownloadURL string + SizeBytes int64 + Category int + // Indexer is the tracker BEHIND the source, as the source named it. + Indexer string + // Source is the acquisition_services code that was asked: where a + // .torrent fetch must be routed back to. + Source string + Published time.Time + // Seeders and Peers as reported. A feed that has not sampled a swarm + // reports a nominal 1, so these are not health unless SwarmKnown. + Seeders int + Peers int + // SwarmKnown says Seeders was measured, which only FabledExchange's own + // search can say. + SwarmKnown bool + // FilesCount is how many files the torrent holds, 0 when not said. + FilesCount int +} + +// Send sends req through c, stopping at a redirect to a magnet (returned as a +// *MagnetLink), and reads at most 8 MB of the answer. A call that never got +// an answer is ErrUnreachable; the status is the caller's to read. +func Send(c *http.Client, req *http.Request) (*http.Response, []byte, error) { + resp, err := StopAtMagnet(c).Do(req) + if m, ok := AsMagnet(err); ok { + return nil, nil, &MagnetLink{URI: m} + } + if err != nil { + return nil, nil, fmt.Errorf("%w: %v", ErrUnreachable, err) + } + defer func() { _ = resp.Body.Close() }() + body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil { + return nil, nil, fmt.Errorf("%w: read body: %v", ErrUnreachable, err) + } + return resp, body, nil +} + +// StatusError is what a non-2xx status means for a search source: a 429 is +// RateLimited naming who, anything else ErrUnreachable. Nil for a 2xx. +func StatusError(resp *http.Response, who string) error { + switch { + case resp.StatusCode == http.StatusTooManyRequests: + return &RateLimited{Who: who, RetryAfter: RetryAfter(resp.Header, time.Now())} + case resp.StatusCode < 200 || resp.StatusCode >= 300: + return fmt.Errorf("%w: HTTP %d", ErrUnreachable, resp.StatusCode) + } + return nil +} + +// NormalizeHash is a torrent's identity in its one canonical spelling. Every +// comparison of infohashes goes through it, so a hash differing only in case +// is never two releases. +func NormalizeHash(s string) string { + return strings.ToLower(strings.TrimSpace(s)) +} + +// IsInfoHash says whether s is shaped like a v1 (40) or v2 (64) hex infohash. +func IsInfoHash(s string) bool { + s = NormalizeHash(s) + if len(s) != 40 && len(s) != 64 { + return false + } + for _, r := range s { + if (r < '0' || r > '9') && (r < 'a' || r > 'f') { + return false + } + } + return true +} + +// MagnetHash pulls the infohash out of a magnet link, or returns "". It does +// not validate it. +func MagnetHash(magnet string) string { + const prefix = "urn:btih:" + at := strings.Index(magnet, prefix) + if at < 0 { + return "" + } + hash := magnet[at+len(prefix):] + if cut := strings.IndexAny(hash, "&/?"); cut >= 0 { + hash = hash[:cut] + } + return strings.ToLower(hash) +} + +// FirstLine is an error's first line, for a sentence on the settings page. +func FirstLine(s string) string { + if i := strings.IndexByte(s, '\n'); i >= 0 { + return s[:i] + } + return s +} diff --git a/internal/indexer/magnet.go b/internal/indexer/magnet.go new file mode 100644 index 00000000..123e61c2 --- /dev/null +++ b/internal/indexer/magnet.go @@ -0,0 +1,58 @@ +package indexer + +import ( + "bytes" + "errors" + "net/http" +) + +// MagnetLink is a download link that turned out to be a magnet. Returned as +// an error by a .torrent fetch so the grab hands the magnet to the download +// client instead. Public trackers do this routinely: Prowlarr's download link +// answers with a redirect to `magnet:?xt=…`, which Go's client would otherwise +// fail on as an unsupported scheme. +type MagnetLink struct{ URI string } + +func (m *MagnetLink) Error() string { return "the download link is a magnet link" } + +// AsMagnet says whether err is a magnet link, and which. +func AsMagnet(err error) (string, bool) { + var m *MagnetLink + if errors.As(err, &m) && m.URI != "" { + return m.URI, true + } + return "", false +} + +// StopAtMagnet returns a copy of c that stops at a redirect to a magnet URI +// and reports it as a *MagnetLink. Every other redirect behaves as c would. +func StopAtMagnet(c *http.Client) *http.Client { + if c == nil { + c = http.DefaultClient + } + out := *c + next := c.CheckRedirect + out.CheckRedirect = func(req *http.Request, via []*http.Request) error { + if req.URL != nil && req.URL.Scheme == "magnet" { + return &MagnetLink{URI: req.URL.String()} + } + if next != nil { + return next(req, via) + } + if len(via) >= 10 { + return errors.New("stopped after 10 redirects") + } + return nil + } + return &out +} + +// MagnetBody is a response whose whole body is a magnet URI, which some +// indexers send with a 200 instead of a redirect. Nil otherwise. +func MagnetBody(body []byte) error { + trimmed := bytes.TrimSpace(body) + if bytes.HasPrefix(trimmed, []byte("magnet:")) && !bytes.ContainsAny(trimmed, " \n<") { + return &MagnetLink{URI: string(trimmed)} + } + return nil +} diff --git a/internal/indexer/ratelimit.go b/internal/indexer/ratelimit.go new file mode 100644 index 00000000..5be3ca9f --- /dev/null +++ b/internal/indexer/ratelimit.go @@ -0,0 +1,50 @@ +package indexer + +import ( + "errors" + "fmt" + "net/http" + "strconv" + "time" +) + +// RateLimited is a source answering 429. Named so the search pacing can hold +// that source back for as long as it asked; still an ErrUnreachable to +// everything else, because this search did not reach it. +type RateLimited struct { + // RetryAfter is what the source asked for; zero when it did not say. + RetryAfter time.Duration + // Who names the source in the sentence, e.g. "Prowlarr". + Who string +} + +func (e *RateLimited) Error() string { + return fmt.Sprintf("%v: HTTP 429, %s is rate-limiting this search", ErrUnreachable, e.Who) +} + +func (e *RateLimited) Unwrap() error { return ErrUnreachable } + +// AsRateLimited finds a RateLimited anywhere in err's chain. +func AsRateLimited(err error) (*RateLimited, bool) { + var rl *RateLimited + if errors.As(err, &rl) { + return rl, true + } + return nil, false +} + +// RetryAfter reads a Retry-After header: seconds, or an HTTP date. Zero when +// absent or unreadable. +func RetryAfter(h http.Header, now time.Time) time.Duration { + v := h.Get("Retry-After") + if v == "" { + return 0 + } + if s, err := strconv.Atoi(v); err == nil && s > 0 { + return time.Duration(s) * time.Second + } + if t, err := http.ParseTime(v); err == nil && t.After(now) { + return t.Sub(now) + } + return 0 +} diff --git a/internal/indexer/torznab.go b/internal/indexer/torznab.go new file mode 100644 index 00000000..8a458b1b --- /dev/null +++ b/internal/indexer/torznab.go @@ -0,0 +1,253 @@ +package indexer + +import ( + "context" + "encoding/xml" + "fmt" + "net/http" + "net/url" + "strconv" + "strings" + "time" +) + +// DefaultTimeout is what an unset Timeout means. Longer than a metadata +// call: a feed may itself fan out to several trackers before answering. +const DefaultTimeout = 30 * time.Second + +// TorznabConfig is one Torznab feed. +type TorznabConfig struct { + // Code is the acquisition_services row this client was built from, + // stamped onto every Result. + Code string + // FeedURL is the feed's whole address up to and including /api, as + // Jackett or a Prowlarr indexer shows it — e.g. + // http://prowlarr:9696/12/api. Query parameters are appended to it. + // One field rather than base + path + indexer id: a pasted URL is more + // reliable than Minstrel being right about a third party's routes. + FeedURL string + // APIKey is optional: Torznab makes it so. A feed that needs one says so + // with Newznab error 100, which is ErrAuth. + APIKey string + Timeout time.Duration +} + +// Torznab searches one Torznab feed. +type Torznab struct { + Config TorznabConfig + // HTTP is injectable for tests. Nil takes http.DefaultClient. + HTTP *http.Client +} + +// Search asks the feed for q. `t=search` is free text, which every indexer +// supports; `t=music` takes artist and album separately, and a release name +// does not reliably say which of its parts is which. +func (c *Torznab) Search(ctx context.Context, q Query) ([]Result, error) { + params := url.Values{"t": {"search"}, "q": {q.Text}} + for _, cat := range q.Categories { + params.Add("cat", strconv.Itoa(cat)) + } + body, err := c.get(ctx, c.endpoint(params)) + if err != nil { + return nil, err + } + out, err := parseFeed(body) + if err != nil { + return nil, err + } + for i := range out { + out[i].Source = c.Config.Code + } + return out, nil +} + +// Download fetches a result's .torrent bytes through the feed, which holds +// the tracker's credentials. A result with a magnet needs none of this. +func (c *Torznab) Download(ctx context.Context, r Result) ([]byte, error) { + if r.DownloadURL == "" { + return nil, fmt.Errorf("%w: %q has no download URL", ErrUnreachable, r.Title) + } + body, err := c.get(ctx, r.DownloadURL) + if err != nil { + return nil, err + } + return torrentBody(r.Title, body) +} + +// torrentBody checks a fetched .torrent. A torrent is a bencoded dictionary +// and starts with 'd'; an expired session gets an HTML login page with HTTP +// 200, which the download client would otherwise reject as an unsupported +// file from a component that did nothing wrong. +func torrentBody(title string, body []byte) ([]byte, error) { + if err := MagnetBody(body); err != nil { + return nil, err + } + if len(body) == 0 || body[0] != 'd' { + return nil, fmt.Errorf("%w: %q returned %d bytes that are not a torrent", ErrUnreachable, title, len(body)) + } + return body, nil +} + +// TorrentBody is torrentBody for the other search clients' fetches. +func TorrentBody(title string, body []byte) ([]byte, error) { return torrentBody(title, body) } + +func (c *Torznab) endpoint(params url.Values) string { + if c.Config.APIKey != "" { + params.Set("apikey", c.Config.APIKey) + } + feed := strings.TrimSpace(c.Config.FeedURL) + sep := "?" + if strings.Contains(feed, "?") { + sep = "&" + } + return feed + sep + params.Encode() +} + +func (c *Torznab) get(ctx context.Context, endpoint string) ([]byte, error) { + if strings.TrimSpace(c.Config.FeedURL) == "" { + return nil, ErrUnconfigured + } + timeout := c.Config.Timeout + if timeout <= 0 { + timeout = DefaultTimeout + } + ctx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil) + if err != nil { + return nil, fmt.Errorf("%w: build request: %v", ErrUnreachable, err) + } + req.Header.Set("User-Agent", UserAgent) + resp, body, err := Send(c.HTTP, req) + if err != nil { + return nil, err + } + if err := StatusError(resp, "the indexer"); err != nil { + return nil, err + } + // Newznab reports errors as HTTP 200 with an document, so a + // client checking only the status reads a refused key as an empty result + // set: search quietly stops finding things. + if err := errorDocument(body); err != nil { + return nil, err + } + return body, nil +} + +// errorDocument reads an answer. Newznab's 1xx +// codes (100 wrong key, 101 suspended, 102 insufficient privileges) are all +// about who is asking, so they are ErrAuth. +func errorDocument(body []byte) error { + head := body + if len(head) > 512 { + head = head[:512] + } + if !strings.Contains(string(head), " + // and repeats it in , either populated by version. + for _, candidate := range []string{it.Link, it.Enclosure.URL} { + if strings.HasPrefix(candidate, "magnet:") && r.Magnet == "" { + r.Magnet = candidate + } else if candidate != "" && r.DownloadURL == "" { + r.DownloadURL = candidate + } + } + // The infohash from the explicit attr, then the magnet, then the GUID + // only when it is shaped like a hash: Prowlarr puts a URL there as + // often as a hash. + r.InfoHash = NormalizeHash(it.attr("infohash")) + if r.InfoHash == "" { + r.InfoHash = MagnetHash(r.Magnet) + } + if r.InfoHash == "" && IsInfoHash(it.GUID) { + r.InfoHash = NormalizeHash(it.GUID) + } + if t, err := time.Parse(time.RFC1123Z, strings.TrimSpace(it.PubDate)); err == nil { + r.Published = t + } else if t, err := time.Parse(time.RFC1123, strings.TrimSpace(it.PubDate)); err == nil { + r.Published = t + } + out = append(out, r) + } + return out, nil +} + +func atoi(s string) int { + n, err := strconv.Atoi(strings.TrimSpace(s)) + if err != nil { + return 0 + } + return n +} diff --git a/internal/indexer/torznab_test.go b/internal/indexer/torznab_test.go new file mode 100644 index 00000000..2855f0af --- /dev/null +++ b/internal/indexer/torznab_test.go @@ -0,0 +1,188 @@ +package indexer + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "testing" + "time" +) + +const sampleFeed = ` + + + + Radiohead - OK Computer (1997) [FLAC] + https://tracker.example/details/1 + https://prowlarr.example/1/download?link=abc + 512000000 + Tue, 07 Oct 2026 10:00:00 +0000 + + + + + + + + Radiohead - Kid A (2000) [MP3 320] + 0123456789abcdef0123456789abcdef01234567 + magnet:?xt=urn:btih:FFEEDDCCBBAA99887766554433221100FFEEDDCC&dn=kid+a + + + + + +` + +func TestTorznabSearch_ParsesTheFeedAndStampsTheSource(t *testing.T) { + reqs := make(chan *http.Request, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + reqs <- r + _, _ = w.Write([]byte(sampleFeed)) + })) + defer srv.Close() + + c := &Torznab{Config: TorznabConfig{Code: "torznab-2", FeedURL: srv.URL + "/12/api", APIKey: "k"}} + results, err := c.Search(context.Background(), Query{Text: "radiohead", Categories: MusicCategories}) + if err != nil { + t.Fatalf("search: %v", err) + } + got := <-reqs + if got.URL.Path != "/12/api" || got.URL.Query().Get("t") != "search" || got.URL.Query().Get("apikey") != "k" { + t.Errorf("request = %s, want the feed URL with t=search and the key", got.URL) + } + if cats := got.URL.Query()["cat"]; len(cats) != 3 || cats[0] != "3000" { + t.Errorf("categories = %v, want the music categories", cats) + } + if len(results) != 2 { + t.Fatalf("results = %d, want 2 (the untitled item dropped)", len(results)) + } + first := results[0] + if first.InfoHash != "aabbccddeeff00112233445566778899aabbccdd" || first.Seeders != 12 || + first.Category != 3040 || first.Indexer != "RED" || first.DownloadURL == "" || first.Source != "torznab-2" { + t.Errorf("first = %+v", first) + } + if first.Published.IsZero() { + t.Error("pubDate was not parsed") + } + // The magnet's hash wins over a GUID shaped like another hash. + if results[1].Magnet == "" || results[1].InfoHash != "ffeeddccbbaa99887766554433221100ffeeddcc" { + t.Errorf("second = %+v, want the magnet's hash", results[1]) + } +} + +func TestTorznabSearch_KeepsAQueryAlreadyInTheFeedURL(t *testing.T) { + reqs := make(chan *http.Request, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + reqs <- r + _, _ = w.Write([]byte(``)) + })) + defer srv.Close() + + c := &Torznab{Config: TorznabConfig{FeedURL: srv.URL + "/api?passkey=p"}} + if _, err := c.Search(context.Background(), Query{Text: "x"}); err != nil { + t.Fatalf("search: %v", err) + } + got := <-reqs + if got.URL.Query().Get("passkey") != "p" || got.URL.Query().Get("q") != "x" { + t.Errorf("request = %s, want the pasted query kept and ours added", got.URL) + } +} + +// Newznab reports a refused key as HTTP 200 with an document. Read +// as a feed, that is an empty result set: search would quietly find nothing. +func TestTorznabSearch_ReadsAnErrorDocument(t *testing.T) { + for name, tc := range map[string]struct { + doc string + want error + }{ + "a wrong key": {``, ErrAuth}, + "anything else": {``, ErrUnreachable}, + } { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`` + tc.doc)) + })) + c := &Torznab{Config: TorznabConfig{FeedURL: srv.URL + "/api"}} + _, err := c.Search(context.Background(), Query{Text: "x"}) + srv.Close() + if !errors.Is(err, tc.want) { + t.Errorf("%s: err = %v, want %v", name, err, tc.want) + } + } +} + +func TestTorznabSearch_A429IsRateLimitedWithItsRetryAfter(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Retry-After", "90") + w.WriteHeader(http.StatusTooManyRequests) + })) + defer srv.Close() + + c := &Torznab{Config: TorznabConfig{FeedURL: srv.URL + "/api"}} + _, err := c.Search(context.Background(), Query{Text: "x"}) + rl, ok := AsRateLimited(err) + if !ok || rl.RetryAfter != 90*time.Second { + t.Fatalf("err = %v, want RateLimited for 90s", err) + } + if !errors.Is(err, ErrUnreachable) { + t.Error("a 429 should still read as unreachable to everything else") + } +} + +func TestTorznabSearch_NoAddressIsUnconfigured(t *testing.T) { + _, err := (&Torznab{}).Search(context.Background(), Query{Text: "x"}) + if !errors.Is(err, ErrUnconfigured) { + t.Fatalf("err = %v, want ErrUnconfigured", err) + } +} + +func TestTorznabDownload_RefusesWhatIsNotATorrent(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/login": + _, _ = w.Write([]byte(`please log in`)) + case "/magnet": + _, _ = w.Write([]byte("magnet:?xt=urn:btih:abc")) + case "/redirect": + http.Redirect(w, r, "magnet:?xt=urn:btih:def", http.StatusFound) + default: + _, _ = w.Write([]byte("d8:announce0:e")) + } + })) + defer srv.Close() + + c := &Torznab{Config: TorznabConfig{FeedURL: srv.URL + "/api"}} + ctx := context.Background() + if _, err := c.Download(ctx, Result{Title: "t", DownloadURL: srv.URL + "/login"}); !errors.Is(err, ErrUnreachable) { + t.Errorf("an HTML page: err = %v, want ErrUnreachable", err) + } + for _, path := range []string{"/magnet", "/redirect"} { + _, err := c.Download(ctx, Result{Title: "t", DownloadURL: srv.URL + path}) + if _, ok := AsMagnet(err); !ok { + t.Errorf("%s: err = %v, want a MagnetLink", path, err) + } + } + body, err := c.Download(ctx, Result{Title: "t", DownloadURL: srv.URL + "/file"}) + if err != nil || string(body) != "d8:announce0:e" { + t.Errorf("a torrent: body %q err %v", body, err) + } +} + +func TestRetryAfter(t *testing.T) { + now := time.Date(2026, 10, 10, 0, 0, 0, 0, time.UTC) + for v, want := range map[string]time.Duration{ + "": 0, + "30": 30 * time.Second, + "soon": 0, + "Sat, 10 Oct 2026 00:02:00 GMT": 2 * time.Minute, + } { + h := http.Header{} + if v != "" { + h.Set("Retry-After", v) + } + if got := RetryAfter(h, now); got != want { + t.Errorf("RetryAfter(%q) = %v, want %v", v, got, want) + } + } +} diff --git a/internal/library/acoustid_lookup.go b/internal/library/acoustid_lookup.go index 0f4dfa15..ce881a0f 100644 --- a/internal/library/acoustid_lookup.go +++ b/internal/library/acoustid_lookup.go @@ -18,17 +18,17 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// AcoustID lookup worker (M401 #3921). +// AcoustID lookup worker. // // A track whose tags carry no recording MBID is invisible to the ListenBrainz // similarity arm: it is never a seed and never comes back as a result. This // worker fingerprints such tracks and asks AcoustID which recording they are, -// writing an id only when the answer is unambiguous (D5). The file's own tag -// always outranks it (D4). +// writing an id only when the answer is unambiguous (chooseRecording). The +// file's own tag always outranks it. // -// It never gates anything else (rule 164). With no key, switched off or -// AcoustID unreachable, it idles and says why through Status; scans and -// playback do not notice. +// It never gates anything else. With no key, switched off or AcoustID +// unreachable, it idles and says why through Status; scans and playback do +// not notice. // acoustIDLookupTick is how often the worker looks for work. A settings save // starts a pass at once, so this only paces new tracks and retries after an @@ -187,7 +187,7 @@ func (w *AcoustIDLookupWorker) Run(ctx context.Context) { } // runOnce contains a pass so that nothing it does (an error, a panic) can stop -// the next one from starting (rule 157). +// the next one from starting. func (w *AcoustIDLookupWorker) runOnce(ctx context.Context) { w.setStatus(func(s *AcoustIDLookupStatus) { s.Running = true }) problem := "" @@ -216,8 +216,8 @@ func (w *AcoustIDLookupWorker) runOnce(ctx context.Context) { "no_match", res.NoMatch, "failed", res.Failed, "inconclusive", res.Inconclusive) } // A recording MBID AcoustID just gave a replacement may be the one its - // missing predecessor carried (M485): most of the operator's stale rows - // became pairable exactly this way. + // missing predecessor carried, which makes the pair foldable + // (PairMissingTracks). if res.Matched > 0 && ctx.Err() == nil { if _, err := PairMissingTracks(ctx, w.pool, w.logger); err != nil && ctx.Err() == nil { w.logger.Warn("acoustid lookup: missing-pair pass failed", "err", err) @@ -322,7 +322,8 @@ func (w *AcoustIDLookupWorker) lookupTrack( if isInconclusive(err) { return "", nil } - // fpcalc ran and rejected the file: a verdict, settled until it changes. + // fpcalc ran and rejected the file: a verdict, settled until it + // changes. return w.store(ctx, row, lookupDecision{state: lookupFailed, detail: err.Error()}), nil } cands, err := w.client.Lookup(ctx, cfg.APIKey, fp.fingerprint, fp.durationSec) @@ -345,6 +346,46 @@ func (w *AcoustIDLookupWorker) lookupTrack( return w.store(ctx, row, chooseRecording(cands, cfg.MinScore, row.Title, row.DurationMs)), nil } +// ErrIdentifyUnavailable says AcoustID cannot be asked now: the lookup is off +// or has no key, the key was refused, or AcoustID could not be reached. The +// caller waits and asks again on a later pass. +var ErrIdentifyUnavailable = errors.New("acoustid: cannot be asked now") + +// IdentifyRecordings asks AcoustID which recordings a file may be, for the +// duplicate resolver: every candidate at or above the minimum score, +// best first. None is an answer: AcoustID does not know the audio. Nothing is +// stored; the file's own lookup is the worker's. +func (w *AcoustIDLookupWorker) IdentifyRecordings(ctx context.Context, path string) ([]string, error) { + if w == nil { + return nil, ErrIdentifyUnavailable + } + cfg := w.settings.Get() + if !cfg.Ready() { + return nil, ErrIdentifyUnavailable + } + fp, err := w.fingerprint(ctx, path) + if err != nil { + if isInconclusive(err) { + return nil, fmt.Errorf("%w: %w", ErrIdentifyUnavailable, err) + } + return nil, fmt.Errorf("fingerprint: %w", err) + } + cands, err := w.client.Lookup(ctx, cfg.APIKey, fp.fingerprint, fp.durationSec) + switch { + case errors.Is(err, acoustid.ErrInvalidKey), errors.Is(err, acoustid.ErrUnavailable): + return nil, fmt.Errorf("%w: %w", ErrIdentifyUnavailable, err) + case err != nil: + return nil, err + } + var ids []string + for _, c := range cands { + if float32(c.Score) >= cfg.MinScore { + ids = append(ids, c.ID) + } + } + return ids, nil +} + // store records a decision and applies it to tracks.mbid in one transaction, // so the gauge never shows a match whose id was not written. A failed write // stores nothing and the track is tried again. @@ -392,7 +433,8 @@ type lookupDecision struct { detail string } -// chooseRecording applies D5 to AcoustID's candidates (best score first): +// chooseRecording picks a recording from AcoustID's candidates (best score +// first): // only recordings at or above minScore count, one of them is a match, and // several are told apart by the track's title and length or not at all. A // wrong MBID would feed similarity the wrong neighbours, which is worse than diff --git a/internal/library/acoustid_lookup_test.go b/internal/library/acoustid_lookup_test.go index be51fd9d..aa0fdc5a 100644 --- a/internal/library/acoustid_lookup_test.go +++ b/internal/library/acoustid_lookup_test.go @@ -24,8 +24,8 @@ func cand(id, title string, sec int, score float64) acoustid.Candidate { return acoustid.Candidate{Recording: acoustid.Recording{ID: id, Title: title, DurationSec: sec}, Score: score} } -// D5, as a table over what decides it: how many candidates clear the -// threshold, and whether title and length leave exactly one of them. +// chooseRecording, as a table over what decides it: how many candidates clear +// the threshold, and whether title and length leave exactly one of them. func TestChooseRecording(t *testing.T) { const ( minScore = 0.85 @@ -154,7 +154,7 @@ func addTrack(t *testing.T, q *dbq.Queries, album dbq.Album, artist dbq.Artist, return tr } -// D4: the file's tag outranks a lookup, in both directions. Falsified by +// The file's tag outranks a lookup, in both directions. Falsified by // restoring `mbid = EXCLUDED.mbid` in UpsertTrack: the first re-read below // then erases the looked-up id. func TestUpsertTrack_KeepsALookedUpMbidUntilATagCarriesOne_Integration(t *testing.T) { @@ -302,7 +302,7 @@ func TestAcoustIDLookup_StopsWhenTheServiceCannotAnswer_Integration(t *testing.T } } -// The point of the milestone (#3921): once a lookup fills an MBID, the +// The point of the lookup: once it fills an MBID, the // similarity worker picks the track as a seed and can map ListenBrainz's // answers back to it. func TestAcoustIDLookup_FilledTrackReachesSimilarity_Integration(t *testing.T) { @@ -352,7 +352,7 @@ func TestAcoustIDLookup_FilledTrackReachesSimilarity_Integration(t *testing.T) { // And it is reachable as a target: an answer cached for another seed // before the lookup filled the MBID now resolves to it, with no new - // fetch (#5296). + // fetch. other, _, _ := seedTrack(t, pool, filepath.Join(dir, "other.mp3")) if _, err := pool.Exec(ctx, `INSERT INTO listenbrainz_similar_recordings (seed_track_id, recording_mbid, score) VALUES ($1, 'rec-played', 0.8)`, other.ID); err != nil { diff --git a/internal/library/acoustid_settings.go b/internal/library/acoustid_settings.go index 8ee310a6..f504219d 100644 --- a/internal/library/acoustid_settings.go +++ b/internal/library/acoustid_settings.go @@ -12,13 +12,13 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// AcoustID lookup settings (M401 #3922). Rule 25: a database row, changed -// without a restart, shared by the worker and the admin API. +// AcoustID lookup settings: a database row, changed without a restart, shared +// by the worker and the admin API. // AcoustIDSettings mirrors the acoustid_settings row. type AcoustIDSettings struct { Enabled bool - // APIKey is the operator's registered application key. Never sent back + // APIKey is the registered AcoustID application key. Never sent back // to the browser; the API reports only whether one is set. APIKey string // MinScore is the AcoustID score a recording needs to be considered. diff --git a/internal/library/album_loudness.go b/internal/library/album_loudness.go index 7d029e0c..17e67fae 100644 --- a/internal/library/album_loudness.go +++ b/internal/library/album_loudness.go @@ -12,7 +12,7 @@ import ( syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) -// Album loudness (M464 #4996). +// Album loudness. // // Album-mode normalization plays a whole album at one gain, so the quiet // interlude stays quieter than the single it sits between. That gain comes @@ -168,7 +168,7 @@ func (w *LoudnessBackfillWorker) storeAlbumLoudness( return fmt.Errorf("read inputs: %w", err) } a := computeAlbumLoudness(inputs) - // Logged before the write, for the reason storeLoudness gives (#2704). A + // Logged before the write, for the reason storeLoudness gives. A // failed log stores nothing, so the digest still differs and the next // pass tries again. if albumVisibleChange(before, a) { diff --git a/internal/library/album_loudness_test.go b/internal/library/album_loudness_test.go index a943e024..93f2cf98 100644 --- a/internal/library/album_loudness_test.go +++ b/internal/library/album_loudness_test.go @@ -185,7 +185,7 @@ func TestAlbumLoudness_Integration(t *testing.T) { t.Fatalf("waiting album = lufs %v, %d/%d settled; want nil, 1/2", lufs, settled, total) } - // Still no album value: clients have nothing new to read (#4997). + // Still no album value: clients have nothing new to read. if n := albumChanges(); n != 0 { t.Fatalf("a waiting album logged %d sync changes, want 0", n) } diff --git a/internal/library/delete.go b/internal/library/delete.go index a12aec3c..8e65edf2 100644 --- a/internal/library/delete.go +++ b/internal/library/delete.go @@ -30,12 +30,12 @@ var ErrTrackNotFound = errors.New("library: track not found") var removeFile = os.Remove // FileRemoveError reports that a track's file exists but could not be removed. -// When DeleteTrackFile returns one, NOTHING was deleted: the row, its likes, its -// play history and its playlist memberships are all intact. +// When DeleteTrackFile returns one, NOTHING was deleted: the row, its likes, +// its play history and its playlist memberships are all intact. type FileRemoveError struct { Path string // UID and GID are the identity the server process runs as — the half of a - // permission problem the operator cannot see from the host side. + // permission problem an admin cannot see from the host side. UID, GID int Err error } @@ -44,15 +44,15 @@ func (e *FileRemoveError) Error() string { return fmt.Sprintf("remove track file func (e *FileRemoveError) Unwrap() error { return e.Err } -// Dir is the directory removal needs write access to. Unlinking a file writes to -// its PARENT, so a world-writable file inside a read-only directory still cannot -// be removed — naming the file's own permissions would send the operator to the -// wrong place. +// Dir is the directory removal needs write access to. Unlinking a file writes +// to its PARENT, so a world-writable file inside a read-only directory still +// cannot be removed — naming the file's own permissions would send an admin to +// the wrong place. func (e *FileRemoveError) Dir() string { return filepath.Dir(e.Path) } // NotWritable reports whether the library is unwritable for this process — a // read-only mount or a permission denial — rather than an I/O fault. It is the -// case the operator can fix, so callers answer it differently. +// case an admin can fix, so callers answer it differently. func (e *FileRemoveError) NotWritable() bool { return errors.Is(e.Err, fs.ErrPermission) || errors.Is(e.Err, syscall.EROFS) } @@ -69,36 +69,38 @@ func (e *FileRemoveError) Reason() string { // DeletedTrack reports what a delete tidied away beyond the track itself. type DeletedTrack struct { - // AlbumID is set when the track was its album's last, so the album went too. + // AlbumID is set when the track was its album's last, so the album went + // too. AlbumID *pgtype.UUID - // ArtistID is set when that album was its artist's last, so the artist went too. + // ArtistID is set when that album was its artist's last, so the artist went + // too. ArtistID *pgtype.UUID } // DeleteTrackFile removes a track's file from disk and then its row, tidying // away an album or artist the delete leaves empty. It is the ONLY path that // deletes a track file: the admin remove-track endpoint and quarantine's Delete -// file both come through here (#3918). +// file both come through here. // // Order is the whole contract. The file goes first, and if it cannot go — a // read-only mount, a permission denial, an I/O error — nothing else happens and -// a *FileRemoveError comes back. Proceeding past that failure is how #3918 lost +// a *FileRemoveError comes back. Proceeding past that failure would lose // history: tracks CASCADEs to play_events, general_likes, contextual_likes, // playlist_tracks, track_tags and playback_errors, so the row and everything -// hanging off it were destroyed while the file survived, and the next scan -// re-imported it as a brand-new track with none of it. +// hanging off it would be destroyed while the file survived, and the next scan +// would re-import it as a brand-new track with none of it. // // A file that is already gone (fs.ErrNotExist) is not a failure; the row is // removed as asked. // // This is NOT the missing-file path. That lifecycle is deliberately -// non-destructive: reconcile stamps missing_since (#2523), selection paths -// filter on it, and a returning file is un-marked or adopted (#2528). This is the -// explicit, irreversible "remove this recording", never the way to tidy up a row -// whose file merely went away. +// non-destructive: reconcile stamps missing_since, selection paths filter on +// it, and a returning file is un-marked or adopted. This is the explicit, +// irreversible "remove this recording", never the way to tidy up a row whose +// file merely went away. // -// dataDir, when set, also clears the cached art of an artist the delete removed. -// logger may be nil. +// dataDir, when set, also clears the cached art of an artist the delete +// removed. logger may be nil. func DeleteTrackFile( ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, trackID pgtype.UUID, ) (DeletedTrack, error) { @@ -163,9 +165,9 @@ func DeleteTrackFile( } // removeTrackFileOnDisk is the one rule for removing a track's file, shared by -// DeleteTrackFile and the duplicate merge. A file already gone is fine; anything -// else comes back as a *FileRemoveError, and the caller must then change nothing -// in the database (#3918). +// DeleteTrackFile and the duplicate merge. A file already gone is fine; +// anything else comes back as a *FileRemoveError, and the caller must then +// change nothing in the database. func removeTrackFileOnDisk(path string) error { if err := removeFile(path); err != nil && !errors.Is(err, fs.ErrNotExist) { return &FileRemoveError{Path: path, UID: os.Getuid(), GID: os.Getgid(), Err: err} diff --git a/internal/library/delete_test.go b/internal/library/delete_test.go index 76cf9ae2..f51af5c3 100644 --- a/internal/library/delete_test.go +++ b/internal/library/delete_test.go @@ -141,7 +141,7 @@ func TestDeleteTrackFile_EmptiedAlbumAndArtistGoToo(t *testing.T) { } } -// The #3918 proof. A file that cannot be removed must leave EVERYTHING in place: +// A file that cannot be removed must leave EVERYTHING in place: // the row is what carries likes, plays and playlist memberships, and the file // surviving means the next scan would re-import it as a stranger. func TestDeleteTrackFile_UnremovableFileDeletesNothing(t *testing.T) { diff --git a/internal/library/duplicate_classify.go b/internal/library/duplicate_classify.go index 70430d73..b4b19817 100644 --- a/internal/library/duplicate_classify.go +++ b/internal/library/duplicate_classify.go @@ -1,26 +1,33 @@ package library import ( + "path/filepath" "regexp" + "sort" "strings" "unicode" ) -// Duplicate classification (M498 #5435). +// Duplicate classification. // // A duplicate group is one recording found more than once. What should happen -// to it depends on where the copies sit, and the measured library (milestone -// 498's body) splits into four shapes: +// to it depends on where the copies sit: // -// - same_release: every copy is on one album. An album holds a song once, so -// the extra copies are the ones to fold away — when Lidarr does not need -// them (the resolver checks). -// - cross_release: the copies are on different albums under one title. A -// single and the album it came from, a deluxe edition and the standard one: -// each copy fulfils its own release in Lidarr, so none is deleted. -// - mismatch: byte-identical audio on different albums under different -// titles. One file carries another song's tags — a wrong-file import. -// - review: anything else. The operator decides. +// - same_release: every copy is on one album under one title. An album holds +// a song once, so the extra copies are the ones to fold away. +// - cross_release: the copies are on different albums. A single and the +// album it came from, a deluxe edition and the standard one, or one +// recording a later release retitled ("The Ragtime Cat", "F.O.D. (live)", +// a romanized title): each copy belongs to its own release, so none is +// deleted (unless it is a video rip; the resolver decides). Copies under +// different titles on different releases are retitles, not different +// songs. +// - distinct: one album, two versions of a song that sound alike, named as +// such ("Lovesick", "Lovesick (Instrumental)"). Not duplicates. +// - mismatch: one album holds the same audio under different song titles: +// one file filling several track slots, a broken download. Which slot is +// right is the resolver's to decide from MusicBrainz's track lengths. +// - review: what none of these fits. An admin decides. // DuplicateClass is a group's shape. The values are duplicate_groups.class. type DuplicateClass string @@ -28,6 +35,7 @@ type DuplicateClass string const ( ClassSameRelease DuplicateClass = "same_release" ClassCrossRelease DuplicateClass = "cross_release" + ClassDistinct DuplicateClass = "distinct" ClassMismatch DuplicateClass = "mismatch" ClassReview DuplicateClass = "review" ) @@ -44,17 +52,17 @@ type ClassifyMember struct { // // Titles are compared by MatchTitleKey, which drops what a release or an upload // adds to a title (featuring credits, remaster notes, video markers) and keeps -// what names a different recording (live, demo, remix, instrumental). So one -// album holding "WWW" and "WWW (instrumental)" as an acoustic match is review, -// not a copy to fold away: the #3885 case, where they were in fact different -// recordings mislabelled. Byte-identical copies on one album are the same file -// twice whatever their tags say. +// what names a different recording (live, demo, remix, instrumental). On one +// album, titles that differ only by such a word are two versions (distinct) +// unless the bytes are identical, which is one file twice: "WWW" and "WWW +// (instrumental)" holding the same file is a wrong slot. func ClassifyDuplicateGroup(tier string, members []ClassifyMember) DuplicateClass { if len(members) < 2 { return ClassReview } - oneAlbum, oneTitle := true, true + oneAlbum, oneTitle, oneSong := true, true, true key := MatchTitleKey(members[0].Title, members[0].ArtistName) + song := songKey(members[0].Title, members[0].ArtistName) for _, m := range members[1:] { if m.AlbumID != members[0].AlbumID { oneAlbum = false @@ -62,26 +70,74 @@ func ClassifyDuplicateGroup(tier string, members []ClassifyMember) DuplicateClas if MatchTitleKey(m.Title, m.ArtistName) != key { oneTitle = false } + if songKey(m.Title, m.ArtistName) != song { + oneSong = false + } } exact := tier == string(tierExact) switch { - case oneAlbum && (oneTitle || exact): - return ClassSameRelease - case oneAlbum: - return ClassReview - case oneTitle: + case !oneAlbum: return ClassCrossRelease - case exact: - return ClassMismatch + case oneTitle: + return ClassSameRelease + case oneSong && !exact: + return ClassDistinct default: - return ClassReview + return ClassMismatch } } +// versionSuffix is a version named after a dash rather than in brackets: +// "Goodbye Yesterday - Instrumental Version", "Song - Radio Edit". +var versionSuffix = regexp.MustCompile( + `(?i)\s-\s[^-]*\b(version|instrumental|remix|mix|live|acoustic|demo|edit|reprise|rework|vip)\b[^-]*$`) + +// versionWord matches a word that names a version of a song. +var versionWord = regexp.MustCompile( + `(?i)\b(clean|explicit|instrumentals?|live|acoustic|unplugged|demo|remix|mix|edit|radio|karaoke|a ?cappella|acapella|reprise)\b`) + +// versionWords lists, sorted and lowercased, the version words in a copy's +// title and file name together. Two copies holding one title tag can still be +// two versions, told apart only by the file name: "Suite-Pee" and +// "Suite-Pee (Clean Version)", "Luv (sic) Grand Finale" and its instrumental. +func versionWords(title, path string) string { + seen := map[string]bool{} + name := strings.TrimSuffix(filepath.Base(path), filepath.Ext(path)) + for _, w := range versionWord.FindAllString(title+" "+name, -1) { + w = strings.ToLower(w) + switch { + case w == "instrumentals": + w = "instrumental" + case strings.Contains(w, "cappella") || w == "acapella": + w = "acappella" + } + seen[w] = true + } + words := make([]string, 0, len(seen)) + for w := range seen { + words = append(words, w) + } + sort.Strings(words) + return strings.Join(words, ",") +} + +// songKey is MatchTitleKey with every bracketed segment and a dashed version +// suffix dropped: the song a title names, whatever version of it. +// "Lovesick (Instrumental)", "Lovesick - Instrumental Version" and "Lovesick" +// share one. +func songKey(title, artist string) string { + t := versionSuffix.ReplaceAllString(bracketed.ReplaceAllString(title, " "), "") + if key := MatchTitleKey(t, artist); key != "" { + return key + } + return MatchTitleKey(title, artist) +} + var ( // A bracketed credit: "(feat. Panda Bear)", "[ft. X]", "(featuring Y)". featBracket = regexp.MustCompile(`(?i)[(\[][^)\]]*\b(feat\.?|ft\.?|featuring)\s[^)\]]*[)\]]`) - // A credit trailing the title without brackets: "Doin' It Right ft. Panda Bear". + // A credit trailing the title without brackets: "Doin' It Right ft. Panda + // Bear". featTrailing = regexp.MustCompile(`(?i)\s(feat\.?|ft\.?|featuring)\s.*$`) // "(2011 Remaster)", "[Remastered]", "- Remastered 2009". remasterBracket = regexp.MustCompile(`(?i)[(\[][^)\]]*remaster[^)\]]*[)\]]`) @@ -94,10 +150,11 @@ var ( // whether two copies carry the same one. // // Dropped: a leading "Artist - " (a video upload's title), bracketed featuring -// credits and remaster notes, and bracketed segments carrying a video-rip marker -// ("(Official Video)", "[HD]"). Kept: every other bracketed word, so "(live)", -// "(demo)", "(acoustic)", "(remix)" and "(instrumental)" still tell recordings -// apart. Then case, "&" against "and", and punctuation and spacing are ignored. +// credits and remaster notes, and bracketed segments carrying a video-rip +// marker ("(Official Video)", "[HD]"). Kept: every other bracketed word, so +// "(live)", "(demo)", "(acoustic)", "(remix)" and "(instrumental)" still tell +// recordings apart. Then case, "&" against "and", and punctuation and spacing +// are ignored. func MatchTitleKey(title, artist string) string { t := strings.TrimSpace(title) if a := strings.TrimSpace(artist); a != "" { @@ -123,13 +180,6 @@ func MatchTitleKey(title, artist string) string { return alnumKey(title) } -// ReleaseTitleKey is the lighter comparison for asking whether one release -// lists a song twice: case, "&" and punctuation only. Every word stays, so a -// box set's "Song" and "Song (demo)" are two songs, as the release means them. -func ReleaseTitleKey(title string) string { - return alnumKey(title) -} - // alnumKey lowercases, reads "&" as "and", and keeps only letters and digits. func alnumKey(s string) string { s = strings.ReplaceAll(strings.ToLower(s), "&", " and ") diff --git a/internal/library/duplicate_classify_test.go b/internal/library/duplicate_classify_test.go index 81f4e9f6..9aea59c3 100644 --- a/internal/library/duplicate_classify_test.go +++ b/internal/library/duplicate_classify_test.go @@ -1,8 +1,13 @@ package library -import "testing" +import ( + "fmt" + "testing" -// The shapes measured on the operator's library (milestone 498's body). + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +// The group shapes the classifier tells apart. func TestClassifyDuplicateGroup(t *testing.T) { m := func(album, title string) ClassifyMember { return ClassifyMember{AlbumID: album, Title: title, ArtistName: "Gorillaz"} @@ -17,18 +22,26 @@ func TestClassifyDuplicateGroup(t *testing.T) { []ClassifyMember{m("humanz", "Saturnz Barz"), m("humanz", "Saturnz Barz (Official Video)")}, ClassSameRelease}, {"one album, video upload title with artist prefix", "acoustic", []ClassifyMember{m("humanz", "Andromeda"), m("humanz", "Gorillaz - Andromeda (Official Audio)")}, ClassSameRelease}, - {"one album, identical bytes under different titles", "exact", - []ClassifyMember{m("www", "WWW"), m("www", "WWW (instrumental)")}, ClassSameRelease}, - {"one album, acoustic match, different recordings named", "acoustic", - []ClassifyMember{m("www", "WWW"), m("www", "WWW (instrumental)")}, ClassReview}, + {"one album, identical bytes under a version title: one file in two slots", "exact", + []ClassifyMember{m("www", "WWW"), m("www", "WWW (instrumental)")}, ClassMismatch}, + {"one album, acoustic match, two versions named", "acoustic", + []ClassifyMember{m("moe", "Lovesick"), m("moe", "Lovesick (Instrumental)")}, ClassDistinct}, + {"Void: one album, one audio under three song titles", "acoustic", + []ClassifyMember{m("void", "Slimewar (into the void)"), m("void", "Void"), m("void", "Reach")}, ClassMismatch}, + {"a version named after a dash", "acoustic", + []ClassifyMember{m("andora", "Goodbye Yesterday"), m("andora", "Goodbye Yesterday - Instrumental Version")}, ClassDistinct}, + {"a dash that names no version", "acoustic", + []ClassifyMember{m("a", "Monday Chill - Part One"), m("a", "Monday Chill - Part Two")}, ClassMismatch}, + {"Slasher and Slasher II are not versions of one song", "acoustic", + []ClassifyMember{m("slasher", "Slasher"), m("slasher", "Slasher II")}, ClassMismatch}, {"Big Me: the single and the album", "exact", []ClassifyMember{m("single", "Big Me"), m("album", "Big Me")}, ClassCrossRelease}, {"remaster and featuring credits are one song", "acoustic", []ClassifyMember{m("a", "Feel Good Inc. (2017 Remaster)"), m("b", "Feel Good Inc feat. De La Soul")}, ClassCrossRelease}, - {"Nervosa / Anorexia: identical bytes, different songs", "exact", - []ClassifyMember{m("a", "Nervosa"), m("b", "Anorexia")}, ClassMismatch}, - {"studio and live across albums", "acoustic", - []ClassifyMember{m("studio", "Charger"), m("live", "Charger (live)")}, ClassReview}, + {"a later release retitles the recording", "acoustic", + []ClassifyMember{m("coco", "Ragtime Cat"), m("single", "The Ragtime Cat")}, ClassCrossRelease}, + {"identical bytes on two albums under two titles", "exact", + []ClassifyMember{m("dookie", "F.O.D."), m("live", "F.O.D. (live)")}, ClassCrossRelease}, {"a lone copy", "exact", []ClassifyMember{m("a", "Song")}, ClassReview}, } for _, tc := range cases { @@ -50,7 +63,8 @@ func TestMatchTitleKey(t *testing.T) { {"Song (live)", "", "songlive"}, {"Song (demo)", "", "songdemo"}, {"Song (Remix)", "", "songremix"}, - // A title that is nothing but a marker keeps its words rather than vanishing. + // A title that is nothing but a marker keeps its words rather than + // vanishing. {"(Official Video)", "", "officialvideo"}, } for _, tc := range cases { @@ -60,12 +74,118 @@ func TestMatchTitleKey(t *testing.T) { } } -// The release check keeps every word: a box set's demo is its own song. -func TestReleaseTitleKey(t *testing.T) { - if ReleaseTitleKey("Song (demo)") == ReleaseTitleKey("Song") { - t.Error("ReleaseTitleKey collapsed a demo into the song") +// Copies whose file names say they are different versions must disagree, so +// the fold never merges them. And copies of one version +// named differently must agree. +func TestVersionWords(t *testing.T) { + differ := [][2][2]string{ + {{"Suite-Pee", "/m/SOAD/Sugar/System of a Down - Sugar - 02 - Suite-Pee.mp3"}, + {"Suite-Pee", "/m/SOAD/Sugar/System of a Down - Sugar - 02 - Suite-Pee (Clean Version).mp3"}}, + {{"Luv (sic) Grand Finale", "/m/Nujabes/Hexalogy/Nujabes - Luv(sic) Hexalogy - 06 - Luv (sic) Grand Finale Instrumentals.mp3"}, + {"Luv (sic) Grand Finale", "/m/Nujabes/Hexalogy/0106 - Luvsic Grand Finale.m4a"}}, + {{"Charger", "/m/Gorillaz/Humanz/Gorillaz - Humanz - 08 - Charger - live in Cologne 2017.mp3"}, + {"Charger", "/m/Gorillaz/Humanz/CD 01/08 Charger.flac"}}, } - if ReleaseTitleKey("Hallelujah Money") != ReleaseTitleKey("hallelujah money!") { - t.Error("ReleaseTitleKey kept case or punctuation") + for _, p := range differ { + a, b := versionWords(p[0][0], p[0][1]), versionWords(p[1][0], p[1][1]) + if a == b { + t.Errorf("versionWords agree (%q) on %s and %s", a, p[0][1], p[1][1]) + } + } + agree := [][2][2]string{ + {{"Heat Waves (Shakur Ahmad Remix)", "/m/GA/Glass Animals - Dreamland Deluxe - 02 - Heat Waves (Shakur Ahmad Remix).mp3"}, + {"Heat Waves (Shakur Ahmad Remix)", "/m/GA/02. Heat Waves (Shakur Ahmad Remix).flac"}}, + {{"Luv (sic) Instrumentals", "/m/Nujabes/0201 - Luvsic instrumentals.m4a"}, + {"Luv (sic) Instrumentals", "/m/Nujabes/Nujabes - Luv(sic) Hexalogy - 01 - Luv (sic) Instrumentals.mp3"}}, + {{"Henrietta", "/m/The Fratellis - Costello Music - 01 - Henrietta.mp3"}, + {"Henrietta", "/m/The Fratellis - Costello Music - 01 - Henrietta.flac"}}, + } + for _, p := range agree { + a, b := versionWords(p[0][0], p[0][1]), versionWords(p[1][0], p[1][1]) + if a != b { + t.Errorf("versionWords %q vs %q on copies of one version: %s, %s", a, b, p[0][1], p[1][1]) + } + } +} + +func TestWrongSlots(t *testing.T) { + track := func(title string, ms int32) dbq.MbReleaseTrack { return dbq.MbReleaseTrack{Title: title, LengthMs: ms} } + copies := func(titles ...string) []slotCopy { + out := make([]slotCopy, len(titles)) + for i, title := range titles { + out[i] = slotCopy{Title: title, Artist: "Savant", DurationMs: 220000} + } + return out + } + void := []dbq.MbReleaseTrack{track("Slimewar (Into the Void)", 300000), track("Reach", 281000), track("Void", 220500)} + cases := []struct { + name string + copies []slotCopy + tracks []dbq.MbReleaseTrack + keep int + losers []int + ok bool + }{ + {"Void: the file is Void's; the other two slots are other songs", copies("Slimewar (into the void)", "Void", "Reach"), void, 1, []int{0, 2}, true}, + {"no track list", copies("Void", "Reach"), nil, -1, nil, false}, + {"a title the track list does not hold", copies("Void", "Bonus"), void, -1, nil, false}, + {"Killjoy held twice beside Dreamland's slot", []slotCopy{ + {Title: "Dreamland", Artist: "Fox Stevenson", DurationMs: 167000}, + {Title: "Killjoy", Artist: "Fox Stevenson", DurationMs: 167000}, + {Title: "Killjoy", Artist: "Fox Stevenson", DurationMs: 166000}, + }, []dbq.MbReleaseTrack{track("Killjoy", 166500), track("Dreamland", 214000)}, 1, []int{0}, true}, + {"two slots fit the file", copies("Void", "Reach"), []dbq.MbReleaseTrack{track("Void", 220000), track("Reach", 221500)}, -1, nil, false}, + {"a slot near but not within the fit", copies("Void", "Reach"), []dbq.MbReleaseTrack{track("Void", 220000), track("Reach", 226000)}, -1, nil, false}, + {"no slot fits, and every slot is clearly another song's", copies("Reach", "Slimewar (into the void)"), void, -1, []int{0, 1}, true}, + {"no slot fits, but one is near", copies("Void", "Reach"), []dbq.MbReleaseTrack{track("Void", 226000), track("Reach", 281000)}, -1, nil, false}, + {"an unknown length counts for nothing", copies("Void", "Reach"), []dbq.MbReleaseTrack{track("Void", 220000), track("Reach", 0)}, -1, nil, false}, + } + for _, tc := range cases { + keep, losers, ok := wrongSlots(tc.copies, tc.tracks) + if keep != tc.keep || ok != tc.ok || fmt.Sprint(losers) != fmt.Sprint(tc.losers) { + t.Errorf("%s: got keep=%d losers=%v ok=%v, want keep=%d losers=%v ok=%v", + tc.name, keep, losers, ok, tc.keep, tc.losers, tc.ok) + } + } +} + +// The lengths cannot place a recording filed as several songs (Savant's +// Mortals, cut to each slot's length); what AcoustID calls it can. +func TestNamedCopies(t *testing.T) { + track := func(title string, ms int32, rec string) dbq.MbReleaseTrack { + return dbq.MbReleaseTrack{Title: title, LengthMs: ms, RecordingMbid: rec} + } + copies := []slotCopy{ + {Title: "Tormentor", Artist: "The Wombats", DurationMs: 220000}, + {Title: "Record Shop", Artist: "The Wombats", DurationMs: 181000}, + } + album := []dbq.MbReleaseTrack{ + track("Tormentor", 221000, "rec-tormentor"), track("Record Shop", 180000, "rec-record-shop"), track("Ice Cream", 200000, "rec-ice"), + } + cases := []struct { + name string + copies []slotCopy + tracks []dbq.MbReleaseTrack + recordings []string + keep int + losers []int + ok bool + }{ + {"AcoustID names Tormentor", copies, album, []string{"rec-tormentor"}, 0, []int{1}, true}, + {"one of several candidates is on the album", copies, album, []string{"rec-elsewhere", "rec-record-shop"}, 1, []int{0}, true}, + {"AcoustID knows none", copies, album, nil, -1, []int{0, 1}, true}, + {"the recording is another song on the album", copies, album, []string{"rec-ice"}, -1, []int{0, 1}, true}, + {"candidates in both copies' slots", copies, album, []string{"rec-tormentor", "rec-record-shop"}, -1, nil, false}, + {"the track list names no recording for a slot", copies, []dbq.MbReleaseTrack{ + track("Tormentor", 221000, "rec-tormentor"), track("Record Shop", 180000, ""), + }, []string{"rec-tormentor"}, -1, nil, false}, + {"every copy filed as the one song named", []slotCopy{copies[0], copies[0]}, album, []string{"rec-tormentor"}, -1, nil, false}, + } + for _, tc := range cases { + keep, losers, ok := namedCopies(tc.copies, tc.tracks, tc.recordings) + if keep != tc.keep || ok != tc.ok || fmt.Sprint(losers) != fmt.Sprint(tc.losers) { + t.Errorf("%s: got keep=%d losers=%v ok=%v, want keep=%d losers=%v ok=%v", + tc.name, keep, losers, ok, tc.keep, tc.losers, tc.ok) + } } } diff --git a/internal/library/duplicate_match.go b/internal/library/duplicate_match.go index d45b83f4..c3dd29fb 100644 --- a/internal/library/duplicate_match.go +++ b/internal/library/duplicate_match.go @@ -6,16 +6,16 @@ import ( "sort" ) -// Duplicate matching (M400 #3909). +// Duplicate matching. // -// Pure functions over fingerprints: no database, no files. This is the part that -// decides whether two tracks in the operator's library are proposed as one -// recording, so every rule in it has to be falsifiable in a unit test. +// Pure functions over fingerprints: no database, no files. This is the part +// that decides whether two tracks in the library are proposed as one recording, +// so every rule in it has to be falsifiable in a unit test. // // Two tiers, answering different questions: // // exact equal audio_stream_sha256 — the same encoded audio bytes. No score, -// no threshold, no false positives (the #3885 pair). +// no threshold, no false positives. // acoustic chromaprint fingerprints that agree closely once aligned — the // same recording at another bitrate or in another codec. // @@ -52,23 +52,22 @@ const minDistinctFraction = 0.3 // fingerprints may show and still be proposed as one recording. Unrelated audio // sits near 0.5; the same recording re-encoded lands well under 0.1. // -// Deliberately conservative. The operator's stated worry is the opposite of a -// missed duplicate: "the same song can appear in different albums, usually it's -// a different recording", and an instrumental shares its vocal version's -// harmony, which chroma features capture. A false merge is the failure that -// matters, and the report is reviewed anyway. This is an unmeasured default: -// calibrate it against real pairs once the backfill (#3908) has populated the -// library, then expose it in Settings (#3913). +// Deliberately conservative. A false merge is the failure that matters, worse +// than a missed duplicate: the same song on different albums is usually a +// different recording, and an instrumental shares its vocal version's harmony, +// which chroma features capture. An admin can change it in Settings +// (FingerprintSettings). const defaultAcousticMaxBitErrorRate = 0.15 -// durationToleranceMs is how far apart two tracks' durations may be and still be -// compared. Encoders pad and trim a little; different edits differ by more. +// durationToleranceMs is how far apart two tracks' durations may be and still +// be compared. Encoders pad and trim a little; different edits differ by more. const durationToleranceMs = 3000 -// maxAcousticGroupSize caps an acoustic group. A cluster bigger than this is far -// more likely a shared jingle, a skit or a low-information pattern than eight -// copies of one recording, and proposing it would bury the real duplicates. -// Exact-tier groups are not capped: identical bytes are identical however many. +// maxAcousticGroupSize caps an acoustic group. A cluster bigger than this is +// far more likely a shared jingle, a skit or a low-information pattern than +// eight copies of one recording, and proposing it would bury the real +// duplicates. Exact-tier groups are not capped: identical bytes are identical +// however many. const maxAcousticGroupSize = 8 // acousticScore is the result of comparing two fingerprints. @@ -84,8 +83,8 @@ type acousticScore struct { // preparedPrint is a fingerprint with the parts every comparison needs worked // out once. The sweep compares each track with every other track within a few -// seconds of its duration, so rebuilding the alignment index for each pair would -// dominate its cost. +// seconds of its duration, so rebuilding the alignment index for each pair +// would dominate its cost. type preparedPrint struct { items []int32 index map[uint32][]int @@ -208,13 +207,13 @@ type duplicateGroup struct { type groupingResult struct { Groups []duplicateGroup // OversizeClusters counts acoustic clusters discarded for exceeding - // maxAcousticGroupSize. Reported rather than silent: a sudden rise means the - // cap or the information floor needs attention. + // maxAcousticGroupSize. Reported rather than silent: a sudden rise means + // the cap or the information floor needs attention. OversizeClusters int } -// groupUnit is one thing the acoustic pass compares: a single track, or an exact -// group standing in for all its byte-identical copies. +// groupUnit is one thing the acoustic pass compares: a single track, or an +// exact group standing in for all its byte-identical copies. type groupUnit struct { ids []string // every member, sorted durationMs int32 @@ -224,11 +223,11 @@ type groupUnit struct { assigned bool } -// streamGrouper is the acoustic pass over units arriving in (durationMs, sortKey) -// order. It holds only the units within durationToleranceMs of the oldest one -// not yet settled, so memory is bounded by the densest few seconds of the -// library rather than by its size — the whole library's fingerprints would be -// hundreds of megabytes. +// streamGrouper is the acoustic pass over units arriving in (durationMs, +// sortKey) order. It holds only the units within durationToleranceMs of the +// oldest one not yet settled, so memory is bounded by the densest few seconds +// of the library rather than by its size — the whole library's fingerprints +// would be hundreds of megabytes. // // A seed can be settled as soon as a unit arrives beyond its window: everything // it could group with has already arrived, and no later seed can reach back to @@ -237,9 +236,9 @@ type groupUnit struct { // // Grouping is COMPLETE-LINKAGE: a unit joins a group only if it matches every // unit already in it, within the duration tolerance and the bit-error limit. -// Single-linkage would let a chain of near-misses — A close to B, B close to C — -// drag A and C, which are not close, into one proposed merge. Complete linkage -// also means any member can be chosen as the survivor (#3911). +// Single-linkage would let a chain of near-misses — A close to B, B close to C +// — drag A and C, which are not close, into one proposed merge. Complete +// linkage also means any member can be chosen as the survivor. // // When an acoustic group absorbs an exact group, the result is tier acoustic: a // group is only as certain as its weakest link. @@ -320,7 +319,8 @@ func (g *streamGrouper) settleOldest() { for _, member := range group { member.assigned = true // The acoustic evidence is untrustworthy; identical bytes are not. - // An exact group caught inside an oversize cluster is still proposed. + // An exact group caught inside an oversize cluster is still + // proposed. if member.exact { g.res.Groups = append(g.res.Groups, duplicateGroup{Tier: tierExact, Members: member.ids}) } diff --git a/internal/library/duplicate_merge.go b/internal/library/duplicate_merge.go index aa5e411b..54f4cd40 100644 --- a/internal/library/duplicate_merge.go +++ b/internal/library/duplicate_merge.go @@ -16,31 +16,20 @@ import ( syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) -// Duplicate merge (M400 #3911). +// Duplicate merge. // ErrDuplicateGroupNotPending means the group was already merged or dismissed, // no longer exists, or no longer has two members to merge. var ErrDuplicateGroupNotPending = errors.New("library: duplicate group is not pending") -// ErrSurvivorNotInGroup means the copy chosen to keep is not a member of the group. +// ErrSurvivorNotInGroup means the copy chosen to keep is not a member of the +// group. var ErrSurvivorNotInGroup = errors.New("library: survivor is not a member of the group") -// ErrCopyTrackedByLidarr means a copy the merge would remove is one Lidarr maps -// to a track of the release it monitors (M498). Removing it would open a hole -// that Lidarr fills by downloading the song again, so the merge refuses and -// changes nothing. -var ErrCopyTrackedByLidarr = errors.New("library: a copy to remove is tracked by Lidarr") - -// RemovableFunc reports whether the file at filePath may be removed. A merge -// asks it about every copy it would remove, before touching anything. -type RemovableFunc func(filePath string) bool - // MergedCopy is one copy a merge kept or removed. type MergedCopy struct { - TrackID pgtype.UUID - FilePath string - TrackMbid *string - AlbumMbid *string + TrackID pgtype.UUID + FilePath string } // MergeResult says what a merge did. @@ -49,7 +38,7 @@ type MergeResult struct { Survivor MergedCopy Removed []MergedCopy - // What moved onto the survivor — reported so the operator, and the audit + // What moved onto the survivor — reported so an admin, and the audit // log, can see that the history was kept rather than take it on trust. PlayEvents int64 SkipEvents int64 @@ -72,30 +61,19 @@ type MergeResult struct { // survivor first, and only then deletes the now-empty row. // // It deletes the removed copies' FILES too, and first, before any row changes -// (#3918, note #3926). A merge that left the file behind would be undone by the -// next scan, which re-imports it as a new track with no history. If a file cannot -// be removed, the *FileRemoveError comes back and nothing in the database changes. -// With several copies to remove, one file may already be gone when a later one -// fails; that copy's row keeps all its history and is marked missing by the next -// scan, and retrying the merge picks up where it stopped. +// (DeleteTrackFile's order). A merge that left the file behind would be undone +// by the next scan, which re-imports it as a new track with no history. If a +// file cannot be removed, the *FileRemoveError comes back and nothing in the +// database changes. With several copies to remove, one file may already be gone +// when a later one fails; that copy's row keeps all its history and is marked +// missing by the next scan, and retrying the merge picks up where it stopped. // -// Everything else happens in one transaction, which holds a lock on the group so -// two merges of it cannot run at once. Sync changes for clients' caches are logged -// inside it, the way the playlists service logs its own. +// Everything else happens in one transaction, which holds a lock on the group +// so two merges of it cannot run at once. Sync changes for clients' caches are +// logged inside it, the way the playlists service logs its own. func MergeDuplicateGroup( ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, groupID, survivorID pgtype.UUID, -) (MergeResult, error) { - return MergeDuplicateGroupGuarded(ctx, pool, logger, dataDir, groupID, survivorID, nil) -} - -// MergeDuplicateGroupGuarded is MergeDuplicateGroup with a check on what it may -// remove: when removable says no to any copy that would go, the merge returns -// ErrCopyTrackedByLidarr (wrapped with the path) and changes nothing. A nil -// removable allows every copy. -func MergeDuplicateGroupGuarded( - ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, - groupID, survivorID pgtype.UUID, removable RemovableFunc, ) (MergeResult, error) { if logger == nil { logger = slog.Default() @@ -130,13 +108,6 @@ func MergeDuplicateGroupGuarded( return MergeResult{}, err } - if removable != nil { - for _, l := range losers { - if !removable(l.FilePath) { - return MergeResult{}, fmt.Errorf("%w: %s", ErrCopyTrackedByLidarr, l.FilePath) - } - } - } for _, l := range losers { if err := removeTrackFileOnDisk(l.FilePath); err != nil { return MergeResult{}, err @@ -182,7 +153,8 @@ func MergeDuplicateGroupGuarded( return MergeResult{}, fmt.Errorf("commit merge: %w", err) } - // After commit, like DeleteTrackFile: a leftover art directory is only disk. + // After commit, like DeleteTrackFile: a leftover art directory is only + // disk. if dataDir != "" { for _, artistID := range res.DeletedArtistIDs { if err := coverart.CleanupArtistArt(dataDir, artistID); err != nil { @@ -206,11 +178,11 @@ type foldResult struct { } // foldTrackInto moves everything the loser row carries onto the survivor, then -// deletes the loser row and tidies an album or artist it leaves empty. It is the -// row half of a merge, shared by the duplicate merge and the missing-pair pass -// (M485): neither touches a file here, and both run it inside their own -// transaction. The sync changes it owes clients go into changes, which the -// caller logs once before committing. +// deletes the loser row and tidies an album or artist it leaves empty. It is +// the row half of a merge, shared by the duplicate merge and the missing-pair +// pass (PairMissingTracks): neither touches a file here, and both run it inside +// their own transaction. The sync changes it owes clients go into changes, +// which the caller logs once before committing. func foldTrackInto( ctx context.Context, tq *dbq.Queries, survivorID, loserID pgtype.UUID, changes *mergeChanges, ) (foldResult, error) { @@ -234,8 +206,8 @@ func foldTrackInto( if _, err := tq.MergeRepointPlaybackErrors(ctx, dbq.MergeRepointPlaybackErrorsParams{SurvivorID: survivorID, LoserID: loserID}); err != nil { return f, fmt.Errorf("move playback errors: %w", err) } - if _, err := tq.MergeRepointLidarrRequests(ctx, dbq.MergeRepointLidarrRequestsParams{SurvivorID: survivorID, LoserID: loserID}); err != nil { - return f, fmt.Errorf("move lidarr request matches: %w", err) + if _, err := tq.MergeRepointRequests(ctx, dbq.MergeRepointRequestsParams{SurvivorID: survivorID, LoserID: loserID}); err != nil { + return f, fmt.Errorf("move request matches: %w", err) } playlists, err := tq.MergeRepointPlaylistTracks(ctx, dbq.MergeRepointPlaylistTracksParams{SurvivorID: survivorID, LoserID: loserID}) @@ -324,7 +296,7 @@ func splitSurvivor( } func mergedCopyOf(m dbq.ListDuplicateGroupMergeMembersRow) MergedCopy { - return MergedCopy{TrackID: m.ID, FilePath: m.FilePath, TrackMbid: m.Mbid, AlbumMbid: m.AlbumMbid} + return MergedCopy{TrackID: m.ID, FilePath: m.FilePath} } // mergeChanges collects the sync-log entries a merge owes clients' caches, each diff --git a/internal/library/duplicate_merge_test.go b/internal/library/duplicate_merge_test.go index 3a6781fe..3d9def22 100644 --- a/internal/library/duplicate_merge_test.go +++ b/internal/library/duplicate_merge_test.go @@ -83,7 +83,8 @@ func newMergeFixture(t *testing.T) mergeFixture { } f.alice, f.bob = user("merge-alice"), user("merge-bob") - // Alice liked both copies, the removed one first; Bob liked only the removed one. + // Alice liked both copies, the removed one first; Bob liked only the + // removed one. f.aliceEarlierLikeOnRem = time.Now().Add(-72 * time.Hour).UTC().Truncate(time.Microsecond) mustExec(`INSERT INTO general_likes (user_id, track_id, liked_at) VALUES ($1, $2, $3), ($1, $4, now()), ($5, $2, now())`, f.alice.ID, f.remove.ID, f.aliceEarlierLikeOnRem, f.keep.ID, f.bob.ID) @@ -144,9 +145,8 @@ func (f mergeFixture) count(t *testing.T, sql string, args ...any) int { return n } -// The #3911 proof: after a merge, every piece of history the removed copy held -// is on the copy kept, nothing is doubled, and the removed copy — row and file — -// is gone. +// After a merge, every piece of history the removed copy held is on the copy +// kept, nothing is doubled, and the removed copy — row and file — is gone. func TestMergeDuplicateGroup_Integration(t *testing.T) { f := newMergeFixture(t) ctx := context.Background() @@ -208,8 +208,8 @@ func TestMergeDuplicateGroup_Integration(t *testing.T) { t.Errorf("the kept copy's own house weight was overwritten") } - // Similarity: rewritten onto the kept copy, no duplicate edge, no self-edge, - // nothing left pointing at the removed copy. + // Similarity: rewritten onto the kept copy, no duplicate edge, no + // self-edge, nothing left pointing at the removed copy. if n := f.count(t, `SELECT count(*) FROM track_similarity WHERE track_a_id = $1 AND track_b_id = $2 AND source = 'listenbrainz'`, f.keep.ID, f.other.ID); n != 1 { t.Errorf("listenbrainz edge keep→other = %d rows, want exactly 1", n) diff --git a/internal/library/duplicate_resolve.go b/internal/library/duplicate_resolve.go index 5dd4b9ee..a41c0480 100644 --- a/internal/library/duplicate_resolve.go +++ b/internal/library/duplicate_resolve.go @@ -2,12 +2,9 @@ package library import ( "context" - "encoding/json" "errors" "fmt" "log/slog" - "sort" - "strings" "time" "github.com/jackc/pgx/v5" @@ -16,152 +13,145 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/audit" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) -// Duplicate resolver (M498 #5435, #5437). +// Duplicate resolver. // // The sweep proposes duplicate groups; the resolver decides what each one is -// and settles the ones that are safe to settle without asking. Safety is -// Lidarr's to define: Lidarr maps one file to each track of the release it -// monitors, and re-downloads any mapped file that disappears. So: +// and settles the ones that are safe to settle without asking. Minstrel is the +// only program writing the library, so no copy is kept because something else +// depends on it. What decides is what the copies are: // -// - A copy Lidarr maps is never removed. -// - On one album, copies Lidarr does not map are merged into the copy it does -// (or, when it maps none, into the best by ProposeSurvivor). -// - When Lidarr maps every copy on one album, the release it monitors lists -// the song twice: the Humanz box set, 14 vinyl sides then the digital -// medium. The resolver moves Lidarr to a release of that album that lists -// each song once. Lidarr then rescans the folder, maps one copy and leaves -// the rest unmapped, and the next pass merges them. -// -// The release change has a fixed point (lesson #4183): it happens only while -// the monitored release repeats a title, and the release it chooses repeats -// none, so the next pass finds nothing to change. - -// LidarrLibrary is the part of Lidarr's API the resolver uses. *lidarr.Client -// satisfies it. -type LidarrLibrary interface { - ListUnmappedTrackFiles(ctx context.Context) ([]lidarr.TrackFile, error) - LookupAlbumByMBID(ctx context.Context, mbid string) (lidarr.LidarrAlbum, error) - ListAlbumTracks(ctx context.Context, albumID int) ([]lidarr.ReleaseTrack, error) - GetAlbumReleases(ctx context.Context, albumID int) ([]lidarr.AlbumRelease, error) - ListReleaseTracks(ctx context.Context, releaseID int) ([]lidarr.ReleaseTrack, error) - SetMonitoredRelease(ctx context.Context, albumID, releaseID int) error -} +// - On one album, a song is held once: the extra copies are merged into the +// best by ProposeSurvivor. +// - Across releases, each copy belongs to its own release (a single and its +// album): nothing is removed, and the copies are linked as one song. An +// album holding the song twice among them still holds it once: its extra +// copies are merged into its best. +// - Except that a video rip is no release's copy. When every copy but one is +// a rip, across releases or under another title, the rips are merged into +// the one that is not. +// - Two versions of a song on one album that sound alike are not duplicates: +// the group is dismissed (duplicate_settle.go). +// - One file filling several of an album's track slots: MusicBrainz's track +// lengths say which slot it belongs to, or AcoustID which song it is when +// each copy was cut to its own slot's length, and the copies in the other +// slots are removed so the album is fetched again (duplicate_settle.go). +// - Anything else waits for an admin. const ( // resolveMergeCap bounds the merges one pass makes. The first pass after - // deploy meets the whole backlog (433 groups measured on 2026-10-09); a - // cap spreads it over a few hours, each pass short. + // deploy meets the whole backlog; a cap spreads it over a few hours, each + // pass short. resolveMergeCap = 200 - // resolveReleaseCap bounds the Lidarr release changes one pass makes. Each - // makes Lidarr rescan an artist folder. - resolveReleaseCap = 10 - // releaseChangeSettle is how long after a release change the resolver - // leaves that album's groups alone. Lidarr unmaps every file of the album - // and rescans; until the rescan lands, every copy reads as unmapped, and a - // merge then could remove the very file Lidarr is about to map. - releaseChangeSettle = 24 * time.Hour // duplicateResolveTick is how often the worker runs a pass. duplicateResolveTick = time.Hour + // scanWaitRetry is how soon a pass put off for a running scan is tried + // again; a boot scan is short. + scanWaitRetry = time.Minute ) -const ( - lidarrStateTracked = "tracked" - lidarrStateUnmapped = "unmapped" -) - -// ReleaseChange is one Lidarr release the resolver switched. -type ReleaseChange struct { - AlbumTitle string - ArtistName string - From, To lidarr.AlbumRelease -} - // DuplicateResolveResult tallies one pass. type DuplicateResolveResult struct { - Groups int - Classes map[DuplicateClass]int - LidarrConsulted bool - Merged int - MergeFailed int - ReleaseChanges []ReleaseChange + Groups int + Classes map[DuplicateClass]int + Merged int + MergeFailed int // SongsLinked counts cross-release groups newly linked as one song. SongsLinked int + // Dismissed counts groups of distinct versions dismissed; WrongFiles the + // copies removed from track slots that were not theirs. + Dismissed, WrongFiles int // HeldBack counts tracks newly held back from radio and the mixes as video // rips; Released counts held-back tracks whose name no longer says so. HeldBack, Released int64 } -// LidarrPathKey is the part of a path Minstrel and Lidarr agree on: the last -// three components (artist folder, album folder, file). Both see the library -// under their own mount; on the operator's deploy both happen to say /music, -// and the key keeps the match from depending on that. -func LidarrPathKey(p string) string { - parts := strings.Split(strings.Trim(p, "/"), "/") - if len(parts) > 3 { - parts = parts[len(parts)-3:] - } - return strings.Join(parts, "/") -} - // resolveGroup is one pending group and its present copies. type resolveGroup struct { - id pgtype.UUID - tier string - members []dbq.ListDuplicateGroupsForResolveRow - class DuplicateClass - note string - states []string // per member: tracked, unmapped, or "" when Lidarr was not asked + id pgtype.UUID + tier string + // worstBER is how far apart the least alike copies' fingerprints are, for + // an acoustic group; nil for an exact one. + worstBER *float32 + members []dbq.ListDuplicateGroupsForResolveRow + class DuplicateClass + note string } -// autoMergeable says whether the resolver may merge the group on its own: when -// every copy it would remove is one Lidarr does not map (D-a rule 1). On one -// album that is any group with at most one mapped copy. Across releases each -// mapped copy fulfils its own release (D-b), so only a group with exactly one -// mapped copy qualifies: the others fulfil nothing — a rip beside the clean -// copy on another release, or a wrong-file import Lidarr holds unmapped -// (#5439) — and the mapped copy is the clear one to keep. -func autoMergeable(g *resolveGroup) bool { +// isRip says whether a copy is a video rip: held back as one, or named like +// one and not kept by an admin. The name is read as well as the verdict +// because a pass reads the groups before it flags new rips. +func isRip(m dbq.ListDuplicateGroupsForResolveRow) bool { + if m.SourceVerdict != nil { + return *m.SourceVerdict == "suspect" + } + return len(SourceMarkersFor(m.FilePath)) > 0 +} + +// lone returns the index of the group's only copy that is not a rip, or -1 +// when there is none or more than one. +func (g *resolveGroup) lone() int { + at := -1 + for i, m := range g.members { + if isRip(m) { + continue + } + if at >= 0 { + return -1 + } + at = i + } + return at +} + +// mergePlan says whether the resolver may merge the group on its own, and into +// which copy (an index into members; -1 means ProposeSurvivor decides). +func mergePlan(g *resolveGroup) (ok bool, survivor int) { switch g.class { case ClassSameRelease: - return g.tracked() <= 1 - case ClassCrossRelease, ClassMismatch: - return g.tracked() == 1 - default: - return false - } -} - -// settling says whether any of the group's albums had its Lidarr release -// changed recently, so Lidarr may still be remapping its files. -func (g *resolveGroup) settling(albums map[string]bool) bool { - for _, m := range g.members { - if albums[syncpkg.FormatUUID(m.AlbumID)] { - return true + return true, -1 + case ClassCrossRelease: + if i := g.lone(); i >= 0 { + return true, i } } - return false + return false, -1 } -func (g *resolveGroup) tracked() int { - n := 0 - for _, s := range g.states { - if s == lidarrStateTracked { - n++ - } - } - return n +// TrackListFetcher reads a release group's releases and track lists from +// MusicBrainz into the catalogue (catalogue.Service.RefreshReleaseGroup). +type TrackListFetcher func(ctx context.Context, releaseGroupMBID string) error + +// RecordingIdentifier asks AcoustID which recordings a file may be +// (AcoustIDLookupWorker.IdentifyRecordings). +type RecordingIdentifier func(ctx context.Context, path string) ([]string, error) + +// ResolveSources are what a pass may ask beyond the library, a few times per +// pass each. Either may be nil, and the pass decides from what is held. +type ResolveSources struct { + // TrackLists reads the track list of an album whose wrong slot the pass + // must decide and the catalogue has never read (trackListFetchesPerPass). + TrackLists TrackListFetcher + // Identify names the recording cut to several slots' lengths, when no + // copy's own lookup has (identifiesPerPass). + Identify RecordingIdentifier } -// ResolveDuplicates runs one pass. lid is nil when Lidarr is disabled; act is -// the operator's auto-resolve setting. Without Lidarr, or with act off, the -// pass only classifies: nothing is removed when Lidarr cannot say what is safe. +// ResolveDuplicates runs one pass. act is the auto-resolve setting; +// with it off, the pass only classifies. It decides from what the library and +// catalogue hold; ResolveDuplicatesWith may also ask MusicBrainz and AcoustID. func ResolveDuplicates( - ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, lid LidarrLibrary, act bool, + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, act bool, +) (DuplicateResolveResult, error) { + return ResolveDuplicatesWith(ctx, pool, logger, dataDir, act, ResolveSources{}) +} + +// ResolveDuplicatesWith is ResolveDuplicates that may ask src. +func ResolveDuplicatesWith( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, act bool, src ResolveSources, ) (DuplicateResolveResult, error) { if logger == nil { logger = slog.Default() @@ -169,42 +159,15 @@ func ResolveDuplicates( q := dbq.New(pool) res := DuplicateResolveResult{Classes: map[DuplicateClass]int{}} - rows, err := q.ListDuplicateGroupsForResolve(ctx) + rows, err := q.ListDuplicateGroupsForResolve(ctx, pgtype.UUID{}) if err != nil { return res, fmt.Errorf("list duplicate groups: %w", err) } groups := foldResolveGroups(rows) res.Groups = len(groups) - var unmapped map[string]bool - if lid != nil { - files, err := lid.ListUnmappedTrackFiles(ctx) - if err != nil { - logger.Warn("duplicate resolve: Lidarr's unmapped files unavailable; classifying only", "err", err) - } else { - unmapped = make(map[string]bool, len(files)) - for _, f := range files { - unmapped[LidarrPathKey(f.Path)] = true - } - res.LidarrConsulted = true - } - } - for _, g := range groups { - cm := make([]ClassifyMember, len(g.members)) - g.states = make([]string, len(g.members)) - for i, m := range g.members { - cm[i] = ClassifyMember{AlbumID: syncpkg.FormatUUID(m.AlbumID), Title: m.Title, ArtistName: m.ArtistName} - if res.LidarrConsulted { - if unmapped[LidarrPathKey(m.FilePath)] { - g.states[i] = lidarrStateUnmapped - } else { - g.states[i] = lidarrStateTracked - } - } - } - g.class = ClassifyDuplicateGroup(g.tier, cm) - g.note = resolveNote(g) + g.classify() res.Classes[g.class]++ } if err := writeResolveVerdicts(ctx, q, groups); err != nil { @@ -212,34 +175,20 @@ func ResolveDuplicates( } if !act { + notifyNewForReview(ctx, groups, nil) return res, nil } - // Holding rips back and linking songs remove nothing and need nothing from - // Lidarr. if res.HeldBack, err = q.FlagSuspectSourceTracks(ctx, SuspectSourcePattern); err != nil { return res, fmt.Errorf("hold back video rips: %w", err) } if res.Released, err = q.ClearStaleSuspectSourceTracks(ctx, SuspectSourcePattern); err != nil { return res, fmt.Errorf("release renamed tracks: %w", err) } - linked, err := linkCrossRelease(ctx, pool, logger, groups) - res.SongsLinked = linked - if err != nil { - return res, err - } - if !res.LidarrConsulted { - return res, nil - } - - settling, err := recentlyChangedAlbums(ctx, q, time.Now()) - if err != nil { - return res, err - } - - // Merges first, on the state Lidarr reported at the start of the pass. - removable := func(p string) bool { return unmapped[LidarrPathKey(p)] } + // Merges before links, so a rip merged away is not linked first. + merged := map[pgtype.UUID]bool{} + var mergedNames []string for _, g := range groups { if res.Merged >= resolveMergeCap { break @@ -247,42 +196,178 @@ func ResolveDuplicates( if ctx.Err() != nil { return res, ctx.Err() } - if !autoMergeable(g) || g.settling(settling) { + ok, survivor := mergePlan(g) + if !ok { continue } - if err := autoMerge(ctx, pool, logger, dataDir, g, removable); err != nil { + if err := autoMerge(ctx, pool, logger, dataDir, g, survivor); err != nil { res.MergeFailed++ logger.Warn("duplicate resolve: merge failed", "group_id", syncpkg.FormatUUID(g.id), "err", err) continue } + merged[g.id] = true res.Merged++ + if len(mergedNames) < notifyItems { + mergedNames = append(mergedNames, TrackLabel(g.members[0].ArtistName, g.members[0].Title)) + } } - // Then release changes, for albums where Lidarr maps every copy. - changes, err := changeRepeatingReleases(ctx, q, pool, logger, lid, groups, settling) - res.ReleaseChanges = changes + folded, foldedNames := foldAlbumCopies(ctx, pool, logger, dataDir, groups, merged, resolveMergeCap-res.Merged) + res.Merged += folded + mergedNames = append(mergedNames, foldedNames...) + + res.Dismissed = dismissDistinct(ctx, pool, logger, groups, merged) + wrong, names := settleWrongSlots(ctx, pool, logger, groups, merged, src) + res.WrongFiles = wrong + mergedNames = append(mergedNames, names...) + + linked, err := linkCrossRelease(ctx, pool, logger, groups, merged) + res.SongsLinked = linked if err != nil { return res, err } - if res.Merged > 0 || len(changes) > 0 { + if settledCopies := res.Merged + res.WrongFiles; settledCopies > 0 { notifyAdmins(ctx, notifications.KindDuplicatesResolved, notifications.Payload{ - Count: int64(res.Merged + len(changes)), - Detail: resolvedDetail(res.Merged, changes), + Count: int64(settledCopies), Items: mergedNames[:min(len(mergedNames), notifyItems)], }) } + notifyNewForReview(ctx, groups, merged) return res, nil } -// linkCrossRelease links each cross-release group's copies as one song -// (#5438): a like on one is a like on all, and the Liked list, shuffle and the -// mixes count the song once. Linking an already-linked group changes nothing, +// foldAlbumCopies merges, in each cross-release group, the extra copies one +// album holds of the song into the best of that album's copies: the one that is +// not a video rip when only one is not, else ProposeSurvivor's choice. Copies +// whose title or file name name different versions (versionWords) are not +// folded, even when they carry one title tag: a song and its clean version are +// both kept. The copies on other releases stay and are linked as one song. At +// most budget copies go. It returns how many went and the songs they were, and +// drops them from their groups. +func foldAlbumCopies( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, + groups []*resolveGroup, merged map[pgtype.UUID]bool, budget int, +) (int, []string) { + n := 0 + var names []string + for _, g := range groups { + if g.class != ClassCrossRelease || merged[g.id] || ctx.Err() != nil || n >= budget { + continue + } + onAlbum := map[string][]int{} + for i, m := range g.members { + k := syncpkg.FormatUUID(m.AlbumID) + "\x00" + MatchTitleKey(m.Title, m.ArtistName) + + "\x00" + versionWords(m.Title, m.FilePath) + onAlbum[k] = append(onAlbum[k], i) + } + gone := map[int]bool{} + for _, copies := range onAlbum { + if len(copies) < 2 { + continue + } + keep, reason := albumCopySurvivor(g, copies) + for _, i := range copies { + if i == keep || n >= budget { + continue + } + if err := foldAlbumCopy(ctx, pool, logger, dataDir, g, i, keep, reason); err != nil { + logger.Warn("duplicate resolve: merging an album's extra copy failed", + "group_id", syncpkg.FormatUUID(g.id), "path", g.members[i].FilePath, "err", err) + continue + } + gone[i] = true + n++ + if len(names) < notifyItems { + names = append(names, TrackLabel(g.members[i].ArtistName, g.members[i].Title)) + } + } + } + if len(gone) > 0 { + kept := g.members[:0:0] + for i, m := range g.members { + if !gone[i] { + kept = append(kept, m) + } + } + g.members = kept + } + } + return n, names +} + +// albumCopySurvivor picks which of one album's copies (indexes into the +// group's members) to keep, and why. +func albumCopySurvivor(g *resolveGroup, copies []int) (int, string) { + notRip := -1 + for _, i := range copies { + if isRip(g.members[i]) { + continue + } + if notRip >= 0 { + notRip = -1 + break + } + notRip = i + } + if notRip >= 0 { + return notRip, "the copy that is not a video rip" + } + all := survivorCandidates(g) + cands := make([]SurvivorCandidate, len(copies)) + for j, i := range copies { + cands[j] = all[i] + } + id, reason := ProposeSurvivor(cands) + for _, i := range copies { + if all[i].TrackID == id { + return i, reason + } + } + return copies[0], reason +} + +// foldAlbumCopy replaces one of an album's copies with the copy kept, history +// and all, and records it as the resolver's merge. +func foldAlbumCopy( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, + g *resolveGroup, gone, keep int, reason string, +) error { + from, into := g.members[gone], g.members[keep] + moved, err := ReplaceTrack(ctx, pool, logger, dataDir, from.TrackID, into.TrackID) + if err != nil { + return err + } + audit.WriteOrLog(ctx, pool, logger, pgtype.UUID{}, pgtype.UUID{}, audit.ActionDuplicateMerge, map[string]any{ + "automatic": true, + "group_id": syncpkg.FormatUUID(g.id), + "tier": g.tier, + "class": string(g.class), + "survivor_track_id": syncpkg.FormatUUID(into.TrackID), + "survivor_path": into.FilePath, + "survivor_reason": reason + "; the album held the song twice", + "removed": []map[string]string{ + {"track_id": syncpkg.FormatUUID(from.TrackID), "file_path": from.FilePath}, + }, + "moved": map[string]any{ + "play_events": moved.PlayEvents, "skip_events": moved.SkipEvents, + "likes": moved.Likes, "playlist_entries": moved.PlaylistEntries, + }, + }) + return nil +} + +// linkCrossRelease links each cross-release group's copies as one song: a +// like on one is a like on all, and the Liked list, shuffle and the +// mixes count the song once. A group the pass merged has nothing left to link. +// Linking an already-linked group changes nothing, // so this runs every pass; likes are shared only when a link is new, since a // like made after it reaches every copy as it is made. -func linkCrossRelease(ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, groups []*resolveGroup) (int, error) { +func linkCrossRelease( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, groups []*resolveGroup, merged map[pgtype.UUID]bool, +) (int, error) { linked := 0 for _, g := range groups { - if g.class != ClassCrossRelease { + if g.class != ClassCrossRelease || merged[g.id] { continue } if ctx.Err() != nil { @@ -338,7 +423,7 @@ func foldResolveGroups(rows []dbq.ListDuplicateGroupsForResolveRow) []*resolveGr var groups []*resolveGroup for _, r := range rows { if n := len(groups); n == 0 || groups[n-1].id != r.GroupID { - groups = append(groups, &resolveGroup{id: r.GroupID, tier: r.Tier}) + groups = append(groups, &resolveGroup{id: r.GroupID, tier: r.Tier, worstBER: r.WorstBitErrorRate}) } g := groups[len(groups)-1] g.members = append(g.members, r) @@ -353,34 +438,36 @@ func foldResolveGroups(rows []dbq.ListDuplicateGroupsForResolveRow) []*resolveGr return out } -// resolveNote is what the report says beside a group the resolver leaves for -// the operator, when there is something to say. +// classify sets the group's class and the note the report shows beside it. +func (g *resolveGroup) classify() { + cm := make([]ClassifyMember, len(g.members)) + for i, m := range g.members { + cm[i] = ClassifyMember{AlbumID: syncpkg.FormatUUID(m.AlbumID), Title: m.Title, ArtistName: m.ArtistName} + } + g.class = ClassifyDuplicateGroup(g.tier, cm) + g.note = resolveNote(g) +} + +// resolveNote is what the report says beside a group, when there is something +// to say. func resolveNote(g *resolveGroup) string { switch g.class { - case ClassSameRelease: - if g.tracked() > 1 { - return "Lidarr tracks every copy as its own track, so removing one would make Lidarr download it again." - } case ClassCrossRelease: - return "Each copy belongs to its own release, and Lidarr keeps each one. Nothing is removed; they count as one song." + if g.lone() < 0 { + return "Each copy belongs to its own release, so none is removed; they count as one song." + } case ClassMismatch: - return "Identical audio filed under different titles: one file carries another song's tags." + return "One recording fills several of this album's track slots, and MusicBrainz's track lengths could not say which is its own." } return "" } func writeResolveVerdicts(ctx context.Context, q *dbq.Queries, groups []*resolveGroup) error { classes := dbq.SetDuplicateGroupClassesParams{} - states := dbq.SetDuplicateMemberLidarrStatesParams{} for _, g := range groups { classes.GroupIds = append(classes.GroupIds, g.id) classes.Classes = append(classes.Classes, string(g.class)) classes.Notes = append(classes.Notes, g.note) - for i, m := range g.members { - states.GroupIds = append(states.GroupIds, g.id) - states.TrackIds = append(states.TrackIds, m.TrackID) - states.States = append(states.States, g.states[i]) - } } if len(classes.GroupIds) == 0 { return nil @@ -388,9 +475,6 @@ func writeResolveVerdicts(ctx context.Context, q *dbq.Queries, groups []*resolve if err := q.SetDuplicateGroupClasses(ctx, classes); err != nil { return fmt.Errorf("write duplicate classes: %w", err) } - if err := q.SetDuplicateMemberLidarrStates(ctx, states); err != nil { - return fmt.Errorf("write lidarr states: %w", err) - } return nil } @@ -399,29 +483,31 @@ func survivorCandidates(g *resolveGroup) []SurvivorCandidate { cands := make([]SurvivorCandidate, len(g.members)) for i, m := range g.members { cands[i] = SurvivorCandidate{ - TrackID: syncpkg.FormatUUID(m.TrackID), - FileFormat: m.FileFormat, - FileSize: m.FileSize, - AddedAt: m.AddedAt.Time, - LidarrTracked: g.states[i] == lidarrStateTracked, - TagFit: TagFitScore(m.TrackNumber != nil, m.PositionClash, m.FilePath, m.HasMbid), + TrackID: syncpkg.FormatUUID(m.TrackID), + FileFormat: m.FileFormat, + FileSize: m.FileSize, + AddedAt: m.AddedAt.Time, + TagFit: TagFitScore(m.TrackNumber != nil, m.PositionClash, m.FilePath, m.HasMbid), } } return cands } -// autoMerge merges one group into the copy ProposeSurvivor picks, which is the -// copy Lidarr tracks when there is one. removable is the guard: a copy Lidarr -// maps can never be among those removed, whatever the survivor rule said. +// autoMerge merges one group into the copy at index survivor, or the copy +// ProposeSurvivor picks when survivor is -1, and records the merge as the +// resolver's. func autoMerge( - ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, g *resolveGroup, removable RemovableFunc, + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, g *resolveGroup, survivor int, ) error { survivorKey, reason := ProposeSurvivor(survivorCandidates(g)) + if survivor >= 0 { + survivorKey, reason = syncpkg.FormatUUID(g.members[survivor].TrackID), "the copy that is not a video rip" + } var survivorID pgtype.UUID if err := survivorID.Scan(survivorKey); err != nil { return fmt.Errorf("parse survivor id: %w", err) } - res, err := MergeDuplicateGroupGuarded(ctx, pool, logger, dataDir, g.id, survivorID, removable) + res, err := MergeDuplicateGroup(ctx, pool, logger, dataDir, g.id, survivorID) if err != nil { return err } @@ -449,343 +535,86 @@ func autoMerge( return nil } -// recentlyChangedAlbums is the Minstrel albums whose Lidarr release the -// resolver changed within releaseChangeSettle, read from the audit log so a -// restart does not forget them. -func recentlyChangedAlbums(ctx context.Context, q *dbq.Queries, now time.Time) (map[string]bool, error) { - rows, err := q.ListAuditLogByActions(ctx, dbq.ListAuditLogByActionsParams{ - Actions: []string{string(audit.ActionLidarrReleaseChange)}, SystemOnly: true, - PageLimit: 200, PageOffset: 0, - }) - if err != nil { - return nil, fmt.Errorf("read recent release changes: %w", err) - } - out := map[string]bool{} - for _, r := range rows { - if now.Sub(r.CreatedAt.Time) > releaseChangeSettle { - break // newest first - } - var meta struct { - AlbumID string `json:"album_id"` - } - if json.Unmarshal(r.Metadata, &meta) == nil && meta.AlbumID != "" { - out[meta.AlbumID] = true - } - } - return out, nil -} - -// repeatingAlbum is a Minstrel album where Lidarr maps more than one copy of a -// song, and the groups that showed it. -type repeatingAlbum struct { - albumID pgtype.UUID - releaseGroupMbid string - title, artist string - groups []*resolveGroup -} - -func changeRepeatingReleases( - ctx context.Context, q *dbq.Queries, pool *pgxpool.Pool, logger *slog.Logger, - lid LidarrLibrary, groups []*resolveGroup, settling map[string]bool, -) ([]ReleaseChange, error) { - albums := map[string]*repeatingAlbum{} - for _, g := range groups { - if g.class != ClassSameRelease || g.tracked() < 2 { - continue - } - m := g.members[0] - key := syncpkg.FormatUUID(m.AlbumID) - if settling[key] || m.ReleaseGroupMbid == nil || *m.ReleaseGroupMbid == "" { - continue - } - a := albums[key] - if a == nil { - a = &repeatingAlbum{albumID: m.AlbumID, releaseGroupMbid: *m.ReleaseGroupMbid, title: m.AlbumTitle, artist: m.ArtistName} - albums[key] = a - } - a.groups = append(a.groups, g) - } - keys := make([]string, 0, len(albums)) - for k := range albums { - keys = append(keys, k) - } - sort.Strings(keys) - - var changes []ReleaseChange - for _, k := range keys { - if len(changes) >= resolveReleaseCap { - break - } - if ctx.Err() != nil { - return changes, ctx.Err() - } - a := albums[k] - change, note, err := changeRelease(ctx, q, lid, a) - if err != nil { - logger.Warn("duplicate resolve: Lidarr release check failed", "album", a.title, "err", err) - continue - } - if change != nil { - changes = append(changes, *change) - audit.WriteOrLog(ctx, pool, logger, pgtype.UUID{}, pgtype.UUID{}, audit.ActionLidarrReleaseChange, map[string]any{ - "automatic": true, - "album_id": k, - "album_title": a.title, - "artist_name": a.artist, - "from_release": releaseLabel(change.From), - "to_release": releaseLabel(change.To), - }) - } - if note != "" { - if err := setGroupNotes(ctx, q, a.groups, note); err != nil { - return changes, err - } - } - } - return changes, nil -} - -// changeRelease checks one album's monitored release and, when it lists a -// song twice, moves Lidarr to the release that lists each song once and best -// covers what is on disk. It returns the change made (nil when none) and the -// note for the album's groups. -func changeRelease( - ctx context.Context, q *dbq.Queries, lid LidarrLibrary, a *repeatingAlbum, -) (*ReleaseChange, string, error) { - la, err := lid.LookupAlbumByMBID(ctx, a.releaseGroupMbid) - if errors.Is(err, lidarr.ErrNotFound) { - return nil, "Lidarr does not hold this album, so its copies are left for you.", nil - } - if err != nil { - return nil, "", err - } - current, err := lid.ListAlbumTracks(ctx, la.ID) - if err != nil { - return nil, "", err - } - if !releaseRepeats(current) { - return nil, "Lidarr maps each copy to a different track of this album, so none can go without a download.", nil - } - releases, err := lid.GetAlbumReleases(ctx, la.ID) - if err != nil { - return nil, "", err - } - titles, err := q.ListAlbumPresentTrackTitles(ctx, a.albumID) - if err != nil { - return nil, "", fmt.Errorf("list album titles: %w", err) - } - onDisk := map[string]bool{} - for _, t := range titles { - if k := ReleaseTitleKey(t); k != "" { - onDisk[k] = true - } - } - - var from lidarr.AlbumRelease - var candidates []releaseCandidate - for _, r := range releases { - if r.Monitored { - from = r - continue - } - tracks, err := lid.ListReleaseTracks(ctx, r.ID) - if err != nil { - return nil, "", err - } - if len(tracks) == 0 || releaseRepeats(tracks) { - continue - } - candidates = append(candidates, releaseCandidate{release: r, coverage: coverage(tracks, onDisk)}) - } - best, ok := pickRelease(candidates, coverage(current, onDisk)) - if !ok { - return nil, "Lidarr's release of this album lists songs twice, and no other release lists each once while keeping what is on disk.", nil - } - if err := lid.SetMonitoredRelease(ctx, la.ID, best.ID); err != nil { - return nil, "", err - } - note := fmt.Sprintf("Lidarr now monitors the %s release, which lists each song once. The extra copies are merged once Lidarr has rescanned.", releaseLabel(best)) - return &ReleaseChange{AlbumTitle: a.title, ArtistName: a.artist, From: from, To: best}, note, nil -} - -type releaseCandidate struct { - release lidarr.AlbumRelease - coverage int -} - -// pickRelease chooses among releases that list each song once: the one -// covering most of the titles on disk, then the fewest tracks (the plainest -// edition that holds them), then a digital one, then the lowest id so the -// choice is stable. It refuses a release covering fewer titles on disk than -// the current one does: the change must not drop a song Lidarr now keeps. -func pickRelease(cands []releaseCandidate, currentCoverage int) (lidarr.AlbumRelease, bool) { - if len(cands) == 0 { - return lidarr.AlbumRelease{}, false - } - sort.SliceStable(cands, func(i, j int) bool { - a, b := cands[i], cands[j] - if a.coverage != b.coverage { - return a.coverage > b.coverage - } - if a.release.TrackCount != b.release.TrackCount { - return a.release.TrackCount < b.release.TrackCount - } - if da, db := isDigital(a.release), isDigital(b.release); da != db { - return da - } - return a.release.ID < b.release.ID - }) - if cands[0].coverage < currentCoverage { - return lidarr.AlbumRelease{}, false - } - return cands[0].release, true -} - -func isDigital(r lidarr.AlbumRelease) bool { - return strings.Contains(strings.ToLower(r.Format), "digital") -} - -// releaseRepeats reports whether a release lists one title more than once. -func releaseRepeats(tracks []lidarr.ReleaseTrack) bool { - seen := map[string]bool{} - for _, t := range tracks { - k := ReleaseTitleKey(t.Title) - if k == "" { - continue - } - if seen[k] { - return true - } - seen[k] = true - } - return false -} - -// coverage counts the distinct titles on disk that the release lists. -func coverage(tracks []lidarr.ReleaseTrack, onDisk map[string]bool) int { - hit := map[string]bool{} - for _, t := range tracks { - if k := ReleaseTitleKey(t.Title); onDisk[k] { - hit[k] = true - } - } - return len(hit) -} - -func releaseLabel(r lidarr.AlbumRelease) string { - label := r.Format - if label == "" { - label = r.Title - } - if r.Disambiguation != "" { - label = r.Disambiguation + ", " + label - } - return fmt.Sprintf("%s (%d tracks)", label, r.TrackCount) -} - -func setGroupNotes(ctx context.Context, q *dbq.Queries, groups []*resolveGroup, note string) error { - p := dbq.SetDuplicateGroupClassesParams{} - for _, g := range groups { - g.note = note - p.GroupIds = append(p.GroupIds, g.id) - p.Classes = append(p.Classes, string(g.class)) - p.Notes = append(p.Notes, note) - } - if err := q.SetDuplicateGroupClasses(ctx, p); err != nil { - return fmt.Errorf("write duplicate notes: %w", err) - } - return nil -} - -func resolvedDetail(merged int, changes []ReleaseChange) string { - var parts []string - if merged > 0 { - parts = append(parts, fmt.Sprintf("Merged %d %s Lidarr does not need.", merged, pluralWord(merged, "copy", "copies"))) - } - switch len(changes) { - case 0: - case 1: - c := changes[0] - parts = append(parts, fmt.Sprintf("Lidarr now monitors the %s release of %s by %s, which lists each song once.", - releaseLabel(c.To), c.AlbumTitle, c.ArtistName)) - default: - parts = append(parts, fmt.Sprintf("Lidarr now monitors a release that lists each song once for %d albums.", len(changes))) - } - return strings.Join(parts, " ") -} - -func pluralWord(n int, one, many string) string { - if n == 1 { - return one - } - return many -} - // DuplicateResolveWorker runs a resolver pass every hour. type DuplicateResolveWorker struct { pool *pgxpool.Pool logger *slog.Logger dataDir string settings *FingerprintSettingsService - lidarr func() LidarrLibrary + src ResolveSources tick time.Duration } -// NewDuplicateResolveWorker builds a worker with the production cadence. -// lidarrFn is asked each pass, so a Lidarr setting saved in admin takes effect -// without a restart; it returns nil while Lidarr is disabled. +// NewDuplicateResolveWorker builds a worker with the production cadence. src +// is what a pass may ask beyond the library. func NewDuplicateResolveWorker( - pool *pgxpool.Pool, logger *slog.Logger, dataDir string, - settings *FingerprintSettingsService, lidarrFn func() LidarrLibrary, + pool *pgxpool.Pool, logger *slog.Logger, dataDir string, settings *FingerprintSettingsService, src ResolveSources, ) *DuplicateResolveWorker { return &DuplicateResolveWorker{ - pool: pool, logger: logger, dataDir: dataDir, settings: settings, lidarr: lidarrFn, tick: duplicateResolveTick, + pool: pool, logger: logger, dataDir: dataDir, settings: settings, src: src, tick: duplicateResolveTick, } } -// Run blocks until ctx is cancelled, running a pass at start and then each tick. +// Run blocks until ctx is cancelled, running a pass at start and then each +// tick. A pass put off for a running scan is tried again after scanWaitRetry. func (w *DuplicateResolveWorker) Run(ctx context.Context) { - w.tickOnce(ctx) t := time.NewTicker(w.tick) defer t.Stop() + soon := time.NewTimer(0) + defer soon.Stop() for { select { case <-ctx.Done(): return case <-t.C: - w.tickOnce(ctx) + case <-soon.C: + } + if w.tickOnce(ctx) { + soon.Reset(scanWaitRetry) } } } -// tickOnce contains one pass so nothing it does can stop the next tick (rule 157). -func (w *DuplicateResolveWorker) tickOnce(ctx context.Context) { +// tickOnce contains one pass so nothing it does can stop the next tick (rule +// 157). It reports whether the pass was put off for a running scan. +func (w *DuplicateResolveWorker) tickOnce(ctx context.Context) (waitForScan bool) { defer func() { if r := recover(); r != nil { w.logger.Error("duplicate resolve: tick panicked", "panic", r) } }() + q := dbq.New(w.pool) // A sweep in flight is rewriting the groups; the next tick sees its result. - _, err := dbq.New(w.pool).GetInFlightDuplicateSweep(ctx) + _, err := q.GetInFlightDuplicateSweep(ctx) switch { case err == nil: - return + return false case !errors.Is(err, pgx.ErrNoRows): if ctx.Err() == nil { w.logger.Warn("duplicate resolve: sweep check failed", "err", err) } - return + return false } - var lid LidarrLibrary - if w.lidarr != nil { - lid = w.lidarr() + // A scan in flight has listed the files it will reconcile: a copy removed + // now is marked present again when it does. A run past StuckScanThreshold + // is a dead one. + run, err := q.GetInFlightScanRun(ctx) + switch { + case err == nil && time.Since(run.StartedAt.Time) <= StuckScanThreshold: + return true + case err != nil && !errors.Is(err, pgx.ErrNoRows): + if ctx.Err() == nil { + w.logger.Warn("duplicate resolve: scan check failed", "err", err) + } + return false } - res, err := ResolveDuplicates(ctx, w.pool, w.logger, w.dataDir, lid, w.settings.Get().AutoResolve) + res, err := ResolveDuplicatesWith(ctx, w.pool, w.logger, w.dataDir, w.settings.Get().AutoResolve, w.src) if err != nil && ctx.Err() == nil { w.logger.Warn("duplicate resolve: pass failed", "err", err) } w.logger.Info("duplicate resolve complete", - "groups", res.Groups, "lidarr", res.LidarrConsulted, "merged", res.Merged, - "merge_failed", res.MergeFailed, "release_changes", len(res.ReleaseChanges), "songs_linked", res.SongsLinked, + "groups", res.Groups, "merged", res.Merged, "merge_failed", res.MergeFailed, + "songs_linked", res.SongsLinked, "dismissed", res.Dismissed, "wrong_files", res.WrongFiles, "held_back", res.HeldBack, "released", res.Released) + return false } diff --git a/internal/library/duplicate_resolve_test.go b/internal/library/duplicate_resolve_test.go index d1b67a83..5cd41a74 100644 --- a/internal/library/duplicate_resolve_test.go +++ b/internal/library/duplicate_resolve_test.go @@ -2,7 +2,8 @@ package library import ( "context" - "errors" + "io" + "log/slog" "os" "path/filepath" "testing" @@ -11,119 +12,9 @@ import ( "github.com/jackc/pgx/v5/pgxpool" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) -func TestLidarrPathKey(t *testing.T) { - // Minstrel and Lidarr see the library under different mounts; the key is - // what they agree on. - a := LidarrPathKey("/music/Gorillaz/Humanz (2017)/Gorillaz - Humanz - 01 - Ascension.mp3") - b := LidarrPathKey("/data/media/music/Gorillaz/Humanz (2017)/Gorillaz - Humanz - 01 - Ascension.mp3") - if a != b || a != "Gorillaz/Humanz (2017)/Gorillaz - Humanz - 01 - Ascension.mp3" { - t.Errorf("keys differ or are wrong: %q, %q", a, b) - } - if LidarrPathKey("/music/Gorillaz/Humanz (2017)/a.mp3") == LidarrPathKey("/music/Gorillaz/Demon Days (2005)/a.mp3") { - t.Error("two albums' files share a key") - } -} - -func rt(titles ...string) []lidarr.ReleaseTrack { - out := make([]lidarr.ReleaseTrack, len(titles)) - for i, title := range titles { - out[i] = lidarr.ReleaseTrack{ID: i + 1, Title: title} - } - return out -} - -func TestReleaseRepeats(t *testing.T) { - if !releaseRepeats(rt("Ascension", "Strobelite", "Ascension")) { - t.Error("the box set lists Ascension twice, but no repeat was found") - } - if releaseRepeats(rt("Charger", "Charger (demo)")) { - t.Error("a demo was read as a repeat of the song") - } -} - -// The Humanz releases as Lidarr lists them (2026-10-09): the box set is -// monitored, and the 40-track digital release is the one that lists each song -// once and covers what is on disk. -func TestPickRelease(t *testing.T) { - digital40 := lidarr.AlbumRelease{ID: 10149, Format: "Digital Media", TrackCount: 40} - deluxe26 := lidarr.AlbumRelease{ID: 10150, Format: "CD", TrackCount: 26} - standard20 := lidarr.AlbumRelease{ID: 10151, Format: "Digital Media", TrackCount: 20} - - got, ok := pickRelease([]releaseCandidate{ - {release: standard20, coverage: 20}, {release: deluxe26, coverage: 26}, {release: digital40, coverage: 34}, - }, 34) - if !ok || got.ID != digital40.ID { - t.Errorf("picked %+v (ok=%v), want the 40-track digital release", got, ok) - } - - // Equal coverage: the plainer edition, then digital. - cd := lidarr.AlbumRelease{ID: 1, Format: "CD", TrackCount: 26} - web := lidarr.AlbumRelease{ID: 2, Format: "Digital Media", TrackCount: 26} - if got, _ := pickRelease([]releaseCandidate{{release: cd, coverage: 26}, {release: web, coverage: 26}}, 26); got.ID != web.ID { - t.Errorf("picked %+v, want the digital one of two equal releases", got) - } - - // Never a release that keeps fewer of the songs on disk than the current one. - if got, ok := pickRelease([]releaseCandidate{{release: deluxe26, coverage: 26}}, 34); ok { - t.Errorf("picked %+v, which covers fewer songs on disk than the current release", got) - } -} - -// fakeLidarr is one Lidarr album with several releases. Switching the -// monitored release changes what ListAlbumTracks answers, as in Lidarr. -type fakeLidarr struct { - unmapped []string - unmappedErr error - album lidarr.LidarrAlbum - releases []lidarr.AlbumRelease - releaseTracks map[int][]lidarr.ReleaseTrack - setCalls [][2]int -} - -func (f *fakeLidarr) ListUnmappedTrackFiles(context.Context) ([]lidarr.TrackFile, error) { - out := make([]lidarr.TrackFile, len(f.unmapped)) - for i, p := range f.unmapped { - out[i] = lidarr.TrackFile{ID: i + 1, Path: p} - } - return out, f.unmappedErr -} - -func (f *fakeLidarr) LookupAlbumByMBID(_ context.Context, mbid string) (lidarr.LidarrAlbum, error) { - if f.album.ForeignAlbumID != mbid { - return lidarr.LidarrAlbum{}, lidarr.ErrNotFound - } - return f.album, nil -} - -func (f *fakeLidarr) ListAlbumTracks(context.Context, int) ([]lidarr.ReleaseTrack, error) { - for _, r := range f.releases { - if r.Monitored { - return f.releaseTracks[r.ID], nil - } - } - return nil, nil -} - -func (f *fakeLidarr) GetAlbumReleases(context.Context, int) ([]lidarr.AlbumRelease, error) { - return append([]lidarr.AlbumRelease(nil), f.releases...), nil -} - -func (f *fakeLidarr) ListReleaseTracks(_ context.Context, releaseID int) ([]lidarr.ReleaseTrack, error) { - return f.releaseTracks[releaseID], nil -} - -func (f *fakeLidarr) SetMonitoredRelease(_ context.Context, albumID, releaseID int) error { - f.setCalls = append(f.setCalls, [2]int{albumID, releaseID}) - for i := range f.releases { - f.releases[i].Monitored = f.releases[i].ID == releaseID - } - return nil -} - // resolveFixture is the Humanz shape: one album, one song twice, both copies on // disk under an artist/album folder, in one pending exact-tier group. type resolveFixture struct { @@ -168,7 +59,6 @@ func newResolveFixture(t *testing.T) resolveFixture { } } exec(`UPDATE tracks SET title = 'Saturnz Barz' WHERE id = $1`, f.clean.ID) - exec(`UPDATE albums SET release_group_mbid = 'rg-humanz' WHERE id = $1`, album.ID) if err := pool.QueryRow(ctx, `INSERT INTO duplicate_groups (member_key, tier) VALUES ('resolve-fixture', 'exact') RETURNING id`, ).Scan(&f.groupID); err != nil { @@ -178,9 +68,6 @@ func newResolveFixture(t *testing.T) resolveFixture { return f } -// lidarrPath is where Lidarr, mounted elsewhere, sees a fixture file. -func lidarrPath(p string) string { return "/music/" + LidarrPathKey(p) } - func (f resolveFixture) group(t *testing.T) (status string, class *string, note *string, auto bool) { t.Helper() if err := f.pool.QueryRow(context.Background(), @@ -193,18 +80,31 @@ func (f resolveFixture) group(t *testing.T) (status string, class *string, note func exists(p string) bool { _, err := os.Stat(p); return err == nil } -// Lidarr maps the clean copy and holds the rip unmapped: the rip goes, the -// clean copy stays, and the merge is recorded as the resolver's. -func TestResolveDuplicates_MergesTheUnmappedCopy_Integration(t *testing.T) { - f := newResolveFixture(t) - lid := &fakeLidarr{unmapped: []string{lidarrPath(f.ripPath)}} +// autoMergeAudit reads the resolver's audit row for a group: how many there +// are, and the reason the last one gave for its survivor. +func autoMergeAudit(t *testing.T, pool *pgxpool.Pool, groupID pgtype.UUID) (n int, reason string) { + t.Helper() + if err := pool.QueryRow(context.Background(), + `SELECT count(*), coalesce(max(metadata->>'survivor_reason'), '') FROM audit_log + WHERE action = 'duplicate_merge' AND actor_id IS NULL + AND metadata->>'group_id' = $1 AND (metadata->>'automatic')::boolean`, + syncpkg.FormatUUID(groupID)).Scan(&n, &reason); err != nil { + t.Fatal(err) + } + return n, reason +} - res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", lid, true) +// One album holding a song twice: the extra copy goes, the copy whose name and +// tags fit the album stays, and the merge is recorded as the resolver's: +// Minstrel is the only program writing the library. +func TestResolveDuplicates_MergesASameReleaseCopy_Integration(t *testing.T) { + f := newResolveFixture(t) + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) if err != nil { t.Fatalf("resolve: %v", err) } - if res.Merged != 1 || !res.LidarrConsulted { - t.Fatalf("result = %+v, want one merge with Lidarr consulted", res) + if res.Merged != 1 || res.Classes[ClassSameRelease] != 1 { + t.Fatalf("result = %+v, want the same-release group merged", res) } if exists(f.ripPath) || !exists(f.cleanPath) { t.Errorf("rip exists=%v clean exists=%v; want only the clean copy left", exists(f.ripPath), exists(f.cleanPath)) @@ -213,138 +113,24 @@ func TestResolveDuplicates_MergesTheUnmappedCopy_Integration(t *testing.T) { if status != "merged" || class == nil || *class != string(ClassSameRelease) || !auto { t.Errorf("group = (%s, %v, auto=%v), want merged same_release by the resolver", status, class, auto) } - var audits int - if err := f.pool.QueryRow(context.Background(), - `SELECT count(*) FROM audit_log WHERE action = 'duplicate_merge' AND actor_id IS NULL - AND metadata->>'group_id' = $1 AND (metadata->>'automatic')::boolean`, - syncpkg.FormatUUID(f.groupID)).Scan(&audits); err != nil { - t.Fatal(err) - } - if audits != 1 { - t.Errorf("automatic merge audit rows = %d, want 1", audits) + if n, _ := autoMergeAudit(t, f.pool, f.groupID); n != 1 { + t.Errorf("automatic merge audit rows = %d, want 1", n) } } -// Lidarr maps both copies, and its release lists the song twice: nothing is -// deleted, the release moves to the one listing each song once, and the next -// pass leaves it there (lesson #4183's fixed point). -func TestResolveDuplicates_ChangesARepeatingRelease_Integration(t *testing.T) { +// With auto-resolve off, the pass classifies and removes nothing. +func TestResolveDuplicates_ClassifiesOnlyWithoutPermission_Integration(t *testing.T) { f := newResolveFixture(t) - lid := &fakeLidarr{ - album: lidarr.LidarrAlbum{ID: 4723, ForeignAlbumID: "rg-humanz", Title: "Humanz"}, - releases: []lidarr.AlbumRelease{ - {ID: 10148, Format: "14x12\" Vinyl, Digital Media", TrackCount: 4, Monitored: true}, - {ID: 10149, Format: "Digital Media", TrackCount: 2}, - }, - releaseTracks: map[int][]lidarr.ReleaseTrack{ - 10148: rt("Saturnz Barz", "Ascension", "Saturnz Barz", "Ascension"), - 10149: rt("Saturnz Barz", "Ascension"), - }, - } - - res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", lid, true) - if err != nil { - t.Fatalf("resolve: %v", err) - } - if res.Merged != 0 || !exists(f.ripPath) || !exists(f.cleanPath) { - t.Fatalf("merged %d; a copy Lidarr maps was touched", res.Merged) - } - if len(lid.setCalls) != 1 || lid.setCalls[0] != [2]int{4723, 10149} { - t.Fatalf("release changes = %v, want album 4723 to release 10149", lid.setCalls) - } - status, _, note, _ := f.group(t) - if status != "pending" || note == nil || *note == "" { - t.Errorf("group = (%s, note %v), want pending with a note saying what changed", status, note) - } - - // The next pass: Lidarr has not rescanned yet, so both copies still read - // as tracked. The album is settling, so nothing changes. - if _, err := ResolveDuplicates(context.Background(), f.pool, nil, "", lid, true); err != nil { - t.Fatalf("second resolve: %v", err) - } - if len(lid.setCalls) != 1 { - t.Errorf("release changed again: %v", lid.setCalls) - } - // And once it has settled, the check itself finds nothing to do: the - // release it chose lists each song once. - change, _, err := changeRelease(context.Background(), dbq.New(f.pool), lid, - &repeatingAlbum{albumID: f.clean.AlbumID, releaseGroupMbid: "rg-humanz"}) - if err != nil || change != nil || len(lid.setCalls) != 1 { - t.Errorf("after the change, changeRelease = (%+v, %v) with %d calls; want no change", change, err, len(lid.setCalls)) - } -} - -// After a release change Lidarr unmaps every file of the album until it has -// rescanned. A pass in that window must not read "every copy unmapped" as -// licence to merge: the copy it would remove may be the one Lidarr maps next. -func TestResolveDuplicates_LeavesAnAlbumAloneWhileLidarrRescans_Integration(t *testing.T) { - f := newResolveFixture(t) - lid := &fakeLidarr{ - album: lidarr.LidarrAlbum{ID: 4723, ForeignAlbumID: "rg-humanz"}, - releases: []lidarr.AlbumRelease{ - {ID: 10148, TrackCount: 2, Monitored: true}, {ID: 10149, TrackCount: 1}, - }, - releaseTracks: map[int][]lidarr.ReleaseTrack{10148: rt("Saturnz Barz", "Saturnz Barz"), 10149: rt("Saturnz Barz")}, - } - if _, err := ResolveDuplicates(context.Background(), f.pool, nil, "", lid, true); err != nil { - t.Fatal(err) - } - lid.unmapped = []string{lidarrPath(f.cleanPath), lidarrPath(f.ripPath)} - res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", lid, true) + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", false) if err != nil { t.Fatal(err) } - if res.Merged != 0 || !exists(f.ripPath) || !exists(f.cleanPath) { - t.Errorf("merged %d during the rescan window", res.Merged) + if res.Merged != 0 || !exists(f.ripPath) { + t.Errorf("merged %d", res.Merged) } -} - -// Without Lidarr, or with auto-resolve off, the pass classifies and removes -// nothing. -func TestResolveDuplicates_ClassifiesOnlyWithoutLidarrOrPermission_Integration(t *testing.T) { - for name, run := range map[string]func(resolveFixture) (DuplicateResolveResult, error){ - "lidarr disabled": func(f resolveFixture) (DuplicateResolveResult, error) { - return ResolveDuplicates(context.Background(), f.pool, nil, "", nil, true) - }, - "lidarr unreachable": func(f resolveFixture) (DuplicateResolveResult, error) { - return ResolveDuplicates(context.Background(), f.pool, nil, "", &fakeLidarr{unmappedErr: errors.New("timeout")}, true) - }, - "auto-resolve off": func(f resolveFixture) (DuplicateResolveResult, error) { - lid := &fakeLidarr{unmapped: []string{lidarrPath(f.ripPath)}} - return ResolveDuplicates(context.Background(), f.pool, nil, "", lid, false) - }, - } { - t.Run(name, func(t *testing.T) { - f := newResolveFixture(t) - res, err := run(f) - if err != nil { - t.Fatal(err) - } - if res.Merged != 0 || !exists(f.ripPath) { - t.Errorf("merged %d", res.Merged) - } - status, class, _, _ := f.group(t) - if status != "pending" || class == nil || *class != string(ClassSameRelease) { - t.Errorf("group = (%s, %v), want pending and classified", status, class) - } - }) - } -} - -// The guard itself: a merge asked to remove a copy the check refuses changes -// nothing, file or row. -func TestMergeDuplicateGroupGuarded_RefusesATrackedCopy_Integration(t *testing.T) { - f := newResolveFixture(t) - refuse := func(p string) bool { return p != f.ripPath } - _, err := MergeDuplicateGroupGuarded(context.Background(), f.pool, nil, "", f.groupID, f.clean.ID, refuse) - if !errors.Is(err, ErrCopyTrackedByLidarr) { - t.Fatalf("err = %v, want ErrCopyTrackedByLidarr", err) - } - if !exists(f.ripPath) { - t.Error("the refused copy's file was removed") - } - if status, _, _, _ := f.group(t); status != "pending" { - t.Errorf("group status = %s, want pending", status) + status, class, _, _ := f.group(t) + if status != "pending" || class == nil || *class != string(ClassSameRelease) { + t.Errorf("group = (%s, %v), want pending and classified", status, class) } } @@ -406,12 +192,11 @@ func newCrossReleaseRipFixture(t *testing.T) crossReleaseRipFixture { return f } -// The rip fulfils nothing in Lidarr while the clean copy on the album does: -// the rip goes and the album copy stays (#5439, D-a rule 1). -func TestResolveDuplicates_MergesAnUnmappedRipAcrossReleases_Integration(t *testing.T) { +// A video rip on one release with the clean copy on another: the rip is no +// release's copy, so it goes and the album cut stays. +func TestResolveDuplicates_MergesARipAcrossReleases_Integration(t *testing.T) { f := newCrossReleaseRipFixture(t) - lid := &fakeLidarr{unmapped: []string{lidarrPath(f.ripPath)}} - res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", lid, true) + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) if err != nil { t.Fatal(err) } @@ -421,30 +206,515 @@ func TestResolveDuplicates_MergesAnUnmappedRipAcrossReleases_Integration(t *test if exists(f.ripPath) || !exists(f.cleanPath) { t.Errorf("rip exists %v, clean exists %v; want only the clean copy", exists(f.ripPath), exists(f.cleanPath)) } + if _, reason := autoMergeAudit(t, f.pool, f.groupID); reason != "the copy that is not a video rip" { + t.Errorf("survivor reason %q, want the rip rule", reason) + } } -// When Lidarr maps both copies, each fulfils its own release: nothing is -// removed, and the copies are linked as one song instead. -func TestResolveDuplicates_KeepsCopiesLidarrMapsOnTwoReleases_Integration(t *testing.T) { +// Copies on two releases that are both clean each belong to their release: +// nothing is removed, and they are linked as one song. A rip an admin +// marked fine counts as clean. +func TestResolveDuplicates_KeepsCleanCopiesOnTwoReleases_Integration(t *testing.T) { + for name, prepare := range map[string]func(f crossReleaseRipFixture) (string, []any){ + "renamed": func(f crossReleaseRipFixture) (string, []any) { + return `UPDATE tracks SET title = 'Feel Good Inc.', file_path = $2 WHERE id = $1`, + []any{f.rip.ID, filepath.Join(filepath.Dir(f.ripPath), "01 - Feel Good Inc.mp3")} + }, + "kept as fine": func(f crossReleaseRipFixture) (string, []any) { + return `UPDATE tracks SET source_verdict = 'fine' WHERE id = $1`, []any{f.rip.ID} + }, + } { + t.Run(name, func(t *testing.T) { + f := newCrossReleaseRipFixture(t) + sql, args := prepare(f) + if _, err := f.pool.Exec(context.Background(), sql, args...); err != nil { + t.Fatal(err) + } + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) + if err != nil { + t.Fatal(err) + } + if res.Merged != 0 || !exists(f.ripPath) { + t.Errorf("merged %d; want both copies kept", res.Merged) + } + if res.SongsLinked != 1 { + t.Errorf("linked %d, want the pair linked as one song", res.SongsLinked) + } + }) + } +} + +// Identical audio under two titles on two albums, neither a rip: a later +// release retitled the recording ("F.O.D." and "F.O.D. (live)"). Each copy +// belongs to its release, so nothing is removed and they count as one song. +func TestResolveDuplicates_LinksOneRecordingUnderTwoTitles_Integration(t *testing.T) { f := newCrossReleaseRipFixture(t) - res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", &fakeLidarr{}, true) + ctx := context.Background() + if _, err := f.pool.Exec(ctx, `UPDATE duplicate_groups SET tier = 'exact' WHERE id = $1`, f.groupID); err != nil { + t.Fatal(err) + } + if _, err := f.pool.Exec(ctx, `UPDATE tracks SET title = 'DARE', file_path = $2 WHERE id = $1`, + f.rip.ID, filepath.Join(filepath.Dir(f.ripPath), "01 - DARE.mp3")); err != nil { + t.Fatal(err) + } + res, err := ResolveDuplicates(ctx, f.pool, nil, "", true) if err != nil { t.Fatal(err) } - if res.Merged != 0 || !exists(f.ripPath) { - t.Errorf("merged %d; want both copies kept", res.Merged) - } - if res.SongsLinked != 1 { - t.Errorf("linked %d, want the pair linked as one song", res.SongsLinked) + if res.Classes[ClassCrossRelease] != 1 || res.Merged != 0 || res.SongsLinked != 1 { + t.Fatalf("classes %v, merged %d, linked %d; want the pair linked and kept", res.Classes, res.Merged, res.SongsLinked) } } -// A rip is held back from radio and the mixes; the operator's "fine" sticks +// oneAlbumFixture is one album holding two tracks in one pending group, each +// with a file on disk. +type oneAlbumFixture struct { + pool *pgxpool.Pool + album dbq.Album + a, b dbq.Track + aPath string + bPath string + groupID pgtype.UUID +} + +func newOneAlbumFixture(t *testing.T, tier, titleA, titleB string) oneAlbumFixture { + t.Helper() + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + dir := filepath.Join(t.TempDir(), "Savant", "Void") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + f := oneAlbumFixture{pool: pool, aPath: filepath.Join(dir, "15 - a.m4a"), bPath: filepath.Join(dir, "17 - b.m4a")} + for _, p := range []string{f.aPath, f.bPath} { + if err := os.WriteFile(p, []byte("audio"), 0o644); err != nil { + t.Fatal(err) + } + } + var artist dbq.Artist + f.a, f.album, artist = seedTrack(t, pool, f.aPath) + b, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: titleB, AlbumID: f.album.ID, ArtistID: artist.ID, + DurationMs: 220000, FilePath: f.bPath, FileSize: 100, FileFormat: "m4a", + }) + if err != nil { + t.Fatal(err) + } + f.b = b + if _, err := pool.Exec(ctx, `UPDATE tracks SET title = $2, duration_ms = 220000 WHERE id = $1`, f.a.ID, titleA); err != nil { + t.Fatal(err) + } + if err := pool.QueryRow(ctx, + `INSERT INTO duplicate_groups (member_key, tier) VALUES ($1, $2) RETURNING id`, t.Name(), tier, + ).Scan(&f.groupID); err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, `INSERT INTO duplicate_group_members (group_id, track_id) VALUES ($1, $2), ($1, $3)`, + f.groupID, f.a.ID, f.b.ID); err != nil { + t.Fatal(err) + } + return f +} + +func auditCount(t *testing.T, pool *pgxpool.Pool, action string, groupID pgtype.UUID) int { + t.Helper() + var n int + if err := pool.QueryRow(context.Background(), + `SELECT count(*) FROM audit_log WHERE action = $1 AND metadata->>'group_id' = $2`, + action, syncpkg.FormatUUID(groupID)).Scan(&n); err != nil { + t.Fatal(err) + } + return n +} + +// Two versions of a song on one album that sound alike are not duplicates: +// the group is dismissed and both files stay. +func TestResolveDuplicates_DismissesTwoVersionsOfASong_Integration(t *testing.T) { + f := newOneAlbumFixture(t, "acoustic", "Lovesick", "Lovesick (Instrumental)") + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) + if err != nil { + t.Fatal(err) + } + if res.Dismissed != 1 || res.Merged != 0 || !exists(f.aPath) || !exists(f.bPath) { + t.Fatalf("result %+v; want the group dismissed and both files kept", res) + } + if status, _, _, auto := (resolveFixture{pool: f.pool, groupID: f.groupID}).group(t); status != "dismissed" || !auto { + t.Errorf("group = (%s, auto=%v), want dismissed by the resolver", status, auto) + } + if n := auditCount(t, f.pool, "duplicate_dismiss", f.groupID); n != 1 { + t.Errorf("dismiss audit rows = %d, want 1", n) + } +} + +// seedTrackList holds a release group whose one release lists the songs with +// their lengths, and points the album at it. +func seedTrackList(t *testing.T, pool *pgxpool.Pool, album dbq.Album, songs map[string]int) { + t.Helper() + ctx := context.Background() + rg := "rg-" + syncpkg.FormatUUID(album.ID) + rel := "rel-" + syncpkg.FormatUUID(album.ID) + for _, stmt := range []struct { + sql string + args []any + }{ + {`INSERT INTO mb_release_groups (mbid, title) VALUES ($1, 'Void')`, []any{rg}}, + {`INSERT INTO mb_releases (mbid, release_group_mbid, title) VALUES ($1, $2, 'Void')`, []any{rel, rg}}, + {`UPDATE albums SET release_group_mbid = $2 WHERE id = $1`, []any{album.ID, rg}}, + } { + if _, err := pool.Exec(ctx, stmt.sql, stmt.args...); err != nil { + t.Fatalf("exec %q: %v", stmt.sql, err) + } + } + pos := 1 + for title, ms := range songs { + if _, err := pool.Exec(ctx, + `INSERT INTO mb_release_tracks (release_mbid, medium_position, position, title, length_ms) VALUES ($1, 1, $2, $3, $4)`, + rel, pos, title, ms); err != nil { + t.Fatal(err) + } + pos++ + } +} + +// One recording in two slots of an album: MusicBrainz says Void is as long as +// the file and Reach is a minute longer, so the copy in Reach's slot goes. Its +// track stays, marked missing, for the album to be fetched again. +func TestResolveDuplicates_RemovesACopyFromAnotherSongsSlot_Integration(t *testing.T) { + f := newOneAlbumFixture(t, "acoustic", "Void", "Reach") + seedTrackList(t, f.pool, f.album, map[string]int{"Void": 220500, "Reach": 281000}) + // The wrong file is Void's recording, so it carries Void's MBID. + if _, err := f.pool.Exec(context.Background(), + `UPDATE tracks SET mbid = 'rec-void', mbid_source = 'tag' WHERE id = ANY($1)`, []pgtype.UUID{f.a.ID, f.b.ID}); err != nil { + t.Fatal(err) + } + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) + if err != nil { + t.Fatal(err) + } + // Folded by that MBID into the copy kept, the slot would never be wanted + // back. + if pairs, err := PairMissingTracks(context.Background(), f.pool, nil); err != nil || pairs.Folded() != 0 { + t.Fatalf("missing-pair pass = %+v, %v; want the emptied slot left alone", pairs, err) + } + if res.WrongFiles != 1 || !exists(f.aPath) || exists(f.bPath) { + t.Fatalf("result %+v, void exists %v, reach exists %v; want only Reach's copy removed", + res, exists(f.aPath), exists(f.bPath)) + } + var missing bool + if err := f.pool.QueryRow(context.Background(), + `SELECT missing_since IS NOT NULL FROM tracks WHERE id = $1`, f.b.ID).Scan(&missing); err != nil { + t.Fatalf("Reach's track is gone: %v", err) + } + if !missing { + t.Error("Reach's track is not marked missing, so nothing will fetch it again") + } + if status, _, _, auto := (resolveFixture{pool: f.pool, groupID: f.groupID}).group(t); status != "merged" || !auto { + t.Errorf("group = (%s, auto=%v), want settled by the resolver", status, auto) + } + if n := auditCount(t, f.pool, "duplicate_wrong_file", f.groupID); n != 1 { + t.Errorf("wrong-file audit rows = %d, want 1", n) + } +} + +// A slot emptied of another song's file folds into a copy of its own song the +// album already holds (Savant's VOID DLC "Weeper" beside Void's). A copy under +// the title that is not as long as the song is another song: the slot waits for +// a copy fetched again, and folds into that. +func TestResolveDuplicates_EmptiedSlotFoldsOnlyIntoItsOwnSong_Integration(t *testing.T) { + for _, tc := range []struct { + name string + heldMs int32 + foldsFirst int + }{ + {"the album holds Reach", 281500, 1}, + {"the album holds a file titled Reach that is not", 220000, 0}, + } { + t.Run(tc.name, func(t *testing.T) { + f := newOneAlbumFixture(t, "acoustic", "Void", "Reach") + ctx := context.Background() + q := dbq.New(f.pool) + seedTrackList(t, f.pool, f.album, map[string]int{"Void": 220500, "Reach": 281000}) + held := func(name string, ms int32) { + t.Helper() + if _, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: "Reach", AlbumID: f.album.ID, ArtistID: f.a.ArtistID, + DurationMs: ms, FilePath: filepath.Join(filepath.Dir(f.aPath), name), FileSize: 100, FileFormat: "m4a", + }); err != nil { + t.Fatal(err) + } + } + held("17 - Reach (other edition).m4a", tc.heldMs) + res, err := ResolveDuplicates(ctx, f.pool, nil, "", true) + if err != nil || res.WrongFiles != 1 { + t.Fatalf("resolve = %+v, %v; want Reach's slot emptied", res, err) + } + pairs, err := PairMissingTracks(ctx, f.pool, nil) + if err != nil || pairs.Folded() != tc.foldsFirst { + t.Fatalf("missing-pair pass = %+v, %v; want %d folded", pairs, err, tc.foldsFirst) + } + if tc.foldsFirst == 1 { + return + } + held("17 - Reach (fetched again).m4a", 281200) + if pairs, err := PairMissingTracks(ctx, f.pool, nil); err != nil || pairs.Folded() != 1 { + t.Fatalf("missing-pair pass after the fetch = %+v, %v; want the slot folded into the copy fetched", pairs, err) + } + }) + } +} + +// One recording filling two slots whose songs are both clearly something else +// (Shnabubula's whole album in one file): both copies go, nothing is kept. +func TestResolveDuplicates_RemovesEveryCopyWhenNoSlotIsTheirs_Integration(t *testing.T) { + f := newOneAlbumFixture(t, "acoustic", "Void", "Reach") + seedTrackList(t, f.pool, f.album, map[string]int{"Void": 282000, "Reach": 421000}) + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) + if err != nil { + t.Fatal(err) + } + if res.WrongFiles != 2 || exists(f.aPath) || exists(f.bPath) { + t.Fatalf("result %+v, void exists %v, reach exists %v; want both copies removed", + res, exists(f.aPath), exists(f.bPath)) + } + if status, _, _, auto := (resolveFixture{pool: f.pool, groupID: f.groupID}).group(t); status != "merged" || !auto { + t.Errorf("group = (%s, auto=%v), want settled by the resolver", status, auto) + } + if n := auditCount(t, f.pool, "duplicate_wrong_file", f.groupID); n != 1 { + t.Errorf("wrong-file audit rows = %d, want 1", n) + } +} + +// seedTrimmedAlbum is newOneAlbumFixture for one recording cut to both its +// slots' lengths: identical audio, as Tormentor and as Record Shop, each as +// long as its own slot's song, and the track list naming each slot's +// recording. The files came from a download Minstrel imported. +func seedTrimmedAlbum(t *testing.T) oneAlbumFixture { + t.Helper() + f := newOneAlbumFixture(t, "exact", "Tormentor", "Record Shop") + ctx := context.Background() + seedTrackList(t, f.pool, f.album, map[string]int{"Tormentor": 221000, "Record Shop": 219000}) + for _, stmt := range []struct { + sql string + args []any + }{ + {`UPDATE mb_release_tracks SET recording_mbid = 'rec-' || lower(replace(title, ' ', '-'))`, nil}, + {`INSERT INTO grabs (info_hash, name, key, client, save_path, origin, import_state, destination) + VALUES ('abc123', 'The Wombats - Void [WEB]', 'k', 'qbit', '/dl', 'schedule', 'imported', $1)`, + []any{filepath.Dir(f.aPath)}}, + } { + if _, err := f.pool.Exec(ctx, stmt.sql, stmt.args...); err != nil { + t.Fatalf("exec %q: %v", stmt.sql, err) + } + } + return f +} + +func releaseBlocked(t *testing.T, pool *pgxpool.Pool, hash string) bool { + t.Helper() + var n int + if err := pool.QueryRow(context.Background(), + `SELECT count(*) FROM release_blocks WHERE info_hash = $1 AND reason = 'files'`, hash).Scan(&n); err != nil { + t.Fatal(err) + } + return n == 1 +} + +// AcoustID matched one copy to Tormentor's recording: that copy stays, Record +// Shop's goes, and the download they came from is set aside. +func TestResolveDuplicates_KeepsTheCopyAcoustIDNames_Integration(t *testing.T) { + f := seedTrimmedAlbum(t) + if _, err := f.pool.Exec(context.Background(), + `INSERT INTO track_acoustid_lookups (track_id, state, candidates, recording_mbid) VALUES ($1, 'matched', 1, 'rec-tormentor')`, + f.a.ID); err != nil { + t.Fatal(err) + } + asked := 0 + res, err := ResolveDuplicatesWith(context.Background(), f.pool, nil, "", true, ResolveSources{ + Identify: func(context.Context, string) ([]string, error) { asked++; return nil, nil }, + }) + if err != nil { + t.Fatal(err) + } + if asked != 0 { + t.Errorf("AcoustID asked %d times; the copy's own lookup already answered", asked) + } + if res.WrongFiles != 1 || !exists(f.aPath) || exists(f.bPath) { + t.Fatalf("result %+v, tormentor exists %v, record shop exists %v; want only Record Shop's copy removed", + res, exists(f.aPath), exists(f.bPath)) + } + if !releaseBlocked(t, f.pool, "abc123") { + t.Error("the download the files came from was not set aside, so the album would be fetched from it again") + } +} + +// Copies whose own lookups all name Record Shop's recording are that +// recording, though their fingerprints differ a little and one is six seconds +// from Tormentor's length, too near for the lengths to decide. With no +// lookups, copies that far apart are not asked about. +func TestResolveDuplicates_AgreeingLookupsDecideAnAcousticGroup_Integration(t *testing.T) { + setup := func(t *testing.T) oneAlbumFixture { + f := seedTrimmedAlbum(t) + for _, stmt := range []string{ + `UPDATE duplicate_groups SET tier = 'acoustic', worst_bit_error_rate = 0.04`, + `UPDATE mb_release_tracks SET length_ms = 226000 WHERE title = 'Tormentor'`, + } { + if _, err := f.pool.Exec(context.Background(), stmt); err != nil { + t.Fatalf("exec %q: %v", stmt, err) + } + } + return f + } + t.Run("agreeing", func(t *testing.T) { + f := setup(t) + if _, err := f.pool.Exec(context.Background(), + `INSERT INTO track_acoustid_lookups (track_id, state, candidates, recording_mbid) + VALUES ($1, 'matched', 1, 'rec-record-shop'), ($2, 'matched', 1, 'rec-record-shop')`, + f.a.ID, f.b.ID); err != nil { + t.Fatal(err) + } + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) + if err != nil { + t.Fatal(err) + } + if res.WrongFiles != 1 || exists(f.aPath) || !exists(f.bPath) { + t.Fatalf("result %+v, tormentor exists %v, record shop exists %v; want only Tormentor's copy removed", + res, exists(f.aPath), exists(f.bPath)) + } + }) + t.Run("unknown", func(t *testing.T) { + f := setup(t) + asked := 0 + res, err := ResolveDuplicatesWith(context.Background(), f.pool, nil, "", true, ResolveSources{ + Identify: func(context.Context, string) ([]string, error) { asked++; return nil, nil }, + }) + if err != nil { + t.Fatal(err) + } + if asked != 0 || res.WrongFiles != 0 || !exists(f.aPath) || !exists(f.bPath) { + t.Fatalf("asked %d, result %+v; want nothing asked or removed for copies that may be two takes", asked, res) + } + }) +} + +// A pass while a scan is walking the library waits for it: a copy removed +// under the walk is marked present again when the scan reconciles. A run +// stuck past StuckScanThreshold does not hold the resolver up. +func TestDuplicateResolveWorker_WaitsForARunningScan_Integration(t *testing.T) { + f := seedTrimmedAlbum(t) + ctx := context.Background() + w := NewDuplicateResolveWorker(f.pool, slog.New(slog.NewTextHandler(io.Discard, nil)), "", nil, ResolveSources{}) + if _, err := f.pool.Exec(ctx, `INSERT INTO scan_runs DEFAULT VALUES`); err != nil { + t.Fatal(err) + } + if !w.tickOnce(ctx) { + t.Error("a pass ran under a running scan") + } + if !exists(f.aPath) || !exists(f.bPath) { + t.Error("a copy was removed while the scan was walking the library") + } + if _, err := f.pool.Exec(ctx, `UPDATE scan_runs SET started_at = now() - interval '2 hours'`); err != nil { + t.Fatal(err) + } + if w.tickOnce(ctx) { + t.Error("a scan stuck for two hours held the resolver up") + } +} + +// AcoustID knows the recording as no song of the album: every copy goes. With +// AcoustID unreachable, nothing is decided and the group waits. +func TestResolveDuplicates_TrimmedCopiesAcoustIDCannotName_Integration(t *testing.T) { + t.Run("no match", func(t *testing.T) { + f := seedTrimmedAlbum(t) + asked := 0 + res, err := ResolveDuplicatesWith(context.Background(), f.pool, nil, "", true, ResolveSources{ + Identify: func(context.Context, string) ([]string, error) { + asked++ + return []string{"rec-elsewhere"}, nil + }, + }) + if err != nil { + t.Fatal(err) + } + if asked != 1 || res.WrongFiles != 2 || exists(f.aPath) || exists(f.bPath) { + t.Fatalf("asked %d, result %+v; want one lookup and both copies removed", asked, res) + } + if status, _, _, _ := (resolveFixture{pool: f.pool, groupID: f.groupID}).group(t); status != "merged" { + t.Errorf("group = %s, want settled", status) + } + }) + t.Run("unreachable", func(t *testing.T) { + f := seedTrimmedAlbum(t) + res, err := ResolveDuplicatesWith(context.Background(), f.pool, nil, "", true, ResolveSources{ + Identify: func(context.Context, string) ([]string, error) { return nil, ErrIdentifyUnavailable }, + }) + if err != nil { + t.Fatal(err) + } + if res.WrongFiles != 0 || !exists(f.aPath) || !exists(f.bPath) || releaseBlocked(t, f.pool, "abc123") { + t.Fatalf("result %+v; want nothing removed or set aside while AcoustID cannot answer", res) + } + if status, _, _, _ := (resolveFixture{pool: f.pool, groupID: f.groupID}).group(t); status != "pending" { + t.Errorf("group = %s, want pending", status) + } + }) +} + +// An album whose track list the catalogue never read: the resolver asks for +// it, once, and decides with what comes back. +func TestResolveDuplicates_FetchesAMissingTrackList_Integration(t *testing.T) { + f := newOneAlbumFixture(t, "acoustic", "Void", "Reach") + rg := "rg-" + syncpkg.FormatUUID(f.album.ID) + if _, err := f.pool.Exec(context.Background(), + `UPDATE albums SET release_group_mbid = $2 WHERE id = $1`, f.album.ID, rg); err != nil { + t.Fatal(err) + } + calls := 0 + fetch := func(_ context.Context, asked string) error { + calls++ + if asked != rg { + t.Errorf("asked for %s, want %s", asked, rg) + } + seedTrackList(t, f.pool, f.album, map[string]int{"Void": 220500, "Reach": 281000}) + return nil + } + res, err := ResolveDuplicatesWith(context.Background(), f.pool, nil, "", true, ResolveSources{TrackLists: fetch}) + if err != nil { + t.Fatal(err) + } + if calls != 1 || res.WrongFiles != 1 || exists(f.bPath) { + t.Fatalf("fetches %d, result %+v; want one fetch and Reach's copy removed", calls, res) + } +} + +// Without a track list that decides, nothing is removed. +func TestResolveDuplicates_LeavesAnUndecidedSlotForTheOperator_Integration(t *testing.T) { + f := newOneAlbumFixture(t, "acoustic", "Void", "Reach") + seedTrackList(t, f.pool, f.album, map[string]int{"Void": 220500, "Reach": 221000}) + res, err := ResolveDuplicates(context.Background(), f.pool, nil, "", true) + if err != nil { + t.Fatal(err) + } + if res.WrongFiles != 0 || !exists(f.aPath) || !exists(f.bPath) { + t.Fatalf("result %+v; want both copies kept when the lengths cannot tell", res) + } + if status, class, _, _ := (resolveFixture{pool: f.pool, groupID: f.groupID}).group(t); status != "pending" || + class == nil || *class != string(ClassMismatch) { + t.Errorf("group = (%s, %v), want a pending mismatch", status, class) + } +} + +// A rip is held back from radio and the mixes; an admin's "fine" sticks // through later passes; a renamed file is released. func TestResolveDuplicates_HoldsBackRips_Integration(t *testing.T) { f := newCrossReleaseRipFixture(t) ctx := context.Background() q := dbq.New(f.pool) + // Without its group the rip has no clean copy to merge into, so it stays + // on disk and only its verdict changes. + if _, err := f.pool.Exec(ctx, `DELETE FROM duplicate_groups WHERE id = $1`, f.groupID); err != nil { + t.Fatal(err) + } verdict := func(id pgtype.UUID) *string { t.Helper() var v *string @@ -455,7 +725,7 @@ func TestResolveDuplicates_HoldsBackRips_Integration(t *testing.T) { } resolve := func() DuplicateResolveResult { t.Helper() - res, err := ResolveDuplicates(ctx, f.pool, nil, "", &fakeLidarr{}, true) + res, err := ResolveDuplicates(ctx, f.pool, nil, "", true) if err != nil { t.Fatal(err) } diff --git a/internal/library/duplicate_settle.go b/internal/library/duplicate_settle.go new file mode 100644 index 00000000..d44b0c4f --- /dev/null +++ b/internal/library/duplicate_settle.go @@ -0,0 +1,618 @@ +package library + +import ( + "context" + "errors" + "fmt" + "log/slog" + "slices" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/audit" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" + "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" + syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" +) + +// Duplicates that settle without a merge, so they do not wait on an admin. +// Retitles across releases are linked like any cross-release group. What is +// left on one album is three shapes: +// +// - distinct: two versions of a song that sound alike ("Lovesick" and +// "Lovesick (Instrumental)"). Not duplicates, so the group is dismissed. +// - mismatch: one file filling several track slots (one recording as +// three of an album's tracks). The album's MusicBrainz track list +// says how long each slot's song is. When exactly one slot's song is as +// long as the file and every other slot's song is clearly not, the copies +// in the other slots are removed and their tracks marked missing, so the +// reacquisition sweeper wants the album back and a good copy fills them. +// When every slot's song is clearly not the file, every copy goes. +// - named: the lengths cannot say (one recording filed as many tracks, each +// cut to its own slot's length), but AcoustID +// can. When every copy's lookup names one recording, or the copies are +// the same audio and a lookup names any, the track list says whose slot +// that recording is: that copy is kept and the rest go, or every copy +// when it is in none of their slots. Same audio that AcoustID does not +// know at all is no copy's song, and every copy goes. +// +// Whenever copies go, the release the files came from is set aside, so the +// album fetched again is not the same files. +// +// Anything less certain stays with an admin, who settles it the same way +// with their own answer (SettleWrongSlot). + +const ( + // trackListFetchesPerPass is how many albums' track lists a pass may ask + // MusicBrainz for, when the catalogue has never read them. The resolver + // runs hourly, so a backlog drains over a few hours without a burst. + trackListFetchesPerPass = 5 + // slotFit is how close a file must be to a slot's MusicBrainz length to be + // that slot's song, and slotOther how far away every length of a slot's + // title must be for the file to be another song's. + slotFit = 3000 + slotOther = 10000 + // identifiesPerPass is how many groups a pass may ask AcoustID about. The + // client keeps to AcoustID's rate; this keeps a pass short. + identifiesPerPass = 5 + // sameAudioBER is the worst bit error rate at which an acoustic group's + // copies are one recording cut differently rather than takes that sound + // alike: the groups measured as one recording trimmed to several slots + // read 0.000, and two takes of one song read far above this. + sameAudioBER = 0.02 +) + +// slotCopy is what deciding a wrong slot needs to know about one copy. +type slotCopy struct { + Title, Artist string + DurationMs int32 +} + +// wrongSlots decides which copies of a one-album mismatch sit in a slot that +// is not theirs. keep is the copy whose slot fits the recording, or -1 when +// every copy is clearly another song's (Shnabubula's whole album, 45 minutes +// in one file, filed as both "Fading Light" and "Looking for Answers"): then +// all of them go. ok is false when the track list cannot say, and nothing +// should be removed. +// +// Several copies may fit when they carry the same title ("Killjoy" twice +// beside "Dreamland"): they are one song held twice, not a wrong slot, so they +// are neither kept nor lost here. Once the wrong slot is emptied, the next +// sweep finds the pair on its own, and the resolver merges it as any song held +// twice on one album. +func wrongSlots(copies []slotCopy, tracks []dbq.MbReleaseTrack) (keep int, losers []int, ok bool) { + keep = -1 + keepKey := "" + for i, c := range copies { + key := MatchTitleKey(c.Title, c.Artist) + fits, other, known := false, true, false + for _, t := range tracks { + if t.LengthMs <= 0 || MatchTitleKey(t.Title, c.Artist) != key { + continue + } + known = true + d := absMs(t.LengthMs - c.DurationMs) + if d <= slotFit { + fits = true + } + if d <= slotOther { + other = false + } + } + switch { + case fits && keep >= 0: + if key != keepKey { + return -1, nil, false + } + case fits: + keep, keepKey = i, key + case known && other: + losers = append(losers, i) + default: + return -1, nil, false + } + } + // A copy that neither fits nor is clearly another song's has returned + // above, so with no keep every copy is a loser. + if len(losers) == 0 { + return -1, nil, false + } + return keep, losers, true +} + +func absMs(d int32) int32 { + if d < 0 { + return -d + } + return d +} + +// dismissDistinct dismisses every group of distinct versions and records it +// for the Resolved automatically tab. It adds what it settled to settled. +func dismissDistinct( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, groups []*resolveGroup, settled map[pgtype.UUID]bool, +) int { + q := dbq.New(pool) + n := 0 + for _, g := range groups { + if g.class != ClassDistinct || settled[g.id] || ctx.Err() != nil { + continue + } + rows, err := q.SettleDuplicateGroupAutomatically(ctx, dbq.SettleDuplicateGroupAutomaticallyParams{ + ID: g.id, Status: "dismissed", + }) + if err != nil { + logger.Warn("duplicate resolve: dismiss failed", "group_id", syncpkg.FormatUUID(g.id), "err", err) + continue + } + if rows == 0 { + continue + } + settled[g.id] = true + n++ + titles := make([]string, len(g.members)) + for i, m := range g.members { + titles[i] = m.Title + } + audit.WriteOrLog(ctx, pool, logger, pgtype.UUID{}, pgtype.UUID{}, audit.ActionDuplicateDismiss, map[string]any{ + "automatic": true, + "group_id": syncpkg.FormatUUID(g.id), + "artist": g.members[0].ArtistName, + "titles": titles, + }) + } + return n +} + +// settleWrongSlots removes the copies that sit in another song's slot, for +// every one-album mismatch the track list, or AcoustID with it, decides. It +// returns how many copies went and the songs whose slots were emptied, and +// adds the groups it settled to settled. +func settleWrongSlots( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, groups []*resolveGroup, settled map[pgtype.UUID]bool, + src ResolveSources, +) (int, []string) { + q := dbq.New(pool) + tracksByGroup := map[string][]dbq.MbReleaseTrack{} + fetches, identifies := trackListFetchesPerPass, identifiesPerPass + if src.TrackLists == nil { + fetches = 0 + } + if src.Identify == nil { + identifies = 0 + } + removed := 0 + var names []string + for _, g := range groups { + if g.class != ClassMismatch || settled[g.id] || ctx.Err() != nil { + continue + } + rg := g.members[0].ReleaseGroupMbid + if rg == "" { + continue + } + tracks, seen := tracksByGroup[rg] + if !seen { + var err error + if tracks, err = q.ListMBReleaseTracksForGroup(ctx, rg); err != nil { + logger.Warn("duplicate resolve: reading the album's track list failed", "release_group", rg, "err", err) + continue + } + if len(tracks) == 0 && fetches > 0 && neverRead(ctx, q, rg) { + fetches-- + if err := src.TrackLists(ctx, rg); err != nil { + logger.Info("duplicate resolve: fetching the album's track list failed", "release_group", rg, "err", err) + if errors.Is(err, musicbrainz.ErrUnreachable) { + fetches = 0 + } + } else if tracks, err = q.ListMBReleaseTracksForGroup(ctx, rg); err != nil { + logger.Warn("duplicate resolve: reading the album's track list failed", "release_group", rg, "err", err) + continue + } + } + tracksByGroup[rg] = tracks + } + copies := make([]slotCopy, len(g.members)) + for i, m := range g.members { + copies[i] = slotCopy{Title: m.Title, Artist: m.ArtistName, DurationMs: m.DurationMs} + } + keep, losers, ok := wrongSlots(copies, tracks) + if !ok { + recordings, known := groupRecordings(ctx, logger, g, src.Identify, &identifies) + if !known { + continue + } + if keep, losers, ok = namedCopies(copies, tracks, recordings); !ok { + logger.Info("duplicate resolve: AcoustID's answer does not place the recording in one slot; left for review", + "group_id", syncpkg.FormatUUID(g.id), "recordings", recordings) + } + } + if !ok { + continue + } + // Set aside before the files go: a pass that dies between the two + // still does not fetch the same files again. + blocked := blockReleasesHolding(ctx, pool, logger, g) + emptied := emptiedSlots(ctx, q, logger, g, losers, tracks) + n, err := removeWrongSlots(ctx, pool, logger, g, keep, losers, emptied, blocked, pgtype.UUID{}) + removed += n + if err != nil { + logger.Warn("duplicate resolve: removing wrong-slot copies failed", "group_id", syncpkg.FormatUUID(g.id), "err", err) + continue + } + settled[g.id] = true + for _, i := range losers { + names = append(names, TrackLabel(g.members[i].ArtistName, g.members[i].Title)) + } + } + return removed, names +} + +// ErrNotWrongSlot means a group is not one recording filling several of an +// album's track slots, so there is no slot to empty and nothing to fetch again. +var ErrNotWrongSlot = errors.New("library: duplicate group is not one recording in several slots") + +// WrongSlotResult is what settling a wrong slot by hand did. Kept is nil when +// every copy went. +type WrongSlotResult struct { + Kept *MergedCopy + Removed []MergedCopy + // Blocked names the releases set aside. None when the album was imported + // before Minstrel downloaded anything: the same files may come back. + Blocked []string +} + +// SettleWrongSlot settles a one-album mismatch the resolver left for the +// admin, with their answer in place of MusicBrainz's lengths and +// AcoustID's. keepID is the copy whose title they say the recording is; an +// invalid id means they cannot say, and every copy goes. Either way it is the +// resolver's own path: the release the files came from is set aside, the +// copies removed are marked missing, and the album is wanted again for them. +// A merge does none of that, and keeps one copy's title whether or not it is +// the audio's. +// +// Copies sharing the kept copy's title are one song held twice and stay, as in +// wrongSlots: the next sweep finds the pair and the resolver merges it. +func SettleWrongSlot( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, groupID, keepID, actorID pgtype.UUID, +) (WrongSlotResult, error) { + if logger == nil { + logger = slog.Default() + } + q := dbq.New(pool) + rows, err := q.ListDuplicateGroupsForResolve(ctx, groupID) + if err != nil { + return WrongSlotResult{}, fmt.Errorf("read duplicate group: %w", err) + } + groups := foldResolveGroups(rows) + if len(groups) == 0 { + return WrongSlotResult{}, ErrDuplicateGroupNotPending + } + g := groups[0] + if g.classify(); g.class != ClassMismatch { + return WrongSlotResult{}, ErrNotWrongSlot + } + keep := -1 + if keepID.Valid { + keep = slices.IndexFunc(g.members, func(m dbq.ListDuplicateGroupsForResolveRow) bool { return m.TrackID == keepID }) + if keep < 0 { + return WrongSlotResult{}, ErrSurvivorNotInGroup + } + } + var losers []int + for i, m := range g.members { + if keep >= 0 && MatchTitleKey(m.Title, m.ArtistName) == MatchTitleKey(g.members[keep].Title, m.ArtistName) { + continue + } + losers = append(losers, i) + } + if len(losers) == 0 { + return WrongSlotResult{}, ErrNotWrongSlot + } + var tracks []dbq.MbReleaseTrack + if rg := g.members[0].ReleaseGroupMbid; rg != "" { + if tracks, err = q.ListMBReleaseTracksForGroup(ctx, rg); err != nil { + return WrongSlotResult{}, fmt.Errorf("read the album's track list: %w", err) + } + } + + res := WrongSlotResult{Blocked: blockReleasesHolding(ctx, pool, logger, g)} + emptied := emptiedSlots(ctx, q, logger, g, losers, tracks) + n, err := removeWrongSlots(ctx, pool, logger, g, keep, losers, emptied, res.Blocked, actorID) + if keep >= 0 { + res.Kept = &MergedCopy{TrackID: g.members[keep].TrackID, FilePath: g.members[keep].FilePath} + } + for _, i := range losers[:n] { + res.Removed = append(res.Removed, MergedCopy{TrackID: g.members[i].TrackID, FilePath: g.members[i].FilePath}) + } + return res, err +} + +// sameAudio reports whether a group's copies are one recording, cut however: +// identical audio, or fingerprints all but identical (sameAudioBER). +func sameAudio(g *resolveGroup) bool { + return g.tier == "exact" || (g.worstBER != nil && *g.worstBER <= sameAudioBER) +} + +// namedCopies decides a one-album mismatch the lengths could not, from the +// recordings AcoustID says it may be. The track list names each slot's +// recording: the copy in a slot of one of them is kept and the others are +// losers. When none of the copies' slots is one of them, or AcoustID knows +// none (recordings empty), every copy is another song's and all go. ok is +// false when they fall in more than one copy's slot, the track list names no +// recording for a copy's slot, or the copies are all filed as one song. +// +// Copies sharing the kept copy's title are one song held twice, as in +// wrongSlots: neither kept nor lost here. +func namedCopies(copies []slotCopy, tracks []dbq.MbReleaseTrack, recordings []string) (keep int, losers []int, ok bool) { + keys := make([]string, len(copies)) + theirs := map[string]bool{} + for i, c := range copies { + keys[i] = MatchTitleKey(c.Title, c.Artist) + named := false + for _, t := range tracks { + if t.RecordingMbid == "" || MatchTitleKey(t.Title, c.Artist) != keys[i] { + continue + } + named = true + if slices.Contains(recordings, t.RecordingMbid) { + theirs[keys[i]] = true + } + } + if !named { + return -1, nil, false + } + } + if len(theirs) > 1 { + return -1, nil, false + } + keep = -1 + for i, key := range keys { + switch { + case theirs[key] && keep < 0: + keep = i + case theirs[key]: + default: + losers = append(losers, i) + } + } + if len(losers) == 0 { + return -1, nil, false + } + return keep, losers, true +} + +// groupRecordings is what AcoustID says a group's recording may be. Copies +// whose own lookups all name one recording are that recording, however their +// fingerprints compare. Otherwise only copies that are the same audio +// (sameAudio) are asked about: the recordings any copy's lookup matched, none +// when every copy's lookup found no match, or else a lookup of one copy now, +// while the pass has lookups left. known is false when there is no answer +// yet, and the group waits. +func groupRecordings( + ctx context.Context, logger *slog.Logger, g *resolveGroup, identify RecordingIdentifier, left *int, +) (recordings []string, known bool) { + matched, noMatch := 0, 0 + for _, m := range g.members { + switch { + case m.AcoustidState == nil: + case *m.AcoustidState == lookupMatched && m.AcoustidRecording != nil: + matched++ + if !slices.Contains(recordings, *m.AcoustidRecording) { + recordings = append(recordings, *m.AcoustidRecording) + } + case *m.AcoustidState == lookupNoMatch: + noMatch++ + } + } + if matched == len(g.members) && len(recordings) == 1 { + return recordings, true + } + if !sameAudio(g) { + return nil, false + } + if len(recordings) > 0 || noMatch == len(g.members) { + return recordings, true + } + if identify == nil || *left <= 0 { + return nil, false + } + *left-- + ids, err := identify(ctx, g.members[0].FilePath) + if err != nil { + if errors.Is(err, ErrIdentifyUnavailable) { + *left = 0 + } + logger.Info("duplicate resolve: asking AcoustID which song a recording is failed", + "group_id", syncpkg.FormatUUID(g.id), "err", err) + return nil, false + } + return ids, true +} + +// blockReleasesHolding sets aside the downloads a group's files were imported +// from, and returns their names. An album imported before Minstrel downloaded +// anything has none. +func blockReleasesHolding(ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, g *resolveGroup) []string { + q := dbq.New(pool) + seen := map[string]bool{} + var names []string + for _, m := range g.members { + grabs, err := q.ListImportedGrabsHolding(ctx, m.FilePath) + if err != nil { + logger.Warn("duplicate resolve: finding the download a file came from failed", "path", m.FilePath, "err", err) + continue + } + for _, gr := range grabs { + if seen[gr.InfoHash] { + continue + } + seen[gr.InfoHash] = true + if err := acquire.NewMemory(pool).Block(ctx, dbq.BlockReleaseParams{ + InfoHash: gr.InfoHash, Name: gr.Name, Reason: acquire.BlockFiles, + Detail: "one recording fills several of the album's tracks.", WantID: gr.WantID, + }); err != nil { + logger.Warn("duplicate resolve: setting the release aside failed", "release", gr.Name, "err", err) + continue + } + names = append(names, gr.Name) + } + } + return names +} + +// emptiedSlots is the losers whose song the album holds no other copy of: +// no present track on it under that title as long as the song's slot. Those +// wait for a copy fetched again (migration 0095). The rest are the song held +// twice, and missing-pair folding folds them into the copy held: Savant's +// VOID DLC "Weeper" beside Void's. A failed read counts every loser as +// emptied, which only makes the album wait for a copy it may not need. +func emptiedSlots( + ctx context.Context, q *dbq.Queries, logger *slog.Logger, g *resolveGroup, losers []int, tracks []dbq.MbReleaseTrack, +) []pgtype.UUID { + held, err := q.ListTracksByAlbum(ctx, dbq.ListTracksByAlbumParams{AlbumID: g.members[0].AlbumID}) + if err != nil { + logger.Warn("duplicate resolve: reading the album's tracks failed", "group_id", syncpkg.FormatUUID(g.id), "err", err) + } + inGroup := map[pgtype.UUID]bool{} + for _, m := range g.members { + inGroup[m.TrackID] = true + } + var emptied []pgtype.UUID + for _, i := range losers { + m := g.members[i] + key := MatchTitleKey(m.Title, m.ArtistName) + covered := false + for _, h := range held { + if covered || h.MissingSince.Valid || inGroup[h.ID] || MatchTitleKey(h.Title, m.ArtistName) != key { + continue + } + for _, t := range tracks { + if t.LengthMs > 0 && MatchTitleKey(t.Title, m.ArtistName) == key && absMs(t.LengthMs-h.DurationMs) <= slotFit { + covered = true + break + } + } + } + if !covered { + emptied = append(emptied, m.TrackID) + } + } + return emptied +} + +// neverRead reports whether the catalogue has never read a release group's +// releases. A group read and found empty is not asked about again: the +// catalogue's own refresh revisits it. +func neverRead(ctx context.Context, q *dbq.Queries, rg string) bool { + g, err := q.GetMBReleaseGroup(ctx, rg) + return errors.Is(err, pgx.ErrNoRows) || (err == nil && !g.ReleasesRefreshedAt.Valid) +} + +// removeWrongSlots deletes each loser's file, then marks its track missing in +// the same transaction that closes the group. The row is kept, with its likes +// and plays: it is the slot, and a good copy fetched again takes it over. Its +// recording MBID is dropped, being the wrong file's, so missing-pair folding +// does not fold the slot into the copy that was kept. The slots in emptied +// are stamped to wait for a copy fetched again (emptiedSlots). blocked names +// the releases set aside, for the record. A file that cannot be removed stops +// the group; the ones already gone are still marked, and the group waits for +// the next pass. actorID is the admin settling it by hand, invalid for the +// resolver. +func removeWrongSlots( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, g *resolveGroup, keep int, losers []int, + emptied []pgtype.UUID, blocked []string, actorID pgtype.UUID, +) (int, error) { + var gone []pgtype.UUID + var removedFiles []map[string]string + var removeErr error + for _, i := range losers { + m := g.members[i] + if err := removeTrackFileOnDisk(m.FilePath); err != nil { + removeErr = err + break + } + gone = append(gone, m.TrackID) + removedFiles = append(removedFiles, map[string]string{ + "track_id": syncpkg.FormatUUID(m.TrackID), "file_path": m.FilePath, "title": m.Title, + }) + } + if len(gone) == 0 { + return 0, removeErr + } + err := pgx.BeginFunc(ctx, pool, func(tx pgx.Tx) error { + tq := dbq.New(tx) + if _, err := tq.MarkTracksMissing(ctx, gone); err != nil { + return fmt.Errorf("mark missing: %w", err) + } + if err := tq.ClearTrackMbids(ctx, gone); err != nil { + return fmt.Errorf("clear mbids: %w", err) + } + if err := tq.MarkTracksEmptied(ctx, emptied); err != nil { + return fmt.Errorf("mark emptied: %w", err) + } + if err := logTrackChanges(ctx, tq, gone); err != nil { + return fmt.Errorf("log changes: %w", err) + } + if removeErr != nil { + return nil + } + if actorID.Valid { + _, err := tq.MarkDuplicateGroupMerged(ctx, g.id) + return err + } + _, err := tq.SettleDuplicateGroupAutomatically(ctx, dbq.SettleDuplicateGroupAutomaticallyParams{ + ID: g.id, Status: "merged", + }) + return err + }) + if err != nil { + return 0, err + } + details := map[string]any{ + "automatic": !actorID.Valid, + "group_id": syncpkg.FormatUUID(g.id), + "artist": g.members[0].ArtistName, + "removed": removedFiles, + } + if len(blocked) > 0 { + details["blocked"] = blocked + } + keptPath := "" + if keep >= 0 { + kept := g.members[keep] + details["kept_title"], details["kept_path"] = kept.Title, kept.FilePath + keptPath = kept.FilePath + } + audit.WriteOrLog(ctx, pool, logger, actorID, pgtype.UUID{}, audit.ActionDuplicateWrongFile, details) + logger.Info("duplicate resolve: removed copies from slots that were not theirs", + "group_id", syncpkg.FormatUUID(g.id), "kept", keptPath, "removed", len(gone)) + return len(gone), removeErr +} + +// notifyNewForReview tells admins about groups this pass classified for the +// first time and left for them. A group the resolver settled, linked or will +// link is not news, and would bury the groups that need them. +func notifyNewForReview(ctx context.Context, groups []*resolveGroup, settled map[pgtype.UUID]bool) { + var count int64 + var items []string + for _, g := range groups { + if g.members[0].PriorClass != nil || settled[g.id] { + continue + } + if g.class != ClassReview && g.class != ClassMismatch { + continue + } + count++ + if len(items) < notifyItems { + items = append(items, TrackLabel(g.members[0].ArtistName, g.members[0].Title)) + } + } + if count > 0 { + notifyAdmins(ctx, notifications.KindDuplicatesFound, notifications.Payload{Count: count, Items: items}) + } +} diff --git a/internal/library/duplicate_settle_test.go b/internal/library/duplicate_settle_test.go new file mode 100644 index 00000000..bf14ffc3 --- /dev/null +++ b/internal/library/duplicate_settle_test.go @@ -0,0 +1,148 @@ +package library + +import ( + "context" + "errors" + "path/filepath" + "strings" + "testing" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" + syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" +) + +// undecidedSlot is one recording filed as Void and as Reach, two songs +// MusicBrainz lists at lengths that cannot tell them apart, from a download +// Minstrel imported: the resolver leaves it pending for an admin. +func undecidedSlot(t *testing.T) (oneAlbumFixture, dbq.User) { + t.Helper() + f := newOneAlbumFixture(t, "acoustic", "Void", "Reach") + ctx := context.Background() + seedTrackList(t, f.pool, f.album, map[string]int{"Void": 220500, "Reach": 221000}) + if _, err := f.pool.Exec(ctx, + `INSERT INTO grabs (info_hash, name, key, client, save_path, origin, import_state, destination) + VALUES ('abc123', 'Savant - Void [WEB]', 'k', 'qbit', '/dl', 'schedule', 'imported', $1)`, + filepath.Dir(f.aPath)); err != nil { + t.Fatal(err) + } + if res, err := ResolveDuplicates(ctx, f.pool, nil, "", true); err != nil || res.WrongFiles != 0 { + t.Fatalf("resolve = %+v, %v; want the group left for the operator", res, err) + } + name := strings.NewReplacer("/", "-", " ", "-").Replace(t.Name()) + u, err := dbq.New(f.pool).CreateUser(ctx, dbq.CreateUserParams{ + Username: dbtest.TestUserPrefix + name, PasswordHash: "x", ApiTokenHash: name + "-token", + }) + if err != nil { + t.Fatal(err) + } + return f, u +} + +// emptiedTrack says whether a copy's track is marked missing and waiting for +// the album to be fetched again. +func emptiedTrack(t *testing.T, f oneAlbumFixture, id pgtype.UUID) bool { + t.Helper() + var emptied bool + if err := f.pool.QueryRow(context.Background(), + `SELECT missing_since IS NOT NULL AND emptied_at IS NOT NULL FROM tracks WHERE id = $1`, id).Scan(&emptied); err != nil { + t.Fatalf("track %s is gone: %v", syncpkg.FormatUUID(id), err) + } + return emptied +} + +// settledByHand checks the group closed as the admin's and the audit row +// names them. +func settledByHand(t *testing.T, f oneAlbumFixture, actor dbq.User) { + t.Helper() + if status, _, _, auto := (resolveFixture{pool: f.pool, groupID: f.groupID}).group(t); status != "merged" || auto { + t.Errorf("group = (%s, auto=%v), want merged by the operator", status, auto) + } + var n int + if err := f.pool.QueryRow(context.Background(), + `SELECT count(*) FROM audit_log WHERE action = 'duplicate_wrong_file' AND actor_id = $1 + AND metadata->>'group_id' = $2 AND NOT (metadata->>'automatic')::boolean`, + actor.ID, syncpkg.FormatUUID(f.groupID)).Scan(&n); err != nil { + t.Fatal(err) + } + if n != 1 { + t.Errorf("operator's wrong-file audit rows = %d, want 1", n) + } +} + +// "Get it again": the admin cannot say which song the recording is, so +// every copy goes, the download they came from is set aside, and both slots +// wait for the album fetched again. A merge would have kept one title. +func TestSettleWrongSlot_GetItAgainRemovesEveryCopy_Integration(t *testing.T) { + f, op := undecidedSlot(t) + ctx := context.Background() + res, err := SettleWrongSlot(ctx, f.pool, nil, f.groupID, pgtype.UUID{}, op.ID) + if err != nil { + t.Fatal(err) + } + if res.Kept != nil || len(res.Removed) != 2 || exists(f.aPath) || exists(f.bPath) { + t.Fatalf("result %+v, void exists %v, reach exists %v; want both copies removed", + res, exists(f.aPath), exists(f.bPath)) + } + for _, id := range []pgtype.UUID{f.a.ID, f.b.ID} { + if !emptiedTrack(t, f, id) { + t.Errorf("track %s is not waiting to be fetched again", syncpkg.FormatUUID(id)) + } + } + if !releaseBlocked(t, f.pool, "abc123") || len(res.Blocked) != 1 { + t.Errorf("blocked %v; want the download the files came from set aside", res.Blocked) + } + settledByHand(t, f, op) + + if _, err := SettleWrongSlot(ctx, f.pool, nil, f.groupID, pgtype.UUID{}, op.ID); !errors.Is(err, ErrDuplicateGroupNotPending) { + t.Errorf("settling it twice: err = %v, want ErrDuplicateGroupNotPending", err) + } +} + +// "This one is Reach": the copy the admin names stays, and only Void's slot +// is emptied and wanted again. +func TestSettleWrongSlot_KeepsTheCopyTheOperatorNames_Integration(t *testing.T) { + f, op := undecidedSlot(t) + res, err := SettleWrongSlot(context.Background(), f.pool, nil, f.groupID, f.b.ID, op.ID) + if err != nil { + t.Fatal(err) + } + if res.Kept == nil || res.Kept.TrackID != f.b.ID || len(res.Removed) != 1 || exists(f.aPath) || !exists(f.bPath) { + t.Fatalf("result %+v, void exists %v, reach exists %v; want only Void's copy removed", + res, exists(f.aPath), exists(f.bPath)) + } + if !emptiedTrack(t, f, f.a.ID) { + t.Error("Void's track is not waiting to be fetched again") + } + if !releaseBlocked(t, f.pool, "abc123") { + t.Error("the download the files came from is not set aside") + } + settledByHand(t, f, op) +} + +// Only a group of one recording in several slots is settled this way, and only +// with a copy from it. +func TestSettleWrongSlot_RefusesWhatItCannotSettle_Integration(t *testing.T) { + ctx := context.Background() + t.Run("two versions of a song", func(t *testing.T) { + f := newOneAlbumFixture(t, "acoustic", "Lovesick", "Lovesick (Instrumental)") + if _, err := SettleWrongSlot(ctx, f.pool, nil, f.groupID, pgtype.UUID{}, pgtype.UUID{}); !errors.Is(err, ErrNotWrongSlot) { + t.Errorf("err = %v, want ErrNotWrongSlot", err) + } + if !exists(f.aPath) || !exists(f.bPath) { + t.Error("a file was removed") + } + }) + t.Run("a copy from another group", func(t *testing.T) { + f, op := undecidedSlot(t) + other := pgtype.UUID{Bytes: [16]byte{1}, Valid: true} + if _, err := SettleWrongSlot(ctx, f.pool, nil, f.groupID, other, op.ID); !errors.Is(err, ErrSurvivorNotInGroup) { + t.Errorf("err = %v, want ErrSurvivorNotInGroup", err) + } + if !exists(f.aPath) || !exists(f.bPath) || releaseBlocked(t, f.pool, "abc123") { + t.Error("something changed") + } + }) +} diff --git a/internal/library/duplicate_survivor.go b/internal/library/duplicate_survivor.go index aaaa0d28..bfb8a021 100644 --- a/internal/library/duplicate_survivor.go +++ b/internal/library/duplicate_survivor.go @@ -14,23 +14,19 @@ type SurvivorCandidate struct { FileSize int64 AddedAt time.Time - // LidarrTracked: Lidarr maps this file to a track of the release it - // monitors. Removing it opens a hole Lidarr downloads again (M498), so a - // tracked copy outranks everything else. False when Lidarr was not asked. - LidarrTracked bool // TagFit is TagFitScore: how well the copy's tags and name fit the album. TagFit int } -// TagFitScore counts the signs that a copy belongs where it is filed (M498 -// #5436), one point each: +// TagFitScore counts the signs that a copy belongs where it is filed, one +// point each: // - it has a track number that no other track on its album also claims // - its file name carries no video-rip marker (SourceMarkersFor) // - it has a MusicBrainz recording id // // Two copies of one song differ in these far more often than in anything a -// listener hears: the Humanz rips differ by a few bytes, and file size picked -// the wrong one in 6 of 21 groups. +// listener hears: rips of one song can differ by a few bytes, which makes file +// size a poor guide. func TagFitScore(hasTrackNumber, positionClash bool, filePath string, hasMBID bool) int { score := 0 if hasTrackNumber && !positionClash { @@ -52,20 +48,21 @@ func TagFitScore(hasTrackNumber, positionClash bool, filePath string, hasMBID bo var losslessFormats = map[string]bool{"flac": true, "wav": true} // ProposeSurvivor picks which copy of a duplicate group to keep, and gives the -// reason in words the operator reads beside it. It is a default, not a verdict: -// the report shows it and the merge (#3911) lets the operator choose another. +// reason in words an admin reads beside it. It is a default, not a verdict: +// the report shows it and the merge lets an admin choose another. // // In order: -// 0. the copy Lidarr maps, then the copy whose tags fit the album best -// (TagFitScore). Both are zero for every copy when the caller does not -// know them, and the order falls through to the quality rules below -// 1. lossless over lossy — the one difference no later step can recover -// 2. the larger file — for one recording at one duration that is the higher +// 1. lossless over lossy — the one difference no later step can recover. It +// comes before tag fit: tags can be fixed, a deleted FLAC cannot +// 2. the copy whose tags fit the album best (TagFitScore). It is zero for +// every copy when the caller does not know it, and the order falls through +// to the rules below +// 3. the larger file — for one recording at one duration that is the higher // bitrate. The scanner does not record bitrate (tracks.bitrate is never // filled), so file size is the signal that actually exists -// 3. the copy in the library longest — the one most likely to carry the play +// 4. the copy in the library longest — the one most likely to carry the play // history and likes, so the merge moves the least -// 4. the lowest track id, so the choice is stable between page loads +// 5. the lowest track id, so the choice is stable between page loads func ProposeSurvivor(cands []SurvivorCandidate) (trackID, reason string) { if len(cands) == 0 { return "", "" @@ -81,12 +78,10 @@ func ProposeSurvivor(cands []SurvivorCandidate) (trackID, reason string) { // runner-up — the rule that actually decided, not every rule it passed. next := ranked[1] switch { - case best.LidarrTracked != next.LidarrTracked: - return best.TrackID, "the copy Lidarr tracks" - case best.TagFit != next.TagFit: - return best.TrackID, "tags fit the album" case isLossless(best) != isLossless(next): return best.TrackID, "lossless (" + strings.ToLower(best.FileFormat) + ")" + case best.TagFit != next.TagFit: + return best.TrackID, "tags fit the album" case best.FileSize != next.FileSize: return best.TrackID, "largest file" case !best.AddedAt.Equal(next.AddedAt): @@ -97,15 +92,12 @@ func ProposeSurvivor(cands []SurvivorCandidate) (trackID, reason string) { } func survivorBefore(a, b SurvivorCandidate) bool { - if a.LidarrTracked != b.LidarrTracked { - return a.LidarrTracked + if isLossless(a) != isLossless(b) { + return isLossless(a) } if a.TagFit != b.TagFit { return a.TagFit > b.TagFit } - if isLossless(a) != isLossless(b) { - return isLossless(a) - } if a.FileSize != b.FileSize { return a.FileSize > b.FileSize } diff --git a/internal/library/duplicate_survivor_test.go b/internal/library/duplicate_survivor_test.go index 4062716f..ad0020f9 100644 --- a/internal/library/duplicate_survivor_test.go +++ b/internal/library/duplicate_survivor_test.go @@ -42,7 +42,7 @@ func TestProposeSurvivor(t *testing.T) { wantID: "320k", wantReason: "largest file", }, { - // The #3885 pair: identical audio, sizes equal but for the tags. + // Identical audio, sizes equal but for the tags. name: "the longest-standing copy wins when size ties", cands: []SurvivorCandidate{ {TrackID: "www-02", FileFormat: "mp3", FileSize: 6_900_000, AddedAt: newer}, @@ -90,11 +90,11 @@ func TestProposeSurvivor_ReasonIsTheDecidingRule(t *testing.T) { } } -// M498 #5436: the copy Lidarr maps outranks everything, then the copy whose -// tags fit the album; quality decides only between copies equal on both. Each -// case is built so the older rules would choose the other copy, so a reorder -// that drops the new rules fails. -func TestProposeSurvivor_LidarrThenTagFit(t *testing.T) { +// The copy whose tags fit the album outranks every quality rule but lossless, +// which decides first: a deleted FLAC cannot be recovered, so it never goes +// for an mp3 whose only edge is a recording MBID. Each tag +// fit case is built so file size would choose the other copy. +func TestProposeSurvivor_LosslessThenTagFit(t *testing.T) { at := time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC) cases := []struct { name string @@ -103,12 +103,20 @@ func TestProposeSurvivor_LidarrThenTagFit(t *testing.T) { wantReason string }{ { - name: "tracked beats lossless and better tags", + name: "lossless beats tag fit", cands: []SurvivorCandidate{ - {TrackID: "flac", FileFormat: "flac", FileSize: 40_000_000, AddedAt: at, TagFit: 3}, - {TrackID: "tracked", FileFormat: "mp3", FileSize: 5_000_000, AddedAt: at, TagFit: 1, LidarrTracked: true}, + {TrackID: "fits", FileFormat: "mp3", FileSize: 5_000_000, AddedAt: at, TagFit: 3}, + {TrackID: "flac", FileFormat: "flac", FileSize: 40_000_000, AddedAt: at, TagFit: 1}, }, - wantID: "tracked", wantReason: "the copy Lidarr tracks", + wantID: "flac", wantReason: "lossless (flac)", + }, + { + name: "tag fit beats a larger file", + cands: []SurvivorCandidate{ + {TrackID: "big", FileFormat: "mp3", FileSize: 9_000_000, AddedAt: at, TagFit: 1}, + {TrackID: "fits", FileFormat: "m4a", FileSize: 5_000_000, AddedAt: at, TagFit: 3}, + }, + wantID: "fits", wantReason: "tags fit the album", }, { // The Humanz shape: two rips a few bytes apart, one carrying a diff --git a/internal/library/duplicate_sweep.go b/internal/library/duplicate_sweep.go index 46e6cba7..df54351e 100644 --- a/internal/library/duplicate_sweep.go +++ b/internal/library/duplicate_sweep.go @@ -14,41 +14,41 @@ import ( "github.com/jackc/pgx/v5/pgxpool" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) -// Duplicate sweep (M400 #3910). +// Duplicate sweep. // -// Reads fingerprints, runs them through the matcher and records what it proposes -// in duplicate_groups. It never merges or deletes anything the operator has not -// asked for: a group is a proposal, reviewed in the admin report (#3912). +// Reads fingerprints, runs them through the matcher and records what it +// proposes in duplicate_groups. It never merges or deletes anything itself: a +// group is a proposal, for the resolver and the admin report. // -// Blocked on duration, deliberately not on title: the #3885 pair are titled -// "WWW" and "WWW (instrumental)", so a title block would have missed the case -// that started the milestone. Candidates stream in (duration_ms, id) order and -// the grouper holds only a few seconds of durations at a time. +// Blocked on duration, deliberately not on title: one file filed as "WWW" and +// "WWW (instrumental)" would escape a title block. Candidates stream in +// (duration_ms, id) order and the grouper holds only a few seconds of durations +// at a time. // duplicateCandidatePage is how many candidates one query returns. Each row // carries a ~4 KB fingerprint, so a page is about 2 MB. const duplicateCandidatePage = 500 // duplicateSweepTick is how often the worker checks whether a sweep is due. The -// operator's sweep interval (#3913) is the least time between sweeps; the tick -// only bounds how late past it one starts. With nothing due, a tick is two cheap -// aggregate queries. +// sweep interval setting is the least time between sweeps; the tick only bounds +// how late past it one starts. With nothing due, a tick is two cheap aggregate +// queries. const duplicateSweepTick = time.Hour // sweepIntervalSlack absorbs the moment between a tick and the sweep it starts -// stamping started_at. Without it a one-hour interval checked on a one-hour tick -// would find the last sweep a moment under an hour old, and skip every other tick. +// stamping started_at. Without it a one-hour interval checked on a one-hour +// tick would find the last sweep a moment under an hour old, and skip every +// other tick. const sweepIntervalSlack = 5 * time.Minute // staleDuplicateSweepThreshold is the age past which an in-flight sweep is -// assumed dead — a crash mid-sweep leaves finished_at NULL for ever — and another -// may start. Twice the library scan's threshold, because a sweep compares -// fingerprints across the whole library and can legitimately run long on a big -// one. +// assumed dead — a crash mid-sweep leaves finished_at NULL for ever — and +// another may start. Twice the library scan's threshold, because a sweep +// compares fingerprints across the whole library and can legitimately run long +// on a big one. const staleDuplicateSweepThreshold = 2 * time.Hour // duplicateSweepFinishTimeout bounds recording that a sweep ended. It runs on a @@ -61,7 +61,7 @@ type DuplicateSweepResult struct { Candidates int // tracks with a chromaprint that were streamed Groups int // groups the matcher found Proposed int // written as pending, new or refreshed - Suppressed int // not proposed: already dismissed or resolved by the operator + Suppressed int // not proposed: already dismissed or resolved Retired int // pending proposals this sweep did not find again, removed Oversize int // acoustic clusters too large to propose } @@ -102,37 +102,12 @@ func runDuplicateSweep( } } - if runErr == nil { - notifyNewDuplicates(finishCtx, q, sweep.StartedAt, logger) - } - logger.Info("duplicate sweep complete", "candidates", res.Candidates, "groups", res.Groups, "proposed", res.Proposed, "suppressed", res.Suppressed, "retired", res.Retired, "oversize", res.Oversize, "err", runErr) return res, runErr } -// notifyNewDuplicates tells admins when a sweep proposed a group it had not -// proposed before (M489), counting every proposal awaiting review. A sweep -// that only re-finds known groups says nothing, so reading the notice once -// is enough until something new turns up. -func notifyNewDuplicates(ctx context.Context, q *dbq.Queries, sweepStarted pgtype.Timestamptz, logger *slog.Logger) { - fresh, err := q.CountDuplicateGroupsDetectedSince(ctx, sweepStarted) - if err != nil { - logger.Warn("duplicate sweep: counting new proposals failed", "err", err) - return - } - if fresh == 0 { - return - } - pending, err := q.CountPendingDuplicateGroups(ctx) - if err != nil { - logger.Warn("duplicate sweep: counting pending proposals failed", "err", err) - return - } - notifyAdmins(ctx, notifications.KindDuplicatesFound, notifications.Payload{Count: pending}) -} - func sweepDuplicates( ctx context.Context, q *dbq.Queries, sweepID pgtype.UUID, cfg FingerprintSettings, pageSize int32, ) (DuplicateSweepResult, error) { @@ -154,15 +129,17 @@ func sweepDuplicates( exactSeen := make([]bool, len(exactRows)) // Acoustic tier, streamed in duration order. The first member of an exact - // group the stream meets stands in for the whole group; the rest are skipped. - // Only prints at the current length are streamed: a print at another length - // cannot be compared, and is waiting on the backfill to be re-derived. + // group the stream meets stands in for the whole group; the rest are + // skipped. Only prints at the current length are streamed: a print at + // another length cannot be compared, and is waiting on the backfill to be + // re-derived. grouper := newStreamGrouper(cfg.AcousticMaxBitErrorRate) params := dbq.ListDuplicateCandidatesParams{ CurrentVersion: fingerprintVersion, ChromaprintLengthSec: cfg.ChromaprintLengthSec, - // Durations are never negative, and the all-zero uuid sorts first: every - // row is after this cursor. Valid must be true, or "> NULL" matches nothing. + // Durations are never negative, and the all-zero uuid sorts first: + // every row is after this cursor. Valid must be true, or "> NULL" + // matches nothing. AfterDurationMs: -1, AfterID: pgtype.UUID{Valid: true}, PageLimit: pageSize, @@ -268,7 +245,7 @@ func sweepDuplicates( } // coveredByDismissal reports whether every member of a proposal sat together in -// one group the operator dismissed. A subset counts: dismissing {A, B, C} said +// one group an admin dismissed. A subset counts: dismissing {A, B, C} said // none of them are copies of each other, so proposing {A, B} again would be // asking the same question twice. A superset does not count: a new copy joining // is new evidence, and worth asking about. @@ -331,7 +308,7 @@ func TryStartDuplicateSweep( } // DuplicateSweepWorker sweeps whenever its input has changed, at most once per -// the operator's sweep interval. +// sweep interval. type DuplicateSweepWorker struct { pool *pgxpool.Pool logger *slog.Logger @@ -362,7 +339,7 @@ func (w *DuplicateSweepWorker) Run(ctx context.Context) { } } -// tickOnce contains one check so nothing it does can stop the next tick (rule 157). +// tickOnce contains one check so nothing it does can stop the next tick. func (w *DuplicateSweepWorker) tickOnce(ctx context.Context) { defer func() { if r := recover(); r != nil { @@ -403,12 +380,12 @@ func duplicateSweepDue(ctx context.Context, q *dbq.Queries, cfg FingerprintSetti } // sweepIsDue reports whether a sweep should start: something it reads has -// changed since the last sweep started, and the operator's interval has passed. +// changed since the last sweep started, and the sweep interval has passed. // // Two things can change its answer. Fingerprints are its input, so any written // after the last sweep began count; while the backfill runs that is true every -// tick, which is what the interval is for. And a settings save counts, because a -// new threshold or length changes what the same fingerprints group into. +// tick, which is what the interval is for. And a settings save counts, because +// a new threshold or length changes what the same fingerprints group into. func sweepIsDue(latestFingerprint, lastSweepStart pgtype.Timestamptz, cfg FingerprintSettings, now time.Time) bool { if !latestFingerprint.Valid { return false // nothing fingerprinted yet diff --git a/internal/library/duplicate_sweep_test.go b/internal/library/duplicate_sweep_test.go index 15bc478b..676425ff 100644 --- a/internal/library/duplicate_sweep_test.go +++ b/internal/library/duplicate_sweep_test.go @@ -37,8 +37,9 @@ func TestCoveredByDismissal(t *testing.T) { } } -// TestDuplicateSweep_Integration pins what the sweep proposes, what it leaves out, -// and how re-sweeping treats a dismissal and a proposal that no longer holds. +// TestDuplicateSweep_Integration pins what the sweep proposes, what it leaves +// out, and how re-sweeping treats a dismissal and a proposal that no longer +// holds. func TestDuplicateSweep_Integration(t *testing.T) { pool := newPool(t) ctx := context.Background() @@ -175,9 +176,10 @@ func TestDuplicateSweep_Integration(t *testing.T) { t.Fatalf("latest sweep = %+v, want finished without error", last) } - // 5. Prints taken at another length are never compared (#3913). Every print - // here was taken at the default length, so a sweep at 60s has nothing to read, - // rather than scoring 120s prints against each other as if they were 60s ones. + // 5. Prints taken at another length are never compared. Every print + // here was taken at the default length, so a sweep at 60s has nothing to + // read, rather than scoring 120s prints against each other as if they were + // 60s ones. atOtherLength := DefaultFingerprintSettings atOtherLength.ChromaprintLengthSec = 60 res, err = runDuplicateSweep(ctx, pool, logger, atOtherLength, duplicateCandidatePage) diff --git a/internal/library/eventbus.go b/internal/library/eventbus.go index 6a1c805a..aacb603d 100644 --- a/internal/library/eventbus.go +++ b/internal/library/eventbus.go @@ -8,7 +8,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/eventbus" ) -// Package-level bus pointer for scan lifecycle events (#392). Set once +// Package-level bus pointer for scan lifecycle events. Set once // at startup via SetEventBus; reads are nil-safe so test contexts that // never call SetEventBus simply skip publishing. // diff --git a/internal/library/fingerprint.go b/internal/library/fingerprint.go index 12890a78..ff35fdb5 100644 --- a/internal/library/fingerprint.go +++ b/internal/library/fingerprint.go @@ -18,7 +18,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Acoustic identity (M400). +// Acoustic identity. // // Two values per track, because they answer different questions: // @@ -26,7 +26,7 @@ import ( // same audio bytes, whatever the tags or container around // them say. No threshold and no false positives — this is // what catches two copies of one MP3 that differ only in -// their ID3 (#3885). +// their ID3. // // chromaprint fpcalc's raw fingerprint. Close means the same // recording, even at another bitrate or in another codec @@ -39,9 +39,9 @@ import ( // fingerprintTimeout bounds one ffmpeg hash or fpcalc call. Longer than // probeTimeout because both read the audio rather than a header: the hash reads -// every packet and fpcalc decodes up to its -length. 60s leaves room for a large -// lossless file on a slow network mount; a call needing more is a stall, not a -// big file. +// every packet and fpcalc decodes up to its -length. 60s leaves room for a +// large lossless file on a slow network mount; a call needing more is a stall, +// not a big file. const fingerprintTimeout = 60 * time.Second // fingerprintWaitDelay bounds how long Output may keep waiting on the tool's @@ -52,9 +52,10 @@ const fingerprintWaitDelay = 5 * time.Second // fingerprintVersion stamps how a track_fingerprints row was derived. Bump it // whenever the derivation changes — the hash arguments or fpcalc's flags — and // the backfill re-derives every row below it. Fingerprints taken by two methods -// are not comparable, and nothing else would reveal that the library held a mix. +// are not comparable, and nothing else would reveal that the library held a +// mix. // -// The length is deliberately not part of it: it is an operator setting (#3913), +// The length is deliberately not part of it: it is an admin setting, // so each row records the length it was taken at and readers compare only rows // at the current one. See fingerprint_settings.go. const fingerprintVersion int16 = 1 @@ -63,7 +64,7 @@ const fingerprintVersion int16 = 1 // failed exit because a stall is a fact about the mount, not about the file. var errFingerprintTimeout = errors.New("fingerprint tool timed out") -// defaultChromaprintLengthSec is the shipped value of the length setting (#3913): +// defaultChromaprintLengthSec is the shipped value of the length setting: // how many seconds of audio fpcalc fingerprints. 120 is fpcalc's own default. const defaultChromaprintLengthSec = 120 @@ -99,10 +100,10 @@ func streamHashArgs(path string) []string { // fpcalcArgs asks for the raw fingerprint as SIGNED integers. // // -raw because the matcher compares items bit by bit, which the compressed form -// cannot do without being unpacked first. -signed because the column is Postgres -// integer[], which is signed: fpcalc's default prints uint32, and half of those -// values do not fit. Signed output is the same 32 bits with no reinterpretation -// step left to get wrong. +// cannot do without being unpacked first. -signed because the column is +// Postgres integer[], which is signed: fpcalc's default prints uint32, and half +// of those values do not fit. Signed output is the same 32 bits with no +// reinterpretation step left to get wrong. func fpcalcArgs(path string, lengthSec int) []string { return []string{ "-raw", "-signed", @@ -112,7 +113,7 @@ func fpcalcArgs(path string, lengthSec int) []string { } // lookupFingerprintLengthSec is how much audio the AcoustID lookup print -// covers (M401). Fixed at fpcalc's default rather than following the +// covers. Fixed at fpcalc's default rather than following the // duplicate-detection length setting: AcoustID's database is built from // prints that taggers such as Picard take at the default, and a lookup print // of a different span matches it less well. @@ -120,7 +121,7 @@ const lookupFingerprintLengthSec = 120 // fpcalcLookupArgs asks for the COMPRESSED fingerprint, fpcalc's default // output and the string AcoustID's lookup takes. A second decode beside the -// raw print, chosen over re-encoding the stored raw array in Go (M401 D1): +// raw print, chosen over re-encoding the stored raw array in Go: // fpcalc is the reference encoder, while a subtle bug in a port would still // produce a well-formed string that AcoustID answers with "no match". func fpcalcLookupArgs(path string) []string { @@ -147,8 +148,8 @@ type fingerprintResult struct { } // fingerprintOptions is what the settings decide for one attempt. Captured once -// per file, so the length a chromaprint was taken at is the length stored with it -// even if the setting changes mid-attempt. +// per file, so the length a chromaprint was taken at is the length stored with +// it even if the setting changes mid-attempt. type fingerprintOptions struct { lengthSec int32 // chromaprint false takes the stream hash alone: a demux, no decode. @@ -208,11 +209,11 @@ const ( ) // storeFingerprint records one attempt, for the scan (new or changed bytes) and -// the backfill (#3908) alike, so there is one rule for what gets written. It +// the backfill alike, so there is one rule for what gets written. It // never fails its caller: a missing fingerprint only keeps a track out of // duplicate detection, which is not worth dropping a scan or a pass over. // -// lengthSec is the length fp's chromaprint was taken at, stored with it (#3913). +// lengthSec is the length fp's chromaprint was taken at, stored with it. func storeFingerprint( ctx context.Context, q *dbq.Queries, logger *slog.Logger, trackID pgtype.UUID, path string, fp fingerprintResult, lengthSec int32, @@ -264,7 +265,7 @@ func computeAudioStreamSHA256(ctx context.Context, path string) ([]byte, error) // lengthSec seconds of the file. func computeChromaprint(ctx context.Context, path string, lengthSec int32) ([]int32, error) { out, err := runFingerprintTool(ctx, "fpcalc", fpcalcArgs(path, int(lengthSec))) - if err != nil { + if err != nil && !fpcalcReadToEnd(err, out, int(lengthSec)) { return nil, err } return parseFpcalcRaw(out) @@ -276,7 +277,7 @@ func computeChromaprint(ctx context.Context, path string, lengthSec int32) ([]in // rejects as a verdict. func computeLookupFingerprint(ctx context.Context, path string) (lookupFingerprint, error) { out, err := runFingerprintTool(ctx, "fpcalc", fpcalcLookupArgs(path)) - if err != nil { + if err != nil && !fpcalcReadToEnd(err, out, lookupFingerprintLengthSec) { return lookupFingerprint{}, err } return parseFpcalcCompressed(out) @@ -284,11 +285,14 @@ func computeLookupFingerprint(ctx context.Context, path string) (lookupFingerpri // runFingerprintTool runs one tool under fingerprintTimeout. // -// Any non-zero exit is an error, and that deliberately includes fpcalc's exit 3: -// "reading failed, but here is a fingerprint of what I got". A partial +// Any non-zero exit is an error, and that deliberately includes fpcalc's exit +// 3: "reading failed, but here is a fingerprint of what I got". A partial // fingerprint of a damaged file is not that file's identity. Stored, it would // score against a healthy copy over whatever prefix survived, and could group // or fail to group either way. Absent is better than wrong. +// +// A tool that ran and exited non-zero returns its stdout beside the +// *toolExitError, for the one exit 3 that is not damage (fpcalcReadToEnd). func runFingerprintTool(ctx context.Context, name string, args []string) ([]byte, error) { runCtx, cancel := context.WithTimeout(ctx, fingerprintTimeout) defer cancel() @@ -311,11 +315,57 @@ func runFingerprintTool(ctx context.Context, name string, args []string) ([]byte } var exitErr *exec.ExitError if errors.As(err, &exitErr) { - return nil, fmt.Errorf("%s exited %d: %s", name, exitErr.ExitCode(), stderrTail(exitErr.Stderr)) + return out, &toolExitError{name: name, code: exitErr.ExitCode(), stderr: stderrTail(exitErr.Stderr)} } return nil, fmt.Errorf("%s: %w", name, err) } +// toolExitError is a fingerprint tool that ran and exited non-zero: a verdict +// on the file, unlike a stall or a missing tool. +type toolExitError struct { + name string + code int + stderr []byte // the tail, as stderrTail keeps it +} + +func (e *toolExitError) Error() string { + return fmt.Sprintf("%s exited %d: %s", e.name, e.code, e.stderr) +} + +// fpcalcExitPartial is fpcalc's exit for "reading failed, but here is a +// fingerprint of what I got". +const fpcalcExitPartial = 3 + +// fpcalcReadToEnd reports whether fpcalc's partial exit only means it decoded +// the whole file. The image's fpcalc 1.5.1 reports the decoder's end of file +// as "Error decoding audio frame (End of file)" and exits 3 whenever it +// reaches the end before -length, which is every file shorter than the span +// asked for. That print covers the whole file, so it is the file's identity. +// +// A damaged file can stop at an end of file too. Its DURATION, the length the +// container declares, is then longer than the span asked for, and it stays a +// failure. +func fpcalcReadToEnd(err error, out []byte, lengthSec int) bool { + var exitErr *toolExitError + if !errors.As(err, &exitErr) || exitErr.code != fpcalcExitPartial || + !bytes.Contains(exitErr.stderr, []byte("(End of file)")) { + return false + } + d, ok := fpcalcDuration(out) + return ok && d > 0 && d <= lengthSec +} + +// fpcalcDuration reads the DURATION= line of fpcalc's output, in whole seconds. +func fpcalcDuration(out []byte) (int, bool) { + for _, line := range strings.Split(string(out), "\n") { + if v, ok := strings.CutPrefix(strings.TrimSpace(line), "DURATION="); ok { + d, err := strconv.Atoi(v) + return d, err == nil + } + } + return 0, false +} + // stderrTail keeps the END of a failing tool's stderr. ffmpeg and fpcalc print // the actual reason last, after any banner or per-frame warnings, so a cap that // kept the head would log the noise and drop the cause. @@ -422,3 +472,16 @@ func notURLBase64(r rune) bool { } return true } + +// LookupFingerprint is the compressed fingerprint AcoustID's lookup takes and +// the file's whole length in seconds, for a file not in the library yet: the +// import checks a download with it. IsInconclusive tells a +// missing or stalled fpcalc from a file it rejected. +func LookupFingerprint(ctx context.Context, path string) (fingerprint string, durationSec int, err error) { + fp, err := computeLookupFingerprint(ctx, path) + return fp.fingerprint, fp.durationSec, err +} + +// IsInconclusive reports that a fingerprint tool could not give an answer +// (missing, timed out), as opposed to rejecting the file. +func IsInconclusive(err error) bool { return isInconclusive(err) } diff --git a/internal/library/fingerprint_backfill.go b/internal/library/fingerprint_backfill.go index eb19c1c8..7f25ab9c 100644 --- a/internal/library/fingerprint_backfill.go +++ b/internal/library/fingerprint_backfill.go @@ -13,11 +13,11 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Fingerprint backfill (M400 #3908). +// Fingerprint backfill. // -// The scan fingerprints only bytes it has not seen (see scanFile), so every track -// imported before fingerprinting existed — and every row derived by an older -// fingerprintVersion — needs a pass of its own. That pass is this worker. +// The scan fingerprints only bytes it has not seen (see scanFile), so every +// track imported before fingerprinting existed — and every row derived by an +// older fingerprintVersion — needs a pass of its own. That pass is this worker. // // Its own worker rather than a stage in RunScan, for two reasons, both about // time: @@ -28,20 +28,23 @@ import ( // - A long stage holds the scan run in flight, and a manual rescan answers 409 // for as long as it runs. // -// Progress is read live (FingerprintCoverage, the admin gauge) rather than from a -// scan_runs tally: the work spans many passes with no single run to attach to. +// Progress is read live (FingerprintCoverage, the admin gauge) rather than from +// a scan_runs tally: the work spans many passes with no single run to attach +// to. // fingerprintBackfillTick is how often the worker looks for work. Once the -// library has caught up, a tick is one indexed query; mostly the hour bounds how -// long a file that timed out on a slow mount waits before it is tried again. +// library has caught up, a tick is one indexed query; mostly the hour bounds +// how long a file that timed out on a slow mount waits before it is tried +// again. const fingerprintBackfillTick = time.Hour -// fingerprintBackfillBatch is how many tracks one query hands the worker. Small, -// so tracks the scan adds mid-pass are not stuck behind one enormous page. +// fingerprintBackfillBatch is how many tracks one query hands the worker. +// Small, so tracks the scan adds mid-pass are not stuck behind one enormous +// page. const fingerprintBackfillBatch = 50 -// fingerprintBackfillConcurrency is the shipped value of the concurrency setting -// (#3913): how many files are decoded at once. Two is deliberately low: fpcalc +// fingerprintBackfillConcurrency is the shipped value of the concurrency +// setting: how many files are decoded at once. Two is deliberately low: fpcalc // and the stream hash compete with playback transcoding for CPU and with // streaming for the mount, and a backfill that makes playback stutter is worse // than one that takes longer. @@ -75,7 +78,8 @@ type FingerprintBackfillWorker struct { tick time.Duration batch int32 // fingerprint is a field for the same reason as Scanner.fingerprint: an - // integration test pins which tracks a pass touches, not what the tools print. + // integration test pins which tracks a pass touches, not what the tools + // print. fingerprint func(ctx context.Context, path string, opts fingerprintOptions) fingerprintResult } @@ -110,8 +114,8 @@ func (w *FingerprintBackfillWorker) Run(ctx context.Context) { } } -// runOnce contains a pass so that nothing it does — an error, a panic — can stop -// the next tick from firing (rule 157). +// runOnce contains a pass so that nothing it does — an error, a panic — can +// stop the next tick from firing. func (w *FingerprintBackfillWorker) runOnce(ctx context.Context) { defer func() { if r := recover(); r != nil { @@ -135,10 +139,10 @@ func (w *FingerprintBackfillWorker) runOnce(ctx context.Context) { // retried forever within the pass. // // Settings are read before every batch, so a save takes effect within a batch -// rather than an hour (#3913): switching fingerprinting off ends the pass, a new -// concurrency applies to the next batch, and a new length restarts the walk from -// the top at that length, because every row written at the old one went stale -// the moment it changed. +// rather than an hour: switching fingerprinting off ends the pass, a new +// concurrency applies to the next batch, and a new length restarts the walk +// from the top at that length, because every row written at the old one went +// stale the moment it changed. func (w *FingerprintBackfillWorker) pass(ctx context.Context) (BackfillFingerprintsResult, error) { q := dbq.New(w.pool) var ( @@ -146,7 +150,8 @@ func (w *FingerprintBackfillWorker) pass(ctx context.Context) (BackfillFingerpri mu sync.Mutex ) // The all-zero uuid sorts before every real id. Valid must be true: a NULL - // cursor would make "id > NULL" match nothing and every pass a silent no-op. + // cursor would make "id > NULL" match nothing and every pass a silent + // no-op. start := pgtype.UUID{Valid: true} after := start lengthSec := w.settings.Get().ChromaprintLengthSec diff --git a/internal/library/fingerprint_backfill_test.go b/internal/library/fingerprint_backfill_test.go index fb3a166a..ae37c662 100644 --- a/internal/library/fingerprint_backfill_test.go +++ b/internal/library/fingerprint_backfill_test.go @@ -151,11 +151,11 @@ func TestFingerprintBackfill_Integration(t *testing.T) { t.Errorf("coverage buckets %+v do not sum to the total", cov) } - // 5. Changing the length (#3913) makes every stored row stale at once — the + // 5. Changing the length makes every stored row stale at once — the // gauge shows the whole library pending before the backfill has touched a // file — and the next pass re-derives each at the new length. The failure - // this prevents is invisible from the UI: prints at two lengths that silently - // never match. + // this prevents is invisible from the UI: prints at two lengths that + // silently never match. shorter := DefaultFingerprintSettings shorter.ChromaprintLengthSec = 60 if _, err := settings.Set(ctx, shorter); err != nil { diff --git a/internal/library/fingerprint_scan_test.go b/internal/library/fingerprint_scan_test.go index 1e4c1d0a..8d505438 100644 --- a/internal/library/fingerprint_scan_test.go +++ b/internal/library/fingerprint_scan_test.go @@ -193,9 +193,10 @@ func TestScanner_FingerprintsOnlyNewOrChangedBytes_Integration(t *testing.T) { t.Fatalf("rejected file stored %+v, want both halves NULL at version %d", got, fingerprintVersion) } - // 6. With fingerprinting off (#3913) the scan decodes nothing. It still asks - // for the stream hash — a demux, and what recognises a moved file — but stores - // no row, and a changed file's old row goes: it describes bytes that are gone. + // 6. With fingerprinting off the scan decodes nothing. It still asks + // for the stream hash — a demux, and what recognises a moved file — but + // stores no row, and a changed file's old row goes: it describes bytes that + // are gone. result = fingerprintResult{streamSHA256: sum, chromaprint: chroma} off := DefaultFingerprintSettings off.Enabled = false diff --git a/internal/library/fingerprint_settings.go b/internal/library/fingerprint_settings.go index b451441a..ff13c24b 100644 --- a/internal/library/fingerprint_settings.go +++ b/internal/library/fingerprint_settings.go @@ -12,10 +12,10 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Fingerprinting settings (M400 #3913). +// Fingerprinting settings. // -// Rule 25: what an operator might tune lives in the database and changes without -// a restart. One service instance is shared by the scanner, the backfill, the +// What an admin might tune lives in the database and changes without a +// restart. One service instance is shared by the scanner, the backfill, the // duplicate sweep and the admin API, so a save reaches all of them at once. // // The length is the dangerous knob: chromaprints taken at two lengths cannot be @@ -29,8 +29,9 @@ import ( // Bounds for each setting, mirrored by migration 0061's CHECKs. const ( // Below 30s a print has too few items to align at the matcher's larger - // offsets and still overlap by minOverlapItems. Above 600s each print passes - // 20 KB and a candidate page of them grows past what one sweep should hold. + // offsets and still overlap by minOverlapItems. Above 600s each print + // passes 20 KB and a candidate page of them grows past what one sweep + // should hold. minChromaprintLengthSec = 30 maxChromaprintLengthSec = 600 // Unrelated audio sits near 0.5; a ceiling of 0.35 keeps a loosened @@ -55,9 +56,9 @@ type FingerprintSettings struct { AcousticMaxBitErrorRate float64 BackfillConcurrency int32 SweepIntervalHours int32 - // AutoResolve lets the duplicate resolver (M498) act on its own: merge - // copies Lidarr does not need and change a Lidarr release that lists songs - // twice. Off, it only classifies, and every group waits for the operator. + // AutoResolve lets the duplicate resolver act on its own: merge copies no + // release needs. Off, it only classifies, and every group waits for an + // admin. AutoResolve bool // UpdatedAt is when the settings were last saved. Set by the database; // ignored by Set. @@ -143,7 +144,8 @@ func validateFingerprintSettings(in FingerprintSettings) error { case in.ChromaprintLengthSec < minChromaprintLengthSec || in.ChromaprintLengthSec > maxChromaprintLengthSec: return fmt.Errorf("%w: chromaprint_length_sec must be %d-%d", ErrFingerprintSettingOutOfRange, minChromaprintLengthSec, maxChromaprintLengthSec) - // Written as a negated range so NaN, which fails every comparison, is refused. + // Written as a negated range so NaN, which fails every comparison, is + // refused. case !(in.AcousticMaxBitErrorRate >= minAcousticMaxBitErrorRate && in.AcousticMaxBitErrorRate <= maxAcousticMaxBitErrorRate): return fmt.Errorf("%w: acoustic_max_bit_error_rate must be %.2f-%.2f", ErrFingerprintSettingOutOfRange, minAcousticMaxBitErrorRate, maxAcousticMaxBitErrorRate) diff --git a/internal/library/fingerprint_settings_test.go b/internal/library/fingerprint_settings_test.go index 65c33d4f..d77b9b59 100644 --- a/internal/library/fingerprint_settings_test.go +++ b/internal/library/fingerprint_settings_test.go @@ -84,8 +84,8 @@ func TestFingerprintSettingsService_Integration(t *testing.T) { t.Fatalf("load: %v", err) } // ResetDB puts every column back to its migration default, so this pins the - // Go defaults to migration 0061's. Were they to drift, a database that could - // not be read would fingerprint differently from one that could. + // Go defaults to migration 0061's. Were they to drift, a database that + // could not be read would fingerprint differently from one that could. loaded := svc.Get() if loaded.UpdatedAt.IsZero() { t.Fatal("loaded settings carry no updated_at") diff --git a/internal/library/fingerprint_test.go b/internal/library/fingerprint_test.go index a2bf4d56..1e9d3fe8 100644 --- a/internal/library/fingerprint_test.go +++ b/internal/library/fingerprint_test.go @@ -122,7 +122,8 @@ func TestFpcalcLookupArgs_RequestCompressedOutput(t *testing.T) { } func TestParseStreamHash(t *testing.T) { - // The real value ffmpeg printed for both files of the #3885 pair. + // The real value ffmpeg printed for two copies of one MP3 differing only + // in their ID3. const www = "24e2daa3b4a534ff1a8d1a76f67810205869daf89f728d83a16625da4d28a18e" got, err := parseStreamHash([]byte("SHA256=" + www + "\n")) @@ -146,7 +147,8 @@ func TestParseStreamHash(t *testing.T) { } } -// followedBy reports whether flag appears in args immediately followed by value. +// followedBy reports whether flag appears in args immediately followed by +// value. func followedBy(args []string, flag, value string) bool { for i := 0; i+1 < len(args); i++ { if args[i] == flag && args[i+1] == value { @@ -253,3 +255,45 @@ func TestFingerprintResult_InconclusiveIfEitherHalfIs(t *testing.T) { } } } + +// fpcalc 1.5.1 exits 3 with "(End of file)" on every file shorter than the +// span asked for. That print covers the whole file and is kept; a file +// whose declared length runs past the span stopped early, and is not. +func TestFpcalcReadToEnd(t *testing.T) { + eof := []byte("ERROR: Error decoding audio frame (End of file)") + out := func(sec int) []byte { return []byte(fmt.Sprintf("DURATION=%d\nFINGERPRINT=AQAB\n", sec)) } + for name, tc := range map[string]struct { + err error + out []byte + want bool + }{ + "short file read to its end": {&toolExitError{name: "fpcalc", code: 3, stderr: eof}, out(37), true}, + "exactly the span": {&toolExitError{name: "fpcalc", code: 3, stderr: eof}, out(120), true}, + "declared longer than read": {&toolExitError{name: "fpcalc", code: 3, stderr: eof}, out(150), false}, + "partial for another reason": {&toolExitError{name: "fpcalc", code: 3, stderr: []byte("ERROR: invalid data")}, out(37), false}, + "a different exit": {&toolExitError{name: "fpcalc", code: 2, stderr: eof}, out(37), false}, + "no duration printed": {&toolExitError{name: "fpcalc", code: 3, stderr: eof}, []byte("FINGERPRINT=AQAB\n"), false}, + "not a tool exit (a stall)": {fmt.Errorf("fpcalc: %w", errFingerprintTimeout), out(37), false}, + "wrapped exit is still judged": {fmt.Errorf("lookup: %w", &toolExitError{name: "fpcalc", code: 3, stderr: eof}), out(37), true}, + } { + if got := fpcalcReadToEnd(tc.err, tc.out, 120); got != tc.want { + t.Errorf("%s: fpcalcReadToEnd = %v, want %v", name, got, tc.want) + } + } +} + +// The output of a tool that exited non-zero reaches the caller, or +// fpcalcReadToEnd has nothing to read; the message is unchanged. +func TestRunFingerprintTool_KeepsOutputOnExit(t *testing.T) { + out, err := runFingerprintTool(context.Background(), "sh", + []string{"-c", "echo DURATION=37; echo 'ERROR: Error decoding audio frame (End of file)' >&2; exit 3"}) + if err == nil { + t.Fatal("exit 3 returned no error") + } + if want := "sh exited 3: ERROR: Error decoding audio frame (End of file)"; err.Error() != want { + t.Errorf("err = %q, want %q", err, want) + } + if d, ok := fpcalcDuration(out); !ok || d != 37 { + t.Errorf("output = %q, want the DURATION line kept", out) + } +} diff --git a/internal/library/genre.go b/internal/library/genre.go index 7981e026..e84b549f 100644 --- a/internal/library/genre.go +++ b/internal/library/genre.go @@ -33,7 +33,7 @@ func extractGenres(meta tag.Metadata, rs io.ReadSeeker) (genres []string, fellBa // dhowden/tag will have nothing either — not worth flagging. fellBack = meta.Genre() != "" case tag.VORBIS, tag.MP4: - // dhowden/tag keeps only the last of a repeated field here (#2500): + // dhowden/tag keeps only the last of a repeated field here: // see vorbisgenre.go and mp4genre.go. var values []string var err error @@ -84,9 +84,8 @@ func normaliseGenres(values []string) []string { // A value may be: // - plain text ("Alternative Rock") — passed through // - a bare ID3v1 index ("17") — resolved to "Rock". This is what the spec -// says a numeric TCON means, and what ffmpeg does. It is why the operator's -// library showed genres like "4017" and "526617": several numeric values -// welded together by the old reader. +// says a numeric TCON means, and what ffmpeg does. Several of them +// welded together read as genres like "4017". // - ID3v2.3 refinement syntax ("(17)", "(51)(39)", "(17)Hard Rock", "(RX)") // — each parenthesised index becomes its own genre, and trailing text // becomes one more. @@ -134,7 +133,8 @@ func normaliseGenreValue(v string) []string { } if n, err := strconv.Atoi(v); err == nil { if name, ok := id3v1GenreName(n); ok { - // Canonical table name, deliberately NOT re-cased — see id3v1Genres. + // Canonical table name, deliberately NOT re-cased — see + // id3v1Genres. return append(out, name) } return out @@ -143,8 +143,7 @@ func normaliseGenreValue(v string) []string { } // genreAcronyms are tokens that belong in caps. Tag editors that title-case the -// whole genre field turn "EDM" into "Edm" and "UK Garage" into "Uk Garage", and -// the operator's library carries all of these (#2468). +// whole genre field turn "EDM" into "Edm" and "UK Garage" into "Uk Garage". // // Matched case-INSENSITIVELY, so "edm", "Edm" and "EDM" all land on "EDM". That // is a deliberate, narrow exception to the project's rule that genre case is @@ -169,8 +168,8 @@ var genreAcronyms = map[string]string{ // how "Children's Music" became "Children'S Music". // // Deliberately a fixed list rather than "lowercase whatever follows an -// apostrophe": that broader rule would break "O'Brien" and "D'Angelo", which are -// correctly capitalised after the apostrophe. +// apostrophe": that broader rule would break "O'Brien" and "D'Angelo", which +// are correctly capitalised after the apostrophe. var contractionSuffixes = map[string]bool{ "s": true, "t": true, "re": true, "ll": true, "ve": true, "d": true, "m": true, } @@ -178,7 +177,7 @@ var contractionSuffixes = map[string]bool{ // trueUpCasing repairs casing damage done by tag editors that title-case the // genre field. It only ever changes case, never the letters — so it cannot // silently turn one genre into a different one, which is what separates it from -// the label-remapping idea #2468 rejected. +// remapping labels. func trueUpCasing(s string) string { if s == "" { return s diff --git a/internal/library/genre_test.go b/internal/library/genre_test.go index 9be5da86..66576036 100644 --- a/internal/library/genre_test.go +++ b/internal/library/genre_test.go @@ -67,7 +67,7 @@ func utf8Frame(values ...string) []byte { return append([]byte{0x03}, []byte(strings.Join(values, "\x00"))...) } -// TestReadID3v2GenreValues_MultiValue is the #2499 regression. dhowden/tag +// TestReadID3v2GenreValues_MultiValue: dhowden/tag // rejoins these values with the empty string, producing "Alternative RockRock"; // the whole point of our own reader is that they stay separate. func TestReadID3v2GenreValues_MultiValue(t *testing.T) { @@ -88,7 +88,7 @@ func TestReadID3v2GenreValues_MultiValue(t *testing.T) { } } -// The operator's worst case: eight values welded into one 70-character token. +// Eight values that a joining reader would weld into one 70-character token. func TestReadID3v2GenreValues_ManyValues(t *testing.T) { values := []string{ "Boom Bap", "Downtempo", "Hip Hop", "Instrumental", @@ -268,7 +268,7 @@ func TestNormaliseGenreValue(t *testing.T) { {"empty", "", nil}, {"whitespace only", " ", nil}, - // The operator's digit soup, one value at a time. + // Numeric references, one value at a time. {"bare numeric", "17", []string{"Rock"}}, {"bare numeric pop", "13", []string{"Pop"}}, {"bare numeric electronic", "52", []string{"Electronic"}}, @@ -334,7 +334,7 @@ func TestExtractGenres_EndToEnd(t *testing.T) { if err != nil { t.Fatalf("tag.ReadFrom: %v", err) } - // Confirm the upstream behaviour this fix exists for is still present — + // Confirm the upstream behaviour this reader works around is still present — // if dhowden ever fixes it, this test tells us the workaround can go. if welded := meta.Genre(); welded != "Alternative RockRock" { t.Logf("note: dhowden/tag no longer welds multi-values (got %q)", welded) @@ -426,7 +426,7 @@ func TestTrueUpCasing(t *testing.T) { in string want string }{ - // The damage actually present in the operator's library (#2468). + // Casing damage tag editors leave. {"edm acronym", "Edm", "EDM"}, {"idm acronym", "Idm", "IDM"}, {"aor acronym", "Aor", "AOR"}, diff --git a/internal/library/id3v2genre.go b/internal/library/id3v2genre.go index 7dbb15ce..49de328d 100644 --- a/internal/library/id3v2genre.go +++ b/internal/library/id3v2genre.go @@ -16,8 +16,8 @@ import ( // // — it splits on the ID3v2 null separator and rejoins with the EMPTY string, so // a file tagged "Alternative Rock" + "Rock" comes back as the single token -// "Alternative RockRock" (#2499). We stored that verbatim, which corrupted the -// genre browse axis and polluted the taste profile's tag vocabulary. +// "Alternative RockRock". Stored verbatim, that would corrupt the genre +// browse axis and pollute the taste profile's tag vocabulary. // // ffprobe is not an escape hatch either: ffmpeg's read_ttag calls decode_str // exactly once with no loop, so it keeps only the FIRST value and silently diff --git a/internal/library/leveled.go b/internal/library/leveled.go index b7abdd11..d13360f7 100644 --- a/internal/library/leveled.go +++ b/internal/library/leveled.go @@ -21,7 +21,7 @@ import ( "golang.org/x/sync/singleflight" ) -// Leveled streams (M464 #5001). +// Leveled streams. // // Sonos and other UPnP renderers fetch a track's URL themselves, so the phone // that sent them there cannot change what they play. For them the server diff --git a/internal/library/library_paths.go b/internal/library/library_paths.go new file mode 100644 index 00000000..fe0a9a5f --- /dev/null +++ b/internal/library/library_paths.go @@ -0,0 +1,231 @@ +package library + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "slices" + "strings" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +// The library's folders. They are set in Settings and changed without +// a restart; MINSTREL_LIBRARY_SCAN_PATHS names them until the first save. The +// Scanner holds the live list, and the watcher and the importer read it there. +// +// Changing a folder's path is a move, not a new library: when the files that +// were under /music are now under /data/music, every row under the old path is +// rewritten to the new one, so the tracks keep their ids and their history. +// The scan never sees the change as a mass deletion followed by a mass add. + +// LoadRoots is the folders the server starts with: the saved ones, or env +// when nothing has been saved. fromSettings says which. +func LoadRoots(ctx context.Context, pool *pgxpool.Pool, env []string) (roots []string, fromSettings bool, err error) { + row, err := dbq.New(pool).GetLibrarySettings(ctx) + if err != nil { + return env, false, fmt.Errorf("library settings: load: %w", err) + } + if len(row.ScanPaths) == 0 { + return env, false, nil + } + return row.ScanPaths, true, nil +} + +// RootChange is one folder of a save. Was is the folder it replaces, whose +// files have moved to Path; "" is a folder added. +type RootChange struct { + Path string + Was string +} + +// RerootResult is what a save rewrote. +type RerootResult struct { + Roots []string + Tracks int64 +} + +// RootsError is a save the folders refuse. Its text says why, in a sentence +// for an admin. +type RootsError struct{ msg string } + +func (e *RootsError) Error() string { return e.msg } + +func badRoots(format string, args ...any) error { + return &RootsError{msg: fmt.Sprintf(format, args...)} +} + +// moveSample is how many present tracks a move checks for at the new path, +// and moveSampleMinFound how many of them must be there. +const ( + moveSample = 20 + moveSampleMinFound = 0.5 +) + +// SetRoots validates the folders, rewrites the paths of any that moved, saves +// them and starts reading them. Nothing is written when any check fails. +func (s *Scanner) SetRoots(ctx context.Context, changes []RootChange) (RerootResult, error) { + s.rootsMu.Lock() + defer s.rootsMu.Unlock() + + roots, moves, err := planRoots(s.paths, changes) + if err != nil { + return RerootResult{}, err + } + q := dbq.New(s.pool) + // A scan walking the old folders would put their paths back, or mark the + // moved rows missing when it reconciles. + _, err = q.GetInFlightScanRun(ctx) + if err == nil { + return RerootResult{}, badRoots("A library scan is running; save again once it finishes.") + } + if !errors.Is(err, pgx.ErrNoRows) { + return RerootResult{}, fmt.Errorf("in-flight scan check: %w", err) + } + for _, m := range moves { + if err := checkMove(ctx, q, m); err != nil { + return RerootResult{}, err + } + } + + var res RerootResult + err = pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + tq := dbq.New(tx) + for _, m := range moves { + n, err := tq.RerootTrackPaths(ctx, dbq.RerootTrackPathsParams{NewRoot: m.Path, OldRoot: m.Was}) + if err != nil { + if isUniqueViolation(err) { + return badRoots("Some tracks under %s are already in the library at %s, "+ + "so its tracks cannot move there.", m.Was, m.Path) + } + return fmt.Errorf("reroot tracks: %w", err) + } + res.Tracks += n + if _, err := tq.RerootAlbumCoverPaths(ctx, dbq.RerootAlbumCoverPathsParams{NewRoot: m.Path, OldRoot: m.Was}); err != nil { + return fmt.Errorf("reroot covers: %w", err) + } + if _, err := tq.RerootGrabDestinations(ctx, dbq.RerootGrabDestinationsParams{NewRoot: m.Path, OldRoot: m.Was}); err != nil { + return fmt.Errorf("reroot downloads: %w", err) + } + if _, err := tq.RerootImportRoot(ctx, dbq.RerootImportRootParams{NewRoot: m.Path, OldRoot: m.Was}); err != nil { + return fmt.Errorf("reroot import folder: %w", err) + } + } + _, err := tq.UpdateLibraryScanPaths(ctx, roots) + return err + }) + if err != nil { + return RerootResult{}, err + } + for _, m := range moves { + s.logger.Info("library folder moved", "from", m.Was, "to", m.Path) + } + s.paths = roots + select { + case s.rootsChanged <- struct{}{}: + default: + } + res.Roots = slices.Clone(roots) + return res, nil +} + +// RootsChanged is signalled after a save changes the folders. A Scanner built +// without New never signals. +func (s *Scanner) RootsChanged() <-chan struct{} { return s.rootsChanged } + +// planRoots checks a save against the current folders and returns the new +// list and the folders that moved. +func planRoots(current []string, changes []RootChange) ([]string, []RootChange, error) { + if len(changes) == 0 { + return nil, nil, badRoots("Name at least one folder.") + } + roots := make([]string, 0, len(changes)) + var moves []RootChange + replaced := map[string]bool{} + for _, c := range changes { + p := strings.TrimSpace(c.Path) + if !filepath.IsAbs(p) { + return nil, nil, badRoots("%q is not a full path; it starts with /.", c.Path) + } + p = filepath.Clean(p) + if p == "/" { + return nil, nil, badRoots("The library cannot be the whole filesystem.") + } + if err := checkDir(p); err != nil { + return nil, nil, err + } + for _, r := range roots { + if r == p { + return nil, nil, badRoots("%s is listed twice.", p) + } + if within(p, r) || within(r, p) { + return nil, nil, badRoots("%s and %s overlap; name one of them.", r, p) + } + } + roots = append(roots, p) + if c.Was == "" || c.Was == p { + continue + } + if !slices.Contains(current, c.Was) { + return nil, nil, badRoots("%s is not one of the library's folders.", c.Was) + } + if replaced[c.Was] { + return nil, nil, badRoots("%s can only move to one place.", c.Was) + } + replaced[c.Was] = true + moves = append(moves, RootChange{Path: p, Was: c.Was}) + } + return roots, moves, nil +} + +// checkDir refuses a folder the server cannot read: a typo, or a mount the +// container does not have. +func checkDir(p string) error { + info, err := os.Stat(p) + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return badRoots("%s does not exist on the server. Is it mounted into the container?", p) + } + return badRoots("%s cannot be read: %v.", p, err) + } + if !info.IsDir() { + return badRoots("%s is a file, not a folder.", p) + } + if _, err := os.ReadDir(p); err != nil { + return badRoots("%s cannot be read: %v.", p, err) + } + return nil +} + +// checkMove looks for a sample of the folder's tracks at the new path, so a +// move to the wrong folder is refused before any row is rewritten. +func checkMove(ctx context.Context, q *dbq.Queries, m RootChange) error { + sample, err := q.SampleTrackPathsUnder(ctx, dbq.SampleTrackPathsUnderParams{Root: m.Was, N: moveSample}) + if err != nil { + return fmt.Errorf("sample tracks: %w", err) + } + if len(sample) == 0 { + return nil + } + found := 0 + for _, p := range sample { + if _, err := os.Stat(m.Path + strings.TrimPrefix(p, m.Was)); err == nil { + found++ + } + } + if float64(found) < moveSampleMinFound*float64(len(sample)) { + return badRoots("Only %d of %d tracks checked from %s are at the same place under %s. "+ + "Is that the folder they moved to?", found, len(sample), m.Was, m.Path) + } + return nil +} + +// within reports whether p is inside dir. +func within(p, dir string) bool { + return strings.HasPrefix(p, dir+"/") +} diff --git a/internal/library/library_paths_test.go b/internal/library/library_paths_test.go new file mode 100644 index 00000000..aa3cd5e2 --- /dev/null +++ b/internal/library/library_paths_test.go @@ -0,0 +1,148 @@ +package library + +import ( + "context" + "errors" + "io" + "log/slog" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +func mkdir(t *testing.T, parts ...string) string { + t.Helper() + p := filepath.Join(parts...) + if err := os.MkdirAll(p, 0o755); err != nil { + t.Fatal(err) + } + return p +} + +func TestPlanRoots(t *testing.T) { + base := t.TempDir() + music := mkdir(t, base, "music") + data := mkdir(t, base, "data", "music") + more := mkdir(t, base, "more") + file := filepath.Join(base, "a.mp3") + if err := os.WriteFile(file, []byte("x"), 0o600); err != nil { + t.Fatal(err) + } + current := []string{music, more} + + roots, moves, err := planRoots(current, []RootChange{{Path: data + "/", Was: music}, {Path: more, Was: more}}) + if err != nil { + t.Fatal(err) + } + if !slices.Equal(roots, []string{data, more}) { + t.Errorf("roots %v", roots) + } + if len(moves) != 1 || moves[0] != (RootChange{Path: data, Was: music}) { + t.Errorf("moves %v; want only %s → %s", moves, music, data) + } + + for name, tc := range map[string]struct { + changes []RootChange + mentions string + }{ + "none": {nil, "at least one"}, + "relative": {[]RootChange{{Path: "music"}}, "full path"}, + "root": {[]RootChange{{Path: "/"}}, "whole filesystem"}, + "absent": {[]RootChange{{Path: filepath.Join(base, "nope")}}, "does not exist"}, + "a file": {[]RootChange{{Path: file}}, "not a folder"}, + "twice": {[]RootChange{{Path: music}, {Path: music}}, "twice"}, + "nested": {[]RootChange{{Path: base}, {Path: music}}, "overlap"}, + "unknown was": {[]RootChange{{Path: data, Was: "/elsewhere"}}, "not one of"}, + "split a move": {[]RootChange{{Path: data, Was: music}, {Path: more, Was: music}}, "one place"}, + } { + _, _, err := planRoots(current, tc.changes) + var re *RootsError + if !errors.As(err, &re) || !strings.Contains(err.Error(), tc.mentions) { + t.Errorf("%s: err %v; want a RootsError mentioning %q", name, err, tc.mentions) + } + } +} + +// clearSavedRoots puts the folders back to env afterwards. It also finishes any +// scan an earlier test left running: dbtest.ResetDB keeps scan_runs, and a +// save refuses while a scan is in flight. +func clearSavedRoots(t *testing.T, pool *pgxpool.Pool) { + t.Helper() + if _, err := pool.Exec(context.Background(), + `UPDATE scan_runs SET finished_at = now() WHERE finished_at IS NULL`); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _, _ = pool.Exec(context.Background(), `UPDATE library_settings SET scan_paths = NULL`) + }) +} + +func TestSetRoots_TracksMoveWithTheirFolder_Integration(t *testing.T) { + pool := newPool(t) + clearSavedRoots(t, pool) + ctx := context.Background() + base := t.TempDir() + old := filepath.Join(base, "music") // gone: the mount moved + moved := mkdir(t, base, "data", "music", "Artist") + if err := os.WriteFile(filepath.Join(moved, "a.mp3"), []byte("x"), 0o600); err != nil { + t.Fatal(err) + } + newRoot := filepath.Dir(moved) + track, _, _ := seedTrack(t, pool, filepath.Join(old, "Artist", "a.mp3")) + + s := New(pool, slog.New(slog.NewTextHandler(io.Discard, nil)), []string{old}, nil) + res, err := s.SetRoots(ctx, []RootChange{{Path: newRoot, Was: old}}) + if err != nil { + t.Fatal(err) + } + if res.Tracks != 1 { + t.Errorf("moved %d tracks; want 1", res.Tracks) + } + got, err := dbq.New(pool).GetTrackByPath(ctx, filepath.Join(newRoot, "Artist", "a.mp3")) + if err != nil { + t.Fatalf("the track is not at its new path: %v", err) + } + if got.ID != track.ID { + t.Error("the move made a new track instead of keeping the old one") + } + if !slices.Equal(s.Roots(), []string{newRoot}) { + t.Errorf("roots %v", s.Roots()) + } + select { + case <-s.RootsChanged(): + default: + t.Error("the watcher was not told the folders changed") + } + roots, saved, err := LoadRoots(ctx, pool, []string{old}) + if err != nil || !saved || !slices.Equal(roots, []string{newRoot}) { + t.Errorf("after a restart: %v saved=%v err=%v; want the saved folder over env", roots, saved, err) + } +} + +func TestSetRoots_RefusesAMoveToAFolderWithoutTheFiles_Integration(t *testing.T) { + pool := newPool(t) + clearSavedRoots(t, pool) + ctx := context.Background() + base := t.TempDir() + old := filepath.Join(base, "music") + wrong := mkdir(t, base, "elsewhere") + seedTrack(t, pool, filepath.Join(old, "a.mp3")) + + s := New(pool, slog.New(slog.NewTextHandler(io.Discard, nil)), []string{old}, nil) + var re *RootsError + if _, err := s.SetRoots(ctx, []RootChange{{Path: wrong, Was: old}}); !errors.As(err, &re) { + t.Fatalf("err %v; want a RootsError", err) + } + if _, err := dbq.New(pool).GetTrackByPath(ctx, filepath.Join(old, "a.mp3")); err != nil { + t.Errorf("a refused move rewrote the track: %v", err) + } + if !slices.Equal(s.Roots(), []string{old}) { + t.Errorf("a refused move changed the folders to %v", s.Roots()) + } +} diff --git a/internal/library/loudness.go b/internal/library/loudness.go index 78893553..ac6c8daa 100644 --- a/internal/library/loudness.go +++ b/internal/library/loudness.go @@ -20,7 +20,7 @@ import ( syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) -// Loudness analysis (M464 #4995). +// Loudness analysis. // // Every track is measured with ffmpeg's EBU R128 filter, which reports the // ITU-R BS.1770 values loudness normalization works from: @@ -39,7 +39,7 @@ import ( // 100 ms). Those are kept as a histogram, because an album's loudness is the // gated loudness of every block on the album, not an average of its tracks' // values. With the histograms stored, album loudness is exact and needs no -// second decode (#4996). +// second decode. // loudnessVersion stamps how a track_loudness row was measured. Bump it when // the measurement changes (the filter's options, the histogram's bins) and the @@ -136,7 +136,7 @@ func (h blockHistogram) empty() bool { return len(h.counts) == 0 } // loudness of the blocks that passed it. ok is false when nothing passes. // // The same function measures an album, over the sum of its tracks' -// histograms (#4996). +// histograms. func (h blockHistogram) gatedLoudness() (lufs float64, ok bool) { energy := func(i int) float64 { l := loudnessHistFloor + float64(int(h.start)+i)/loudnessHistPerLU @@ -354,7 +354,7 @@ const ( // value from an older method is a better gain than none until it is redone. // // A stored measurement is logged to the sync feed as a track upsert, so -// clients that cache the library (Android) pick up the new gain (#4997). If +// clients that cache the library (Android) pick up the new gain. If // the log fails nothing is stored, and the backfill tries the track again. func storeLoudness( ctx context.Context, db dbq.DBTX, logger *slog.Logger, @@ -394,7 +394,7 @@ func storeLoudness( } } } - // Logged before the write, as reconcile does (#2704): a log that then + // Logged before the write, as reconcile does: a log that then // finds the write failed costs clients one wasted re-read, but a write // that lands with no log is never retried, since the track is settled, and // clients would never learn its gain. diff --git a/internal/library/loudness_backfill.go b/internal/library/loudness_backfill.go index 3379c973..afcae46f 100644 --- a/internal/library/loudness_backfill.go +++ b/internal/library/loudness_backfill.go @@ -13,15 +13,14 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Loudness backfill (M464 #4995). +// Loudness backfill. // // Every track is measured here, the new ones included: the scan only deletes a // changed file's measurement (see scanFile), and this worker measures it again. -// Measuring inline in the scan was the first plan and was dropped, because the -// analysis decodes the whole file. Added to the scan, a large import would run -// several times longer and could pass StuckScanThreshold (1h), at which point -// the run is reaped and a second scan started beside it. The fingerprint -// backfill is a worker of its own for the same reason. +// Not inline in the scan, because the analysis decodes the whole file: a large +// import would run several times longer and could pass StuckScanThreshold +// (1h), at which point the run is reaped and a second scan started beside it. +// The fingerprint backfill is a worker of its own for the same reason. // // Until a track is measured it plays with no gain adjustment, which is how // every track played before normalization existed. @@ -107,7 +106,7 @@ func (w *LoudnessBackfillWorker) Run(ctx context.Context) { } // runOnce contains a pass so that nothing it does (an error, a panic) can stop -// the next tick from firing (rule 157). +// the next tick from firing. func (w *LoudnessBackfillWorker) runOnce(ctx context.Context) { defer func() { if r := recover(); r != nil { @@ -123,7 +122,7 @@ func (w *LoudnessBackfillWorker) runOnce(ctx context.Context) { "processed", res.Processed, "measured", res.Measured, "silent", res.Silent, "unreadable", res.Unreadable, "inconclusive", res.Inconclusive) } - // Album loudness follows the tracks (#4996). It runs even with analysis + // Album loudness follows the tracks. It runs even with analysis // switched off: it decodes nothing, and membership still changes as the // library does. albums, err := w.albumPass(ctx) diff --git a/internal/library/loudness_backfill_test.go b/internal/library/loudness_backfill_test.go index 88451475..b650f733 100644 --- a/internal/library/loudness_backfill_test.go +++ b/internal/library/loudness_backfill_test.go @@ -132,7 +132,7 @@ func TestLoudnessBackfill_Integration(t *testing.T) { } // Each stored measurement told the sync feed, so caching clients re-read - // the track and pick up its gain (#4997). + // the track and pick up its gain. var logged int if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_changes WHERE entity_type = 'track' AND op = 'upsert' AND entity_id = $1`, diff --git a/internal/library/loudness_settings.go b/internal/library/loudness_settings.go index 7f6f73d8..21fe1bad 100644 --- a/internal/library/loudness_settings.go +++ b/internal/library/loudness_settings.go @@ -12,9 +12,9 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Loudness analysis settings (M464 #4995). Rule 25: an operator setting is a -// database row, changed without a restart. One instance is shared by the -// backfill and the admin API, so a save reaches the worker at once. +// Loudness analysis settings: a database row, changed without a restart. One +// instance is shared by the backfill and the admin API, so a save reaches the +// worker at once. // LoudnessSettings mirrors the loudness_settings row. type LoudnessSettings struct { @@ -23,14 +23,15 @@ type LoudnessSettings struct { Enabled bool BackfillConcurrency int32 // LeveledCacheMB caps the disk the leveled-stream renders for Sonos and - // UPnP speakers may use (#5001); the least recently played go first. + // UPnP speakers may use; the least recently played go first. LeveledCacheMB int32 // UpdatedAt is set by the database; ignored by Set. UpdatedAt time.Time } -// DefaultLoudnessSettings mirrors migrations 0065 and 0068's column defaults, so a -// database that cannot be read still analyzes the way a fresh install does. +// DefaultLoudnessSettings mirrors migrations 0065 and 0068's column defaults, +// so a database that cannot be read still analyzes the way a fresh install +// does. var DefaultLoudnessSettings = LoudnessSettings{ Enabled: true, BackfillConcurrency: loudnessBackfillConcurrency, diff --git a/internal/library/loudness_test.go b/internal/library/loudness_test.go index 6d4ac4da..8e2c9c03 100644 --- a/internal/library/loudness_test.go +++ b/internal/library/loudness_test.go @@ -65,7 +65,7 @@ func TestEbur128Parser_RealOutput(t *testing.T) { t.Errorf("histogram not trimmed to its occupied bins: %v", r.hist.counts) } - // Album loudness is computed from these histograms (#4996), so the + // Album loudness is computed from these histograms, so the // histogram has to reproduce ffmpeg's own figure. got, ok := r.hist.gatedLoudness() if !ok || math.Abs(got-(-10.7)) > 0.05 { diff --git a/internal/library/mbidbackfill.go b/internal/library/mbidbackfill.go index 5d79ff96..2ebffced 100644 --- a/internal/library/mbidbackfill.go +++ b/internal/library/mbidbackfill.go @@ -18,9 +18,9 @@ import ( const pgUniqueViolation = "23505" // isUniqueViolation reports whether err is a Postgres unique-constraint -// conflict. Used to distinguish "another row already owns this MBID" -// (a duplicate album/artist row in our DB, the operator should merge -// it) from genuine write failures. +// conflict. Used to distinguish "another row already owns this MBID" (a +// duplicate album/artist row in our DB, for an admin to merge) from genuine +// write failures. func isUniqueViolation(err error) bool { var pgErr *pgconn.PgError return errors.As(err, &pgErr) && pgErr.Code == pgUniqueViolation @@ -31,7 +31,7 @@ type BackfillMBIDsResult struct { Processed int // Albums considered (mbid IS NULL). Healed int // Albums whose mbid we set. Skipped int // Albums where the file was missing or had no MBID tag. - Duplicates int // Albums whose mbid is already held by another row (operator-resolvable). + Duplicates int // Albums whose mbid another row already holds; an admin merges. } // BackfillMBIDs walks albums with NULL mbid, opens one track per album, @@ -165,7 +165,8 @@ func readMBIDsForFile(path string, logger *slog.Logger) (albumMBID, artistMBID s logger.Warn("mbid backfill: tag read failed", "path", path, "err", err) return "", "" } - return extractMBIDs(meta) + ids := ReadTagMBIDs(meta) + return ids.Release, ids.Artist } // BackfillTrackMBIDsResult tallies the track recording-MBID backfill. @@ -258,5 +259,5 @@ func readRecordingMBIDForFile(path string, logger *slog.Logger) string { logger.Warn("track mbid backfill: tag read failed", "path", path, "err", err) return "" } - return extractRecordingMBID(meta) + return ReadTagMBIDs(meta).Recording } diff --git a/internal/library/mbidbackfill_test.go b/internal/library/mbidbackfill_test.go index a645abe4..990d6e45 100644 --- a/internal/library/mbidbackfill_test.go +++ b/internal/library/mbidbackfill_test.go @@ -13,8 +13,8 @@ import ( ) // The MBID backfills read tags from files, so a track whose file is gone has -// nothing to give them. Before #5139 they opened it anyway, and every scan -// logged an "open failed" warning per missing track. +// nothing to give them: they skip it rather than log an "open failed" warning +// per missing track on every scan. func TestMBIDBackfill_SkipsMissingTracks_Integration(t *testing.T) { pool := newPool(t) ctx := context.Background() diff --git a/internal/library/mbids.go b/internal/library/mbids.go index e63550b7..85bad411 100644 --- a/internal/library/mbids.go +++ b/internal/library/mbids.go @@ -7,54 +7,56 @@ import ( "github.com/dhowden/tag/mbz" ) -// extractMBIDs reads MusicBrainz album and artist IDs from a tag.Metadata. +// TagMBIDs is what a file's MusicBrainz Picard tags name. Empty means the +// tag is absent. +type TagMBIDs struct { + // Release is musicbrainz_albumid: one edition, what albums.mbid stores. + Release string + Artist string + // ReleaseGroup is musicbrainz_releasegroupid. Wants name albums by + // release group, so re-acquisition and request completion key on it. + ReleaseGroup string + // Recording is what tracks.mbid stores and the ListenBrainz Labs + // similar-recordings API keys on. + Recording string +} + +// ReadTagMBIDs reads a file's MusicBrainz IDs, for the scanner, the MBID +// backfills and the importer alike. // // Delegates to dhowden/tag/mbz, which knows the per-format Raw() conventions: -// - Vorbis (FLAC/OGG) keys are lowercased by the parser. +// - Vorbis (FLAC/OGG/Opus) keys are lowercased by the parser. // - ID3v2 (MP3) TXXX frames are stored under "TXXX"/"TXXX_N" with // a *tag.Comm value whose Description holds the Picard tag name. // - MP4 (M4A) freeform iTunes atoms are stored under their bare sub-name. // -// Then sanitizes: dhowden's readTextWithDescrFrame (used for ID3v2 TXXX) -// does NOT strip the trailing/embedded NUL bytes that ID3v2.4 uses as -// frame terminator and multi-value separator — so the Text often comes -// out as "uuid\x00" or "uuid1\x00uuid2". Postgres rejects NULs in text -// columns (SQLSTATE 22021). We split on NUL and take the first non-empty -// segment — Minstrel uses the primary release artist for MBCAA lookup, -// so first-wins matches the original semantics. -func extractMBIDs(m tag.Metadata) (albumMBID, artistMBID string) { +// mbz maps keys by name alone, and Picard names the recording differently +// in Vorbis: MUSICBRAINZ_TRACKID there is the recording (the release track +// is MUSICBRAINZ_RELEASETRACKID), so mbz files it under mbz.Track, its name +// for the release track. Read mbz.Recording alone and every FLAC has no +// recording id. +func ReadTagMBIDs(m tag.Metadata) TagMBIDs { info := mbz.Extract(m) - return cleanMBID(info.Get(mbz.Album)), cleanMBID(info.Get(mbz.Artist)) -} - -// extractRecordingMBID reads the MusicBrainz *recording* ID — Picard's -// musicbrainz_recordingid, surfaced by dhowden/tag as mbz.Recording -// (tag display name "MusicBrainz Track Id"). This is the id the -// ListenBrainz Labs similar-recordings API keys on and what tracks.mbid -// stores. -// -// Deliberately NOT mbz.Track ("MusicBrainz Release Track Id"): that is -// per-release-track, whereas similarity is per-recording. Kept separate -// from extractMBIDs so its existing (album, artist) signature and unit -// tests stay untouched. -func extractRecordingMBID(m tag.Metadata) string { - return cleanMBID(mbz.Extract(m).Get(mbz.Recording)) -} - -// extractReleaseGroupMBID reads the MusicBrainz *release-group* ID — -// Picard's musicbrainz_releasegroupid, surfaced by dhowden/tag as -// mbz.ReleaseGroup. mbz.Album (what albums.mbid stores) is the release: one -// edition. Lidarr names albums by release group, so re-acquisition and -// request completion key on this one (#5241). Separate from extractMBIDs for -// the same reason extractRecordingMBID is. -func extractReleaseGroupMBID(m tag.Metadata) string { - return cleanMBID(mbz.Extract(m).Get(mbz.ReleaseGroup)) -} - -func cleanMBID(s string) string { - if s == "" { - return "" + ids := TagMBIDs{ + Release: cleanMBID(info.Get(mbz.Album)), + Artist: cleanMBID(info.Get(mbz.Artist)), + ReleaseGroup: cleanMBID(info.Get(mbz.ReleaseGroup)), + Recording: cleanMBID(info.Get(mbz.Recording)), } + if ids.Recording == "" && m.Format() == tag.VORBIS { + ids.Recording = cleanMBID(info.Get(mbz.Track)) + } + return ids +} + +// cleanMBID takes the first non-empty NUL-separated value. dhowden's +// readTextWithDescrFrame (used for ID3v2 TXXX) does NOT strip the +// trailing/embedded NUL bytes that ID3v2.4 uses as frame terminator and +// multi-value separator — so the Text often comes out as "uuid\x00" or +// "uuid1\x00uuid2". Postgres rejects NULs in text columns (SQLSTATE 22021). +// Minstrel uses the primary release artist for MBCAA lookup, so first-wins +// matches the original semantics. +func cleanMBID(s string) string { for _, part := range strings.Split(s, "\x00") { if t := strings.TrimSpace(part); t != "" { return t diff --git a/internal/library/mbids_test.go b/internal/library/mbids_test.go index db5225dc..7f3471d4 100644 --- a/internal/library/mbids_test.go +++ b/internal/library/mbids_test.go @@ -40,7 +40,8 @@ func TestExtractMBIDs_Vorbis(t *testing.T) { "musicbrainz_artistid": "def", }, } - a, ar := extractMBIDs(m) + ids := ReadTagMBIDs(m) + a, ar := ids.Release, ids.Artist if a != "abc" || ar != "def" { t.Errorf("got (%q, %q), want (\"abc\", \"def\")", a, ar) } @@ -62,7 +63,8 @@ func TestExtractMBIDs_ID3v2(t *testing.T) { }, }, } - a, ar := extractMBIDs(m) + ids := ReadTagMBIDs(m) + a, ar := ids.Release, ids.Artist if a != "abc-123" || ar != "def-456" { t.Errorf("got (%q, %q), want (\"abc-123\", \"def-456\")", a, ar) } @@ -78,7 +80,8 @@ func TestExtractMBIDs_MP4(t *testing.T) { "MusicBrainz Artist Id": "def", }, } - a, ar := extractMBIDs(m) + ids := ReadTagMBIDs(m) + a, ar := ids.Release, ids.Artist if a != "abc" || ar != "def" { t.Errorf("got (%q, %q), want (\"abc\", \"def\")", a, ar) } @@ -86,17 +89,17 @@ func TestExtractMBIDs_MP4(t *testing.T) { func TestExtractMBIDs_MissingTagsReturnEmpty(t *testing.T) { m := stubMeta{format: tag.VORBIS, raw: map[string]interface{}{}} - a, ar := extractMBIDs(m) + ids := ReadTagMBIDs(m) + a, ar := ids.Release, ids.Artist if a != "" || ar != "" { t.Errorf("got (%q, %q), want empty", a, ar) } } func TestExtractMBIDs_VorbisUppercaseKeysMiss(t *testing.T) { - // Regression guard: real dhowden never produces uppercase Vorbis - // keys. If a future change starts looking up "MUSICBRAINZ_ALBUMID" - // directly, this test must keep returning empty so we don't - // silently re-introduce the v0 bug where every album skipped. + // Real dhowden never produces uppercase Vorbis keys. Looking up + // "MUSICBRAINZ_ALBUMID" directly would find nothing, and every album + // would be skipped; this test keeps that lookup returning empty. m := stubMeta{ format: tag.VORBIS, raw: map[string]interface{}{ @@ -104,7 +107,8 @@ func TestExtractMBIDs_VorbisUppercaseKeysMiss(t *testing.T) { "MUSICBRAINZ_ARTISTID": "wrong-case", }, } - a, ar := extractMBIDs(m) + ids := ReadTagMBIDs(m) + a, ar := ids.Release, ids.Artist if a != "" || ar != "" { t.Errorf("uppercase Vorbis keys must not match dhowden output; got (%q, %q)", a, ar) } @@ -127,7 +131,8 @@ func TestExtractMBIDs_ID3v2StripsTrailingNUL(t *testing.T) { }, }, } - a, ar := extractMBIDs(m) + ids := ReadTagMBIDs(m) + a, ar := ids.Release, ids.Artist if a != "abc-123" || ar != "def-456" { t.Errorf("got (%q, %q), want trailing NUL stripped", a, ar) } @@ -146,7 +151,7 @@ func TestExtractMBIDs_ID3v2MultiValueTakesFirst(t *testing.T) { }, }, } - _, ar := extractMBIDs(m) + ar := ReadTagMBIDs(m).Artist if ar != "primary-mbid" { t.Errorf("got %q, want %q", ar, "primary-mbid") } @@ -164,7 +169,7 @@ func TestCleanMBID_OnlyNULReturnsEmpty(t *testing.T) { } } -// #5241: the release-group id is read beside the release id, from each +// The release-group id is read beside the release id, from each // format's own key, and the two are never confused. func TestExtractReleaseGroupMBID(t *testing.T) { cases := []struct { @@ -186,17 +191,71 @@ func TestExtractReleaseGroupMBID(t *testing.T) { } for _, c := range cases { t.Run(c.name, func(t *testing.T) { - if got := extractReleaseGroupMBID(c.meta); got != "group-1" { + if got := ReadTagMBIDs(c.meta).ReleaseGroup; got != "group-1" { t.Errorf("release group = %q, want group-1", got) } - if album, _ := extractMBIDs(c.meta); album != "release-1" { + if album := ReadTagMBIDs(c.meta).Release; album != "release-1" { t.Errorf("album (release) = %q, want release-1", album) } }) } none := stubMeta{format: tag.VORBIS, raw: map[string]interface{}{"musicbrainz_albumid": "release-1"}} - if got := extractReleaseGroupMBID(none); got != "" { + if got := ReadTagMBIDs(none).ReleaseGroup; got != "" { t.Errorf("untagged release group = %q, want empty", got) } } + +// Each format's recording id lands in Recording, and the release-track +// id never does. Picard calls the recording MUSICBRAINZ_TRACKID in Vorbis +// comments, the name mbz gives the release track. +func TestReadTagMBIDs_Recording(t *testing.T) { + cases := []struct { + name string + meta stubMeta + }{ + {"vorbis", stubMeta{format: tag.VORBIS, raw: map[string]interface{}{ + "musicbrainz_trackid": "rec-1", + "musicbrainz_releasetrackid": "reltrack-1", + }}}, + {"id3v2 ufid", stubMeta{format: tag.ID3v2_4, raw: map[string]interface{}{ + "UFID": &tag.UFID{Provider: "http://musicbrainz.org", Identifier: []byte("rec-1")}, + "TXXX_0": &tag.Comm{Description: "MusicBrainz Release Track Id", Text: "reltrack-1\x00"}, + }}}, + {"mp4", stubMeta{format: tag.MP4, raw: map[string]interface{}{ + "MusicBrainz Track Id": "rec-1", + "MusicBrainz Release Track Id": "reltrack-1", + }}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := ReadTagMBIDs(c.meta).Recording; got != "rec-1" { + t.Errorf("recording = %q, want rec-1", got) + } + }) + } + + // A Vorbis file carrying only the release-track id has no recording id. + only := stubMeta{format: tag.VORBIS, raw: map[string]interface{}{"musicbrainz_releasetrackid": "reltrack-1"}} + if got := ReadTagMBIDs(only).Recording; got != "" { + t.Errorf("release track only: recording = %q, want empty", got) + } +} + +// A Vorbis TRACKNUMBER of "3/20" is track 3: dhowden/tag reads it as 0. +func TestReadTrackAndDisc(t *testing.T) { + for _, tc := range []struct { + name string + m stubMeta + track, disc int + }{ + {"vorbis with counts", stubMeta{format: tag.VORBIS, raw: map[string]interface{}{"tracknumber": "3/20", "discnumber": "2/2"}}, 3, 2}, + {"vorbis bare", stubMeta{format: tag.VORBIS, raw: map[string]interface{}{"tracknumber": " 7 "}}, 7, 0}, + {"vorbis junk", stubMeta{format: tag.VORBIS, raw: map[string]interface{}{"tracknumber": "A1", "discnumber": "-1"}}, 0, 0}, + {"id3 is the library's own reading", stubMeta{format: tag.ID3v2_4, raw: map[string]interface{}{"tracknumber": "3/20"}}, 0, 0}, + } { + if track, disc := ReadTrackAndDisc(tc.m); track != tc.track || disc != tc.disc { + t.Errorf("%s: got track %d disc %d, want %d and %d", tc.name, track, disc, tc.track, tc.disc) + } + } +} diff --git a/internal/library/missing_pairs.go b/internal/library/missing_pairs.go index 9d91d624..bcde5643 100644 --- a/internal/library/missing_pairs.go +++ b/internal/library/missing_pairs.go @@ -13,30 +13,29 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Missing-pair folding (M485). +// Missing-pair folding. // // A track marked missing whose replacement already sits on disk under another // row is folded into that replacement: its likes, plays, playlist entries and -// tags move across and the missing row goes. On the operator's library this was -// 355 of 451 missing tracks (#5274), nearly all Lidarr quality upgrades — the -// mp3 removed, a flac imported at a new path. +// tags move across and the missing row goes. The common case is a quality +// upgrade — the mp3 removed, a flac imported at a new path. // // Move adoption (moved.go) can't catch these. It links a new file to a missing // row when the file is first inserted, matching on the recording MBID or the -// audio hash. A re-encode never shares the hash, and the replacements arrived -// untagged, so there was nothing to match; AcoustID filled many of the MBIDs -// weeks later, by which time both rows existed. Adoption keeps the old row and -// re-points its path, which only works while the new file has no row of its own. -// Here both do, so this folds instead, with the duplicate merge's row half. +// audio hash. A re-encode never shares the hash, and an untagged replacement +// has nothing to match until AcoustID fills its MBID, by which time both rows +// exist. Adoption keeps the old row and re-points its path, which only works +// while the new file has no row of its own. Here both do, so this folds +// instead, with the duplicate merge's row half. // -// The rules are in ListMissingTrackPairs. Only a pair where each side has exactly -// one candidate is folded: a wrong fold can't be undone, a skipped one is only a -// row left in the missing list. +// The rules are in ListMissingTrackPairs. Only a pair where each side has +// exactly one candidate is folded: a wrong fold can't be undone, a skipped one +// is only a row left in the missing list. // -// Unlike the duplicate merge this runs without review (operator, 2026-10-07): -// no file is touched, the missing row has nothing to play, and adoption, its -// closest relative, is automatic too. Each fold is logged with both paths so a -// wrong pair can be read back. +// Unlike the duplicate merge this runs without review: no file is touched, the +// missing row has nothing to play, and adoption, its closest relative, is +// automatic too. Each fold is logged with both paths so a wrong pair can be +// read back. // PairResult tallies one pass. type PairResult struct { @@ -112,8 +111,8 @@ func PairMissingTracks(ctx context.Context, pool *pgxpool.Pool, logger *slog.Log return res, nil } -// foldMissingPair folds one pair in its own transaction, after locking both rows -// and checking the pair still holds. +// foldMissingPair folds one pair in its own transaction, after locking both +// rows and checking the pair still holds. func foldMissingPair(ctx context.Context, pool *pgxpool.Pool, missingID, presentID pgtype.UUID) (foldResult, error) { var f foldResult err := pgx.BeginFunc(ctx, pool, func(tx pgx.Tx) error { diff --git a/internal/library/missing_pairs_test.go b/internal/library/missing_pairs_test.go index 01126f9f..7e2b24fc 100644 --- a/internal/library/missing_pairs_test.go +++ b/internal/library/missing_pairs_test.go @@ -73,7 +73,8 @@ func (f pairFixture) album(title, releaseGroup string) dbq.Album { return a } -// track adds a row; mbid "" leaves it without one, and missing marks it missing. +// track adds a row; mbid "" leaves it without one, and missing marks it +// missing. func (f pairFixture) track(album dbq.Album, title, path, mbid string, missing bool) dbq.Track { f.t.Helper() tr, err := f.q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ @@ -110,7 +111,7 @@ func (f pairFixture) count(sql string, args ...any) int { return n } -// The M485 proof on rows: each rule folds the pair it should, history moves onto +// On rows: each rule folds the pair it should, history moves onto // the replacement, and every shape the pass must leave alone is left alone. func TestPairMissingTracks_Integration(t *testing.T) { pool := newPool(t) @@ -126,13 +127,13 @@ func TestPairMissingTracks_Integration(t *testing.T) { edition := f.album("Pair Album (Deluxe)", "rg-pair") // same release group as albumA other := f.album("Other Album", "rg-other") - // Rule title: the mp3 went, the flac came, no MBID on either, and the title's - // case differs as it does between old and new tags. + // Rule title: the mp3 went, the flac came, no MBID on either, and the + // title's case differs as it does between old and new tags. oldMp3 := f.track(albumA, "Song One", "/m/a/01 - Song One.mp3", "", true) newFlac := f.track(albumA, "song one", "/m/a/01 - song one.flac", "", false) - // Rule mbid: the same recording, retitled and filed under another edition of - // the same release group. + // Rule mbid: the same recording, retitled and filed under another edition + // of the same release group. oldByMbid := f.track(albumA, "Old Name", "/m/a/02 - Old Name.mp3", "rec-2", true) newByMbid := f.track(edition, "New Name", "/m/d/02 - New Name.flac", "rec-2", false) @@ -231,10 +232,9 @@ func TestPairMissingTracks_Integration(t *testing.T) { } } -// The case that made M485: a Lidarr quality upgrade. The mp3 is removed and a -// re-encode of it arrives at a new path with no recording MBID, so move adoption -// has nothing to match. The scan must still end with one row, carrying the old -// row's history. +// The common case: a quality upgrade. The mp3 is removed and a re-encode of it +// arrives at a new path with no recording MBID, so move adoption has nothing to +// match. The scan must still end with one row, carrying the old row's history. func TestScanner_FoldsUpgradedFile_Integration(t *testing.T) { pool := newPool(t) ctx := context.Background() diff --git a/internal/library/moved.go b/internal/library/moved.go index eb453ccd..55b5eb99 100644 --- a/internal/library/moved.go +++ b/internal/library/moved.go @@ -9,48 +9,46 @@ import ( syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) -// Move detection (#2528). +// Move detection. // // Track identity is file_path: UpsertTrack conflicts on it, and the reconcile // pass in reconcile.go clears a missing mark when the walk sees that same path // again. So a file that comes back exactly where it was restores cleanly, but a -// file that comes back RENAMED or in a different directory looked, to the +// file that comes back RENAMED or in a different directory looks, to the // scanner, like a deletion plus an unrelated new track: // -// - the old row stayed marked missing, holding the like and every play_event -// - a fresh row appeared with no history -// - nothing connected them +// - the old row stays marked missing, holding the like and every play_event +// - a fresh row appears with no history +// - nothing connects them // -// A liked song read as unliked after a retag, its play count reset to zero, and -// Rediscover could offer it as a discovery. All silently. Renumbering an album -// was enough to do it — which is exactly what happened on the operator's copy of -// Minutes to Midnight. +// A liked song would read as unliked after a retag, its play count reset to +// zero, and Rediscover could offer it as a discovery. Renumbering an album is +// enough to do it. // -// The fix adopts the existing row rather than inserting: re-point its file_path -// at the new location and clear the mark. The caller's normal UpsertTrack then -// conflicts on file_path and updates THAT row, so the track id survives and -// likes, plays and playlist memberships travel with it. Clients see an update -// rather than a delete-and-create, so no cache churn either. +// So the scan adopts the existing row rather than inserting: re-point its +// file_path at the new location and clear the mark. The caller's normal +// UpsertTrack then conflicts on file_path and updates THAT row, so the track id +// survives and likes, plays and playlist memberships travel with it. Clients +// see an update rather than a delete-and-create, so no cache churn either. // // Only rows already marked missing are eligible. A row whose file is present // elsewhere is a duplicate, not a move, and re-pointing it would corrupt the // copy that still exists. That constraint is what makes this safe, and the -// marking added in #2523 is what makes it expressible. +// missing mark is what makes it expressible. -// trackAdopter is the slice of dbq.Queries move detection needs, narrowed so the -// match/ambiguity logic can be tested against a fake. +// trackAdopter is the slice of dbq.Queries move detection needs, narrowed so +// the match/ambiguity logic can be tested against a fake. type trackAdopter interface { FindMissingTrackByMbid(ctx context.Context, mbid string) ([]dbq.FindMissingTrackByMbidRow, error) FindMissingTrackByAudioHash(ctx context.Context, audioStreamSha256 []byte) ([]dbq.FindMissingTrackByAudioHashRow, error) AdoptTrackPath(ctx context.Context, arg dbq.AdoptTrackPathParams) (int64, error) } -// adoptMovedTrack looks for a missing row that is the same recording as the file -// at newPath and re-points it there. Reports whether a row was adopted. +// adoptMovedTrack looks for a missing row that is the same recording as the +// file at newPath and re-points it there. Reports whether a row was adopted. // -// Never returns an error: failing to detect a move is a missed optimisation, not -// a broken scan. The caller carries on and inserts a fresh row, which is the -// pre-#2528 behaviour. +// Never returns an error: failing to detect a move is a missed optimisation, +// not a broken scan. The caller carries on and inserts a fresh row. func (s *Scanner) adoptMovedTrack( ctx context.Context, q trackAdopter, newPath string, audioHash []byte, recordingMBID string, @@ -67,10 +65,10 @@ func (s *Scanner) adoptMovedTrack( } } - // Audio-hash fallback for untagged files (#3914): the encoded audio itself, - // which a rename, a move or a retag leaves unchanged. Absent when the hash - // could not be taken, and then nothing is matched — an empty hash must never - // be looked up, or every unhashable file would pair with every other. + // Audio-hash fallback for untagged files: the encoded audio itself, which a + // rename, a move or a retag leaves unchanged. Absent when the hash could + // not be taken, and then nothing is matched — an empty hash must never be + // looked up, or every unhashable file would pair with every other. if len(audioHash) > 0 { rows, err := q.FindMissingTrackByAudioHash(ctx, audioHash) if err != nil { @@ -84,7 +82,8 @@ func (s *Scanner) adoptMovedTrack( return false } -// candidate is the shared shape of both lookups, so uniqueMatch is written once. +// candidate is the shared shape of both lookups, so uniqueMatch is written +// once. type candidate struct { id pgtype.UUID filePath string @@ -107,9 +106,10 @@ func rowsFromAudioHash(rows []dbq.FindMissingTrackByAudioHashRow) []candidate { } // uniqueMatch requires exactly one candidate. Adopting an arbitrary row out of -// several would attach this file's future history to a coin flip, which is worse -// than starting a fresh row — a fork is recoverable later, a wrong merge isn't. -// Libraries with genuine duplicates hit this, so it's logged rather than silent. +// several would attach this file's future history to a coin flip, which is +// worse than starting a fresh row — a fork is recoverable later, a wrong merge +// isn't. Libraries with genuine duplicates hit this, so it's logged rather than +// silent. func (s *Scanner) uniqueMatch( cands []candidate, newPath, via string, ) (candidate, bool) { @@ -130,8 +130,9 @@ func (s *Scanner) adopt( ) bool { n, err := q.AdoptTrackPath(ctx, dbq.AdoptTrackPathParams{ID: c.id, FilePath: newPath}) if err != nil { - // A unique violation on file_path means something else claimed this path - // first. Fall through to a normal insert rather than failing the file. + // A unique violation on file_path means something else claimed this + // path first. Fall through to a normal insert rather than failing the + // file. s.logger.Warn("library scan: adopting moved track failed", "path", newPath, "via", via, "err", err) return false @@ -141,7 +142,7 @@ func (s *Scanner) adopt( // update, so its mark was already cleared. return false } - // Logged with both paths: this is the operator's only window onto a + // Logged with both paths: this is an admin's only window onto a // reorganisation being understood as a move rather than a new track. s.logger.Info("library scan: track moved, history preserved", "from", c.filePath, "to", newPath, "via", via, "track_id", syncpkg.FormatUUID(c.id)) diff --git a/internal/library/moved_test.go b/internal/library/moved_test.go index 561dfff2..0c32383a 100644 --- a/internal/library/moved_test.go +++ b/internal/library/moved_test.go @@ -12,8 +12,8 @@ import ( ) // fakeAdopter answers the audio-hash lookup only for the hash it holds, the way -// the real query does. A fake that returned its rows for any hash could not tell -// adoption by identity apart from adoption by coincidence. +// the real query does. A fake that returned its rows for any hash could not +// tell adoption by identity apart from adoption by coincidence. type fakeAdopter struct { byMbid []dbq.FindMissingTrackByMbidRow hash []byte @@ -107,10 +107,9 @@ func TestAdoptMovedTrack_FallsBackToAudioHash(t *testing.T) { } } -// The case the old (file_size, duration_ms) pair got wrong: an unrelated file -// that happens to share a size and a duration with a missing track. Size and -// duration are no longer inputs at all; only the audio itself can match, and a -// different recording has a different hash. +// An unrelated file that happens to share a size and a duration with a missing +// track is not adopted. Size and duration are not inputs at all; only the +// audio itself can match, and a different recording has a different hash. func TestAdoptMovedTrack_DifferentAudioIsNotAdopted(t *testing.T) { s := testScanner(t) q := &fakeAdopter{hash: audioHash(0xAA), byHash: []dbq.FindMissingTrackByAudioHashRow{hashRow(1, oldPath)}, adoptRows: 1} @@ -209,8 +208,9 @@ func TestAdoptMovedTrack_NoCandidates(t *testing.T) { } } -// The row's mark was cleared between lookup and update — another file adopted it -// first. AdoptTrackPath's `missing_since IS NOT NULL` predicate reports 0 rows. +// The row's mark was cleared between lookup and update — another file adopted +// it first. AdoptTrackPath's `missing_since IS NOT NULL` predicate reports 0 +// rows. func TestAdoptMovedTrack_LostRaceReportsNotAdopted(t *testing.T) { s := testScanner(t) q := &fakeAdopter{ @@ -224,7 +224,7 @@ func TestAdoptMovedTrack_LostRaceReportsNotAdopted(t *testing.T) { } // Failing to detect a move must never fail the file: the caller falls back to -// inserting a fresh row, which is the pre-#2528 behaviour. +// inserting a fresh row. func TestAdoptMovedTrack_ToleratesQueryErrors(t *testing.T) { sentinel := errors.New("db down") tests := []struct { diff --git a/internal/library/mp4genre.go b/internal/library/mp4genre.go index d2096e12..94204997 100644 --- a/internal/library/mp4genre.go +++ b/internal/library/mp4genre.go @@ -7,15 +7,15 @@ import ( "strings" ) -// MP4 has the same gap as Vorbis comments (#2500). A tag writer stores several -// genres as several "data" atoms inside one ©gen atom (mutagen and Picard do), -// or occasionally as repeated ©gen atoms. dhowden/tag keeps atoms in a map keyed +// MP4 has the same gap as Vorbis comments. A tag writer stores several genres +// as several "data" atoms inside one ©gen atom (mutagen and Picard do), or +// occasionally as repeated ©gen atoms. dhowden/tag keeps atoms in a map keyed // by name (mp4.go: m.data[name] = data), so one value survives either way. // // Path to the values: moov > udta > meta > ilst > ©gen > data. A file that -// carries only the numeric "gnre" atom has no ©gen; this reader returns -// (nil, nil) for it, and extractGenres falls back to dhowden/tag, which resolves -// gnre to a name. +// carries only the numeric "gnre" atom has no ©gen; this reader returns (nil, +// nil) for it, and extractGenres falls back to dhowden/tag, which resolves gnre +// to a name. // maxMP4GenreAtomSize caps how much of one ©gen atom is buffered. Genre text is // bytes; the cap only stops a corrupt size from making the scanner allocate. diff --git a/internal/library/normalization_prefs.go b/internal/library/normalization_prefs.go index 22e68cc3..2a1b1585 100644 --- a/internal/library/normalization_prefs.go +++ b/internal/library/normalization_prefs.go @@ -12,10 +12,9 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// NormalizationPrefs is one user's loudness-normalization preference (M464 -// #4998). It lives on the server so every player applies the same one: the -// web player and the Android app read it, and the leveled stream for -// UPnP/Sonos (#5001) renders with it. +// NormalizationPrefs is one user's loudness-normalization preference. It lives +// on the server so every player applies the same one: the web player and the +// Android app read it, and the leveled stream for UPnP/Sonos renders with it. type NormalizationPrefs struct { // Mode is off, auto, track or album. Auto uses album gain while an album // plays in order and track gain otherwise. diff --git a/internal/library/notify.go b/internal/library/notify.go index 01b4f361..cee533dd 100644 --- a/internal/library/notify.go +++ b/internal/library/notify.go @@ -6,10 +6,11 @@ import ( "github.com/jackc/pgx/v5/pgtype" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" ) -// Package-level notifier for library-health notices to admins (M489): a +// Package-level notifier for library-health notices to admins: a // scan that failed, tracks gone missing, new duplicates. Set once at startup // beside SetEventBus, for the same reason the bus is package-level. Nil, as in // tests that never set it, means nobody is told. @@ -34,6 +35,42 @@ func notifyAdmins(ctx context.Context, kind notifications.Kind, p notifications. n.NotifyLogged(ctx, kind, notifications.ToAdmins(pgtype.UUID{}), p.Map()) } +// notifyItems is how many names a notice carries; the notifications package +// keeps the newest ten across a coalesced burst. +const notifyItems = 10 + +// TrackLabel names a track as a notice does: "Artist – Title", or the title +// alone when the artist is unknown. +func TrackLabel(artist, title string) string { + if artist == "" { + return title + } + return artist + " – " + title +} + +// notifyStillMissing tells admins which of the tracks a scan marked missing +// are still gone after it adopted moves and folded replacements. A marked +// track the scan healed is not news: reporting it would make a quality +// upgrade read as lost music. +func (s *Scanner) notifyStillMissing(ctx context.Context, marked []pgtype.UUID) { + if len(marked) == 0 || ctx.Err() != nil { + return + } + rows, err := dbq.New(s.pool).ListStillMissingTrackLabels(ctx, marked) + if err != nil { + s.logger.Warn("library scan: listing missing tracks for the notice failed", "err", err) + return + } + if len(rows) == 0 { + return + } + items := make([]string, 0, min(len(rows), notifyItems)) + for _, r := range rows[:min(len(rows), notifyItems)] { + items = append(items, TrackLabel(r.ArtistName, r.Title)) + } + notifyAdmins(ctx, notifications.KindTracksMissing, notifications.Payload{Count: int64(len(rows)), Items: items}) +} + // notifyScanFinished tells admins a scan run ended in error. A scan cut short // by shutdown is not a failure anyone needs telling about. func notifyScanFinished(ctx context.Context, errMsg string) { diff --git a/internal/library/notify_test.go b/internal/library/notify_test.go index 0ca635e2..1885f878 100644 --- a/internal/library/notify_test.go +++ b/internal/library/notify_test.go @@ -1,7 +1,6 @@ package library import ( - "bytes" "context" "encoding/json" "fmt" @@ -10,6 +9,7 @@ import ( "path/filepath" "testing" + "github.com/jackc/pgx/v5/pgtype" "github.com/jackc/pgx/v5/pgxpool" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" @@ -36,6 +36,7 @@ type notice struct { kind string count int64 body string + items []string } func unreadNotices(t *testing.T, pool *pgxpool.Pool, user dbq.User) []notice { @@ -55,7 +56,7 @@ func unreadNotices(t *testing.T, pool *pgxpool.Pool, user dbq.User) []notice { if err := json.Unmarshal(r.Payload, &p); err != nil { t.Fatalf("payload: %v", err) } - out = append(out, notice{kind: r.Kind, count: p.Count, body: p.Detail}) + out = append(out, notice{kind: r.Kind, count: p.Count, body: p.Detail, items: p.Items}) } return out } @@ -86,95 +87,121 @@ func TestNotifyScanFinished_FailuresCoalesceAndShutdownIsSilent(t *testing.T) { } } -func TestReconcileMissing_NotifiesAdminsWithARunningCount(t *testing.T) { +// Reconcile only records what it marked. The scan tells admins after +// it has adopted moves and folded replacements, so a file it marked and then +// healed is never reported as lost. +func TestReconcileMissing_RecordsTheMarksWithoutNotifying(t *testing.T) { pool := newPool(t) admin := notifyingAdmin(t, pool) s := testScanner(t, populatedRoot(t)) - // Two scans, each losing 2 of 10 tracks (under the mark cap). - for pass := 0; pass < 2; pass++ { - rows := make([]dbq.ListTrackPathsForReconcileRow, 0, 10) - seen := map[string]struct{}{} - for i := 0; i < 10; i++ { - p := fmt.Sprintf("/music/pass-%d-%02d.mp3", pass, i) - rows = append(rows, row(byte(pass*10+i), p, false)) - if i >= 2 { - seen[p] = struct{}{} - } - } - var stats Stats - if err := s.reconcileMissing(context.Background(), &fakeReconciler{rows: rows}, seen, &stats); err != nil { - t.Fatalf("reconcile pass %d: %v", pass, err) + rows := make([]dbq.ListTrackPathsForReconcileRow, 0, 10) + seen := map[string]struct{}{} + for i := 0; i < 10; i++ { + p := fmt.Sprintf("/music/%02d.mp3", i) + rows = append(rows, row(byte(i), p, false)) + if i >= 2 { + seen[p] = struct{}{} } } - - got := unreadNotices(t, pool, admin) - if len(got) != 1 || got[0].kind != string(notifications.KindTracksMissing) || got[0].count != 4 { - t.Fatalf("notices = %+v, want one tracks_missing counting 4", got) + var stats Stats + if err := s.reconcileMissing(context.Background(), &fakeReconciler{rows: rows}, seen, &stats); err != nil { + t.Fatalf("reconcile: %v", err) + } + if len(stats.marked) != 2 { + t.Errorf("marked %d tracks; want the 2 the walk did not see", len(stats.marked)) + } + if got := unreadNotices(t, pool, admin); len(got) != 0 { + t.Fatalf("reconcile notified before the scan could heal anything: %+v", got) } } -func TestDuplicateSweep_NotifiesOnlyWhenSomethingNewIsProposed(t *testing.T) { +func TestNotifyStillMissing_NamesOnlyTheTracksStillGone(t *testing.T) { pool := newPool(t) admin := notifyingAdmin(t, pool) ctx := context.Background() q := dbq.New(pool) dir := t.TempDir() - logger := slog.New(slog.NewTextHandler(io.Discard, nil)) - - _, album, artist := seedTrack(t, pool, filepath.Join(dir, "seed.mp3")) - // Same audio-stream hash: an exact pair, whatever the prints say. - pair := func(name string, b byte, seed uint64) { - t.Helper() - for i := 1; i <= 2; i++ { - tr, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ - Title: name, AlbumID: album.ID, ArtistID: artist.ID, DurationMs: 200000, - FilePath: filepath.Join(dir, fmt.Sprintf("%s-%d.mp3", name, i)), FileSize: 100, FileFormat: "mp3", - }) - if err != nil { - t.Fatalf("track %s: %v", name, err) - } - if err := q.UpsertTrackFingerprint(ctx, dbq.UpsertTrackFingerprintParams{ - TrackID: tr.ID, AudioStreamSha256: bytes.Repeat([]byte{b}, 32), - Chromaprint: randomPrint(seed+uint64(i), printLen), FingerprintVersion: fingerprintVersion, - ChromaprintLengthSec: defaultChromaprintLengthSec, - }); err != nil { - t.Fatalf("fingerprint %s: %v", name, err) - } - } + first, album, artist := seedTrack(t, pool, filepath.Join(dir, "a.mp3")) + second, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: "Another Track", AlbumID: album.ID, ArtistID: artist.ID, + DurationMs: 1000, FilePath: filepath.Join(dir, "b.mp3"), FileSize: 100, FileFormat: "mp3", + }) + if err != nil { + t.Fatal(err) } - sweep := func() { - t.Helper() - if _, err := runDuplicateSweep(ctx, pool, logger, DefaultFingerprintSettings, duplicateCandidatePage); err != nil { - t.Fatalf("sweep: %v", err) - } - } - markAllRead := func() { - t.Helper() - if _, err := q.MarkAllNotificationsRead(ctx, dbq.MarkAllNotificationsReadParams{UserID: admin.ID}); err != nil { - t.Fatalf("mark read: %v", err) - } + // The scan marked both; the second came back (a move it adopted). + marked := []pgtype.UUID{first.ID, second.ID} + if _, err := q.MarkTracksMissing(ctx, []pgtype.UUID{first.ID}); err != nil { + t.Fatal(err) } - pair("first", 1, 100) - sweep() + s := New(pool, slog.New(slog.NewTextHandler(io.Discard, nil)), nil, nil) + s.notifyStillMissing(ctx, marked) got := unreadNotices(t, pool, admin) - if len(got) != 1 || got[0].kind != string(notifications.KindDuplicatesFound) || got[0].count != 1 { - t.Fatalf("after the first sweep notices = %+v, want one duplicates_found counting 1", got) + if len(got) != 1 || got[0].kind != string(notifications.KindTracksMissing) || got[0].count != 1 || + len(got[0].items) != 1 || got[0].items[0] != "Delete Test Artist – Delete Test Track" { + t.Fatalf("notices = %+v, want one tracks_missing naming the one track still gone", got) } - // Read, then swept again with nothing new: no reminder. - markAllRead() - sweep() + // Nothing still gone: nothing to say. + if _, err := q.MarkAllNotificationsRead(ctx, dbq.MarkAllNotificationsReadParams{UserID: admin.ID}); err != nil { + t.Fatal(err) + } + s.notifyStillMissing(ctx, []pgtype.UUID{second.ID}) if got := unreadNotices(t, pool, admin); len(got) != 0 { - t.Fatalf("a sweep that found nothing new notified: %+v", got) + t.Fatalf("a healed track was reported: %+v", got) } +} - // A new pair is news, and the count is everything awaiting review. - pair("second", 2, 200) - sweep() - got = unreadNotices(t, pool, admin) - if len(got) != 1 || got[0].count != 2 { - t.Fatalf("after a new pair notices = %+v, want one counting 2", got) +// Admins hear about a duplicate group once, when a resolver pass first leaves +// it for them. A group the resolver settles or links is never news, and a +// group already seen by an earlier pass is not news again. +func TestResolveDuplicates_TellsAdminsOnlyOfNewGroupsLeftForThem(t *testing.T) { + pool := newPool(t) + admin := notifyingAdmin(t, pool) + ctx := context.Background() + q := dbq.New(pool) + dir := t.TempDir() + first, album, artist := seedTrack(t, pool, filepath.Join(dir, "a.mp3")) + if _, err := pool.Exec(ctx, `UPDATE tracks SET title = 'Alpha' WHERE id = $1`, first.ID); err != nil { + t.Fatal(err) + } + second, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: "Beta", AlbumID: album.ID, ArtistID: artist.ID, + DurationMs: 1000, FilePath: filepath.Join(dir, "b.mp3"), FileSize: 100, FileFormat: "mp3", + }) + if err != nil { + t.Fatal(err) + } + // One album, one recording under two song titles, no track list to decide. + var groupID pgtype.UUID + if err := pool.QueryRow(ctx, + `INSERT INTO duplicate_groups (member_key, tier) VALUES ('notify-review', 'exact') RETURNING id`, + ).Scan(&groupID); err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, `INSERT INTO duplicate_group_members (group_id, track_id) VALUES ($1, $2), ($1, $3)`, + groupID, first.ID, second.ID); err != nil { + t.Fatal(err) + } + + if _, err := ResolveDuplicates(ctx, pool, nil, "", true); err != nil { + t.Fatal(err) + } + got := unreadNotices(t, pool, admin) + if len(got) != 1 || got[0].kind != string(notifications.KindDuplicatesFound) || got[0].count != 1 || + len(got[0].items) != 1 || got[0].items[0] == "" { + t.Fatalf("after the first pass notices = %+v, want one duplicates_found naming the group", got) + } + + if _, err := q.MarkAllNotificationsRead(ctx, dbq.MarkAllNotificationsReadParams{UserID: admin.ID}); err != nil { + t.Fatal(err) + } + if _, err := ResolveDuplicates(ctx, pool, nil, "", true); err != nil { + t.Fatal(err) + } + if got := unreadNotices(t, pool, admin); len(got) != 0 { + t.Fatalf("a group an earlier pass saw was told again: %+v", got) } } diff --git a/internal/library/reconcile.go b/internal/library/reconcile.go index 1f13474b..18e05b9b 100644 --- a/internal/library/reconcile.go +++ b/internal/library/reconcile.go @@ -9,7 +9,6 @@ import ( "github.com/jackc/pgx/v5/pgtype" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) @@ -23,14 +22,13 @@ type trackReconciler interface { InsertLibraryChanges(ctx context.Context, arg dbq.InsertLibraryChangesParams) error } -// Reconcile marks tracks whose files have disappeared (#2523). +// Reconcile marks tracks whose files have disappeared. // -// Why this exists: nothing in Minstrel used to notice a deleted file. The walk -// only visits paths that exist, so a row whose file is gone was never scanned, -// never errored, never counted — permanently invisible. The watcher ignores -// removals by design (see classifyEvent), and the safety-net scan is the same -// walk, so it covers additions only. Rows accumulated forever, kept being -// offered to recommendations, and failed at playback. +// The walk only visits paths that exist, so without this a row whose file is +// gone would never be scanned, never error, never be counted. The watcher +// ignores removals by design (see classifyEvent), and the safety-net scan is +// the same walk, so it covers additions only. Unmarked, such rows would keep +// being offered to recommendations and fail at playback. // // Why it MARKS rather than deletes: a missing file is a claim about the // filesystem, and the filesystem lies transiently — an unmounted volume, a @@ -63,9 +61,9 @@ func (s *Scanner) reconcileMissing( if err := s.verifyRootsPresent(); err != nil { return err } - // Roots resolved but the walk found nothing. Either the library is genuinely - // empty — in which case there is nothing to reconcile — or the mount is - // hollow. Both mean: don't act. + // Roots resolved but the walk found nothing. Either the library is + // genuinely empty — in which case there is nothing to reconcile — or the + // mount is hollow. Both mean: don't act. if len(seen) == 0 { return errors.New("walk saw no audio files; refusing to reconcile") } @@ -124,25 +122,24 @@ func (s *Scanner) reconcileMissing( return fmt.Errorf("mark tracks missing: %w", err) } stats.Missing = int(n) - // Warn, not Info: every one of these is a library entry the operator - // probably didn't intend to lose, and the only place it surfaces today is - // this line. + // Warn, not Info: every one of these is a library entry an admin probably + // didn't intend to lose. s.logger.Warn("library scan: tracks marked missing (files not found)", "count", n, "library_total", len(rows)) - if n > 0 { - notifyAdmins(ctx, notifications.KindTracksMissing, notifications.Payload{Count: n}) - } + // Admins are told after the scan has adopted moves and folded + // replacements, about the ones still gone (notifyStillMissing). + stats.marked = toMark return nil } // logTrackChanges tells the delta sync that these tracks changed, so clients -// pick up the missing mark (#2704). +// pick up the missing mark. // -// Without this the mark was invisible to every client: MarkTracksMissing and -// ClearTracksMissing are plain UPDATEs, and /api/library/sync is a change-log -// feed — a row that never produces a change is never re-sent, so a client -// would keep its stale copy until an unrelated edit touched the track or the -// cursor fell out of the retention window. +// Without this the mark would be invisible to every client: MarkTracksMissing +// and ClearTracksMissing are plain UPDATEs, and /api/library/sync is a +// change-log feed — a row that never produces a change is never re-sent, so a +// client would keep its stale copy until an unrelated edit touched the track or +// the cursor fell out of the retention window. // // Logged BEFORE the mutation, deliberately, which is the opposite of the // scanner's log-after-success pattern. The two failure modes are not @@ -162,7 +159,7 @@ func logTrackChanges(ctx context.Context, q trackReconciler, ids []pgtype.UUID) } // OpUpsert, not OpDelete: the track still exists and keeps its history — // only its playability changed. A delete would tell clients to drop the - // row, which is the behaviour #2704 deliberately rejected. + // row, and a missing file is expected to come back. return q.InsertLibraryChanges(ctx, dbq.InsertLibraryChangesParams{ EntityType: string(syncpkg.EntityTrack), EntityIds: strIDs, @@ -170,16 +167,17 @@ func logTrackChanges(ctx context.Context, q trackReconciler, ids []pgtype.UUID) }) } -// verifyRootsPresent is the first and most important guard. If a configured root -// doesn't resolve to a readable directory, the walk beneath it found nothing and -// every row under it would look deleted. An unmounted media volume is the -// obvious case, and it is common enough — a container restart racing its volume -// mount does exactly this. +// verifyRootsPresent is the first and most important guard. If a configured +// root doesn't resolve to a readable directory, the walk beneath it found +// nothing and every row under it would look deleted. An unmounted media volume +// is the obvious case, and it is common enough — a container restart racing its +// volume mount does exactly this. func (s *Scanner) verifyRootsPresent() error { - if len(s.paths) == 0 { + roots := s.Roots() + if len(roots) == 0 { return errors.New("no scan roots configured") } - for _, root := range s.paths { + for _, root := range roots { info, err := os.Stat(root) if err != nil { return fmt.Errorf("scan root %q unavailable: %w", root, err) diff --git a/internal/library/reconcile_test.go b/internal/library/reconcile_test.go index 55cb8e45..5d043565 100644 --- a/internal/library/reconcile_test.go +++ b/internal/library/reconcile_test.go @@ -26,7 +26,7 @@ type fakeReconciler struct { listErr error markErr error clearErr error - // Change rows the reconcile pass asked the delta sync to emit (#2704). + // Change rows the reconcile pass asked the delta sync to emit. loggedChanges []dbq.InsertLibraryChangesParams } @@ -186,8 +186,8 @@ func TestReconcileMissing_DoesNotRemarkAlreadyMissingRows(t *testing.T) { } // The guard that matters most. A half-attached mount makes the walk succeed -// while seeing only part of the library — evidence indistinguishable from a mass -// deletion, so reconcile must refuse rather than guess. +// while seeing only part of the library — evidence indistinguishable from a +// mass deletion, so reconcile must refuse rather than guess. func TestReconcileMissing_RefusesWhenTooMuchWouldBeMarked(t *testing.T) { root := populatedRoot(t) s := testScanner(t, root) @@ -288,8 +288,8 @@ func TestReconcileMissing_RefusesWhenRootEmpty(t *testing.T) { } } -// Several roots, one detached. Marking must not proceed on partial evidence just -// because the other roots looked fine. +// Several roots, one detached. Marking must not proceed on partial evidence +// just because the other roots looked fine. func TestReconcileMissing_RefusesWhenAnyRootMissing(t *testing.T) { good := populatedRoot(t) s := testScanner(t, good, filepath.Join(t.TempDir(), "detached")) @@ -336,8 +336,8 @@ func TestVerifyRootsPresent_NoRootsConfigured(t *testing.T) { // Without a change row the mark is invisible to every client: // /api/library/sync is a change-log feed, so a plain UPDATE never reaches -// anyone. This was the gap that made #2704's wire field inert — the flag -// existed and nothing ever told a client to re-read the track. +// anyone, and the wire field would exist with nothing ever telling a client to +// re-read the track. func TestReconcileMissing_MarkingEmitsSyncChanges(t *testing.T) { s := testScanner(t, populatedRoot(t)) @@ -366,8 +366,8 @@ func TestReconcileMissing_MarkingEmitsSyncChanges(t *testing.T) { t.Errorf("entity_type = %q, want track", got.EntityType) } // Upsert, not delete: the track still exists and keeps its history. A - // delete would tell clients to drop the row, which is the behaviour - // #2704 deliberately rejected. + // delete would tell clients to drop the row, and a missing file is + // expected to come back. if got.Op != "upsert" { t.Errorf("op = %q, want upsert — the row survives, only its state changed", got.Op) } diff --git a/internal/library/replace.go b/internal/library/replace.go new file mode 100644 index 00000000..344b03d6 --- /dev/null +++ b/internal/library/replace.go @@ -0,0 +1,83 @@ +package library + +import ( + "context" + "errors" + "fmt" + "log/slog" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/coverart" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" +) + +// Replacing a track with a better copy. +// +// An upgrade files the better copy first, as a track of its own, and only +// then replaces the old one: its file is removed, then its likes, plays, +// playlist entries and tags move onto the new row and the old row goes. It +// is the duplicate merge with the survivor chosen by the import, and it +// keeps the merge's order: if the old file cannot be removed nothing +// in the database changes, and the next scan finds the two copies as a +// duplicate pair. + +// ErrReplaceSame: the old and new track are one row. +var ErrReplaceSame = errors.New("library: a track cannot replace itself") + +// Replaced is what one replacement moved onto the new track. +type Replaced struct { + PlayEvents int64 + SkipEvents int64 + Likes int + PlaylistEntries int +} + +// ReplaceTrack replaces oldID with newID: the old file is removed, then the +// old row folded into the new one. dataDir, when set, also clears the cached +// art of an artist the fold left with nothing. logger may be nil. +func ReplaceTrack( + ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, dataDir string, oldID, newID pgtype.UUID, +) (Replaced, error) { + if logger == nil { + logger = slog.Default() + } + if oldID == newID { + return Replaced{}, ErrReplaceSame + } + old, err := dbq.New(pool).GetTrackByID(ctx, oldID) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + return Replaced{}, ErrTrackNotFound + } + return Replaced{}, fmt.Errorf("get track: %w", err) + } + if err := removeTrackFileOnDisk(old.FilePath); err != nil { + return Replaced{}, err + } + + var f foldResult + err = pgx.BeginFunc(ctx, pool, func(tx pgx.Tx) error { + changes := mergeChanges{} + var ferr error + f, ferr = foldTrackInto(ctx, dbq.New(tx), newID, oldID, &changes) + if ferr != nil { + return ferr + } + return changes.log(ctx, tx) + }) + if err != nil { + return Replaced{}, fmt.Errorf("fold the old track into the new: %w", err) + } + if f.DeletedArtistID != nil && dataDir != "" { + if err := coverart.CleanupArtistArt(dataDir, *f.DeletedArtistID); err != nil { + logger.Warn("track replace: artist-art cleanup failed", + "artist_id", syncpkg.FormatUUID(*f.DeletedArtistID), "err", err) + } + } + return Replaced{PlayEvents: f.PlayEvents, SkipEvents: f.SkipEvents, Likes: len(f.Likers), + PlaylistEntries: f.PlaylistEntries}, nil +} diff --git a/internal/library/replaygain.go b/internal/library/replaygain.go index 8c98ae51..04e7af35 100644 --- a/internal/library/replaygain.go +++ b/internal/library/replaygain.go @@ -9,7 +9,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// ReplayGain delivery (M464 #4997). +// ReplayGain delivery. // // Clients are handed loudness in the ReplayGain 2.0 form every player already // understands: a gain in dB that brings the track (or album) to the RG2 diff --git a/internal/library/scanner.go b/internal/library/scanner.go index 4fcd73c9..f2cff895 100644 --- a/internal/library/scanner.go +++ b/internal/library/scanner.go @@ -12,6 +12,7 @@ import ( "path/filepath" "strconv" "strings" + "sync" "time" "github.com/dhowden/tag" @@ -27,8 +28,7 @@ import ( // `internal/api/media.go` MIME detection — the stream handler must be // able to serve every extension the scanner indexes, and there is no // point in adding extensions to the stream handler that the scanner -// will silently skip. Drift #571 caught the divergence after .opus, -// .aac, and .wav were added to media.go but not here. +// will silently skip. var audioExtensions = map[string]bool{ ".mp3": true, ".m4a": true, @@ -42,7 +42,7 @@ var audioExtensions = map[string]bool{ // tagReadVersion is the version of this package's tag-extraction logic. Rows // whose tracks.tag_read_version is lower get their tags re-read on the next // scan even when the file itself hasn't changed, so a fix reaches an existing -// library without the operator rebuilding it (migration 0054). +// library without anyone rebuilding it (migration 0054). // // Bump this whenever a change to tag extraction should reach already-indexed // files, and say why below. @@ -51,20 +51,34 @@ var audioExtensions = map[string]bool{ // dhowden/tag welds null-separated multi-values into one token // ("Alternative Rock" + "Rock" -> "Alternative RockRock"), which corrupted // the genre browse axis and polluted the taste profile's tag vocabulary, -// and left bare ID3v1 numeric references unresolved (#2499). -// 2: acronym and apostrophe casing repaired on genre values (#2468) — "Edm" -> +// and left bare ID3v1 numeric references unresolved. +// 2: acronym and apostrophe casing repaired on genre values — "Edm" -> // "EDM", "Children'S Music" -> "Children's Music". Bumped rather than left // to new files only because taste_profile.sql reads tracks.genre directly, // so a half-repaired library would carry both spellings as separate tags. -// 3: the MusicBrainz release-group id is read into albums.release_group_mbid -// (#5241). Lidarr names albums by release group, not by the release id +// 3: the MusicBrainz release-group id is read into albums.release_group_mbid. +// Wants name albums by release group, not by the release id // albums.mbid holds, so re-acquisition and request completion need it for // the albums already in the library, not only for files added from now on. // 4: every value of a repeated genre field is read for FLAC, Ogg Vorbis, Opus -// and MP4 (#2500). dhowden/tag kept only the last, and 3,658 of the -// operator's 4,092 FLACs declare more than one, so most of their genres -// were never stored. -const tagReadVersion int16 = 4 +// and MP4. dhowden/tag kept only the last, and most FLACs that carry +// genres declare more than one. +// 5: the recording id is read from FLAC, Ogg Vorbis and Opus. Picard +// calls it MUSICBRAINZ_TRACKID there and dhowden/tag files that under the +// release-track name, so every Vorbis-tagged track came in without its +// tag id and waited on an AcoustID lookup. A re-read stores the tag id, +// over a looked-up one. +// 6: a Vorbis TRACKNUMBER or DISCNUMBER written as "3/20" is read. +// dhowden/tag reads it as 0, and Minstrel's own importer wrote it that +// way, so the tracks it imported came in with no track or disc number. +// 7: an album of the same name in another release group gets its own row +// (resolveAlbum). Self-titled albums by one artist shared a row, so their +// merges, wants and upgrades all counted as one album; a re-read files +// each existing track under its own album. +// 8: a file tagged with no release group joins the same-named album that +// already holds a track from its folder. Under 7 it took the oldest row, +// which could file a bonus track on a later album of the same name. +const tagReadVersion int16 = 8 type Stats struct { Scanned int `json:"scanned"` @@ -72,29 +86,38 @@ type Stats struct { Updated int `json:"updated"` Skipped int `json:"skipped"` Errored int `json:"errored"` - // Missing / Restored come from the reconcile pass, not the walk (#2523): + // Missing / Restored come from the reconcile pass, not the walk: // rows whose file the walk didn't find, and rows whose file came back. // Only a full Scan sets these — see reconcileMissing. Missing int `json:"missing"` Restored int `json:"restored"` + // marked are the tracks this scan's reconcile marked missing, so the scan + // can tell admins which of them are still gone once it has adopted moves + // and folded replacements. + marked []pgtype.UUID } type Scanner struct { pool *pgxpool.Pool logger *slog.Logger - paths []string - // fingerprint derives a file's acoustic identity (M400). A field so an + // paths are the library's folders, changed by SetRoots. Read them + // through Roots. + paths []string + rootsMu sync.RWMutex + rootsChanged chan struct{} + // fingerprint derives a file's acoustic identity. A field so an // integration test can substitute a deterministic one: CI has no real audio // to fingerprint, and what the test pins is WHEN the scan fingerprints, not // what the tools print. Call it through fingerprintFile. fingerprint func(ctx context.Context, path string, opts fingerprintOptions) fingerprintResult - // settings is the operator's fingerprinting policy (#3913), shared with the + // settings is the fingerprinting policy, shared with the // workers and the admin API. Nil fingerprints with the defaults. settings *FingerprintSettingsService } func New(pool *pgxpool.Pool, logger *slog.Logger, paths []string, settings *FingerprintSettingsService) *Scanner { - return &Scanner{pool: pool, logger: logger, paths: paths, fingerprint: computeFingerprint, settings: settings} + return &Scanner{pool: pool, logger: logger, paths: paths, rootsChanged: make(chan struct{}, 1), + fingerprint: computeFingerprint, settings: settings} } // Scan walks every configured root and upserts any audio file whose mtime is @@ -102,7 +125,7 @@ func New(pool *pgxpool.Pool, logger *slog.Logger, paths []string, settings *Fing // are logged + counted; the scan keeps going. // // It then reconciles: rows whose file the walk never saw get marked missing, -// and rows whose file has come back get un-marked (#2523). Only a FULL scan may +// and rows whose file has come back get un-marked. Only a FULL scan may // do this — the walk's set of seen paths is the evidence, and a partial // (watcher-driven) scan has no basis for concluding anything about files it // didn't look at. That's why ScanFiles does not reconcile. @@ -115,23 +138,24 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro start := time.Now() // PHASE 1 — enumerate. Collect every audio path without touching tags or - // ffprobe. Cheap: WalkDir already stats each entry, so this adds a directory - // traversal and nothing else. + // ffprobe. Cheap: WalkDir already stats each entry, so this adds a + // directory traversal and nothing else. // // The order matters and is the whole reason enumeration is separate. // Reconcile has to mark disappeared rows BEFORE any file is processed, - // because move detection (#2528) can only adopt a row that is already marked - // missing. A rename performed while the server was down surfaces the deletion - // and the addition in the SAME scan — so if reconcile ran at the end, the new - // path would insert a fresh row first and the fork would be permanent. + // because move detection can only adopt a row that is already marked + // missing. A rename performed while the server was down surfaces the + // deletion and the addition in the SAME scan — so if reconcile ran at the + // end, the new path would insert a fresh row first and the fork would be + // permanent. paths, walkErrs := s.enumerate(ctx, progressCb, &stats) stats.Errored += walkErrs if err := ctx.Err(); err != nil { return stats, err } - // PHASE 2 — reconcile. Only ever on a COMPLETE enumeration: a cancelled walk - // has a partial view and would mark everything it hadn't reached. + // PHASE 2 — reconcile. Only ever on a COMPLETE enumeration: a cancelled + // walk has a partial view and would mark everything it hadn't reached. seen := make(map[string]struct{}, len(paths)) for _, p := range paths { seen[p] = struct{}{} @@ -141,10 +165,9 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro // evidence, and that refusal arrives here as an error. // // The consequence is named explicitly because it is not obvious: move - // detection (#2528) can only adopt a row that is already marked missing, - // so a refused reconcile also means renamed files insert fresh rows and - // fork their history. That's the pre-#2528 behaviour rather than a new - // failure, but it's worth knowing which scan it happened on. It bites + // detection can only adopt a row that is already marked missing, so a + // refused reconcile also means renamed files insert fresh rows and fork + // their history. It's worth knowing which scan it happened on. It bites // hardest when a large fraction of a small library is reorganised at // once, which trips the mark cap. s.logger.Warn("library scan: reconcile skipped — moved files will fork rather than adopt", @@ -181,30 +204,33 @@ func (s *Scanner) Scan(ctx context.Context, progressCb func(Stats)) (Stats, erro } // PHASE 4 — fold missing rows into replacements this scan brought in or - // that were already here (M485). After processing, so both this scan's + // that were already here. After processing, so both this scan's // marks and its arrivals are in. Not fatal: an unfolded pair is only a row // left in the missing list, and the next scan tries again. if _, err := PairMissingTracks(ctx, s.pool, s.logger); err != nil && ctx.Err() == nil { s.logger.Warn("library scan: missing-pair pass failed", "err", err) } + // Told only now, so a file that moved or was upgraded, which this scan + // marked and then healed, is not reported as lost. + s.notifyStillMissing(ctx, stats.marked) return stats, nil } // enumerate walks every configured root and returns the audio paths found, in // walk order, plus a count of walk errors. // -// A path is recorded even if it will later fail to parse: an unreadable file is a -// broken file, not a missing one, and letting reconcile mark it missing would -// hide it from the operator behind the wrong explanation. +// A path is recorded even if it will later fail to parse: an unreadable file is +// a broken file, not a missing one, and letting reconcile mark it missing would +// hide it from an admin behind the wrong explanation. func (s *Scanner) enumerate( ctx context.Context, progressCb func(Stats), stats *Stats, ) ([]string, int) { paths := make([]string, 0, 8192) errs := 0 - for _, root := range s.paths { - // WalkDir's own error return is folded into the per-entry handler below, - // so a bad root is counted rather than aborting the whole scan — one - // unreadable root shouldn't discard the others' results. + for _, root := range s.Roots() { + // WalkDir's own error return is folded into the per-entry handler + // below, so a bad root is counted rather than aborting the whole scan — + // one unreadable root shouldn't discard the others' results. _ = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { if ctx.Err() != nil { return fs.SkipAll @@ -251,11 +277,10 @@ func (s *Scanner) scanFile( } // Incremental skip: only when the file hasn't changed AND we already have a // real duration AND the row's tag-derived columns were written by the - // current extraction logic. The duration clause lets older scans that - // recorded duration_ms=0 (before ffprobe was wired) get backfilled without - // forcing the operator to wipe the library; the tag-version clause does the - // same job for tag-extraction fixes (#2499). Once both are current, - // subsequent scans short-circuit as before. + // current extraction logic. The duration clause backfills rows that + // recorded duration_ms=0 without anyone wiping the library; the + // tag-version clause does the same job for tag-extraction fixes. Once both + // are current, subsequent scans short-circuit. unchanged := knownTrack && !existing.UpdatedAt.Time.Before(mtime) if unchanged && existing.DurationMs > 0 && existing.TagReadVersion >= tagReadVersion { stats.Skipped++ @@ -272,9 +297,9 @@ func (s *Scanner) scanFile( if err != nil { return pgtype.UUID{}, false, fmt.Errorf("tag read: %w", err) } - albumMBID, artistMBID := extractMBIDs(meta) - recordingMBID := extractRecordingMBID(meta) - releaseGroupMBID := extractReleaseGroupMBID(meta) + ids := ReadTagMBIDs(meta) + albumMBID, artistMBID := ids.Release, ids.Artist + recordingMBID, releaseGroupMBID := ids.Recording, ids.ReleaseGroup artistName := meta.Artist() if artistName == "" { @@ -293,13 +318,12 @@ func (s *Scanner) scanFile( if err != nil { return pgtype.UUID{}, false, fmt.Errorf("artist: %w", err) } - album, err := s.resolveAlbum(ctx, q, artist.ID, albumTitle, meta.Year(), albumMBID, releaseGroupMBID) + album, err := s.resolveAlbum(ctx, q, artist.ID, albumTitle, meta.Year(), albumMBID, releaseGroupMBID, filepath.Dir(path)) if err != nil { return pgtype.UUID{}, false, fmt.Errorf("album: %w", err) } - trackNum, _ := meta.Track() - discNum, _ := meta.Disc() + trackNum, discNum := ReadTrackAndDisc(meta) // An unchanged file being re-read only to refresh tag-derived columns // doesn't need another ffprobe: the stored duration is still accurate, and @@ -326,15 +350,15 @@ func (s *Scanner) scanFile( // above — a tagReadVersion bump must stay bound by tag reads, not become a // decode of the whole library. // - // An unchanged file with NO fingerprint yet is the backfill's job (#3908), + // An unchanged file with NO fingerprint yet is the backfill's job, // deliberately not the scan's. Folding it into the skip check would make // the first scan after an upgrade re-decode every track and push a sync // change to every client for each one. // - // Computed before move adoption so adoption can match on the audio hash - // (#3914); stored after the upsert, once the row id is known. + // Computed before move adoption so adoption can match on the audio hash; + // stored after the upsert, once the row id is known. // - // With fingerprinting switched off (#3913) the scan decodes nothing, but it + // With fingerprinting switched off the scan decodes nothing, but it // still takes the stream hash: a demux rather than a decode, and the only // thing that recognises a moved file with no MBID. var fp fingerprintResult @@ -348,13 +372,13 @@ func (s *Scanner) scanFile( } // A path we've never seen might not be a new track — it might be one that - // moved or was renamed (#2528). Adopting re-points the existing row at this + // moved or was renamed. Adopting re-points the existing row at this // path and clears its missing mark, so the UpsertTrack below conflicts on // file_path and updates THAT row: same track id, likes and play history // intact. Without this, renumbering an album forks every track on it. // - // Runs after fingerprinting because, for a file with no MBID, adoption matches - // on its audio hash (#3914), and only for genuinely unknown paths — a known + // Runs after fingerprinting because, for a file with no MBID, adoption + // matches on its audio hash, and only for genuinely unknown paths — a known // path is already the row we're going to update. if !knownTrack { if s.adoptMovedTrack(ctx, q, path, fp.streamSHA256, recordingMBID) { @@ -416,7 +440,8 @@ func (s *Scanner) scanFile( storeFingerprint(ctx, q, s.logger, track.ID, path, fp, fpCfg.ChromaprintLengthSec) } else if err := q.DeleteTrackFingerprint(ctx, track.ID); err != nil { // Off, nothing is stored — but a row describing the previous bytes - // must not outlive them, or the sweep would compare audio that is gone. + // must not outlive them, or the sweep would compare audio that is + // gone. s.logger.Warn("fingerprint: clearing stale fingerprint failed", "path", path, "err", err) } // Loudness is measured by its own worker, never inline: it decodes the @@ -426,7 +451,7 @@ func (s *Scanner) scanFile( if err := q.DeleteTrackLoudness(ctx, track.ID); err != nil { s.logger.Warn("loudness: clearing stale measurement failed", "path", path, "err", err) } - // Likewise an AcoustID lookup (M401): the worker looks the new bytes + // Likewise an AcoustID lookup: the worker looks the new bytes // up again. A looked-up MBID stays meanwhile and is taken back only if // the new lookup does not match it. if err := q.DeleteAcoustIDLookup(ctx, track.ID); err != nil { @@ -479,7 +504,7 @@ func (s *Scanner) ScanFiles(ctx context.Context, paths []string) ([]pgtype.UUID, "errored", stats.Errored, ) // A file the watcher just brought in may be the replacement for a row - // an earlier scan marked missing (M485). + // an earlier scan marked missing. if _, err := PairMissingTracks(ctx, s.pool, s.logger); err != nil && ctx.Err() == nil { s.logger.Warn("library watch: missing-pair pass failed", "err", err) } @@ -500,12 +525,11 @@ func (s *Scanner) resolveArtist(ctx context.Context, q *dbq.Queries, name, mbid if isUniqueViolation(uerr) { // Another artist row already owns this MBID — two rows that // should be merged (usually two spellings of one name). - // Expected, not a fault: leave NULL and let the operator - // merge. Mirrors resolveAlbum, which has always handled it - // this way — without this branch the identical benign - // condition logged a generic warning plus a Postgres ERROR - // line on every scan, which teaches an operator to ignore - // database errors (#2524). + // Expected, not a fault: leave NULL and let an admin + // merge. Mirrors resolveAlbum. Logged at Info, because a + // warning plus a Postgres ERROR line on every scan for a + // benign condition teaches an admin to ignore database + // errors. s.logger.Info("library scan: duplicate artist mbid (canonical row already owns it)", "artist_id", existing.ID, "artist", name, "mbid", mbid) } else { @@ -540,8 +564,13 @@ func (s *Scanner) resolveArtist(ctx context.Context, q *dbq.Queries, name, mbid return artist, nil } -func (s *Scanner) resolveAlbum(ctx context.Context, q *dbq.Queries, artistID pgtype.UUID, title string, year int, mbid, releaseGroupMBID string) (dbq.Album, error) { - existing, err := q.GetAlbumByArtistAndTitle(ctx, dbq.GetAlbumByArtistAndTitleParams{ArtistID: artistID, Title: title}) +func (s *Scanner) resolveAlbum(ctx context.Context, q *dbq.Queries, artistID pgtype.UUID, title string, year int, mbid, releaseGroupMBID, dir string) (dbq.Album, error) { + lookup := dbq.GetAlbumByArtistAndTitleParams{ArtistID: artistID, Title: title, Dir: dir} + if releaseGroupMBID != "" { + rg := releaseGroupMBID + lookup.ReleaseGroupMbid = &rg + } + existing, err := q.GetAlbumByArtistAndTitle(ctx, lookup) if err == nil { // Heal the release group the same way. Not unique, so no conflict to // handle: every release of a group may carry it. @@ -674,3 +703,23 @@ func probeDurationMs(ctx context.Context, path string) (int32, error) { } return int32(ms), nil } + +// ProbeDurationMs is a file's length in milliseconds, read by ffprobe. +func ProbeDurationMs(ctx context.Context, path string) (int32, error) { + return probeDurationMs(ctx, path) +} + +// IsAudioFile reports whether the scanner reads a file with this name. +func IsAudioFile(path string) bool { + return audioExtensions[strings.ToLower(filepath.Ext(path))] +} + +// Roots are the folders the scanner reads: the library. +func (s *Scanner) Roots() []string { + if s == nil { + return nil + } + s.rootsMu.RLock() + defer s.rootsMu.RUnlock() + return append([]string(nil), s.paths...) +} diff --git a/internal/library/scanner_test.go b/internal/library/scanner_test.go index b5404c63..fbceb4dc 100644 --- a/internal/library/scanner_test.go +++ b/internal/library/scanner_test.go @@ -148,8 +148,8 @@ func TestScanner_Integration(t *testing.T) { t.Fatalf("GetTrackByPath after ScanFiles: %v", err) } - // Re-running ScanFiles on the now-unchanged file (duration probed) skips it, - // so no album is reported as changed. + // Re-running ScanFiles on the now-unchanged file (duration probed) skips + // it, so no album is reported as changed. if _, err := pool.Exec(ctx, "UPDATE tracks SET duration_ms = 1000 WHERE duration_ms = 0"); err != nil { t.Fatalf("seed durations (2): %v", err) } @@ -206,14 +206,14 @@ func writeTestMP3(t *testing.T, path string, frames map[string]string) { } } -// TestScanner_AdoptsMovedFile_Integration is the #2528 proof: a renamed file +// TestScanner_AdoptsMovedFile_Integration: a renamed file // must keep its existing tracks row — same id, so likes, play history and // playlist memberships travel with it — rather than forking into a marked ghost // plus a fresh zero-history row. // -// Uses the MBID path. The synthetic MP3s here carry no real audio, so their audio -// stream hash cannot be relied on — which is why the recording MBID is the signal -// under test. The audio-hash path is covered by moved_test.go. +// Uses the MBID path. The synthetic MP3s here carry no real audio, so their +// audio stream hash cannot be relied on — which is why the recording MBID is +// the signal under test. The audio-hash path is covered by moved_test.go. // // Eight tracks with one rename keeps the marked fraction at 12.5%, under // missingMarkMaxFraction. That is load-bearing: if the rename exceeded the cap, @@ -309,7 +309,7 @@ func TestScanner_AdoptsMovedFile_Integration(t *testing.T) { } } -// TestScanner_ReleaseGroupMBID_Integration is #5241's scan half: a new album +// TestScanner_ReleaseGroupMBID_Integration: a new album // takes its release-group id from the tag, and an album indexed before this // read existed gets it on the next scan, through the tagReadVersion bump, // without its file changing. @@ -385,3 +385,103 @@ func TestScanner_ReleaseGroupMBID_Integration(t *testing.T) { t.Errorf("album A release group changed to %v", rg) } } + +// Two albums of one name by one artist, in different release groups, are two +// albums, as Weezer's self-titled albums are. A +// track of the same group joins its album, and a track filed under the other +// album's row by the old lookup moves to its own on the tagReadVersion re-read. +func TestScanner_SameTitledAlbumsOfTwoGroupsAreTwoAlbums_Integration(t *testing.T) { + if testing.Short() { + t.Skip("skipping scanner integration in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + ctx := context.Background() + logger := slog.New(slog.NewTextHandler(io.Discard, nil)) + if err := db.Migrate(dsn, logger); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(ctx, dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + if _, err := pool.Exec(ctx, "TRUNCATE tracks, albums, artists RESTART IDENTITY CASCADE"); err != nil { + t.Fatalf("truncate: %v", err) + } + + const ( + blue = "aaaaaaaa-1994-2222-3333-444444444444" + green = "bbbbbbbb-2001-2222-3333-444444444444" + ) + root := t.TempDir() + tagged := func(rel, title, group string) string { + p := filepath.Join(root, rel) + writeTestMP3(t, p, map[string]string{ + "TIT2": title, "TPE1": "Weezer", "TALB": "Weezer", + "TXXX": "MusicBrainz Release Group Id\x00" + group, + }) + return p + } + tagged("Weezer (1994)/01.mp3", "My Name Is Jonas", blue) + tagged("Weezer (1994)/02.mp3", "No One Else", blue) + greenPath := tagged("Weezer (2001)/01.mp3", "Don't Let Go", green) + + scanner := New(pool, logger, []string{root}, nil) + if _, err := scanner.Scan(ctx, nil); err != nil { + t.Fatalf("scan: %v", err) + } + albumsByGroup := func() map[string]int { + t.Helper() + rows, err := pool.Query(ctx, `SELECT a.release_group_mbid, count(t.id) FROM albums a + JOIN tracks t ON t.album_id = a.id WHERE a.title = 'Weezer' GROUP BY a.id, a.release_group_mbid`) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + out := map[string]int{} + for rows.Next() { + var g *string + var n int + if err := rows.Scan(&g, &n); err != nil { + t.Fatal(err) + } + if g == nil { + t.Fatal("an album with no release group") + } + out[*g] = n + } + return out + } + if got := albumsByGroup(); len(got) != 2 || got[blue] != 2 || got[green] != 1 { + t.Fatalf("albums by group = %v, want blue with 2 tracks and green with 1", got) + } + + // A bonus track tagged with no release group joins the album its folder + // holds, whichever row is older. + writeTestMP3(t, filepath.Join(root, "Weezer (2001)/02.mp3"), map[string]string{ + "TIT2": "Photograph (live)", "TPE1": "Weezer", "TALB": "Weezer", + }) + if _, err := scanner.Scan(ctx, nil); err != nil { + t.Fatalf("scan the bonus track: %v", err) + } + if got := albumsByGroup(); len(got) != 2 || got[blue] != 2 || got[green] != 2 { + t.Fatalf("albums by group = %v, want the ungrouped 2001 bonus track on green", got) + } + + // The green track as the old lookup left it: on the blue album's row, + // under an older tag-read version. + if _, err := pool.Exec(ctx, ` + WITH b AS (SELECT id FROM albums WHERE release_group_mbid = $1) + UPDATE tracks SET album_id = (SELECT id FROM b), tag_read_version = 6 WHERE file_path = $2`, blue, greenPath); err != nil { + t.Fatal(err) + } + if _, err := scanner.Scan(ctx, nil); err != nil { + t.Fatalf("rescan: %v", err) + } + if got := albumsByGroup(); len(got) != 2 || got[blue] != 2 || got[green] != 2 { + t.Errorf("after the re-read, albums by group = %v, want the green track on its own album", got) + } +} diff --git a/internal/library/scanrun.go b/internal/library/scanrun.go index f25081f8..752b8bd2 100644 --- a/internal/library/scanrun.go +++ b/internal/library/scanrun.go @@ -24,14 +24,22 @@ type LibraryStageTallies struct { Updated int `json:"updated"` Skipped int `json:"skipped"` Errored int `json:"errored"` - // Reconcile results (#2523). Surfaced in the scan record because a track - // disappearing from the library is something the operator should be able to + // Reconcile results. Surfaced in the scan record because a track + // disappearing from the library is something an admin should be able to // see happened, rather than discovering it when a mix comes up short. Missing int `json:"missing"` Restored int `json:"restored"` } -// MBIDBackfillStageTallies wires BackfillMBIDsResult into the scan_runs jsonb column. +func (st Stats) tallies() LibraryStageTallies { + return LibraryStageTallies{ + Scanned: st.Scanned, Added: st.Added, Updated: st.Updated, Skipped: st.Skipped, + Errored: st.Errored, Missing: st.Missing, Restored: st.Restored, + } +} + +// MBIDBackfillStageTallies wires BackfillMBIDsResult into the scan_runs jsonb +// column. type MBIDBackfillStageTallies struct { Processed int `json:"processed"` Healed int `json:"healed"` @@ -39,7 +47,8 @@ type MBIDBackfillStageTallies struct { Duplicates int `json:"duplicates"` } -// CoverEnrichStageTallies wires EnrichBatch tallies into the scan_runs jsonb column. +// CoverEnrichStageTallies wires EnrichBatch tallies into the scan_runs jsonb +// column. type CoverEnrichStageTallies struct { Processed int `json:"processed"` Succeeded int `json:"succeeded"` @@ -57,13 +66,12 @@ type ArtistArtEnrichStageTallies struct { // RunScanConfig assembles the knobs for the orchestrator. BackfillCap is // the album/artist-MBID-backfill batch cap (5000); EnrichCap / // ArtistEnrichCap gate cover/artist-art enrichment (0 = off, <0 = -// unbounded — #388 removed the global cover-art cap; remote providers -// self-throttle per provider). +// unbounded; remote providers pace themselves). type RunScanConfig struct { BackfillCap int EnrichCap int - ArtistEnrichCap int // M7 cover-sources: cap for the artist-art enrich stage - DataDir string // M7 cover-sources: where to write artist-art files + ArtistEnrichCap int // cap for the artist-art enrich stage + DataDir string // where to write artist-art files } // RunScan is the unified scan-chain orchestrator. Creates a scan_runs row, @@ -105,7 +113,7 @@ func RunScan( var lastStats Stats libPub := newStagePublisher(ctx, 500, time.Second, func() []byte { - blob, _ := json.Marshal(LibraryStageTallies(lastStats)) + blob, _ := json.Marshal(lastStats.tallies()) return blob }, func(blob []byte) error { @@ -228,7 +236,8 @@ func RunScan( } } - // Finalize: set finished_at + error message (or empty string if all stages succeeded). + // Finalize: set finished_at + error message (or empty string if all stages + // succeeded). errMsg := "" if firstErr != nil { errMsg = firstErr.Error() @@ -236,7 +245,8 @@ func RunScan( if ferr := q.FinishScanRun(ctx, dbq.FinishScanRunParams{ ID: row.ID, Column2: errMsg, }); ferr != nil { - // Even the finalize failed; log loudly, return the original error if any. + // Even the finalize failed; log loudly, return the original error if + // any. logger.Error("scan run finalize failed", "id", row.ID, "err", ferr) if firstErr != nil { return row.ID, firstErr diff --git a/internal/library/song_link_test.go b/internal/library/song_link_test.go index 97bf3aab..f42581ca 100644 --- a/internal/library/song_link_test.go +++ b/internal/library/song_link_test.go @@ -2,6 +2,8 @@ package library import ( "context" + "os" + "path/filepath" "testing" "github.com/jackc/pgx/v5/pgtype" @@ -78,7 +80,7 @@ func TestCrossReleaseCopiesCountAsOneSong_Integration(t *testing.T) { q := dbq.New(pool) f := newSongLinkFixture(t, pool) - res, err := ResolveDuplicates(ctx, pool, nil, "", nil, true) + res, err := ResolveDuplicates(ctx, pool, nil, "", true) if err != nil { t.Fatal(err) } @@ -118,7 +120,7 @@ func TestCrossReleaseCopiesCountAsOneSong_Integration(t *testing.T) { } // A second pass changes nothing. - if res, err := ResolveDuplicates(ctx, pool, nil, "", nil, true); err != nil || res.SongsLinked != 0 { + if res, err := ResolveDuplicates(ctx, pool, nil, "", true); err != nil || res.SongsLinked != 0 { t.Errorf("second pass linked %d (%v), want 0", res.SongsLinked, err) } @@ -149,13 +151,13 @@ func TestCrossReleaseCopiesCountAsOneSong_Integration(t *testing.T) { } } -// Without the operator's permission the resolver links nothing. +// With auto-resolve off the resolver links nothing. func TestCrossReleaseLinkNeedsAutoResolve_Integration(t *testing.T) { pool := newPool(t) ctx := context.Background() q := dbq.New(pool) f := newSongLinkFixture(t, pool) - if _, err := ResolveDuplicates(ctx, pool, nil, "", nil, false); err != nil { + if _, err := ResolveDuplicates(ctx, pool, nil, "", false); err != nil { t.Fatal(err) } keys := songKeys(t, q, f.single.ID, f.albumCut.ID) @@ -164,29 +166,41 @@ func TestCrossReleaseLinkNeedsAutoResolve_Integration(t *testing.T) { } } -// A merge keeps the removed copy's link to the song on another release. +// A merge keeps the removed copy's link to the song on another release, +// whichever copy's id names the song. That copy has no song_id of its own, +// and its link must survive whichever way the random ids fall, including when +// they make it the key. func TestMergeInheritsSongLink_Integration(t *testing.T) { - pool := newPool(t) - ctx := context.Background() - q := dbq.New(pool) - f := newSongLinkFixture(t, pool) - if _, err := q.LinkTracksAsSong(ctx, []pgtype.UUID{f.single.ID, f.albumCut.ID}); err != nil { - t.Fatal(err) - } - // A third, unlinked copy of the album cut stands in for a survivor. - other, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ - Title: "Feel Good Inc.", AlbumID: f.albumCut.AlbumID, ArtistID: f.albumCut.ArtistID, - DurationMs: 1000, FilePath: "/music/Link Artist/Demon Days/06 Feel Good Inc (1).mp3", FileSize: 90, FileFormat: "mp3", - }) - if err != nil { - t.Fatal(err) - } - if err := q.MergeInheritSongLink(ctx, dbq.MergeInheritSongLinkParams{SurvivorID: other.ID, LoserID: f.albumCut.ID}); err != nil { - t.Fatal(err) - } - keys := songKeys(t, q, other.ID, f.single.ID) - if keys[0] != keys[1] { - t.Error("survivor did not join the song") + for _, removeAlbumCut := range []bool{true, false} { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + f := newSongLinkFixture(t, pool) + if _, err := q.LinkTracksAsSong(ctx, []pgtype.UUID{f.single.ID, f.albumCut.ID}); err != nil { + t.Fatal(err) + } + loser, kept := f.albumCut, f.single + if !removeAlbumCut { + loser, kept = f.single, f.albumCut + } + // A third, unlinked copy of the removed track stands in for a survivor. + other, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: loser.Title, AlbumID: loser.AlbumID, ArtistID: loser.ArtistID, + DurationMs: 1000, FilePath: loser.FilePath + ".copy.mp3", FileSize: 90, FileFormat: "mp3", + }) + if err != nil { + t.Fatal(err) + } + if err := q.MergeInheritSongLink(ctx, dbq.MergeInheritSongLinkParams{SurvivorID: other.ID, LoserID: loser.ID}); err != nil { + t.Fatal(err) + } + if _, err := q.DeleteTrack(ctx, loser.ID); err != nil { + t.Fatal(err) + } + keys := songKeys(t, q, other.ID, kept.ID) + if keys[0] != keys[1] { + t.Errorf("removing the %s: survivor did not join the song", map[bool]string{true: "album cut", false: "single"}[removeAlbumCut]) + } } } @@ -206,3 +220,74 @@ func songKeys(t *testing.T, q *dbq.Queries, ids ...pgtype.UUID) []pgtype.UUID { } return out } + +// A cross-release group whose album holds the song twice (Humanz: Saturnz +// Barz as a FLAC and a YouTube mp3, beside the single): the album's +// extra copy is merged into its FLAC, the single stays, and the two copies +// left are linked as one song. +func TestResolveDuplicates_FoldsAnAlbumsExtraCopyInACrossReleaseGroup_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + dir := t.TempDir() + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "Gorillaz", SortName: "Gorillaz"}) + if err != nil { + t.Fatal(err) + } + // UpsertAlbum folds only by MBID, so each album is made once. + albums := map[string]dbq.Album{} + track := func(album, name, format string) dbq.Track { + t.Helper() + a, ok := albums[album] + if !ok { + var err error + if a, err = q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: album, SortTitle: album, ArtistID: artist.ID}); err != nil { + t.Fatal(err) + } + albums[album] = a + } + path := filepath.Join(dir, album, name) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte("audio"), 0o644); err != nil { + t.Fatal(err) + } + tr, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: "Saturnz Barz", AlbumID: a.ID, ArtistID: artist.ID, + DurationMs: 181000, FilePath: path, FileSize: 100, FileFormat: format, + }) + if err != nil { + t.Fatal(err) + } + return tr + } + flac := track("Humanz", "04 Saturnz Barz.flac", "flac") + mp3 := track("Humanz", "06 Saturnz Barz (feat. Popcaan).mp3", "mp3") + single := track("Saturnz Barz", "01 Saturnz Barz.mp3", "mp3") + var groupID pgtype.UUID + if err := pool.QueryRow(ctx, + `INSERT INTO duplicate_groups (member_key, tier) VALUES ('humanz-saturnz', 'exact') RETURNING id`, + ).Scan(&groupID); err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, `INSERT INTO duplicate_group_members (group_id, track_id) VALUES ($1, $2), ($1, $3), ($1, $4)`, + groupID, flac.ID, mp3.ID, single.ID); err != nil { + t.Fatal(err) + } + + res, err := ResolveDuplicates(ctx, pool, nil, "", true) + if err != nil { + t.Fatal(err) + } + if res.Merged != 1 || exists(mp3.FilePath) || !exists(flac.FilePath) || !exists(single.FilePath) { + t.Fatalf("result %+v, mp3 %v flac %v single %v; want only the album's mp3 merged away", + res, exists(mp3.FilePath), exists(flac.FilePath), exists(single.FilePath)) + } + if _, err := q.GetTrackByID(ctx, mp3.ID); err == nil { + t.Error("the album's extra copy is still a track") + } + if res.SongsLinked != 1 { + t.Errorf("songs linked = %d, want the FLAC and the single linked as one song", res.SongsLinked) + } +} diff --git a/internal/library/source_markers.go b/internal/library/source_markers.go index 0bc5a825..dec5ecf1 100644 --- a/internal/library/source_markers.go +++ b/internal/library/source_markers.go @@ -8,7 +8,7 @@ import ( // sourceMarker is one sign that a file was ripped from a video rather than // taken from a release: words a video title carries and an album track does -// not (#5410). +// not. // // Each pattern is used twice, so it is written in the subset that Postgres's // regex flavour and Go's RE2 read the same way: no \b (Postgres spells word @@ -21,9 +21,9 @@ type sourceMarker struct { pattern string } -// suspectSourceMarkers was calibrated against the operator's library on -// 2026-10-08. Two candidates were left out on purpose: -// - "live in"/"live at": 229 files, nearly all from real live albums. +// suspectSourceMarkers was calibrated against a real library. Two candidates +// were left out on purpose: +// - "live in"/"live at": nearly all from real live albums. // - a bare "reaction": it caught Beck's "Chain Reaction". The marker below // wants the phrases a reaction video actually uses. var suspectSourceMarkers = []sourceMarker{ diff --git a/internal/library/source_markers_test.go b/internal/library/source_markers_test.go index 2282a696..6eb5086d 100644 --- a/internal/library/source_markers_test.go +++ b/internal/library/source_markers_test.go @@ -5,7 +5,7 @@ import ( "testing" ) -// Real basenames from the operator's library (2026-10-08), each with the +// Real basenames, each with the // labels it should carry. The negatives are the near misses that shaped the // list: a song called "Chain Reaction", a live album, an ordinary track. func TestSourceMarkersFor(t *testing.T) { diff --git a/internal/library/stage_publisher_test.go b/internal/library/stage_publisher_test.go index ffbe969d..f04e651e 100644 --- a/internal/library/stage_publisher_test.go +++ b/internal/library/stage_publisher_test.go @@ -81,7 +81,8 @@ func TestStagePublisher_Tick_SwallowsPersistErrors(t *testing.T) { }, ) - // Ticks should not panic, should keep firing the persist (which keeps erroring). + // Ticks should not panic, should keep firing the persist (which keeps + // erroring). for i := 0; i < 5; i++ { pub.Tick() } diff --git a/internal/library/tracknumber.go b/internal/library/tracknumber.go new file mode 100644 index 00000000..51d27b80 --- /dev/null +++ b/internal/library/tracknumber.go @@ -0,0 +1,40 @@ +package library + +import ( + "strconv" + "strings" + + "github.com/dhowden/tag" +) + +// ReadTrackAndDisc reads a file's track and disc numbers. dhowden/tag reads a +// Vorbis TRACKNUMBER or DISCNUMBER only as a bare number, but taggers also +// write "3/20" there (as Minstrel's own importer once did), and that +// reads as 0. For Vorbis the number before the slash is taken instead. +func ReadTrackAndDisc(m tag.Metadata) (track, disc int) { + track, _ = m.Track() + disc, _ = m.Disc() + if m.Format() != tag.VORBIS { + return track, disc + } + raw := m.Raw() + if track == 0 { + track = leadingNumber(raw["tracknumber"]) + } + if disc == 0 { + disc = leadingNumber(raw["discnumber"]) + } + return track, disc +} + +// leadingNumber is the number in a tag value of "3" or "3/20"; 0 for +// anything else. +func leadingNumber(v any) int { + s, _ := v.(string) + before, _, _ := strings.Cut(s, "/") + n, err := strconv.Atoi(strings.TrimSpace(before)) + if err != nil || n < 0 { + return 0 + } + return n +} diff --git a/internal/library/trigger_test.go b/internal/library/trigger_test.go index fbbec894..24f20049 100644 --- a/internal/library/trigger_test.go +++ b/internal/library/trigger_test.go @@ -115,7 +115,8 @@ func TestTryStartScan_StaleInFlight_Reaps(t *testing.T) { t.Errorf("existing = %v, want nil", existing) } - // Verify the stale row was reaped: finished_at non-NULL, error_message contains "reaped". + // Verify the stale row was reaped: finished_at non-NULL, error_message + // contains "reaped". var finishedAt pgtype.Timestamptz var errMsg *string if err := pool.QueryRow(context.Background(), diff --git a/internal/library/vorbisgenre.go b/internal/library/vorbisgenre.go index fa3027ff..590ba10c 100644 --- a/internal/library/vorbisgenre.go +++ b/internal/library/vorbisgenre.go @@ -17,10 +17,9 @@ import ( // // dhowden/tag stores comments in a map keyed by field name // (vorbis.go: m.c[strings.ToLower(k)] = v), so each repeat overwrites the last -// and only the final value survives. On the operator's library that was 3,658 -// of 4,092 FLAC files: Kupla's Life Forms declares eight genres and was stored -// as "Instrumental Hip Hop" alone (#2500). Nothing looked corrupt; the other -// seven were simply never seen by browse or the taste profile. +// and only the final value survives. An album that declares eight genres +// would be stored under the last alone; nothing would look corrupt, and the +// other seven would never be seen by browse or the taste profile. // // The comment block is read here directly, as the ID3v2 genre frame is in // id3v2genre.go. Every other field still comes from dhowden/tag. @@ -43,7 +42,8 @@ func readFLACGenreValues(rs io.ReadSeeker) ([]string, error) { if _, err := io.ReadFull(rs, magic[:]); err != nil { return nil, errNoGenreFrame } - // Some taggers prepend an ID3v2 tag to a FLAC. The FLAC stream starts after it. + // Some taggers prepend an ID3v2 tag to a FLAC. The FLAC stream starts after + // it. if string(magic[:3]) == "ID3" { var rest [6]byte if _, err := io.ReadFull(rs, rest[:]); err != nil { @@ -172,7 +172,8 @@ func oggCommentGenres(packet []byte) ([]string, error) { // vorbisCommentGenres reads a comment block (vendor string, then a count of // length-prefixed "KEY=value" entries, little-endian lengths throughout) and // returns the GENRE values in file order. Field names are case-insensitive by -// spec. Anything after the last entry, such as Vorbis's framing bit, is ignored. +// spec. Anything after the last entry, such as Vorbis's framing bit, is +// ignored. func vorbisCommentGenres(b []byte) ([]string, error) { next := func() ([]byte, bool) { if len(b) < 4 { diff --git a/internal/library/vorbisgenre_test.go b/internal/library/vorbisgenre_test.go index 1fe0f0be..9277c92c 100644 --- a/internal/library/vorbisgenre_test.go +++ b/internal/library/vorbisgenre_test.go @@ -161,7 +161,7 @@ func genreFields(values ...string) []string { return out } -// The #2500 regression, shaped like the operator's Kupla - Life Forms files: +// Shaped like a real multi-genre FLAC: // eight GENRE fields, of which dhowden/tag keeps the last. func TestExtractGenres_FLACRepeatedFields(t *testing.T) { data := buildFLAC(false, append([]string{"TITLE=Eons"}, genreFields(lifeForms...)...)...) @@ -170,7 +170,7 @@ func TestExtractGenres_FLACRepeatedFields(t *testing.T) { if err != nil { t.Fatalf("tag.ReadFrom: %v", err) } - // Confirm the upstream behaviour this fix exists for is still present. + // Confirm the upstream behaviour this reader works around is still present. if got := meta.Genre(); got != "Instrumental Hip Hop" { t.Logf("note: dhowden/tag no longer keeps only the last GENRE (got %q)", got) } diff --git a/internal/library/watcher.go b/internal/library/watcher.go index dc6946c7..c22a4c87 100644 --- a/internal/library/watcher.go +++ b/internal/library/watcher.go @@ -60,25 +60,23 @@ func classifyEvent(op fsnotify.Op, isDir, isAudio bool) watchAction { // because the walk alone only ever gave one of them: the walk re-visits every // path that EXISTS, which catches additions and edits, and the reconcile pass // that runs with it compares the whole tracks table against what the walk saw, -// which is what catches removals (#2523). classifyEvent still ignores fsnotify +// which is what catches removals. classifyEvent still ignores fsnotify // removals by design, so a deleted file surfaces at the next scan, not the // next inotify event. type Watcher struct { scanner *Scanner enricher *coverart.Enricher logger *slog.Logger - roots []string debounce time.Duration } -func NewWatcher( - scanner *Scanner, enricher *coverart.Enricher, logger *slog.Logger, roots []string, -) *Watcher { +// NewWatcher watches the scanner's folders, and moves to the new ones when +// they change in Settings. +func NewWatcher(scanner *Scanner, enricher *coverart.Enricher, logger *slog.Logger) *Watcher { return &Watcher{ scanner: scanner, enricher: enricher, logger: logger, - roots: roots, debounce: watcherDebounce, } } @@ -87,8 +85,32 @@ func NewWatcher( // fsnotify watcher cannot be created at all; per-path watch failures and // fsnotify errors are logged and tolerated. func (w *Watcher) Run(ctx context.Context) error { - if len(w.roots) == 0 { - w.logger.Info("library watcher: no scan paths configured, watcher disabled") + for { + if err := w.watch(ctx, w.scanner.Roots()); err != nil { + return err + } + if ctx.Err() != nil { + return nil + } + } +} + +// watch watches roots until ctx is cancelled or the folders change. +func (w *Watcher) watch(ctx context.Context, roots []string) error { + ctx, cancel := context.WithCancel(ctx) + defer cancel() + go func() { + select { + case <-w.scanner.RootsChanged(): + w.logger.Info("library watcher: folders changed, restarting") + cancel() + case <-ctx.Done(): + } + }() + + if len(roots) == 0 { + w.logger.Info("library watcher: no scan paths configured, watcher idle") + <-ctx.Done() return nil } fsw, err := fsnotify.NewWatcher() @@ -98,10 +120,10 @@ func (w *Watcher) Run(ctx context.Context) error { } defer func() { _ = fsw.Close() }() - for _, root := range w.roots { + for _, root := range roots { w.addRecursive(fsw, root) } - w.logger.Info("library watcher: started", "roots", len(w.roots)) + w.logger.Info("library watcher: started", "roots", len(roots)) batches := make(chan []string, batchQueueDepth) go w.processLoop(ctx, batches) diff --git a/internal/lidarr/client.go b/internal/lidarr/client.go deleted file mode 100644 index 9d90cc51..00000000 --- a/internal/lidarr/client.go +++ /dev/null @@ -1,511 +0,0 @@ -package lidarr - -import ( - "bytes" - "context" - "encoding/json" - "errors" - "fmt" - "io" - "net/http" - "net/url" - "strconv" - "strings" - "time" -) - -// Client wraps Lidarr's v1 HTTP API. BaseURL is the Lidarr instance -// (e.g. http://lidarr.lan:8686), APIKey comes from Lidarr's settings. -type Client struct { - BaseURL string - APIKey string - HTTP *http.Client -} - -// NewClient builds a Client with sensible HTTP defaults (30s timeout, -// matching the pattern in internal/scrobble/listenbrainz). Call sites -// that need a custom client can construct the Client struct directly. -func NewClient(baseURL, apiKey string) *Client { - return &Client{ - BaseURL: baseURL, - APIKey: apiKey, - HTTP: &http.Client{Timeout: 30 * time.Second}, - } -} - -// get executes an authenticated GET request and maps HTTP status codes to -// typed errors. The caller is responsible for closing the returned body. -func (c *Client) get(ctx context.Context, path string, q url.Values) (*http.Response, error) { - u, err := url.Parse(c.BaseURL) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) - } - u.Path = strings.TrimRight(u.Path, "/") + path - if q != nil { - u.RawQuery = q.Encode() - } - req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil) - if err != nil { - return nil, err - } - req.Header.Set("X-Api-Key", c.APIKey) - resp, err := c.HTTP.Do(req) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) - } - if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden { - _ = resp.Body.Close() - return nil, ErrAuthFailed - } - if resp.StatusCode >= 500 { - body := readBodySnippet(resp.Body) - _ = resp.Body.Close() - return nil, fmt.Errorf("%w: %d: %s", ErrServerError, resp.StatusCode, body) - } - if resp.StatusCode >= 400 { - body := readBodySnippet(resp.Body) - _ = resp.Body.Close() - return nil, fmt.Errorf("%w: %d: %s", ErrLookupFailed, resp.StatusCode, body) - } - return resp, nil -} - -// readBodySnippet reads up to 512 bytes from an error response body so the -// wrapped error message can include Lidarr's actual complaint (e.g. -// `{"errorMessage":"Artist already exists in your library"}`). Without this, -// callers see only the status-code bucket and can't diagnose without -// cross-referencing Lidarr's own logs. -func readBodySnippet(r io.Reader) string { - buf, _ := io.ReadAll(io.LimitReader(r, 512)) - return strings.TrimSpace(string(buf)) -} - -// post is the shared POST helper. body must be marshaled JSON. It maps HTTP -// status codes to typed errors; the caller is responsible for closing the -// returned body. -func (c *Client) post(ctx context.Context, path string, body []byte) (*http.Response, error) { - return c.bodyRequest(ctx, http.MethodPost, path, body) -} - -// put is the shared PUT helper. Body must be marshaled JSON. Mirrors post() — -// same status-code mapping, same caller-closes-body contract. -func (c *Client) put(ctx context.Context, path string, body []byte) (*http.Response, error) { - return c.bodyRequest(ctx, http.MethodPut, path, body) -} - -// bodyRequest is the shared core for POST and PUT. Lidarr's monitor toggle -// is PUT-shaped, so factoring it out keeps the post/put helpers from -// duplicating the status-code mapping. -func (c *Client) bodyRequest(ctx context.Context, method, path string, body []byte) (*http.Response, error) { - u, err := url.Parse(c.BaseURL) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) - } - u.Path = strings.TrimRight(u.Path, "/") + path - req, err := http.NewRequestWithContext(ctx, method, u.String(), bytes.NewReader(body)) - if err != nil { - return nil, err - } - req.Header.Set("X-Api-Key", c.APIKey) - req.Header.Set("Content-Type", "application/json") - resp, err := c.HTTP.Do(req) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) - } - if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden { - _ = resp.Body.Close() - return nil, ErrAuthFailed - } - if resp.StatusCode >= 500 { - body := readBodySnippet(resp.Body) - _ = resp.Body.Close() - return nil, fmt.Errorf("%w: %d: %s", ErrServerError, resp.StatusCode, body) - } - if resp.StatusCode >= 400 { - body := readBodySnippet(resp.Body) - _ = resp.Body.Close() - return nil, fmt.Errorf("%w: %d: %s", ErrLookupFailed, resp.StatusCode, body) - } - return resp, nil -} - -// LookupArtist hits GET /api/v1/artist/lookup?term=. Returns normalized -// LookupResults; Secondary is "{first genre} · {albumCount} albums" with -// missing parts skipped. -func (c *Client) LookupArtist(ctx context.Context, term string) ([]LookupResult, error) { - resp, err := c.get(ctx, "/api/v1/artist/lookup", url.Values{"term": {term}}) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - var raw []struct { - ForeignArtistID string `json:"foreignArtistId"` - ArtistName string `json:"artistName"` - Genres []string `json:"genres"` - AlbumCount int `json:"albumCount"` - Images []lidarrImage `json:"images"` - } - if err := json.Unmarshal(body, &raw); err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - out := make([]LookupResult, 0, len(raw)) - for _, r := range raw { - secondary := "" - if len(r.Genres) > 0 { - secondary = r.Genres[0] - } - if r.AlbumCount > 0 { - if secondary != "" { - secondary += " · " - } - secondary += strconv.Itoa(r.AlbumCount) + " albums" - } - out = append(out, LookupResult{ - MBID: r.ForeignArtistID, - Name: r.ArtistName, - Secondary: secondary, - ImageURL: pickPosterImage(r.Images), - }) - } - return out, nil -} - -// LookupAlbum hits GET /api/v1/album/lookup?term=. Returns normalized -// LookupResults; Secondary is "{artistName} · {year} · {trackCount} tracks". -func (c *Client) LookupAlbum(ctx context.Context, term string) ([]LookupResult, error) { - resp, err := c.get(ctx, "/api/v1/album/lookup", url.Values{"term": {term}}) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - var raw []struct { - ForeignAlbumID string `json:"foreignAlbumId"` - ForeignArtistID string `json:"foreignArtistId"` - Title string `json:"title"` - ArtistName string `json:"artistName"` - ReleaseDate string `json:"releaseDate"` - TrackCount int `json:"trackCount"` - Images []lidarrImage `json:"images"` - } - if err := json.Unmarshal(body, &raw); err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - out := make([]LookupResult, 0, len(raw)) - for _, r := range raw { - year := "" - if len(r.ReleaseDate) >= 4 { - year = r.ReleaseDate[:4] - } - secondary := "" - if r.ArtistName != "" { - secondary = r.ArtistName - } - if year != "" { - if secondary != "" { - secondary += " · " - } - secondary += year - } - if r.TrackCount > 0 { - if secondary != "" { - secondary += " · " - } - secondary += strconv.Itoa(r.TrackCount) + " tracks" - } - out = append(out, LookupResult{ - MBID: r.ForeignAlbumID, - ArtistMBID: r.ForeignArtistID, - Name: r.Title, - Secondary: secondary, - ImageURL: pickPosterImage(r.Images), - }) - } - return out, nil -} - -// LookupTrack hits GET /api/v1/track/lookup?term=. Lidarr's track lookup is -// per-album under the hood. Secondary is "{albumTitle} · {artistName}". -func (c *Client) LookupTrack(ctx context.Context, term string) ([]LookupResult, error) { - resp, err := c.get(ctx, "/api/v1/track/lookup", url.Values{"term": {term}}) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - var raw []struct { - ForeignTrackID string `json:"foreignTrackId"` - ForeignAlbumID string `json:"foreignAlbumId"` - ForeignArtistID string `json:"foreignArtistId"` - Title string `json:"title"` - AlbumTitle string `json:"albumTitle"` - ArtistName string `json:"artistName"` - } - if err := json.Unmarshal(body, &raw); err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - out := make([]LookupResult, 0, len(raw)) - for _, r := range raw { - secondary := r.AlbumTitle - if r.ArtistName != "" { - if secondary != "" { - secondary += " · " - } - secondary += r.ArtistName - } - out = append(out, LookupResult{ - MBID: r.ForeignTrackID, - ArtistMBID: r.ForeignArtistID, - AlbumMBID: r.ForeignAlbumID, - Name: r.Title, - Secondary: secondary, - }) - } - return out, nil -} - -// AddArtist posts to POST /api/v1/artist. MonitorAll=true monitors every -// album ("all"); false monitors only releases from now on ("future"). New -// releases are monitored either way. Returns nil on 2xx; typed error -// otherwise. -// -// All four of artistName, foreignArtistId, qualityProfileId, and -// metadataProfileId are required by Lidarr. Omitting any one produces a -// 400 with field-level validation messages (e.g. "'Metadata Profile Id' -// must be greater than '0'"). -// -// The monitor choice belongs in addOptions (AddArtistOptions, a -// MonitoringOptions). ArtistResource has no top-level `monitor`, so a -// choice sent there is dropped, AddOptions.Monitor stays Unknown and -// Lidarr skips applying it (#5239). -func (c *Client) AddArtist(ctx context.Context, p AddArtistParams) error { - monitor := "future" - if p.MonitorAll { - monitor = "all" - } - body, err := json.Marshal(map[string]any{ - "foreignArtistId": p.ForeignArtistID, - "artistName": p.ArtistName, - "qualityProfileId": p.QualityProfileID, - "metadataProfileId": p.MetadataProfileID, - "rootFolderPath": p.RootFolderPath, - "monitored": true, - "monitorNewItems": "all", - "addOptions": map[string]any{ - "monitor": monitor, - "monitored": true, - "searchForMissingAlbums": true, - }, - }) - if err != nil { - return fmt.Errorf("%w: marshal: %v", ErrInvalidPayload, err) - } - resp, err := c.post(ctx, "/api/v1/artist", body) - if err != nil { - if isAlreadyExists(err) { - return ErrAlreadyExists - } - return err - } - _ = resp.Body.Close() - return nil -} - -// isAlreadyExists detects Lidarr's 400 "already in your library" rejection -// from an add endpoint. The post() helper buckets it as ErrLookupFailed -// but embeds Lidarr's errorMessage body snippet; matching the phrase lets -// the idempotent retry path treat a re-add as success instead of error. -func isAlreadyExists(err error) bool { - if !errors.Is(err, ErrLookupFailed) { - return false - } - m := strings.ToLower(err.Error()) - return strings.Contains(m, "already exists") || - strings.Contains(m, "already been added") || - strings.Contains(m, "already in your library") -} - -// AddAlbum adds one album to Lidarr the way Lidarr's own UI does: look the -// album up by MBID, then POST that album resource back with `monitored`, -// `addOptions` and — when Lidarr doesn't have the artist yet — the -// artist's profiles, root folder and monitoring filled in. -// -// POST /api/v1/album validates `artist` as a nested resource (#5234: a -// flat payload is refused with "'Artist' must not be empty"), and the -// lookup is what supplies it: the existing Lidarr artist when there is one, -// otherwise a new one carrying its metadata. A new artist monitors only -// this album (monitor "none" with albumsToMonitor, which Lidarr applies in -// preference to the monitor type) and no future releases, so approving one -// album never subscribes the library to the artist's whole catalogue. -// -// Returns nil on 2xx, ErrAlreadyExists when Lidarr already has the album, -// ErrNotFound when Lidarr's metadata has no album under that MBID. -func (c *Client) AddAlbum(ctx context.Context, p AddAlbumParams) error { - album, err := c.lookupAlbumResource(ctx, p.ForeignAlbumID) - if err != nil { - return err - } - album["monitored"] = true - album["addOptions"] = map[string]any{"searchForNewAlbum": true} - - artist, _ := album["artist"].(map[string]any) - if artist == nil { - artist = map[string]any{ - "foreignArtistId": p.ForeignArtistID, - "artistName": p.ArtistName, - } - album["artist"] = artist - } - if id, _ := artist["id"].(float64); id == 0 { - artist["qualityProfileId"] = p.QualityProfileID - artist["metadataProfileId"] = p.MetadataProfileID - artist["rootFolderPath"] = p.RootFolderPath - artist["monitored"] = true - artist["monitorNewItems"] = "none" - artist["addOptions"] = map[string]any{ - "monitor": "none", - "albumsToMonitor": []string{p.ForeignAlbumID}, - "monitored": true, - "searchForMissingAlbums": false, - } - } - - body, err := json.Marshal(album) - if err != nil { - return fmt.Errorf("%w: marshal: %v", ErrInvalidPayload, err) - } - resp, err := c.post(ctx, "/api/v1/album", body) - if err != nil { - if isAlreadyExists(err) { - return ErrAlreadyExists - } - return err - } - _ = resp.Body.Close() - return nil -} - -// ListMetadataProfiles hits GET /api/v1/metadataprofile and returns the -// list of metadata profiles configured in Lidarr. The Approve flow uses -// the first profile as a default when the operator hasn't pinned one. -func (c *Client) ListMetadataProfiles(ctx context.Context) ([]MetadataProfile, error) { - resp, err := c.get(ctx, "/api/v1/metadataprofile", nil) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - var raw []struct { - ID int `json:"id"` - Name string `json:"name"` - } - if err := json.Unmarshal(body, &raw); err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - out := make([]MetadataProfile, len(raw)) - for i, r := range raw { - out[i] = MetadataProfile{ID: r.ID, Name: r.Name} - } - return out, nil -} - -// ListQualityProfiles hits GET /api/v1/qualityprofile and returns the full -// list of quality profiles configured in Lidarr. -func (c *Client) ListQualityProfiles(ctx context.Context) ([]QualityProfile, error) { - resp, err := c.get(ctx, "/api/v1/qualityprofile", nil) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - var raw []struct { - ID int `json:"id"` - Name string `json:"name"` - } - if err := json.Unmarshal(body, &raw); err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - out := make([]QualityProfile, len(raw)) - for i, r := range raw { - out[i] = QualityProfile{ID: r.ID, Name: r.Name} - } - return out, nil -} - -// ListRootFolders hits GET /api/v1/rootfolder and returns the list of root -// folders configured in Lidarr. -func (c *Client) ListRootFolders(ctx context.Context) ([]RootFolder, error) { - resp, err := c.get(ctx, "/api/v1/rootfolder", nil) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - body, err := io.ReadAll(resp.Body) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - var raw []struct { - Path string `json:"path"` - Accessible bool `json:"accessible"` - FreeSpace int64 `json:"freeSpace"` - } - if err := json.Unmarshal(body, &raw); err != nil { - return nil, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - out := make([]RootFolder, len(raw)) - for i, r := range raw { - out[i] = RootFolder{Path: r.Path, Accessible: r.Accessible, FreeSpace: r.FreeSpace} - } - return out, nil -} - -// Ping hits GET /api/v1/system/status to verify connectivity and returns -// the Lidarr version string. -func (c *Client) Ping(ctx context.Context) (PingResult, error) { - resp, err := c.get(ctx, "/api/v1/system/status", nil) - if err != nil { - return PingResult{}, err - } - defer func() { _ = resp.Body.Close() }() - body, err := io.ReadAll(resp.Body) - if err != nil { - return PingResult{}, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - var raw struct { - Version string `json:"version"` - } - if err := json.Unmarshal(body, &raw); err != nil { - return PingResult{}, fmt.Errorf("%w: %v", ErrInvalidPayload, err) - } - return PingResult{Version: raw.Version}, nil -} - -// pickPosterImage returns the remote URL (preferred) or local URL for the -// first image with coverType=="poster". Returns "" when none match. -func pickPosterImage(imgs []lidarrImage) string { - for _, img := range imgs { - if img.CoverType == "poster" { - if img.RemoteURL != "" { - return img.RemoteURL - } - return img.URL - } - } - return "" -} diff --git a/internal/lidarr/client_test.go b/internal/lidarr/client_test.go deleted file mode 100644 index 43e7b0af..00000000 --- a/internal/lidarr/client_test.go +++ /dev/null @@ -1,892 +0,0 @@ -package lidarr - -import ( - "context" - "encoding/json" - "errors" - "net/http" - "net/http/httptest" - "os" - "testing" -) - -// newTestClient creates an httptest.Server and a Client wired to it. -func newTestClient(handler http.HandlerFunc) (*Client, *httptest.Server) { - srv := httptest.NewServer(handler) - return &Client{BaseURL: srv.URL, APIKey: "key123", HTTP: srv.Client()}, srv -} - -// mustReadFixture reads a file from testdata/ or fatals the test. -func mustReadFixture(t *testing.T, name string) []byte { - t.Helper() - b, err := os.ReadFile("testdata/" + name) - if err != nil { - t.Fatalf("read fixture %q: %v", name, err) - } - return b -} - -// --- LookupArtist --- - -func TestLookupArtist_HappyPath(t *testing.T) { - body := mustReadFixture(t, "lookup_artist.json") - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if got := r.Header.Get("X-Api-Key"); got != "key123" { - t.Errorf("X-Api-Key = %q, want key123", got) - } - if r.URL.Path != "/api/v1/artist/lookup" { - t.Errorf("path = %q, want /api/v1/artist/lookup", r.URL.Path) - } - if got := r.URL.Query().Get("term"); got != "boards" { - t.Errorf("term = %q, want boards", got) - } - _, _ = w.Write(body) - }) - defer srv.Close() - - got, err := c.LookupArtist(context.Background(), "boards") - if err != nil { - t.Fatalf("LookupArtist: %v", err) - } - if len(got) != 2 { - t.Fatalf("len = %d, want 2", len(got)) - } - - // First result — rich data. - if got[0].MBID != "069b64b6-7884-4f6a-94cc-e4c1d6c87a01" { - t.Errorf("MBID = %q", got[0].MBID) - } - if got[0].Name != "Boards of Canada" { - t.Errorf("Name = %q", got[0].Name) - } - if got[0].Secondary != "Electronic · 18 albums" { - t.Errorf("Secondary = %q, want %q", got[0].Secondary, "Electronic · 18 albums") - } - if got[0].ImageURL != "https://example.invalid/boc.jpg" { - t.Errorf("ImageURL = %q", got[0].ImageURL) - } - - // Second result — empty genres and zero album count → empty secondary. - if got[1].Secondary != "" { - t.Errorf("expected empty secondary for no genres + 0 albums; got %q", got[1].Secondary) - } - if got[1].ImageURL != "" { - t.Errorf("expected empty ImageURL; got %q", got[1].ImageURL) - } -} - -func TestLookupArtist_AuthFailed(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusUnauthorized) - }) - defer srv.Close() - _, err := c.LookupArtist(context.Background(), "boards") - if !errors.Is(err, ErrAuthFailed) { - t.Fatalf("err = %v, want ErrAuthFailed", err) - } -} - -func TestLookupArtist_Forbidden(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusForbidden) - }) - defer srv.Close() - _, err := c.LookupArtist(context.Background(), "boards") - if !errors.Is(err, ErrAuthFailed) { - t.Fatalf("err = %v, want ErrAuthFailed (403 must map to auth)", err) - } -} - -func TestLookupArtist_ServerError(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusInternalServerError) - }) - defer srv.Close() - _, err := c.LookupArtist(context.Background(), "boards") - if !errors.Is(err, ErrServerError) { - t.Fatalf("err = %v, want ErrServerError", err) - } -} - -func TestLookupArtist_Unreachable(t *testing.T) { - c := &Client{BaseURL: "http://127.0.0.1:1", APIKey: "x", HTTP: &http.Client{}} - _, err := c.LookupArtist(context.Background(), "boards") - if !errors.Is(err, ErrUnreachable) { - t.Fatalf("err = %v, want ErrUnreachable", err) - } -} - -func TestLookupArtist_BadJSON(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write([]byte(`not json`)) - }) - defer srv.Close() - _, err := c.LookupArtist(context.Background(), "boards") - if !errors.Is(err, ErrInvalidPayload) { - t.Fatalf("err = %v, want ErrInvalidPayload", err) - } -} - -// --- LookupAlbum --- - -func TestLookupAlbum_HappyPath(t *testing.T) { - body := mustReadFixture(t, "lookup_album.json") - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/album/lookup" { - t.Errorf("path = %q, want /api/v1/album/lookup", r.URL.Path) - } - if got := r.URL.Query().Get("term"); got != "geogaddi" { - t.Errorf("term = %q, want geogaddi", got) - } - _, _ = w.Write(body) - }) - defer srv.Close() - - got, err := c.LookupAlbum(context.Background(), "geogaddi") - if err != nil { - t.Fatalf("LookupAlbum: %v", err) - } - if len(got) != 2 { - t.Fatalf("len = %d, want 2", len(got)) - } - - // First result: year from releaseDate, trackCount in secondary. - if got[0].MBID != "a1b2c3d4-e5f6-7890-abcd-ef1234567890" { - t.Errorf("MBID = %q", got[0].MBID) - } - if got[0].ArtistMBID != "069b64b6-7884-4f6a-94cc-e4c1d6c87a01" { - t.Errorf("ArtistMBID = %q, want 069b64b6-7884-4f6a-94cc-e4c1d6c87a01", got[0].ArtistMBID) - } - if got[0].AlbumMBID != "" { - t.Errorf("AlbumMBID = %q, want empty on album result", got[0].AlbumMBID) - } - if got[0].Name != "Music Has the Right to Children" { - t.Errorf("Name = %q", got[0].Name) - } - want0 := "Boards of Canada · 1998 · 18 tracks" - if got[0].Secondary != want0 { - t.Errorf("Secondary = %q, want %q", got[0].Secondary, want0) - } - // Poster image from second image entry. - if got[0].ImageURL != "https://example.invalid/mhtrtc-poster.jpg" { - t.Errorf("ImageURL = %q", got[0].ImageURL) - } - - // Second result. - want1 := "Boards of Canada · 2002 · 23 tracks" - if got[1].Secondary != want1 { - t.Errorf("Secondary = %q, want %q", got[1].Secondary, want1) - } - if got[1].ImageURL != "" { - t.Errorf("expected empty ImageURL; got %q", got[1].ImageURL) - } -} - -func TestLookupAlbum_BadJSON(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write([]byte(`{}`)) - }) - defer srv.Close() - _, err := c.LookupAlbum(context.Background(), "x") - if !errors.Is(err, ErrInvalidPayload) { - t.Fatalf("err = %v, want ErrInvalidPayload (object, not array)", err) - } -} - -// --- LookupTrack --- - -func TestLookupTrack_HappyPath(t *testing.T) { - body := mustReadFixture(t, "lookup_track.json") - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/track/lookup" { - t.Errorf("path = %q, want /api/v1/track/lookup", r.URL.Path) - } - _, _ = w.Write(body) - }) - defer srv.Close() - - got, err := c.LookupTrack(context.Background(), "roygbiv") - if err != nil { - t.Fatalf("LookupTrack: %v", err) - } - if len(got) != 2 { - t.Fatalf("len = %d, want 2", len(got)) - } - if got[0].Name != "Roygbiv" { - t.Errorf("Name = %q", got[0].Name) - } - want := "Music Has the Right to Children · Boards of Canada" - if got[0].Secondary != want { - t.Errorf("Secondary = %q, want %q", got[0].Secondary, want) - } - if got[0].ArtistMBID != "069b64b6-7884-4f6a-94cc-e4c1d6c87a01" { - t.Errorf("ArtistMBID = %q, want 069b64b6-7884-4f6a-94cc-e4c1d6c87a01", got[0].ArtistMBID) - } - if got[0].AlbumMBID != "a1b2c3d4-e5f6-7890-abcd-ef1234567890" { - t.Errorf("AlbumMBID = %q, want a1b2c3d4-e5f6-7890-abcd-ef1234567890", got[0].AlbumMBID) - } - if got[0].ImageURL != "" { - t.Errorf("tracks carry no images; ImageURL = %q", got[0].ImageURL) - } -} - -func TestLookupTrack_NoArtistName(t *testing.T) { - // Verify secondary degrades gracefully when artistName is absent. - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write([]byte(`[{"foreignTrackId":"abc","title":"X","albumTitle":"AlbumY","artistName":""}]`)) - }) - defer srv.Close() - got, err := c.LookupTrack(context.Background(), "x") - if err != nil { - t.Fatalf("err: %v", err) - } - if got[0].Secondary != "AlbumY" { - t.Errorf("Secondary = %q, want %q", got[0].Secondary, "AlbumY") - } -} - -// --- AddArtist --- - -func TestAddArtist_PostsCorrectBody(t *testing.T) { - var decoded map[string]any - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodPost { - t.Errorf("method = %q, want POST", r.Method) - } - if r.URL.Path != "/api/v1/artist" { - t.Errorf("path = %q, want /api/v1/artist", r.URL.Path) - } - if ct := r.Header.Get("Content-Type"); ct != "application/json" { - t.Errorf("Content-Type = %q, want application/json", ct) - } - if err := json.NewDecoder(r.Body).Decode(&decoded); err != nil { - t.Errorf("decode body: %v", err) - } - w.WriteHeader(http.StatusCreated) - }) - defer srv.Close() - - err := c.AddArtist(context.Background(), AddArtistParams{ - ForeignArtistID: "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - QualityProfileID: 2, - RootFolderPath: "/music", - MonitorAll: true, - }) - if err != nil { - t.Fatalf("AddArtist: %v", err) - } - - if decoded["foreignArtistId"] != "069b64b6-7884-4f6a-94cc-e4c1d6c87a01" { - t.Errorf("foreignArtistId = %v", decoded["foreignArtistId"]) - } - if decoded["qualityProfileId"] != float64(2) { - t.Errorf("qualityProfileId = %v", decoded["qualityProfileId"]) - } - if decoded["rootFolderPath"] != "/music" { - t.Errorf("rootFolderPath = %v", decoded["rootFolderPath"]) - } - if decoded["monitored"] != true { - t.Errorf("monitored = %v, want true", decoded["monitored"]) - } - if decoded["monitorNewItems"] != "all" { - t.Errorf("monitorNewItems = %v, want all", decoded["monitorNewItems"]) - } - // #5239: ArtistResource has no top-level monitor; Lidarr reads the - // choice from addOptions and silently drops it anywhere else. - if _, set := decoded["monitor"]; set { - t.Errorf("monitor sent at top level (%v), where Lidarr ignores it", decoded["monitor"]) - } - opts, ok := decoded["addOptions"].(map[string]any) - if !ok { - t.Fatalf("addOptions missing or wrong type: %T", decoded["addOptions"]) - } - if opts["monitor"] != "all" { - t.Errorf("addOptions.monitor = %v, want all (MonitorAll=true)", opts["monitor"]) - } - if opts["monitored"] != true { - t.Errorf("addOptions.monitored = %v, want true", opts["monitored"]) - } - if opts["searchForMissingAlbums"] != true { - t.Errorf("addOptions.searchForMissingAlbums = %v, want true", opts["searchForMissingAlbums"]) - } -} - -func TestAddArtist_MonitorFuture(t *testing.T) { - var decoded map[string]any - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - _ = json.NewDecoder(r.Body).Decode(&decoded) - w.WriteHeader(http.StatusCreated) - }) - defer srv.Close() - - _ = c.AddArtist(context.Background(), AddArtistParams{MonitorAll: false}) - if opts, _ := decoded["addOptions"].(map[string]any); opts["monitor"] != "future" { - t.Errorf("addOptions.monitor = %v, want future (MonitorAll=false)", opts["monitor"]) - } -} - -func TestAddArtist_ServerError(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusInternalServerError) - }) - defer srv.Close() - err := c.AddArtist(context.Background(), AddArtistParams{}) - if !errors.Is(err, ErrServerError) { - t.Fatalf("err = %v, want ErrServerError", err) - } -} - -// --- AddAlbum --- - -const ( - addAlbumMBID = "a1b2c3d4-e5f6-7890-abcd-ef1234567890" - addArtistMBID = "069b64b6-7884-4f6a-94cc-e4c1d6c87a01" -) - -// albumLookupStub answers Lidarr's album lookup with lookupBody and records -// the body of the album POST that follows it. -func albumLookupStub(t *testing.T, lookupBody string, postStatus int, postBody string) (*Client, *httptest.Server, *map[string]any, *int) { - t.Helper() - var posted map[string]any - posts := 0 - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - switch { - case r.Method == http.MethodGet && r.URL.Path == "/api/v1/album/lookup": - if got := r.URL.Query().Get("term"); got != "lidarr:"+addAlbumMBID { - t.Errorf("lookup term = %q, want lidarr:%s", got, addAlbumMBID) - } - _, _ = w.Write([]byte(lookupBody)) - case r.Method == http.MethodPost && r.URL.Path == "/api/v1/album": - posts++ - _ = json.NewDecoder(r.Body).Decode(&posted) - w.WriteHeader(postStatus) - _, _ = w.Write([]byte(postBody)) - default: - t.Errorf("unexpected %s %s", r.Method, r.URL.Path) - w.WriteHeader(http.StatusNotFound) - } - }) - return c, srv, &posted, &posts -} - -var addAlbumParams = AddAlbumParams{ - ForeignAlbumID: addAlbumMBID, - ForeignArtistID: addArtistMBID, - ArtistName: "Boards of Canada", - QualityProfileID: 1, - MetadataProfileID: 3, - RootFolderPath: "/music", -} - -// #5234: Lidarr refuses a flat album add with "'Artist' must not be empty". -// The add has to be the looked-up album resource with its artist nested. -func TestAddAlbum_NewArtist_PostsTheLookedUpAlbumWithTheArtistNested(t *testing.T) { - lookup := `[ - {"foreignAlbumId": "someone-else", "title": "Other"}, - {"foreignAlbumId": "` + addAlbumMBID + `", "title": "Music Has the Right to Children", - "releases": [{"foreignReleaseId": "r1", "monitored": true}], - "artist": {"foreignArtistId": "` + addArtistMBID + `", "artistName": "Boards of Canada"}} - ]` - c, srv, posted, _ := albumLookupStub(t, lookup, http.StatusCreated, `{"id":9}`) - defer srv.Close() - - if err := c.AddAlbum(context.Background(), addAlbumParams); err != nil { - t.Fatalf("AddAlbum: %v", err) - } - body := *posted - if body["foreignAlbumId"] != addAlbumMBID || body["title"] != "Music Has the Right to Children" { - t.Errorf("posted album = %v / %v, want the looked-up album", body["foreignAlbumId"], body["title"]) - } - if rel, _ := body["releases"].([]any); len(rel) != 1 { - t.Errorf("releases = %v, want the lookup's releases passed through", body["releases"]) - } - if body["monitored"] != true { - t.Errorf("monitored = %v, want true", body["monitored"]) - } - if opts, _ := body["addOptions"].(map[string]any); opts["searchForNewAlbum"] != true { - t.Errorf("addOptions = %v, want searchForNewAlbum true", body["addOptions"]) - } - - artist, ok := body["artist"].(map[string]any) - if !ok { - t.Fatalf("artist = %T, want a nested object", body["artist"]) - } - for k, want := range map[string]any{ - "foreignArtistId": addArtistMBID, - "qualityProfileId": float64(1), - "metadataProfileId": float64(3), - "rootFolderPath": "/music", - "monitored": true, - "monitorNewItems": "none", - } { - if artist[k] != want { - t.Errorf("artist.%s = %v, want %v", k, artist[k], want) - } - } - // A new artist monitors this album only: albumsToMonitor wins over the - // monitor type in Lidarr's AlbumMonitoredService. - aOpts, _ := artist["addOptions"].(map[string]any) - if aOpts["monitor"] != "none" || aOpts["searchForMissingAlbums"] != false { - t.Errorf("artist.addOptions = %v, want monitor none, no catalogue search", aOpts) - } - if only, _ := aOpts["albumsToMonitor"].([]any); len(only) != 1 || only[0] != addAlbumMBID { - t.Errorf("artist.addOptions.albumsToMonitor = %v, want [%s]", aOpts["albumsToMonitor"], addAlbumMBID) - } -} - -// An artist Lidarr already has keeps its own profiles and monitoring. -func TestAddAlbum_ExistingArtist_IsLeftAsLidarrHasIt(t *testing.T) { - lookup := `[{"foreignAlbumId": "` + addAlbumMBID + `", - "artist": {"id": 42, "foreignArtistId": "` + addArtistMBID + `", "qualityProfileId": 5, - "metadataProfileId": 2, "rootFolderPath": "/lib", "monitorNewItems": "all"}}]` - c, srv, posted, _ := albumLookupStub(t, lookup, http.StatusCreated, `{"id":9}`) - defer srv.Close() - - if err := c.AddAlbum(context.Background(), addAlbumParams); err != nil { - t.Fatalf("AddAlbum: %v", err) - } - artist, _ := (*posted)["artist"].(map[string]any) - if artist["qualityProfileId"] != float64(5) || artist["rootFolderPath"] != "/lib" || artist["monitorNewItems"] != "all" { - t.Errorf("existing artist was rewritten: %v", artist) - } - if _, set := artist["addOptions"]; set { - t.Errorf("existing artist got addOptions %v", artist["addOptions"]) - } -} - -func TestAddAlbum_NotInLidarrsMetadata_IsNotFoundAndPostsNothing(t *testing.T) { - c, srv, _, posts := albumLookupStub(t, `[{"foreignAlbumId": "someone-else"}]`, http.StatusCreated, "") - defer srv.Close() - - err := c.AddAlbum(context.Background(), addAlbumParams) - if !errors.Is(err, ErrNotFound) { - t.Fatalf("err = %v, want ErrNotFound", err) - } - if *posts != 0 { - t.Errorf("posted %d times, want 0", *posts) - } -} - -func TestAddAlbum_AlreadyAdded_IsErrAlreadyExists(t *testing.T) { - lookup := `[{"foreignAlbumId": "` + addAlbumMBID + `", "artist": {"id": 42}}]` - c, srv, _, _ := albumLookupStub(t, lookup, http.StatusBadRequest, - `[{"propertyName":"ForeignAlbumId","errorMessage":"This album has already been added."}]`) - defer srv.Close() - - if err := c.AddAlbum(context.Background(), addAlbumParams); !errors.Is(err, ErrAlreadyExists) { - t.Fatalf("err = %v, want ErrAlreadyExists", err) - } -} - -func TestAddAlbum_AuthFailed(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusUnauthorized) - }) - defer srv.Close() - err := c.AddAlbum(context.Background(), addAlbumParams) - if !errors.Is(err, ErrAuthFailed) { - t.Fatalf("err = %v, want ErrAuthFailed", err) - } -} - -// --- ListQualityProfiles --- - -func TestListQualityProfiles_HappyPath(t *testing.T) { - body := mustReadFixture(t, "quality_profiles.json") - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/qualityprofile" { - t.Errorf("path = %q, want /api/v1/qualityprofile", r.URL.Path) - } - _, _ = w.Write(body) - }) - defer srv.Close() - - got, err := c.ListQualityProfiles(context.Background()) - if err != nil { - t.Fatalf("ListQualityProfiles: %v", err) - } - if len(got) != 3 { - t.Fatalf("len = %d, want 3", len(got)) - } - if got[0].ID != 1 || got[0].Name != "Any" { - t.Errorf("first profile = %+v, want {ID:1 Name:Any}", got[0]) - } - if got[1].ID != 2 || got[1].Name != "Lossless" { - t.Errorf("second profile = %+v", got[1]) - } -} - -func TestListQualityProfiles_BadJSON(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write([]byte(`not json`)) - }) - defer srv.Close() - _, err := c.ListQualityProfiles(context.Background()) - if !errors.Is(err, ErrInvalidPayload) { - t.Fatalf("err = %v, want ErrInvalidPayload", err) - } -} - -// --- LookupArtist (additional) --- - -func TestLookupArtist_LookupFailed(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusNotFound) - })) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "x", HTTP: srv.Client()} - _, err := c.LookupArtist(context.Background(), "boards") - if !errors.Is(err, ErrLookupFailed) { - t.Fatalf("err = %v, want ErrLookupFailed", err) - } -} - -func TestGet_BaseURLWithTrailingSlash(t *testing.T) { - body, _ := os.ReadFile("testdata/lookup_artist.json") - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - // Verify path is /api/v1/artist/lookup, NOT //api/v1/artist/lookup - if r.URL.Path != "/api/v1/artist/lookup" { - t.Errorf("got path %q; trailing-slash BaseURL produced double-slash", r.URL.Path) - } - _, _ = w.Write(body) - })) - defer srv.Close() - c := &Client{BaseURL: srv.URL + "/", APIKey: "k", HTTP: srv.Client()} - _, err := c.LookupArtist(context.Background(), "boards") - if err != nil { - t.Fatalf("LookupArtist: %v", err) - } -} - -// --- AddArtist (additional) --- - -func TestAddArtist_PostFailed(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusUnprocessableEntity) - })) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "x", HTTP: srv.Client()} - err := c.AddArtist(context.Background(), AddArtistParams{ForeignArtistID: "x", QualityProfileID: 1, RootFolderPath: "/m"}) - if !errors.Is(err, ErrLookupFailed) { - t.Fatalf("err = %v, want ErrLookupFailed", err) - } -} - -// --- ListRootFolders --- - -func TestListRootFolders_HappyPath(t *testing.T) { - body := mustReadFixture(t, "root_folders.json") - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/rootfolder" { - t.Errorf("path = %q, want /api/v1/rootfolder", r.URL.Path) - } - _, _ = w.Write(body) - }) - defer srv.Close() - - got, err := c.ListRootFolders(context.Background()) - if err != nil { - t.Fatalf("ListRootFolders: %v", err) - } - if len(got) != 3 { - t.Fatalf("len = %d, want 3", len(got)) - } - if got[0].Path != "/music" { - t.Errorf("Path = %q", got[0].Path) - } - if !got[0].Accessible { - t.Errorf("Accessible = false for /music, want true") - } - if got[0].FreeSpace != 107374182400 { - t.Errorf("FreeSpace = %d", got[0].FreeSpace) - } - if got[2].Accessible { - t.Errorf("expected /music-offline to be inaccessible") - } -} - -func TestListRootFolders_NullBody(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write([]byte(`null`)) - }) - defer srv.Close() - // null decodes to a nil slice, no error — but len check should yield 0. - got, err := c.ListRootFolders(context.Background()) - if err != nil { - t.Fatalf("unexpected error on null body: %v", err) - } - if len(got) != 0 { - t.Errorf("expected empty slice from null body, got %d", len(got)) - } -} - -func TestListRootFolders_BadJSON(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write([]byte("{not valid json")) - })) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "x", HTTP: srv.Client()} - _, err := c.ListRootFolders(context.Background()) - if !errors.Is(err, ErrInvalidPayload) { - t.Fatalf("err = %v, want ErrInvalidPayload", err) - } -} - -// --- Ping --- - -func TestPing_ReturnsVersion(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/system/status" { - t.Errorf("path = %q, want /api/v1/system/status", r.URL.Path) - } - _, _ = w.Write([]byte(`{"version":"2.4.3.3795","buildTime":"2024-01-15T00:00:00Z"}`)) - }) - defer srv.Close() - - res, err := c.Ping(context.Background()) - if err != nil { - t.Fatalf("Ping: %v", err) - } - if res.Version != "2.4.3.3795" { - t.Errorf("Version = %q, want 2.4.3.3795", res.Version) - } -} - -func TestPing_AuthFailed(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusUnauthorized) - }) - defer srv.Close() - _, err := c.Ping(context.Background()) - if !errors.Is(err, ErrAuthFailed) { - t.Fatalf("err = %v, want ErrAuthFailed", err) - } -} - -func TestPing_Unreachable(t *testing.T) { - c := &Client{BaseURL: "http://127.0.0.1:1", APIKey: "x", HTTP: &http.Client{}} - _, err := c.Ping(context.Background()) - if !errors.Is(err, ErrUnreachable) { - t.Fatalf("err = %v, want ErrUnreachable", err) - } -} - -func TestPing_BadJSON(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - _, _ = w.Write([]byte(`{bad`)) - }) - defer srv.Close() - _, err := c.Ping(context.Background()) - if !errors.Is(err, ErrInvalidPayload) { - t.Fatalf("err = %v, want ErrInvalidPayload", err) - } -} - -// --- pickPosterImage --- - -func TestPickPosterImage_PreferRemoteURL(t *testing.T) { - imgs := []lidarrImage{ - {CoverType: "poster", RemoteURL: "https://remote.example/img.jpg", URL: "https://local.example/img.jpg"}, - } - got := pickPosterImage(imgs) - if got != "https://remote.example/img.jpg" { - t.Errorf("got %q, want remote URL", got) - } -} - -func TestPickPosterImage_FallsBackToLocalURL(t *testing.T) { - imgs := []lidarrImage{ - {CoverType: "poster", RemoteURL: "", URL: "https://local.example/img.jpg"}, - } - got := pickPosterImage(imgs) - if got != "https://local.example/img.jpg" { - t.Errorf("got %q, want local URL", got) - } -} - -func TestPickPosterImage_SkipsNonPoster(t *testing.T) { - imgs := []lidarrImage{ - {CoverType: "banner", RemoteURL: "https://banner.example/img.jpg"}, - {CoverType: "cover", RemoteURL: "https://cover.example/img.jpg"}, - } - got := pickPosterImage(imgs) - if got != "" { - t.Errorf("got %q, want empty (no poster)", got) - } -} - -// --- UnmonitorTrack --- - -// unmonitorMux is the three-endpoint stub UnmonitorTrack walks. The flags it -// captures are inspected by the individual tests. -type unmonitorMux struct { - t *testing.T - wantAlbumMbid string - albumStatus int // override status for album lookup; 0 → 200 with body - tracksStatus int // override status for tracks list; 0 → 200 with body - putStatus int // override status for PUT /track/monitor; 0 → 200 - albumBody []byte // when albumStatus is 0 - tracksBody []byte // when tracksStatus is 0 - putBody map[string]any - putHit bool -} - -func (m *unmonitorMux) ServeHTTP(w http.ResponseWriter, r *http.Request) { - switch r.URL.Path { - case "/api/v1/album": - if got := r.URL.Query().Get("foreignAlbumId"); got != m.wantAlbumMbid { - m.t.Errorf("album lookup foreignAlbumId = %q, want %q", got, m.wantAlbumMbid) - } - if m.albumStatus != 0 { - w.WriteHeader(m.albumStatus) - return - } - _, _ = w.Write(m.albumBody) - case "/api/v1/track": - if got := r.URL.Query().Get("albumId"); got != "42" { - m.t.Errorf("track list albumId = %q, want 42", got) - } - if m.tracksStatus != 0 { - w.WriteHeader(m.tracksStatus) - return - } - _, _ = w.Write(m.tracksBody) - case "/api/v1/track/monitor": - if r.Method != http.MethodPut { - m.t.Errorf("monitor method = %q, want PUT", r.Method) - } - m.putHit = true - if m.putStatus != 0 { - w.WriteHeader(m.putStatus) - return - } - _ = json.NewDecoder(r.Body).Decode(&m.putBody) - w.WriteHeader(http.StatusAccepted) - default: - m.t.Errorf("unexpected path: %s", r.URL.Path) - w.WriteHeader(http.StatusNotFound) - } -} - -func TestUnmonitorTrack_HappyPath(t *testing.T) { - mux := &unmonitorMux{ - t: t, - wantAlbumMbid: "album-mbid", - albumBody: []byte(`[{"id":42,"foreignAlbumId":"album-mbid","title":"X","artistId":7}]`), - tracksBody: []byte(`[{"id":1001,"foreignTrackId":"other-mbid"},{"id":1002,"foreignTrackId":"track-mbid"}]`), - } - srv := httptest.NewServer(mux) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "k", HTTP: srv.Client()} - - if err := c.UnmonitorTrack(context.Background(), "track-mbid", "album-mbid"); err != nil { - t.Fatalf("UnmonitorTrack: %v", err) - } - if !mux.putHit { - t.Fatal("PUT /api/v1/track/monitor was not called") - } - if mux.putBody["monitored"] != false { - t.Errorf("PUT body monitored = %v, want false", mux.putBody["monitored"]) - } - ids, ok := mux.putBody["trackIds"].([]any) - if !ok || len(ids) != 1 { - t.Fatalf("PUT body trackIds = %v, want [1002]", mux.putBody["trackIds"]) - } - if ids[0] != float64(1002) { - t.Errorf("PUT trackIds[0] = %v, want 1002 (the matching foreignTrackId)", ids[0]) - } -} - -func TestUnmonitorTrack_EmptyMbid(t *testing.T) { - c := &Client{BaseURL: "http://127.0.0.1:1", APIKey: "k", HTTP: &http.Client{}} - if err := c.UnmonitorTrack(context.Background(), "", "album-mbid"); err == nil { - t.Error("UnmonitorTrack with empty trackMbid: want error, got nil") - } - if err := c.UnmonitorTrack(context.Background(), "track-mbid", ""); err == nil { - t.Error("UnmonitorTrack with empty albumMbid: want error, got nil") - } -} - -func TestUnmonitorTrack_AlbumNotInLidarr(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/album" { - t.Errorf("unexpected path %q", r.URL.Path) - } - _, _ = w.Write([]byte(`[]`)) - })) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "k", HTTP: srv.Client()} - err := c.UnmonitorTrack(context.Background(), "track-mbid", "album-mbid") - if !errors.Is(err, ErrNotFound) { - t.Fatalf("err = %v, want ErrNotFound", err) - } -} - -func TestUnmonitorTrack_TrackNotInAlbum(t *testing.T) { - mux := &unmonitorMux{ - t: t, - wantAlbumMbid: "album-mbid", - albumBody: []byte(`[{"id":42,"foreignAlbumId":"album-mbid","title":"X","artistId":7}]`), - // No track with foreignTrackId="track-mbid" - tracksBody: []byte(`[{"id":1001,"foreignTrackId":"other-mbid"}]`), - } - srv := httptest.NewServer(mux) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "k", HTTP: srv.Client()} - err := c.UnmonitorTrack(context.Background(), "track-mbid", "album-mbid") - if !errors.Is(err, ErrNotFound) { - t.Fatalf("err = %v, want ErrNotFound", err) - } - if mux.putHit { - t.Error("PUT was called even though track wasn't found") - } -} - -func TestUnmonitorTrack_ServerError(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusInternalServerError) - })) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "k", HTTP: srv.Client()} - err := c.UnmonitorTrack(context.Background(), "track-mbid", "album-mbid") - if !errors.Is(err, ErrServerError) { - t.Fatalf("err = %v, want ErrServerError", err) - } -} - -func TestUnmonitorTrack_AuthFailed(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusUnauthorized) - })) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "k", HTTP: srv.Client()} - err := c.UnmonitorTrack(context.Background(), "track-mbid", "album-mbid") - if !errors.Is(err, ErrAuthFailed) { - t.Fatalf("err = %v, want ErrAuthFailed", err) - } -} - -func TestUnmonitorTrack_Unreachable(t *testing.T) { - c := &Client{BaseURL: "http://127.0.0.1:1", APIKey: "k", HTTP: &http.Client{}} - err := c.UnmonitorTrack(context.Background(), "track-mbid", "album-mbid") - if !errors.Is(err, ErrUnreachable) { - t.Fatalf("err = %v, want ErrUnreachable", err) - } -} - -func TestUnmonitorTrack_PutServerError(t *testing.T) { - mux := &unmonitorMux{ - t: t, - wantAlbumMbid: "album-mbid", - albumBody: []byte(`[{"id":42,"foreignAlbumId":"album-mbid","title":"X","artistId":7}]`), - tracksBody: []byte(`[{"id":1002,"foreignTrackId":"track-mbid"}]`), - putStatus: http.StatusInternalServerError, - } - srv := httptest.NewServer(mux) - defer srv.Close() - c := &Client{BaseURL: srv.URL, APIKey: "k", HTTP: srv.Client()} - err := c.UnmonitorTrack(context.Background(), "track-mbid", "album-mbid") - if !errors.Is(err, ErrServerError) { - t.Fatalf("err = %v, want ErrServerError on PUT 5xx", err) - } -} diff --git a/internal/lidarr/delete.go b/internal/lidarr/delete.go deleted file mode 100644 index 9755c680..00000000 --- a/internal/lidarr/delete.go +++ /dev/null @@ -1,69 +0,0 @@ -package lidarr - -import ( - "context" - "fmt" - "net/http" - "net/url" - "strconv" - "strings" -) - -// del is the shared DELETE helper. Mirrors get()/post() in client.go: builds -// the URL inline, sets the API key, maps status codes to typed errors. The -// caller is responsible for closing the returned body. -func (c *Client) del(ctx context.Context, path string, q url.Values) (*http.Response, error) { - u, err := url.Parse(c.BaseURL) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) - } - u.Path = strings.TrimRight(u.Path, "/") + path - if q != nil { - u.RawQuery = q.Encode() - } - req, err := http.NewRequestWithContext(ctx, http.MethodDelete, u.String(), nil) - if err != nil { - return nil, err - } - req.Header.Set("X-Api-Key", c.APIKey) - - resp, err := c.HTTP.Do(req) - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrUnreachable, err) - } - if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden { - _ = resp.Body.Close() - return nil, ErrAuthFailed - } - if resp.StatusCode >= 500 { - _ = resp.Body.Close() - return nil, ErrServerError - } - if resp.StatusCode >= 400 { - _ = resp.Body.Close() - return nil, ErrLookupFailed - } - return resp, nil -} - -// DeleteAlbum removes an album from Lidarr's library. -// - deleteFiles=true also removes the audio files from disk. -// - addImportListExclusion=true tells Lidarr to never re-add this album -// via import-list scans. -// -// M5b's admin "delete via Lidarr" action always passes both `true`. -func (c *Client) DeleteAlbum(ctx context.Context, lidarrAlbumID int, deleteFiles, addImportListExclusion bool) error { - if lidarrAlbumID == 0 { - return fmt.Errorf("lidarr: zero album id") - } - q := url.Values{ - "deleteFiles": []string{strconv.FormatBool(deleteFiles)}, - "addImportListExclusion": []string{strconv.FormatBool(addImportListExclusion)}, - } - resp, err := c.del(ctx, "/api/v1/album/"+strconv.Itoa(lidarrAlbumID), q) - if err != nil { - return err - } - _ = resp.Body.Close() - return nil -} diff --git a/internal/lidarr/delete_test.go b/internal/lidarr/delete_test.go deleted file mode 100644 index eb0edf0d..00000000 --- a/internal/lidarr/delete_test.go +++ /dev/null @@ -1,214 +0,0 @@ -package lidarr - -import ( - "context" - "errors" - "net/http" - "net/http/httptest" - "os" - "testing" -) - -func TestLookupAlbumByMBID_HappyPath(t *testing.T) { - body, err := os.ReadFile("testdata/album_lookup_by_mbid.json") - if err != nil { - t.Fatalf("read fixture: %v", err) - } - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/album" { - t.Errorf("path = %q, want /api/v1/album", r.URL.Path) - } - if got := r.URL.Query().Get("foreignAlbumId"); got != "3a2c2c8c-7e6f-4f8a-b1d2-9a8b6c4e3f1d" { - t.Errorf("foreignAlbumId = %q", got) - } - if got := r.Header.Get("X-Api-Key"); got != "key123" { - t.Errorf("X-Api-Key = %q, want key123", got) - } - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - _, _ = w.Write(body) - }) - defer srv.Close() - - got, err := c.LookupAlbumByMBID(context.Background(), "3a2c2c8c-7e6f-4f8a-b1d2-9a8b6c4e3f1d") - if err != nil { - t.Fatalf("LookupAlbumByMBID: %v", err) - } - if got.ID != 42 || got.Title != "Music Has The Right To Children" { - t.Errorf("got = %+v", got) - } -} - -func TestLookupAlbumByMBID_EmptyArrayReturnsErrNotFound(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte("[]")) - }) - defer srv.Close() - - _, err := c.LookupAlbumByMBID(context.Background(), "x") - if !errors.Is(err, ErrNotFound) { - t.Errorf("err = %v, want ErrNotFound", err) - } -} - -func TestLookupAlbumByMBID_AuthFailed(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusUnauthorized) - }) - defer srv.Close() - - _, err := c.LookupAlbumByMBID(context.Background(), "x") - if !errors.Is(err, ErrAuthFailed) { - t.Errorf("err = %v, want ErrAuthFailed", err) - } -} - -func TestLookupAlbumByMBID_5xxReturnsErrServerError(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusInternalServerError) - }) - defer srv.Close() - - _, err := c.LookupAlbumByMBID(context.Background(), "x") - if !errors.Is(err, ErrServerError) { - t.Errorf("err = %v, want ErrServerError", err) - } -} - -func TestLookupAlbumByMBID_BadJSON(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte("not json")) - }) - defer srv.Close() - - _, err := c.LookupAlbumByMBID(context.Background(), "x") - if !errors.Is(err, ErrInvalidPayload) { - t.Errorf("err = %v, want ErrInvalidPayload", err) - } -} - -func TestLookupAlbumByMBID_EmptyMBIDRejected(t *testing.T) { - c, srv := newTestClient(func(http.ResponseWriter, *http.Request) { - t.Error("server should not be called with empty mbid") - }) - defer srv.Close() - - if _, err := c.LookupAlbumByMBID(context.Background(), ""); err == nil { - t.Error("err = nil, want empty-mbid rejection") - } -} - -func TestLookupArtistByMBID_EmptyArrayReturnsErrNotFound(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - _, _ = w.Write([]byte("[]")) - }) - defer srv.Close() - - _, err := c.LookupArtistByMBID(context.Background(), "x") - if !errors.Is(err, ErrNotFound) { - t.Errorf("err = %v, want ErrNotFound", err) - } -} - -func TestLookupArtistByMBID_EmptyMBIDRejected(t *testing.T) { - c, srv := newTestClient(func(http.ResponseWriter, *http.Request) { - t.Error("server should not be called with empty mbid") - }) - defer srv.Close() - - if _, err := c.LookupArtistByMBID(context.Background(), ""); err == nil { - t.Error("err = nil, want empty-mbid rejection") - } -} - -func TestLookupArtistByMBID_HappyPath(t *testing.T) { - body, err := os.ReadFile("testdata/artist_lookup_by_mbid.json") - if err != nil { - t.Fatalf("read fixture: %v", err) - } - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/artist" { - t.Errorf("path = %q, want /api/v1/artist", r.URL.Path) - } - if got := r.URL.Query().Get("mbId"); got != "069b64b6-7884-4f6a-94cc-e4c1d6c87a01" { - t.Errorf("mbId = %q", got) - } - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - _, _ = w.Write(body) - }) - defer srv.Close() - - got, err := c.LookupArtistByMBID(context.Background(), "069b64b6-7884-4f6a-94cc-e4c1d6c87a01") - if err != nil { - t.Fatalf("LookupArtistByMBID: %v", err) - } - if got.ID != 7 || got.ArtistName != "Boards of Canada" { - t.Errorf("got = %+v", got) - } -} - -func TestDeleteAlbum_PassesBothFlags(t *testing.T) { - var captured *http.Request - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - captured = r - w.WriteHeader(http.StatusOK) - }) - defer srv.Close() - - if err := c.DeleteAlbum(context.Background(), 42, true, true); err != nil { - t.Fatalf("DeleteAlbum: %v", err) - } - if captured == nil || captured.Method != http.MethodDelete { - t.Fatalf("method = %v, want DELETE", captured) - } - if captured.URL.Path != "/api/v1/album/42" { - t.Errorf("path = %q", captured.URL.Path) - } - if got := captured.URL.Query().Get("deleteFiles"); got != "true" { - t.Errorf("deleteFiles = %q", got) - } - if got := captured.URL.Query().Get("addImportListExclusion"); got != "true" { - t.Errorf("addImportListExclusion = %q", got) - } -} - -func TestDeleteAlbum_5xxReturnsErrServerError(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusInternalServerError) - }) - defer srv.Close() - - err := c.DeleteAlbum(context.Background(), 42, true, true) - if !errors.Is(err, ErrServerError) { - t.Errorf("err = %v, want ErrServerError", err) - } -} - -func TestDeleteAlbum_NetworkErrorReturnsErrUnreachable(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) - srv.Close() // server is closed; client should fail to connect - c := NewClient(srv.URL, "key123") - - err := c.DeleteAlbum(context.Background(), 42, true, true) - if !errors.Is(err, ErrUnreachable) { - t.Errorf("err = %v, want ErrUnreachable", err) - } -} - -func TestDeleteAlbum_ZeroIDRejected(t *testing.T) { - c, srv := newTestClient(func(_ http.ResponseWriter, _ *http.Request) { - t.Error("server should not be called with zero id") - }) - defer srv.Close() - - if err := c.DeleteAlbum(context.Background(), 0, true, true); err == nil { - t.Error("err = nil, want zero-id rejection") - } -} diff --git a/internal/lidarr/errors.go b/internal/lidarr/errors.go deleted file mode 100644 index fc27d359..00000000 --- a/internal/lidarr/errors.go +++ /dev/null @@ -1,33 +0,0 @@ -package lidarr - -import ( - "errors" - - "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" -) - -// Sentinel errors. Callers branch on these via errors.Is, not on -// HTTP status codes — the client maps codes to errors. -var ( - ErrUnreachable = errors.New("lidarr: unreachable") - ErrAuthFailed = errors.New("lidarr: auth failed") // 401 / 403 - ErrLookupFailed = errors.New("lidarr: lookup failed") // 4xx other than 401/403 - ErrServerError = errors.New("lidarr: server error") // 5xx - ErrInvalidPayload = errors.New("lidarr: invalid payload") - // ErrAlreadyExists is returned by AddArtist/AddAlbum when Lidarr - // rejects the add with a 400 because the artist/album is already in - // its library. This is the desired end state, not a failure — the - // idempotent retry path (reconciler re-sending an add after a partial - // success) treats it as confirmed rather than spinning forever. - ErrAlreadyExists = errors.New("lidarr: already exists") - // ErrNotFound is returned by LookupArtistByMBID and LookupAlbumByMBID - // when Lidarr returns 200 with an empty array — i.e., the MBID isn't in - // Lidarr's monitored set. Distinguished from network/auth errors so admin - // handlers can surface it as `lidarr_album_lookup_failed` (502) instead - // of `lidarr_unreachable` (503). - // - // Aliased to apierror.ErrNotFound so handler error mapping (errors.Is - // against the shared sentinel) and existing per-package callsites both - // resolve to the same pointer. - ErrNotFound = apierror.ErrNotFound -) diff --git a/internal/lidarr/lookup_mbid.go b/internal/lidarr/lookup_mbid.go deleted file mode 100644 index f419597c..00000000 --- a/internal/lidarr/lookup_mbid.go +++ /dev/null @@ -1,81 +0,0 @@ -package lidarr - -import ( - "context" - "encoding/json" - "fmt" - "net/url" -) - -// LookupArtistByMBID returns the artist Lidarr has indexed under that -// MBID. Returns ErrNotFound if Lidarr returns an empty array. -func (c *Client) LookupArtistByMBID(ctx context.Context, mbid string) (LidarrArtist, error) { - if mbid == "" { - return LidarrArtist{}, fmt.Errorf("lidarr: empty mbid") - } - q := url.Values{"mbId": []string{mbid}} - resp, err := c.get(ctx, "/api/v1/artist", q) - if err != nil { - return LidarrArtist{}, err - } - defer func() { _ = resp.Body.Close() }() - - var rows []LidarrArtist - if err := json.NewDecoder(resp.Body).Decode(&rows); err != nil { - return LidarrArtist{}, fmt.Errorf("%w: decode artist: %v", ErrInvalidPayload, err) - } - if len(rows) == 0 { - return LidarrArtist{}, ErrNotFound - } - return rows[0], nil -} - -// LookupAlbumByMBID returns the album Lidarr has indexed under that -// MBID. Returns ErrNotFound on empty result. -func (c *Client) LookupAlbumByMBID(ctx context.Context, mbid string) (LidarrAlbum, error) { - if mbid == "" { - return LidarrAlbum{}, fmt.Errorf("lidarr: empty mbid") - } - q := url.Values{"foreignAlbumId": []string{mbid}} - resp, err := c.get(ctx, "/api/v1/album", q) - if err != nil { - return LidarrAlbum{}, err - } - defer func() { _ = resp.Body.Close() }() - - var rows []LidarrAlbum - if err := json.NewDecoder(resp.Body).Decode(&rows); err != nil { - return LidarrAlbum{}, fmt.Errorf("%w: decode album: %v", ErrInvalidPayload, err) - } - if len(rows) == 0 { - return LidarrAlbum{}, ErrNotFound - } - return rows[0], nil -} - -// lookupAlbumResource fetches the album resource Lidarr's metadata holds -// under mbid (GET /api/v1/album/lookup?term=lidarr:), kept as raw -// JSON so AddAlbum can POST it back with every field Lidarr sent — the -// round trip Lidarr's own add-album UI makes. Returns ErrNotFound when no -// result carries that MBID. -func (c *Client) lookupAlbumResource(ctx context.Context, mbid string) (map[string]any, error) { - if mbid == "" { - return nil, fmt.Errorf("lidarr: empty album mbid") - } - resp, err := c.get(ctx, "/api/v1/album/lookup", url.Values{"term": {"lidarr:" + mbid}}) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - - var rows []map[string]any - if err := json.NewDecoder(resp.Body).Decode(&rows); err != nil { - return nil, fmt.Errorf("%w: decode album lookup: %v", ErrInvalidPayload, err) - } - for _, row := range rows { - if id, _ := row["foreignAlbumId"].(string); id == mbid { - return row, nil - } - } - return nil, fmt.Errorf("%w: album %s in Lidarr's metadata", ErrNotFound, mbid) -} diff --git a/internal/lidarr/releases.go b/internal/lidarr/releases.go deleted file mode 100644 index 2513af12..00000000 --- a/internal/lidarr/releases.go +++ /dev/null @@ -1,154 +0,0 @@ -package lidarr - -import ( - "context" - "encoding/json" - "fmt" - "net/url" - "strconv" -) - -// What Lidarr holds on disk and how it maps it (Scribe milestone #498). -// -// Lidarr maps one file to each track of the ONE release it monitors per album. -// A file it holds but maps to no track is "unmapped": removing it opens no hole. -// Removing a mapped file does, and Lidarr fills the hole by downloading again — -// which is why Minstrel's duplicate resolver asks before it deletes anything. - -// TrackFile is a file Lidarr holds on disk. -type TrackFile struct { - ID int `json:"id"` - Path string `json:"path"` - AlbumID int `json:"albumId"` -} - -// ListUnmappedTrackFiles hits GET /api/v1/trackfile?unmapped=true: files under -// Lidarr's root folders that it has scanned but matched to no track. -func (c *Client) ListUnmappedTrackFiles(ctx context.Context) ([]TrackFile, error) { - resp, err := c.get(ctx, "/api/v1/trackfile", url.Values{"unmapped": {"true"}}) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - var out []TrackFile - if err := json.NewDecoder(resp.Body).Decode(&out); err != nil { - return nil, fmt.Errorf("%w: decode unmapped track files: %v", ErrInvalidPayload, err) - } - return out, nil -} - -// AlbumRelease is one edition of an album as Lidarr knows it: the 14×12" vinyl -// box set, the 2×CD deluxe, the digital standard. Exactly one is Monitored. -type AlbumRelease struct { - ID int `json:"id"` - Title string `json:"title"` - Disambiguation string `json:"disambiguation"` - Format string `json:"format"` - TrackCount int `json:"trackCount"` - MediumCount int `json:"mediumCount"` - Monitored bool `json:"monitored"` -} - -// ReleaseTrack is one track of a release. -type ReleaseTrack struct { - ID int `json:"id"` - Title string `json:"title"` - MediumNumber int `json:"mediumNumber"` - TrackFileID int `json:"trackFileId"` -} - -// GetAlbumReleases hits GET /api/v1/album/{id} and returns the album's releases. -func (c *Client) GetAlbumReleases(ctx context.Context, albumID int) ([]AlbumRelease, error) { - resp, err := c.get(ctx, "/api/v1/album/"+strconv.Itoa(albumID), nil) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - var album struct { - Releases []AlbumRelease `json:"releases"` - } - if err := json.NewDecoder(resp.Body).Decode(&album); err != nil { - return nil, fmt.Errorf("%w: decode album: %v", ErrInvalidPayload, err) - } - return album.Releases, nil -} - -// ListAlbumTracks hits GET /api/v1/track?albumId= : the tracks of the release -// Lidarr monitors for that album. -func (c *Client) ListAlbumTracks(ctx context.Context, albumID int) ([]ReleaseTrack, error) { - return c.listTracks(ctx, url.Values{"albumId": {strconv.Itoa(albumID)}}) -} - -// ListReleaseTracks hits GET /api/v1/track?albumReleaseId= : the tracks of one -// release, monitored or not. -func (c *Client) ListReleaseTracks(ctx context.Context, releaseID int) ([]ReleaseTrack, error) { - return c.listTracks(ctx, url.Values{"albumReleaseId": {strconv.Itoa(releaseID)}}) -} - -func (c *Client) listTracks(ctx context.Context, q url.Values) ([]ReleaseTrack, error) { - resp, err := c.get(ctx, "/api/v1/track", q) - if err != nil { - return nil, err - } - defer func() { _ = resp.Body.Close() }() - var out []ReleaseTrack - if err := json.NewDecoder(resp.Body).Decode(&out); err != nil { - return nil, fmt.Errorf("%w: decode tracks: %v", ErrInvalidPayload, err) - } - return out, nil -} - -// SetMonitoredRelease makes releaseID the album's monitored release. -// -// It is the round trip Lidarr's own "edit album" makes: GET the album resource, -// flip `monitored` on its releases, PUT the whole resource back. Lidarr's -// UpdateAlbum publishes AlbumEditedEvent, and when the monitored release changed, -// AlbumEditedService unlinks the album's track files and queues -// RescanFoldersCommand for the artist's folder, which maps the files on disk to -// the new release's tracks. Files the new release has no track for come out of -// that rescan unmapped. (Lidarr source, develop: Music/Services/ -// AlbumEditedService.cs and AlbumService.UpdateAlbum.) -// -// The resource is kept as raw JSON so every field Lidarr sent goes back -// unchanged; only the releases' monitored flags differ. Returns ErrNotFound when -// the album has no release with that id. -func (c *Client) SetMonitoredRelease(ctx context.Context, albumID, releaseID int) error { - path := "/api/v1/album/" + strconv.Itoa(albumID) - resp, err := c.get(ctx, path, nil) - if err != nil { - return err - } - var album map[string]any - err = json.NewDecoder(resp.Body).Decode(&album) - _ = resp.Body.Close() - if err != nil { - return fmt.Errorf("%w: decode album: %v", ErrInvalidPayload, err) - } - - releases, _ := album["releases"].([]any) - found := false - for _, r := range releases { - rel, ok := r.(map[string]any) - if !ok { - continue - } - id, _ := rel["id"].(float64) - match := int(id) == releaseID - rel["monitored"] = match - found = found || match - } - if !found { - return fmt.Errorf("%w: release %d on album %d", ErrNotFound, releaseID, albumID) - } - - body, err := json.Marshal(album) - if err != nil { - return fmt.Errorf("%w: marshal album: %v", ErrInvalidPayload, err) - } - putResp, err := c.put(ctx, path, body) - if err != nil { - return err - } - _ = putResp.Body.Close() - return nil -} diff --git a/internal/lidarr/releases_test.go b/internal/lidarr/releases_test.go deleted file mode 100644 index 5e3c769e..00000000 --- a/internal/lidarr/releases_test.go +++ /dev/null @@ -1,116 +0,0 @@ -package lidarr - -import ( - "context" - "encoding/json" - "errors" - "io" - "net/http" - "testing" -) - -func TestListUnmappedTrackFiles(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/trackfile" || r.URL.Query().Get("unmapped") != "true" { - t.Errorf("request = %s?%s, want /api/v1/trackfile?unmapped=true", r.URL.Path, r.URL.RawQuery) - } - _, _ = w.Write([]byte(`[{"id":7,"path":"/music/A/B (2017)/A - B - 02 - X.mp3","albumId":0}]`)) - }) - defer srv.Close() - - got, err := c.ListUnmappedTrackFiles(context.Background()) - if err != nil { - t.Fatalf("ListUnmappedTrackFiles: %v", err) - } - if len(got) != 1 || got[0].ID != 7 || got[0].Path != "/music/A/B (2017)/A - B - 02 - X.mp3" { - t.Errorf("got %+v", got) - } -} - -func TestReleaseReads(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - switch { - case r.URL.Path == "/api/v1/album/4723": - _, _ = w.Write([]byte(`{"id":4723,"releases":[ - {"id":10148,"title":"Humanz","format":"14x12\" Vinyl, Digital Media","trackCount":68,"mediumCount":15,"monitored":true}, - {"id":10149,"title":"Humanz","format":"Digital Media","trackCount":40,"mediumCount":1,"monitored":false}]}`)) - case r.URL.Path == "/api/v1/track" && r.URL.Query().Get("albumId") == "4723": - _, _ = w.Write([]byte(`[{"id":1,"title":"Ascension","mediumNumber":1,"trackFileId":9}]`)) - case r.URL.Path == "/api/v1/track" && r.URL.Query().Get("albumReleaseId") == "10149": - _, _ = w.Write([]byte(`[{"id":2,"title":"Ascension","mediumNumber":1,"trackFileId":0}]`)) - default: - t.Errorf("unexpected request %s?%s", r.URL.Path, r.URL.RawQuery) - w.WriteHeader(http.StatusNotFound) - } - }) - defer srv.Close() - ctx := context.Background() - - rels, err := c.GetAlbumReleases(ctx, 4723) - if err != nil { - t.Fatalf("GetAlbumReleases: %v", err) - } - if len(rels) != 2 || !rels[0].Monitored || rels[0].MediumCount != 15 || rels[1].TrackCount != 40 { - t.Errorf("releases = %+v", rels) - } - tr, err := c.ListAlbumTracks(ctx, 4723) - if err != nil || len(tr) != 1 || tr[0].TrackFileID != 9 { - t.Errorf("ListAlbumTracks = %+v, %v", tr, err) - } - tr, err = c.ListReleaseTracks(ctx, 10149) - if err != nil || len(tr) != 1 || tr[0].Title != "Ascension" { - t.Errorf("ListReleaseTracks = %+v, %v", tr, err) - } -} - -// The PUT carries the whole resource back, with only the releases' monitored -// flags changed: exactly the chosen release is monitored, and a field Minstrel -// does not model survives the round trip. -func TestSetMonitoredRelease_RoundTripsTheResource(t *testing.T) { - var put map[string]any - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/v1/album/4723" { - t.Errorf("path = %q", r.URL.Path) - } - switch r.Method { - case http.MethodGet: - _, _ = w.Write([]byte(`{"id":4723,"anyReleaseOk":false,"secondaryTypes":["Studio"],"releases":[ - {"id":10148,"monitored":true},{"id":10149,"monitored":false}]}`)) - case http.MethodPut: - b, _ := io.ReadAll(r.Body) - if err := json.Unmarshal(b, &put); err != nil { - t.Errorf("PUT body: %v", err) - } - w.WriteHeader(http.StatusAccepted) - } - }) - defer srv.Close() - - if err := c.SetMonitoredRelease(context.Background(), 4723, 10149); err != nil { - t.Fatalf("SetMonitoredRelease: %v", err) - } - rels := put["releases"].([]any) - if m := rels[0].(map[string]any)["monitored"]; m != false { - t.Errorf("old release monitored = %v, want false", m) - } - if m := rels[1].(map[string]any)["monitored"]; m != true { - t.Errorf("new release monitored = %v, want true", m) - } - if st, _ := put["secondaryTypes"].([]any); len(st) != 1 || st[0] != "Studio" { - t.Errorf("secondaryTypes = %v, want it carried through unchanged", put["secondaryTypes"]) - } -} - -func TestSetMonitoredRelease_UnknownReleaseSendsNothing(t *testing.T) { - c, srv := newTestClient(func(w http.ResponseWriter, r *http.Request) { - if r.Method == http.MethodPut { - t.Error("PUT sent for a release the album does not have") - } - _, _ = w.Write([]byte(`{"id":4723,"releases":[{"id":10148,"monitored":true}]}`)) - }) - defer srv.Close() - - if err := c.SetMonitoredRelease(context.Background(), 4723, 999); !errors.Is(err, ErrNotFound) { - t.Errorf("err = %v, want ErrNotFound", err) - } -} diff --git a/internal/lidarr/testdata/album_lookup_by_mbid.json b/internal/lidarr/testdata/album_lookup_by_mbid.json deleted file mode 100644 index 90d38049..00000000 --- a/internal/lidarr/testdata/album_lookup_by_mbid.json +++ /dev/null @@ -1,8 +0,0 @@ -[ - { - "id": 42, - "foreignAlbumId": "3a2c2c8c-7e6f-4f8a-b1d2-9a8b6c4e3f1d", - "title": "Music Has The Right To Children", - "artistId": 7 - } -] diff --git a/internal/lidarr/testdata/artist_lookup_by_mbid.json b/internal/lidarr/testdata/artist_lookup_by_mbid.json deleted file mode 100644 index 74d78984..00000000 --- a/internal/lidarr/testdata/artist_lookup_by_mbid.json +++ /dev/null @@ -1,7 +0,0 @@ -[ - { - "id": 7, - "foreignArtistId": "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - "artistName": "Boards of Canada" - } -] diff --git a/internal/lidarr/testdata/lookup_album.json b/internal/lidarr/testdata/lookup_album.json deleted file mode 100644 index b1bb6e35..00000000 --- a/internal/lidarr/testdata/lookup_album.json +++ /dev/null @@ -1,23 +0,0 @@ -[ - { - "foreignAlbumId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "foreignArtistId": "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - "title": "Music Has the Right to Children", - "artistName": "Boards of Canada", - "releaseDate": "1998-04-27", - "trackCount": 18, - "images": [ - {"coverType": "cover", "remoteUrl": "https://example.invalid/mhtrtc.jpg", "url": ""}, - {"coverType": "poster", "remoteUrl": "https://example.invalid/mhtrtc-poster.jpg", "url": ""} - ] - }, - { - "foreignAlbumId": "b2c3d4e5-f6a7-8901-bcde-f12345678901", - "foreignArtistId": "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - "title": "Geogaddi", - "artistName": "Boards of Canada", - "releaseDate": "2002-02-11", - "trackCount": 23, - "images": [] - } -] diff --git a/internal/lidarr/testdata/lookup_artist.json b/internal/lidarr/testdata/lookup_artist.json deleted file mode 100644 index d3caab26..00000000 --- a/internal/lidarr/testdata/lookup_artist.json +++ /dev/null @@ -1,19 +0,0 @@ -[ - { - "foreignArtistId": "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - "artistName": "Boards of Canada", - "genres": ["Electronic", "IDM"], - "albumCount": 18, - "images": [ - {"coverType": "poster", "remoteUrl": "https://example.invalid/boc.jpg", "url": ""}, - {"coverType": "banner", "remoteUrl": "https://example.invalid/boc-banner.jpg", "url": ""} - ] - }, - { - "foreignArtistId": "f54ba20c-7da3-4b8a-9b12-22f09b9e2c1c", - "artistName": "Bored of Education", - "genres": [], - "albumCount": 0, - "images": [] - } -] diff --git a/internal/lidarr/testdata/lookup_track.json b/internal/lidarr/testdata/lookup_track.json deleted file mode 100644 index 10751762..00000000 --- a/internal/lidarr/testdata/lookup_track.json +++ /dev/null @@ -1,18 +0,0 @@ -[ - { - "foreignTrackId": "t1a2b3c4-d5e6-7890-abcd-ef1234567890", - "foreignAlbumId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "foreignArtistId": "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - "title": "Roygbiv", - "albumTitle": "Music Has the Right to Children", - "artistName": "Boards of Canada" - }, - { - "foreignTrackId": "t2b3c4d5-e6f7-8901-bcde-f12345678901", - "foreignAlbumId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "foreignArtistId": "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - "title": "Aquarius", - "albumTitle": "Music Has the Right to Children", - "artistName": "Boards of Canada" - } -] diff --git a/internal/lidarr/testdata/quality_profiles.json b/internal/lidarr/testdata/quality_profiles.json deleted file mode 100644 index b645820e..00000000 --- a/internal/lidarr/testdata/quality_profiles.json +++ /dev/null @@ -1,5 +0,0 @@ -[ - {"id": 1, "name": "Any"}, - {"id": 2, "name": "Lossless"}, - {"id": 3, "name": "Standard"} -] diff --git a/internal/lidarr/testdata/root_folders.json b/internal/lidarr/testdata/root_folders.json deleted file mode 100644 index b75eef9e..00000000 --- a/internal/lidarr/testdata/root_folders.json +++ /dev/null @@ -1,5 +0,0 @@ -[ - {"path": "/music", "accessible": true, "freeSpace": 107374182400}, - {"path": "/music-lossy", "accessible": true, "freeSpace": 53687091200}, - {"path": "/music-offline", "accessible": false, "freeSpace": 0} -] diff --git a/internal/lidarr/types.go b/internal/lidarr/types.go deleted file mode 100644 index 3fd9db36..00000000 --- a/internal/lidarr/types.go +++ /dev/null @@ -1,97 +0,0 @@ -// Package lidarr is a typed HTTP client for Lidarr's v1 API. It is the -// only place in the codebase that knows about Lidarr's wire format. -// Callers receive value structs, never raw JSON. -package lidarr - -// LookupResult is the normalized shape returned by Lookup{Artist,Album,Track}. -// MBID is the kind's primary identifier (foreignArtistId / foreignAlbumId / -// foreignTrackId from Lidarr). ArtistMBID and AlbumMBID carry parent -// identifiers when the kind has them — request creation and admin -// approval need them to construct the lidarr_requests row and the -// AddArtist/AddAlbum API calls. -type LookupResult struct { - MBID string // foreignArtistId / foreignAlbumId / foreignTrackId - ArtistMBID string // parent artist's foreignArtistId — set on album/track results, empty on artist results - AlbumMBID string // parent album's foreignAlbumId — set on track results only - Name string // artist name; album/track returns Title here too - Secondary string // genre + album count for artist; year for album; album for track - ImageURL string // cover-art URL Lidarr surfaced (may be empty) -} - -// lidarrImage is the wire shape for image entries in Lidarr's API responses. -type lidarrImage struct { - CoverType string `json:"coverType"` - RemoteURL string `json:"remoteUrl"` - URL string `json:"url"` -} - -// QualityProfile is the dropdown choice in /admin/integrations. -type QualityProfile struct { - ID int - Name string -} - -// MetadataProfile controls which release types Lidarr tracks for an artist -// (album / single / EP / compilation). It's a separate Lidarr concept from -// quality profile and is required on POST /api/v1/artist — Lidarr 4xx's -// the request with "'Metadata Profile Id' must be greater than '0'" if -// it's missing or zero. -type MetadataProfile struct { - ID int - Name string -} - -// RootFolder is the dropdown choice in /admin/integrations. -type RootFolder struct { - Path string - Accessible bool - FreeSpace int64 -} - -// AddArtistParams are the fields Lidarr requires on POST /api/v1/artist. -// ArtistName and MetadataProfileID are both required; omitting either -// produces a 400 with field-validation errors. -type AddArtistParams struct { - ForeignArtistID string - ArtistName string - QualityProfileID int - MetadataProfileID int - RootFolderPath string - MonitorAll bool // true => addOptions.monitor="all"; false => "future" -} - -// AddAlbumParams identify the album to add and carry the settings Lidarr -// needs when it has to create the parent artist on the fly (an album by an -// artist not yet in Lidarr). The artist fields are a fallback only: the -// album lookup normally supplies the artist itself. -type AddAlbumParams struct { - ForeignAlbumID string - ForeignArtistID string // Lidarr requires the artist's foreign id too - ArtistName string - QualityProfileID int - MetadataProfileID int - RootFolderPath string -} - -// PingResult is the response shape from GET /api/v1/system/status. -type PingResult struct { - Version string -} - -// LidarrArtist is the subset of Lidarr's artist resource used by M5b -// admin actions. The "id" field is Lidarr's internal numeric ID — needed -// for DELETE /api/v1/artist/{id} calls. -type LidarrArtist struct { - ID int `json:"id"` - ForeignArtistID string `json:"foreignArtistId"` // MBID - ArtistName string `json:"artistName"` -} - -// LidarrAlbum is the subset of Lidarr's album resource used by M5b -// admin actions. -type LidarrAlbum struct { - ID int `json:"id"` - ForeignAlbumID string `json:"foreignAlbumId"` // MBID - Title string `json:"title"` - ArtistID int `json:"artistId"` -} diff --git a/internal/lidarr/unmonitor.go b/internal/lidarr/unmonitor.go deleted file mode 100644 index 81d541c7..00000000 --- a/internal/lidarr/unmonitor.go +++ /dev/null @@ -1,91 +0,0 @@ -package lidarr - -import ( - "context" - "encoding/json" - "fmt" - "net/url" - "strconv" -) - -// UnmonitorTrack flips Lidarr's `monitored` flag to false on the track -// matching trackMbid (a MusicBrainz id, aka Lidarr's `foreignTrackId`). -// Used by the M7 #372 admin "Remove from library" flow when the operator -// requests `?unmonitor=true` so Lidarr's monitor doesn't search for a -// replacement after we delete the local file + DB rows. -// -// Lidarr's monitor API takes its own internal numeric id, not the mbid, -// so this is a three-step walk: -// -// 1. GET /api/v1/album?foreignAlbumId={albumMbid} — find the album's -// internal Lidarr id. albumMbid is the parent album's mbid; the -// caller (tracks service) passes it because looking it up from the -// trackMbid alone would mean either a dedicated Lidarr endpoint -// that doesn't exist or scanning the entire library. -// 2. GET /api/v1/track?albumId={lidarr_album_id} — list the album's -// tracks, each with its internal id and foreignTrackId. -// 3. PUT /api/v1/track/monitor — body `{trackIds:[id], monitored:false}`. -// -// Returns ErrNotFound when the album or track isn't in Lidarr's library -// (already removed, or the track was never imported). Network / auth / -// 5xx errors propagate as typed sentinel errors so the tracks service -// can decide whether to surface a `lidarr_unmonitor_failed` flag. -func (c *Client) UnmonitorTrack(ctx context.Context, trackMbid, albumMbid string) error { - if trackMbid == "" { - return fmt.Errorf("lidarr: empty track mbid") - } - if albumMbid == "" { - return fmt.Errorf("lidarr: empty album mbid") - } - - // Step 1: resolve album mbid → Lidarr album id. - album, err := c.LookupAlbumByMBID(ctx, albumMbid) - if err != nil { - return err - } - if album.ID == 0 { - return ErrNotFound - } - - // Step 2: list tracks under that album, find the one whose - // foreignTrackId matches. - q := url.Values{"albumId": []string{strconv.Itoa(album.ID)}} - resp, err := c.get(ctx, "/api/v1/track", q) - if err != nil { - return err - } - defer func() { _ = resp.Body.Close() }() - - var tracks []struct { - ID int `json:"id"` - ForeignTrackID string `json:"foreignTrackId"` - } - if err := json.NewDecoder(resp.Body).Decode(&tracks); err != nil { - return fmt.Errorf("%w: decode tracks: %v", ErrInvalidPayload, err) - } - var lidarrTrackID int - for _, t := range tracks { - if t.ForeignTrackID == trackMbid { - lidarrTrackID = t.ID - break - } - } - if lidarrTrackID == 0 { - return ErrNotFound - } - - // Step 3: PUT the monitor toggle. - body, err := json.Marshal(map[string]any{ - "trackIds": []int{lidarrTrackID}, - "monitored": false, - }) - if err != nil { - return fmt.Errorf("%w: marshal: %v", ErrInvalidPayload, err) - } - putResp, err := c.put(ctx, "/api/v1/track/monitor", body) - if err != nil { - return err - } - _ = putResp.Body.Close() - return nil -} diff --git a/internal/lidarrconfig/service.go b/internal/lidarrconfig/service.go deleted file mode 100644 index 05e02693..00000000 --- a/internal/lidarrconfig/service.go +++ /dev/null @@ -1,94 +0,0 @@ -// Package lidarrconfig is a thin wrapper over the singleton lidarr_config -// row. Get returns a typed Config; Save updates it. Callers branch on -// Config.Enabled — never on raw NULL fields. -package lidarrconfig - -import ( - "context" - "fmt" - - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" -) - -// Config is the typed projection of lidarr_config (no NULL fields exposed -// to callers — empty strings / zero ints carry the "unset" meaning). -type Config struct { - Enabled bool - BaseURL string - APIKey string - DefaultQualityProfileID int - DefaultMetadataProfileID int - DefaultRootFolderPath string -} - -// Service reads and writes the singleton. -type Service struct { - pool *pgxpool.Pool -} - -func New(pool *pgxpool.Pool) *Service { return &Service{pool: pool} } - -func (s *Service) Get(ctx context.Context) (Config, error) { - row, err := dbq.New(s.pool).GetLidarrConfig(ctx) - if err != nil { - return Config{}, fmt.Errorf("lidarrconfig: %w", err) - } - cfg := Config{Enabled: row.Enabled} - if row.BaseUrl != nil { - cfg.BaseURL = *row.BaseUrl - } - if row.ApiKey != nil { - cfg.APIKey = *row.ApiKey - } - if row.DefaultQualityProfileID != nil { - cfg.DefaultQualityProfileID = int(*row.DefaultQualityProfileID) - } - if row.DefaultMetadataProfileID != nil { - cfg.DefaultMetadataProfileID = int(*row.DefaultMetadataProfileID) - } - if row.DefaultRootFolderPath != nil { - cfg.DefaultRootFolderPath = *row.DefaultRootFolderPath - } - return cfg, nil -} - -// Save writes the entire row. Callers pass the full Config they want -// stored — this is not a partial update. -func (s *Service) Save(ctx context.Context, cfg Config) error { - var ( - baseURL = strPtr(cfg.BaseURL) - apiKey = strPtr(cfg.APIKey) - qpID = int32Ptr(cfg.DefaultQualityProfileID) - mpID = int32Ptr(cfg.DefaultMetadataProfileID) - rootPath = strPtr(cfg.DefaultRootFolderPath) - ) - _, err := dbq.New(s.pool).UpdateLidarrConfig(ctx, dbq.UpdateLidarrConfigParams{ - Enabled: cfg.Enabled, - BaseUrl: baseURL, - ApiKey: apiKey, - DefaultQualityProfileID: qpID, - DefaultMetadataProfileID: mpID, - DefaultRootFolderPath: rootPath, - }) - if err != nil { - return fmt.Errorf("lidarrconfig: %w", err) - } - return nil -} - -func strPtr(s string) *string { - if s == "" { - return nil - } - return &s -} - -func int32Ptr(i int) *int32 { - if i == 0 { - return nil - } - v := int32(i) - return &v -} diff --git a/internal/lidarrconfig/service_test.go b/internal/lidarrconfig/service_test.go deleted file mode 100644 index 627e4be1..00000000 --- a/internal/lidarrconfig/service_test.go +++ /dev/null @@ -1,88 +0,0 @@ -package lidarrconfig - -import ( - "context" - "io" - "log/slog" - "os" - "testing" - - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db" -) - -func newTestPool(t *testing.T) *pgxpool.Pool { - t.Helper() - if testing.Short() { - t.Skip("skipping integration test in -short mode") - } - dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") - if dsn == "" { - t.Skip("MINSTREL_TEST_DATABASE_URL not set") - } - if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { - t.Fatalf("migrate: %v", err) - } - pool, err := pgxpool.New(context.Background(), dsn) - if err != nil { - t.Fatalf("pool: %v", err) - } - t.Cleanup(pool.Close) - // Reset singleton to default state before each test by replacing - // the row contents via UPDATE (TRUNCATE would violate the CHECK). - if _, err := pool.Exec(context.Background(), - "UPDATE lidarr_config SET enabled=false, base_url=NULL, api_key=NULL, default_quality_profile_id=NULL, default_root_folder_path=NULL WHERE id=1", - ); err != nil { - t.Fatalf("reset: %v", err) - } - return pool -} - -func TestGet_DefaultRowReturnsZeroValueConfig(t *testing.T) { - pool := newTestPool(t) - cfg, err := New(pool).Get(context.Background()) - if err != nil { - t.Fatalf("Get: %v", err) - } - if cfg.Enabled || cfg.BaseURL != "" || cfg.APIKey != "" { - t.Errorf("expected zero-value Config, got %+v", cfg) - } -} - -func TestSaveThenGet_RoundTrip(t *testing.T) { - pool := newTestPool(t) - s := New(pool) - want := Config{ - Enabled: true, - BaseURL: "http://lidarr.lan:8686", - APIKey: "secret", - DefaultQualityProfileID: 4, - DefaultRootFolderPath: "/music", - } - if err := s.Save(context.Background(), want); err != nil { - t.Fatalf("Save: %v", err) - } - got, err := s.Get(context.Background()) - if err != nil { - t.Fatalf("Get: %v", err) - } - if got != want { - t.Errorf("round-trip mismatch:\n got = %+v\nwant = %+v", got, want) - } -} - -func TestSave_EmptyValuesPersistAsNULL(t *testing.T) { - pool := newTestPool(t) - s := New(pool) - if err := s.Save(context.Background(), Config{Enabled: false}); err != nil { - t.Fatalf("Save: %v", err) - } - got, err := s.Get(context.Background()) - if err != nil { - t.Fatalf("Get: %v", err) - } - if got.BaseURL != "" || got.APIKey != "" || got.DefaultRootFolderPath != "" { - t.Errorf("expected empty strings on round-trip; got %+v", got) - } -} diff --git a/internal/lidarrquarantine/service.go b/internal/lidarrquarantine/service.go deleted file mode 100644 index 54a97abe..00000000 --- a/internal/lidarrquarantine/service.go +++ /dev/null @@ -1,360 +0,0 @@ -// Package lidarrquarantine owns the per-user track quarantine workflow. -// Users flag a track as broken (Flag/Unflag), the SPA hides the track -// from their views, and admins resolve the resulting reports via the -// Service's admin actions (Resolve / DeleteFile / DeleteViaLidarr). -package lidarrquarantine - -import ( - "context" - "errors" - "fmt" - "strings" - - "github.com/jackc/pgx/v5" - "github.com/jackc/pgx/v5/pgconn" - "github.com/jackc/pgx/v5/pgtype" - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/library" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" -) - -// Public errors. Handlers map these to API codes. -var ( - ErrBadReason = errors.New("lidarrquarantine: invalid reason") - ErrTrackNotFound = errors.New("lidarrquarantine: track not found") - ErrQuarantineNotFound = errors.New("lidarrquarantine: quarantine row not found") - ErrAlbumMBIDMissing = errors.New("lidarrquarantine: track has no parent album mbid") - ErrLidarrAlbumNotFound = errors.New("lidarrquarantine: lidarr has no album for that mbid") - ErrLidarrDisabled = errors.New("lidarrquarantine: lidarr is not configured") -) - -// Service owns the request lifecycle. clientFn is a per-call factory so -// config changes in lidarrconfig take effect immediately. clientFn returns -// nil when Lidarr is disabled. -type Service struct { - // dataDir lets a Delete file that empties an artist clear that artist's - // cached art, the same as the admin remove-track path. - dataDir string - pool *pgxpool.Pool - lidarrCfg *lidarrconfig.Service - clientFn func() *lidarr.Client -} - -// NewService constructs a Service. Pass nil for clientFn to disable the -// Lidarr-using methods (DeleteViaLidarr will return ErrLidarrDisabled). -func NewService(pool *pgxpool.Pool, cfg *lidarrconfig.Service, clientFn func() *lidarr.Client, dataDir string) *Service { - if clientFn == nil { - clientFn = func() *lidarr.Client { return nil } - } - return &Service{pool: pool, lidarrCfg: cfg, clientFn: clientFn, dataDir: dataDir} -} - -// Flag inserts or updates a quarantine row for the caller. Re-flagging -// the same (user, track) overwrites reason+notes (and bumps created_at). -func (s *Service) Flag(ctx context.Context, userID, trackID pgtype.UUID, reason string, notes string) (dbq.LidarrQuarantine, error) { - if !validReason(reason) { - return dbq.LidarrQuarantine{}, ErrBadReason - } - var notesPtr *string - if notes != "" { - notesPtr = ¬es - } - row, err := dbq.New(s.pool).UpsertQuarantine(ctx, dbq.UpsertQuarantineParams{ - UserID: userID, - TrackID: trackID, - Reason: dbq.LidarrQuarantineReason(reason), - Notes: notesPtr, - }) - if err != nil { - // FK violation on track_id (no such track) → ErrTrackNotFound so the - // handler can return 404 instead of a generic 500. Code 23503 is the - // Postgres SQLSTATE for foreign_key_violation; the constraint check - // guards against future FKs (e.g. user_id) firing the same code. - var pgErr *pgconn.PgError - if errors.As(err, &pgErr) && pgErr.Code == "23503" && - (pgErr.ConstraintName == "" || strings.Contains(pgErr.ConstraintName, "track")) { - return dbq.LidarrQuarantine{}, ErrTrackNotFound - } - return dbq.LidarrQuarantine{}, fmt.Errorf("upsert: %w", err) - } - return row, nil -} - -// Unflag removes the caller's row. Returns ErrQuarantineNotFound if no -// row exists for that (user, track). -func (s *Service) Unflag(ctx context.Context, userID, trackID pgtype.UUID) error { - _, err := dbq.New(s.pool).DeleteQuarantine(ctx, dbq.DeleteQuarantineParams{ - UserID: userID, TrackID: trackID, - }) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - return ErrQuarantineNotFound - } - return fmt.Errorf("delete: %w", err) - } - return nil -} - -// ListMine returns the caller's quarantines with track + album + artist -// detail. Drives /library/hidden. Ordered newest-first. -func (s *Service) ListMine(ctx context.Context, userID pgtype.UUID) ([]dbq.ListQuarantineForUserRow, error) { - return dbq.New(s.pool).ListQuarantineForUser(ctx, userID) -} - -// AdminQueueRow is the assembled aggregated row served by the admin -// queue endpoint. The handler post-processes the SQL result + a per-track -// ListQuarantineReports call to materialize reason_counts and the -// per-user reports list. -type AdminQueueRow struct { - TrackID pgtype.UUID - TrackTitle string - ArtistName string - AlbumTitle string - AlbumID pgtype.UUID - LidarrAlbumMBID *string - ReportCount int32 - LatestAt pgtype.Timestamptz - ReasonCounts map[string]int - Reports []UserReport -} - -// UserReport is the per-user detail under an aggregated admin row. -type UserReport struct { - UserID pgtype.UUID - Username string - Reason string - Notes *string - CreatedAt pgtype.Timestamptz -} - -// ListAdminQueue returns the aggregated admin queue. One row per track. -// Performs an N+1 (one ListQuarantineReportsForTrack per aggregated row) -// because materializing reason_counts in pure SQL is awkward; the queue -// is small in practice (≲100 tracks at household scale). Revisit with -// json_object_agg if the queue grows. -func (s *Service) ListAdminQueue(ctx context.Context) ([]AdminQueueRow, error) { - q := dbq.New(s.pool) - aggregated, err := q.ListAdminQuarantineQueue(ctx) - if err != nil { - return nil, fmt.Errorf("aggregate: %w", err) - } - out := make([]AdminQueueRow, 0, len(aggregated)) - for _, r := range aggregated { - reports, err := q.ListQuarantineReportsForTrack(ctx, r.TrackID) - if err != nil { - return nil, fmt.Errorf("reports for track %v: %w", r.TrackID, err) - } - rc := make(map[string]int, len(reports)) - userReports := make([]UserReport, 0, len(reports)) - for _, rep := range reports { - rc[string(rep.Reason)]++ - userReports = append(userReports, UserReport{ - UserID: rep.UserID, - Username: rep.Username, - Reason: string(rep.Reason), - Notes: rep.Notes, - CreatedAt: rep.CreatedAt, - }) - } - out = append(out, AdminQueueRow{ - TrackID: r.TrackID, - TrackTitle: r.TrackTitle, - ArtistName: r.ArtistName, - AlbumTitle: r.AlbumTitle, - AlbumID: r.AlbumID, - LidarrAlbumMBID: r.LidarrAlbumMbid, - ReportCount: r.ReportCount, - LatestAt: r.LatestAt, - ReasonCounts: rc, - Reports: userReports, - }) - } - return out, nil -} - -func validReason(r string) bool { - switch r { - case "bad_rip", "wrong_file", "wrong_tags", "duplicate", "other": - return true - } - return false -} - -// Resolve clears all per-user quarantine rows for a track and writes an -// audit log row. Idempotent — a track with no rows still writes an audit -// entry with affected_users=0 (so admin can see "I clicked resolve on a -// track that already had no reports"). -func (s *Service) Resolve(ctx context.Context, trackID, adminID pgtype.UUID) (dbq.LidarrQuarantineAction, error) { - q := dbq.New(s.pool) - track, err := q.GetTrackByID(ctx, trackID) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - return dbq.LidarrQuarantineAction{}, ErrTrackNotFound - } - return dbq.LidarrQuarantineAction{}, fmt.Errorf("get track: %w", err) - } - snap, err := s.snapshot(ctx, q, track) - if err != nil { - return dbq.LidarrQuarantineAction{}, err - } - - affected, err := q.CountQuarantineForTrack(ctx, trackID) - if err != nil { - return dbq.LidarrQuarantineAction{}, fmt.Errorf("count: %w", err) - } - if err := q.DeleteQuarantineForTrack(ctx, trackID); err != nil { - return dbq.LidarrQuarantineAction{}, fmt.Errorf("delete rows: %w", err) - } - // LidarrAlbumMbid stays nil here: Lidarr was not contacted. The audit - // log uses the field as a "this row triggered Lidarr" marker, so only - // DeleteViaLidarr sets it. - return q.WriteQuarantineAction(ctx, dbq.WriteQuarantineActionParams{ - TrackID: trackID, TrackTitle: snap.TrackTitle, ArtistName: snap.ArtistName, - AlbumTitle: snap.AlbumTitle, Action: dbq.LidarrQuarantineActionKindResolved, - AdminID: adminID, LidarrAlbumMbid: nil, AffectedUsers: affected, - }) -} - -// quarantineSnapshot is the audit-row-ready snapshot of track + album + -// artist text columns. Used by the three admin actions. -type quarantineSnapshot struct { - TrackTitle string - ArtistName string - AlbumTitle *string - LidarrAlbumMBID *string -} - -// snapshot pulls the audit-row text columns from track / album / artist. -// AlbumTitle is materialized as *string because WriteQuarantineActionParams -// expects nullable; we coerce the schema's NOT NULL value into a pointer. -func (s *Service) snapshot(ctx context.Context, q *dbq.Queries, track dbq.Track) (quarantineSnapshot, error) { - album, err := q.GetAlbumByID(ctx, track.AlbumID) - if err != nil { - return quarantineSnapshot{}, fmt.Errorf("get album: %w", err) - } - artist, err := q.GetArtistByID(ctx, track.ArtistID) - if err != nil { - return quarantineSnapshot{}, fmt.Errorf("get artist: %w", err) - } - titlePtr := album.Title - return quarantineSnapshot{ - TrackTitle: track.Title, - ArtistName: artist.Name, - AlbumTitle: &titlePtr, - LidarrAlbumMBID: album.Mbid, - }, nil -} - -// DeleteFile removes the track file from disk and the tracks row (tidying away -// an album or artist that leaves empty), then — via FK ON DELETE CASCADE — -// clears every per-user quarantine row for that track and writes an audit row. -// -// If the file cannot be removed, nothing is deleted and the per-user rows stay -// so the admin can retry: the error wraps a *library.FileRemoveError, which the -// handler turns into an answer naming the cause (#3918). No partial state. -func (s *Service) DeleteFile(ctx context.Context, trackID, adminID pgtype.UUID) (dbq.LidarrQuarantineAction, error) { - q := dbq.New(s.pool) - track, err := q.GetTrackByID(ctx, trackID) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - return dbq.LidarrQuarantineAction{}, ErrTrackNotFound - } - return dbq.LidarrQuarantineAction{}, fmt.Errorf("get track: %w", err) - } - snap, err := s.snapshot(ctx, q, track) - if err != nil { - return dbq.LidarrQuarantineAction{}, err - } - - affected, err := q.CountQuarantineForTrack(ctx, trackID) - if err != nil { - return dbq.LidarrQuarantineAction{}, fmt.Errorf("count: %w", err) - } - if _, err := library.DeleteTrackFile(ctx, s.pool, nil, s.dataDir, trackID); err != nil { - return dbq.LidarrQuarantineAction{}, fmt.Errorf("delete file: %w", err) - } - // tracks row is gone; ON DELETE CASCADE on lidarr_quarantine.track_id - // already cleared the per-user rows. We don't call - // DeleteQuarantineForTrack separately. LidarrAlbumMbid stays nil — - // Lidarr was not contacted. - return q.WriteQuarantineAction(ctx, dbq.WriteQuarantineActionParams{ - TrackID: trackID, TrackTitle: snap.TrackTitle, ArtistName: snap.ArtistName, - AlbumTitle: snap.AlbumTitle, Action: dbq.LidarrQuarantineActionKindDeletedFile, - AdminID: adminID, LidarrAlbumMbid: nil, AffectedUsers: affected, - }) -} - -// DeleteViaLidarr is the destructive admin path: tells Lidarr to remove -// the parent album with deleteFiles=true + addImportListExclusion=true, -// then removes Minstrel rows for all tracks of that album. The cascade -// on lidarr_quarantine.track_id clears per-user rows automatically. -// -// On Lidarr failure (unreachable, auth-failed, lookup-empty), nothing -// changes locally. Admin retries. -// -// Returns the audit row + the count of tracks deleted from Minstrel. -func (s *Service) DeleteViaLidarr(ctx context.Context, trackID, adminID pgtype.UUID) (dbq.LidarrQuarantineAction, int, error) { - cfg, err := s.lidarrCfg.Get(ctx) - if err != nil { - return dbq.LidarrQuarantineAction{}, 0, fmt.Errorf("load config: %w", err) - } - client := s.clientFn() - if !cfg.Enabled || client == nil { - return dbq.LidarrQuarantineAction{}, 0, ErrLidarrDisabled - } - - q := dbq.New(s.pool) - track, err := q.GetTrackByID(ctx, trackID) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - return dbq.LidarrQuarantineAction{}, 0, ErrTrackNotFound - } - return dbq.LidarrQuarantineAction{}, 0, fmt.Errorf("get track: %w", err) - } - snap, err := s.snapshot(ctx, q, track) - if err != nil { - return dbq.LidarrQuarantineAction{}, 0, err - } - if snap.LidarrAlbumMBID == nil || *snap.LidarrAlbumMBID == "" { - return dbq.LidarrQuarantineAction{}, 0, ErrAlbumMBIDMissing - } - - affected, err := q.CountQuarantineForTrack(ctx, trackID) - if err != nil { - return dbq.LidarrQuarantineAction{}, 0, fmt.Errorf("count: %w", err) - } - - // Lidarr lookup: MBID -> internal album ID. - album, err := client.LookupAlbumByMBID(ctx, *snap.LidarrAlbumMBID) - if err != nil { - if errors.Is(err, lidarr.ErrNotFound) { - return dbq.LidarrQuarantineAction{}, 0, ErrLidarrAlbumNotFound - } - return dbq.LidarrQuarantineAction{}, 0, fmt.Errorf("lidarr lookup: %w", err) - } - - // Lidarr DELETE — both flags true. - if err := client.DeleteAlbum(ctx, album.ID, true, true); err != nil { - return dbq.LidarrQuarantineAction{}, 0, fmt.Errorf("lidarr delete: %w", err) - } - - // Now remove the local rows. CASCADE on lidarr_quarantine.track_id - // handles per-user rows. - res, err := s.pool.Exec(ctx, "DELETE FROM tracks WHERE album_id = $1", track.AlbumID) - if err != nil { - // Lidarr is already done; we couldn't update locally. Operator- - // recoverable: re-running picks up where we left off. - return dbq.LidarrQuarantineAction{}, 0, fmt.Errorf("delete tracks: %w", err) - } - deletedCount := int(res.RowsAffected()) - - action, err := q.WriteQuarantineAction(ctx, dbq.WriteQuarantineActionParams{ - TrackID: trackID, TrackTitle: snap.TrackTitle, ArtistName: snap.ArtistName, - AlbumTitle: snap.AlbumTitle, Action: dbq.LidarrQuarantineActionKindDeletedViaLidarr, - AdminID: adminID, LidarrAlbumMbid: snap.LidarrAlbumMBID, AffectedUsers: affected, - }) - return action, deletedCount, err -} diff --git a/internal/lidarrrequests/reconciler.go b/internal/lidarrrequests/reconciler.go deleted file mode 100644 index eac3c103..00000000 --- a/internal/lidarrrequests/reconciler.go +++ /dev/null @@ -1,404 +0,0 @@ -package lidarrrequests - -import ( - "context" - "errors" - "fmt" - "log/slog" - "time" - - "github.com/jackc/pgx/v5" - "github.com/jackc/pgx/v5/pgtype" - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/eventbus" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" - "git.fabledsword.com/bvandeusen/minstrel/internal/tags" -) - -// Reconciler is a background worker that periodically scans approved -// lidarr_requests. For rows whose Lidarr add hasn't been confirmed it -// (re)sends the add idempotently until it sticks; for confirmed rows it -// transitions them to completed when their target track/album/artist has -// appeared in the local library (matched by MBID). Errors per-row are -// logged at WARN and do not abort the tick. -type Reconciler struct { - pool *pgxpool.Pool - lidarrCfg *lidarrconfig.Service - clientFn func() *lidarr.Client - logger *slog.Logger - bus *eventbus.Bus - notifier *notifications.Notifier // nil: no inbox notifications - tick time.Duration - batch int32 - // releaseGroup names the MusicBrainz release group of a release id, to - // repair a request stored under a release id Lidarr does not index - // (#5241). A field so tests can stand in for MusicBrainz. - releaseGroup func(ctx context.Context, releaseMBID string) (string, error) -} - -// NewReconciler constructs a Reconciler with production defaults: -// 5-minute tick, batch size 50. clientFn is the per-tick Lidarr client -// factory (same pattern as Service) so config changes apply without a -// restart; pass nil to disable the add-retry path. Pass nil for bus when -// no SSE publishing is desired (tests, or environments without the event -// stream enabled). -func NewReconciler(pool *pgxpool.Pool, cfg *lidarrconfig.Service, clientFn func() *lidarr.Client, logger *slog.Logger, bus *eventbus.Bus) *Reconciler { - if clientFn == nil { - clientFn = func() *lidarr.Client { return nil } - } - return &Reconciler{ - pool: pool, - lidarrCfg: cfg, - clientFn: clientFn, - logger: logger, - bus: bus, - tick: 5 * time.Minute, - batch: 50, - - releaseGroup: tags.ReleaseGroupForRelease, - } -} - -// SetNotifier makes completions land in the requester's notifications inbox -// (M489). Without one, completions are only broadcast on the bus. -func (r *Reconciler) SetNotifier(n *notifications.Notifier) { r.notifier = n } - -// publishCompleted tells the requester their request arrived: a -// request.status_changed event for open screens, and a notification that -// outlives the connection. -func (r *Reconciler) publishCompleted(ctx context.Context, row dbq.LidarrRequest) { - p := notifications.Payload{ - RequestID: formatUUIDForBus(row.ID), - RequestKind: string(row.Kind), - Name: DisplayName(row), - Artist: row.ArtistName, - Title: requestedTitle(row), - } - if row.MatchedAlbumID.Valid { - p.AlbumID = formatUUIDForBus(row.MatchedAlbumID) - } else if row.MatchedArtistID.Valid { - p.ArtistID = formatUUIDForBus(row.MatchedArtistID) - } - r.notifier.NotifyLogged(ctx, notifications.KindRequestCompleted, notifications.ToUser(row.UserID), p.Map()) - - if r.bus == nil { - return - } - r.bus.Publish(eventbus.Event{ - Kind: "request.status_changed", - UserID: formatUUIDForBus(row.UserID), - Data: map[string]any{ - "request_id": formatUUIDForBus(row.ID), - "status": "completed", - "kind": string(row.Kind), - }, - }) -} - -// formatUUIDForBus renders a pgtype.UUID as the canonical 8-4-4-4-12 hex -// string. Mirrors helpers in internal/api/convert.go and other packages; -// inlined here to avoid a back-edge dependency from lidarrrequests onto -// the api layer. -func formatUUIDForBus(u pgtype.UUID) string { - if !u.Valid { - return "" - } - const hex = "0123456789abcdef" - out := make([]byte, 36) - pos := 0 - for i, b := range u.Bytes { - if i == 4 || i == 6 || i == 8 || i == 10 { - out[pos] = '-' - pos++ - } - out[pos] = hex[b>>4] - out[pos+1] = hex[b&0xf] - pos += 2 - } - return string(out) -} - -// Run blocks until ctx is cancelled, ticking every r.tick. Errors from -// tickOnce are logged at WARN and never propagated. -func (r *Reconciler) Run(ctx context.Context) { - t := time.NewTicker(r.tick) - defer t.Stop() - for { - select { - case <-ctx.Done(): - return - case <-t.C: - if err := r.tickOnce(ctx); err != nil { - r.logger.Warn("lidarrrequests: reconciler tick failed", "err", err) - } - } - } -} - -// tickOnce loads approved requests (oldest first, up to r.batch) and -// transitions any whose target MBID is now present in the local library. -// It is exported-for-tests via the lowercase name (package-internal). -func (r *Reconciler) tickOnce(ctx context.Context) error { - cfg, err := r.lidarrCfg.Get(ctx) - if err != nil { - return err - } - if !cfg.Enabled { - r.logger.Debug("lidarrrequests: reconciler skipping tick — lidarr disabled") - return nil - } - - q := dbq.New(r.pool) - rows, err := q.ListApprovedLidarrRequestsForReconcile(ctx, r.batch) - if err != nil { - return err - } - - for _, row := range rows { - if err := r.reconcileRow(ctx, q, cfg, row); err != nil { - r.logger.Warn("lidarrrequests: reconciler row failed", - "request_id", row.ID, - "kind", row.Kind, - "err", err, - ) - } - } - return nil -} - -// reconcileRow drives a request forward by one step. If Lidarr hasn't -// confirmed the add yet it (re)sends it and returns — the import can't -// land before Lidarr has the add, so there's nothing to match this tick. -// Once confirmed, it checks the library for a match and, if found, calls -// CompleteLidarrRequest. Returns an error only for unexpected failures -// (the add retry's transient error included, so tickOnce logs WARN and -// it's retried next tick — "Lidarr just keeps trying"); a no-match is -// not an error. -func (r *Reconciler) reconcileRow(ctx context.Context, q *dbq.Queries, cfg lidarrconfig.Config, row dbq.LidarrRequest) error { - if !row.LidarrAddConfirmedAt.Valid { - if err := r.ensureLidarrAdd(ctx, q, cfg, row); err != nil { - return err - } - // ensureLidarrAdd either confirmed the add or no-op'd (Lidarr - // client unavailable). Fall through to import-matching so an - // item already present in the library still completes this tick. - } - switch row.Kind { - case dbq.LidarrRequestKindArtist: - return r.reconcileArtist(ctx, q, row) - case dbq.LidarrRequestKindAlbum: - return r.reconcileAlbum(ctx, q, row) - case dbq.LidarrRequestKindTrack: - return r.reconcileTrack(ctx, q, row) - default: - r.logger.Warn("lidarrrequests: reconciler unknown kind", "kind", row.Kind) - return nil - } -} - -// ensureLidarrAdd (re)sends the Lidarr add for an approved request whose -// add hasn't been confirmed. On success (including ErrAlreadyExists, -// handled inside sendLidarrAdd) it stamps lidarr_add_confirmed_at so this -// stops re-sending and the next tick moves to import-matching. A failure -// is returned so tickOnce logs WARN and the row is retried next tick; -// there is intentionally no failed-state or attempt cap — per product -// decision Lidarr just keeps trying and the operator monitors Lidarr. -func (r *Reconciler) ensureLidarrAdd(ctx context.Context, q *dbq.Queries, cfg lidarrconfig.Config, row dbq.LidarrRequest) error { - client := r.clientFn() - if client == nil { - // Lidarr disabled mid-flight; tickOnce already gates on - // cfg.Enabled, so this is just defensive — nothing to do. - return nil - } - err := sendLidarrAdd(ctx, client, cfg, row) - if errors.Is(err, lidarr.ErrNotFound) { - // Lidarr's metadata has no album under this id. Requests made before - // albums carried a release group (#5241) name the album by its - // MusicBrainz release id, which Lidarr does not index: repoint the - // request at the release group and try once more. - if repaired, ok := r.repointAtReleaseGroup(ctx, q, row); ok { - row = repaired - err = sendLidarrAdd(ctx, client, cfg, row) - } - } - if err != nil { - return fmt.Errorf("lidarr add retry: %w", err) - } - return q.MarkLidarrRequestAddConfirmed(ctx, row.ID) -} - -// repointAtReleaseGroup reads an album or track request's album id as a -// MusicBrainz release and, when its release group can be named, rewrites the -// request to it. The library's own albums answer first; MusicBrainz is asked -// only for a release no album row knows the group of, and its answer is cached -// onto that album. Reports false when there is nothing to repoint: not an -// album request, the group is unknown, or the id already is the group. -func (r *Reconciler) repointAtReleaseGroup(ctx context.Context, q *dbq.Queries, row dbq.LidarrRequest) (dbq.LidarrRequest, bool) { - if row.Kind == dbq.LidarrRequestKindArtist || row.LidarrAlbumMbid == nil || *row.LidarrAlbumMbid == "" { - return row, false - } - release := *row.LidarrAlbumMbid - group, err := q.GetReleaseGroupForReleaseMbid(ctx, &release) - if err != nil { - if !isNoRows(err) { - r.logger.Warn("lidarrrequests: release group lookup failed", "request_id", row.ID, "err", err) - return row, false - } - group, err = r.releaseGroup(ctx, release) - if err != nil { - if !errors.Is(err, tags.ErrNotFound) { - r.logger.Warn("lidarrrequests: MusicBrainz release group lookup failed", - "request_id", row.ID, "release_mbid", release, "err", err) - } - return row, false - } - if cerr := q.SetReleaseGroupForReleaseMbidIfNull(ctx, dbq.SetReleaseGroupForReleaseMbidIfNullParams{ - ReleaseGroupMbid: group, - ReleaseMbid: release, - }); cerr != nil { - r.logger.Warn("lidarrrequests: cache release group failed", "release_mbid", release, "err", cerr) - } - } - if group == "" || group == release { - return row, false - } - if err := q.SetLidarrRequestAlbumMbid(ctx, dbq.SetLidarrRequestAlbumMbidParams{ - ID: row.ID, - LidarrAlbumMbid: &group, - }); err != nil { - r.logger.Warn("lidarrrequests: repoint request failed", "request_id", row.ID, "err", err) - return row, false - } - r.logger.Info("lidarrrequests: request repointed from release to release group", - "request_id", row.ID, "release_mbid", release, "release_group_mbid", group) - row.LidarrAlbumMbid = &group - return row, true -} - -// albumForRequest finds the library album a request's album id ($1) names, by -// release id or release group, once the request ($2 = requested_at) has been -// answered: the album has a track on disk AND either -// -// - a track arrived after the request (a new album, or Lidarr fetching -// another release of the group into its own row), or -// - no track that was already missing when the request was made is still -// missing (the lost files came back in place or were adopted at a new path). -// -// "Has a track on disk" alone is not enough (#5263): re-acquisition targets -// albums with ANY track missing, so the tracks that never left satisfied it -// the moment Lidarr accepted the add. added_at is the arrival clock because -// nothing rewrites it; updated_at moves on every tag re-read. -// An exact release match is preferred over another release of the group. -const albumForRequest = ` -SELECT a.id - FROM albums a - WHERE (a.mbid = $1 OR a.release_group_mbid = $1) - AND EXISTS (SELECT 1 FROM tracks t WHERE t.album_id = a.id AND t.missing_since IS NULL) - AND ( - EXISTS (SELECT 1 FROM tracks t - WHERE t.album_id = a.id AND t.missing_since IS NULL AND t.added_at > $2) - OR NOT EXISTS (SELECT 1 FROM tracks t - WHERE t.album_id = a.id AND t.missing_since IS NOT NULL - AND t.missing_since < $2) - ) - ORDER BY (a.mbid = $1) DESC, a.id - LIMIT 1` - -func (r *Reconciler) reconcileArtist(ctx context.Context, q *dbq.Queries, row dbq.LidarrRequest) error { - var artistID pgtype.UUID - err := r.pool.QueryRow(ctx, - "SELECT id FROM artists WHERE mbid = $1", - row.LidarrArtistMbid, - ).Scan(&artistID) - if err != nil { - if isNoRows(err) { - return nil // not in library yet - } - return err - } - completed, err := q.CompleteLidarrRequest(ctx, dbq.CompleteLidarrRequestParams{ - ID: row.ID, - MatchedArtistID: artistID, - MatchedAlbumID: pgtype.UUID{}, - MatchedTrackID: pgtype.UUID{}, - }) - if err != nil { - return err - } - r.publishCompleted(ctx, completed) - return nil -} - -func (r *Reconciler) reconcileAlbum(ctx context.Context, q *dbq.Queries, row dbq.LidarrRequest) error { - if row.LidarrAlbumMbid == nil { - return nil - } - var albumID pgtype.UUID - err := r.pool.QueryRow(ctx, albumForRequest, *row.LidarrAlbumMbid, row.RequestedAt).Scan(&albumID) - if err != nil { - if isNoRows(err) { - return nil - } - return err - } - completed, err := q.CompleteLidarrRequest(ctx, dbq.CompleteLidarrRequestParams{ - ID: row.ID, - MatchedAlbumID: albumID, - MatchedArtistID: pgtype.UUID{}, - MatchedTrackID: pgtype.UUID{}, - }) - if err != nil { - return err - } - r.publishCompleted(ctx, completed) - return nil -} - -func (r *Reconciler) reconcileTrack(ctx context.Context, q *dbq.Queries, row dbq.LidarrRequest) error { - if row.LidarrAlbumMbid == nil { - return nil - } - // Track-kind requests match via their parent album's MBID, not track.mbid. - var albumID pgtype.UUID - err := r.pool.QueryRow(ctx, albumForRequest, *row.LidarrAlbumMbid, row.RequestedAt).Scan(&albumID) - if err != nil { - if isNoRows(err) { - return nil - } - return err - } - - // Load a track from that album to set matched_track_id, newest arrival first. - var trackID pgtype.UUID - err = r.pool.QueryRow(ctx, - "SELECT id FROM tracks WHERE album_id = $1 AND missing_since IS NULL ORDER BY added_at DESC, id LIMIT 1", - albumID, - ).Scan(&trackID) - if err != nil { - if isNoRows(err) { - // Album is in the library but no tracks ingested yet — try again next tick. - return nil - } - return err - } - - completed, err := q.CompleteLidarrRequest(ctx, dbq.CompleteLidarrRequestParams{ - ID: row.ID, - MatchedAlbumID: albumID, - MatchedTrackID: trackID, - MatchedArtistID: pgtype.UUID{}, - }) - if err != nil { - return err - } - r.publishCompleted(ctx, completed) - return nil -} - -func isNoRows(err error) bool { - return err == pgx.ErrNoRows -} diff --git a/internal/lidarrrequests/reconciler_integration_test.go b/internal/lidarrrequests/reconciler_integration_test.go deleted file mode 100644 index 6458f83f..00000000 --- a/internal/lidarrrequests/reconciler_integration_test.go +++ /dev/null @@ -1,444 +0,0 @@ -package lidarrrequests - -import ( - "context" - "io" - "log/slog" - "testing" - "time" - - "github.com/jackc/pgx/v5/pgtype" - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" -) - -// seedApprovedRequestDirect inserts a pending request and immediately -// approves it via ApproveLidarrRequest, bypassing the Lidarr API call. -func seedApprovedRequestDirect(t *testing.T, q *dbq.Queries, userID pgtype.UUID, p CreateParams) dbq.LidarrRequest { - t.Helper() - ctx := context.Background() - row, err := q.CreateLidarrRequest(ctx, dbq.CreateLidarrRequestParams{ - UserID: userID, - Kind: dbq.LidarrRequestKind(p.Kind), - LidarrArtistMbid: p.LidarrArtistMBID, - LidarrAlbumMbid: nilableStr(p.LidarrAlbumMBID), - LidarrTrackMbid: nilableStr(p.LidarrTrackMBID), - ArtistName: p.ArtistName, - AlbumTitle: nilableStr(p.AlbumTitle), - TrackTitle: nilableStr(p.TrackTitle), - }) - if err != nil { - t.Fatalf("seedApprovedRequestDirect create: %v", err) - } - approved, err := q.ApproveLidarrRequest(ctx, dbq.ApproveLidarrRequestParams{ - ID: row.ID, - QualityProfileID: nil, - RootFolderPath: nil, - DecidedBy: userID, - }) - if err != nil { - t.Fatalf("seedApprovedRequestDirect approve: %v", err) - } - return approved -} - -func nilableStr(s string) *string { - if s == "" { - return nil - } - return &s -} - -func seedArtist(t *testing.T, q *dbq.Queries, name, mbid string) dbq.Artist { - t.Helper() - a, err := q.UpsertArtist(context.Background(), dbq.UpsertArtistParams{ - Name: name, SortName: name, Mbid: &mbid, - }) - if err != nil { - t.Fatalf("seedArtist: %v", err) - } - return a -} - -func seedAlbum(t *testing.T, q *dbq.Queries, artistID pgtype.UUID, title, mbid string) dbq.Album { - t.Helper() - a, err := q.UpsertAlbum(context.Background(), dbq.UpsertAlbumParams{ - Title: title, SortTitle: title, - ArtistID: artistID, - Mbid: &mbid, - }) - if err != nil { - t.Fatalf("seedAlbum: %v", err) - } - return a -} - -func seedTrack(t *testing.T, q *dbq.Queries, albumID, artistID pgtype.UUID, title, filePath string) dbq.Track { - t.Helper() - tr, err := q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ - Title: title, AlbumID: albumID, ArtistID: artistID, - DurationMs: 180000, FilePath: filePath, - FileSize: 1024, FileFormat: "flac", - }) - if err != nil { - t.Fatalf("seedTrack: %v", err) - } - return tr -} - -func enableLidarrForPool(t *testing.T, pool *pgxpool.Pool) { - t.Helper() - if err := lidarrconfig.New(pool).Save(context.Background(), lidarrconfig.Config{ - Enabled: true, BaseURL: "http://lidarr:8686", APIKey: "test", - }); err != nil { - t.Fatalf("enableLidarrForPool: %v", err) - } -} - -func newTestLogger() *slog.Logger { - return slog.New(slog.NewTextHandler(io.Discard, nil)) -} - -// --------------------------------------------------------------------------- -// Tests -// --------------------------------------------------------------------------- - -// TestReconciler_MatchesArtistByMBID: an approved artist-kind request whose -// MBID is in the artists table gets completed with matched_artist_id set. -func TestReconciler_MatchesArtistByMBID(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - artistMBID := "069b64b6-7884-4f6a-94cc-e4c1d6c87a01" - artist := seedArtist(t, q, "Boards of Canada", artistMBID) - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "artist", LidarrArtistMBID: artistMBID, ArtistName: "Boards of Canada", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusCompleted { - t.Errorf("status = %v, want completed", got.Status) - } - if got.MatchedArtistID != artist.ID { - t.Errorf("matched_artist_id = %v, want %v", got.MatchedArtistID, artist.ID) - } -} - -// TestReconciler_MatchesAlbumByMBID: an approved album-kind request whose -// album MBID is in the albums table gets completed with matched_album_id set. -func TestReconciler_MatchesAlbumByMBID(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - artistMBID := "a-artist-mbid-album-test" - albumMBID := "b-album-mbid-album-test" - - artist := seedArtist(t, q, "Test Artist", artistMBID) - album := seedAlbum(t, q, artist.ID, "Test Album", albumMBID) - _ = seedTrack(t, q, album.ID, artist.ID, "Album Track", "/music/album-test/01.flac") - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: artistMBID, ArtistName: "Test Artist", - LidarrAlbumMBID: albumMBID, AlbumTitle: "Test Album", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusCompleted { - t.Errorf("status = %v, want completed", got.Status) - } - if got.MatchedAlbumID != album.ID { - t.Errorf("matched_album_id = %v, want %v", got.MatchedAlbumID, album.ID) - } -} - -// TestReconciler_MatchesTrackViaAlbumMBID: an approved track-kind request -// matches when the parent album appears in the library. matched_track_id is -// also set to a track from that album. The track MBID on the request does -// not need to match any tracks.mbid — matching is via the parent album. -func TestReconciler_MatchesTrackViaAlbumMBID(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - artistMBID := "a-artist-mbid-track-test" - albumMBID := "b-album-mbid-track-test" - trackMBID := "c-track-mbid-does-not-exist-in-tracks-table" - - artist := seedArtist(t, q, "Track Test Artist", artistMBID) - album := seedAlbum(t, q, artist.ID, "Track Test Album", albumMBID) - track := seedTrack(t, q, album.ID, artist.ID, "Track One", "/music/track-test/01.flac") - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "track", LidarrArtistMBID: artistMBID, ArtistName: "Track Test Artist", - LidarrAlbumMBID: albumMBID, AlbumTitle: "Track Test Album", - LidarrTrackMBID: trackMBID, TrackTitle: "Track One", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusCompleted { - t.Errorf("status = %v, want completed", got.Status) - } - if got.MatchedAlbumID != album.ID { - t.Errorf("matched_album_id = %v, want %v", got.MatchedAlbumID, album.ID) - } - if got.MatchedTrackID != track.ID { - t.Errorf("matched_track_id = %v, want %v", got.MatchedTrackID, track.ID) - } -} - -// TestReconciler_NoMatchLeavesPending: an approved request whose MBID is not -// in any library table is left unchanged after tickOnce. -func TestReconciler_NoMatchLeavesPending(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "artist", LidarrArtistMBID: "nonexistent-mbid-xyz", ArtistName: "Ghost Artist", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusApproved { - t.Errorf("status = %v, want approved (unchanged)", got.Status) - } - if got.MatchedArtistID.Valid { - t.Errorf("matched_artist_id should be NULL, got %v", got.MatchedArtistID) - } -} - -// TestReconciler_AlreadyCompletedRowNotReprocessed: a completed row is not -// touched by the reconciler because ListApprovedLidarrRequestsForReconcile -// filters on status='approved'. -func TestReconciler_AlreadyCompletedRowNotReprocessed(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - artistMBID := "already-done-artist-mbid" - artist := seedArtist(t, q, "Already Done Artist", artistMBID) - - // Seed approved, then complete it before running tickOnce. - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "artist", LidarrArtistMBID: artistMBID, ArtistName: "Already Done Artist", - }) - completed, err := q.CompleteLidarrRequest(ctx, dbq.CompleteLidarrRequestParams{ - ID: req.ID, - MatchedArtistID: artist.ID, - MatchedAlbumID: pgtype.UUID{}, - MatchedTrackID: pgtype.UUID{}, - }) - if err != nil { - t.Fatalf("CompleteLidarrRequest setup: %v", err) - } - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusCompleted { - t.Errorf("status = %v, want completed (unchanged)", got.Status) - } - if !got.UpdatedAt.Time.Equal(completed.UpdatedAt.Time) { - t.Errorf("updated_at moved: was %v, now %v — reconciler should not have touched completed row", - completed.UpdatedAt.Time, got.UpdatedAt.Time) - } -} - -// TestReconciler_DisabledIsNoOp: when lidarr_config.enabled=false the -// reconciler skips the entire tick, leaving approved requests untouched. -func TestReconciler_DisabledIsNoOp(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - // lidarr_config is left disabled (newPool sets it to false). - user := seedUser(t, pool) - artistMBID := "disabled-noop-mbid" - // Seed the matching artist so it WOULD be found if lidarr were enabled. - seedArtist(t, q, "No-Op Artist", artistMBID) - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "artist", LidarrArtistMBID: artistMBID, ArtistName: "No-Op Artist", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusApproved { - t.Errorf("status = %v, want approved — reconciler should no-op when disabled", got.Status) - } -} - -// TestReconciler_RunRunsAtLeastOneTick exercises Run's goroutine loop with a -// short tick, verifies it picks up an approved row and transitions it to -// completed before the test cancels the context. -func TestReconciler_RunRunsAtLeastOneTick(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - pool := newPool(t) - enableLidarrForPool(t, pool) - q := dbq.New(pool) - user := seedUser(t, pool) - - const mbid = "run-mbid" - seedArtist(t, q, "Run Artist", mbid) - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "artist", LidarrArtistMBID: mbid, ArtistName: "Run Artist", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - rec.tick = 25 * time.Millisecond - - done := make(chan struct{}) - go func() { - rec.Run(ctx) - close(done) - }() - - // Poll the row up to ~2s for the reconciler to flip it. Beats sleeping - // a fixed amount on slow CI. - deadline := time.Now().Add(2 * time.Second) - for { - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err == nil && got.Status == dbq.LidarrRequestStatusCompleted { - break - } - if time.Now().After(deadline) { - t.Fatalf("status not completed within 2s; last err=%v", err) - } - time.Sleep(10 * time.Millisecond) - } - - cancel() - select { - case <-done: - case <-time.After(time.Second): - t.Fatal("Run did not exit within 1s of cancel") - } -} - -// TestReconciler_TrackKindNoTracksInAlbumYet: the parent album exists in the -// library but no track rows have been ingested yet. Reconciler should leave -// the row approved so a subsequent tick can pick it up once tracks land. -func TestReconciler_TrackKindNoTracksInAlbumYet(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - artistMBID := "ar-mbid-tracks-pending" - albumMBID := "al-mbid-tracks-pending" - - artist := seedArtist(t, q, "Pending Tracks Artist", artistMBID) - _ = seedAlbum(t, q, artist.ID, "Album Without Tracks", albumMBID) - // No tracks seeded. - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "track", LidarrArtistMBID: artistMBID, ArtistName: "Pending Tracks Artist", - LidarrAlbumMBID: albumMBID, AlbumTitle: "Album Without Tracks", - LidarrTrackMBID: "t-mbid-not-yet", TrackTitle: "Some Track", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusApproved { - t.Errorf("status = %v, want approved (album present, no tracks yet)", got.Status) - } -} - -// TestReconciler_AlbumKindNoMatchInAlbumsTable exercises the album branch's -// no-rows return path when the request's album_mbid isn't in the library yet. -func TestReconciler_AlbumKindNoMatchInAlbumsTable(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: "ar-no-match", ArtistName: "Nope Artist", - LidarrAlbumMBID: "al-not-in-library", AlbumTitle: "Nope Album", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatalf("GetLidarrRequestByID: %v", err) - } - if got.Status != dbq.LidarrRequestStatusApproved { - t.Errorf("status = %v, want approved (album not yet in library)", got.Status) - } -} diff --git a/internal/lidarrrequests/reconciler_notify_test.go b/internal/lidarrrequests/reconciler_notify_test.go deleted file mode 100644 index 53e71751..00000000 --- a/internal/lidarrrequests/reconciler_notify_test.go +++ /dev/null @@ -1,52 +0,0 @@ -package lidarrrequests - -import ( - "context" - "encoding/json" - "testing" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" -) - -// M489: an album request that arrives lands in the requester's inbox once, -// linked to the album it matched. -func TestReconciler_CompletionNotifiesRequester(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - artist := seedArtist(t, q, "Notify Artist", "notify-artist-mbid") - album := seedAlbum(t, q, artist.ID, "Notify Album", "notify-album-mbid") - _ = seedTrack(t, q, album.ID, artist.ID, "Notify Track", "/music/notify/01.flac") - seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: "notify-artist-mbid", LidarrAlbumMBID: "notify-album-mbid", - ArtistName: "Notify Artist", AlbumTitle: "Notify Album", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - rec.SetNotifier(notifications.New(pool, nil, nil)) - for i := 0; i < 2; i++ { // the second tick finds nothing left to complete - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - } - - rows, err := q.ListNotifications(ctx, dbq.ListNotificationsParams{UserID: user, PageLimit: 10}) - if err != nil { - t.Fatalf("list notifications: %v", err) - } - if len(rows) != 1 || rows[0].Kind != string(notifications.KindRequestCompleted) { - t.Fatalf("inbox = %+v, want one request_completed", rows) - } - var p notifications.Payload - if err := json.Unmarshal(rows[0].Payload, &p); err != nil { - t.Fatal(err) - } - if p.Name != "Notify Artist – Notify Album" || p.AlbumID != formatUUIDForBus(album.ID) || p.ArtistID != "" { - t.Errorf("payload = %+v", p) - } -} diff --git a/internal/lidarrrequests/reconciler_release_group_test.go b/internal/lidarrrequests/reconciler_release_group_test.go deleted file mode 100644 index 696e2a4d..00000000 --- a/internal/lidarrrequests/reconciler_release_group_test.go +++ /dev/null @@ -1,373 +0,0 @@ -package lidarrrequests - -import ( - "context" - "encoding/json" - "io/fs" - "net/http" - "net/http/httptest" - "strings" - "sync" - "testing" - - "github.com/jackc/pgx/v5/pgtype" - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db" - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/tags" -) - -// #5241: Lidarr names albums by MusicBrainz release group. A request carrying -// a group id completes against the album row whose tags named that group, -// once a track of it is on disk. -func TestReconciler_AlbumMatchesByReleaseGroup(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - - const artistMBID, release, group = "rg-artist-match", "rg-release-match", "rg-group-match" - artist := seedArtist(t, q, "RG Artist", artistMBID) - album := seedAlbum(t, q, artist.ID, "RG Album", release) - if err := q.SetAlbumReleaseGroupMbidIfNull(ctx, dbq.SetAlbumReleaseGroupMbidIfNullParams{ - ID: album.ID, ReleaseGroupMbid: nilableStr(group), - }); err != nil { - t.Fatal(err) - } - _ = seedTrack(t, q, album.ID, artist.ID, "RG One", "/music/rg-match/01.flac") - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: artistMBID, ArtistName: "RG Artist", - LidarrAlbumMBID: group, AlbumTitle: "RG Album", - }) - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatal(err) - } - if got.Status != dbq.LidarrRequestStatusCompleted || got.MatchedAlbumID != album.ID { - t.Errorf("status %v matched %v, want completed on %v", got.Status, got.MatchedAlbumID, album.ID) - } -} - -// A re-acquisition request names an album whose row never went away. The row -// alone must not complete it: nothing has come back until a track is on disk. -func TestReconciler_AlbumWithEveryTrackMissingStaysApproved(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - - const artistMBID, release = "rg-artist-missing", "rg-release-missing" - artist := seedArtist(t, q, "Missing Artist", artistMBID) - album := seedAlbum(t, q, artist.ID, "Missing Album", release) - tr := seedTrack(t, q, album.ID, artist.ID, "Gone", "/music/rg-missing/01.flac") - if _, err := pool.Exec(ctx, "UPDATE tracks SET missing_since = now() WHERE id = $1", tr.ID); err != nil { - t.Fatal(err) - } - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: artistMBID, ArtistName: "Missing Artist", - LidarrAlbumMBID: release, AlbumTitle: "Missing Album", - }) - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatal(err) - } - if got.Status != dbq.LidarrRequestStatusApproved { - t.Errorf("status = %v, want approved while every track is missing", got.Status) - } -} - -// fakeAlbumLidarr answers album lookups only for knownGroup (Lidarr's metadata -// is keyed by release group) and records every add. -type fakeAlbumLidarr struct { - knownGroup string - mu sync.Mutex - added []string -} - -func (f *fakeAlbumLidarr) handler(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - switch { - case strings.HasSuffix(r.URL.Path, "/metadataprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Standard"}]`)) - case strings.HasSuffix(r.URL.Path, "/album/lookup"): - if r.URL.Query().Get("term") == "lidarr:"+f.knownGroup { - _, _ = w.Write([]byte(`[{"foreignAlbumId":"` + f.knownGroup + `","artist":{"id":42}}]`)) - return - } - _, _ = w.Write([]byte(`[]`)) - case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/album"): - var body map[string]any - _ = json.NewDecoder(r.Body).Decode(&body) - f.mu.Lock() - f.added = append(f.added, body["foreignAlbumId"].(string)) - f.mu.Unlock() - w.WriteHeader(http.StatusCreated) - _, _ = w.Write([]byte(`{"id":9}`)) - default: - w.WriteHeader(http.StatusNotFound) - } -} - -// The ~49 requests on the deploy were stored under release ids. The reconciler -// resolves the group (MusicBrainz here, since no album row knows it), rewrites -// the request, adds it, and caches the group onto the album holding the -// release. -func TestReconciler_RepointsAReleaseIdRequestAtItsReleaseGroup(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - user := seedUser(t, pool) - - const artistMBID, release, group = "rg-artist-repoint", "rg-release-repoint", "rg-group-repoint" - fake := &fakeAlbumLidarr{knownGroup: group} - srv := httptest.NewServer(http.HandlerFunc(fake.handler)) - t.Cleanup(srv.Close) - if err := lidarrconfig.New(pool).Save(ctx, lidarrconfig.Config{ - Enabled: true, BaseURL: srv.URL, APIKey: "k", - DefaultQualityProfileID: 7, DefaultRootFolderPath: "/music", - }); err != nil { - t.Fatal(err) - } - - artist := seedArtist(t, q, "Repoint Artist", artistMBID) - album := seedAlbum(t, q, artist.ID, "Repoint Album", release) - tr := seedTrack(t, q, album.ID, artist.ID, "Lost", "/music/rg-repoint/01.flac") - if _, err := pool.Exec(ctx, "UPDATE tracks SET missing_since = now() WHERE id = $1", tr.ID); err != nil { - t.Fatal(err) - } - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: artistMBID, ArtistName: "Repoint Artist", - LidarrAlbumMBID: release, AlbumTitle: "Repoint Album", - }) - - rec := NewReconciler(pool, lidarrconfig.New(pool), - func() *lidarr.Client { return lidarr.NewClient(srv.URL, "k") }, newTestLogger(), nil) - asked := 0 - rec.releaseGroup = func(_ context.Context, id string) (string, error) { - asked++ - if id == release { - return group, nil - } - return "", tags.ErrNotFound - } - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - - got, err := q.GetLidarrRequestByID(ctx, req.ID) - if err != nil { - t.Fatal(err) - } - if got.LidarrAlbumMbid == nil || *got.LidarrAlbumMbid != group { - t.Errorf("request album mbid = %v, want the release group %s", got.LidarrAlbumMbid, group) - } - if !got.LidarrAddConfirmedAt.Valid { - t.Error("add not confirmed after the repoint") - } - fake.mu.Lock() - added := append([]string(nil), fake.added...) - fake.mu.Unlock() - if len(added) != 1 || added[0] != group { - t.Errorf("Lidarr adds = %v, want [%s]", added, group) - } - var cached *string - if err := pool.QueryRow(ctx, "SELECT release_group_mbid FROM albums WHERE id = $1", album.ID).Scan(&cached); err != nil { - t.Fatal(err) - } - if cached == nil || *cached != group { - t.Errorf("album release group = %v, want %s cached", cached, group) - } - if got.Status != dbq.LidarrRequestStatusApproved { - t.Errorf("status = %v, want approved until a track is back", got.Status) - } - - // The next tick finds the group on the request and the album: no second - // MusicBrainz question. - before := asked - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("second tick: %v", err) - } - if asked != before { - t.Errorf("MusicBrainz asked again on the next tick (%d -> %d)", before, asked) - } -} - -// markMissing stamps a track missing since well before any request a test makes. -func markMissing(t *testing.T, pool *pgxpool.Pool, trackID pgtype.UUID) { - t.Helper() - if _, err := pool.Exec(context.Background(), - "UPDATE tracks SET missing_since = now() - interval '2 days' WHERE id = $1", trackID); err != nil { - t.Fatal(err) - } -} - -func requestStatus(t *testing.T, q *dbq.Queries, id pgtype.UUID) dbq.LidarrRequest { - t.Helper() - got, err := q.GetLidarrRequestByID(context.Background(), id) - if err != nil { - t.Fatal(err) - } - return got -} - -// #5263: re-acquisition targets albums with SOME tracks missing. The tracks -// that never left must not complete the request; it completes once the lost -// track is back. -func TestReconciler_PartlyMissingAlbumCompletesWhenTheLostTrackReturns(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - - const artistMBID, release = "rg-artist-partial", "rg-release-partial" - artist := seedArtist(t, q, "Partial Artist", artistMBID) - album := seedAlbum(t, q, artist.ID, "Partial Album", release) - _ = seedTrack(t, q, album.ID, artist.ID, "Kept", "/music/rg-partial/01.flac") - lost := seedTrack(t, q, album.ID, artist.ID, "Lost", "/music/rg-partial/02.flac") - markMissing(t, pool, lost.ID) - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: artistMBID, ArtistName: "Partial Artist", - LidarrAlbumMBID: release, AlbumTitle: "Partial Album", - }) - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - if got := requestStatus(t, q, req.ID); got.Status != dbq.LidarrRequestStatusApproved { - t.Fatalf("status = %v while a track is still missing, want approved", got.Status) - } - - if _, err := pool.Exec(ctx, "UPDATE tracks SET missing_since = NULL WHERE id = $1", lost.ID); err != nil { - t.Fatal(err) - } - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("second tick: %v", err) - } - got := requestStatus(t, q, req.ID) - if got.Status != dbq.LidarrRequestStatusCompleted || got.MatchedAlbumID != album.ID { - t.Errorf("status %v matched %v after the track returned, want completed on %v", - got.Status, got.MatchedAlbumID, album.ID) - } -} - -// Lidarr may fetch a different release of the group, which the scanner files -// as its own album row. A track arriving there after the request completes it, -// even though the old row's lost tracks stay missing. -func TestReconciler_PartlyMissingAlbumCompletesOnAnotherReleaseArriving(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - enableLidarrForPool(t, pool) - user := seedUser(t, pool) - - const artistMBID, oldRelease, newRelease, group = "rg-artist-edition", "rg-release-old", "rg-release-new", "rg-group-edition" - artist := seedArtist(t, q, "Edition Artist", artistMBID) - old := seedAlbum(t, q, artist.ID, "Edition Album", oldRelease) - if err := q.SetAlbumReleaseGroupMbidIfNull(ctx, dbq.SetAlbumReleaseGroupMbidIfNullParams{ - ID: old.ID, ReleaseGroupMbid: nilableStr(group), - }); err != nil { - t.Fatal(err) - } - _ = seedTrack(t, q, old.ID, artist.ID, "Kept", "/music/rg-edition-old/01.flac") - lost := seedTrack(t, q, old.ID, artist.ID, "Lost", "/music/rg-edition-old/02.flac") - markMissing(t, pool, lost.ID) - - req := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: artistMBID, ArtistName: "Edition Artist", - LidarrAlbumMBID: group, AlbumTitle: "Edition Album", - }) - rec := NewReconciler(pool, lidarrconfig.New(pool), nil, newTestLogger(), nil) - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("tickOnce: %v", err) - } - if got := requestStatus(t, q, req.ID); got.Status != dbq.LidarrRequestStatusApproved { - t.Fatalf("status = %v before anything arrived, want approved", got.Status) - } - - arrived := seedAlbum(t, q, artist.ID, "Edition Album (Deluxe)", newRelease) - if err := q.SetAlbumReleaseGroupMbidIfNull(ctx, dbq.SetAlbumReleaseGroupMbidIfNullParams{ - ID: arrived.ID, ReleaseGroupMbid: nilableStr(group), - }); err != nil { - t.Fatal(err) - } - _ = seedTrack(t, q, arrived.ID, artist.ID, "Lost", "/music/rg-edition-new/02.flac") - if err := rec.tickOnce(ctx); err != nil { - t.Fatalf("second tick: %v", err) - } - got := requestStatus(t, q, req.ID) - if got.Status != dbq.LidarrRequestStatusCompleted || got.MatchedAlbumID != arrived.ID { - t.Errorf("status %v matched %v, want completed on the arrived release %v", - got.Status, got.MatchedAlbumID, arrived.ID) - } -} - -// Migration 0071 reopens requests the old check completed against a partly -// missing album, and leaves a genuine completion alone. -func TestMigration0071_ReopensFalseCompletions(t *testing.T) { - pool := newPool(t) - q := dbq.New(pool) - ctx := context.Background() - user := seedUser(t, pool) - - artist := seedArtist(t, q, "Reopen Artist", "rg-artist-reopen") - partial := seedAlbum(t, q, artist.ID, "Partial", "rg-release-reopen-partial") - _ = seedTrack(t, q, partial.ID, artist.ID, "Kept", "/music/rg-reopen-partial/01.flac") - lost := seedTrack(t, q, partial.ID, artist.ID, "Lost", "/music/rg-reopen-partial/02.flac") - markMissing(t, pool, lost.ID) - falseReq := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: "rg-artist-reopen", ArtistName: "Reopen Artist", - LidarrAlbumMBID: "rg-release-reopen-partial", AlbumTitle: "Partial", - }) - - genuineReq := seedApprovedRequestDirect(t, q, user, CreateParams{ - Kind: "album", LidarrArtistMBID: "rg-artist-reopen", ArtistName: "Reopen Artist", - LidarrAlbumMBID: "rg-release-reopen-new", AlbumTitle: "New", - }) - arrived := seedAlbum(t, q, artist.ID, "New", "rg-release-reopen-new") - _ = seedTrack(t, q, arrived.ID, artist.ID, "Fresh", "/music/rg-reopen-new/01.flac") - - for _, c := range []struct { - id pgtype.UUID - album pgtype.UUID - }{{falseReq.ID, partial.ID}, {genuineReq.ID, arrived.ID}} { - if _, err := q.CompleteLidarrRequest(ctx, dbq.CompleteLidarrRequestParams{ - ID: c.id, MatchedAlbumID: c.album, - }); err != nil { - t.Fatal(err) - } - } - - up, err := fs.ReadFile(db.MigrationsFS(), "0071_reopen_false_reacquisition_completions.up.sql") - if err != nil { - t.Fatal(err) - } - if _, err := pool.Exec(ctx, string(up)); err != nil { - t.Fatalf("migration: %v", err) - } - - got := requestStatus(t, q, falseReq.ID) - if got.Status != dbq.LidarrRequestStatusApproved || got.CompletedAt.Valid || got.MatchedAlbumID.Valid { - t.Errorf("false completion: status %v completed_at %v matched %v, want approved and cleared", - got.Status, got.CompletedAt, got.MatchedAlbumID) - } - if got := requestStatus(t, q, genuineReq.ID); got.Status != dbq.LidarrRequestStatusCompleted { - t.Errorf("genuine completion reopened: status %v", got.Status) - } -} diff --git a/internal/lidarrrequests/service.go b/internal/lidarrrequests/service.go deleted file mode 100644 index b177ae33..00000000 --- a/internal/lidarrrequests/service.go +++ /dev/null @@ -1,382 +0,0 @@ -// Package lidarrrequests owns the lifecycle of user requests to add -// music via Lidarr. The synchronous Service handles Create/List/ -// Approve/Reject/Cancel; the async Reconciler (separate file) closes -// approved requests once their target track lands in the library. -package lidarrrequests - -import ( - "context" - "errors" - "fmt" - - "github.com/jackc/pgx/v5" - "github.com/jackc/pgx/v5/pgtype" - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" -) - -// Public errors. Handlers map these to API error codes. -var ( - ErrInvalidKindFields = errors.New("lidarrrequests: missing required fields for kind") - ErrNotPending = errors.New("lidarrrequests: request is not pending") - // ErrNotFound aliases apierror.ErrNotFound so handlers can errors.Is - // against the shared sentinel; existing lidarrrequests.ErrNotFound - // callsites still resolve to the same pointer. - ErrNotFound = apierror.ErrNotFound - ErrLidarrDisabled = errors.New("lidarrrequests: lidarr not configured") - // ErrDefaultsIncomplete fires when Lidarr is enabled but the operator - // hasn't picked a default quality profile or root folder. Without those, - // Lidarr's add-artist/add-album endpoints reject the payload with a - // generic 5xx — surfacing a typed error here lets the admin UI tell the - // operator exactly what to fix. - ErrDefaultsIncomplete = errors.New("lidarrrequests: default quality profile or root folder not configured") -) - -// CreateParams is the input for a new request from a user. -type CreateParams struct { - Kind string // "artist", "album", or "track" - LidarrArtistMBID string - LidarrAlbumMBID string // required for kind=album/track - LidarrTrackMBID string // required for kind=track - ArtistName string - AlbumTitle string // required for kind=album/track - TrackTitle string // required for kind=track -} - -// ApproveOverrides lets the admin override the snapshot defaults for one -// approval. Zero values mean "use config default." -type ApproveOverrides struct { - QualityProfileID int - RootFolderPath string -} - -// Service owns the request lifecycle. clientFn is a factory called per -// Approve so that BaseURL/APIKey changes in lidarrconfig take effect -// immediately without restarting. scanFn is called after a successful -// Approve to trigger a library scan. -type Service struct { - pool *pgxpool.Pool - lidarrCfg *lidarrconfig.Service - clientFn func() *lidarr.Client // factory, called per Approve - scanFn func() -} - -// NewService constructs a Service. Pass nil for clientFn to disable Lidarr -// (Approve returns ErrLidarrDisabled). Pass nil for scanFn to no-op. -func NewService(pool *pgxpool.Pool, cfg *lidarrconfig.Service, clientFn func() *lidarr.Client, scanFn func()) *Service { - if scanFn == nil { - scanFn = func() {} - } - if clientFn == nil { - clientFn = func() *lidarr.Client { return nil } - } - return &Service{pool: pool, lidarrCfg: cfg, clientFn: clientFn, scanFn: scanFn} -} - -// Create validates the kind→required-fields invariant and inserts a -// pending row. -func (s *Service) Create(ctx context.Context, userID pgtype.UUID, p CreateParams) (dbq.LidarrRequest, error) { - row, _, err := s.CreateTracked(ctx, userID, p) - return row, err -} - -// CreateTracked is Create, also reporting whether a row was inserted. A -// request that dedupes into one already in flight is not new, so callers -// that announce new requests (M489) say nothing for it. -func (s *Service) CreateTracked(ctx context.Context, userID pgtype.UUID, p CreateParams) (dbq.LidarrRequest, bool, error) { - if err := validateKindFields(p); err != nil { - return dbq.LidarrRequest{}, false, err - } - q := dbq.New(s.pool) - // Dedup: if a non-terminal request for this MBID already exists, - // return it instead of inserting a duplicate. The Discover/search UI - // already hides requested candidates; this backstops races and direct - // API callers so we don't accrue duplicate request rows. Match on the - // new request's own-kind MBID (mirrors HasNonTerminalRequestForMBID). - dedupMBID := p.LidarrArtistMBID - switch p.Kind { - case "album": - dedupMBID = p.LidarrAlbumMBID - case "track": - dedupMBID = p.LidarrTrackMBID - } - if existing, derr := q.GetNonTerminalRequestForMBID(ctx, dedupMBID); derr == nil { - return existing, false, nil - } else if !errors.Is(derr, pgx.ErrNoRows) { - return dbq.LidarrRequest{}, false, fmt.Errorf("create: dedup check: %w", derr) - } - row, err := q.CreateLidarrRequest(ctx, dbq.CreateLidarrRequestParams{ - UserID: userID, - Kind: dbq.LidarrRequestKind(p.Kind), - LidarrArtistMbid: p.LidarrArtistMBID, - LidarrAlbumMbid: strPtr(p.LidarrAlbumMBID), - LidarrTrackMbid: strPtr(p.LidarrTrackMBID), - ArtistName: p.ArtistName, - AlbumTitle: strPtr(p.AlbumTitle), - TrackTitle: strPtr(p.TrackTitle), - }) - if err != nil { - return dbq.LidarrRequest{}, false, fmt.Errorf("create: %w", err) - } - return row, true, nil -} - -// DisplayName is a request as a person would say it: "Moe Shop" for an -// artist, "Moe Shop – WWW" for an album or a track. -func DisplayName(row dbq.LidarrRequest) string { - switch row.Kind { - case dbq.LidarrRequestKindTrack: - if row.TrackTitle != nil && *row.TrackTitle != "" { - return row.ArtistName + " – " + *row.TrackTitle - } - case dbq.LidarrRequestKindAlbum: - if row.AlbumTitle != nil && *row.AlbumTitle != "" { - return row.ArtistName + " – " + *row.AlbumTitle - } - } - return row.ArtistName -} - -// requestedTitle is the album or track a request names, or "" for an -// artist request. DisplayName is the artist and this, joined. -func requestedTitle(row dbq.LidarrRequest) string { - switch { - case row.Kind == dbq.LidarrRequestKindTrack && row.TrackTitle != nil: - return *row.TrackTitle - case row.Kind == dbq.LidarrRequestKindAlbum && row.AlbumTitle != nil: - return *row.AlbumTitle - } - return "" -} - -func validateKindFields(p CreateParams) error { - if p.LidarrArtistMBID == "" || p.ArtistName == "" { - return fmt.Errorf("%w: artist_mbid and artist_name are always required", ErrInvalidKindFields) - } - switch p.Kind { - case "artist": - // fine - case "album": - if p.LidarrAlbumMBID == "" || p.AlbumTitle == "" { - return fmt.Errorf("%w: album kind requires album_mbid and album_title", ErrInvalidKindFields) - } - case "track": - if p.LidarrAlbumMBID == "" || p.AlbumTitle == "" { - return fmt.Errorf("%w: track kind requires album_mbid and album_title (track will be promoted)", ErrInvalidKindFields) - } - if p.LidarrTrackMBID == "" || p.TrackTitle == "" { - return fmt.Errorf("%w: track kind requires track_mbid and track_title", ErrInvalidKindFields) - } - default: - return fmt.Errorf("%w: unknown kind %q", ErrInvalidKindFields, p.Kind) - } - return nil -} - -// ListPending returns pending requests ordered by requested_at DESC. -func (s *Service) ListPending(ctx context.Context, limit int32) ([]dbq.LidarrRequest, error) { - return dbq.New(s.pool).ListLidarrRequestsByStatus(ctx, dbq.ListLidarrRequestsByStatusParams{ - Status: dbq.LidarrRequestStatusPending, Limit: limit, - }) -} - -// ListByStatus returns requests with the given status, ordered by requested_at DESC. -func (s *Service) ListByStatus(ctx context.Context, status string, limit int32) ([]dbq.LidarrRequest, error) { - return dbq.New(s.pool).ListLidarrRequestsByStatus(ctx, dbq.ListLidarrRequestsByStatusParams{ - Status: dbq.LidarrRequestStatus(status), Limit: limit, - }) -} - -// ListForUser returns all requests by a user, ordered by requested_at DESC. -func (s *Service) ListForUser(ctx context.Context, userID pgtype.UUID, limit int32) ([]dbq.LidarrRequest, error) { - return dbq.New(s.pool).ListLidarrRequestsForUser(ctx, dbq.ListLidarrRequestsForUserParams{ - UserID: userID, Limit: limit, - }) -} - -// Approve records the admin's approval DURABLY first (snapshotting the -// chosen quality profile + root folder), then makes a best-effort Lidarr -// add and triggers a library scan. The add is no longer a gate: if Lidarr -// is unreachable the request still becomes approved and the reconciler -// idempotently (re)sends the add on each tick until it sticks (see -// sendLidarrAdd + Reconciler.ensureLidarrAdd) — "Lidarr just keeps -// trying" without the admin re-clicking. Pre-conditions (lidarr disabled, -// not pending, defaults incomplete) still fail fast before any approval. -func (s *Service) Approve(ctx context.Context, requestID pgtype.UUID, adminID pgtype.UUID, ov ApproveOverrides) (dbq.LidarrRequest, error) { - cfg, err := s.lidarrCfg.Get(ctx) - if err != nil { - return dbq.LidarrRequest{}, fmt.Errorf("approve: load config: %w", err) - } - client := s.clientFn() - if !cfg.Enabled || client == nil { - return dbq.LidarrRequest{}, ErrLidarrDisabled - } - row, err := dbq.New(s.pool).GetLidarrRequestByID(ctx, requestID) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - return dbq.LidarrRequest{}, ErrNotFound - } - return dbq.LidarrRequest{}, fmt.Errorf("approve: get: %w", err) - } - if row.Status != dbq.LidarrRequestStatusPending { - return dbq.LidarrRequest{}, ErrNotPending - } - - qp := cfg.DefaultQualityProfileID - if ov.QualityProfileID != 0 { - qp = ov.QualityProfileID - } - rf := cfg.DefaultRootFolderPath - if ov.RootFolderPath != "" { - rf = ov.RootFolderPath - } - if qp == 0 || rf == "" { - return dbq.LidarrRequest{}, ErrDefaultsIncomplete - } - - // Record the approval durably FIRST so a Lidarr outage can't lose the - // admin's decision. The snapshotted qp/rf also feed the reconciler's - // retry, so a deferred add uses exactly what the admin approved with. - approved, err := dbq.New(s.pool).ApproveLidarrRequest(ctx, dbq.ApproveLidarrRequestParams{ - ID: requestID, - QualityProfileID: int32Ptr(qp), - RootFolderPath: strPtr(rf), - DecidedBy: adminID, - }) - if err != nil { - return dbq.LidarrRequest{}, fmt.Errorf("approve: persist: %w", err) - } - - // Best-effort immediate add for the common (Lidarr-up) path. Any - // failure is deliberately swallowed: the request is already durably - // approved and the reconciler retries the add on its next tick, - // leaving lidarr_add_confirmed_at NULL until it succeeds. - if addErr := sendLidarrAdd(ctx, client, cfg, approved); addErr == nil { - _ = dbq.New(s.pool).MarkLidarrRequestAddConfirmed(ctx, requestID) - } - s.scanFn() - return approved, nil -} - -// sendLidarrAdd performs the idempotent Lidarr add for an approved -// request. Used by Service.Approve (best-effort, common path) and by the -// Reconciler (retry path). qp/rf come from the request's snapshot (set at -// Approve time) with a config-default fallback; the metadata profile is -// resolved from config, falling back to Lidarr's first profile for -// operators who upgraded from before metadata-profile support. Treats -// lidarr.ErrAlreadyExists as success — the artist/album is already in -// Lidarr, which is exactly the desired end state. -func sendLidarrAdd(ctx context.Context, client *lidarr.Client, cfg lidarrconfig.Config, row dbq.LidarrRequest) error { - qp := cfg.DefaultQualityProfileID - if row.QualityProfileID != nil && *row.QualityProfileID != 0 { - qp = int(*row.QualityProfileID) - } - rf := cfg.DefaultRootFolderPath - if row.RootFolderPath != nil && *row.RootFolderPath != "" { - rf = *row.RootFolderPath - } - if qp == 0 || rf == "" { - return ErrDefaultsIncomplete - } - mdProfileID := cfg.DefaultMetadataProfileID - if mdProfileID == 0 { - mdProfiles, mdErr := client.ListMetadataProfiles(ctx) - if mdErr != nil { - return fmt.Errorf("list metadata profiles: %w", mdErr) - } - if len(mdProfiles) == 0 { - return fmt.Errorf("no metadata profiles configured in lidarr") - } - mdProfileID = mdProfiles[0].ID - } - var err error - switch row.Kind { - case dbq.LidarrRequestKindArtist: - err = client.AddArtist(ctx, lidarr.AddArtistParams{ - ForeignArtistID: row.LidarrArtistMbid, - ArtistName: row.ArtistName, - QualityProfileID: qp, - MetadataProfileID: mdProfileID, - RootFolderPath: rf, - MonitorAll: true, - }) - case dbq.LidarrRequestKindAlbum, dbq.LidarrRequestKindTrack: - // Track-kind requests promote to album-add; the spec is explicit. - albumMBID := "" - if row.LidarrAlbumMbid != nil { - albumMBID = *row.LidarrAlbumMbid - } - err = client.AddAlbum(ctx, lidarr.AddAlbumParams{ - ForeignAlbumID: albumMBID, - ForeignArtistID: row.LidarrArtistMbid, - ArtistName: row.ArtistName, - QualityProfileID: qp, - MetadataProfileID: mdProfileID, - RootFolderPath: rf, - }) - } - if err != nil && !errors.Is(err, lidarr.ErrAlreadyExists) { - return err - } - return nil -} - -// Reject transitions a pending request to rejected, recording notes and -// who decided. Returns ErrNotPending if the request is not pending, -// ErrNotFound if no such request exists. -func (s *Service) Reject(ctx context.Context, requestID pgtype.UUID, adminID pgtype.UUID, notes string) (dbq.LidarrRequest, error) { - row, err := dbq.New(s.pool).RejectLidarrRequest(ctx, dbq.RejectLidarrRequestParams{ - ID: requestID, Notes: strPtr(notes), DecidedBy: adminID, - }) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - // Either not found OR not pending — check which. - cur, gerr := dbq.New(s.pool).GetLidarrRequestByID(ctx, requestID) - if gerr != nil { - return dbq.LidarrRequest{}, ErrNotFound - } - if cur.Status != dbq.LidarrRequestStatusPending { - return dbq.LidarrRequest{}, ErrNotPending - } - return dbq.LidarrRequest{}, ErrNotFound - } - return dbq.LidarrRequest{}, fmt.Errorf("reject: %w", err) - } - return row, nil -} - -// Cancel lets a user withdraw their own pending request. The SQL WHERE -// clause enforces both ownership (user_id = $2) and pending status -// (status = 'pending'). A zero-rows result always means ErrNotPending -// from the caller's perspective. -func (s *Service) Cancel(ctx context.Context, requestID pgtype.UUID, userID pgtype.UUID) (dbq.LidarrRequest, error) { - row, err := dbq.New(s.pool).CancelLidarrRequest(ctx, dbq.CancelLidarrRequestParams{ - ID: requestID, DecidedBy: userID, - }) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - return dbq.LidarrRequest{}, ErrNotPending - } - return dbq.LidarrRequest{}, fmt.Errorf("cancel: %w", err) - } - return row, nil -} - -func strPtr(s string) *string { - if s == "" { - return nil - } - return &s -} - -func int32Ptr(i int) *int32 { - if i == 0 { - return nil - } - v := int32(i) - return &v -} diff --git a/internal/lidarrrequests/service_test.go b/internal/lidarrrequests/service_test.go deleted file mode 100644 index c1c90149..00000000 --- a/internal/lidarrrequests/service_test.go +++ /dev/null @@ -1,343 +0,0 @@ -package lidarrrequests - -import ( - "context" - "errors" - "io" - "log/slog" - "net/http" - "net/http/httptest" - "os" - "strings" - "testing" - - "github.com/jackc/pgx/v5/pgtype" - "github.com/jackc/pgx/v5/pgxpool" - - "git.fabledsword.com/bvandeusen/minstrel/internal/db" - "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" -) - -func newPool(t *testing.T) *pgxpool.Pool { - t.Helper() - if testing.Short() { - t.Skip("skipping integration test in -short mode") - } - dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") - if dsn == "" { - t.Skip("MINSTREL_TEST_DATABASE_URL not set") - } - if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { - t.Fatalf("migrate: %v", err) - } - pool, err := pgxpool.New(context.Background(), dsn) - if err != nil { - t.Fatalf("pool: %v", err) - } - t.Cleanup(pool.Close) - dbtest.ResetDB(t, pool) - if _, err := pool.Exec(context.Background(), - "DELETE FROM lidarr_requests; UPDATE lidarr_config SET enabled=false, base_url=NULL, api_key=NULL WHERE id=1", - ); err != nil { - t.Fatalf("reset lidarr tables: %v", err) - } - return pool -} - -func seedUser(t *testing.T, pool *pgxpool.Pool) pgtype.UUID { - t.Helper() - u, err := dbq.New(pool).CreateUser(context.Background(), dbq.CreateUserParams{ - Username: dbtest.TestUserPrefix + "rqtester", PasswordHash: "x", ApiTokenHash: "x", IsAdmin: false, - }) - if err != nil { - t.Fatalf("seed user: %v", err) - } - return u.ID -} - -func TestCreate_HappyPath_Artist(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - r, err := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", - LidarrArtistMBID: "069b64b6-7884-4f6a-94cc-e4c1d6c87a01", - ArtistName: "Boards of Canada", - }) - if err != nil { - t.Fatalf("Create: %v", err) - } - if r.Status != dbq.LidarrRequestStatusPending { - t.Errorf("status = %v", r.Status) - } -} - -func TestCreate_TrackKindRequiresAlbumMBID(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - _, err := svc.Create(context.Background(), user, CreateParams{ - Kind: "track", - LidarrArtistMBID: "a-mbid", ArtistName: "X", - LidarrTrackMBID: "t-mbid", TrackTitle: "Y", - // missing album fields - }) - if !errors.Is(err, ErrInvalidKindFields) { - t.Fatalf("err = %v, want ErrInvalidKindFields", err) - } -} - -func TestApprove_NotConfigured(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - r, _ := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", LidarrArtistMBID: "a-mbid", ArtistName: "X", - }) - _, err := svc.Approve(context.Background(), r.ID, user, ApproveOverrides{}) - if !errors.Is(err, ErrLidarrDisabled) { - t.Fatalf("err = %v, want ErrLidarrDisabled", err) - } -} - -func TestReject_TransitionsToRejected(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - r, _ := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", LidarrArtistMBID: "a-mbid", ArtistName: "X", - }) - rejected, err := svc.Reject(context.Background(), r.ID, user, "low quality") - if err != nil { - t.Fatalf("Reject: %v", err) - } - if rejected.Status != dbq.LidarrRequestStatusRejected { - t.Errorf("status = %v", rejected.Status) - } -} - -func TestReject_AlreadyRejectedReturnsErrNotPending(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - r, _ := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", LidarrArtistMBID: "a-mbid", ArtistName: "X", - }) - _, _ = svc.Reject(context.Background(), r.ID, user, "first") - _, err := svc.Reject(context.Background(), r.ID, user, "second") - if !errors.Is(err, ErrNotPending) { - t.Fatalf("err = %v, want ErrNotPending", err) - } -} - -func TestCancel_OwnPendingOnly(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - r, _ := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", LidarrArtistMBID: "a-mbid", ArtistName: "X", - }) - if _, err := svc.Cancel(context.Background(), r.ID, user); err != nil { - t.Fatalf("Cancel: %v", err) - } - // Second cancel hits "not pending" because we just rejected it. - if _, err := svc.Cancel(context.Background(), r.ID, user); !errors.Is(err, ErrNotPending) { - t.Errorf("err = %v, want ErrNotPending", err) - } -} - -func TestCreate_AlbumKindRequiresAlbumMBID(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - _, err := svc.Create(context.Background(), user, CreateParams{ - Kind: "album", - LidarrArtistMBID: "a-mbid", ArtistName: "X", - // missing album_mbid + album_title - }) - if !errors.Is(err, ErrInvalidKindFields) { - t.Fatalf("err = %v, want ErrInvalidKindFields", err) - } -} - -func TestCreate_UnknownKindRejected(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - _, err := svc.Create(context.Background(), user, CreateParams{ - Kind: "playlist", LidarrArtistMBID: "a-mbid", ArtistName: "X", - }) - if !errors.Is(err, ErrInvalidKindFields) { - t.Fatalf("err = %v, want ErrInvalidKindFields", err) - } -} - -func TestListPending_AndListByStatus(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool) - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - for _, name := range []string{"A", "B"} { - if _, err := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", LidarrArtistMBID: name + "-mbid", ArtistName: name, - }); err != nil { - t.Fatalf("Create(%s): %v", name, err) - } - } - pending, err := svc.ListPending(context.Background(), 50) - if err != nil { - t.Fatalf("ListPending: %v", err) - } - if len(pending) != 2 { - t.Errorf("ListPending len = %d, want 2", len(pending)) - } - rejected, err := svc.ListByStatus(context.Background(), "rejected", 50) - if err != nil { - t.Fatalf("ListByStatus(rejected): %v", err) - } - if len(rejected) != 0 { - t.Errorf("ListByStatus(rejected) len = %d, want 0", len(rejected)) - } -} - -func TestListForUser_OnlyOwnRows(t *testing.T) { - pool := newPool(t) - alice := seedUser(t, pool) - bob, err := dbq.New(pool).CreateUser(context.Background(), dbq.CreateUserParams{ - Username: dbtest.TestUserPrefix + "bob", PasswordHash: "x", ApiTokenHash: "x2", IsAdmin: false, - }) - if err != nil { - t.Fatalf("seed bob: %v", err) - } - svc := NewService(pool, lidarrconfig.New(pool), nil, nil) - if _, err := svc.Create(context.Background(), alice, CreateParams{ - Kind: "artist", LidarrArtistMBID: "a-mbid", ArtistName: "Alice's request", - }); err != nil { - t.Fatalf("Create(alice): %v", err) - } - if _, err := svc.Create(context.Background(), bob.ID, CreateParams{ - Kind: "artist", LidarrArtistMBID: "b-mbid", ArtistName: "Bob's request", - }); err != nil { - t.Fatalf("Create(bob): %v", err) - } - rows, err := svc.ListForUser(context.Background(), alice, 50) - if err != nil { - t.Fatalf("ListForUser: %v", err) - } - if len(rows) != 1 || rows[0].ArtistName != "Alice's request" { - t.Errorf("ListForUser = %+v, want only Alice's row", rows) - } -} - -// approveTestSetup wires Service against a stub Lidarr server. Returns the -// Service, an admin user id, and the captured Lidarr request body for the -// most recent call (handler updates it in-place). -func approveTestSetup(t *testing.T) (*Service, pgtype.UUID, *pgxpool.Pool, *httptest.Server) { - t.Helper() - pool := newPool(t) - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - // Approve resolves metadata/quality profiles before the add; - // those list endpoints return JSON arrays, not the {"id":1} - // object the add endpoints return. - switch { - case strings.Contains(r.URL.Path, "/metadataprofile"): - _, _ = w.Write([]byte(`[{"id":1,"name":"Standard"}]`)) - case strings.Contains(r.URL.Path, "/qualityprofile"): - _, _ = w.Write([]byte(`[{"id":7,"name":"Lossless"}]`)) - default: - _, _ = w.Write([]byte(`{"id":1}`)) - } - })) - t.Cleanup(stub.Close) - - cfg := lidarrconfig.New(pool) - if err := cfg.Save(context.Background(), lidarrconfig.Config{ - Enabled: true, - BaseURL: stub.URL, - APIKey: "k", - DefaultQualityProfileID: 7, - DefaultRootFolderPath: "/music", - }); err != nil { - t.Fatalf("save config: %v", err) - } - clientFn := func() *lidarr.Client { return lidarr.NewClient(stub.URL, "k") } - svc := NewService(pool, cfg, clientFn, nil) - user := seedUser(t, pool) - return svc, user, pool, stub -} - -func TestApprove_HappyPath_ArtistKind(t *testing.T) { - svc, user, _, _ := approveTestSetup(t) - r, err := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", LidarrArtistMBID: "ar-mbid", ArtistName: "Approvee", - }) - if err != nil { - t.Fatalf("Create: %v", err) - } - approved, err := svc.Approve(context.Background(), r.ID, user, ApproveOverrides{}) - if err != nil { - t.Fatalf("Approve: %v", err) - } - if approved.Status != dbq.LidarrRequestStatusApproved { - t.Errorf("status = %v, want approved", approved.Status) - } - // Default snapshot from config. - if approved.QualityProfileID == nil || *approved.QualityProfileID != 7 { - t.Errorf("quality_profile_id = %v, want 7", approved.QualityProfileID) - } - if approved.RootFolderPath == nil || *approved.RootFolderPath != "/music" { - t.Errorf("root_folder_path = %v, want /music", approved.RootFolderPath) - } -} - -func TestApprove_HappyPath_AlbumKindWithOverride(t *testing.T) { - svc, user, _, _ := approveTestSetup(t) - r, err := svc.Create(context.Background(), user, CreateParams{ - Kind: "album", - LidarrArtistMBID: "ar-mbid", ArtistName: "X", - LidarrAlbumMBID: "al-mbid", AlbumTitle: "Y", - }) - if err != nil { - t.Fatalf("Create: %v", err) - } - approved, err := svc.Approve(context.Background(), r.ID, user, ApproveOverrides{ - QualityProfileID: 99, - RootFolderPath: "/elsewhere", - }) - if err != nil { - t.Fatalf("Approve: %v", err) - } - if approved.QualityProfileID == nil || *approved.QualityProfileID != 99 { - t.Errorf("quality_profile_id = %v, want 99 (override)", approved.QualityProfileID) - } - if approved.RootFolderPath == nil || *approved.RootFolderPath != "/elsewhere" { - t.Errorf("root_folder_path = %v, want /elsewhere", approved.RootFolderPath) - } -} - -func TestApprove_AlreadyApprovedReturnsErrNotPending(t *testing.T) { - svc, user, _, _ := approveTestSetup(t) - r, _ := svc.Create(context.Background(), user, CreateParams{ - Kind: "artist", LidarrArtistMBID: "ar-mbid", ArtistName: "X", - }) - if _, err := svc.Approve(context.Background(), r.ID, user, ApproveOverrides{}); err != nil { - t.Fatalf("first approve: %v", err) - } - if _, err := svc.Approve(context.Background(), r.ID, user, ApproveOverrides{}); !errors.Is(err, ErrNotPending) { - t.Errorf("second approve err = %v, want ErrNotPending", err) - } -} - -func TestApprove_NotFound(t *testing.T) { - svc, user, _, _ := approveTestSetup(t) - var bogus pgtype.UUID - bogus.Bytes = [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16} - bogus.Valid = true - if _, err := svc.Approve(context.Background(), bogus, user, ApproveOverrides{}); !errors.Is(err, ErrNotFound) { - t.Errorf("err = %v, want ErrNotFound", err) - } -} diff --git a/internal/lifecycle/lifecycle.go b/internal/lifecycle/lifecycle.go new file mode 100644 index 00000000..23102f4d --- /dev/null +++ b/internal/lifecycle/lifecycle.go @@ -0,0 +1,682 @@ +// Package lifecycle looks after a download from the grab to its removal. +// +// - A download the client has failed is given up at once. +// - One nobody has shared, or still fetching its file list, after the +// stall setting is retried: the trackers asked again and the files +// re-read, three times a pass apart. Then it is given up. +// - Giving up removes the download with its files, holds the release back +// for a week, ends the grab and searches for the album again. There is +// no limit per album: what stops it churning is the hold on each release. +// - A running download the client no longer lists, for half an hour, ends +// and its album is searched again. The release is not held back: nothing +// was wrong with it. +// - An imported download seeds (its files were copied) until its share +// ratio reaches the setting or its days are up, then is removed. +// +// Every condition and every act goes in the intervention ledger, one open +// row per (client, download, condition), so the Activity tab can say what +// is wrong with a download and what was done. +package lifecycle + +import ( + "context" + "errors" + "fmt" + "log/slog" + "strings" + "sync" + "time" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// Conditions, matching download_interventions_trigger_check. +const ( + NoPeers = "no_peers" + NoMetadata = "no_metadata" + ClientError = "client_error" + ClientUnreachable = "client_unreachable" + Gone = "gone" + NotRemoved = "not_removed" +) + +// Acts, matching download_interventions_action_check. +const ( + actRetried = "retried" + actGivenUp = "given_up" + actRemoved = "removed" +) + +// End reasons a grab is given with, matching grabs_end_reason_check. +const ( + EndStalled = "stalled" + EndFailed = "failed" + EndAbandoned = "abandoned" + EndGone = "gone" +) + +const ( + // interval: three retries a pass apart is half an hour of asking before + // a stalled download is given up. + interval = 10 * time.Minute + passTimeout = 5 * time.Minute + // Retries is how many times one stuck download is nudged first. + Retries = 3 + // StallBlock is how long a release given up on is held back. + StallBlock = 7 * 24 * time.Hour + // goneAfter: a client restarting can answer without a download for a + // while; half an hour missing is gone. + goneAfter = 30 * time.Minute + // goneWindow: a release gone twice within it is held back. + goneWindow = 7 * 24 * time.Hour +) + +// running are the import states of a grab still downloading: the lifecycle +// acts only on these, so an import under way is never pulled from under. +var running = []string{"", "downloading", "waiting"} + +// Settings are the limits, from acquisition_settings. +type Settings struct { + // SeedRatio is the share ratio an imported download seeds to; 0 removes + // it once imported. + SeedRatio float64 + // SeedTime is the longest it seeds; 0 is no limit. + SeedTime time.Duration + // Patience is how long a download may sit with nobody sharing it before + // it is retried, then given up; 0 never gives up for that. + Patience time.Duration +} + +// Defaults are FL's rulings. +var Defaults = Settings{SeedRatio: 2, SeedTime: 14 * 24 * time.Hour, Patience: 6 * time.Hour} + +// Service runs the lifecycle passes. +type Service struct { + pool *pgxpool.Pool + // ClientFor builds a download client by its code. Defaults to + // acquire.DownloadClientFor. + ClientFor func(ctx context.Context, code string) (downloader.Driver, error) + // Now is the clock; nil is time.Now. + Now func() time.Time + logger *slog.Logger + mu sync.Mutex // one pass at a time +} + +// New returns a Service. +func New(pool *pgxpool.Pool, logger *slog.Logger) *Service { + if logger == nil { + logger = slog.Default() + } + return &Service{pool: pool, logger: logger, + ClientFor: func(ctx context.Context, code string) (downloader.Driver, error) { + return acquire.DownloadClientFor(ctx, pool, code) + }} +} + +func (s *Service) now() time.Time { + if s.Now != nil { + return s.Now() + } + return time.Now() +} + +// Run passes a minute after start, then every ten minutes, until ctx ends. +// Each pass is bounded, so a client that never answers costs one pass. +func (s *Service) Run(ctx context.Context) { + first := time.NewTimer(time.Minute) + defer first.Stop() + select { + case <-ctx.Done(): + return + case <-first.C: + } + t := time.NewTicker(interval) + defer t.Stop() + for { + rep := s.Pass(ctx) + if rep.Retried+rep.GivenUp+rep.Gone+rep.Released+rep.Removed > 0 || rep.Error != "" { + s.logger.Info("lifecycle: pass", "retried", rep.Retried, "given_up", rep.GivenUp, "gone", rep.Gone, + "released", rep.Released, "removed", rep.Removed, "raised", rep.Raised, "settled", rep.Settled, + "error", rep.Error) + } + select { + case <-ctx.Done(): + return + case <-t.C: + } + } +} + +// Report is one pass. +type Report struct { + // Raised and Settled count ledger writes; the rest count acts. + Raised, Settled int + Retried, GivenUp, Gone, Released, Removed int + Error string +} + +// Pass looks at every running download, every seeding one and every one +// left behind, once. +func (s *Service) Pass(ctx context.Context) Report { + s.mu.Lock() + defer s.mu.Unlock() + ctx, cancel := context.WithTimeout(ctx, passTimeout) + defer cancel() + var rep Report + q := dbq.New(s.pool) + set := s.settings(ctx, q) + + if _, err := q.DeleteExpiredBlocks(ctx); err != nil { + s.logger.Warn("lifecycle: lift expired holds", "err", err) + } + if n, err := q.SettleEndedGrabs(ctx); err == nil { + rep.Settled += int(n) + } + live, err := q.ListLiveGrabs(ctx) + if err != nil { + rep.Error = err.Error() + return rep + } + seeds, err := q.ListSeedingGrabs(ctx) + if err != nil { + rep.Error = err.Error() + return rep + } + left, err := q.ListLeftovers(ctx) + if err != nil { + rep.Error = err.Error() + return rep + } + + work := map[string]*clientWork{} + var order []string + of := func(code string) *clientWork { + if w, ok := work[code]; ok { + return w + } + order = append(order, code) + work[code] = &clientWork{} + return work[code] + } + for _, g := range live { + if g.InfoHash != "" { + of(g.Client).live = append(of(g.Client).live, g) + } + } + for _, g := range seeds { + if g.InfoHash != "" { + of(g.Client).seeds = append(of(g.Client).seeds, g) + } + } + for _, l := range left { + of(l.Client).left = append(of(l.Client).left, l) + } + for _, code := range order { + if ctx.Err() != nil { + break + } + s.passClient(ctx, q, code, work[code], set, &rep) + } + return rep +} + +type clientWork struct { + live []dbq.ListLiveGrabsRow + seeds []dbq.ListSeedingGrabsRow + left []dbq.DownloadIntervention +} + +func (s *Service) settings(ctx context.Context, q *dbq.Queries) Settings { + row, err := q.GetAcquisitionSettings(ctx) + if err != nil { + // A blip keeps the defaults rather than stopping the pass. + return Defaults + } + return Settings{ + SeedRatio: float64(row.SeedRatio), + SeedTime: time.Duration(row.SeedDays) * 24 * time.Hour, + Patience: time.Duration(row.StallHours) * time.Hour, + } +} + +func (s *Service) passClient(ctx context.Context, q *dbq.Queries, code string, w *clientWork, set Settings, rep *Report) { + d, err := s.ClientFor(ctx, code) + if err != nil { + s.raise(ctx, q, code, "", pgtype.UUID{}, "", ClientUnreachable, firstLine(err.Error()), rep) + return + } + var hashes []string + for _, g := range w.live { + hashes = append(hashes, g.InfoHash) + } + for _, g := range w.seeds { + hashes = append(hashes, g.InfoHash) + } + for _, l := range w.left { + hashes = append(hashes, l.InfoHash) + } + ts, err := d.Torrents(ctx, hashes...) + if err != nil { + // Nothing about any one download is observed through a client that + // cannot answer. + s.raise(ctx, q, code, "", pgtype.UUID{}, "", ClientUnreachable, firstLine(err.Error()), rep) + return + } + for _, t := range ts { + if t.EngineDown != "" { + s.raise(ctx, q, code, "", pgtype.UUID{}, "", ClientUnreachable, t.EngineDown, rep) + return + } + } + s.settle(ctx, q, code, "", ClientUnreachable, rep) + + byHash := make(map[string]downloader.Torrent, len(ts)) + for _, t := range ts { + byHash[indexer.NormalizeHash(t.Hash)] = t + } + // ABSENCE IS ONLY A MEASUREMENT FROM A CLIENT THAT LISTS SOMETHING. An + // engine switched off, or a store that would not read, answers with an + // empty list; then nothing is concluded from what is missing. + listed := len(ts) > 0 + if !listed { + all, err := d.Torrents(ctx) + listed = err == nil && len(all) > 0 + } + + for _, l := range w.left { + t, ok := byHash[indexer.NormalizeHash(l.InfoHash)] + s.removeAgain(ctx, q, d, l, t, ok, listed, rep) + } + for _, g := range w.live { + t, ok := byHash[indexer.NormalizeHash(g.InfoHash)] + s.judge(ctx, q, d, g, t, ok, listed, set, rep) + } + for _, g := range w.seeds { + t, ok := byHash[indexer.NormalizeHash(g.InfoHash)] + s.seed(ctx, q, d, g, t, ok, set, rep) + } +} + +// judge decides what, if anything, is wrong with one running download, and +// answers each condition either way: raised when true, settled when not. +func (s *Service) judge(ctx context.Context, q *dbq.Queries, d downloader.Driver, g dbq.ListLiveGrabsRow, + t downloader.Torrent, ok, listed bool, set Settings, rep *Report) { + if !isRunning(g.ImportState) { + return // the importer's, or a person's + } + if !ok { + if !listed { + return + } + row, raised := s.raise(ctx, q, g.Client, g.InfoHash, g.ID, g.Name, Gone, + "The download client no longer lists it.", rep) + if raised && s.now().Sub(row.FirstSeen.Time) >= goneAfter { + s.endGone(ctx, q, g, rep) + } + return + } + s.settle(ctx, q, g.Client, g.InfoHash, Gone, rep) + + if t.Done() { + // Finished: whatever was wrong while it downloaded is over, and the + // import is the importer's. + for _, c := range []string{ClientError, NoMetadata, NoPeers} { + s.settle(ctx, q, g.Client, g.InfoHash, c, rep) + } + return + } + + // The client's own verdict, given up on at once: it has already made + // the decision a retry would ask about. + if t.Failed() { + why := t.FailReason + if why == "" { + why = fmt.Sprintf("the download client reports it %q", t.State) + } + if _, raised := s.raise(ctx, q, g.Client, g.InfoHash, g.ID, g.Name, ClientError, why, rep); raised { + s.giveUp(ctx, q, d, g, ClientError, EndFailed, why, rep) + } + return + } + s.settle(ctx, q, g.Client, g.InfoHash, ClientError, rep) + + // Below here the stall setting applies, measured from the grab. With it + // off nothing is raised and what was raised clears. + age := s.now().Sub(g.GrabbedAt.Time) + patient := set.Patience > 0 && age > set.Patience + + if patient && t.Fetching() { + row, raised := s.raise(ctx, q, g.Client, g.InfoHash, g.ID, g.Name, NoMetadata, + fmt.Sprintf("Still fetching its file list after %s.", span(age)), rep) + if raised { + s.ladder(ctx, q, d, g, NoMetadata, row.Remediations, age, rep) + } + } else { + s.settle(ctx, q, g.Client, g.InfoHash, NoMetadata, rep) + } + // Not while fetching: a torrent with no file list has no swarm to be + // alone in. + if patient && !t.Fetching() && t.Alone() { + row, raised := s.raise(ctx, q, g.Client, g.InfoHash, g.ID, g.Name, NoPeers, + fmt.Sprintf("Nobody has shared it in %s; %.0f%% downloaded.", span(age), t.Progress*100), rep) + if raised { + s.ladder(ctx, q, d, g, NoPeers, row.Remediations, age, rep) + } + } else { + s.settle(ctx, q, g.Client, g.InfoHash, NoPeers, rep) + } +} + +// ladder retries a stuck download until the retries are spent, then gives +// it up. The rung is the ledger row's, so a restart hands out no more. +func (s *Service) ladder(ctx context.Context, q *dbq.Queries, d downloader.Driver, g dbq.ListLiveGrabsRow, + trigger string, done int32, age time.Duration, rep *Report) { + if int(done) < Retries { + // Counted before it happens: a retry that happened and was not + // counted is a bound that never binds. + n, err := q.CountRemediation(ctx, dbq.CountRemediationParams{ + Client: g.Client, InfoHash: g.InfoHash, Trigger: trigger, Action: actRetried}) + if err != nil { + return + } + // Exchange schedules its own announces and has no recheck: for it a + // retry is the time between passes, and the ladder still climbs. + if err := d.Reannounce(ctx, g.InfoHash); err != nil && !errors.Is(err, downloader.ErrUnsupported) { + s.logger.Warn("lifecycle: reannounce", "grab", g.Name, "err", err) + } + if err := d.Recheck(ctx, g.InfoHash); err != nil && !errors.Is(err, downloader.ErrUnsupported) { + s.logger.Warn("lifecycle: recheck", "grab", g.Name, "err", err) + } + s.logger.Info("lifecycle: retried a stalled download", "grab", g.Name, "trigger", trigger, "attempt", n) + rep.Retried++ + return + } + why := fmt.Sprintf("taken %s ago and never finished: %s", span(age), reasonFor(trigger)) + s.giveUp(ctx, q, d, g, trigger, EndStalled, why, rep) +} + +// giveUp ends the grab, holds the release back for StallBlock, removes the +// download with its files and searches for the album again. +func (s *Service) giveUp(ctx context.Context, q *dbq.Queries, d downloader.Driver, g dbq.ListLiveGrabsRow, + trigger, end, why string, rep *Report) { + until := s.now().Add(StallBlock) + detail := capital(why) + fmt.Sprintf(". Held back until %s; the album is searched for again.", until.Format("2 Jan")) + // The update is the lock: an import that claimed the download meanwhile + // keeps it. + if _, err := q.GiveUpGrab(ctx, dbq.GiveUpGrabParams{ID: g.ID, EndReason: end, EndDetail: detail, States: running}); err != nil { + return + } + // Held back before it is removed: if the pass dies between the two, the + // release is still not taken again. + if err := acquire.NewMemory(s.pool).Block(ctx, dbq.BlockReleaseParams{ + InfoHash: g.InfoHash, Name: g.Name, Reason: acquire.BlockStalled, Detail: capital(why) + ".", + WantID: g.WantID, ExpiresAt: pgtype.Timestamptz{Time: until, Valid: true}, + }); err != nil { + s.logger.Warn("lifecycle: hold the release back", "grab", g.Name, "err", err) + } + removed := s.remove(ctx, q, d, g.Client, g.InfoHash, g.ID, g.Name, rep) + if g.WantID.Valid { + if err := q.SearchWantAgain(ctx, g.WantID); err != nil { + s.logger.Warn("lifecycle: search again", "grab", g.Name, "err", err) + } + } + if _, err := q.CountRemediation(ctx, dbq.CountRemediationParams{ + Client: g.Client, InfoHash: g.InfoHash, Trigger: trigger, Action: actGivenUp}); err != nil { + s.logger.Warn("lifecycle: stamp the ledger", "grab", g.Name, "err", err) + } + if removed { + if n, err := q.SettleInterventionsFor(ctx, dbq.SettleInterventionsForParams{ + Client: g.Client, InfoHash: g.InfoHash}); err == nil { + rep.Settled += int(n) + } + } + s.logger.Info("lifecycle: gave up on a download", "grab", g.Name, "trigger", trigger, "why", why) + rep.GivenUp++ +} + +// endGone ends a grab whose download left the client, and searches again. +// The first time, the release is not held back: nothing was found wrong with +// it. A release that leaves its client again within goneWindow is held back for +// acquire.GoneBlock: whatever keeps the client from listing it would otherwise +// have it taken again on every pass. +func (s *Service) endGone(ctx context.Context, q *dbq.Queries, g dbq.ListLiveGrabsRow, rep *Report) { + detail := "The download client no longer lists it: removed outside Minstrel. The album is searched for again." + before, err := q.CountGoneGrabs(ctx, dbq.CountGoneGrabsParams{ + InfoHash: g.InfoHash, Since: pgtype.Timestamptz{Time: s.now().Add(-goneWindow), Valid: true}}) + if err != nil { + s.logger.Warn("lifecycle: count earlier gone grabs", "grab", g.Name, "err", err) + } + again := before > 0 + until := s.now().Add(acquire.GoneBlock) + if again { + detail = fmt.Sprintf("The download client no longer lists it, again: held back until %s; the album is searched for again.", + until.Format("2 Jan 15:04")) + } + if _, err := q.GiveUpGrab(ctx, dbq.GiveUpGrabParams{ID: g.ID, EndReason: EndGone, EndDetail: detail, States: running}); err != nil { + return + } + if again { + if err := acquire.NewMemory(s.pool).Block(ctx, dbq.BlockReleaseParams{ + InfoHash: g.InfoHash, Name: g.Name, Reason: acquire.BlockGone, + Detail: "It left its download client twice without finishing.", + WantID: g.WantID, ExpiresAt: pgtype.Timestamptz{Time: until, Valid: true}, + }); err != nil { + s.logger.Warn("lifecycle: hold the release back", "grab", g.Name, "err", err) + } + } + if g.WantID.Valid { + if err := q.SearchWantAgain(ctx, g.WantID); err != nil { + s.logger.Warn("lifecycle: search again", "grab", g.Name, "err", err) + } + } + if _, err := q.CountRemediation(ctx, dbq.CountRemediationParams{ + Client: g.Client, InfoHash: g.InfoHash, Trigger: Gone, Action: actGivenUp}); err != nil { + s.logger.Warn("lifecycle: stamp the ledger", "grab", g.Name, "err", err) + } + if n, err := q.SettleInterventionsFor(ctx, dbq.SettleInterventionsForParams{ + Client: g.Client, InfoHash: g.InfoHash}); err == nil { + rep.Settled += int(n) + } + s.logger.Info("lifecycle: a download left its client", "grab", g.Name) + rep.Gone++ +} + +// seed removes an imported download once it has shared enough. +// +// Absence is not a verdict: a restarting client answers with a partial list. +// A seeding download missing from it is left until its time is up, then +// recorded as released; there is nothing to delete. +func (s *Service) seed(ctx context.Context, q *dbq.Queries, d downloader.Driver, g dbq.ListSeedingGrabsRow, + t downloader.Torrent, ok bool, set Settings, rep *Report) { + overdue := set.SeedTime > 0 && g.ImportedAt.Valid && s.now().Sub(g.ImportedAt.Time) >= set.SeedTime + if !ok { + if overdue { + s.release(ctx, q, g.ID, rep) + } + return + } + if err := q.SetGrabRatio(ctx, dbq.SetGrabRatioParams{ID: g.ID, Ratio: float32(t.Ratio)}); err != nil { + s.logger.Warn("lifecycle: record the ratio", "grab", g.Name, "err", err) + } + if t.Ratio < set.SeedRatio && !overdue { + return + } + s.remove(ctx, q, d, g.Client, g.InfoHash, g.ID, g.Name, rep) + s.logger.Info("lifecycle: seeded enough, removed the download", "grab", g.Name, "ratio", t.Ratio, "overdue", overdue) + s.release(ctx, q, g.ID, rep) +} + +func (s *Service) release(ctx context.Context, q *dbq.Queries, id pgtype.UUID, rep *Report) { + if err := q.ReleaseSeed(ctx, id); err != nil { + s.logger.Warn("lifecycle: record the seeding ended", "err", err) + return + } + rep.Released++ +} + +// remove deletes a download and its files. A client that refuses leaves an +// open not_removed row, which every pass tries again. +func (s *Service) remove(ctx context.Context, q *dbq.Queries, d downloader.Driver, code, hash string, + grab pgtype.UUID, name string, rep *Report) bool { + if err := d.Delete(ctx, true, hash); err != nil { + s.logger.Warn("lifecycle: remove the download", "grab", name, "err", err) + s.raise(ctx, q, code, hash, grab, name, NotRemoved, notRemoved(err), rep) + return false + } + return true +} + +// removeAgain asks the client once more to remove a download left behind, +// unless it has been taken again since. +func (s *Service) removeAgain(ctx context.Context, q *dbq.Queries, d downloader.Driver, l dbq.DownloadIntervention, + t downloader.Torrent, ok, listed bool, rep *Report) { + if !ok { + if listed { + s.settle(ctx, q, l.Client, l.InfoHash, NotRemoved, rep) + } + return + } + live, err := q.LiveGrabOnHash(ctx, l.InfoHash) + if err != nil { + return + } + if live { + s.settle(ctx, q, l.Client, l.InfoHash, NotRemoved, rep) + return + } + if err := d.Delete(ctx, true, l.InfoHash); err != nil { + s.raise(ctx, q, l.Client, l.InfoHash, l.GrabID, t.Name, NotRemoved, notRemoved(err), rep) + return + } + if _, err := q.CountRemediation(ctx, dbq.CountRemediationParams{ + Client: l.Client, InfoHash: l.InfoHash, Trigger: NotRemoved, Action: actRemoved}); err != nil { + s.logger.Warn("lifecycle: stamp the ledger", "hash", l.InfoHash, "err", err) + } + s.logger.Info("lifecycle: removed a download left behind", "name", l.Name) + rep.Removed++ +} + +// raise records a condition or agrees with the open row that says it. The +// bool is false when the write failed: then nothing is acted on, since +// without the row's count there is no bound. +func (s *Service) raise(ctx context.Context, q *dbq.Queries, code, hash string, grab pgtype.UUID, name, trigger, + detail string, rep *Report) (dbq.DownloadIntervention, bool) { + row, err := q.RaiseIntervention(ctx, dbq.RaiseInterventionParams{ + Client: code, InfoHash: hash, GrabID: grab, Name: name, Trigger: trigger, Detail: detail}) + if err != nil { + s.logger.Warn("lifecycle: record", "client", code, "hash", hash, "trigger", trigger, "err", err) + return row, false + } + rep.Raised++ + return row, true +} + +func (s *Service) settle(ctx context.Context, q *dbq.Queries, code, hash, trigger string, rep *Report) { + n, err := q.SettleIntervention(ctx, dbq.SettleInterventionParams{Client: code, InfoHash: hash, Trigger: trigger}) + if err != nil { + s.logger.Warn("lifecycle: settle", "client", code, "hash", hash, "trigger", trigger, "err", err) + return + } + rep.Settled += int(n) +} + +// ErrNotAbandonable: the download is not running; an import, or a person's +// decision on one, has it. +var ErrNotAbandonable = errors.New("lifecycle: that download is not running") + +// Abandon is a person giving up on a running download: the release is set +// aside for good (their word about it), the download removed with its files, +// and the album searched for again. +func (s *Service) Abandon(ctx context.Context, grabID, by pgtype.UUID) (dbq.Grab, error) { + q := dbq.New(s.pool) + g, err := q.GetGrab(ctx, grabID) + if err != nil { + return dbq.Grab{}, err + } + if g.EndedAt.Valid || !isRunning(g.ImportState) { + return dbq.Grab{}, ErrNotAbandonable + } + const why = "Given up by you before it finished. The album is searched for again." + ended, err := q.GiveUpGrab(ctx, dbq.GiveUpGrabParams{ID: g.ID, EndReason: EndAbandoned, EndDetail: why, States: running}) + if err != nil { + return dbq.Grab{}, ErrNotAbandonable + } + if g.InfoHash != "" { + if err := acquire.NewMemory(s.pool).Block(ctx, dbq.BlockReleaseParams{ + InfoHash: g.InfoHash, Name: g.Name, Reason: acquire.BlockOperator, Detail: "Given up by you before it finished.", + WantID: g.WantID, BlockedBy: by, + }); err != nil { + s.logger.Warn("lifecycle: set the release aside", "grab", g.Name, "err", err) + } + var rep Report + if d, err := s.ClientFor(ctx, g.Client); err != nil { + s.raise(ctx, q, g.Client, g.InfoHash, g.ID, g.Name, NotRemoved, notRemoved(err), &rep) + } else if s.remove(ctx, q, d, g.Client, g.InfoHash, g.ID, g.Name, &rep) { + _, _ = q.SettleInterventionsFor(ctx, dbq.SettleInterventionsForParams{Client: g.Client, InfoHash: g.InfoHash}) + } + } + if g.WantID.Valid { + if err := q.SearchWantAgain(ctx, g.WantID); err != nil { + s.logger.Warn("lifecycle: search again", "grab", g.Name, "err", err) + } + } + return ended, nil +} + +func isRunning(state string) bool { + for _, r := range running { + if state == r { + return true + } + } + return false +} + +// reasonFor is a condition in a person's words. +func reasonFor(trigger string) string { + switch trigger { + case NoPeers: + return "nobody was sharing it" + case NoMetadata: + return "its file list never arrived" + case ClientError: + return "the download client gave up on it" + } + return "it stalled" +} + +func notRemoved(err error) string { + return "Given up on, but the download client would not remove it: " + firstLine(err.Error()) + + ". Minstrel asks again every ten minutes." +} + +// span is a duration as a person says it. +func span(d time.Duration) string { + switch { + case d < 2*time.Hour: + return fmt.Sprintf("%d minutes", int(d.Minutes())) + case d < 48*time.Hour: + return fmt.Sprintf("%d hours", int(d.Hours())) + } + return fmt.Sprintf("%d days", int(d.Hours()/24)) +} + +func capital(s string) string { + if s == "" { + return s + } + return strings.ToUpper(s[:1]) + s[1:] +} + +func firstLine(s string) string { + if i := strings.IndexByte(s, '\n'); i >= 0 { + return strings.TrimSpace(s[:i]) + } + return strings.TrimSpace(s) +} diff --git a/internal/lifecycle/lifecycle_test.go b/internal/lifecycle/lifecycle_test.go new file mode 100644 index 00000000..b5e9277b --- /dev/null +++ b/internal/lifecycle/lifecycle_test.go @@ -0,0 +1,488 @@ +package lifecycle + +import ( + "context" + "errors" + "io" + "log/slog" + "os" + "strings" + "testing" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/db" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" + "git.fabledsword.com/bvandeusen/minstrel/internal/downloader" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" +) + +func newPool(t *testing.T) *pgxpool.Pool { + t.Helper() + if testing.Short() { + t.Skip("skipping integration test in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(context.Background(), dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + dbtest.ResetDB(t, pool) + return pool +} + +const ( + hash = "0123456789abcdef0123456789abcdef01234567" + other = "fedcba9876543210fedcba9876543210fedcba98" +) + +// fakeClient lists the torrents it holds and records what it is asked. +type fakeClient struct { + downloader.Driver + held map[string]downloader.Torrent + err error + deleteErr error + deleted []string + reannounce int + recheck int +} + +func (c *fakeClient) Torrents(_ context.Context, hashes ...string) ([]downloader.Torrent, error) { + if c.err != nil { + return nil, c.err + } + var out []downloader.Torrent + if len(hashes) == 0 { + for _, t := range c.held { + out = append(out, t) + } + return out, nil + } + for _, h := range hashes { + if t, ok := c.held[h]; ok { + out = append(out, t) + } + } + return out, nil +} + +func (c *fakeClient) Delete(_ context.Context, _ bool, hashes ...string) error { + if c.deleteErr != nil { + return c.deleteErr + } + c.deleted = append(c.deleted, hashes...) + for _, h := range hashes { + delete(c.held, h) + } + return nil +} + +func (c *fakeClient) Reannounce(context.Context, ...string) error { c.reannounce++; return nil } +func (c *fakeClient) Recheck(context.Context, ...string) error { c.recheck++; return nil } + +type stack struct { + q *dbq.Queries + s *Service + client *fakeClient + want dbq.Want + grab dbq.Grab + clock time.Time +} + +// newStack wants RAM and has a grab of it running in a fake client holding +// t under hash. +func newStack(t *testing.T, tor downloader.Torrent) *stack { + t.Helper() + ctx := context.Background() + pool := newPool(t) + q := dbq.New(pool) + w, err := wanted.WantWith(ctx, q, wanted.WantParams{GroupMBID: "rg-ram", ArtistMBID: "mb-dp", ArtistName: "Daft Punk", + Title: "Random Access Memories", PrimaryType: "Album", FirstReleaseDate: "2013-05-17", Reason: wanted.ReasonRequest}) + if err != nil { + t.Fatal(err) + } + g, err := acquire.NewMemory(pool).Grabbed(ctx, dbq.InsertGrabParams{WantID: w.ID, InfoHash: hash, + Name: "Daft Punk - RAM [FLAC]", Key: hash, Client: "qbittorrent", SavePath: "/downloads", Origin: acquire.OriginSchedule}, + pgtype.UUID{}, "") + if err != nil { + t.Fatal(err) + } + if err := q.MarkWantSearched(ctx, w.ID); err != nil { + t.Fatal(err) + } + tor.Hash = hash + st := &stack{q: q, client: &fakeClient{held: map[string]downloader.Torrent{hash: tor}}, want: w, grab: g, + clock: time.Now()} + st.s = New(pool, slog.New(slog.NewTextHandler(io.Discard, nil))) + st.s.ClientFor = func(context.Context, string) (downloader.Driver, error) { return st.client, nil } + st.s.Now = func() time.Time { return st.clock } + return st +} + +func (st *stack) grabNow(t *testing.T) dbq.Grab { + t.Helper() + g, err := st.q.GetGrab(context.Background(), st.grab.ID) + if err != nil { + t.Fatal(err) + } + return g +} + +func (st *stack) searchedAgain(t *testing.T) bool { + t.Helper() + w, err := st.q.GetWantByReleaseGroup(context.Background(), "rg-ram") + if err != nil { + t.Fatal(err) + } + return !w.SearchedAt.Valid +} + +func (st *stack) ledger(t *testing.T) []dbq.DownloadIntervention { + t.Helper() + rows, err := st.s.pool.Query(context.Background(), `SELECT * FROM download_interventions ORDER BY first_seen, id`) + if err != nil { + t.Fatal(err) + } + out, err := pgx.CollectRows(rows, pgx.RowToStructByPos[dbq.DownloadIntervention]) + if err != nil { + t.Fatal(err) + } + return out +} + +func TestLifecycle_AFailedDownloadIsGivenUpAtOnce_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.4, State: "error", + FailReason: "No space left on device"}) + rep := st.s.Pass(context.Background()) + if rep.GivenUp != 1 { + t.Fatalf("pass = %+v", rep) + } + g := st.grabNow(t) + if !g.EndedAt.Valid || g.EndReason != EndFailed { + t.Fatalf("grab ended=%v reason=%q", g.EndedAt.Valid, g.EndReason) + } + if len(st.client.deleted) != 1 || !st.searchedAgain(t) { + t.Errorf("deleted %v, searched again %v", st.client.deleted, st.searchedAgain(t)) + } + blocks, err := st.q.ListReleaseBlocks(context.Background()) + if err != nil || len(blocks) != 1 || blocks[0].Reason != acquire.BlockStalled || !blocks[0].ExpiresAt.Valid { + t.Fatalf("blocks = %+v %v", blocks, err) + } + if d := blocks[0].ExpiresAt.Time.Sub(st.clock); d < StallBlock-time.Minute || d > StallBlock+time.Minute { + t.Errorf("held back for %s", d) + } + rows := st.ledger(t) + if len(rows) != 1 || rows[0].Trigger != ClientError || rows[0].Action != actGivenUp || rows[0].Outcome != "resolved" || + rows[0].Detail != "No space left on device" { + t.Errorf("ledger = %+v", rows) + } +} + +func TestLifecycle_ADownloadNobodySharesIsRetriedThenGivenUp_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.3, State: "stalledDL"}) + ctx := context.Background() + + // Within the stall setting: nothing is wrong yet. + if rep := st.s.Pass(ctx); rep.Retried+rep.GivenUp != 0 || len(st.ledger(t)) != 0 { + t.Fatalf("a young download was judged: %+v", rep) + } + + st.clock = st.clock.Add(7 * time.Hour) + for i := 1; i <= Retries; i++ { + if rep := st.s.Pass(ctx); rep.Retried != 1 || rep.GivenUp != 0 { + t.Fatalf("pass %d = %+v", i, rep) + } + } + if st.client.reannounce != Retries || st.client.recheck != Retries { + t.Errorf("reannounced %d, rechecked %d", st.client.reannounce, st.client.recheck) + } + if st.grabNow(t).EndedAt.Valid { + t.Fatal("given up before the retries ran out") + } + if rep := st.s.Pass(ctx); rep.GivenUp != 1 { + t.Fatalf("last pass = %+v", rep) + } + g := st.grabNow(t) + if g.EndReason != EndStalled || len(st.client.deleted) != 1 || !st.searchedAgain(t) { + t.Errorf("grab %q: %s; deleted %v", g.EndReason, g.EndDetail, st.client.deleted) + } + rows := st.ledger(t) + if len(rows) != 1 || rows[0].Trigger != NoPeers || rows[0].Remediations != Retries+1 || rows[0].Outcome != "resolved" { + t.Errorf("ledger = %+v", rows) + } +} + +func TestLifecycle_AStalledDownloadThatRecoversIsSettled_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.3, State: "stalledDL"}) + ctx := context.Background() + st.clock = st.clock.Add(7 * time.Hour) + st.s.Pass(ctx) + t2 := st.client.held[hash] + t2.NumSeeds = 3 + st.client.held[hash] = t2 + st.s.Pass(ctx) + rows := st.ledger(t) + if len(rows) != 1 || rows[0].Outcome != "resolved" || rows[0].Action != actRetried { + t.Fatalf("ledger = %+v", rows) + } + if st.grabNow(t).EndedAt.Valid { + t.Error("a download that found peers was given up") + } +} + +func TestLifecycle_TheStallSettingOffNeverGivesUp_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.3, State: "stalledDL"}) + ctx := context.Background() + if _, err := st.s.pool.Exec(ctx, `UPDATE acquisition_settings SET stall_hours = 0`); err != nil { + t.Fatal(err) + } + st.clock = st.clock.Add(30 * 24 * time.Hour) + for range Retries + 2 { + if rep := st.s.Pass(ctx); rep.Retried+rep.GivenUp != 0 { + t.Fatalf("pass = %+v", rep) + } + } +} + +func TestLifecycle_ADownloadGoneFromTheClientEndsAfterHalfAnHour_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.3, State: "downloading", NumSeeds: 4}) + ctx := context.Background() + delete(st.client.held, hash) + st.client.held[other] = downloader.Torrent{Hash: other, Name: "Someone else's", Progress: 1, State: "uploading"} + + if rep := st.s.Pass(ctx); rep.Gone != 0 { + t.Fatalf("ended at first sight: %+v", rep) + } + st.clock = st.clock.Add(31 * time.Minute) + if rep := st.s.Pass(ctx); rep.Gone != 1 { + t.Fatalf("pass = %+v", rep) + } + g := st.grabNow(t) + if g.EndReason != EndGone || !st.searchedAgain(t) { + t.Errorf("grab %q, searched again %v", g.EndReason, st.searchedAgain(t)) + } + if blocks, _ := st.q.ListReleaseBlocks(ctx); len(blocks) != 0 { + t.Errorf("a release nothing was wrong with was held back: %+v", blocks) + } +} + +// A release that leaves its client a second time is held back, whatever the +// cause, so it is not taken again on every pass. +func TestLifecycle_AReleaseGoneTwiceIsHeldBack_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.3, State: "downloading", NumSeeds: 4}) + ctx := context.Background() + delete(st.client.held, hash) + st.client.held[other] = downloader.Torrent{Hash: other, Name: "Someone else's", Progress: 1, State: "uploading"} + + st.s.Pass(ctx) + st.clock = st.clock.Add(31 * time.Minute) + if rep := st.s.Pass(ctx); rep.Gone != 1 { + t.Fatalf("first gone: %+v", rep) + } + if blocks, _ := st.q.ListReleaseBlocks(ctx); len(blocks) != 0 { + t.Fatalf("held back after one gone: %+v", blocks) + } + + again, err := acquire.NewMemory(st.s.pool).Grabbed(ctx, dbq.InsertGrabParams{WantID: st.want.ID, InfoHash: hash, + Name: "Daft Punk - RAM [FLAC]", Key: hash, Client: "qbittorrent", SavePath: "/downloads", Origin: acquire.OriginSchedule}, + pgtype.UUID{}, "") + if err != nil { + t.Fatal(err) + } + // first_seen is the database's clock and the test's runs ahead of it, so + // the second grab may end on either pass; count both. + gone := st.s.Pass(ctx).Gone + st.clock = st.clock.Add(31 * time.Minute) + if gone += st.s.Pass(ctx).Gone; gone != 1 { + t.Fatalf("second grab ended gone %d times, want 1", gone) + } + blocks, err := st.q.ListReleaseBlocks(ctx) + if err != nil { + t.Fatal(err) + } + if len(blocks) != 1 || blocks[0].Reason != acquire.BlockGone || !blocks[0].ExpiresAt.Valid { + t.Fatalf("blocks = %+v, want one timed gone block", blocks) + } + g, err := st.q.GetGrab(ctx, again.ID) + if err != nil { + t.Fatal(err) + } + if g.EndReason != EndGone || !strings.Contains(g.EndDetail, "held back until") { + t.Errorf("second grab %q: %s", g.EndReason, g.EndDetail) + } +} + +func TestLifecycle_AnEmptyListIsNotAMeasurement_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.3, State: "downloading"}) + ctx := context.Background() + delete(st.client.held, hash) + st.clock = st.clock.Add(24 * time.Hour) + for range 3 { + if rep := st.s.Pass(ctx); rep.Gone != 0 { + t.Fatalf("pass = %+v", rep) + } + } + if len(st.ledger(t)) != 0 || st.grabNow(t).EndedAt.Valid { + t.Error("a client listing nothing was taken to have lost the download") + } +} + +func TestLifecycle_AnUnreachableClientIsRecordedAndSettled_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.3, State: "stalledDL"}) + ctx := context.Background() + st.clock = st.clock.Add(7 * time.Hour) + st.client.err = errors.New("dial tcp: connection refused") + st.s.Pass(ctx) + down, err := st.q.ListClientsDown(ctx) + if err != nil || len(down) != 1 || down[0].Client != "qbittorrent" { + t.Fatalf("clients down = %+v %v", down, err) + } + if st.client.reannounce != 0 { + t.Error("a download was judged through a client that did not answer") + } + st.client.err = nil + st.s.Pass(ctx) + if down, _ := st.q.ListClientsDown(ctx); len(down) != 0 { + t.Errorf("still down: %+v", down) + } +} + +func TestLifecycle_AnImportedDownloadSeedsThenIsRemoved_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 1, State: "uploading", Ratio: 0.8}) + ctx := context.Background() + if _, err := st.q.RecordImport(ctx, dbq.RecordImportParams{ID: st.grab.ID, ImportState: "imported", + ImportReason: "Imported.", ImportDetail: []byte("{}"), ImportedFiles: 3, EndReason: "imported"}); err != nil { + t.Fatal(err) + } + if !st.grabNow(t).Seeding { + t.Fatal("an import did not leave its download seeding") + } + if rep := st.s.Pass(ctx); rep.Released != 0 || len(st.client.deleted) != 0 { + t.Fatalf("removed before the ratio: %+v", rep) + } + if g := st.grabNow(t); g.Ratio < 0.79 || g.Ratio > 0.81 { + t.Errorf("ratio recorded %v", g.Ratio) + } + tor := st.client.held[hash] + tor.Ratio = 2.1 + st.client.held[hash] = tor + if rep := st.s.Pass(ctx); rep.Released != 1 || len(st.client.deleted) != 1 { + t.Fatalf("pass = %+v, deleted %v", rep, st.client.deleted) + } + if !st.grabNow(t).SeedReleasedAt.Valid { + t.Error("the release was not recorded") + } +} + +func TestLifecycle_AnImportedDownloadIsRemovedWhenItsDaysAreUp_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 1, State: "uploading", Ratio: 0.1}) + ctx := context.Background() + if _, err := st.q.RecordImport(ctx, dbq.RecordImportParams{ID: st.grab.ID, ImportState: "imported", + ImportReason: "Imported.", ImportDetail: []byte("{}"), ImportedFiles: 3, EndReason: "imported"}); err != nil { + t.Fatal(err) + } + st.clock = st.clock.Add(15 * 24 * time.Hour) + if rep := st.s.Pass(ctx); rep.Released != 1 || len(st.client.deleted) != 1 { + t.Fatalf("pass = %+v", rep) + } +} + +func TestLifecycle_ADownloadTheClientWouldNotRemoveIsAskedAgain_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.4, State: "error"}) + ctx := context.Background() + st.client.deleteErr = errors.New("HTTP 500") + if rep := st.s.Pass(ctx); rep.GivenUp != 1 { + t.Fatalf("pass = %+v", rep) + } + if !st.grabNow(t).EndedAt.Valid { + t.Fatal("a refused removal held the grab") + } + open := 0 + for _, r := range st.ledger(t) { + if r.Trigger == NotRemoved && r.Outcome == "open" { + open++ + } + } + if open != 1 { + t.Fatalf("ledger = %+v", st.ledger(t)) + } + st.client.deleteErr = nil + if rep := st.s.Pass(ctx); rep.Removed != 1 || len(st.client.deleted) != 1 { + t.Fatalf("pass = %+v", rep) + } + for _, r := range st.ledger(t) { + if r.Outcome == "open" { + t.Errorf("still open: %+v", r) + } + } +} + +func TestLifecycle_APersonGivingUpSetsTheReleaseAsideForGood_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "Daft Punk - RAM [FLAC]", Progress: 0.4, State: "downloading"}) + ctx := context.Background() + if _, err := st.s.Abandon(ctx, st.grab.ID, pgtype.UUID{}); err != nil { + t.Fatal(err) + } + g := st.grabNow(t) + if g.EndReason != EndAbandoned || len(st.client.deleted) != 1 || !st.searchedAgain(t) { + t.Errorf("grab %q, deleted %v", g.EndReason, st.client.deleted) + } + blocks, _ := st.q.ListReleaseBlocks(ctx) + if len(blocks) != 1 || blocks[0].Reason != acquire.BlockOperator || blocks[0].ExpiresAt.Valid { + t.Errorf("blocks = %+v", blocks) + } + if _, err := st.s.Abandon(ctx, st.grab.ID, pgtype.UUID{}); !errors.Is(err, ErrNotAbandonable) { + t.Errorf("abandoned twice: %v", err) + } +} + +func TestLifecycle_AHoldExpiresButABlockForGoodIsNeverShortened_Integration(t *testing.T) { + st := newStack(t, downloader.Torrent{Name: "x", Progress: 0.4, State: "downloading"}) + ctx := context.Background() + mem := acquire.NewMemory(st.s.pool) + past := pgtype.Timestamptz{Time: time.Now().Add(-time.Hour), Valid: true} + if err := mem.Block(ctx, dbq.BlockReleaseParams{InfoHash: other, Name: "old", Reason: acquire.BlockStalled, ExpiresAt: past}); err != nil { + t.Fatal(err) + } + if blocks, _ := st.q.ListReleaseBlocks(ctx); len(blocks) != 0 { + t.Errorf("an expired hold still holds: %+v", blocks) + } + if n, err := st.q.DeleteExpiredBlocks(ctx); err != nil || n != 1 { + t.Errorf("lifted %d %v", n, err) + } + + if err := mem.Block(ctx, dbq.BlockReleaseParams{InfoHash: hash, Name: "x", Reason: acquire.BlockOperator}); err != nil { + t.Fatal(err) + } + soon := pgtype.Timestamptz{Time: time.Now().Add(time.Hour), Valid: true} + if err := mem.Block(ctx, dbq.BlockReleaseParams{InfoHash: hash, Name: "x", Reason: acquire.BlockStalled, ExpiresAt: soon}); err != nil { + t.Fatal(err) + } + blocks, _ := st.q.ListReleaseBlocks(ctx) + if len(blocks) != 1 || blocks[0].Reason != acquire.BlockOperator || blocks[0].ExpiresAt.Valid { + t.Errorf("a block for good was shortened: %+v", blocks) + } +} + +func TestSpan(t *testing.T) { + for _, c := range []struct { + d time.Duration + want string + }{{45 * time.Minute, "45 minutes"}, {7 * time.Hour, "7 hours"}, {3 * 24 * time.Hour, "3 days"}} { + if got := span(c.d); got != c.want { + t.Errorf("span(%s) = %q, want %q", c.d, got, c.want) + } + } +} diff --git a/internal/logging/logging.go b/internal/logging/logging.go index d0561466..80ca1041 100644 --- a/internal/logging/logging.go +++ b/internal/logging/logging.go @@ -23,7 +23,8 @@ func New(w io.Writer, level, format string) (*slog.Logger, error) { default: return nil, fmt.Errorf("unknown log format %q (want text|json)", format) } - return slog.New(h), nil + // Errors are also kept for the admin Overview. + return slog.New(newRingHandler(h, Errors)), nil } func parseLevel(s string) (slog.Level, error) { diff --git a/internal/logging/recent.go b/internal/logging/recent.go new file mode 100644 index 00000000..55c5f8a6 --- /dev/null +++ b/internal/logging/recent.go @@ -0,0 +1,122 @@ +package logging + +import ( + "context" + "fmt" + "log/slog" + "strings" + "sync" + "time" +) + +// The server's recent errors, kept in memory for the admin Overview: +// what has gone wrong since the process started, without reading the +// container's logs. Only ERROR and above, and only the newest few; the full +// record is still the log stream. + +// RecentErrorsKept is how many recent errors are kept. +const RecentErrorsKept = 50 + +// LoggedError is one error the server logged. +type LoggedError struct { + At time.Time + Message string + // Attrs is the record's attributes as key=value, the way the text log + // writes them. + Attrs string +} + +// ErrorRing keeps the newest errors and counts all of them. +type ErrorRing struct { + mu sync.Mutex + buf []LoggedError + next int + total int64 +} + +// Errors is the process's ring, filled by every logger New returns. +var Errors = NewErrorRing(RecentErrorsKept) + +// NewErrorRing returns a ring that keeps the newest n errors. +func NewErrorRing(n int) *ErrorRing { + return &ErrorRing{buf: make([]LoggedError, 0, n)} +} + +func (r *ErrorRing) add(e LoggedError) { + r.mu.Lock() + defer r.mu.Unlock() + r.total++ + if len(r.buf) < cap(r.buf) { + r.buf = append(r.buf, e) + return + } + r.buf[r.next] = e + r.next = (r.next + 1) % len(r.buf) +} + +// Snapshot returns the kept errors, newest first, and how many have been +// logged in all since the process started. +func (r *ErrorRing) Snapshot() ([]LoggedError, int64) { + r.mu.Lock() + defer r.mu.Unlock() + out := make([]LoggedError, 0, len(r.buf)) + for i := len(r.buf) - 1; i >= 0; i-- { + out = append(out, r.buf[(r.next+i)%len(r.buf)]) + } + return out, r.total +} + +// ringHandler passes every record on, and copies the errors into a ring. +type ringHandler struct { + inner slog.Handler + ring *ErrorRing + // attrs are the logger's own attributes (component=… and the like), + // already rendered, group-qualified. + attrs []string + groups []string +} + +func newRingHandler(inner slog.Handler, ring *ErrorRing) *ringHandler { + return &ringHandler{inner: inner, ring: ring} +} + +func (h *ringHandler) Enabled(ctx context.Context, l slog.Level) bool { + return h.inner.Enabled(ctx, l) +} + +func (h *ringHandler) Handle(ctx context.Context, rec slog.Record) error { + if rec.Level >= slog.LevelError { + parts := append([]string(nil), h.attrs...) + rec.Attrs(func(a slog.Attr) bool { + parts = append(parts, h.render(a)) + return true + }) + h.ring.add(LoggedError{At: rec.Time, Message: rec.Message, Attrs: strings.Join(parts, " ")}) + } + return h.inner.Handle(ctx, rec) +} + +func (h *ringHandler) WithAttrs(attrs []slog.Attr) slog.Handler { + next := *h + next.inner = h.inner.WithAttrs(attrs) + next.attrs = append([]string(nil), h.attrs...) + for _, a := range attrs { + next.attrs = append(next.attrs, h.render(a)) + } + return &next +} + +func (h *ringHandler) WithGroup(name string) slog.Handler { + next := *h + next.inner = h.inner.WithGroup(name) + next.groups = append(append([]string(nil), h.groups...), name) + return &next +} + +func (h *ringHandler) render(a slog.Attr) string { + key := a.Key + if len(h.groups) > 0 { + key = strings.Join(h.groups, ".") + "." + key + } + return fmt.Sprintf("%s=%v", key, a.Value.Resolve()) +} diff --git a/internal/logging/recent_test.go b/internal/logging/recent_test.go new file mode 100644 index 00000000..dacbbbb7 --- /dev/null +++ b/internal/logging/recent_test.go @@ -0,0 +1,49 @@ +package logging + +import ( + "bytes" + "log/slog" + "testing" +) + +func TestRingHandler_KeepsErrorsNewestFirst(t *testing.T) { + ring := NewErrorRing(3) + var out bytes.Buffer + log := slog.New(newRingHandler(slog.NewTextHandler(&out, nil), ring)).With("component", "scan") + + log.Info("fine") + log.Warn("odd") + for _, msg := range []string{"one", "two", "three", "four"} { + log.WithGroup("job").Error(msg, "n", len(msg)) + } + + got, total := ring.Snapshot() + if total != 4 { + t.Errorf("total = %d, want 4 (info and warn are not errors)", total) + } + if len(got) != 3 { + t.Fatalf("kept %d, want 3", len(got)) + } + for i, want := range []string{"four", "three", "two"} { + if got[i].Message != want { + t.Errorf("kept[%d] = %q, want %q", i, got[i].Message, want) + } + } + if got[0].Attrs != "component=scan job.n=4" { + t.Errorf("attrs = %q", got[0].Attrs) + } + // Every record still reaches the log stream. + if n := bytes.Count(out.Bytes(), []byte("\n")); n != 6 { + t.Errorf("log lines = %d, want 6", n) + } +} + +func TestErrorRing_BeforeItFills(t *testing.T) { + ring := NewErrorRing(5) + ring.add(LoggedError{Message: "a"}) + ring.add(LoggedError{Message: "b"}) + got, total := ring.Snapshot() + if total != 2 || len(got) != 2 || got[0].Message != "b" || got[1].Message != "a" { + t.Errorf("got %+v total %d, want b, a", got, total) + } +} diff --git a/internal/mailer/mailer.go b/internal/mailer/mailer.go index feea93dc..090017d5 100644 --- a/internal/mailer/mailer.go +++ b/internal/mailer/mailer.go @@ -74,8 +74,8 @@ func (s *SMTPSender) Send(ctx context.Context, to, subject, textBody, htmlBody s // We respect the use_tls flag: if true, we require STARTTLS via // smtp.SendMail's default behavior (which negotiates STARTTLS when // the server advertises it). For implicit-TLS (port 465 style), - // we'd need a custom dial — skip that complexity for v1 and rely on - // STARTTLS via port 587 (the default). + // we'd need a custom dial; instead this relies on STARTTLS via port + // 587 (the default). dialCtx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() _ = dialCtx // smtp.SendMail doesn't accept ctx; the timeout is enforced via Dialer below diff --git a/internal/mood/mood.go b/internal/mood/mood.go index 7e855643..032a2743 100644 --- a/internal/mood/mood.go +++ b/internal/mood/mood.go @@ -1,5 +1,5 @@ -// Package mood maps folksonomy tags (from the track_tags cache, #1490) to a -// small set of canonical mood buckets (#1534, milestone #160 Opt 2b). Both the +// Package mood maps folksonomy tags (from the track_tags cache) to a small +// set of canonical mood buckets. Both the // taste-profile builder and the recommendation scorer classify a track's tags // through Of() so a track's mood is derived the same way on both sides. // diff --git a/internal/musicbrainz/client.go b/internal/musicbrainz/client.go new file mode 100644 index 00000000..86721968 --- /dev/null +++ b/internal/musicbrainz/client.go @@ -0,0 +1,209 @@ +package musicbrainz + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "strconv" + "strings" + "time" +) + +// DefaultBaseURL is the public MusicBrainz web service. +const DefaultBaseURL = "https://musicbrainz.org/ws/2" + +// UserAgent identifies Minstrel. MusicBrainz refuses requests without a +// descriptive one. +const UserAgent = "Minstrel/1.0 ( https://git.fabledsword.com/bvandeusen/minstrel )" + +// DefaultTimeout bounds one request. A browse page with track +// lists is the slowest thing asked for. +const DefaultTimeout = 20 * time.Second + +// pageSize is the most a browse returns per page. +const pageSize = 100 + +// maxPages bounds a browse, so an artist with thousands of release groups +// cannot hold the pacer for an hour. 25 pages is 2,500 entries. +const maxPages = 25 + +var ( + // ErrUnreachable: MusicBrainz could not be asked, or kept answering with + // its throttle or a server error. Callers degrade visibly: they say + // MusicBrainz is unreachable rather than show an empty answer. + ErrUnreachable = errors.New("musicbrainz: unreachable") + // ErrNotFound: MusicBrainz has nothing under that id. + ErrNotFound = errors.New("musicbrainz: not found") + // ErrBadResponse: MusicBrainz answered with something that is not the + // JSON asked for. + ErrBadResponse = errors.New("musicbrainz: unexpected response") +) + +// Client asks MusicBrainz. The zero value is usable: it talks to the public +// service through the shared pacer. +type Client struct { + BaseURL string + HTTP *http.Client + // Pace spaces requests; nil means Shared(). Tests pass a zero Pacer. + Pace *Pacer + // Retries is how many times a throttled or failed request is tried + // again. Zero means 2; negative means none. + Retries int + // Backoff is the first retry's wait, doubled each time. Zero means 1s. + Backoff time.Duration +} + +func (c *Client) base() string { + if c.BaseURL != "" { + return strings.TrimRight(c.BaseURL, "/") + } + return DefaultBaseURL +} + +func (c *Client) httpClient() *http.Client { + if c.HTTP != nil { + return c.HTTP + } + return &http.Client{Timeout: DefaultTimeout} +} + +func (c *Client) pacer() *Pacer { + if c.Pace != nil { + return c.Pace + } + return Shared() +} + +func (c *Client) retries() int { + switch { + case c.Retries < 0: + return 0 + case c.Retries == 0: + return 2 + } + return c.Retries +} + +func (c *Client) backoff(attempt int) time.Duration { + b := c.Backoff + if b <= 0 { + b = time.Second + } + return b << attempt +} + +// get fetches path?query (fmt=json added) into out, waiting on the pacer +// before every attempt. +func (c *Client) get(ctx context.Context, path string, query url.Values, out any) error { + if query == nil { + query = url.Values{} + } + query.Set("fmt", "json") + full := c.base() + path + "?" + query.Encode() + + var last error + for attempt := 0; attempt <= c.retries(); attempt++ { + if attempt > 0 { + if err := sleep(ctx, c.backoff(attempt-1)); err != nil { + return err + } + } + if err := c.pacer().Wait(ctx); err != nil { + return err + } + retry, err := c.once(ctx, full, out) + if !retry { + return err + } + last = err + } + return last +} + +// once makes one request. retry reports whether trying again could help. +func (c *Client) once(ctx context.Context, full string, out any) (retry bool, err error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, full, nil) + if err != nil { + return false, fmt.Errorf("musicbrainz: build request: %w", err) + } + req.Header.Set("Accept", "application/json") + req.Header.Set("User-Agent", UserAgent) + + resp, err := c.httpClient().Do(req) + if err != nil { + if ctx.Err() != nil { + return false, ctx.Err() + } + return true, fmt.Errorf("%w: %v", ErrUnreachable, err) + } + defer func() { _ = resp.Body.Close() }() + + switch { + case resp.StatusCode == http.StatusNotFound: + return false, ErrNotFound + case resp.StatusCode == http.StatusServiceUnavailable, + resp.StatusCode == http.StatusTooManyRequests, + resp.StatusCode >= 500: + // 503 is how MusicBrainz says "slow down". + return true, fmt.Errorf("%w: HTTP %d", ErrUnreachable, resp.StatusCode) + case resp.StatusCode != http.StatusOK: + return false, fmt.Errorf("%w: HTTP %d", ErrBadResponse, resp.StatusCode) + } + body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil { + return true, fmt.Errorf("%w: read: %v", ErrUnreachable, err) + } + if err := json.Unmarshal(body, out); err != nil { + return false, fmt.Errorf("%w: %v", ErrBadResponse, err) + } + return false, nil +} + +func sleep(ctx context.Context, d time.Duration) error { + t := time.NewTimer(d) + defer t.Stop() + select { + case <-ctx.Done(): + return ctx.Err() + case <-t.C: + return nil + } +} + +// luceneSpecial are the characters MusicBrainz's search syntax gives meaning. +const luceneSpecial = `+-&|!(){}[]^"~*?:\/` + +// Escape makes text typed by a person safe to put in a search query, so +// "AC/DC" or "Sunn O)))" searches for what was typed rather than failing to +// parse. +func Escape(s string) string { + var b strings.Builder + for _, r := range s { + if strings.ContainsRune(luceneSpecial, r) { + b.WriteByte('\\') + } + b.WriteRune(r) + } + return b.String() +} + +// fieldQuery searches each field for the escaped text. +func fieldQuery(text string, fields ...string) string { + t := Escape(strings.TrimSpace(text)) + parts := make([]string, len(fields)) + for i, f := range fields { + parts[i] = f + ":(" + t + ")" + } + return strings.Join(parts, " OR ") +} + +func searchLimit(limit int) string { + if limit <= 0 || limit > 100 { + limit = 25 + } + return strconv.Itoa(limit) +} diff --git a/internal/musicbrainz/client_test.go b/internal/musicbrainz/client_test.go new file mode 100644 index 00000000..ded2815f --- /dev/null +++ b/internal/musicbrainz/client_test.go @@ -0,0 +1,223 @@ +package musicbrainz + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strconv" + "sync/atomic" + "testing" + "time" +) + +// testClient talks to srv with no pacing and fast retries. +func testClient(srv *httptest.Server) *Client { + return &Client{BaseURL: srv.URL, Pace: &Pacer{}, Backoff: time.Millisecond} +} + +func TestEscape_KeepsWhatWasTyped(t *testing.T) { + for in, want := range map[string]string{ + "Radiohead": "Radiohead", + "AC/DC": `AC\/DC`, + "Sunn O)))": `Sunn O\)\)\)`, + `"Weird" Al`: `\"Weird\" Al`, + } { + if got := Escape(in); got != want { + t.Errorf("Escape(%q) = %q, want %q", in, got, want) + } + } +} + +func TestSearchReleaseGroups_AsksTitleAndArtistAndReadsCredits(t *testing.T) { + var query, agent atomic.Value + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + query.Store(r.URL.Query().Get("query")) + agent.Store(r.Header.Get("User-Agent")) + _, _ = w.Write([]byte(`{"release-groups": [{ + "id": "rg1", "title": "OK Computer", "primary-type": "Album", "secondary-types": [], + "first-release-date": "1997-05-21", + "artist-credit": [{"name": "Radiohead", "joinphrase": "", "artist": {"id": "a1", "name": "Radiohead"}}] + }]}`)) + })) + defer srv.Close() + + got, err := testClient(srv).SearchReleaseGroups(context.Background(), "ok computer", 10) + if err != nil { + t.Fatal(err) + } + if q := query.Load(); q != "releasegroup:(ok computer) OR artist:(ok computer)" { + t.Errorf("query = %q", q) + } + if agent.Load() != UserAgent { + t.Errorf("User-Agent = %q", agent.Load()) + } + if len(got) != 1 || got[0].MBID != "rg1" || got[0].Year() != "1997" || got[0].ArtistCredit != "Radiohead" || got[0].ArtistMBIDs[0] != "a1" { + t.Errorf("got %+v", got) + } +} + +func TestSearchReleaseGroupsBy_AsksForBothTitleAndArtist(t *testing.T) { + var query atomic.Value + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + query.Store(r.URL.Query().Get("query")) + _, _ = w.Write([]byte(`{"release-groups": [{"id": "rg1", "title": "Highway to Hell", + "artist-credit": [{"name": "AC/DC", "artist": {"id": "a1"}}]}]}`)) + })) + defer srv.Close() + + got, err := testClient(srv).SearchReleaseGroupsBy(context.Background(), " AC/DC ", "Highway to Hell", 10) + if err != nil { + t.Fatal(err) + } + if q := query.Load(); q != `releasegroup:(Highway to Hell) AND artist:(AC\/DC)` { + t.Errorf("query = %q", q) + } + if len(got) != 1 || got[0].MBID != "rg1" || got[0].ArtistMBIDs[0] != "a1" { + t.Errorf("got %+v", got) + } +} + +func TestSearchRecordings_FilesASongUnderItsAlbum(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"recordings": [{ + "id": "rec1", "title": "Karma Police", "length": 264000, + "artist-credit": [{"name": "Radiohead", "artist": {"id": "a1"}}], + "releases": [ + {"id": "r1", "status": "Official", "release-group": {"id": "rg-live", "title": "Live", "primary-type": "Album", "secondary-types": ["Live"]}}, + {"id": "r2", "status": "Official", "release-group": {"id": "rg1", "title": "OK Computer", "primary-type": "Album"}} + ] + }]}`)) + })) + defer srv.Close() + + got, err := testClient(srv).SearchRecordings(context.Background(), "karma police", 5) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].ReleaseGroup.MBID != "rg1" || got[0].LengthMs != 264000 { + t.Errorf("got %+v, want it filed under the studio album", got) + } +} + +func TestArtistReleaseGroups_PagesUntilTheCount(t *testing.T) { + var calls atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + offset, _ := strconv.Atoi(r.URL.Query().Get("offset")) + if r.URL.Query().Get("artist") != "a1" { + t.Errorf("artist = %q", r.URL.Query().Get("artist")) + } + // 3 in all, two per page. + var items string + for i := offset; i < offset+2 && i < 3; i++ { + if items != "" { + items += "," + } + items += fmt.Sprintf(`{"id": "rg%d", "title": "T%d", "primary-type": "Album"}`, i, i) + } + _, _ = fmt.Fprintf(w, `{"release-group-count": 3, "release-groups": [%s]}`, items) + })) + defer srv.Close() + + got, err := testClient(srv).ArtistReleaseGroups(context.Background(), "a1") + if err != nil { + t.Fatal(err) + } + if len(got) != 3 || calls.Load() != 2 { + t.Errorf("got %d groups in %d calls, want 3 in 2", len(got), calls.Load()) + } +} + +func TestReleaseGroupReleases_ReadsMediaAndTracks(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Query().Get("inc") != "media recordings" { + t.Errorf("inc = %q", r.URL.Query().Get("inc")) + } + _, _ = w.Write([]byte(`{"release-count": 1, "releases": [{ + "id": "r1", "title": "OK Computer", "status": "Official", "date": "1997-06-16", "country": "GB", + "media": [ + {"position": 1, "format": "CD", "tracks": [ + {"position": 1, "number": "1", "title": "Airbag", "length": 284000, "recording": {"id": "rec-a"}}, + {"position": 2, "number": "2", "title": "Paranoid Android", "length": null, "recording": {"id": "rec-b", "length": 383000}} + ]}, + {"position": 2, "format": "CD", "tracks": []} + ] + }]}`)) + })) + defer srv.Close() + + got, err := testClient(srv).ReleaseGroupReleases(context.Background(), "rg1") + if err != nil { + t.Fatal(err) + } + if len(got) != 1 { + t.Fatalf("got %+v", got) + } + r := got[0] + if r.TrackCount() != 2 || r.Formats() != "2×CD" { + t.Errorf("tracks %d, formats %q", r.TrackCount(), r.Formats()) + } + if tr := r.Media[0].Tracks[1]; tr.LengthMs != 383000 || tr.RecordingMBID != "rec-b" { + t.Errorf("a track with no length of its own should take the recording's: %+v", tr) + } +} + +func TestGet_ThrottleIsRetriedThenUnreachable(t *testing.T) { + var calls atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if calls.Add(1) == 1 { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + _, _ = w.Write([]byte(`{"artists": [{"id": "a1", "name": "Radiohead"}]}`)) + })) + defer srv.Close() + + got, err := testClient(srv).SearchArtists(context.Background(), "radiohead", 5) + if err != nil || len(got) != 1 || calls.Load() != 2 { + t.Fatalf("a throttled first answer should be retried: %v, %+v, %d calls", err, got, calls.Load()) + } + + down := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusBadGateway) + })) + defer down.Close() + if _, err := testClient(down).SearchArtists(context.Background(), "x", 5); !errors.Is(err, ErrUnreachable) { + t.Errorf("err = %v, want ErrUnreachable", err) + } +} + +func TestGet_NotFoundAndNoServer(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusNotFound) + })) + if _, err := testClient(srv).Artist(context.Background(), "nope"); !errors.Is(err, ErrNotFound) { + t.Errorf("err = %v, want ErrNotFound", err) + } + srv.Close() + c := testClient(srv) + c.Retries = -1 + if _, err := c.Artist(context.Background(), "a1"); !errors.Is(err, ErrUnreachable) { + t.Errorf("a closed server: err = %v, want ErrUnreachable", err) + } +} + +func TestPacer_SpacesCallers(t *testing.T) { + p := &Pacer{Interval: 30 * time.Millisecond} + start := time.Now() + for range 3 { + if err := p.Wait(context.Background()); err != nil { + t.Fatal(err) + } + } + if elapsed := time.Since(start); elapsed < 60*time.Millisecond { + t.Errorf("three waits took %v, want at least two intervals", elapsed) + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if err := p.Wait(ctx); err == nil { + t.Error("a cancelled wait should return the context's error") + } +} diff --git a/internal/musicbrainz/entities.go b/internal/musicbrainz/entities.go new file mode 100644 index 00000000..b7acd786 --- /dev/null +++ b/internal/musicbrainz/entities.go @@ -0,0 +1,402 @@ +package musicbrainz + +import ( + "context" + "net/url" + "strconv" + "strings" +) + +// Artist is a MusicBrainz artist. +type Artist struct { + MBID string + Name string + SortName string + Disambiguation string + Type string // Person, Group, Orchestra, …; may be empty + Country string +} + +// ReleaseGroup is what a person calls an album: one work, released any +// number of times. +type ReleaseGroup struct { + MBID string + Title string + PrimaryType string // Album, Single, EP, Broadcast, Other; may be empty + SecondaryTypes []string // Compilation, Live, Remix, Soundtrack, … + // FirstReleaseDate is MusicBrainz's partial date: "", "1997", + // "1997-05" or "1997-05-21". + FirstReleaseDate string + ArtistCredit string // the credit as printed, joinphrases included + ArtistMBIDs []string +} + +// Year is the first release's year, or "". +func (g ReleaseGroup) Year() string { + if len(g.FirstReleaseDate) >= 4 { + return g.FirstReleaseDate[:4] + } + return "" +} + +// Release is one issue of a release group: a pressing, an edition, a +// digital release, each with its own track list. +type Release struct { + MBID string + Title string + Status string // Official, Promotion, Bootleg, Pseudo-Release; may be empty + Date string // partial, like ReleaseGroup.FirstReleaseDate + Country string + Disambiguation string + Media []Medium +} + +// TrackCount totals the tracks across every medium. +func (r Release) TrackCount() int { + n := 0 + for _, m := range r.Media { + n += len(m.Tracks) + } + return n +} + +// Formats summarises the media, e.g. "CD", "2×CD" or "CD + DVD". +func (r Release) Formats() string { + var parts []string + counts := map[string]int{} + for _, m := range r.Media { + f := m.Format + if f == "" { + f = "Unknown" + } + if counts[f] == 0 { + parts = append(parts, f) + } + counts[f]++ + } + for i, f := range parts { + if counts[f] > 1 { + parts[i] = strconv.Itoa(counts[f]) + "×" + f + } + } + return strings.Join(parts, " + ") +} + +// Medium is one disc (or side, or file set) of a release. +type Medium struct { + Position int + Format string + Tracks []Track +} + +// Track is a position on a medium, pointing at the recording heard there. +type Track struct { + Position int + Number string // as printed: "1", "A1", … + Title string + LengthMs int // 0 when unknown + RecordingMBID string +} + +// Recording is a piece of audio, as a search returns it. +type Recording struct { + MBID string + Title string + LengthMs int + ArtistCredit string + ArtistMBIDs []string + // ReleaseGroup is the group the recording is best known from: the first + // official album it appears on, else the first release listed. + ReleaseGroup ReleaseGroup +} + +// --- wire shapes --- + +type wireCredit struct { + Name string `json:"name"` + JoinPhrase string `json:"joinphrase"` + Artist struct { + ID string `json:"id"` + Name string `json:"name"` + } `json:"artist"` +} + +func credit(cs []wireCredit) (string, []string) { + var b strings.Builder + ids := make([]string, 0, len(cs)) + for _, c := range cs { + name := c.Name + if name == "" { + name = c.Artist.Name + } + b.WriteString(name) + b.WriteString(c.JoinPhrase) + if c.Artist.ID != "" { + ids = append(ids, c.Artist.ID) + } + } + return b.String(), ids +} + +type wireArtist struct { + ID string `json:"id"` + Name string `json:"name"` + SortName string `json:"sort-name"` + Disambiguation string `json:"disambiguation"` + Type string `json:"type"` + Country string `json:"country"` +} + +func (w wireArtist) artist() Artist { + return Artist{MBID: w.ID, Name: w.Name, SortName: w.SortName, Disambiguation: w.Disambiguation, Type: w.Type, Country: w.Country} +} + +type wireReleaseGroup struct { + ID string `json:"id"` + Title string `json:"title"` + PrimaryType string `json:"primary-type"` + SecondaryTypes []string `json:"secondary-types"` + FirstReleaseDate string `json:"first-release-date"` + ArtistCredit []wireCredit `json:"artist-credit"` +} + +func (w wireReleaseGroup) group() ReleaseGroup { + name, ids := credit(w.ArtistCredit) + return ReleaseGroup{ + MBID: w.ID, Title: w.Title, PrimaryType: w.PrimaryType, + SecondaryTypes: append([]string{}, w.SecondaryTypes...), FirstReleaseDate: w.FirstReleaseDate, + ArtistCredit: name, ArtistMBIDs: ids, + } +} + +type wireTrack struct { + Position int `json:"position"` + Number string `json:"number"` + Title string `json:"title"` + Length *int `json:"length"` + Recording struct { + ID string `json:"id"` + Length *int `json:"length"` + } `json:"recording"` +} + +type wireRelease struct { + ID string `json:"id"` + Title string `json:"title"` + Status string `json:"status"` + Date string `json:"date"` + Country string `json:"country"` + Disambiguation string `json:"disambiguation"` + Media []struct { + Position int `json:"position"` + Format string `json:"format"` + Tracks []wireTrack `json:"tracks"` + } `json:"media"` + ReleaseGroup *wireReleaseGroup `json:"release-group"` +} + +func (w wireRelease) release() Release { + r := Release{MBID: w.ID, Title: w.Title, Status: w.Status, Date: w.Date, Country: w.Country, Disambiguation: w.Disambiguation} + for _, m := range w.Media { + med := Medium{Position: m.Position, Format: m.Format} + for _, t := range m.Tracks { + length := 0 + switch { + case t.Length != nil: + length = *t.Length + case t.Recording.Length != nil: + length = *t.Recording.Length + } + med.Tracks = append(med.Tracks, Track{ + Position: t.Position, Number: t.Number, Title: t.Title, + LengthMs: length, RecordingMBID: t.Recording.ID, + }) + } + r.Media = append(r.Media, med) + } + return r +} + +// --- searches --- + +// SearchArtists finds artists by name or alias. +func (c *Client) SearchArtists(ctx context.Context, text string, limit int) ([]Artist, error) { + var resp struct { + Artists []wireArtist `json:"artists"` + } + q := url.Values{"query": {fieldQuery(text, "artist", "alias")}, "limit": {searchLimit(limit)}} + if err := c.get(ctx, "/artist", q, &resp); err != nil { + return nil, err + } + out := make([]Artist, 0, len(resp.Artists)) + for _, a := range resp.Artists { + out = append(out, a.artist()) + } + return out, nil +} + +// SearchReleaseGroups finds albums by title or by artist, so "radiohead ok +// computer" and "ok computer" both find OK Computer. +func (c *Client) SearchReleaseGroups(ctx context.Context, text string, limit int) ([]ReleaseGroup, error) { + var resp struct { + ReleaseGroups []wireReleaseGroup `json:"release-groups"` + } + q := url.Values{"query": {fieldQuery(text, "releasegroup", "artist")}, "limit": {searchLimit(limit)}} + if err := c.get(ctx, "/release-group", q, &resp); err != nil { + return nil, err + } + out := make([]ReleaseGroup, 0, len(resp.ReleaseGroups)) + for _, g := range resp.ReleaseGroups { + out = append(out, g.group()) + } + return out, nil +} + +// SearchReleaseGroupsBy finds an artist's albums by title: both must +// match, where SearchReleaseGroups takes either. It is for naming an album +// already in hand, not for a person's search box. +func (c *Client) SearchReleaseGroupsBy(ctx context.Context, artist, title string, limit int) ([]ReleaseGroup, error) { + var resp struct { + ReleaseGroups []wireReleaseGroup `json:"release-groups"` + } + query := "releasegroup:(" + Escape(strings.TrimSpace(title)) + ") AND artist:(" + Escape(strings.TrimSpace(artist)) + ")" + q := url.Values{"query": {query}, "limit": {searchLimit(limit)}} + if err := c.get(ctx, "/release-group", q, &resp); err != nil { + return nil, err + } + out := make([]ReleaseGroup, 0, len(resp.ReleaseGroups)) + for _, g := range resp.ReleaseGroups { + out = append(out, g.group()) + } + return out, nil +} + +// SearchRecordings finds songs by title or by artist. +func (c *Client) SearchRecordings(ctx context.Context, text string, limit int) ([]Recording, error) { + var resp struct { + Recordings []struct { + ID string `json:"id"` + Title string `json:"title"` + Length *int `json:"length"` + ArtistCredit []wireCredit `json:"artist-credit"` + Releases []wireRelease `json:"releases"` + } `json:"recordings"` + } + q := url.Values{"query": {fieldQuery(text, "recording", "artist")}, "limit": {searchLimit(limit)}} + if err := c.get(ctx, "/recording", q, &resp); err != nil { + return nil, err + } + out := make([]Recording, 0, len(resp.Recordings)) + for _, r := range resp.Recordings { + name, ids := credit(r.ArtistCredit) + rec := Recording{MBID: r.ID, Title: r.Title, ArtistCredit: name, ArtistMBIDs: ids} + if r.Length != nil { + rec.LengthMs = *r.Length + } + rec.ReleaseGroup = bestKnownFrom(r.Releases) + out = append(out, rec) + } + return out, nil +} + +// bestKnownFrom picks the release group a recording should be filed under +// in a search result: the first official album, else the first release. +func bestKnownFrom(releases []wireRelease) ReleaseGroup { + var first *wireReleaseGroup + for _, r := range releases { + if r.ReleaseGroup == nil || r.ReleaseGroup.ID == "" { + continue + } + if first == nil { + first = r.ReleaseGroup + } + if r.Status == "Official" && r.ReleaseGroup.PrimaryType == "Album" && len(r.ReleaseGroup.SecondaryTypes) == 0 { + return r.ReleaseGroup.group() + } + } + if first == nil { + return ReleaseGroup{} + } + return first.group() +} + +// --- lookups and browses --- + +// Artist looks one artist up by MBID. +func (c *Client) Artist(ctx context.Context, mbid string) (Artist, error) { + var w wireArtist + if err := c.get(ctx, "/artist/"+url.PathEscape(mbid), nil, &w); err != nil { + return Artist{}, err + } + if w.ID == "" { + return Artist{}, ErrBadResponse + } + return w.artist(), nil +} + +// ArtistReleaseGroups lists every release group the artist is credited on, +// paging through MusicBrainz's browse. +func (c *Client) ArtistReleaseGroups(ctx context.Context, artistMBID string) ([]ReleaseGroup, error) { + var out []ReleaseGroup + for page := 0; page < maxPages; page++ { + var resp struct { + Count int `json:"release-group-count"` + ReleaseGroups []wireReleaseGroup `json:"release-groups"` + } + q := url.Values{ + "artist": {artistMBID}, "inc": {"artist-credits"}, + "limit": {strconv.Itoa(pageSize)}, "offset": {strconv.Itoa(len(out))}, + } + if err := c.get(ctx, "/release-group", q, &resp); err != nil { + return nil, err + } + for _, g := range resp.ReleaseGroups { + out = append(out, g.group()) + } + if len(resp.ReleaseGroups) == 0 || len(out) >= resp.Count { + break + } + } + return out, nil +} + +// ReleaseGroupReleases lists every release of a release group with its +// media and track lists. +func (c *Client) ReleaseGroupReleases(ctx context.Context, groupMBID string) ([]Release, error) { + var out []Release + for page := 0; page < maxPages; page++ { + var resp struct { + Count int `json:"release-count"` + Releases []wireRelease `json:"releases"` + } + q := url.Values{ + "release-group": {groupMBID}, "inc": {"media recordings"}, + "limit": {strconv.Itoa(pageSize)}, "offset": {strconv.Itoa(len(out))}, + } + if err := c.get(ctx, "/release", q, &resp); err != nil { + return nil, err + } + for _, r := range resp.Releases { + out = append(out, r.release()) + } + // Advance by what came back: MusicBrainz may return fewer than the + // page size when track lists are large. + if len(resp.Releases) == 0 || len(out) >= resp.Count { + break + } + } + return out, nil +} + +// ReleaseGroup looks one release group up by MBID, with its artist credit. +func (c *Client) ReleaseGroup(ctx context.Context, mbid string) (ReleaseGroup, error) { + var w wireReleaseGroup + q := url.Values{"inc": {"artist-credits"}} + if err := c.get(ctx, "/release-group/"+url.PathEscape(mbid), q, &w); err != nil { + return ReleaseGroup{}, err + } + if w.ID == "" { + return ReleaseGroup{}, ErrBadResponse + } + return w.group(), nil +} diff --git a/internal/musicbrainz/pace.go b/internal/musicbrainz/pace.go new file mode 100644 index 00000000..c000c75d --- /dev/null +++ b/internal/musicbrainz/pace.go @@ -0,0 +1,59 @@ +// Package musicbrainz is Minstrel's client for the MusicBrainz web service: +// searches and browses for the release catalogue, and the one +// pacer every MusicBrainz request in the process waits on. +package musicbrainz + +import ( + "context" + "sync" + "time" +) + +// MinInterval honours MusicBrainz's 1 request/second ceiling, with headroom. +// The limit is per client address, not per caller, which is why there is one +// pacer for the whole process. +const MinInterval = 1100 * time.Millisecond + +// Pacer spaces requests at least Interval apart. Safe for concurrent use; +// callers queue in the order they claim slots. +type Pacer struct { + Interval time.Duration + + mu sync.Mutex + next time.Time +} + +// Wait blocks until this caller's slot comes up, or ctx ends. A zero +// Interval never waits. +func (p *Pacer) Wait(ctx context.Context) error { + if p == nil || p.Interval <= 0 { + return ctx.Err() + } + p.mu.Lock() + now := time.Now() + slot := p.next + if slot.Before(now) { + slot = now + } + p.next = slot.Add(p.Interval) + p.mu.Unlock() + + wait := time.Until(slot) + if wait <= 0 { + return ctx.Err() + } + timer := time.NewTimer(wait) + defer timer.Stop() + select { + case <-ctx.Done(): + return ctx.Err() + case <-timer.C: + return nil + } +} + +var shared = &Pacer{Interval: MinInterval} + +// Shared is the pacer every MusicBrainz request in the process goes through: +// the catalogue's and tag enrichment's alike. +func Shared() *Pacer { return shared } diff --git a/internal/netsettings/service.go b/internal/netsettings/service.go index 49d23d1b..8fe4d31a 100644 --- a/internal/netsettings/service.go +++ b/internal/netsettings/service.go @@ -1,6 +1,6 @@ // Package netsettings holds the DB-backed network settings the request path -// needs. Today that's the trusted reverse-proxy depth used to pull a real -// client address out of X-Forwarded-For (#2453). +// needs: the trusted reverse-proxy depth used to pull a real client address +// out of X-Forwarded-For, and the public address links are built from. // // Values are cached under an RWMutex and refreshed on write. That isn't an // optimisation: auth.ClientIP runs in the RequireUser middleware for every @@ -53,7 +53,7 @@ type Service struct { // It ALWAYS returns a usable Service, even alongside a non-nil error. The // value it holds sits on the authenticated request path, so a boot-time // database hiccup must degrade to the default rather than take every request -// down with it (rule #131). The error is returned so the caller can log that +// down with it. The error is returned so the caller can log that // the cache holds a default rather than stored state. func New(ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger) (*Service, error) { s := &Service{pool: pool, logger: logger, hops: DefaultTrustedProxyHops} @@ -84,7 +84,7 @@ func (s *Service) Hops() int { } // SetHops persists a new depth and refreshes the cache, so an admin change -// takes effect on the next request with no restart (rule #25). +// takes effect on the next request with no restart. func (s *Service) SetHops(ctx context.Context, hops int) error { // Range first, availability second. The argument is wrong regardless of // whether the database is reachable, and the distinction is user-visible: @@ -115,7 +115,7 @@ func (s *Service) SetHops(ctx context.Context, hops int) error { return nil } -// PublicURL returns the operator-set address users reach Minstrel at, with no +// PublicURL returns the admin-set address users reach Minstrel at, with no // trailing slash, or "" when it hasn't been set. Links that leave the app (a // password-reset email) are built from this and never from the request's // Host header, which the requester controls. Nil-safe like Hops. diff --git a/internal/notifications/digest.go b/internal/notifications/digest.go index a1cd60c2..94b44211 100644 --- a/internal/notifications/digest.go +++ b/internal/notifications/digest.go @@ -15,10 +15,8 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/mailer" ) -// The email digest (#5346). Nothing is emailed per event. Operator, -// 2026-10-08: "music and stuff coming in should land as a summary email and -// not everytime same for a approvals the emails should be grouped by a -// reasonable amount of time like an hour or so." +// The email digest. Nothing is emailed per event: a steady stream of new +// music or approvals would otherwise be a stream of emails. // // - EmailSummary kinds (new music) go out at most once a day, at the summary // hour in the user's own timezone, listing everything since the last one. diff --git a/internal/notifications/digest_render.go b/internal/notifications/digest_render.go index 59213582..5c9ae146 100644 --- a/internal/notifications/digest_render.go +++ b/internal/notifications/digest_render.go @@ -48,7 +48,7 @@ type digestVars struct { // renderDigest renders one group's email. base is the public address; with // none set, the email still goes out, without links. func renderDigest(group EmailGroup, name string, items []pendingItem, base string) (digestEmail, error) { - vars := digestVars{Name: name, SettingsURL: joinURL(base, "/settings#notifications")} + vars := digestVars{Name: name, SettingsURL: joinURL(base, "/settings/notifications")} var subject string if group == EmailSummary { vars.Artists = summaryArtists(items, base) diff --git a/internal/notifications/digest_test.go b/internal/notifications/digest_test.go index b89539ff..60799a13 100644 --- a/internal/notifications/digest_test.go +++ b/internal/notifications/digest_test.go @@ -154,7 +154,7 @@ func TestRenderDigest_SummaryGroupsByArtist(t *testing.T) { require.Contains(t, e.Text, " - Pure\n https://music.example/albums/al-3") require.Contains(t, e.Text, "Boards of Canada\n https://music.example/artists/ar-2") require.Contains(t, e.Text, "Tycho\n - Awake") - require.Contains(t, e.Text, "https://music.example/settings#notifications") + require.Contains(t, e.Text, "https://music.example/settings/notifications") require.Contains(t, e.HTML, ` 1 { title = fmt.Sprintf("Library scan failed %d times", p.Count) } - body := p.Detail - if body == "" { - body = "The last scan stopped with an error." + body := "The last scan stopped with an error." + if p.Detail != "" { + body = "The last scan stopped: " + p.Detail } - return Rendered{Title: title, Body: body, Link: "/admin"} + return Rendered{Title: title, Body: body, Link: "/admin/library"} case KindTracksMissing: + body := "Their files are no longer where the library expects them." + if names := nameList(p.Items, p.Count); names != "" { + body = "No longer on disk: " + names + "." + } return Rendered{ Title: plural(p.Count, "track went missing", "tracks went missing"), - Body: "Their files are no longer where the library expects them.", - Link: "/admin/missing-files", + Body: body, + Link: "/collection/missing", } case KindDuplicatesFound: + body := "The duplicate sweep found tracks holding the same recording." + if names := nameList(p.Items, p.Count); names != "" { + body = "Held more than once: " + names + "." + } return Rendered{ - Title: plural(p.Count, "duplicate group to review", "duplicate groups to review"), - Body: "The duplicate sweep found tracks holding the same recording.", - Link: "/admin/duplicates", + Title: plural(p.Count, "new duplicate to review", "new duplicates to review"), + Body: body, + Link: "/collection/duplicates", } case KindDuplicatesResolved: - body := "Copies Lidarr does not need were merged into the one it tracks." - if p.Detail != "" { + body := "Extra copies of a song were merged into the one the album keeps." + switch names := nameList(p.Items, p.Count); { + case names != "": + body = "Extra copies removed, the right one kept: " + names + "." + case p.Detail != "": body = p.Detail } return Rendered{ - Title: plural(p.Count, "duplicate resolved automatically", "duplicates resolved automatically"), + Title: plural(p.Count, "duplicate merged automatically", "duplicates merged automatically"), Body: body, - Link: "/admin/duplicates?view=resolved", + Link: "/collection/duplicates?view=resolved", } case KindPlaybackErrors: + body := "A track failed to play for someone." + if names := nameList(p.Items, p.Count); names != "" { + body = "Failed to play: " + names + "." + } return Rendered{ Title: plural(p.Count, "playback error reported", "playback errors reported"), - Body: "A track failed to play for someone.", - Link: "/admin/playback-errors", + Body: body, + Link: "/collection/reports", } default: return Rendered{Title: "Notification", Link: "/"} } } -func plural(n int64, one, many string) string { - if n == 1 { - return "1 " + one +// namesShown is how many of a notice's items its body names. +const namesShown = 3 + +// nameList names the first few items, then how many more of the total there +// are: "A, B, C and 4 more". Empty when there are no items. +func nameList(items []string, total int64) string { + if len(items) == 0 { + return "" } + shown := items[:min(len(items), namesShown)] + more := max(total, int64(len(items))) - int64(len(shown)) + if more > 0 { + return strings.Join(shown, ", ") + " and " + words.Count(more, "more", "more") + } + if len(shown) == 1 { + return shown[0] + } + return strings.Join(shown[:len(shown)-1], ", ") + " and " + shown[len(shown)-1] +} + +func plural(n int64, one, many string) string { if n < 1 { // A coalesced kind always carries a count; this only guards a // malformed row from reading "0 tracks went missing". return "Some " + many } - return fmt.Sprintf("%d %s", n, many) + return words.Count(n, one, many) } diff --git a/internal/notifications/render_test.go b/internal/notifications/render_test.go index 6bd6444a..c8376e85 100644 --- a/internal/notifications/render_test.go +++ b/internal/notifications/render_test.go @@ -45,6 +45,36 @@ func TestRender_CoalescedCountsReadNaturally(t *testing.T) { require.Equal(t, "Some tracks went missing", Render(KindTracksMissing, []byte(`{}`)).Title) } +// A notice says which, not only how many. +func TestRender_NamesWhatChanged(t *testing.T) { + require.Equal(t, "No longer on disk: Weezer – Buddy Holly.", + render(KindTracksMissing, Payload{Count: 1, Items: []string{"Weezer – Buddy Holly"}}).Body) + require.Equal(t, "No longer on disk: A – 1 and B – 2.", + render(KindTracksMissing, Payload{Count: 2, Items: []string{"A – 1", "B – 2"}}).Body) + // Three are named; the rest of the count, not only the rest of the + // stored names, is "more". + require.Equal(t, "Held more than once: A, B, C and 9 more.", + render(KindDuplicatesFound, Payload{Count: 12, Items: []string{"A", "B", "C", "D", "E"}}).Body) + require.Equal(t, "12 new duplicates to review", render(KindDuplicatesFound, Payload{Count: 12}).Title) + require.Equal(t, "Extra copies removed, the right one kept: A and B.", + render(KindDuplicatesResolved, Payload{Count: 2, Items: []string{"A", "B"}}).Body) + require.Equal(t, "Failed to play: Weezer – Buddy Holly (alice).", + render(KindPlaybackErrors, Payload{Count: 1, Items: []string{"Weezer – Buddy Holly (alice)"}}).Body) +} + +func TestRender_RowsFromBeforeItemsReadAsTheyDid(t *testing.T) { + require.Equal(t, "Their files are no longer where the library expects them.", + render(KindTracksMissing, Payload{Count: 4}).Body) + require.Equal(t, "Merged 8 duplicate copies.", + render(KindDuplicatesResolved, Payload{Count: 716, Detail: "Merged 8 duplicate copies."}).Body) +} + +func TestRender_AScanFailureSaysWhatStopped(t *testing.T) { + r := render(KindScanFailed, Payload{Count: 1, Detail: "scan root \"/music\" unavailable"}) + require.Equal(t, `The last scan stopped: scan root "/music" unavailable`, r.Body) + require.Equal(t, "/admin/library", r.Link) +} + func TestRender_AnUnreadablePayloadStillRenders(t *testing.T) { r := Render(KindRequestApproved, []byte(`not json`)) require.Equal(t, "Request approved", r.Title) diff --git a/internal/playevents/writer.go b/internal/playevents/writer.go index f83fb099..5fbb9eb6 100644 --- a/internal/playevents/writer.go +++ b/internal/playevents/writer.go @@ -33,7 +33,7 @@ type Writer struct { } // NewWriter constructs a writer. timeout is the play-session window; the two -// remaining args are the skip rule thresholds from spec §6. +// remaining args are the skip rule's thresholds. func NewWriter( pool *pgxpool.Pool, logger *slog.Logger, @@ -78,23 +78,12 @@ func strPtrOrNil(s string) *string { } // systemPlaylistSources are the play_events.source values that count -// against a system playlist's rotation (Fable #415). Mirrors the -// `playlists_kind_variant_consistent` CHECK in migration -// 0028_discovery_mix_variants.up.sql — every variant in that CHECK -// list that ships as a refreshable system mix needs to be here so -// the rotation reporter sees Android + web plays from that surface. -// -// Drift #563: this map drifted behind the migrations. It had only -// for_you + discover but migrations 0021 + 0028 added 6 more -// variants. Plays from Rediscover / Deep Cuts / Songs Like X / -// New for You / On This Day / First Listens didn't advance the -// per-user rotation, so "unplayed first" ordering staled on those -// mixes — same tracks kept surfacing. -// -// Future variant additions should update this map AND the CHECK in -// the matching migration in the same change; the -// `db_check_constraint_for_new_variants` standing rule covers the -// CHECK half. +// against a system playlist's rotation. Every variant in the +// `playlists_kind_variant_consistent` CHECK that ships as a refreshable +// system mix must be here: a variant missing from this map never advances +// its per-user rotation, so "unplayed first" goes stale and the same tracks +// keep surfacing. A new variant updates this map and that CHECK (in its +// migration) in the same change. var systemPlaylistSources = map[string]bool{ "for_you": true, "discover": true, @@ -107,10 +96,10 @@ var systemPlaylistSources = map[string]bool{ } // lookupSystemPickKind resolves a system-playlist play's pick_kind -// (taste/fresh, Discover's bucket, or a rule-#131 tier) from the user's +// (taste/fresh, Discover's bucket, or an eligibility tier) from the user's // live snapshot of that variant at ingestion time. Attribution must be // frozen now — snapshots rebuild daily, so it can't be reconstructed at -// metrics-read time (#1249, generalized in #1270). Source strings +// metrics-read time. Source strings // double as system_variant values (see systemPlaylistSources). Returns // nil (unattributed) when the track isn't in the current snapshot — // plays predating the feature, or an offline replay arriving after the @@ -140,8 +129,8 @@ func lookupSystemPickKind( // tag identifying which surface the play came from. When source is a // known system-playlist kind the track is appended to that user's // rotation set so shuffle-on-play can prefer the unplayed tail on a -// re-play within the day. Empty source → behaves exactly like the -// pre-#415 path (NULL column, no rotation write). +// re-play within the day. An empty source behaves exactly like +// RecordPlayStarted (NULL column, no rotation write). func (w *Writer) RecordPlayStartedWithSource( ctx context.Context, userID, trackID pgtype.UUID, @@ -209,7 +198,7 @@ func (w *Writer) RecordPlayStartedWithSource( } // RecordPlayEnded closes a play_event with duration + computed completion -// ratio. Applies the spec §6 skip rule (AND of completion < threshold AND +// ratio. Applies the skip rule (AND of completion < threshold AND // duration_played < threshold) — both conditions must hold for the row to // be marked was_skipped=true. If skipped, also writes a skip_events row. func (w *Writer) RecordPlayEnded( @@ -232,7 +221,7 @@ func (w *Writer) RecordPlayEnded( // row is already closed, re-applying the close is harmless (the // UPDATE is last-write-wins) but a second skip_events insert must // NOT fire — a lost-response retry would otherwise double the - // skip signal. See the 2026-06-11 contract audit. + // skip signal. alreadyEnded := ev.EndedAt.Valid ratio := 0.0 if track.DurationMs > 0 { @@ -382,17 +371,15 @@ func (w *Writer) RecordSyntheticCompletedPlay( } // RecordOfflinePlay writes a complete start+end play in one call from -// a caller-supplied timestamp + duration. This is the replay path for -// the Flutter offline mutation queue (#426 part B): a play that -// happened with no/flaky connectivity is captured locally as a -// completed unit and replayed here once back online, preserving the -// original `at`. +// a caller-supplied timestamp + duration. This is the replay path for a +// client's offline queue: a play that happened with no or flaky +// connectivity is captured locally as a completed unit and replayed here +// once back online, preserving the original `at`. // // Generalizes RecordSyntheticCompletedPlay: instead of assuming full -// completion it applies the spec §6 skip rule from the real -// durationPlayed vs track length (same AND-of-thresholds as -// RecordPlayEnded), and threads `source` so #415 rotation advances -// for system-playlist plays just like the live path. +// completion it applies the skip rule from the real durationPlayed vs +// track length (same AND-of-thresholds as RecordPlayEnded), and threads +// `source` so system-playlist rotation advances just like the live path. // // durationPlayedMs is clamped to [0, track duration] to absorb // client clock / position drift. @@ -414,7 +401,7 @@ func (w *Writer) RecordOfflinePlay( // after commit makes the client re-fire the same play. The replayed // payload carries the original `at`, so an exact (user, track, // started_at) match means we already recorded it — no-op instead of a - // duplicate history row. See the 2026-06-11 contract audit. + // duplicate history row. var dupID pgtype.UUID err = tx.QueryRow(ctx, `SELECT id FROM play_events diff --git a/internal/playevents/writer_test.go b/internal/playevents/writer_test.go index 24893d50..49d7136f 100644 --- a/internal/playevents/writer_test.go +++ b/internal/playevents/writer_test.go @@ -220,8 +220,8 @@ func TestRecordPlayEnded_DurationOver50Percent_NotSkip(t *testing.T) { } // A lost-response retry of an offline play (same user/track/started_at) -// must not create a second history row. Guards the at-least-once replay -// path added in the 2026-06-11 audit. +// must not create a second history row: the offline replay is +// at-least-once. func TestRecordOfflinePlay_DedupsByUserTrackStartedAt(t *testing.T) { f := newFixture(t, 200_000) ctx := context.Background() @@ -491,7 +491,7 @@ func TestRecordPlayStarted_NoRotationWithoutSource(t *testing.T) { // seedSystemSnapshot creates a system playlist of the given variant // containing the fixture track with the given pick_kind, mimicking -// what the builder persists (#1249, generalized #1270). +// what the builder persists. func seedSystemSnapshot(t *testing.T, f fixture, variant, pickKind string) { t.Helper() pl, err := f.q.CreateSystemPlaylist(context.Background(), dbq.CreateSystemPlaylistParams{ @@ -567,7 +567,7 @@ func TestRecordPlayStartedWithSource_CrossVariant_PickKindNull(t *testing.T) { } func TestRecordPlayStartedWithSource_StampsDiscoverBucket(t *testing.T) { - // Discover stamps its candidate bucket (#1270); a discover play whose + // Discover stamps its candidate bucket; a discover play whose // track is in the live snapshot freezes that bucket onto the play. f := newFixture(t, 200_000) seedSystemSnapshot(t, f, "discover", "dormant") diff --git a/internal/playlists/discover.go b/internal/playlists/discover.go index e30d3c31..a33cc1bd 100644 --- a/internal/playlists/discover.go +++ b/internal/playlists/discover.go @@ -11,7 +11,7 @@ import ( const ( discoverTotalSlots = 100 - // Bucket allocation, pushed toward *targeted* novelty (Scribe #1488): + // Bucket allocation, pushed toward *targeted* novelty: // the taste-matched-unheard arm gets the plurality, dormant stays a // familiar anchor, and the crude random arm is trimmed to a // serendipity sliver. On a single-user server cross_user is empty and @@ -28,8 +28,8 @@ const ( // discoverTrack is the common shape used by the bucket allocator. The // three sqlc-generated row types collapse into this internal struct so // downstream functions don't need to be generic over them. PickKind is -// the originating bucket (taste_unheard/dormant/cross_user/random, -// #1270/#1488), stamped in the row adapters so it survives the +// the originating bucket (taste_unheard/dormant/cross_user/random), +// stamped in the row adapters so it survives the // interleave — bucket identity is what makes the slot allocation // measurable in the metrics. type discoverTrack struct { @@ -95,8 +95,8 @@ type discoverPools struct { } func loadDiscoverPools(ctx context.Context, q *dbq.Queries, logger *slog.Logger, userID pgtype.UUID, dateStr string) discoverPools { - // The caps go into the query too (#5356): applied only here, after its - // LIMIT, they left a heavily tagged artist or two owning the whole bucket. + // The caps go into the query too: applied only here, after its LIMIT, they + // would leave a heavily tagged artist or two owning the whole bucket. tasteRows, err := q.ListTasteUnheardTracksForDiscover(ctx, dbq.ListTasteUnheardTracksForDiscoverParams{ UserID: userID, DateSeed: dateStr, MaxPerAlbum: discoverMaxTracksPerAlbum, MaxPerArtist: discoverMaxTracksPerArtist, @@ -229,12 +229,10 @@ type bucketRequest struct { // peer's supply ran out mid-distribution and the residual needs // to roll to other peers in a later pass. // -// Bug history (May 2026): an earlier version recomputed -// `deficit = b.want - final[i]` every pass without subtracting -// previously-redistributed amounts, so a bucket with permanently -// 0 final and want=30 kept handing out 30 to peers each pass. The -// `redistributed` array below tracks per-source absorbed deficit so -// subsequent passes only move the residual. +// The `redistributed` array tracks how much of each source's deficit has +// been absorbed, so later passes move only the residual: recomputing +// `b.want - final[i]` each pass would hand out a permanently empty bucket's +// whole want again on every pass. func redistributeSlots(buckets []bucketRequest) []int { n := len(buckets) final := make([]int, n) diff --git a/internal/playlists/discover_taste_db_test.go b/internal/playlists/discover_taste_db_test.go index 81772972..a6084d61 100644 --- a/internal/playlists/discover_taste_db_test.go +++ b/internal/playlists/discover_taste_db_test.go @@ -29,11 +29,11 @@ func seedTasteTag(t *testing.T, pool *pgxpool.Pool, userID pgtype.UUID, tag stri } } -// TestListTasteUnheardTracksForDiscover_CapsBeforeLimit is the deploy's shape -// (#5356): one artist whose every track carries the whole taste profile -// outscores everything on summed weight. Capped only after the LIMIT, that -// artist took all 120 rows and the bucket shrank to its 3. Capped before, the -// artist keeps 3 and the single-tag artists below it fill the rest. +// TestListTasteUnheardTracksForDiscover_CapsBeforeLimit: one artist whose every +// track carries the whole taste profile outscores everything on summed +// weight. Capped only after the LIMIT, that artist would take all 120 rows +// and the bucket shrink to its 3. Capped before, the artist keeps 3 and the +// single-tag artists below it fill the rest. func TestListTasteUnheardTracksForDiscover_CapsBeforeLimit(t *testing.T) { pool := newPool(t) ctx := context.Background() diff --git a/internal/playlists/discover_test.go b/internal/playlists/discover_test.go index fc1d54a8..61583f8c 100644 --- a/internal/playlists/discover_test.go +++ b/internal/playlists/discover_test.go @@ -163,7 +163,7 @@ func TestInterleaveBuckets_DedupsAcrossBuckets(t *testing.T) { } func TestInterleaveBuckets_PreservesBucketPickKind(t *testing.T) { - // Bucket provenance (#1270) is stamped on discoverTrack before the + // Bucket provenance is stamped on discoverTrack before the // interleave and must survive it — including dedup, where a track in // two buckets keeps the stamp of the bucket it was taken from. dormant := []discoverTrack{ diff --git a/internal/playlists/foryou_test.go b/internal/playlists/foryou_test.go index 2a102f9e..398dccd5 100644 --- a/internal/playlists/foryou_test.go +++ b/internal/playlists/foryou_test.go @@ -246,7 +246,7 @@ func TestPickTopN_DiversityCap(t *testing.T) { func TestPickHeadAndTail_MarksPickKinds(t *testing.T) { // 100-deep pool with headN=20, tailN=5: the 20 head entries are the - // taste picks, the 5 tail entries the freshness injection (#1249). + // taste picks, the 5 tail entries the freshness injection. in := make([]recommendation.Candidate, 0, 100) for i := 0; i < 100; i++ { in = append(in, makeCand(i+1, i+1, i+1, float64(100-i))) @@ -323,7 +323,7 @@ func TestHeadQuotas(t *testing.T) { } func TestPickQuotaHead_BlendsSeeds(t *testing.T) { - // Seed 0's neighborhood out-scores everything (#1269): without + // Seed 0's neighborhood out-scores everything: without // quotas the head would be all seed-0. With headN=10 and quotas // [5,3,2], the walk takes the best 5 from seed 0, then seeds 1 and // 2 get their guaranteed slots from deeper ranks. @@ -385,7 +385,7 @@ func TestPickQuotaHead_ThinSeedSpillsBestFirst(t *testing.T) { } func TestPickWeightedTail_PrefersEarlyRanks(t *testing.T) { - // The fresh tail's sample is rank-weighted (#1269): across many + // The fresh tail's sample is rank-weighted: across many // days, the mean selected rank must sit well below the uniform // draw's ~99.5 for a 200-deep pool. pool := make([]recommendation.Candidate, 0, 200) diff --git a/internal/playlists/scheduler.go b/internal/playlists/scheduler.go index c68d2da7..ee23f7e1 100644 --- a/internal/playlists/scheduler.go +++ b/internal/playlists/scheduler.go @@ -1,5 +1,4 @@ -// Per-user system-playlist scheduler (#392 Half B). Replaces the -// 24h-anchored cron loop in system_cron.go (deleted in the same arc). +// Per-user system-playlist scheduler. // // Each active user gets one gocron daily job at 03:00 in their stored // timezone. The scheduler reconciles its in-memory job set with the @@ -34,7 +33,7 @@ import ( // dailyRebuildHour is the local-time hour at which each user's system // playlists rebuild. 03:00 is well past most "today" listening and -// before morning use; matches the operator framing in the design doc. +// before morning use. const dailyRebuildHour = 3 // reconciliationInterval drives the hourly pass that brings the @@ -48,7 +47,7 @@ type Scheduler struct { pool *pgxpool.Pool logger *slog.Logger dataDir string - bus *eventbus.Bus // #968: announce rebuilds so clients self-refresh + bus *eventbus.Bus // announces rebuilds so clients self-refresh mu sync.Mutex jobs map[pgtype.UUID]uuid.UUID // user_id → gocron job id @@ -222,9 +221,8 @@ func (s *Scheduler) reconcile(ctx context.Context) { } // runStartupCatchUp fires BuildSystemPlaylists once for every active -// user at startup, regardless of scheduled time. Matches the -// behavior of the previous system_cron.go startup runOnce so users -// whose scheduled fire was missed during downtime get caught up. +// user at startup, regardless of scheduled time, so users whose +// scheduled fire was missed during downtime get caught up. func (s *Scheduler) runStartupCatchUp(ctx context.Context, users []dbq.ListActiveUsersWithTimezonesRow) { for _, u := range users { s.rebuildUserDaily(ctx, u.ID, time.Now()) @@ -245,10 +243,10 @@ func (s *Scheduler) rebuildUserDaily(ctx context.Context, userID pgtype.UUID, no "user_id", uuidStringPL(userID), "err", err) return } - // #968: tell the user's connected clients their home content was - // regenerated, so a tab/app left open across the rebuild refreshes its - // system-playlist + You-might-like views (and a stale active queue can - // re-pull) instead of serving yesterday's snapshot until a manual reload. + // Tell the user's connected clients their home content was regenerated, so + // a tab/app left open across the rebuild refreshes its system-playlist + + // You-might-like views (and a stale active queue can re-pull) instead of + // serving yesterday's snapshot until a manual reload. s.publishRebuilt(userID) } diff --git a/internal/playlists/seed_missing_db_test.go b/internal/playlists/seed_missing_db_test.go index 25826b09..22cbaa09 100644 --- a/internal/playlists/seed_missing_db_test.go +++ b/internal/playlists/seed_missing_db_test.go @@ -28,7 +28,7 @@ func markMissing(t *testing.T, pool *pgxpool.Pool, trackID pgtype.UUID) { } // TestPickTopPlayedTracksForUser_LikedTierSkipsMissing: with no plays, For -// You seeds from likes. A liked track whose file is gone is not a seed (#2701). +// You seeds from likes. A liked track whose file is gone is not a seed. func TestPickTopPlayedTracksForUser_LikedTierSkipsMissing(t *testing.T) { pool := newPool(t) u := seedUser(t, pool, "seedmissing") diff --git a/internal/playlists/seed_selection_test.go b/internal/playlists/seed_selection_test.go index 14822422..9a95509f 100644 --- a/internal/playlists/seed_selection_test.go +++ b/internal/playlists/seed_selection_test.go @@ -48,9 +48,9 @@ func TestPickSeedArtistsFromRows_Empty(t *testing.T) { } func TestPickKindForSeedTier(t *testing.T) { - // The seed query's fallback tiers map onto the rule-#131 ladder: - // fresh 7-day engagement is the exact desire, everything past the - // 30-day step-back collapses into the far tier (#1255). + // The seed query's fallback tiers map onto the tier ladder: fresh + // 7-day engagement is the exact desire, everything past the 30-day + // step-back collapses into the far tier. cases := []struct { tier int32 want string @@ -99,7 +99,7 @@ func TestUserIDHash_DifferentUserChangesHash(t *testing.T) { // pickDailySeeds shuffles the candidate pool daily-deterministically // and takes up to n — For-You uses n=forYouSeedCount for its -// multi-seed blend (#1269), Songs-like uses n=songsLikeSeedCount via +// multi-seed blend, Songs-like uses n=songsLikeSeedCount via // the pickSeedArtistsForDay wrapper. Verifies determinism within a day, // variation across days, and graceful degradation on small pools. diff --git a/internal/playlists/service.go b/internal/playlists/service.go index b6ae36fe..8661525c 100644 --- a/internal/playlists/service.go +++ b/internal/playlists/service.go @@ -1,6 +1,6 @@ // Package playlists implements the user-facing playlist CRUD service. // -// Visibility model (from the M7 #352 design spec): +// Visibility model: // - Playlists are private by default. Only the owner sees private rows // in List, Get, and is allowed to mutate them via Update / Delete. // - is_public = true makes a playlist readable by any authenticated @@ -112,7 +112,7 @@ type PlaylistTrack struct { AddedAt pgtype.Timestamptz // Unavailable reports that the track still exists in the library but // its file is currently missing from disk (tracks.missing_since is - // set, #2527). The entry keeps its place in the playlist — the user + // set). The entry keeps its place in the playlist — the user // put it there — but nothing should try to play it. Unavailable bool } diff --git a/internal/playlists/service_test.go b/internal/playlists/service_test.go index 1b8f6ec2..bc46d150 100644 --- a/internal/playlists/service_test.go +++ b/internal/playlists/service_test.go @@ -459,7 +459,7 @@ func TestSnapshotPersistsAfterUpstreamTrackDelete(t *testing.T) { pl, _ := svc.Create(context.Background(), user.ID, "Test", "", false) _ = svc.AppendTracks(context.Background(), user.ID, pl.ID, []pgtype.UUID{tk.ID}) - // Delete the track row directly (simulating Lidarr re-import / file removal). + // Delete the track row directly (simulating a re-import / file removal). if _, err := pool.Exec(context.Background(), `DELETE FROM tracks WHERE id = $1`, tk.ID); err != nil { t.Fatalf("delete track: %v", err) } diff --git a/internal/playlists/service_test_helpers_test.go b/internal/playlists/service_test_helpers_test.go index f206cfd3..a2c57df6 100644 --- a/internal/playlists/service_test_helpers_test.go +++ b/internal/playlists/service_test_helpers_test.go @@ -51,7 +51,7 @@ func newPool(t *testing.T) *pgxpool.Pool { } // seedUser inserts a `test-`-prefixed user (so dbtest.ResetDB can -// clean it without touching the operator's admin row) and returns +// clean it without touching the developer's admin row) and returns // the persisted user record. func seedUser(t *testing.T, pool *pgxpool.Pool, name string) dbq.User { t.Helper() diff --git a/internal/playlists/song_copies_test.go b/internal/playlists/song_copies_test.go index 5488b1b2..afdfbffd 100644 --- a/internal/playlists/song_copies_test.go +++ b/internal/playlists/song_copies_test.go @@ -10,9 +10,9 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/playlists" ) -// A mix holds a song once (M498 #5438) and never a held-back video rip -// (#5439): of a single and its album cut, the better-ranked copy stays; a rip -// is left out and the next copy of its song takes the place. +// A mix holds a song once and never a held-back video rip: of a single and its +// album cut, the better-ranked copy stays; a rip is left out and the next copy +// of its song takes the place. func TestOneCopyPerSong_Integration(t *testing.T) { pool := newPool(t) ctx := context.Background() diff --git a/internal/playlists/system.go b/internal/playlists/system.go index 807d38f8..e6fe1940 100644 --- a/internal/playlists/system.go +++ b/internal/playlists/system.go @@ -1,5 +1,5 @@ // Package playlists' system.go implements the system-generated mix -// builder (M7 #352 slice 2). The cron loop and lazy fallback both call +// builder. The daily scheduler and the on-demand refresh both call // BuildSystemPlaylists; the helpers in this file (pickSeedArtists, // pickRepresentativeTrack, tieBreakHash) are the pure parts. package playlists @@ -52,15 +52,10 @@ func pickSeedArtistsFromRows(rows []seedArtistRow) []pgtype.UUID { // daily-deterministic ordering. Same inputs always produce the same // output; different inputs almost always differ. // -// Switched from FNV-1a (May 2026) because two date strings differing -// only in the last character (e.g. "2026-05-07" vs "2026-05-08") gave -// hash values whose RELATIVE ORDERING across our small candidate pools -// was identical — the dateStr-driven divergence concentrated in low -// bits that lost out to high-bit ordering during sort. SHA-256 -// truncated to 8 bytes has full avalanche, so any single-byte input -// change roughly half-flips the output bits and reorders meaningfully. -// The hash isn't security-load-bearing; we just want strong avalanche -// for tiny input deltas. +// SHA-256 truncated to 8 bytes, not FNV-1a: two date strings differing only +// in the last character must reorder the pool, and FNV's divergence for such +// inputs sits in low bits that lose to high-bit ordering during the sort. +// SHA-256 has full avalanche. The hash isn't security-load-bearing. func tieBreakHash(trackID pgtype.UUID, dateStr string) uint64 { h := sha256.New() if trackID.Valid { @@ -111,12 +106,9 @@ func pickDailySeeds(pool []pgtype.UUID, userID pgtype.UUID, dateStr string, n in } // songsLikeSeedCount is how many "Songs like {artist}" mixes to build -// each day (#1491). Bumped from 3 when Songs-like was promoted to its -// own dedicated Home row — the best-performing surface (low skip / high -// completion) was buried as 3 tiles in the shared Playlists carousel, so -// the wider row now shows a wider spread. Drawn from PickSeedArtists' -// deeper top-12 pool via a daily-deterministic shuffle so the set still -// rotates day to day rather than pinning the same six artists. +// each day, enough to fill Songs-like's own Home row. Drawn from +// PickSeedArtists' deeper top-12 pool via a daily-deterministic shuffle so +// the set still rotates day to day rather than pinning the same six artists. const songsLikeSeedCount = 6 // pickSeedArtistsForDay picks up to songsLikeSeedCount seed artists for @@ -127,10 +119,10 @@ func pickSeedArtistsForDay(pool []pgtype.UUID, userID pgtype.UUID, dateStr strin } // rankedCandidate is a (track_id, score) pair used during in-memory -// sorting before insert into playlist_tracks. T5 fills these from +// sorting before insert into playlist_tracks, filled from // recommendation.Candidate scores. PickKind is the track's provenance -// within its mix — For-You's head/tail split (#1249) and Discover's -// buckets (#1270); empty persists as NULL (variant doesn't stamp yet). +// within its mix — For-You's head/tail split and Discover's buckets; +// empty persists as NULL (a variant that doesn't stamp). type rankedCandidate struct { TrackID pgtype.UUID Score float64 @@ -140,8 +132,8 @@ type rankedCandidate struct { // Pick kinds, persisted on playlist_tracks.pick_kind so plays can // attribute skips to the population that sourced the track — For You's // "taste engine missing" vs "freshness tax", Discover's three buckets -// (#1270). Values mirror the CHECK in migration 0041; taste_unheard is -// Discover's taste-targeted-novelty arm (#1488). +// Values mirror the CHECK in migration 0041; taste_unheard is Discover's +// taste-targeted-novelty arm. const ( pickKindTaste = "taste" pickKindFresh = "fresh" @@ -155,7 +147,7 @@ const ( ) // pickKindForSeedTier maps PickSeedArtists' seed-pool fallback tier -// onto the rule-#131 pick-kind ladder: fresh 7-day engagement pins the +// onto the tiered pick-kind ladder: fresh 7-day engagement pins the // mix's exact desire (tier1), the 30-day window steps back a little // (tier2), and all-time / liked-only seeds are the far fallback // (tier3). Stamped mix-wide — seed staleness is a property of the @@ -172,7 +164,7 @@ func pickKindForSeedTier(tier int32) string { } // pickKindForMixTier maps a tiered mix query's 1-based tier column -// (numbered to match rule #131's ladder directly) onto the pick-kind +// (numbered to match the tier ladder directly) onto the pick-kind // vocabulary. Distinct from pickKindForSeedTier, whose 0-based tiers // count fallback steps of the seed pool rather than eligibility rungs. func pickKindForMixTier(tier int32) string { @@ -194,12 +186,11 @@ const systemMixLength = 25 // scores within a day, but near-tied candidates reshuffle across days // so the playlist doesn't feel frozen. // -// DB-tunable since #1250: the recsettings service pushes the current -// daily_mix profile via SetSystemMixWeights at boot and on every admin -// change (coverart Configure() pattern — no signature threading, live -// effect without restart). The literal here is only the pre-push -// value; shipped defaults live in recsettings.ShippedDailyMixWeights, -// which must stay in sync with it. +// DB-tunable: the recsettings service pushes the current daily_mix profile via +// SetSystemMixWeights at boot and on every admin change (coverart Configure() +// pattern — no signature threading, live effect without restart). The literal +// here is only the pre-push value; shipped defaults live in +// recsettings.ShippedDailyMixWeights, which must stay in sync with it. var ( systemTuningMu sync.RWMutex systemMixWeights = recommendation.ScoringWeights{ @@ -210,16 +201,16 @@ var ( JitterMagnitude: 0.1, ContextWeight: 0.5, SimilarityWeight: 1.5, - // Taste profile (#796 phase 2): the daily mixes are the primary + // Taste profile: the daily mixes are the primary // taste-driven surface, so they lean on it. TasteMatchScore is in // [-1,+1], so 1.5 makes a strong taste fit comparable to a like boost // while passive avoidance (negative) gently demotes. TasteWeight: 1.5, - // Time-of-day/weekday context affinity (#1531), in [-1,+1]. Starts - // uniform with radio pending trend data. + // Time-of-day/weekday context affinity, in [-1,+1]. The same as + // radio's until trend data says otherwise. ContextTimeWeight: 1.0, } - // Songs-like's own profile (#3881). Pre-push literal only; shipped + // Songs-like's own profile. Pre-push literal only; shipped // defaults live in recsettings.ShippedSongsLikeWeights and must stay in // sync with it, exactly as systemMixWeights does above. // @@ -239,7 +230,7 @@ var ( } systemTasteConfig = taste.DefaultConfig() - // Sizes the candidate pool to the library (#3880). Cached because the + // Sizes the candidate pool to the library. Cached because the // count is a full table scan and the daily build runs it once per user; // within the TTL every user in a build shares one count. systemLibrarySize = recommendation.NewLibrarySize(nil) @@ -253,13 +244,13 @@ func SetSystemMixWeights(w recommendation.ScoringWeights) { systemMixWeights = w } -// SetSongsLikeWeights installs the songs_like scoring profile (#3881). +// SetSongsLikeWeights installs the songs_like scoring profile. // Same push model as SetSystemMixWeights — recsettings calls it on boot and // after every knob turn, so a tuning change takes effect on the next daily // build with no restart. // // Separate from systemMixWeights because Songs-like and For-You want opposite -// things: For-You roams, Songs-like must not. Sharing one profile is what made +// things: For-You roams, Songs-like must not. Sharing one profile makes // "Songs like X" wander. func SetSongsLikeWeights(w recommendation.ScoringWeights) { systemTuningMu.Lock() @@ -268,7 +259,7 @@ func SetSongsLikeWeights(w recommendation.ScoringWeights) { } // SetTasteConfig installs the taste-profile build configuration -// (half-life + engagement curve, #1250). Same push model as +// (half-life + engagement curve). Same push model as // SetSystemMixWeights. func SetTasteConfig(c taste.Config) { systemTuningMu.Lock() @@ -276,13 +267,11 @@ func SetTasteConfig(c taste.Config) { systemTasteConfig = c } -// dailyOrderSeed is the value the randomised candidate arms order by (#3889). +// dailyOrderSeed is the value the randomised candidate arms order by. // -// Per (user, day) so a same-day rebuild draws the SAME set — which is what -// TestBuildSystemPlaylists_DailyNonceDeterminism asserts and what those arms -// only ever achieved by accident before, when their limits happened to exceed -// the eligible rows. It changes on the day boundary, so the mixes still move -// daily. +// Per (user, day) so a same-day rebuild draws the SAME set, which is what +// TestBuildSystemPlaylists_DailyNonceDeterminism asserts. It changes on the +// day boundary, so the mixes still move daily. func dailyOrderSeed(userID pgtype.UUID, dateStr string) string { return uuidStringPL(userID) + ":" + dateStr } @@ -326,7 +315,7 @@ const ( ) // forYouSeedCount is how many of the user's top-5 tracks seed each -// day's build (#1269). A single rotating seed made the mix bipolar +// day's build. A single rotating seed would make the mix bipolar // day-to-day — swinging whole neighborhoods as the rotation moved // between dissimilar anchors; blending 3 spans neighborhoods within // one day and smooths the swing across days. @@ -334,7 +323,7 @@ const forYouSeedCount = 3 // headQuotas fixes each seed's share of the For-You head so one // neighborhood can't monopolize it even when its candidates out-score -// the others (#1269): 50/30/20 for three seeds, 60/40 for two, all of +// the others: 50/30/20 for three seeds, 60/40 for two, all of // it for one. Rounding remainder goes to the primary seed; a thin // seed's unfilled quota spills best-first in pickQuotaHead. func headQuotas(numSeeds, headN int) []int { @@ -401,7 +390,7 @@ func pickQuotaHead(capped []recommendation.Candidate, seedOf map[pgtype.UUID]int return head } -// forYouTailHalfLifeRanks tunes the fresh tail's rank bias (#1269): a +// forYouTailHalfLifeRanks tunes the fresh tail's rank bias: a // candidate's sampling weight halves every 50 ranks, so the 101st-best // candidate is far likelier than the 401st-best instead of equal — // freshness keeps its "you'll probably enjoy this" half. @@ -450,8 +439,8 @@ func pickWeightedTail(tailPool []recommendation.Candidate, dateStr string, tailN // because this sort IS the selection: the caller caps and truncates in the // order this returns, so whatever profile ranks here decides which tracks // reach the playlist. Scoring with daily_mix here and re-scoring with -// songs_like afterwards would have let the new profile relabel tracks it had -// no part in choosing — inert where it matters (#3881). +// songs_like afterwards would let the profile relabel tracks it had no part +// in choosing — inert where it matters. func scoreAndSortCandidates(cands []recommendation.Candidate, userID pgtype.UUID, dateStr string, now time.Time, weights recommendation.ScoringWeights) []recommendation.Candidate { rng := rand.New(rand.NewSource(int64(userIDHash(userID, dateStr)))) type scored struct { @@ -603,9 +592,9 @@ func systemForYouSourceLimits() recommendation.CandidateSourceLimits { RandomFill: 150, // For-You / You-might-like are the taste-driven surfaces, so pull a // deep slice of the user's top taste-profile artists into the pool - // (#796 phase 2b). Empty for cold-start users (no profile yet). + // Empty for cold-start users (no profile yet). TasteOverlap: 80, - // Household co-play (#1533): a deeper collaborative slice for the + // Household co-play: a deeper collaborative slice for the // taste surfaces. Empty on single-user servers. UserCoplay: 40, } @@ -613,7 +602,7 @@ func systemForYouSourceLimits() recommendation.CandidateSourceLimits { // produceForYou: blend candidate pools from up to forYouSeedCount of // the user's top-5 played tracks (rotating daily via pickDailySeeds), -// then head+tail composition with per-seed head quotas (#1269). The +// then head+tail composition with per-seed head quotas. The // base seed query failing is fatal; a per-seed candidate-load failure // is logged and that seed just contributes nothing. func produceForYou( @@ -667,7 +656,7 @@ func produceForYou( // produceSeedMixes: up to songsLikeSeedCount "Songs like {artist}" // mixes. Seed artists rotate daily-deterministically; the seed query falls back -// through widening engagement windows (#1255) and every returned row +// through widening engagement windows and every returned row // shares the winning tier, stamped onto the built tracks as their // pick_kind. The base seed-artist query failing is fatal; per-artist // failures are logged + skipped. @@ -714,12 +703,12 @@ func produceSeedMixes( continue } zeroVec := recommendation.SessionVector{Seed: true} - // Songs-like's own pool shape, not the default (#3881). Same total - // size; the composition shifts toward arms that actually measure - // distance from the seed. The default gave ~29% of candidates a - // sim_score of literally 0. + // Songs-like's own pool shape, not the default. Same total size; the + // composition shifts toward arms that actually measure distance from + // the seed, where the default gives ~29% of candidates a sim_score of + // 0. // - // Then scaled to the library (#3880): a bigger collection should put + // Then scaled to the library: a bigger collection should put // more genuinely-similar candidates in reach, not the same ~170 // regardless. The weights still rank sim_score-0 arms last, so the // growth reaches coherence rather than working against it. @@ -737,16 +726,11 @@ func produceSeedMixes( } // The seed artist's own songs are ELIGIBLE here, deliberately. // - // This used to filter them out — "Songs like X excludes X's own - // songs" — which reads as obviously right and is not. The seed is a - // TRACK, and the tracks most likely to sound like it are usually the - // rest of that artist's catalogue; excluding them threw away the - // nearest neighbours of the very thing the mix is built around, and - // then reached further out to replace them. On a surface whose whole - // job is staying in one neighbourhood, that is backwards. - // - // Operator, 2026-09-10: "it should also be able to include music from - // the same artist." + // "Songs like X excludes X's own songs" reads as obviously right and + // is not. The seed is a TRACK, and the tracks most likely to sound + // like it are usually the rest of that artist's catalogue; excluding + // them would throw away the nearest neighbours of the very thing the + // mix is built around, and reach further out to replace them. // // Domination is bounded by the cap rather than by exclusion, which is // the distinction that makes this safe: capCandidatesByAlbumAndArtist @@ -791,12 +775,12 @@ func produceDiscover( return []builtPlaylist{{Name: "Discover", Variant: "discover", Tracks: tracks}}, nil } -// BuildSystemPlaylists builds the user's daily system mixes (one For-You + -// up to songsLikeSeedCount Songs-like-{seed} mixes). Atomic-replace inside one tx; -// concurrency-guarded via system_playlist_runs.in_flight; deterministic -// within a day via tieBreakHash(track_id, now.UTC().Format("2006-01-02")). +// BuildSystemPlaylists builds the user's daily system mixes (one For-You + up +// to songsLikeSeedCount Songs-like-{seed} mixes). Atomic-replace inside one tx; +// concurrency-guarded via system_playlist_runs.in_flight; deterministic within +// a day via tieBreakHash(track_id, now.UTC().Format("2006-01-02")). // -// Callable from both the cron loop (system_cron.go) and the lazy fallback. +// Called by the daily scheduler and by the on-demand refresh endpoints. // Returns nil when the user has no library activity to build from. Errors // are captured in system_playlist_runs.last_error before returning. func BuildSystemPlaylists(ctx context.Context, pool *pgxpool.Pool, logger *slog.Logger, userID pgtype.UUID, now time.Time, dataDir string) error { @@ -929,13 +913,13 @@ func BuildSystemPlaylists(ctx context.Context, pool *pgxpool.Pool, logger *slog. // truncates to n. Used by Songs-like-X (and as the fallback inside // pickHeadAndTail for small pools). func pickTopN(cands []recommendation.Candidate, userID pgtype.UUID, dateStr string, now time.Time, n int) []rankedCandidate { - // songs_like, not daily_mix (#3881). produceSeedMixes is this function's - // only caller, so the switch moves exactly one surface — For-You ranks - // through pickHeadAndTail and keeps the broader daily_mix profile. + // songs_like, not daily_mix. produceSeedMixes is this function's only + // caller, so this moves exactly one surface — For-You ranks through + // pickHeadAndTail and keeps the broader daily_mix profile. // // The SAME profile does the selection sort and the final score. Passing - // one and using the other is the subtle version of this bug: the playlist - // would still be chosen by daily_mix and merely wear songs_like numbers. + // one and using the other would leave the playlist chosen by daily_mix + // and merely wearing songs_like numbers. weights := currentSongsLikeWeights() sorted := scoreAndSortCandidates(cands, userID, dateStr, now, weights) capped := capCandidatesByAlbumAndArtist(sorted) @@ -955,7 +939,7 @@ func pickTopN(cands []recommendation.Candidate, userID pgtype.UUID, dateStr stri // pickHeadAndTail picks headN taste anchors from the score-sorted pool // (under per-seed quotas when seedOf/numSeeds describe a multi-seed -// blend, #1269) plus tailN freshness picks sampled rank-weighted from +// blend) plus tailN freshness picks sampled rank-weighted from // positions 2*headN onward. Caps applied before the head/tail split. // Used by For-You only. // @@ -1022,7 +1006,7 @@ func pickHeadAndTail( // Combine: head first (score-sorted under quotas), then the fresh // sample. "First similar, then surprise" reads naturally in // playback. Head entries are the taste picks; tail entries are the - // freshness injection (#1249) — the split the metrics page + // freshness injection — the split the metrics page // attributes skips to. combined := make([]recommendation.Candidate, 0, len(head)+len(tail)) combined = append(combined, head...) @@ -1044,7 +1028,7 @@ func pickHeadAndTail( } // insertSystemPlaylist inserts a kind='system' playlists row plus its -// playlist_tracks via AppendPlaylistTrack (slice 1's existing query, which +// playlist_tracks via AppendPlaylistTrack (which // does the snapshot lookup from tracks/albums/artists). Then recomputes // rollups. Returns the new playlist's ID so the caller can drive // post-commit collage generation. @@ -1101,8 +1085,8 @@ func insertSystemPlaylist(ctx context.Context, qtx *dbq.Queries, userID pgtype.U } // oneCopyPerSong keeps the first of a song's copies on different releases -// (M498 #5438): a mix holds the song once, in its best-ranked place. A video -// rip held back from the mixes (#5439) is left out, and a clean copy of the +// a mix holds the song once, in its best-ranked place. A video rip held back +// from the mixes is left out, and a clean copy of the // same song can take its place. func oneCopyPerSong(ctx context.Context, qtx *dbq.Queries, tracks []rankedCandidate) ([]rankedCandidate, error) { ids := make([]pgtype.UUID, len(tracks)) diff --git a/internal/playlists/system_mixes.go b/internal/playlists/system_mixes.go index 2ca6ef78..f602ce00 100644 --- a/internal/playlists/system_mixes.go +++ b/internal/playlists/system_mixes.go @@ -11,10 +11,9 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Discovery mixes (#419-423). One generic producer + per-mix spec. -// Replaces the five near-identical produceXxx functions that all -// fetched ranked (id, album_id, artist_id) rows, diversified, -// truncated, and emitted a single playlist. +// Discovery mixes: one generic producer + a per-mix spec. Each mix fetches +// ranked (id, album_id, artist_id) rows, diversifies, truncates, and emits a +// single playlist. // // Day-keying is a per-mix property captured by `dailyRotate`: // @@ -40,7 +39,7 @@ import ( // Discover so shuffle-on-play has a varied pool within a day. const discoveryMixLen = 100 -// Minimum viable mix sizes (issue #1246): below the floor the variant +// Minimum viable mix sizes: below the floor the variant // is withheld entirely so Home renders its "listen more to unlock" // placeholder — a playlist with a couple of songs reads as a build // bug, not a mix. Album-coherent mixes (NewForYou / FirstListens) get @@ -68,7 +67,7 @@ type discoveryMixSpec struct { // minLen is the minimum viable mix size: a finished pool below it // is withheld (no playlist row) so the client renders the locked - // placeholder instead of a mix that looks built-wrong (#1246). + // placeholder instead of a mix that looks built-wrong. minLen int // fetch returns the raw ranked rows. dateStr is supplied for @@ -109,10 +108,10 @@ func produceDiscoveryMix(spec discoveryMixSpec) systemPlaylistProducer { // so each day's downstream truncate-to-N surfaces a different slice // while contiguous-block ordering inside the slice is preserved. // -// Rotation happens WITHIN each contiguous same-pick-kind block -// (#1267): tiered mixes arrive tier-ordered, and a whole-pool +// Rotation happens WITHIN each contiguous same-pick-kind block: +// tiered mixes arrive tier-ordered, and a whole-pool // rotation would hoist tier-3 filler above tier-1's exact fits. -// Untiered pools are a single block, which reduces to the original +// Untiered pools are a single block, which reduces to a plain // whole-pool rotation (same seed, same first draw). Empty / // single-element pools pass through unchanged. func rotateForDay(pool []discoverTrack, userID pgtype.UUID, dateStr string) []discoverTrack { diff --git a/internal/playlists/system_mixes_db_test.go b/internal/playlists/system_mixes_db_test.go index a2a65c09..556150f1 100644 --- a/internal/playlists/system_mixes_db_test.go +++ b/internal/playlists/system_mixes_db_test.go @@ -106,8 +106,8 @@ func likeArtist(t *testing.T, pool *pgxpool.Pool, userID, artistID pgtype.UUID) } func TestListDeepCutsTracks_SkippedTracksExcluded(t *testing.T) { - // #1257: eligibility used to count only unskipped plays, so a track - // skipped repeatedly (and never finished) read as "barely heard". + // Eligibility counts skipped plays too, so a track skipped repeatedly + // (and never finished) does not read as "barely heard". pool := newPool(t) q := dbq.New(pool) u := seedUser(t, pool, "dc1") @@ -150,9 +150,9 @@ func TestListDeepCutsTracks_SkippedTracksExcluded(t *testing.T) { } func TestListOnThisDayTracks_WrapsYearBoundary(t *testing.T) { - // #1256: plain ABS on day-of-year made Dec 28 vs Jan 3 read as 359 - // days apart. The build date parameter anchors "today" so the wrap - // is testable at any run date. + // The day-of-year window wraps the year: plain ABS would make Dec 28 vs Jan + // 3 read as 359 days apart. The build date parameter anchors "today" so the + // wrap is testable at any run date. pool := newPool(t) q := dbq.New(pool) u := seedUser(t, pool, "otd1") @@ -182,7 +182,7 @@ func TestListOnThisDayTracks_WrapsYearBoundary(t *testing.T) { } func TestListNewForYouTracks_TiersAndConsumption(t *testing.T) { - // #1267: tiered eligibility with consume-on-play. Tier 1 = fresh + // Tiered eligibility with consume-on-play. Tier 1 = fresh // affinity albums; tier 2 = affinity albums from the wider 90-day // window; tier 3 = fresh additions without affinity; any attempted // (>=30s) track consumes the album out of the mix entirely. @@ -240,8 +240,8 @@ func TestListNewForYouTracks_TiersAndConsumption(t *testing.T) { } func TestListFirstListensTracks_AttemptThreshold(t *testing.T) { - // #1268: "attempted" is track-level at >=30s. A 2-second brush no - // longer disqualifies an album; barely-attempted albums step back + // "Attempted" is track-level at >=30s. A 2-second brush does not + // disqualify an album; barely-attempted albums step back // to tier 2 with the attempted tracks themselves excluded. pool := newPool(t) q := dbq.New(pool) @@ -297,7 +297,7 @@ func TestListFirstListensTracks_AttemptThreshold(t *testing.T) { } func TestListRediscoverTracks_SkippedResurfacingCoolsDown(t *testing.T) { - // #1258: a skip on a rediscover-sourced play is a declined + // A skip on a rediscover-sourced play is a declined // invitation — the track sits out ~90 days instead of re-qualifying // the next day. Skips from other surfaces don't cool down, and the // cooldown expires. diff --git a/internal/playlists/system_mixes_test.go b/internal/playlists/system_mixes_test.go index 64e863d0..bbb2da08 100644 --- a/internal/playlists/system_mixes_test.go +++ b/internal/playlists/system_mixes_test.go @@ -21,7 +21,7 @@ func idsOf(pool []discoverTrack) []byte { func TestRotateForDay_RotatesWithinTierBlocks(t *testing.T) { // Tiered pools arrive tier-ordered; rotation must vary the daily // slice WITHIN each tier without hoisting tier-3 filler above - // tier-1's exact fits (#1267). + // tier-1's exact fits. pool := []discoverTrack{ mixTrack(1, pickKindTier1), mixTrack(2, pickKindTier1), mixTrack(3, pickKindTier1), mixTrack(4, pickKindTier2), mixTrack(5, pickKindTier2), @@ -80,7 +80,7 @@ func TestRotateForDay_UntieredPoolIsWholeRotation(t *testing.T) { func TestFinishMix_PropagatesPickKind(t *testing.T) { // Tier stamps ride discoverTrack through diversify/truncate into - // the rankedCandidates that insertSystemPlaylist persists (#1267). + // the rankedCandidates that insertSystemPlaylist persists. pool := []discoverTrack{mixTrack(1, pickKindTier1), mixTrack(2, pickKindTier2)} got := finishMix(pool, false) if len(got) != 2 { diff --git a/internal/playlists/system_test.go b/internal/playlists/system_test.go index 48a96719..e5a0719e 100644 --- a/internal/playlists/system_test.go +++ b/internal/playlists/system_test.go @@ -42,11 +42,11 @@ func seedPlayEvent(t *testing.T, pool *pgxpool.Pool, userID, trackID pgtype.UUID } } -// seedQuarantine inserts a lidarr_quarantine row for (user, track). +// seedQuarantine inserts a quarantine row for (user, track). func seedQuarantine(t *testing.T, pool *pgxpool.Pool, userID, trackID pgtype.UUID) { t.Helper() _, err := pool.Exec(context.Background(), ` - INSERT INTO lidarr_quarantine (user_id, track_id, reason) + INSERT INTO quarantine (user_id, track_id, reason) VALUES ($1, $2, 'other') `, userID, trackID) if err != nil { @@ -115,7 +115,7 @@ func TestBuildSystemPlaylists_SufficientActivity(t *testing.T) { t.Errorf("songs_like_artist row should have non-NULL seed_artist_id") } case "discover", "deep_cuts", "rediscover", "new_for_you", "on_this_day", "first_listens": - // Discover + the #411 discovery mixes are all seedless — + // Discover and the discovery mixes are all seedless — // no seed_artist_id required. default: t.Errorf("unknown system_variant=%q", *r.SystemVariant) @@ -124,7 +124,7 @@ func TestBuildSystemPlaylists_SufficientActivity(t *testing.T) { t.Errorf("row %s: empty track_count", uuidString(r.ID)) } // For-You / Discover / the discovery mixes are sized to ~100 - // (#352/#411); songs_like_artist stays at systemMixLength (25). + // songs_like_artist stays at systemMixLength (25). if r.TrackCount > 100 { t.Errorf("row %s: track_count=%d exceeds 100", uuidString(r.ID), r.TrackCount) } @@ -138,10 +138,10 @@ func TestBuildSystemPlaylists_SufficientActivity(t *testing.T) { } func TestBuildSystemPlaylists_StaleActivity_SongsLikeSurvives(t *testing.T) { - // #1255: the seed-artist query's old hard 7-day window emptied the - // pool after a quiet week, and the atomic-replace build then deleted - // every "Songs like X" mix. With the tiered fallback, plays that are - // ~20 days old (outside 7d, inside 30d) must still seed the mixes — + // A hard 7-day seed window would empty the pool after a quiet week, and + // the atomic-replace build would then delete every "Songs like X" mix. + // With the tiered fallback, plays that are ~20 days old (outside 7d, + // inside 30d) must still seed the mixes — // and the built tracks carry the tier2 stamp so metrics can compare // stale-seeded mixes against fresh ones. pool := newPool(t) @@ -352,16 +352,12 @@ func seedSharedArtistLibrary( } // The seed artist's own tracks are ELIGIBLE for its "Songs like" mix, and are -// bounded by the diversity cap rather than excluded outright (#3881). -// -// produceSeedMixes used to filter them out — "Songs like X excludes X's own -// songs" — which threw away the nearest neighbours of the seed TRACK and then -// reached further out to replace them. Operator, 2026-09-10: "it should also -// be able to include music from the same artist." +// bounded by the diversity cap rather than excluded outright: excluding them +// would throw away the nearest neighbours of the seed TRACK. // // Asserted end-to-end rather than by reading the source, because the guard has -// to survive the filter coming back in a different shape — and because an -// absence check would now match the comment explaining why the filter is gone. +// to survive a filter coming back in a different shape — and because an +// absence check would match the comment explaining why there is no filter. func TestBuildSystemPlaylists_SongsLikeIncludesItsSeedArtist(t *testing.T) { pool := newPool(t) logger := discardLogger() @@ -403,7 +399,7 @@ func TestBuildSystemPlaylists_SongsLikeIncludesItsSeedArtist(t *testing.T) { } // The bound is what makes inclusion safe. Without it, "include the // seed artist" becomes "the mix is mostly the seed artist", which is - // the radio failure (#3882) arriving on a different surface. + // the one-artist radio arriving on a different surface. if own > maxPerArtist { t.Errorf("a songs_like mix carries %d tracks by its own seed artist "+ "out of %d; the per-artist cap (%d) is not being applied", diff --git a/internal/playlists/you_might_like.go b/internal/playlists/you_might_like.go index 2e4d4bc1..171657f0 100644 --- a/internal/playlists/you_might_like.go +++ b/internal/playlists/you_might_like.go @@ -1,5 +1,5 @@ -// you_might_like.go builds the per-user "You might like" Home rows -// (#790): in-library albums/artists the listener doesn't actively spin +// you_might_like.go builds the per-user "You might like" Home rows: +// in-library albums/artists the listener doesn't actively spin // but is predicted to enjoy. It reuses the For-You candidate engine // (similarity + like-weighted scoring) and rolls the per-track scores up // to album/artist granularity, gated on a minimum listening history so a @@ -84,7 +84,7 @@ func buildYouMightLike( return youMightLikeResult{built: false} } // One rotating seed is right here (unlike For-You's multi-seed - // blend, #1269): the row is a short shelf, not a mix, and a single + // blend): the row is a short shelf, not a mix, and a single // neighborhood per day keeps it coherent. daily := pickDailySeeds(seeds, userID, dateStr, 1) if len(daily) == 0 { diff --git a/internal/playsessions/service.go b/internal/playsessions/service.go index edb14898..9489e48b 100644 --- a/internal/playsessions/service.go +++ b/internal/playsessions/service.go @@ -1,7 +1,6 @@ -// Package playsessions implements the listening-session lifecycle from -// spec §6: a session is a contiguous sequence of plays by a user with no -// gap longer than the configured timeout (default 30 minutes) between -// events. +// Package playsessions implements the listening-session lifecycle: a session is +// a contiguous sequence of plays by a user with no gap longer than the +// configured timeout (default 30 minutes) between events. package playsessions import ( @@ -35,7 +34,8 @@ func FindOrCreate( return pgtype.UUID{}, err } if err == nil { - // "Within window" includes the exact-boundary case: gap <= timeout extends. + // "Within window" includes the exact-boundary case: gap <= timeout + // extends. gap := eventTime.Sub(prev.LastEventAt.Time) if gap <= timeout { if err := q.TouchPlaySessionLastEvent(ctx, dbq.TouchPlaySessionLastEventParams{ diff --git a/internal/prowlarr/prowlarr.go b/internal/prowlarr/prowlarr.go new file mode 100644 index 00000000..3ca4975b --- /dev/null +++ b/internal/prowlarr/prowlarr.go @@ -0,0 +1,176 @@ +// Package prowlarr searches through Prowlarr's native API. +// +// Native rather than Torznab because Prowlarr has no aggregate Torznab feed: +// `/api/v1/indexer/all/newznab` 404s exactly like an indexer id that does not +// exist. +// `/api/v1/search` is the one place Prowlarr asks every indexer at once. +package prowlarr + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + "strings" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +// SearchPath is Prowlarr's aggregate search. +const SearchPath = "/api/v1/search" + +// DefaultTimeout is longer than a Torznab feed's: Prowlarr waits on every +// indexer it has before answering. +const DefaultTimeout = 60 * time.Second + +// Config is one Prowlarr. +type Config struct { + // Code is the acquisition_services row, stamped onto every Result. + Code string + BaseURL string + // APIKey is required: Prowlarr's API refuses every call without one. + APIKey string + Timeout time.Duration +} + +// Client searches one Prowlarr and fetches .torrent files through it. +type Client struct { + Config Config + // HTTP is injectable for tests. Nil takes http.DefaultClient. + HTTP *http.Client +} + +type searchResult struct { + Title string `json:"title"` + InfoHash string `json:"infoHash"` + GUID string `json:"guid"` + MagnetURL string `json:"magnetUrl"` + DownloadURL string `json:"downloadUrl"` + Size int64 `json:"size"` + Seeders int `json:"seeders"` + Leechers int `json:"leechers"` + Indexer string `json:"indexer"` + PublishDate string `json:"publishDate"` + Categories []struct { + ID int `json:"id"` + } `json:"categories"` +} + +// Search asks every indexer in this Prowlarr. `type=search` is free text; +// `type=music` takes artist and album separately, and a release name does not +// reliably say which part is which. +func (c *Client) Search(ctx context.Context, q indexer.Query) ([]indexer.Result, error) { + params := url.Values{"query": {q.Text}, "type": {"search"}} + for _, cat := range q.Categories { + params.Add("categories", fmt.Sprint(cat)) + } + endpoint := strings.TrimRight(c.Config.BaseURL, "/") + SearchPath + "?" + params.Encode() + body, err := c.get(ctx, endpoint) + if err != nil { + return nil, err + } + out, err := parseResults(body) + if err != nil { + return nil, err + } + for i := range out { + out[i].Source = c.Config.Code + } + return out, nil +} + +func parseResults(body []byte) ([]indexer.Result, error) { + var rows []searchResult + if err := json.Unmarshal(body, &rows); err != nil { + return nil, fmt.Errorf("%w: parse search results: %v", indexer.ErrUnreachable, err) + } + out := make([]indexer.Result, 0, len(rows)) + for _, row := range rows { + title := strings.TrimSpace(row.Title) + if title == "" { + continue + } + r := indexer.Result{ + Title: title, + SizeBytes: row.Size, + Seeders: row.Seeders, + Peers: row.Leechers, + Indexer: strings.TrimSpace(row.Indexer), + Magnet: firstMagnet(row.MagnetURL, row.GUID), + DownloadURL: strings.TrimSpace(row.DownloadURL), + } + if len(row.Categories) > 0 { + r.Category = row.Categories[0].ID + } + r.InfoHash = indexer.NormalizeHash(row.InfoHash) + if r.InfoHash == "" { + r.InfoHash = indexer.MagnetHash(r.Magnet) + } + if r.InfoHash == "" && indexer.IsInfoHash(row.GUID) { + r.InfoHash = indexer.NormalizeHash(row.GUID) + } + if t, err := time.Parse(time.RFC3339, strings.TrimSpace(row.PublishDate)); err == nil { + r.Published = t + } + out = append(out, r) + } + return out, nil +} + +func firstMagnet(candidates ...string) string { + for _, c := range candidates { + c = strings.TrimSpace(c) + if strings.HasPrefix(c, "magnet:") { + return c + } + } + return "" +} + +// Download fetches a result's .torrent through Prowlarr, which holds the +// tracker credentials. A public tracker's link often redirects to a magnet, +// which comes back as an *indexer.MagnetLink. +func (c *Client) Download(ctx context.Context, r indexer.Result) ([]byte, error) { + if r.DownloadURL == "" { + return nil, fmt.Errorf("%w: %q has no download URL", indexer.ErrUnreachable, r.Title) + } + body, err := c.get(ctx, r.DownloadURL) + if err != nil { + return nil, err + } + return indexer.TorrentBody(r.Title, body) +} + +func (c *Client) get(ctx context.Context, endpoint string) ([]byte, error) { + if c.Config.BaseURL == "" || c.Config.APIKey == "" { + return nil, indexer.ErrUnconfigured + } + timeout := c.Config.Timeout + if timeout <= 0 { + timeout = DefaultTimeout + } + ctx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil) + if err != nil { + return nil, fmt.Errorf("%w: build request: %v", indexer.ErrUnreachable, err) + } + req.Header.Set("User-Agent", indexer.UserAgent) + // In a header, not ?apikey=, so the key stays out of URLs, logs and + // error strings. + req.Header.Set("X-Api-Key", c.Config.APIKey) + req.Header.Set("Accept", "application/json") + resp, body, err := indexer.Send(c.HTTP, req) + if err != nil { + return nil, err + } + if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden { + return nil, fmt.Errorf("%w: HTTP %d", indexer.ErrAuth, resp.StatusCode) + } + if err := indexer.StatusError(resp, "Prowlarr"); err != nil { + return nil, err + } + return body, nil +} diff --git a/internal/prowlarr/prowlarr_test.go b/internal/prowlarr/prowlarr_test.go new file mode 100644 index 00000000..2d4f3011 --- /dev/null +++ b/internal/prowlarr/prowlarr_test.go @@ -0,0 +1,78 @@ +package prowlarr + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "testing" + + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" +) + +const sampleResults = `[ + {"title": "Radiohead - In Rainbows (2007) [FLAC]", "infoHash": "AABBCCDDEEFF00112233445566778899AABBCCDD", + "downloadUrl": "http://prowlarr/1/download?link=x", "size": 400000000, "seeders": 30, "leechers": 4, + "indexer": "RED", "publishDate": "2026-10-01T12:00:00Z", "categories": [{"id": 3040}]}, + {"title": "Radiohead - Amnesiac", "guid": "magnet:?xt=urn:btih:ffeeddccbbaa99887766554433221100ffeeddcc"}, + {"title": ""} +]` + +func TestSearch_AsksTheNativeAPIWithTheKeyInAHeader(t *testing.T) { + reqs := make(chan *http.Request, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + reqs <- r + _, _ = w.Write([]byte(sampleResults)) + })) + defer srv.Close() + + c := &Client{Config: Config{Code: "prowlarr", BaseURL: srv.URL + "/", APIKey: "secret"}} + results, err := c.Search(context.Background(), indexer.Query{Text: "radiohead", Categories: []int{3000}}) + if err != nil { + t.Fatalf("search: %v", err) + } + got := <-reqs + if got.URL.Path != SearchPath || got.URL.Query().Get("type") != "search" || got.URL.Query().Get("categories") != "3000" { + t.Errorf("request = %s, want %s with type=search and the categories", got.URL, SearchPath) + } + if got.Header.Get("X-Api-Key") != "secret" || got.URL.Query().Get("apikey") != "" { + t.Error("the key belongs in X-Api-Key, never in the URL") + } + if len(results) != 2 { + t.Fatalf("results = %d, want 2", len(results)) + } + if r := results[0]; r.InfoHash != "aabbccddeeff00112233445566778899aabbccdd" || r.Peers != 4 || + r.Category != 3040 || r.Source != "prowlarr" || r.Published.IsZero() { + t.Errorf("first = %+v", r) + } + if r := results[1]; r.Magnet == "" || r.InfoHash != "ffeeddccbbaa99887766554433221100ffeeddcc" { + t.Errorf("second = %+v, want the GUID's magnet and its hash", r) + } +} + +func TestSearch_Refusals(t *testing.T) { + for name, tc := range map[string]struct { + status int + want error + }{ + "a refused key": {http.StatusUnauthorized, indexer.ErrAuth}, + "an outage": {http.StatusBadGateway, indexer.ErrUnreachable}, + } { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(tc.status) + })) + c := &Client{Config: Config{BaseURL: srv.URL, APIKey: "k"}} + _, err := c.Search(context.Background(), indexer.Query{Text: "x"}) + srv.Close() + if !errors.Is(err, tc.want) { + t.Errorf("%s: err = %v, want %v", name, err, tc.want) + } + } +} + +func TestSearch_NoKeyIsUnconfigured(t *testing.T) { + c := &Client{Config: Config{BaseURL: "http://prowlarr:9696"}} + if _, err := c.Search(context.Background(), indexer.Query{Text: "x"}); !errors.Is(err, indexer.ErrUnconfigured) { + t.Fatalf("err = %v, want ErrUnconfigured", err) + } +} diff --git a/internal/quarantine/service.go b/internal/quarantine/service.go new file mode 100644 index 00000000..1e676508 --- /dev/null +++ b/internal/quarantine/service.go @@ -0,0 +1,267 @@ +// Package quarantine owns the per-user track quarantine workflow. +// Users flag a track as broken (Flag/Unflag), the SPA hides the track +// from their views, and admins resolve the resulting reports via the +// Service's admin actions (Resolve / DeleteFile). +package quarantine + +import ( + "context" + "errors" + "fmt" + "strings" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgconn" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/library" +) + +// Public errors. Handlers map these to API codes. +var ( + ErrBadReason = errors.New("quarantine: invalid reason") + ErrTrackNotFound = errors.New("quarantine: track not found") + ErrQuarantineNotFound = errors.New("quarantine: quarantine row not found") +) + +// Service owns the quarantine lifecycle. +type Service struct { + // dataDir lets a Delete file that empties an artist clear that artist's + // cached art, the same as the admin remove-track path. + dataDir string + pool *pgxpool.Pool +} + +// NewService constructs a Service. +func NewService(pool *pgxpool.Pool, dataDir string) *Service { + return &Service{pool: pool, dataDir: dataDir} +} + +// Flag inserts or updates a quarantine row for the caller. Re-flagging +// the same (user, track) overwrites reason+notes (and bumps created_at). +func (s *Service) Flag(ctx context.Context, userID, trackID pgtype.UUID, reason string, notes string) (dbq.Quarantine, error) { + if !validReason(reason) { + return dbq.Quarantine{}, ErrBadReason + } + var notesPtr *string + if notes != "" { + notesPtr = ¬es + } + row, err := dbq.New(s.pool).UpsertQuarantine(ctx, dbq.UpsertQuarantineParams{ + UserID: userID, + TrackID: trackID, + Reason: dbq.QuarantineReason(reason), + Notes: notesPtr, + }) + if err != nil { + // FK violation on track_id (no such track) → ErrTrackNotFound so the + // handler can return 404 instead of a generic 500. Code 23503 is the + // Postgres SQLSTATE for foreign_key_violation; the constraint check + // guards against future FKs (e.g. user_id) firing the same code. + var pgErr *pgconn.PgError + if errors.As(err, &pgErr) && pgErr.Code == "23503" && + (pgErr.ConstraintName == "" || strings.Contains(pgErr.ConstraintName, "track")) { + return dbq.Quarantine{}, ErrTrackNotFound + } + return dbq.Quarantine{}, fmt.Errorf("upsert: %w", err) + } + return row, nil +} + +// Unflag removes the caller's row. Returns ErrQuarantineNotFound if no +// row exists for that (user, track). +func (s *Service) Unflag(ctx context.Context, userID, trackID pgtype.UUID) error { + _, err := dbq.New(s.pool).DeleteQuarantine(ctx, dbq.DeleteQuarantineParams{ + UserID: userID, TrackID: trackID, + }) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + return ErrQuarantineNotFound + } + return fmt.Errorf("delete: %w", err) + } + return nil +} + +// ListMine returns the caller's quarantines with track + album + artist +// detail. Drives /library/hidden. Ordered newest-first. +func (s *Service) ListMine(ctx context.Context, userID pgtype.UUID) ([]dbq.ListQuarantineForUserRow, error) { + return dbq.New(s.pool).ListQuarantineForUser(ctx, userID) +} + +// AdminQueueRow is the assembled aggregated row served by the admin +// queue endpoint. The handler post-processes the SQL result + a per-track +// ListQuarantineReports call to materialize reason_counts and the +// per-user reports list. +type AdminQueueRow struct { + TrackID pgtype.UUID + TrackTitle string + ArtistName string + AlbumTitle string + AlbumID pgtype.UUID + ReportCount int32 + LatestAt pgtype.Timestamptz + ReasonCounts map[string]int + Reports []UserReport +} + +// UserReport is the per-user detail under an aggregated admin row. +type UserReport struct { + UserID pgtype.UUID + Username string + Reason string + Notes *string + CreatedAt pgtype.Timestamptz +} + +// ListAdminQueue returns the aggregated admin queue. One row per track. +// Performs an N+1 (one ListQuarantineReportsForTrack per aggregated row) +// because materializing reason_counts in pure SQL is awkward; the queue +// is small in practice (≲100 tracks at household scale). Revisit with +// json_object_agg if the queue grows. +func (s *Service) ListAdminQueue(ctx context.Context) ([]AdminQueueRow, error) { + q := dbq.New(s.pool) + aggregated, err := q.ListAdminQuarantineQueue(ctx) + if err != nil { + return nil, fmt.Errorf("aggregate: %w", err) + } + out := make([]AdminQueueRow, 0, len(aggregated)) + for _, r := range aggregated { + reports, err := q.ListQuarantineReportsForTrack(ctx, r.TrackID) + if err != nil { + return nil, fmt.Errorf("reports for track %v: %w", r.TrackID, err) + } + rc := make(map[string]int, len(reports)) + userReports := make([]UserReport, 0, len(reports)) + for _, rep := range reports { + rc[string(rep.Reason)]++ + userReports = append(userReports, UserReport{ + UserID: rep.UserID, + Username: rep.Username, + Reason: string(rep.Reason), + Notes: rep.Notes, + CreatedAt: rep.CreatedAt, + }) + } + out = append(out, AdminQueueRow{ + TrackID: r.TrackID, + TrackTitle: r.TrackTitle, + ArtistName: r.ArtistName, + AlbumTitle: r.AlbumTitle, + AlbumID: r.AlbumID, + ReportCount: r.ReportCount, + LatestAt: r.LatestAt, + ReasonCounts: rc, + Reports: userReports, + }) + } + return out, nil +} + +func validReason(r string) bool { + switch r { + case "bad_rip", "wrong_file", "wrong_tags", "duplicate", "other": + return true + } + return false +} + +// Resolve clears all per-user quarantine rows for a track and writes an +// audit log row. Idempotent — a track with no rows still writes an audit +// entry with affected_users=0 (so admin can see "I clicked resolve on a +// track that already had no reports"). +func (s *Service) Resolve(ctx context.Context, trackID, adminID pgtype.UUID) (dbq.QuarantineAction, error) { + q := dbq.New(s.pool) + track, err := q.GetTrackByID(ctx, trackID) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + return dbq.QuarantineAction{}, ErrTrackNotFound + } + return dbq.QuarantineAction{}, fmt.Errorf("get track: %w", err) + } + snap, err := s.snapshot(ctx, q, track) + if err != nil { + return dbq.QuarantineAction{}, err + } + + affected, err := q.CountQuarantineForTrack(ctx, trackID) + if err != nil { + return dbq.QuarantineAction{}, fmt.Errorf("count: %w", err) + } + if err := q.DeleteQuarantineForTrack(ctx, trackID); err != nil { + return dbq.QuarantineAction{}, fmt.Errorf("delete rows: %w", err) + } + return q.WriteQuarantineAction(ctx, dbq.WriteQuarantineActionParams{ + TrackID: trackID, TrackTitle: snap.TrackTitle, ArtistName: snap.ArtistName, + AlbumTitle: snap.AlbumTitle, Action: dbq.QuarantineActionKindResolved, + AdminID: adminID, AffectedUsers: affected, + }) +} + +// quarantineSnapshot is the audit-row-ready snapshot of track + album + +// artist text columns. Used by the admin actions. +type quarantineSnapshot struct { + TrackTitle string + ArtistName string + AlbumTitle *string +} + +// snapshot pulls the audit-row text columns from track / album / artist. +// AlbumTitle is materialized as *string because WriteQuarantineActionParams +// expects nullable; we coerce the schema's NOT NULL value into a pointer. +func (s *Service) snapshot(ctx context.Context, q *dbq.Queries, track dbq.Track) (quarantineSnapshot, error) { + album, err := q.GetAlbumByID(ctx, track.AlbumID) + if err != nil { + return quarantineSnapshot{}, fmt.Errorf("get album: %w", err) + } + artist, err := q.GetArtistByID(ctx, track.ArtistID) + if err != nil { + return quarantineSnapshot{}, fmt.Errorf("get artist: %w", err) + } + titlePtr := album.Title + return quarantineSnapshot{ + TrackTitle: track.Title, + ArtistName: artist.Name, + AlbumTitle: &titlePtr, + }, nil +} + +// DeleteFile removes the track file from disk and the tracks row (tidying away +// an album or artist that leaves empty), then — via FK ON DELETE CASCADE — +// clears every per-user quarantine row for that track and writes an audit row. +// +// If the file cannot be removed, nothing is deleted and the per-user rows stay +// so the admin can retry: the error wraps a *library.FileRemoveError, which the +// handler turns into an answer naming the cause. No partial state. +func (s *Service) DeleteFile(ctx context.Context, trackID, adminID pgtype.UUID) (dbq.QuarantineAction, error) { + q := dbq.New(s.pool) + track, err := q.GetTrackByID(ctx, trackID) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + return dbq.QuarantineAction{}, ErrTrackNotFound + } + return dbq.QuarantineAction{}, fmt.Errorf("get track: %w", err) + } + snap, err := s.snapshot(ctx, q, track) + if err != nil { + return dbq.QuarantineAction{}, err + } + + affected, err := q.CountQuarantineForTrack(ctx, trackID) + if err != nil { + return dbq.QuarantineAction{}, fmt.Errorf("count: %w", err) + } + if _, err := library.DeleteTrackFile(ctx, s.pool, nil, s.dataDir, trackID); err != nil { + return dbq.QuarantineAction{}, fmt.Errorf("delete file: %w", err) + } + // tracks row is gone; ON DELETE CASCADE on quarantine.track_id already + // cleared the per-user rows. We don't call DeleteQuarantineForTrack + // separately. + return q.WriteQuarantineAction(ctx, dbq.WriteQuarantineActionParams{ + TrackID: trackID, TrackTitle: snap.TrackTitle, ArtistName: snap.ArtistName, + AlbumTitle: snap.AlbumTitle, Action: dbq.QuarantineActionKindDeletedFile, + AdminID: adminID, AffectedUsers: affected, + }) +} diff --git a/internal/lidarrquarantine/service_test.go b/internal/quarantine/service_test.go similarity index 55% rename from internal/lidarrquarantine/service_test.go rename to internal/quarantine/service_test.go index cfa21359..6bb302fb 100644 --- a/internal/lidarrquarantine/service_test.go +++ b/internal/quarantine/service_test.go @@ -1,12 +1,10 @@ -package lidarrquarantine +package quarantine import ( "context" "errors" "io" "log/slog" - "net/http" - "net/http/httptest" "os" "path/filepath" "testing" @@ -17,8 +15,6 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" ) func newPool(t *testing.T) *pgxpool.Pool { @@ -87,7 +83,7 @@ func TestFlag_HappyPath(t *testing.T) { user := seedUser(t, pool, "alice") track, _, _ := seedTrack(t, pool, "Bad Track", "abc") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") row, err := svc.Flag(context.Background(), user.ID, track.ID, "bad_rip", "crackly") if err != nil { t.Fatalf("Flag: %v", err) @@ -105,7 +101,7 @@ func TestFlag_UpsertOnSecondFlag(t *testing.T) { user := seedUser(t, pool, "alice") track, _, _ := seedTrack(t, pool, "T", "x") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") if _, err := svc.Flag(context.Background(), user.ID, track.ID, "bad_rip", "first"); err != nil { t.Fatalf("first flag: %v", err) } @@ -129,7 +125,7 @@ func TestFlag_NonexistentTrackReturnsErrTrackNotFound(t *testing.T) { bogus.Bytes = [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16} bogus.Valid = true - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") _, err := svc.Flag(context.Background(), user.ID, bogus, "bad_rip", "") if !errors.Is(err, ErrTrackNotFound) { t.Errorf("err = %v, want ErrTrackNotFound", err) @@ -141,7 +137,7 @@ func TestFlag_BadReasonRejected(t *testing.T) { user := seedUser(t, pool, "alice") track, _, _ := seedTrack(t, pool, "T", "x") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") _, err := svc.Flag(context.Background(), user.ID, track.ID, "garbage", "") if !errors.Is(err, ErrBadReason) { t.Errorf("err = %v, want ErrBadReason", err) @@ -153,7 +149,7 @@ func TestUnflag_DeletesRow(t *testing.T) { user := seedUser(t, pool, "alice") track, _, _ := seedTrack(t, pool, "T", "x") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") if _, err := svc.Flag(context.Background(), user.ID, track.ID, "bad_rip", ""); err != nil { t.Fatalf("Flag: %v", err) } @@ -171,7 +167,7 @@ func TestListMine_OrderedNewestFirst(t *testing.T) { t1, _, _ := seedTrack(t, pool, "T1", "x") t2, _, _ := seedTrack(t, pool, "T2", "y") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") if _, err := svc.Flag(context.Background(), user.ID, t1.ID, "bad_rip", ""); err != nil { t.Fatalf("Flag t1: %v", err) } @@ -187,8 +183,8 @@ func TestListMine_OrderedNewestFirst(t *testing.T) { t.Fatalf("len = %d, want 2", len(rows)) } // T2 was flagged second — newest first. - if rows[0].LidarrQuarantine.TrackID != t2.ID { - t.Errorf("first row track = %v, want T2 (%v)", rows[0].LidarrQuarantine.TrackID, t2.ID) + if rows[0].Quarantine.TrackID != t2.ID { + t.Errorf("first row track = %v, want T2 (%v)", rows[0].Quarantine.TrackID, t2.ID) } } @@ -199,7 +195,7 @@ func TestListAdminQueue_AggregatesByTrackWithReasonCounts(t *testing.T) { carol := seedUser(t, pool, "carol") track, _, _ := seedTrack(t, pool, "Hot Mess", "abc") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") if _, err := svc.Flag(context.Background(), alice.ID, track.ID, "bad_rip", ""); err != nil { t.Fatalf("alice flag: %v", err) } @@ -235,7 +231,7 @@ func TestResolve_ClearsRowsAndWritesAudit(t *testing.T) { bob := seedUser(t, pool, "bob") track, _, _ := seedTrack(t, pool, "T", "x") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") if _, err := svc.Flag(context.Background(), alice.ID, track.ID, "bad_rip", ""); err != nil { t.Fatalf("alice flag: %v", err) } @@ -250,12 +246,9 @@ func TestResolve_ClearsRowsAndWritesAudit(t *testing.T) { if audit.AffectedUsers != 2 { t.Errorf("affected_users = %d, want 2", audit.AffectedUsers) } - if audit.Action != dbq.LidarrQuarantineActionKindResolved { + if audit.Action != dbq.QuarantineActionKindResolved { t.Errorf("action = %v, want resolved", audit.Action) } - if audit.LidarrAlbumMbid != nil { - t.Errorf("lidarr_album_mbid = %v, want nil for resolve", audit.LidarrAlbumMbid) - } n, _ := dbq.New(pool).CountQuarantineForTrack(context.Background(), track.ID) if n != 0 { t.Errorf("rows after resolve = %d, want 0", n) @@ -267,7 +260,7 @@ func TestResolve_NoExistingRowsStillWritesAudit(t *testing.T) { user := seedUser(t, pool, "alice") track, _, _ := seedTrack(t, pool, "T", "x") - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") // No flags applied — resolve a track with zero quarantine rows. audit, err := svc.Resolve(context.Background(), track.ID, user.ID) if err != nil { @@ -276,12 +269,9 @@ func TestResolve_NoExistingRowsStillWritesAudit(t *testing.T) { if audit.AffectedUsers != 0 { t.Errorf("affected_users = %d, want 0", audit.AffectedUsers) } - if audit.Action != dbq.LidarrQuarantineActionKindResolved { + if audit.Action != dbq.QuarantineActionKindResolved { t.Errorf("action = %v, want resolved", audit.Action) } - if audit.LidarrAlbumMbid != nil { - t.Errorf("lidarr_album_mbid = %v, want nil for resolve", audit.LidarrAlbumMbid) - } } func TestResolve_TrackNotFound(t *testing.T) { @@ -291,7 +281,7 @@ func TestResolve_TrackNotFound(t *testing.T) { bogus.Bytes = [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16} bogus.Valid = true - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") _, err := svc.Resolve(context.Background(), bogus, user.ID) if !errors.Is(err, ErrTrackNotFound) { t.Errorf("err = %v, want ErrTrackNotFound", err) @@ -315,7 +305,7 @@ func TestDeleteFile_RemovesFileAndAuditsAffected(t *testing.T) { DurationMs: 1000, FilePath: path, FileSize: 1, FileFormat: "mp3", }) - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") if _, err := svc.Flag(context.Background(), user.ID, track.ID, "bad_rip", ""); err != nil { t.Fatalf("Flag: %v", err) } @@ -327,12 +317,9 @@ func TestDeleteFile_RemovesFileAndAuditsAffected(t *testing.T) { if audit.AffectedUsers != 1 { t.Errorf("affected_users = %d, want 1", audit.AffectedUsers) } - if audit.Action != dbq.LidarrQuarantineActionKindDeletedFile { + if audit.Action != dbq.QuarantineActionKindDeletedFile { t.Errorf("action = %v, want deleted_file", audit.Action) } - if audit.LidarrAlbumMbid != nil { - t.Errorf("lidarr_album_mbid = %v, want nil for delete_file", audit.LidarrAlbumMbid) - } if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) { t.Errorf("file still exists: %v", err) } @@ -342,156 +329,6 @@ func TestDeleteFile_RemovesFileAndAuditsAffected(t *testing.T) { } } -func TestDeleteViaLidarr_FullCascade(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool, "alice") - - var captured []string - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - captured = append(captured, r.Method+" "+r.URL.Path+"?"+r.URL.RawQuery) - w.Header().Set("Content-Type", "application/json") - if r.URL.Path == "/api/v1/album" && r.Method == http.MethodGet { - _, _ = w.Write([]byte(`[{"id":42,"foreignAlbumId":"al-mbid","title":"Al","artistId":7}]`)) - return - } - // DELETE /api/v1/album/42 -> 200. - w.WriteHeader(http.StatusOK) - })) - t.Cleanup(stub.Close) - - cfg := lidarrconfig.New(pool) - if err := cfg.Save(context.Background(), lidarrconfig.Config{ - Enabled: true, BaseURL: stub.URL, APIKey: "k", - }); err != nil { - t.Fatalf("save config: %v", err) - } - clientFn := func() *lidarr.Client { return lidarr.NewClient(stub.URL, "k") } - svc := NewService(pool, cfg, clientFn, "") - - q := dbq.New(pool) - artist, _ := q.UpsertArtist(context.Background(), dbq.UpsertArtistParams{Name: "A", SortName: "A"}) - albumMBID := "al-mbid" - album, _ := q.UpsertAlbum(context.Background(), dbq.UpsertAlbumParams{ - Title: "Al", SortTitle: "Al", ArtistID: artist.ID, Mbid: &albumMBID, - }) - dir := t.TempDir() - path := filepath.Join(dir, "T.mp3") - if err := os.WriteFile(path, []byte("x"), 0o644); err != nil { - t.Fatal(err) - } - track, _ := q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ - Title: "T", AlbumID: album.ID, ArtistID: artist.ID, - DurationMs: 1000, FilePath: path, FileSize: 1, FileFormat: "mp3", - }) - if _, err := svc.Flag(context.Background(), user.ID, track.ID, "bad_rip", ""); err != nil { - t.Fatalf("Flag: %v", err) - } - - audit, deleted, err := svc.DeleteViaLidarr(context.Background(), track.ID, user.ID) - if err != nil { - t.Fatalf("DeleteViaLidarr: %v", err) - } - if deleted != 1 { - t.Errorf("deleted = %d, want 1 track removed", deleted) - } - if audit.Action != dbq.LidarrQuarantineActionKindDeletedViaLidarr { - t.Errorf("action = %v", audit.Action) - } - if audit.AffectedUsers != 1 { - t.Errorf("affected_users = %d, want 1", audit.AffectedUsers) - } - if audit.LidarrAlbumMbid == nil || *audit.LidarrAlbumMbid != "al-mbid" { - t.Errorf("lidarr_album_mbid = %v", audit.LidarrAlbumMbid) - } - if _, err := q.GetTrackByID(context.Background(), track.ID); err == nil { - t.Errorf("track row still exists") - } - // Verify Lidarr received DELETE with both flags true (query order may vary). - foundDelete := false - for _, c := range captured { - if c == "DELETE /api/v1/album/42?addImportListExclusion=true&deleteFiles=true" || - c == "DELETE /api/v1/album/42?deleteFiles=true&addImportListExclusion=true" { - foundDelete = true - } - } - if !foundDelete { - t.Errorf("Lidarr DELETE not called with both flags true; captured = %v", captured) - } -} - -func TestDeleteViaLidarr_LidarrDisabled(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool, "alice") - track, _, _ := seedTrack(t, pool, "T", "x") - - svc := NewService(pool, lidarrconfig.New(pool), nil, "") - _, _, err := svc.DeleteViaLidarr(context.Background(), track.ID, user.ID) - if !errors.Is(err, ErrLidarrDisabled) { - t.Errorf("err = %v, want ErrLidarrDisabled", err) - } -} - -func TestDeleteViaLidarr_AlbumMBIDMissing(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool, "alice") - - q := dbq.New(pool) - artist, _ := q.UpsertArtist(context.Background(), dbq.UpsertArtistParams{Name: "A", SortName: "A"}) - // Album with NO mbid set. - album, _ := q.UpsertAlbum(context.Background(), dbq.UpsertAlbumParams{ - Title: "Al", SortTitle: "Al", ArtistID: artist.ID, // Mbid: nil - }) - dir := t.TempDir() - path := filepath.Join(dir, "T.mp3") - _ = os.WriteFile(path, []byte("x"), 0o644) - track, _ := q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ - Title: "T", AlbumID: album.ID, ArtistID: artist.ID, - DurationMs: 1000, FilePath: path, FileSize: 1, FileFormat: "mp3", - }) - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusOK) - })) - t.Cleanup(stub.Close) - cfg := lidarrconfig.New(pool) - _ = cfg.Save(context.Background(), lidarrconfig.Config{Enabled: true, BaseURL: stub.URL, APIKey: "k"}) - clientFn := func() *lidarr.Client { return lidarr.NewClient(stub.URL, "k") } - svc := NewService(pool, cfg, clientFn, "") - _, _ = svc.Flag(context.Background(), user.ID, track.ID, "bad_rip", "") - - _, _, err := svc.DeleteViaLidarr(context.Background(), track.ID, user.ID) - if !errors.Is(err, ErrAlbumMBIDMissing) { - t.Errorf("err = %v, want ErrAlbumMBIDMissing", err) - } -} - -func TestDeleteViaLidarr_LidarrAlbumNotFound(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool, "alice") - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/json") - if r.URL.Path == "/api/v1/album" && r.Method == http.MethodGet { - _, _ = w.Write([]byte("[]")) // Lidarr returns empty -> ErrNotFound - return - } - w.WriteHeader(http.StatusOK) - })) - t.Cleanup(stub.Close) - cfg := lidarrconfig.New(pool) - _ = cfg.Save(context.Background(), lidarrconfig.Config{Enabled: true, BaseURL: stub.URL, APIKey: "k"}) - clientFn := func() *lidarr.Client { return lidarr.NewClient(stub.URL, "k") } - svc := NewService(pool, cfg, clientFn, "") - - track, _, _ := seedTrack(t, pool, "T", "x") - _, _ = svc.Flag(context.Background(), user.ID, track.ID, "bad_rip", "") - - _, _, err := svc.DeleteViaLidarr(context.Background(), track.ID, user.ID) - if !errors.Is(err, ErrLidarrAlbumNotFound) { - t.Errorf("err = %v, want ErrLidarrAlbumNotFound", err) - } -} - func TestDeleteFile_TrackNotFound(t *testing.T) { pool := newPool(t) user := seedUser(t, pool, "alice") @@ -499,36 +336,9 @@ func TestDeleteFile_TrackNotFound(t *testing.T) { bogus.Bytes = [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16} bogus.Valid = true - svc := NewService(pool, lidarrconfig.New(pool), nil, "") + svc := NewService(pool, "") _, err := svc.DeleteFile(context.Background(), bogus, user.ID) if !errors.Is(err, ErrTrackNotFound) { t.Errorf("err = %v, want ErrTrackNotFound", err) } } - -func TestDeleteViaLidarr_TrackNotFound(t *testing.T) { - pool := newPool(t) - user := seedUser(t, pool, "alice") - - stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.WriteHeader(http.StatusOK) - })) - t.Cleanup(stub.Close) - cfg := lidarrconfig.New(pool) - if err := cfg.Save(context.Background(), lidarrconfig.Config{ - Enabled: true, BaseURL: stub.URL, APIKey: "k", - }); err != nil { - t.Fatalf("save config: %v", err) - } - clientFn := func() *lidarr.Client { return lidarr.NewClient(stub.URL, "k") } - svc := NewService(pool, cfg, clientFn, "") - - var bogus pgtype.UUID - bogus.Bytes = [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16} - bogus.Valid = true - - _, _, err := svc.DeleteViaLidarr(context.Background(), bogus, user.ID) - if !errors.Is(err, ErrTrackNotFound) { - t.Errorf("err = %v, want ErrTrackNotFound", err) - } -} diff --git a/internal/reacquisition/settings.go b/internal/reacquisition/settings.go index 455f9f36..29bf05ba 100644 --- a/internal/reacquisition/settings.go +++ b/internal/reacquisition/settings.go @@ -1,11 +1,11 @@ -// Package reacquisition turns a missing file back into a Lidarr request -// without anyone pressing anything (milestone #290). +// Package reacquisition wants an album back when its files go missing, +// without anyone pressing anything. // -// The unit of work is the ALBUM, not the track. Lidarr acquires releases; -// there is no meaningful "fetch me one track", and a track-kind request needs -// a recording MBID plenty of files lack. Grouping also does most of the -// safety work: the loss that produced #2523 — three reorganised albums, ~40 -// missing files — becomes three requests rather than forty. +// The unit of work is the ALBUM, not the track. Music is acquired as +// releases; there is no meaningful "fetch me one track", and a recording +// MBID is something plenty of files lack. Grouping also does most of the +// safety work: a reorganised album's forty missing files become one want, +// not forty. package reacquisition import ( @@ -14,23 +14,18 @@ import ( "fmt" "log/slog" "sync" - "time" "github.com/jackc/pgx/v5/pgxpool" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Settings is the operator-tunable policy, mirroring the columns and CHECK +// Settings is the admin-tunable policy, mirroring the columns and CHECK // ranges in migration 0056. type Settings struct { - Enabled bool - GraceHours int32 - BackoffBaseHours int32 - BackoffMaxHours int32 - MaxAttempts int32 - MaxPerPass int32 - AutoApprove bool + Enabled bool + GraceHours int32 + MaxPerPass int32 } // Defaults mirror migration 0056's column defaults. Duplicated here so a @@ -38,13 +33,9 @@ type Settings struct { // zero-valued one — a zero grace window would fire on every transient // unmount, which is the exact failure the grace period exists to prevent. var Defaults = Settings{ - Enabled: true, - GraceHours: 24, - BackoffBaseHours: 6, - BackoffMaxHours: 168, - MaxAttempts: 3, - MaxPerPass: 20, - AutoApprove: true, + Enabled: true, + GraceHours: 24, + MaxPerPass: 20, } // ErrOutOfRange is returned by Set for values migration 0056's CHECKs would @@ -92,13 +83,9 @@ func (s *SettingsService) Set(ctx context.Context, in Settings) (Settings, error return Settings{}, err } row, err := dbq.New(s.pool).UpdateReacquisitionSettings(ctx, dbq.UpdateReacquisitionSettingsParams{ - Enabled: in.Enabled, - GraceHours: in.GraceHours, - BackoffBaseHours: in.BackoffBaseHours, - BackoffMaxHours: in.BackoffMaxHours, - MaxAttempts: in.MaxAttempts, - MaxPerPass: in.MaxPerPass, - AutoApprove: in.AutoApprove, + Enabled: in.Enabled, + GraceHours: in.GraceHours, + MaxPerPass: in.MaxPerPass, }) if err != nil { return Settings{}, fmt.Errorf("reacquisition: save settings: %w", err) @@ -110,43 +97,10 @@ func (s *SettingsService) Set(ctx context.Context, in Settings) (Settings, error return out, nil } -// Backoff is how long to wait before the next attempt on an album that has -// already been tried [attempts] times: base * 2^(attempts-1), clamped to the -// configured maximum. Zero attempts means "never tried", which is always due. -// -// Exported and pure so the schedule is testable without a database, and so -// the admin surface can show the same number the sweeper will act on. -func (s Settings) Backoff(attempts int32) time.Duration { - if attempts <= 0 { - return 0 - } - hours := s.BackoffBaseHours - for i := int32(1); i < attempts; i++ { - hours *= 2 - // Clamp inside the loop as well as after: doubling from a large base - // enough times would overflow int32 before the comparison ran. - if hours >= s.BackoffMaxHours { - return time.Duration(s.BackoffMaxHours) * time.Hour - } - } - if hours > s.BackoffMaxHours { - hours = s.BackoffMaxHours - } - return time.Duration(hours) * time.Hour -} - func validate(in Settings) error { switch { case in.GraceHours < 1 || in.GraceHours > 720: return fmt.Errorf("%w: grace_hours must be 1-720", ErrOutOfRange) - case in.BackoffBaseHours < 1 || in.BackoffBaseHours > 168: - return fmt.Errorf("%w: backoff_base_hours must be 1-168", ErrOutOfRange) - case in.BackoffMaxHours < 1 || in.BackoffMaxHours > 720: - return fmt.Errorf("%w: backoff_max_hours must be 1-720", ErrOutOfRange) - case in.BackoffMaxHours < in.BackoffBaseHours: - return fmt.Errorf("%w: backoff_max_hours must be >= backoff_base_hours", ErrOutOfRange) - case in.MaxAttempts < 1 || in.MaxAttempts > 10: - return fmt.Errorf("%w: max_attempts must be 1-10", ErrOutOfRange) case in.MaxPerPass < 1 || in.MaxPerPass > 200: return fmt.Errorf("%w: max_per_pass must be 1-200", ErrOutOfRange) } @@ -155,12 +109,8 @@ func validate(in Settings) error { func fromRow(row dbq.ReacquisitionSetting) Settings { return Settings{ - Enabled: row.Enabled, - GraceHours: row.GraceHours, - BackoffBaseHours: row.BackoffBaseHours, - BackoffMaxHours: row.BackoffMaxHours, - MaxAttempts: row.MaxAttempts, - MaxPerPass: row.MaxPerPass, - AutoApprove: row.AutoApprove, + Enabled: row.Enabled, + GraceHours: row.GraceHours, + MaxPerPass: row.MaxPerPass, } } diff --git a/internal/reacquisition/settings_test.go b/internal/reacquisition/settings_test.go index 6f7ed3e2..60b599d9 100644 --- a/internal/reacquisition/settings_test.go +++ b/internal/reacquisition/settings_test.go @@ -3,69 +3,8 @@ package reacquisition import ( "errors" "testing" - "time" ) -func TestBackoffSchedule(t *testing.T) { - s := Defaults // 6h base, 168h (one week) cap, 3 attempts - - cases := []struct { - name string - attempts int32 - want time.Duration - }{ - // Never tried is always due — the grace window, not the backoff, is - // what holds the first attempt back. - {"never attempted", 0, 0}, - {"negative is treated as never", -1, 0}, - {"after one attempt", 1, 6 * time.Hour}, - {"after two", 2, 12 * time.Hour}, - {"after three", 3, 24 * time.Hour}, - {"after four", 4, 48 * time.Hour}, - {"after five", 5, 96 * time.Hour}, - // 6h * 2^5 = 192h, past the one-week cap. - {"clamped at the cap", 6, 168 * time.Hour}, - {"still clamped far out", 20, 168 * time.Hour}, - } - for _, c := range cases { - t.Run(c.name, func(t *testing.T) { - if got := s.Backoff(c.attempts); got != c.want { - t.Errorf("Backoff(%d) = %v, want %v", c.attempts, got, c.want) - } - }) - } -} - -// The doubling must not be able to overflow int32 before the clamp is -// consulted — a large base with a high attempt count is the case that would -// wrap negative and make a spent album look due immediately. -func TestBackoffDoesNotOverflow(t *testing.T) { - s := Settings{BackoffBaseHours: 168, BackoffMaxHours: 720} - for attempts := int32(1); attempts <= 40; attempts++ { - got := s.Backoff(attempts) - if got <= 0 { - t.Fatalf("Backoff(%d) = %v, want a positive duration", attempts, got) - } - if got > 720*time.Hour { - t.Fatalf("Backoff(%d) = %v, want <= the 720h cap", attempts, got) - } - } -} - -func TestBackoffRespectsCustomSettings(t *testing.T) { - s := Settings{BackoffBaseHours: 1, BackoffMaxHours: 4} - for attempts, want := range map[int32]time.Duration{ - 1: 1 * time.Hour, - 2: 2 * time.Hour, - 3: 4 * time.Hour, - 4: 4 * time.Hour, // clamped - } { - if got := s.Backoff(attempts); got != want { - t.Errorf("Backoff(%d) = %v, want %v", attempts, got, want) - } - } -} - func TestValidateRejectsWhatTheCheckWouldReject(t *testing.T) { // Each case mirrors a CHECK in migration 0056. Validating in Go as well // means the API answers 400 with a readable message instead of surfacing @@ -77,15 +16,6 @@ func TestValidateRejectsWhatTheCheckWouldReject(t *testing.T) { {"zero grace would fire on every transient unmount", mutate(func(s *Settings) { s.GraceHours = 0 })}, {"grace beyond a month", mutate(func(s *Settings) { s.GraceHours = 721 })}, - {"zero backoff base", mutate(func(s *Settings) { s.BackoffBaseHours = 0 })}, - {"backoff base beyond a week", mutate(func(s *Settings) { s.BackoffBaseHours = 169 })}, - {"zero backoff cap", mutate(func(s *Settings) { s.BackoffMaxHours = 0 })}, - {"cap below base is incoherent", mutate(func(s *Settings) { - s.BackoffBaseHours = 48 - s.BackoffMaxHours = 24 - })}, - {"zero attempts means never try", mutate(func(s *Settings) { s.MaxAttempts = 0 })}, - {"attempts beyond ten", mutate(func(s *Settings) { s.MaxAttempts = 11 })}, {"zero per pass means never sweep", mutate(func(s *Settings) { s.MaxPerPass = 0 })}, {"per pass beyond the cap", mutate(func(s *Settings) { s.MaxPerPass = 201 })}, } @@ -104,21 +34,6 @@ func TestValidateAcceptsDefaults(t *testing.T) { } } -// Equal base and cap is legal — it is how an operator asks for a flat retry -// interval rather than an escalating one. -func TestValidateAcceptsFlatBackoff(t *testing.T) { - s := mutate(func(s *Settings) { - s.BackoffBaseHours = 12 - s.BackoffMaxHours = 12 - }) - if err := validate(s); err != nil { - t.Fatalf("flat backoff should be allowed, got %v", err) - } - if got := s.Backoff(5); got != 12*time.Hour { - t.Errorf("flat backoff gave %v, want 12h at every attempt", got) - } -} - func mutate(f func(*Settings)) Settings { s := Defaults f(&s) diff --git a/internal/reacquisition/sweeper.go b/internal/reacquisition/sweeper.go index fba2a6e3..b5c4118a 100644 --- a/internal/reacquisition/sweeper.go +++ b/internal/reacquisition/sweeper.go @@ -7,61 +7,38 @@ import ( "log/slog" "time" - "github.com/google/uuid" "github.com/jackc/pgx/v5" - "github.com/jackc/pgx/v5/pgtype" "github.com/jackc/pgx/v5/pgxpool" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" - "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" "git.fabledsword.com/bvandeusen/minstrel/internal/tags" ) -// requestCreator is the slice of lidarrrequests.Service the sweeper needs, -// narrowed to an interface so the pass can be tested without a Lidarr client -// or an approval path that talks to one. -type requestCreator interface { - CreateTracked(ctx context.Context, userID pgtype.UUID, p lidarrrequests.CreateParams) (dbq.LidarrRequest, bool, error) - Approve(ctx context.Context, requestID, adminID pgtype.UUID, ov lidarrrequests.ApproveOverrides) (dbq.LidarrRequest, error) -} - -// Sweeper periodically turns albums with long-missing files into Lidarr -// requests (milestone #290). +// Sweeper periodically wants back albums whose files have been missing past +// the grace window. Searching for them is the search schedule's, which +// retries per want. // // Deliberately a periodic worker rather than a hook inside the scanner's -// reconcile pass. Reconcile runs inside a scan and has no business deciding -// to talk to a third-party service; it also re-runs often, which would make -// "attempt once, then back off" awkward to express. A worker paces itself, -// survives a restart, and retries without needing another scan. +// reconcile pass: reconcile runs inside a scan and re-runs often, and a +// filesystem lies transiently (an unmounted volume, a NAS mid-reboot) in ways +// the grace window exists to ride out. type Sweeper struct { pool *pgxpool.Pool settings *SettingsService - requests requestCreator logger *slog.Logger - // notifier tells admins about a request the sweeper filed that still - // needs their approval (M489). Nil: nobody is told. - notifier *notifications.Notifier tick time.Duration // releaseGroup names the MusicBrainz release group of a release id, for an - // album whose tags never carried one (#5241). Lidarr knows albums only by - // release group. A field so tests can stand in for MusicBrainz. + // album whose tags never carried one. Wants are kept by release + // group. A field so tests can stand in for MusicBrainz. releaseGroup func(ctx context.Context, releaseMBID string) (string, error) } -// NewSweeper constructs a Sweeper. The tick is deliberately coarse: the -// smallest meaningful backoff is measured in hours, so waking more often than -// hourly would only re-read settings and find nothing due. -func NewSweeper( - pool *pgxpool.Pool, - settings *SettingsService, - requests requestCreator, - logger *slog.Logger, -) *Sweeper { +// NewSweeper constructs a Sweeper. The tick is deliberately coarse: the grace +// window is measured in hours. +func NewSweeper(pool *pgxpool.Pool, settings *SettingsService, logger *slog.Logger) *Sweeper { return &Sweeper{ pool: pool, settings: settings, - requests: requests, logger: logger, tick: 1 * time.Hour, releaseGroup: tags.ReleaseGroupForRelease, @@ -86,100 +63,50 @@ func (s *Sweeper) Run(ctx context.Context) { // PassResult reports what a single sweep did, for logging and tests. type PassResult struct { - Cleared int64 // albums whose files came back; state dropped - Requested int // requests created this pass - Approved int // of those, sent on to Lidarr - GaveUp int // albums that spent their attempt budget - Unnameable int64 // albums with missing files but no MBID to ask for + Wanted int // albums newly wanted, or wanted again after being held + Unnameable int64 // albums with missing files but no MBID to want them by // Unresolved: albums skipped this pass because MusicBrainz could not name - // their release group (switched off, or no such release). Lidarr cannot be - // asked for them, and no attempt is spent; the next pass tries again. + // their release group; the next pass tries again. Unresolved int } -// SetNotifier makes requests the sweeper files, and cannot approve itself, -// reach the admins' notifications inbox (M489). -func (s *Sweeper) SetNotifier(n *notifications.Notifier) { s.notifier = n } - -// SweepOnce runs one pass. Exported so the admin surface can offer a "run -// now" without waiting out the tick, and so tests drive it directly. +// SweepOnce runs one pass. Exported so tests drive it directly. func (s *Sweeper) SweepOnce(ctx context.Context) error { cfg := s.settings.Get() if !cfg.Enabled { return nil } q := dbq.New(s.pool) - - // Before selecting work: drop state for albums whose files came back, or - // were adopted at a new path (#2528). Doing this first means a recovered - // album cannot be picked in the same pass that would have retried it. - cleared, err := q.ClearRecoveredReacquisitions(ctx) - if err != nil { - return fmt.Errorf("clear recovered: %w", err) - } - res := PassResult{Cleared: cleared} - + var res PassResult // Counted, not acted on: an album MusicBrainz cannot name is not a - // failure to retry, it is a permanent gap the operator should see. + // failure to retry, it is a permanent gap an admin should see. if n, cerr := q.CountAlbumsMissingWithoutMbid(ctx); cerr == nil { res.Unnameable = n } - due, err := q.ListAlbumsDueReacquisition(ctx, dbq.ListAlbumsDueReacquisitionParams{ - GraceHours: cfg.GraceHours, - BackoffBaseHours: cfg.BackoffBaseHours, - BackoffMaxHours: cfg.BackoffMaxHours, - PageLimit: cfg.MaxPerPass, + GraceHours: cfg.GraceHours, + PageLimit: cfg.MaxPerPass, }) if err != nil { return fmt.Errorf("list due: %w", err) } - if len(due) == 0 { - s.logSummary(res) - return nil - } - - // One admin lookup per pass, not per album. A library with no admin at - // all cannot own a request, so the pass stops rather than half-working. - admin, err := q.GetOldestAdmin(ctx) - if err != nil { - if errors.Is(err, pgx.ErrNoRows) { - s.logger.Warn("reacquisition: no admin account to own requests; skipping pass") - return nil - } - return fmt.Errorf("owner lookup: %w", err) - } - for _, album := range due { - if err := s.attempt(ctx, q, cfg, admin.ID, album, &res); err != nil { - // One album's failure must not abandon the rest of the pass: a - // single unmatched MBID or a transient Lidarr error says nothing - // about the next album in the list. - s.logger.Warn("reacquisition: attempt failed", - "album", album.AlbumTitle, "err", err) + if err := s.want(ctx, q, album, &res); err != nil { + // One album's failure must not abandon the rest of the pass. + s.logger.Warn("reacquisition: want failed", "album", album.AlbumTitle, "err", err) } } s.logSummary(res) return nil } -// attempt creates (and optionally approves) the request for one album, then -// records the attempt against its backoff budget. -func (s *Sweeper) attempt( - ctx context.Context, - q *dbq.Queries, - cfg Settings, - adminID pgtype.UUID, - album dbq.ListAlbumsDueReacquisitionRow, - res *PassResult, -) error { - // The query already filters these out; belt and braces, because Create - // would reject the request and burn an attempt for no reason. - if album.AlbumMbid == nil || album.ArtistMbid == nil { +// want records one album as wanted under its release group. +func (s *Sweeper) want(ctx context.Context, q *dbq.Queries, album dbq.ListAlbumsDueReacquisitionRow, res *PassResult) error { + // The query already filters these out; belt and braces. + hasGroup := album.AlbumReleaseGroupMbid != nil && *album.AlbumReleaseGroupMbid != "" + if (album.AlbumMbid == nil && !hasGroup) || album.ArtistMbid == nil { return nil } - - // Lidarr names albums by release group; albums.mbid is the release. group, err := s.albumReleaseGroup(ctx, q, album) if err != nil { if errors.Is(err, tags.ErrNotFound) { @@ -190,77 +117,25 @@ func (s *Sweeper) attempt( } return fmt.Errorf("release group: %w", err) } - - req, created, err := s.requests.CreateTracked(ctx, adminID, lidarrrequests.CreateParams{ - Kind: "album", - LidarrArtistMBID: *album.ArtistMbid, - ArtistName: album.ArtistName, - LidarrAlbumMBID: group, - AlbumTitle: album.AlbumTitle, + _, err = q.WantMissingAlbum(ctx, dbq.WantMissingAlbumParams{ + ReleaseGroupMbid: group, ArtistMbid: *album.ArtistMbid, ArtistName: album.ArtistName, + Title: album.AlbumTitle, AlbumID: album.AlbumID, }) + if errors.Is(err, pgx.ErrNoRows) { + // Already wanted, open: by a request, a follow, or this sweeper. + return nil + } if err != nil { - return fmt.Errorf("create request: %w", err) - } - res.Requested++ - - // Record the attempt even when Create deduped into somebody else's - // existing request: the point of the counter is "how often have we gone - // looking for this album", and a manual request in flight is a reason to - // wait rather than to keep asking. - row, err := q.RecordReacquisitionAttempt(ctx, dbq.RecordReacquisitionAttemptParams{ - AlbumID: album.AlbumID, - LastRequestID: req.ID, - }) - if err != nil { - return fmt.Errorf("record attempt: %w", err) - } - - // A request the sweeper filed and could not approve waits on an admin. - // One it deduped into was announced when it was first filed. - awaitingAdmin := created - if cfg.AutoApprove { - _, aerr := s.requests.Approve(ctx, req.ID, adminID, lidarrrequests.ApproveOverrides{}) - switch { - case aerr == nil: - res.Approved++ - awaitingAdmin = false - case errors.Is(aerr, lidarrrequests.ErrLidarrDisabled): - // Leave it pending rather than treating it as a failure. The - // request is still the right record of intent, and it becomes - // actionable the moment Lidarr is configured. - s.logger.Info("reacquisition: request left pending, Lidarr disabled", - "album", album.AlbumTitle) - case errors.Is(aerr, lidarrrequests.ErrNotPending): - // Deduped onto a request somebody already approved. Nothing to do - // and nothing wrong. - default: - return fmt.Errorf("approve: %w", aerr) - } - } - - if awaitingAdmin { - s.notifier.NotifyLogged(ctx, notifications.KindRequestPending, notifications.ToAdmins(pgtype.UUID{}), - notifications.Payload{ - RequestID: uuid.UUID(req.ID.Bytes).String(), - RequestKind: string(req.Kind), - Name: lidarrrequests.DisplayName(req), - Actor: "Re-acquisition", - }.Map()) - } - - if row.Attempts >= cfg.MaxAttempts { - if err := q.MarkReacquisitionGaveUp(ctx, album.AlbumID); err != nil { - return fmt.Errorf("mark gave up: %w", err) - } - res.GaveUp++ + return fmt.Errorf("want: %w", err) } + res.Wanted++ return nil } // albumReleaseGroup is the album's release group: the one its tags carried, // else MusicBrainz's answer for its release id, cached onto the album so the -// next pass (and request completion) need not ask again. Returns -// tags.ErrNotFound when MusicBrainz cannot name it. +// next pass need not ask again. Returns tags.ErrNotFound when MusicBrainz +// cannot name it. func (s *Sweeper) albumReleaseGroup(ctx context.Context, q *dbq.Queries, album dbq.ListAlbumsDueReacquisitionRow) (string, error) { if album.AlbumReleaseGroupMbid != nil && *album.AlbumReleaseGroupMbid != "" { return *album.AlbumReleaseGroupMbid, nil @@ -281,14 +156,11 @@ func (s *Sweeper) albumReleaseGroup(ctx context.Context, q *dbq.Queries, album d func (s *Sweeper) logSummary(res PassResult) { // Silence when a pass did nothing at all — this runs hourly forever, and // an unconditional line would bury the passes that mattered. - if res.Requested == 0 && res.Cleared == 0 && res.GaveUp == 0 && res.Unresolved == 0 { + if res.Wanted == 0 && res.Unresolved == 0 { return } s.logger.Info("reacquisition: sweep", - "requested", res.Requested, - "approved", res.Approved, - "gave_up", res.GaveUp, - "cleared", res.Cleared, + "wanted", res.Wanted, "unnameable", res.Unnameable, "unresolved", res.Unresolved, ) diff --git a/internal/reacquisition/sweeper_test.go b/internal/reacquisition/sweeper_test.go index 0bb7c732..8d8a0899 100644 --- a/internal/reacquisition/sweeper_test.go +++ b/internal/reacquisition/sweeper_test.go @@ -13,9 +13,6 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" - "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" "git.fabledsword.com/bvandeusen/minstrel/internal/tags" ) @@ -37,24 +34,25 @@ func newSweepPool(t *testing.T) *pgxpool.Pool { } t.Cleanup(pool.Close) dbtest.ResetDB(t, pool) - if _, err := pool.Exec(context.Background(), "DELETE FROM lidarr_requests"); err != nil { - t.Fatalf("reset requests: %v", err) - } return pool } // lostAlbum seeds an album whose only track has been missing for two days, -// named by MusicBrainz release id and (when group is set) release group. +// named by MusicBrainz release id (when release is set) and release group +// (when group is set). func lostAlbum(t *testing.T, pool *pgxpool.Pool, title, release, group string) pgtype.UUID { t.Helper() ctx := context.Background() q := dbq.New(pool) - artistMBID := "artist-" + release + artistMBID := "artist-" + title ar, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: title + " Artist", SortName: title, Mbid: &artistMBID}) if err != nil { t.Fatal(err) } - params := dbq.UpsertAlbumParams{Title: title, SortTitle: title, ArtistID: ar.ID, Mbid: &release} + params := dbq.UpsertAlbumParams{Title: title, SortTitle: title, ArtistID: ar.ID} + if release != "" { + params.Mbid = &release + } if group != "" { params.ReleaseGroupMbid = &group } @@ -64,7 +62,7 @@ func lostAlbum(t *testing.T, pool *pgxpool.Pool, title, release, group string) p } tr, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ Title: "Lost", AlbumID: al.ID, ArtistID: ar.ID, DurationMs: 180000, - FilePath: "/music/" + release + "/01.flac", FileSize: 1, FileFormat: "flac", + FilePath: "/music/" + title + "/01.flac", FileSize: 1, FileFormat: "flac", }) if err != nil { t.Fatal(err) @@ -75,39 +73,44 @@ func lostAlbum(t *testing.T, pool *pgxpool.Pool, title, release, group string) p return al.ID } -// #5241: re-acquisition asks Lidarr for the album's release group, never its -// release id. A tag-supplied group is used as is; otherwise MusicBrainz names -// it and the answer is cached onto the album; an album MusicBrainz cannot -// name is skipped without spending an attempt. -func TestSweep_RequestsAlbumsByReleaseGroup_Integration(t *testing.T) { +// heldWant puts an album on the wanted list, held since ago. +func heldWant(t *testing.T, pool *pgxpool.Pool, group string, album pgtype.UUID, ago string) { + t.Helper() + if _, err := pool.Exec(context.Background(), ` + INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, album_id, fulfilled_at) + VALUES ($1, 'a', 'A', 'T', 'library', $2, now() - $3::interval)`, group, album, ago); err != nil { + t.Fatal(err) + } +} + +// Only an album on the wanted list is fetched back: one taken off it, or +// dropped into the library by hand, is not. An album on the list is wanted +// by its release group, never its release id: a tag-supplied +// group is used as is; otherwise MusicBrainz names it and the answer is +// cached onto the album; an album MusicBrainz cannot name is skipped and +// tried again next pass. +func TestSweep_WantsBackOnlyAlbumsOnTheList_Integration(t *testing.T) { pool := newSweepPool(t) ctx := context.Background() logger := slog.New(slog.NewTextHandler(io.Discard, nil)) q := dbq.New(pool) - admin, err := q.CreateUser(ctx, dbq.CreateUserParams{ - Username: dbtest.TestUserPrefix + "sweepadmin", PasswordHash: "x", ApiTokenHash: "x", IsAdmin: true, - }) - if err != nil { - t.Fatal(err) - } settings, err := NewSettingsService(ctx, pool, logger) if err != nil { t.Fatal(err) } - cfg := Defaults - cfg.AutoApprove = false - if _, err := settings.Set(ctx, cfg); err != nil { - t.Fatal(err) - } - tagged := lostAlbum(t, pool, "Tagged", "release-tagged", "group-tagged") resolved := lostAlbum(t, pool, "Resolved", "release-resolved", "") unknown := lostAlbum(t, pool, "Unknown", "release-unknown", "") + _ = lostAlbum(t, pool, "Off The List", "release-off", "group-off") + // Matched to a group with no release id: still wanted back. + groupOnly := lostAlbum(t, pool, "Group Only", "", "group-only") + heldWant(t, pool, "group-tagged", tagged, "3 days") + heldWant(t, pool, "group-only", groupOnly, "3 days") + heldWant(t, pool, "group-resolved", resolved, "3 days") + heldWant(t, pool, "group-unknown", unknown, "3 days") - requests := lidarrrequests.NewService(pool, lidarrconfig.New(pool), nil, nil) - s := NewSweeper(pool, settings, requests, logger) - s.SetNotifier(notifications.New(pool, nil, nil)) + s := NewSweeper(pool, settings, logger) asked := map[string]int{} s.releaseGroup = func(_ context.Context, release string) (string, error) { asked[release]++ @@ -120,27 +123,27 @@ func TestSweep_RequestsAlbumsByReleaseGroup_Integration(t *testing.T) { t.Fatalf("sweep: %v", err) } - requested := func(album pgtype.UUID) (string, bool) { + wantFor := func(group string, album pgtype.UUID) { t.Helper() - var mbid *string - err := pool.QueryRow(ctx, ` -SELECT lr.lidarr_album_mbid - FROM missing_reacquisitions r - JOIN lidarr_requests lr ON lr.id = r.last_request_id - WHERE r.album_id = $1`, album).Scan(&mbid) - if err != nil || mbid == nil { - return "", false + w, err := q.GetWantByReleaseGroup(ctx, group) + if err != nil { + t.Fatalf("no want for %s: %v", group, err) + } + if w.Reason != "missing" || w.AlbumID != album || w.FulfilledAt.Valid { + t.Errorf("want %s = %+v: an open missing want about the album", group, w) } - return *mbid, true } - if got, ok := requested(tagged); !ok || got != "group-tagged" { - t.Errorf("tagged album requested as %q (ok=%v), want group-tagged", got, ok) + wantFor("group-tagged", tagged) + wantFor("group-resolved", resolved) + wantFor("group-only", groupOnly) + if w, _ := q.GetWantByReleaseGroup(ctx, "group-unknown"); !w.FulfilledAt.Valid { + t.Error("an album MusicBrainz cannot name was reopened under the wrong group") } - if asked["release-tagged"] != 0 { - t.Error("MusicBrainz asked about an album whose tags named its group") + if _, err := q.GetWantByReleaseGroup(ctx, "group-off"); err == nil { + t.Error("an album not on the list was wanted back") } - if got, ok := requested(resolved); !ok || got != "group-resolved" { - t.Errorf("resolved album requested as %q (ok=%v), want group-resolved", got, ok) + if asked["release-tagged"] != 0 || asked["release-off"] != 0 { + t.Errorf("MusicBrainz asked %v: not about a tagged album, nor one off the list", asked) } var cached *string if err := pool.QueryRow(ctx, "SELECT release_group_mbid FROM albums WHERE id = $1", resolved).Scan(&cached); err != nil { @@ -149,32 +152,48 @@ SELECT lr.lidarr_album_mbid if cached == nil || *cached != "group-resolved" { t.Errorf("resolved group not cached on the album: %v", cached) } - var attempts int - if err := pool.QueryRow(ctx, "SELECT count(*) FROM missing_reacquisitions WHERE album_id = $1", unknown).Scan(&attempts); err != nil { + + // The next pass finds both already wanted and asks nothing new. + asked = map[string]int{} + if err := s.SweepOnce(ctx); err != nil { t.Fatal(err) } - if attempts != 0 { - t.Errorf("unresolvable album spent an attempt (%d state rows)", attempts) + if asked["release-resolved"] != 0 || asked["release-unknown"] != 1 { + t.Errorf("second pass asked MusicBrainz %v: only the unresolved album again", asked) } - // M489: the two requests filed without auto-approval wait on an admin, - // so each reaches the admins' inbox; the unresolvable album filed none. - inbox, err := q.ListNotifications(ctx, dbq.ListNotificationsParams{UserID: admin.ID, PageLimit: 10}) +} + +// A want held after the files went missing means the library has a full copy +// despite the missing rows: the album is not wanted again. One held +// before they went missing is reopened. +func TestSweep_HeldWantsAreReopenedOnlyForANewLoss_Integration(t *testing.T) { + pool := newSweepPool(t) + ctx := context.Background() + logger := slog.New(slog.NewTextHandler(io.Discard, nil)) + settings, err := NewSettingsService(ctx, pool, logger) if err != nil { t.Fatal(err) } - if len(inbox) != 2 { - t.Errorf("admin inbox holds %d notifications, want 2", len(inbox)) - } - for _, n := range inbox { - if n.Kind != string(notifications.KindRequestPending) { - t.Errorf("admin inbox holds %q, want request_pending", n.Kind) + stale := lostAlbum(t, pool, "Stale", "release-stale", "group-stale") + lost := lostAlbum(t, pool, "Lost Again", "release-lost", "group-lost") + for _, c := range []struct { + group, heldAgo string + album pgtype.UUID + }{{"group-stale", "1 hour", stale}, {"group-lost", "3 days", lost}} { + if _, err := pool.Exec(ctx, ` + INSERT INTO wants (release_group_mbid, artist_mbid, artist_name, title, reason, album_id, fulfilled_at) + VALUES ($1, 'a', 'A', 'T', 'request', $2, now() - $3::interval)`, c.group, c.album, c.heldAgo); err != nil { + t.Fatal(err) } } - var stray int - if err := pool.QueryRow(ctx, "SELECT count(*) FROM lidarr_requests WHERE lidarr_album_mbid LIKE 'release-%'").Scan(&stray); err != nil { + if err := NewSweeper(pool, settings, logger).SweepOnce(ctx); err != nil { t.Fatal(err) } - if stray != 0 { - t.Errorf("%d request(s) named an album by its release id", stray) + q := dbq.New(pool) + if w, _ := q.GetWantByReleaseGroup(ctx, "group-stale"); !w.FulfilledAt.Valid { + t.Error("a want held since the loss was reopened") + } + if w, _ := q.GetWantByReleaseGroup(ctx, "group-lost"); w.FulfilledAt.Valid || w.Reason != "missing" { + t.Errorf("a want held before the loss should reopen as missing: %+v", w) } } diff --git a/internal/recommendation/candidate_tags_integration_test.go b/internal/recommendation/candidate_tags_integration_test.go index 19cefbc8..ef24982a 100644 --- a/internal/recommendation/candidate_tags_integration_test.go +++ b/internal/recommendation/candidate_tags_integration_test.go @@ -10,11 +10,11 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Slice 5 (#2376): the candidate-artist tag cache for out-of-library Discover -// candidates. These cover the SQL rather than the provider chain — the -// eligibility query is the piece with real risk in it (a GROUP BY over a -// many-rows-per-candidate table, a LEFT JOIN to bookkeeping, and two exclusion -// branches), and it's consumed by this package's slice-6 ranking. +// The candidate-artist tag cache for out-of-library Discover candidates. These +// cover the SQL rather than the provider chain — the eligibility query is the +// piece with real risk in it (a GROUP BY over a many-rows-per-candidate table, +// a LEFT JOIN to bookkeeping, and two exclusion branches), and it feeds this +// package's taste-tag ranking. // // Fixtures live here because the harness and seedUnmatched do. @@ -262,7 +262,7 @@ func TestCandidateTags_GcDropsOrphansAndKeepsLiveOnes(t *testing.T) { } } -// Coverage is the operator's window onto the honest ceiling: processed vs +// Coverage is the admin's window onto the honest ceiling: processed vs // actually-tagged. A big gap means thin upstream data, not a broken worker. func TestCandidateTags_CoverageCountsProcessedAndTagged(t *testing.T) { pool := newPool(t) @@ -282,7 +282,7 @@ func TestCandidateTags_CoverageCountsProcessedAndTagged(t *testing.T) { } } -// --- Slice 6 (#2377): the taste-tag blend, end to end --- +// --- The taste-tag blend, end to end --- // // The pure tests in tagoverlap_test.go cover the scoring maths. These cover the // wiring the pure tests cannot reach: that loadTagInputs actually reads both @@ -348,7 +348,7 @@ func TestSuggestArtists_TasteTagMatchOvertakesStrongerSimilarity(t *testing.T) { } // The same fixture with the knob at 0 must return pure similarity order — the -// operator's off switch, verified against a real DB rather than assumed. +// off switch, verified against a real DB rather than assumed. func TestSuggestArtists_ZeroTagWeightKeepsSimilarityOrder(t *testing.T) { pool := newPool(t) user := seedUser(t, pool, "alice") @@ -367,7 +367,7 @@ func TestSuggestArtists_ZeroTagWeightKeepsSimilarityOrder(t *testing.T) { } // A user with no taste profile must still get a full deck in similarity order: -// the cold-start path, which is every user's first days (rule #131). +// the cold-start path, which is every user's first days. func TestSuggestArtists_NoTasteTagsStillReturnsSimilarityOrder(t *testing.T) { pool := newPool(t) user := seedUser(t, pool, "alice") @@ -388,7 +388,7 @@ func TestSuggestArtists_NoTasteTagsStillReturnsSimilarityOrder(t *testing.T) { } // An untagged candidate must never be dropped or sunk just because another -// candidate has tags — permanently-partial coverage (#2376) must not become a +// candidate has tags — permanently-partial coverage must not become a // permanent ranking penalty. func TestSuggestArtists_UntaggedCandidateSurvivesAlongsideTagged(t *testing.T) { pool := newPool(t) diff --git a/internal/recommendation/candidates.go b/internal/recommendation/candidates.go index ecd1d7e2..ba388f5d 100644 --- a/internal/recommendation/candidates.go +++ b/internal/recommendation/candidates.go @@ -82,17 +82,17 @@ type CandidateSourceLimits struct { TagOverlap int LikesOverlap int RandomFill int - // TasteOverlap (#796 phase 2b): tracks by the user's top positively- + // TasteOverlap: tracks by the user's top positively- // weighted taste-profile artists. 0 disables the arm (e.g. cold-start // users have an empty profile, so it contributes nothing anyway). TasteOverlap int - // UserCoplay (#1533): tracks by artists co-played across the instance + // UserCoplay: tracks by artists co-played across the instance // with the seed's artist (source='user_cooccurrence'). Empty on // single-user servers, so it contributes nothing there. UserCoplay int } -// DefaultCandidateSourceLimits returns the v1 hardcoded constants per spec. +// DefaultCandidateSourceLimits returns the hardcoded per-source limits. func DefaultCandidateSourceLimits() CandidateSourceLimits { return CandidateSourceLimits{ LBSimilar: 30, @@ -105,21 +105,18 @@ func DefaultCandidateSourceLimits() CandidateSourceLimits { } } -// SongsLikeCandidateSourceLimits is the pool shape for "Songs like {X}" -// (#3881). Same total size as the default (~170) — the composition is what -// changes, shifted hard toward arms that actually measure distance from the -// seed. +// SongsLikeCandidateSourceLimits is the pool shape for "Songs like {X}". +// Same total size as the default (~170) — the composition is what changes, +// shifted hard toward arms that actually measure distance from the seed. // -// The surface answers "what sounds like THIS", and it shared the default -// pool with For-You, which answers the much broader "what will they enjoy -// today". Under the default, 50 of ~170 candidates carried sim_score = 0 by -// construction — `taste_overlap` (tracks by the user's top taste artists) and -// `random_fill` (literally any track not already in the pool), both of which -// are seed-INDEPENDENT. Nearly a third of the pool had no relationship to the -// seed at all, and the operator saw it: "I was getting a seeming wide variety -// of music from each one when I was hoping to stay in a certain neighborhood." +// The surface answers "what sounds like THIS", where For-You answers the much +// broader "what will they enjoy today". In the default pool, 50 of ~170 +// candidates carry sim_score = 0 by construction — `taste_overlap` (tracks by +// the user's top taste artists) and `random_fill` (any track not already in +// the pool), both seed-INDEPENDENT — so nearly a third of it has no +// relationship to the seed. // -// TIERED, per rule 131 — a system mix degrades, it never vanishes: +// TIERED — a system mix degrades, it never vanishes: // // tier 1 lb_similar real track-level similarity. The exact promise. // tier 2 similar_artist, tag_overlap, coplay, likes_overlap @@ -127,63 +124,44 @@ func DefaultCandidateSourceLimits() CandidateSourceLimits { // tier 3 taste_overlap, random_fill // seed-independent. The floor, and nothing more. // -// The tiering is enforced by SCORE rather than by a fallback ladder: tier-3 -// arms carry sim_score 0, so under SongsLike weights (SimilarityWeight 4.0, -// everything seed-independent demoted) they rank below any real match and -// surface only when tiers 1–2 cannot fill the mix. That is the rule's -// "fill from tier 1 first, reach down only when a tier cannot fill". +// The tiering is enforced by SCORE rather than by a fallback ladder: each tier +// holds its own sim_score band (recommendation.sql; tier 1 above 0.5, tier 2 +// at or below it), and tier-3 arms carry sim_score 0, so under the SongsLike +// weights (SimilarityWeight 4.0, everything seed-independent demoted) they +// rank below any real match and surface only when tiers 1–2 cannot fill the +// mix: fill from tier 1 first, reach down only when a tier cannot fill. // -// Which is exactly why tier 3 is REDUCED rather than removed. Zeroing those -// two arms was the first instinct and it is the vanish-or-nothing shape rule -// 131 exists to forbid: a seed whose artist has thin ListenBrainz coverage -// would produce a short mix or none at all, and "no playlist" is a worse -// answer than "a few tracks further from the seed than we would like". +// Tier 3 is REDUCED, never removed. Zeroing those two arms would leave a seed +// whose artist has thin ListenBrainz coverage with a short mix or none at +// all, and "no playlist" is a worse answer than "a few tracks further from +// the seed than we would like". The trim is safe because those arms order by +// md5(id || seed): a smaller limit takes a smaller but REPRODUCIBLE slice, so +// same-day rebuilds still agree. // -// The seed-independent arms are trimmed hardest, because on this surface they -// are noise: `taste_overlap` (tracks by the user's top taste artists) and -// `random_fill` (any track not already in the pool) both carry -// `0.0::float8 AS sim_score`, so nearly a third of the default pool had no -// relationship to the seed at all. -// -// THESE TRIMS WERE BLOCKED UNTIL #3889. `likes_overlap` and `random_fill` -// used to end in a bare `ORDER BY random()`, which made their output a stable -// SET only while the limit exceeded the eligible rows — so SHRINKING them -// changed pool membership between same-day rebuilds and broke daily -// determinism. Those arms now order by md5(id || seed), so a smaller limit -// takes a smaller but REPRODUCIBLE slice, and the trim is safe. -// -// Reduced, never removed. Rule 131: the two seed-independent arms are the -// tier-3 floor, and zeroing them would leave a seed with thin ListenBrainz -// coverage producing a short mix or none at all. The weights (SimilarityWeight -// 4.0, everything seed-independent demoted) keep them ranked last, so they -// surface only when the closer tiers cannot fill the mix. -// -// likes_overlap is cut hardest of the tier-2 arms for a specific reason: its -// SQL assigns a FLAT 0.6 sim_score (recommendation.sql) rather than measuring -// anything. It is a collaborative signal wearing similarity's clothes, and a -// raised SimilarityWeight amplifies it — if real ListenBrainz scores commonly -// land below 0.6 it would outrank genuine matches. Halved pending the -// fill-rate measurement in #3879; the honest fix is to stop it claiming a -// similarity score it never computed. +// likes_overlap is cut hardest of the tier-2 arms: its SQL assigns a FLAT 0.3 +// sim_score (recommendation.sql) rather than measuring anything. It is a +// collaborative signal wearing similarity's clothes, and a raised +// SimilarityWeight amplifies it. The tier bands keep it below every +// ListenBrainz match. func SongsLikeCandidateSourceLimits() CandidateSourceLimits { return CandidateSourceLimits{ LBSimilar: 60, // tier 1 — doubled; the only arm that measures the seed SimilarArtist: 40, // tier 2 — raised; growing is always safe TagOverlap: 20, // tier 2 UserCoplay: 20, // tier 2 - LikesOverlap: 10, // tier 2, halved — flat 0.6 sim_score, see above + LikesOverlap: 10, // tier 2, halved — flat 0.3 sim_score, see above TasteOverlap: 10, // tier 3 floor — halved, not removed RandomFill: 10, // tier 3 floor — cut hard, never to zero } } -// LoadCandidatesFromSimilarity is M4c's primary candidate-pool loader. +// LoadCandidatesFromSimilarity is the primary candidate-pool loader. // 5-way SQL UNION (LB-similar / similar-artist tracks / MB-tag overlap / // likes-overlap / random fill) + dedup-by-max sim_score. Returns // []Candidate (same shape as LoadCandidates) so Shuffle is unchanged. // // orderSeed decides whether the randomised arms repeat their draw — see -// Column12 below and #3889. Pass a stable per-(user, day) value where the +// Column12 below. Pass a stable per-(user, day) value where the // selection must be reproducible, and a varying one where it should not be. // // Caller (radio handler) falls back to LoadCandidates on error. @@ -212,11 +190,10 @@ func LoadCandidatesFromSimilarity( Limit_5: int32(limits.RandomFill), Limit_6: int32(limits.TasteOverlap), Limit_7: int32(limits.UserCoplay), - // #3889. Four arms used to end in a bare ORDER BY random(), which made - // their output a stable SET only while the limit exceeded the eligible - // rows. They now order by md5(id || this), so the caller decides - // whether the draw repeats: a per-(user, day) seed for the system - // mixes that promise daily determinism, a fresh one per radio request. + // The randomised arms order by md5(id || this) rather than random(), so + // the caller decides whether the draw repeats: a per-(user, day) seed + // for the system mixes that promise daily determinism, a fresh one per + // radio request. Column12: orderSeed, }) if err != nil { @@ -281,7 +258,7 @@ func LoadCandidatesFromSimilarity( } // loadCandidateMoods fetches the enriched tags for the given candidate tracks -// and reduces each to its canonical mood buckets (internal/mood, #1534), so the +// and reduces each to its canonical mood buckets (internal/mood), so the // scorer can apply the mood facet per candidate. Tracks with no mood-word tags // are absent from the map (→ no mood signal). Empty input short-circuits. func loadCandidateMoods( diff --git a/internal/recommendation/candidates_v2_test.go b/internal/recommendation/candidates_v2_test.go index 29dfc754..e59f17f7 100644 --- a/internal/recommendation/candidates_v2_test.go +++ b/internal/recommendation/candidates_v2_test.go @@ -3,6 +3,7 @@ package recommendation import ( "context" "fmt" + "maps" "reflect" "sort" "strings" @@ -69,8 +70,9 @@ func TestLoadCandidatesFromSimilarity_LBSimilarSourceContributes(t *testing.T) { if found == nil { t.Fatal("LB-similar target missing from candidates") } - if found.Inputs.SimilarityScore < 0.84 || found.Inputs.SimilarityScore > 0.86 { - t.Errorf("LB-similar SimilarityScore = %v, want ~0.85", found.Inputs.SimilarityScore) + // Tier 1's band: 0.5 + 0.5 × 0.85. + if found.Inputs.SimilarityScore < 0.92 || found.Inputs.SimilarityScore > 0.93 { + t.Errorf("LB-similar SimilarityScore = %v, want ~0.925 (0.5 + 0.5 × 0.85)", found.Inputs.SimilarityScore) } } @@ -117,9 +119,9 @@ func TestLoadCandidatesFromSimilarity_TagOverlapContributes(t *testing.T) { } for _, c := range got { if c.Track.ID == target.ID { - // Seed has 2 tags; target shares 1 → jaccard 1/2 = 0.5. - if c.Inputs.SimilarityScore < 0.49 || c.Inputs.SimilarityScore > 0.51 { - t.Errorf("tag-overlap SimilarityScore = %v, want ~0.5", c.Inputs.SimilarityScore) + // Seed has 2 tags; target shares 1 → 1/2, in tier 2's band → 0.25. + if c.Inputs.SimilarityScore < 0.24 || c.Inputs.SimilarityScore > 0.26 { + t.Errorf("tag-overlap SimilarityScore = %v, want ~0.25 (0.5 × 1/2)", c.Inputs.SimilarityScore) } return } @@ -144,10 +146,10 @@ func TestLoadCandidatesFromSimilarity_LikesOverlapContributes(t *testing.T) { } for _, c := range got { if c.Track.ID == liked.ID { - // Both tracks tagged "Rock" → jaccard 1/1 = 1.0 from tag-overlap. - // likes-overlap = 0.6. Max wins = 1.0. - if c.Inputs.SimilarityScore < 0.59 { - t.Errorf("likes-overlap candidate SimilarityScore = %v, want ≥ 0.6", c.Inputs.SimilarityScore) + // Both tracks tagged "Rock" → 0.5 × 1/1 = 0.5 from tag-overlap. + // likes-overlap = 0.3. Max wins = 0.5. + if c.Inputs.SimilarityScore < 0.29 { + t.Errorf("likes-overlap candidate SimilarityScore = %v, want ≥ 0.3", c.Inputs.SimilarityScore) } return } @@ -243,8 +245,8 @@ func TestLoadCandidatesFromSimilarity_DedupTakesMaxScore(t *testing.T) { seed := f.tracks[0] target := f.tracks[1] helperSetTrackGenre(t, f, seed.ID, "Rock") - helperSetTrackGenre(t, f, target.ID, "Rock") // jaccard 1/1 = 1.0 from tag-overlap - helperLBSimilarity(t, f, seed.ID, target.ID, 0.5) // weaker LB signal + helperSetTrackGenre(t, f, target.ID, "Rock") // 0.5 × 1/1 = 0.5 from tag-overlap + helperLBSimilarity(t, f, seed.ID, target.ID, 0.1) // a weak LB edge → 0.5 + 0.05 got, err := LoadCandidatesFromSimilarity( context.Background(), f.q, f.user, seed.ID, 1, SessionVector{Seed: true}, nil, defaultLimits(), "test-seed", ) @@ -255,9 +257,10 @@ func TestLoadCandidatesFromSimilarity_DedupTakesMaxScore(t *testing.T) { for _, c := range got { if c.Track.ID == target.ID { count++ - // tag-overlap (1.0) wins over LB (0.5) per max() — expect ≥ 0.99 - if c.Inputs.SimilarityScore < 0.99 { - t.Errorf("dedup max SimilarityScore = %v, want ≥ 0.99 (tag-overlap should win)", c.Inputs.SimilarityScore) + // max() keeps LB's 0.55. Even a weak ListenBrainz edge is tier 1 and + // outranks a full tag match. + if c.Inputs.SimilarityScore < 0.54 || c.Inputs.SimilarityScore > 0.56 { + t.Errorf("dedup max SimilarityScore = %v, want ~0.55 (the LB edge's band should win)", c.Inputs.SimilarityScore) } } } @@ -266,7 +269,7 @@ func TestLoadCandidatesFromSimilarity_DedupTakesMaxScore(t *testing.T) { } } -// TestLoadCandidatesFromSimilarity_TasteOverlapArm (#796 phase 2b): a track by +// TestLoadCandidatesFromSimilarity_TasteOverlapArm: a track by // a positively-weighted taste-profile artist enters the pool via taste_overlap // even with every other arm disabled; a negatively-weighted artist's track does // not (the WHERE weight > 0 filter). @@ -336,22 +339,16 @@ func TestLoadCandidatesFromSimilarity_EmptyLibrary_NoError(t *testing.T) { } } -// The randomised arms must draw REPRODUCIBLY for a given seed (#3889). +// The randomised arms must draw REPRODUCIBLY for a given seed. // -// Four arms used to end in a bare `ORDER BY random()`. That returned a stable -// set only while the arm's LIMIT exceeded the rows eligible for it — at that -// point it returned all of them and the order stopped mattering, because the -// caller sorts by track id before scoring. Below that threshold it returned a -// random SUBSET, so two calls drew different candidates. +// An arm ordered by bare `random()` returns a stable set only while its LIMIT +// exceeds the rows eligible for it (it returns all of them, and the caller +// sorts by track id). Below that it returns a random SUBSET, so two calls +// draw different candidates — invisible, because a mix that changes after a +// refresh looks like a feature. // -// It therefore held by ACCIDENT, and only for libraries smaller than the -// limits. Any real library is larger, so same-day rebuilds had been drawing -// different mixes since the arm was written — invisible, because a mix that -// changes after a refresh looks like a feature. -// -// Limits deliberately smaller than the fixture, because that is the only -// regime where the bug existed at all: with limits above the eligible count -// the old code passes this too. +// The limits are deliberately smaller than the fixture: with limits above the +// eligible count, an unseeded arm passes this too. func TestLoadCandidatesFromSimilarity_SameSeedDrawsTheSameSet(t *testing.T) { f := newFixture(t, 12) seed := f.tracks[0] @@ -430,10 +427,9 @@ func TestLoadCandidatesFromSimilarity_DifferentSeedsCanDrawDifferently(t *testin } } -// #5297: an artist-level arm takes a track from each related artist before a -// second from any. Ordered by artist score alone, the closest artist's -// catalogue filled the whole LIMIT; on the operator's library the -// similar_artists arm held one artist for every seed measured (#3879). +// An artist-level arm takes a track from each related artist before a second +// from any. Ordered by artist score alone, the closest artist's catalogue +// would fill the whole LIMIT. func TestLoadCandidatesFromSimilarity_ArtistArmsRoundRobin(t *testing.T) { for _, source := range []string{"listenbrainz", "user_cooccurrence"} { t.Run(source, func(t *testing.T) { @@ -495,3 +491,90 @@ func TestLoadCandidatesFromSimilarity_ArtistArmsRoundRobin(t *testing.T) { }) } } + +// The taste arm takes one track from each taste artist before a second from +// any. Ordered by weight then track id, the strongest artist's catalogue would +// fill the arm. +func TestLoadCandidatesFromSimilarity_TasteArmRoundRobinsAcrossArtists(t *testing.T) { + f := newFixture(t, 1) + ctx := context.Background() + seed := f.tracks[0] + artistOf := map[[16]byte]string{} + for i, weight := range []float64{9, 8, 7} { + name := fmt.Sprintf("Taste %d", i) + ar, err := f.q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: name, SortName: name}) + if err != nil { + t.Fatal(err) + } + al, err := f.q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: name, SortTitle: name, ArtistID: ar.ID}) + if err != nil { + t.Fatal(err) + } + // More tracks per artist than the arm's limit. + for j := 0; j < 5; j++ { + tr, err := f.q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: fmt.Sprintf("%s #%d", name, j), AlbumID: al.ID, ArtistID: ar.ID, + FilePath: fmt.Sprintf("/tmp/taste-%d-%d.flac", i, j), DurationMs: 180_000, + }) + if err != nil { + t.Fatal(err) + } + artistOf[tr.ID.Bytes] = name + } + if _, err := f.pool.Exec(ctx, + `INSERT INTO taste_profile_artists (user_id, artist_id, weight) VALUES ($1, $2, $3)`, + f.user, ar.ID, weight); err != nil { + t.Fatal(err) + } + } + got, err := LoadCandidatesFromSimilarity( + ctx, f.q, f.user, seed.ID, 1, SessionVector{Seed: true}, nil, CandidateSourceLimits{TasteOverlap: 3}, "test-seed", + ) + if err != nil { + t.Fatalf("load: %v", err) + } + artists := map[string]int{} + for _, c := range got { + if name, ok := artistOf[c.Track.ID.Bytes]; ok { + artists[name]++ + } + } + if len(got) != 3 || len(artists) != 3 { + t.Errorf("taste arm of 3 drew %d candidates from %d artists (%v), want one from each of 3", + len(got), len(artists), artists) + } +} + +// Tracks tied on tag overlap are taken in the order seed's order: the same +// seed takes the same slice, another seed another. With no tie-break a +// one-tag seed's LIMIT would take whichever rows Postgres reached first. +func TestLoadCandidatesFromSimilarity_TagTiesFollowTheOrderSeed(t *testing.T) { + f := newFixture(t, 21) + ctx := context.Background() + for _, tr := range f.tracks { + helperSetTrackGenre(t, f, tr.ID, "Rock") + } + pick := func(orderSeed string) map[[16]byte]bool { + got, err := LoadCandidatesFromSimilarity( + ctx, f.q, f.user, f.tracks[0].ID, 1, SessionVector{Seed: true}, nil, CandidateSourceLimits{TagOverlap: 5}, orderSeed, + ) + if err != nil { + t.Fatalf("load: %v", err) + } + out := map[[16]byte]bool{} + for _, c := range got { + out[c.Track.ID.Bytes] = true + } + return out + } + a, again, b := pick("day-a"), pick("day-a"), pick("day-b") + if len(a) != 5 { + t.Fatalf("tag arm of 5 drew %d", len(a)) + } + if !maps.Equal(a, again) { + t.Error("the same order seed took a different slice of tied tracks") + } + if maps.Equal(a, b) { + t.Error("two order seeds took the same slice of 20 tied tracks: the ties are not following the seed") + } +} diff --git a/internal/recommendation/context.go b/internal/recommendation/context.go index 6f94345d..ee676576 100644 --- a/internal/recommendation/context.go +++ b/internal/recommendation/context.go @@ -15,7 +15,7 @@ import ( const contextAffinityShrinkage = 5.0 // ContextAffinity is the read-side map of per-artist time-of-day/weekday -// affinity for the CURRENT context (#1531): artist_id → score in [-1, +1]. +// affinity for the CURRENT context: artist_id → score in [-1, +1]. // Absent artists (no play history) score 0, so cold-start candidates stay // neutral. The zero value is a valid all-neutral affinity. type ContextAffinity struct { @@ -29,7 +29,7 @@ func (c ContextAffinity) Affinity(artistID pgtype.UUID) float64 { // LoadContextAffinity computes each artist's affinity for the user's CURRENT // context cell — daypart × weekday, narrowed by deviceClass when non-empty -// (#1551; radio passes the current device, the daily mixes pass "" for a +// (radio passes the current device, the daily mixes pass "" for a // device-agnostic cell). For every artist with completed plays in the window it // compares the share of that artist's plays that fall in the current cell // against the user's overall baseline share, shrinking sparse artists toward diff --git a/internal/recommendation/context_device_integration_test.go b/internal/recommendation/context_device_integration_test.go index 484c5d62..7d76ac2a 100644 --- a/internal/recommendation/context_device_integration_test.go +++ b/internal/recommendation/context_device_integration_test.go @@ -33,7 +33,7 @@ func insertPlayEventDevice( // TestLoadContextAffinity_DeviceNarrowing seeds one artist played only on // mobile and another only on web (all in the current time cell), and verifies -// the current-device parameter narrows the affinity cell (#1551): on 'mobile' +// the current-device parameter narrows the affinity cell: on 'mobile' // the mobile artist out-scores the web artist and vice-versa, while the // device-agnostic (”) pass treats them equally. func TestLoadContextAffinity_DeviceNarrowing(t *testing.T) { diff --git a/internal/recommendation/home_integration_test.go b/internal/recommendation/home_integration_test.go index c4353f28..02b94e8f 100644 --- a/internal/recommendation/home_integration_test.go +++ b/internal/recommendation/home_integration_test.go @@ -158,7 +158,7 @@ func TestHomeData_MostPlayed_RanksByCount_HonorsQuarantine(t *testing.T) { insertPlayEvent(t, pool, user.ID, tQuar.ID, now) } if _, err := pool.Exec(context.Background(), - `INSERT INTO lidarr_quarantine (user_id, track_id, reason) VALUES ($1, $2, 'bad_rip')`, + `INSERT INTO quarantine (user_id, track_id, reason) VALUES ($1, $2, 'bad_rip')`, user.ID, tQuar.ID, ); err != nil { t.Fatalf("insert quarantine: %v", err) diff --git a/internal/recommendation/library_scale.go b/internal/recommendation/library_scale.go index a5298244..1252e3a1 100644 --- a/internal/recommendation/library_scale.go +++ b/internal/recommendation/library_scale.go @@ -11,32 +11,26 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) -// Scaling the candidate pool to the library (#3880). +// Scaling the candidate pool to the library. // -// DefaultCandidateSourceLimits returns what its own comment calls "the v1 -// hardcoded constants per spec" — ~170 candidates, identical for a 500-track -// library and a 100,000-track one. Operator, 2026-09-10: "is the pool that we -// draw from somehow scaled to the amount of music in the library... my earlier -// understanding of the tuning and work may have been skewed by what was in my -// library." -// -// It was not, and the consequence compounds. The pool samples a shrinking -// FRACTION of the library as it grows — 17% of 1,000 tracks, 1.7% of 10,000, -// 0.17% of 100,000 — so the ceiling on how much of a collection can ever -// surface stays flat while the collection does not. RandomFill, whose entire -// job is exploration, becomes a thinner and noisier slice of a more diverse -// corpus at exactly the moment diversity rises. +// DefaultCandidateSourceLimits is ~170 candidates, whatever the library's +// size. Unscaled, the pool samples a shrinking FRACTION of the library as it +// grows — 17% of 1,000 tracks, 1.7% of 10,000, 0.17% of 100,000 — so the +// ceiling on how much of a collection can ever surface stays flat while the +// collection does not, and RandomFill, whose job is exploration, becomes a +// thinner slice of a more diverse corpus. const ( // libraryScaleReference is the library size at which the base limits // apply unchanged. Below it nothing scales, so small libraries keep // today's behaviour exactly. // - // ASSUMED, NOT MEASURED. The size the v1 constants were actually tuned - // against is unrecorded; 5,000 is a plausible mid-size library and a - // deliberately conservative place to start growing. #3879 should replace - // this with the real number, and until it does, this constant is the - // single thing to change. + // An assumption, not a measurement: 5,000 is a plausible mid-size library + // and a deliberately conservative place to start growing. On a large + // library the arms the scale grows (similar_artists, tag_overlap, + // likes_overlap, taste_overlap) hold far more rows than it asks for, while + // lb_similar rarely reaches even its base limit. Change this if a library + // shows a scaled arm running dry or a pool too slow. libraryScaleReference = 5000 // maxLibraryScale bounds growth so a very large library does not turn @@ -51,9 +45,9 @@ const ( // Square-root growth keeps the pool meaningfully proportional while staying // bounded: 1x at the reference, 2x at four times it, 4x at sixteen times. // -// Never returns below 1: the base limits are a floor, not a midpoint. That -// also keeps this safe against #3889 — growing an arm ordered by unseeded -// random() is fine, shrinking one is what breaks same-day determinism. +// Never returns below 1: the base limits are a floor, not a midpoint. Growing +// an arm ordered by unseeded random() is harmless; shrinking one would break +// same-day determinism. func libraryScale(libraryTracks int64) float64 { if libraryTracks <= libraryScaleReference { return 1.0 @@ -101,8 +95,7 @@ func ScaleForLibrary(base CandidateSourceLimits, libraryTracks int64) CandidateS // CountLibraryTracks returns the whole library's track count. // // Reuses CountTracksMatching with an empty pattern — `title ILIKE '%%'` -// matches every row — rather than adding a query, because a new one would -// need sqlc regeneration (rule 28, and see #3889 for where that blocks). +// matches every row — rather than adding a query. // The user id is left invalid so the quarantine anti-join is skipped: this // number sizes a pool, and a per-user view of it would be false precision. func CountLibraryTracks(ctx context.Context, q *dbq.Queries) (int64, error) { @@ -143,11 +136,11 @@ func NewLibrarySize(now func() time.Time) *LibrarySize { // Get returns the cached size, refreshing through load when stale. // // A NIL RECEIVER IS VALID and means "no cache": the count still runs, it is -// just not memoised. That is deliberate rather than defensive habit. The api -// handlers struct is built directly by a dozen tests that cannot know about -// every field, and a nil here previously panicked inside a radio request — -// turning a missing pool-sizing HINT into a 500. Uncached-but-correct is the -// right failure for something whose whole contract is that it degrades. +// just not memoised. The api handlers struct is built directly by a dozen +// tests that cannot know about every field, and a panic here would turn a +// missing pool-sizing HINT into a 500 inside a radio request. +// Uncached-but-correct is the right failure for something whose whole +// contract is that it degrades. func (l *LibrarySize) Get(ctx context.Context, load func(context.Context) (int64, error)) int64 { if l == nil { n, err := loadWithDeadline(ctx, load) @@ -179,8 +172,8 @@ func (l *LibrarySize) Get(ctx context.Context, load func(context.Context) (int64 return l.size } -// loadWithDeadline bounds the count. Rule 156: the caller is a user waiting -// on a radio, and a pool-sizing hint is never worth hanging for. +// loadWithDeadline bounds the count: the caller is a user waiting on a radio, +// and a pool-sizing hint is never worth hanging for. func loadWithDeadline(ctx context.Context, load func(context.Context) (int64, error)) (int64, error) { cctx, cancel := context.WithTimeout(ctx, librarySizeTimeout) defer cancel() diff --git a/internal/recommendation/library_scale_test.go b/internal/recommendation/library_scale_test.go index 918ce409..e06875bb 100644 --- a/internal/recommendation/library_scale_test.go +++ b/internal/recommendation/library_scale_test.go @@ -18,9 +18,7 @@ func TestScaleForLibrary_SmallLibrariesAreUntouched(t *testing.T) { } } -// The complaint, restated as a property: a bigger library must reach further -// into itself. Operator, 2026-09-10 — "is the pool that we draw from somehow -// scaled to the amount of music in the library". +// A bigger library must reach further into itself. func TestScaleForLibrary_BigLibrariesGetABiggerPool(t *testing.T) { base := DefaultCandidateSourceLimits() small := ScaleForLibrary(base, libraryScaleReference) @@ -86,9 +84,9 @@ func TestScaleForLibrary_GrowthIsBounded(t *testing.T) { } } -// Never below the base. The base limits are a floor, not a midpoint — and -// #3889 makes this load-bearing rather than tidy: shrinking an arm ordered by -// unseeded random() changes pool membership between same-day rebuilds. +// Never below the base. The base limits are a floor, not a midpoint, and +// shrinking an arm ordered by unseeded random() would change pool membership +// between same-day rebuilds. func TestScaleForLibrary_NeverShrinksAnArm(t *testing.T) { base := DefaultCandidateSourceLimits() for _, size := range []int64{0, 1, 100, 4999, 5001, 1_000_000} { diff --git a/internal/recommendation/score.go b/internal/recommendation/score.go index 3876519b..d5007cd0 100644 --- a/internal/recommendation/score.go +++ b/internal/recommendation/score.go @@ -1,5 +1,5 @@ -// Package recommendation implements the weighted-shuffle scoring engine -// from spec §6. The Score function is pure and takes an injectable RNG so +// Package recommendation implements the weighted-shuffle scoring engine. +// The Score function is pure and takes an injectable RNG so // tests can pin jitter to deterministic values. package recommendation @@ -20,20 +20,20 @@ type ScoringInputs struct { ContextualMatchScore float64 // [0, 1]; 0 when no signal SimilarityScore float64 // [0, 1]; 0 when no signal (random fill) // TasteMatchScore is the candidate's fit to the user's learned taste - // profile (#796 phase 2), in [-1, +1]: positive draws a track toward the + // profile, in [-1, +1]: positive draws a track toward the // user's taste, negative reflects passive avoidance, 0 when there's no // profile signal (cold start / artist+tags absent from the profile). TasteMatchScore float64 // ContextAffinityScore is the candidate artist's time-of-day/weekday - // affinity for the CURRENT context (#1531), in [-1, +1]: positive when the + // affinity for the CURRENT context, in [-1, +1]: positive when the // artist's plays concentrate in the current daypart × weekday-type cell // more than the user's baseline, negative when under-represented, 0 when // there's no history (cold-start neutral). ContextAffinityScore float64 } -// ScoringWeights are the operator-tunable knobs. Defaults live in -// config.RecommendationConfig and are propagated here per request. +// ScoringWeights are the tunable knobs. The shipped values and any tuned ones +// live in internal/recsettings, which hands them in per request. type ScoringWeights struct { BaseWeight float64 LikeBoost float64 @@ -46,7 +46,7 @@ type ScoringWeights struct { ContextTimeWeight float64 } -// Score computes the weighted-shuffle score per spec §6: +// Score computes the weighted-shuffle score: // // score = base // + (is_general_liked ? LikeBoost : 0) diff --git a/internal/recommendation/sessionvector.go b/internal/recommendation/sessionvector.go index 33e6a645..824ee91c 100644 --- a/internal/recommendation/sessionvector.go +++ b/internal/recommendation/sessionvector.go @@ -13,7 +13,7 @@ type SessionVector struct { Artists []string `json:"artists"` Tags map[string]int `json:"tags"` RecentTrackIDs []string `json:"recent_track_ids"` - // DeviceClass is the current request's device (#1551), set by the radio + // DeviceClass is the current request's device, set by the radio // handler from the user's latest play; drives the device dimension of the // context-affinity term. Empty (omitted) for the daily mixes and for the // snapshot stored on play_events, so it never narrows those. @@ -55,12 +55,6 @@ func uuidString(u pgtype.UUID) string { // multi-genre delimiters (`;`, `,`) used by various tag editors. Trims // whitespace; drops empty fragments. Strings with no delimiter come back // as a single-element slice. -// -// This comment used to blame concatenated inputs like -// "ElectronicComplextroGlitch Hop" on broken tag editors. They were ours: the -// scanner stored dhowden/tag's welded multi-value frames verbatim. Fixed in -// #2499 — the scanner now writes ";"-delimited values, so such tokens only -// survive on rows not yet re-scanned. func splitGenres(s string) []string { parts := strings.FieldsFunc(s, func(r rune) bool { return r == ';' || r == ',' diff --git a/internal/recommendation/shuffle.go b/internal/recommendation/shuffle.go index 91173c22..718c6cb3 100644 --- a/internal/recommendation/shuffle.go +++ b/internal/recommendation/shuffle.go @@ -16,8 +16,7 @@ type Candidate struct { } // DiversityCaps bounds how much of one selection a single artist or album -// may occupy. A zero value means "no cap", which is what Shuffle did -// unconditionally before #3882. +// may occupy. A zero value means "no cap". type DiversityCaps struct { MaxPerArtist int MaxPerAlbum int @@ -53,7 +52,7 @@ func RadioDiversityCaps(limit int) DiversityCaps { } // heldBack says whether the candidate is a video rip held back from radio -// (M498 #5439): it plays when chosen, but radio does not choose it. +// it plays when chosen, but radio does not choose it. func heldBack(c Candidate) bool { return c.Track.SourceVerdict != nil && *c.Track.SourceVerdict == "suspect" } @@ -69,19 +68,13 @@ func heldBack(c Candidate) bool { // So the result holds min(limit, len(candidates)) either way — the caps // change WHICH tracks are chosen, never HOW MANY. (What is not a candidate at // all is dropped first: a held-back video rip, and a second copy of a song -// already chosen from another release, M498.) +// already chosen from another release.) // -// That two-pass shape is the whole design, and a hard cap would have been -// the easy mistake. Radio asks for 50 tracks by default and 200 at most; a -// pool concentrated on a few artists would return six tracks and call it a -// radio. Rule 131's principle — degrade, never vanish — applies past the -// system mixes it was written for. -// -// Before #3882 there was no cap here at all, while discover.go, -// you_might_like.go and home.go all had one. That asymmetry is what let a -// radio session come back entirely from a single artist: nothing between -// the pool and the output bounded any artist's share, so a pool dominated -// by one artist produced an output dominated by it too. +// That two-pass shape is the whole design. A hard cap would be the easy +// mistake: radio asks for 50 tracks by default and 200 at most, and a pool +// concentrated on a few artists would return six tracks and call it a radio. +// Degrade, never vanish. Without any cap, a pool dominated by one artist +// would produce a radio entirely of that artist. func Shuffle( candidates []Candidate, weights ScoringWeights, @@ -112,7 +105,7 @@ func Shuffle( deferred := make([]Candidate, 0, len(scored)-limit) artistCount := map[pgtype.UUID]int{} albumCount := map[pgtype.UUID]int{} - // One song on several releases is played once (M498 #5438): after its + // One song on several releases is played once: after its // best-scoring copy, the others are dropped, not held back. songs := map[pgtype.UUID]bool{} repeat := func(c Candidate) bool { return c.Track.SongKey.Valid && songs[c.Track.SongKey] } diff --git a/internal/recommendation/shuffle_caps_test.go b/internal/recommendation/shuffle_caps_test.go index 9ba78ecd..0fd5f6ab 100644 --- a/internal/recommendation/shuffle_caps_test.go +++ b/internal/recommendation/shuffle_caps_test.go @@ -50,9 +50,6 @@ func artistsOf(picks []Candidate) map[string]int { return out } -// THE BUG. Operator, 2026-09-10: started radio from a song and "literally -// all of the songs in the playlist after that were from a single artist". -// // One artist's tracks all outscore everything else, and there are more of // them than the radio has room for. Without a cap the output is entirely // that artist — nothing between the pool and the result bounded its share. diff --git a/internal/recommendation/shuffle_test.go b/internal/recommendation/shuffle_test.go index 9e36fe22..776a43d2 100644 --- a/internal/recommendation/shuffle_test.go +++ b/internal/recommendation/shuffle_test.go @@ -112,7 +112,7 @@ func TestShuffle_OneCopyPerSong(t *testing.T) { } } -// A video rip held back from radio (M498 #5439) is never chosen, even to fill +// A video rip held back from radio is never chosen, even to fill // the request. func TestShuffle_SkipsHeldBackRips(t *testing.T) { suspect := "suspect" diff --git a/internal/recommendation/similarity.go b/internal/recommendation/similarity.go index 3062a7f3..f4505a0c 100644 --- a/internal/recommendation/similarity.go +++ b/internal/recommendation/similarity.go @@ -1,14 +1,13 @@ package recommendation // SimilarityWeights balances the per-axis contribution to the weighted Jaccard -// score. v1 hardcodes the defaults — operators cannot tune via YAML. If -// telemetry justifies it, expose under recommendation.similarity.* later. +// score. The defaults are hardcoded; there is no config key for them. type SimilarityWeights struct { TagsWeight float64 ArtistsWeight float64 } -// DefaultSimilarityWeights is the v1 axis balance per the M3 design. +// DefaultSimilarityWeights is the shipped axis balance. // Tags carry more signal than artists because a session's "vibe" tracks // genre more directly than artist identity (a session can mix artists // within a genre but rarely mixes genres). diff --git a/internal/recommendation/suggestions.go b/internal/recommendation/suggestions.go index 348ebf6d..b7cb8b4f 100644 --- a/internal/recommendation/suggestions.go +++ b/internal/recommendation/suggestions.go @@ -4,11 +4,11 @@ // through artist_similarity_unmatched via a single CTE, and returns top-N // out-of-library candidates with top-3 attribution seeds resolved to names. // -// Originally M5c, which seeded from raw likes + plays. That signal grew -// without bound and counted skips as engagement, so a few heavily-played -// artists monopolized every slot and the surface entrenched harder the more -// the user listened. Reworked to seed from the taste profile in issue #2367; -// see internal/db/queries/recommendation.sql for the tiering. +// Seeding from the taste profile rather than raw likes + plays matters: raw +// counts grow without bound and count skips as engagement, so a few +// heavily-played artists would monopolize every slot and entrench harder the +// more the user listened. See internal/db/queries/recommendation.sql for the +// tiering. package recommendation import ( @@ -33,12 +33,12 @@ type ArtistSuggestion struct { Attribution []SeedContribution // MatchedTags are the candidate's own tags that overlap the user's taste // profile, strongest first (max 3) — the "matches: shoegaze, melancholic" - // explanation (#2377). Empty when the candidate has no cached tags, which - // is common and not an error: coverage is permanently partial (#2376). + // explanation. Empty when the candidate has no cached tags, which is + // common and not an error: coverage is permanently partial. MatchedTags []string // TagOverlap is the [0,1] share of the candidate's tag mass the user likes. - // Exposed for the admin tuning lab — seeing the term's actual distribution - // is how the operator picks a weight rather than guessing at one. + // Exposed for admin tuning — seeing the term's actual distribution is how + // a weight gets picked rather than guessed. TagOverlap float64 } @@ -71,8 +71,8 @@ func SuggestArtists( q := dbq.New(pool) // Over-fetch so there is something to rotate through. A deterministic // top-N over a score ordering shows the same faces every day until one is - // requested away — the tail of the ranking was unreachable (#2367 - // mechanism 1). Selection from this pool happens in selectSuggestions. + // requested away, leaving the tail of the ranking unreachable. Selection + // from this pool happens in selectSuggestions. rows, err := q.SuggestArtistsForUser(ctx, dbq.SuggestArtistsForUserParams{ UserID: userID, Column2: halfLifeDays, @@ -130,7 +130,7 @@ func SuggestArtists( }) } - // Taste-tag term (#2377). Applied to the whole over-fetched pool BEFORE + // Taste-tag term. Applied to the whole over-fetched pool BEFORE // selection, so the rotation and diversity rules in selectSuggestions // operate on taste-blended scores — boosting only the twelve already // chosen by similarity would leave the actual re-ranking undone. @@ -227,9 +227,8 @@ func rotationDay(now time.Time) string { // Three rules, in order: // // 1. Diversity cap. Walking in score order, a candidate is dropped once its -// dominant seed already owns maxPerSeed slots. Nothing capped this before, -// so all twelve slots could be neighbours of one artist and the surface -// read as a single narrow cluster (#2367 mechanism 4). +// dominant seed already owns maxPerSeed slots, so twelve slots cannot all +// be neighbours of one artist and read as a single narrow cluster. // 2. Head. The best few by score always lead, so the strongest matches are // never rotated out of sight. Mirrors For You's head/tail shape. // 3. Tail. The rest of the slots are drawn from the remaining pool ordered by @@ -263,8 +262,8 @@ func selectSuggestions(pool []ArtistSuggestion, limit int, day string) []ArtistS } // Diversity is a PREFERENCE, not a quota that may starve the deck. A user // whose whole pool hangs off one or two seeds would otherwise get a - // three-card surface, which is worse than the monoculture we were avoiding - // (and is the vanish-or-nothing shape rule #131 exists to prevent). Top up + // three-card surface, which is worse than the monoculture it avoids. + // Top up // in score order from what the cap set aside. for _, s := range overflow { if len(out) >= limit { diff --git a/internal/recommendation/suggestions_integration_test.go b/internal/recommendation/suggestions_integration_test.go index 9c86e76c..e8f50bd8 100644 --- a/internal/recommendation/suggestions_integration_test.go +++ b/internal/recommendation/suggestions_integration_test.go @@ -270,13 +270,13 @@ func TestSuggestArtists_FiltersAlreadyRequested(t *testing.T) { seed := seedArtist(t, pool, "Seed", "") likeArtist(t, pool, user.ID, seed.ID) seedUnmatched(t, pool, seed.ID, "req-mbid", "Pending Request", 0.9) - if _, err := dbq.New(pool).CreateLidarrRequest(context.Background(), dbq.CreateLidarrRequestParams{ - UserID: user.ID, - Kind: dbq.LidarrRequestKindArtist, - LidarrArtistMbid: "req-mbid", - ArtistName: "Pending Request", + if _, err := dbq.New(pool).CreateRequest(context.Background(), dbq.CreateRequestParams{ + UserID: user.ID, + Kind: dbq.RequestKindArtist, + ArtistMbid: "req-mbid", + ArtistName: "Pending Request", }); err != nil { - t.Fatalf("CreateLidarrRequest: %v", err) + t.Fatalf("CreateRequest: %v", err) } out, err := SuggestArtists(context.Background(), pool, user.ID, 30, 12, 0) @@ -294,20 +294,20 @@ func TestSuggestArtists_RejectedRequestStillShown(t *testing.T) { seed := seedArtist(t, pool, "Seed", "") likeArtist(t, pool, user.ID, seed.ID) seedUnmatched(t, pool, seed.ID, "rej-mbid", "Rejected Once", 0.9) - req, err := dbq.New(pool).CreateLidarrRequest(context.Background(), dbq.CreateLidarrRequestParams{ - UserID: user.ID, - Kind: dbq.LidarrRequestKindArtist, - LidarrArtistMbid: "rej-mbid", - ArtistName: "Rejected Once", + req, err := dbq.New(pool).CreateRequest(context.Background(), dbq.CreateRequestParams{ + UserID: user.ID, + Kind: dbq.RequestKindArtist, + ArtistMbid: "rej-mbid", + ArtistName: "Rejected Once", }) if err != nil { - t.Fatalf("CreateLidarrRequest: %v", err) + t.Fatalf("CreateRequest: %v", err) } rejNotes := "wrong artist" - if _, err := dbq.New(pool).RejectLidarrRequest(context.Background(), dbq.RejectLidarrRequestParams{ + if _, err := dbq.New(pool).RejectRequest(context.Background(), dbq.RejectRequestParams{ ID: req.ID, Notes: &rejNotes, DecidedBy: user.ID, }); err != nil { - t.Fatalf("RejectLidarrRequest: %v", err) + t.Fatalf("RejectRequest: %v", err) } out, err := SuggestArtists(context.Background(), pool, user.ID, 30, 12, 0) @@ -331,7 +331,7 @@ func TestSuggestArtists_EmptyForNewUser(t *testing.T) { } } -// --- Slice 1 (#2372): taste-profile seeding, tiering, and the skip fix --- +// --- Taste-profile seeding, tiering, and skipped plays --- func setTasteWeight(t *testing.T, pool *pgxpool.Pool, userID, artistID pgtype.UUID, weight float64) { t.Helper() @@ -365,7 +365,7 @@ func insertSkippedPlayEvent(t *testing.T, pool *pgxpool.Pool, userID, trackID pg } // Tier 1: a taste-profile weight alone seeds a suggestion, with no like and no -// play on the seed artist. Before #2367 the profile was ignored entirely here. +// play on the seed artist. func TestSuggestArtists_TasteProfileWeightSeedsTier1(t *testing.T) { pool := newPool(t) user := seedUser(t, pool, "alice") @@ -449,9 +449,9 @@ func TestSuggestArtists_NonPositiveTasteWeightDoesNotSeed(t *testing.T) { } } -// Tier 2 counts COMPLETED plays only. Previously every play_event counted, so -// skipping an artist repeatedly increased its signal and pushed more of its -// neighbours at the user (#2367 mechanism 3). +// Tier 2 counts COMPLETED plays only. Counting every play_event would make +// skipping an artist repeatedly increase its signal and push more of its +// neighbours at the user. func TestSuggestArtists_SkippedPlaysDoNotSeedTier2(t *testing.T) { pool := newPool(t) user := seedUser(t, pool, "alice") @@ -473,12 +473,11 @@ func TestSuggestArtists_SkippedPlaysDoNotSeedTier2(t *testing.T) { } } -// --- Slice 3 (#2374): time-boxed suggestion snooze --- +// --- Time-boxed suggestion snooze --- // -// The defining property under test is that a snooze EXPIRES. A permanent -// dismissal would pass most of these; only TestSuggestArtists_ExpiredSnooze -// distinguishes the two, and it is the reason this shape was approved over an -// exclusion UI (rule #101). +// The defining property under test is that a snooze EXPIRES: suggestions have +// no permanent "never show" control. A permanent dismissal would pass most of +// these; only TestSuggestArtists_ExpiredSnooze distinguishes the two. // snoozeUntil inserts a snooze row with an explicit absolute expiry, so a // test can place it in the past without depending on the duration arithmetic @@ -546,7 +545,7 @@ func TestSuggestArtists_ExpiredSnoozeShowsCandidateAgain(t *testing.T) { } } -// Rule #47: one household member parking a suggestion must not remove it +// One household member parking a suggestion must not remove it // from anyone else's deck. func TestSuggestArtists_SnoozeIsPerUser(t *testing.T) { pool := newPool(t) diff --git a/internal/recommendation/suggestions_select_test.go b/internal/recommendation/suggestions_select_test.go index c73ad91c..138a3a73 100644 --- a/internal/recommendation/suggestions_select_test.go +++ b/internal/recommendation/suggestions_select_test.go @@ -8,7 +8,7 @@ import ( ) // selectSuggestions is deliberately pure (no DB, no clock), so the rotation and -// diversity rules from slice #2373 are covered here in the fast lane rather +// diversity rules are covered here in the fast lane rather // than behind the integration gate. func seedID(n byte) pgtype.UUID { @@ -95,13 +95,12 @@ func TestSelectSuggestions_HeadIsStableTopScorers(t *testing.T) { } } -// #2367 mechanism 4: all twelve slots could be neighbours of one artist. +// Twelve slots must not all be neighbours of one artist. // // The pool is deliberately SKEWED — one seed owns the entire top of the score // ranking — because that is the only shape where a cap can bite. With scores // spread evenly across seeds the top-N is already diverse and the cap is -// untestable (an earlier version of this test asserted against an even pool and -// could not fail). +// untestable: an even pool cannot fail this test. func TestSelectSuggestions_CapsOneDominantSeed(t *testing.T) { const dominantRun = 20 pool := make([]ArtistSuggestion, 0, 48) @@ -131,7 +130,7 @@ func TestSelectSuggestions_CapsOneDominantSeed(t *testing.T) { } } -// Diversity must not starve the deck (rule #131): a pool hanging off a single +// Diversity must not starve the deck: a pool hanging off a single // seed should still fill, not collapse to maxPerSeed entries. func TestSelectSuggestions_SingleSeedPoolStillFills(t *testing.T) { got := selectSuggestions(poolOf(30, 1), 12, "2026-08-01") diff --git a/internal/recommendation/tagoverlap.go b/internal/recommendation/tagoverlap.go index 8e2d3bbd..afde5eaf 100644 --- a/internal/recommendation/tagoverlap.go +++ b/internal/recommendation/tagoverlap.go @@ -1,13 +1,10 @@ -// tagoverlap.go — the taste-tag term for the Discover request surface -// (#2377, milestone #268 slice 6). +// tagoverlap.go — the taste-tag term for the Discover request surface. // -// Slices 1-2 made the deck stop repeating; this is what makes it *relevant*. -// Before this, a candidate's only claim on a slot was "some artist you play is -// similar to it" — a graph-adjacency fact that says nothing about whether the -// music sounds like anything you actually like. Here the candidate's own -// folksonomy tags (cached by slice 5) are compared against the user's -// taste-profile tags, so the surface can rank on "matches the sound you like" -// and say WHY. +// Similarity alone says "some artist you play is similar to it", a +// graph-adjacency fact that says nothing about whether the music sounds like +// anything you actually like. Here the candidate's own folksonomy tags (from +// the candidate tag cache) are compared against the user's taste-profile +// tags, so the surface can rank on "matches the sound you like" and say WHY. // // Pure by design — no DB, no clock — so the scoring rules are unit-testable in // the fast lane rather than behind the integration gate. @@ -42,9 +39,8 @@ type TagWeights map[string]float64 // // Returns (0, nil) when either side is empty. That is the load-bearing // degradation path: tag coverage for out-of-library candidates is permanently -// partial (#2376), and a cold-start user has no taste tags at all. Both must -// leave the candidate's similarity score untouched rather than sink it — -// rule #131, tiered degradation, never vanish-or-nothing. +// partial, and a cold-start user has no taste tags at all. Both must leave +// the candidate's similarity score untouched rather than sink it. func tagOverlap(candidate, taste TagWeights) (float64, []string) { if len(candidate) == 0 || len(taste) == 0 { return 0, nil @@ -123,7 +119,7 @@ func tagOverlap(candidate, taste TagWeights) (float64, []string) { // weight would let a near-zero-similarity artist outrank a strong match // just for sharing a popular tag, which reads as noise to the user. // - weight 0 disables the feature completely and restores pure similarity -// order, so the operator's knob has a real off position. +// order, so the knob has a real off position. // // Callers must pass the pool in similarity order; it is returned in blended // order. Mutates the elements in place (they're the caller's own slice built @@ -132,8 +128,8 @@ func tagOverlap(candidate, taste TagWeights) (float64, []string) { func applyTagOverlap( pool []ArtistSuggestion, candidateTags map[string]TagWeights, taste TagWeights, weight float64, ) []ArtistSuggestion { - // A zero weight is the operator turning the feature off. Skip the work - // AND the re-sort so the ordering is bit-for-bit the pre-slice-6 result. + // A zero weight turns the feature off. Skip the work AND the re-sort so + // the ordering is bit-for-bit pure similarity order. if weight == 0 || len(taste) == 0 { return pool } @@ -149,7 +145,7 @@ func applyTagOverlap( } // Stable tiebreak by MBID. Without it, two candidates on equal scores // could swap between requests within the same day, which the daily - // rotation (#2373) exists to prevent. + // rotation exists to prevent. return pool[i].MBID < pool[j].MBID }) return pool diff --git a/internal/recommendation/tagoverlap_test.go b/internal/recommendation/tagoverlap_test.go index e66269e4..e18cec86 100644 --- a/internal/recommendation/tagoverlap_test.go +++ b/internal/recommendation/tagoverlap_test.go @@ -5,14 +5,13 @@ import ( "testing" ) -// tagOverlap / applyTagOverlap are pure, so slice 6's ranking rules are covered -// in the fast lane rather than behind the integration gate. +// tagOverlap / applyTagOverlap are pure, so the taste-tag ranking rules are +// covered in the fast lane rather than behind the integration gate. // // Fixtures here are deliberately SKEWED — candidates that match the taste // profile to clearly different degrees. An evenly-matching pool cannot exercise // a re-ranking at all: every candidate gets the same multiplier and the order is -// unchanged whether the blend works or not. Slice 2's first diversity test had -// exactly that defect, so it's called out explicitly here. +// unchanged whether the blend works or not. func TestTagOverlap_FullMatchScoresOne(t *testing.T) { // Every unit of the candidate's tag mass sits on the user's single @@ -68,8 +67,8 @@ func TestTagOverlap_WeakTasteTagScoresLowerThanStrong(t *testing.T) { } // THE degradation path. Tag coverage for out-of-library candidates is -// permanently partial (#2376) and cold-start users have no taste tags, so both -// must be scored 0 — never negative, never dropped (rule #131). +// permanently partial and cold-start users have no taste tags, so both must be +// scored 0 — never negative, never dropped. func TestTagOverlap_EmptyEitherSideIsZeroNotNegative(t *testing.T) { cases := []struct { name string @@ -186,8 +185,8 @@ func TestApplyTagOverlap_UntaggedCandidateScoreIsUnchanged(t *testing.T) { } } -// Weight 0 is the operator's off switch: the ordering must be bit-for-bit the -// pre-slice-6 result, not merely similar. +// Weight 0 is the off switch: the ordering must be bit-for-bit pure similarity +// order, not merely similar. func TestApplyTagOverlap_ZeroWeightIsAnExactNoOp(t *testing.T) { pool := poolFor(spec{"a", 1.0}, spec{"b", 0.7}) candTags := map[string]TagWeights{"b": {"shoegaze": 1.0}} @@ -232,7 +231,7 @@ func TestApplyTagOverlap_BoostIsBoundedByOnePlusWeight(t *testing.T) { // Equal blended scores must resolve deterministically, or two candidates could // swap between requests inside one day — the exact churn the daily rotation -// (#2373) exists to prevent. +// exists to prevent. func TestApplyTagOverlap_TiesBreakDeterministically(t *testing.T) { first := applyTagOverlap(poolFor(spec{"zzz", 1.0}, spec{"aaa", 1.0}), map[string]TagWeights{}, TagWeights{"x": 1}, 1.0) diff --git a/internal/recommendation/taste.go b/internal/recommendation/taste.go index 46fd37f6..004dd6a4 100644 --- a/internal/recommendation/taste.go +++ b/internal/recommendation/taste.go @@ -16,11 +16,11 @@ import ( // rest toward zero (as a per-user max-normalisation would). A weight at the // scale value maps to tanh(1) ≈ 0.76 — "clearly a preference." // -// The era (#1530) and mood (#1534) terms are added on TOP of artist+tag +// The era and mood terms are added on TOP of artist+tag // (shares don't sum to 1) so they're pure nudges that never weaken the existing // artist/tag signal when a track lacks a date or mood tags; clampUnit bounds // the combined result. Their shares are small — both are coarse, partial- -// coverage signals — and re-bakeable if the lab shows they should carry more. +// coverage signals — and re-bakeable if tuning shows they should carry more. const ( tasteArtistScale = 4.0 tasteTagScale = 3.0 diff --git a/internal/recsettings/patch.go b/internal/recsettings/patch.go index f7139cdf..179c1e3b 100644 --- a/internal/recsettings/patch.go +++ b/internal/recsettings/patch.go @@ -171,7 +171,7 @@ const ( tagOverlapWeightMax = 10.0 // Snooze duration: at least a day (anything less isn't a snooze, it's a // flicker), at most a year — past that it's a permanent dismissal wearing a - // snooze's clothes, which is exactly the shape rule #101 rules out. + // snooze's clothes, and suggestions have no permanent "never show" control. snoozeDaysMin = 1.0 snoozeDaysMax = 365.0 ) diff --git a/internal/recsettings/service.go b/internal/recsettings/service.go index 354cddb8..e333ff46 100644 --- a/internal/recsettings/service.go +++ b/internal/recsettings/service.go @@ -1,16 +1,15 @@ // Package recsettings is the DB-backed home of the recommendation -// tuning knobs (#1250): the two scoring-weight profiles (radio / -// daily_mix) and the taste-profile build settings (engagement -// half-life + completion curve). It follows the coverart +// tuning knobs: the scoring-weight profiles (radio, daily_mix, +// songs_like), Discover's settings, and the taste-profile build +// settings (engagement half-life + completion curve). It follows the coverart // SettingsService pattern — boot reconcile seeds shipped defaults for // missing rows, values are cached under a RWMutex, and every change // takes effect live (the daily_mix profile + taste config are pushed // into package playlists; radio reads the cache per request). // -// Framing (decision #1247): this is the defaults-discovery lab. The -// operator turns knobs to FIND good values; found-good values get -// baked back into the Shipped* functions below as new shipped -// defaults. End users and other operators should never need the card. +// This is a lab for finding good defaults: an admin turns knobs to find +// good values, and those get baked back into the Shipped* functions below +// as new shipped defaults. A typical install never needs to touch it. package recsettings import ( @@ -35,24 +34,23 @@ const ( ScopeRadio = "radio" ScopeDailyMix = "daily_mix" ScopeTaste = "taste" - // ScopeDiscover is the Discover request surface (#2377). Its own scope - // rather than columns on taste: SnoozeDays lives here, and a snooze must - // never be read as taste signal (#2374) — filing it under taste would put - // it one careless join from the leak that design forbids. + // ScopeDiscover is the Discover request surface. Its own scope rather + // than columns on taste: SnoozeDays lives here, and a snooze must never + // be read as taste signal — filing it under taste would put it one + // careless join from that leak. ScopeDiscover = "discover" - // ScopeSongsLike is the "Songs like {X}" surface (#3881). It shared - // daily_mix with For-You until 2026-09-10, and that sharing WAS the bug: - // the two surfaces want opposite things. For-You answers "what will they - // enjoy today" and is supposed to roam; Songs-like answers "what sounds - // like THIS" and is the tightest surface in the product. One set of - // weights cannot serve both, and the broad answer was winning. + // ScopeSongsLike is the "Songs like {X}" surface. It is separate from + // daily_mix because the two want opposite things: For-You answers "what + // will they enjoy today" and is supposed to roam; Songs-like answers + // "what sounds like THIS" and is the tightest surface in the product. One + // set of weights cannot serve both. ScopeSongsLike = "songs_like" ) // TasteTuning is the tunable subset of taste.Config: the engagement // half-life, the completion→engagement curve points, the enriched-tag -// weight (how much folksonomy tags count vs raw ID3 genre, #1520), and the -// era-facet weight (how strongly a decade-play imprints, #1530). +// weight (how much folksonomy tags count vs raw ID3 genre), and the +// era-facet weight (how strongly a decade-play imprints). type TasteTuning struct { HalfLifeDays float64 EngagementHardSkip float64 @@ -63,12 +61,10 @@ type TasteTuning struct { MoodScale float64 } -// ShippedRadioWeights are the shipped radio-profile defaults (moved -// here from config.RecommendationConfig — YAML is bootstrap-only, -// rule: config in UI). Radio is seed-directed (the user picked a -// direction), so taste is a lighter nudge than in the daily mixes. -// ContextTimeWeight starts uniform (1.0) across both profiles pending -// trend data (#1531); split them once the metrics view justifies it. +// ShippedRadioWeights are the shipped radio-profile defaults. Radio is +// seed-directed (the user picked a direction), so taste is a lighter nudge +// than in the daily mixes. ContextTimeWeight is 1.0 in both profiles; the +// metrics view is what would justify splitting them. func ShippedRadioWeights() recommendation.ScoringWeights { return recommendation.ScoringWeights{ BaseWeight: 1.0, @@ -99,12 +95,11 @@ func ShippedDailyMixWeights() recommendation.ScoringWeights { } } -// ShippedSongsLikeWeights are the shipped songs_like-profile defaults -// (#3881). The whole point is that SIMILARITY DOMINATES; every other -// profile balances it against taste and engagement, and this one must not. +// ShippedSongsLikeWeights are the shipped songs_like-profile defaults. The +// whole point is that SIMILARITY DOMINATES; every other profile balances it +// against taste and engagement, and this one must not. // -// The failure being corrected, arithmetic from the daily_mix profile that -// this surface used to share: +// Under the daily_mix weights, for example: // // unrelated track, liked, not played recently → 1.0 + 2.0 + 1.0 = 4.0 // PERFECT similarity match, not liked → 1.0 + 1.5 = 2.5 @@ -132,12 +127,10 @@ func ShippedDailyMixWeights() recommendation.ScoringWeights { // stays 2.0 because a track the user skips is still unwanted no matter how // similar it is. // -// These are DEFAULTS, not settings (rule 25) — the operator turns them in the -// admin tuning card and good values get baked back here. They are a -// defensible starting point rather than a measured optimum: the per-arm fill -// rates and the real sim_score distribution are still unknown (#3879), and -// `likes_overlap` contributes a flat 0.6 that a high SimilarityWeight -// amplifies. Expect to move these once that lands. +// These are DEFAULTS, not settings: an admin turns them in the tuning card +// and good values get baked back here. They are a defensible starting point +// rather than a measured optimum, and `likes_overlap` contributes a flat 0.3 +// that a high SimilarityWeight amplifies. func ShippedSongsLikeWeights() recommendation.ScoringWeights { return recommendation.ScoringWeights{ BaseWeight: 1.0, // same for all candidates; differentiates nothing @@ -169,12 +162,12 @@ func shippedWeightsFor(scope string) (recommendation.ScoringWeights, bool) { return recommendation.ScoringWeights{}, false } -// DiscoverTuning is the tunable set for the Discover request surface (#2377). +// DiscoverTuning is the tunable set for the Discover request surface. type DiscoverTuning struct { // TagOverlapWeight scales the taste-tag term: score × (1 + w × overlap). // 0 disables it and restores pure similarity ranking. TagOverlapWeight float64 - // SnoozeDays is the default "not right now" duration (#2374). + // SnoozeDays is the default "not right now" duration. SnoozeDays float64 } @@ -183,12 +176,11 @@ type DiscoverTuning struct { // TagOverlapWeight 1.0 lets a perfect tag match at most double a candidate's // similarity score — enough to reorder the deck meaningfully, not enough for a // popular-tag coincidence to beat a genuinely strong similarity match. It is a -// starting point for the tuning lab, not a tuned value: the honest way to pick -// it is the metrics trend view after some real use. +// starting point, not a tuned value: the way to pick it is the metrics trend +// view after some real use. // -// SnoozeDays 90 matches the operator's approved shape: long enough that a -// parked suggestion stops nagging, short enough that a taste shift brings it -// back on its own. +// SnoozeDays 90: long enough that a parked suggestion stops nagging, short +// enough that a taste shift brings it back on its own. func ShippedDiscoverTuning() DiscoverTuning { return DiscoverTuning{ TagOverlapWeight: 1.0, @@ -337,7 +329,7 @@ func (s *Service) Taste() TasteTuning { // Discover returns the cached Discover-tuning values. Read per request by the // suggestions handler, so an admin change takes effect on the next refresh -// with no restart (rule #25). +// with no restart. func (s *Service) Discover() DiscoverTuning { s.mu.RLock() defer s.mu.RUnlock() diff --git a/internal/recsettings/service_test.go b/internal/recsettings/service_test.go index 083a38bd..37bc3958 100644 --- a/internal/recsettings/service_test.go +++ b/internal/recsettings/service_test.go @@ -282,7 +282,7 @@ func TestUpdate_NoOpWritesNoAudit(t *testing.T) { } } -// --- Discover scope (#2377, milestone #268 slice 6) --- +// --- Discover scope --- func TestNew_SeedsDiscoverDefaults(t *testing.T) { pool := newPool(t) @@ -310,9 +310,8 @@ func TestUpdateDiscover_PersistsAndAudits(t *testing.T) { t.Errorf("SnoozeDays = %v, want 30", got) } - // The audit row must land under the new scope. This is the assertion that - // would have caught a missing rule-#36 CHECK migration: without expanding - // recommendation_tuning_audit's whitelist, this INSERT fails at runtime. + // The audit row must land under the new scope: a scope missing from + // recommendation_tuning_audit's CHECK fails this INSERT at runtime. rows := auditRows(t, pool) if len(rows) != 1 { t.Fatalf("audit rows = %d, want 1", len(rows)) @@ -343,8 +342,7 @@ func TestUpdateDiscover_Validation(t *testing.T) { {"weight past the typo bound", map[string]float64{"tag_overlap_weight": 100}}, // A sub-day snooze isn't a snooze, it's a flicker. {"snooze under a day", map[string]float64{"snooze_days": 0.5}}, - // Past a year it's a permanent dismissal wearing a snooze's clothes — - // the shape rule #101 rules out. + // Past a year it's a permanent dismissal wearing a snooze's clothes. {"snooze past a year", map[string]float64{"snooze_days": 400}}, } for _, c := range cases { @@ -360,9 +358,8 @@ func TestUpdateDiscover_Validation(t *testing.T) { } } -// Weight 0 must be accepted — it's the operator's off switch for the whole -// tag term, so a "must be positive" bound would remove their ability to -// disable the feature. +// Weight 0 must be accepted: it's the off switch for the whole tag term, so +// a "must be positive" bound would leave no way to disable it. func TestUpdateDiscover_ZeroWeightIsAllowed(t *testing.T) { pool := newPool(t) s := newService(t, pool) diff --git a/internal/recsettings/songs_like_weights_test.go b/internal/recsettings/songs_like_weights_test.go index 805b940b..c4830d49 100644 --- a/internal/recsettings/songs_like_weights_test.go +++ b/internal/recsettings/songs_like_weights_test.go @@ -7,10 +7,9 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/recommendation" ) -// The bug, reproduced as a ranking: a track that sounds nothing like the seed -// but which the user liked and has not played lately used to OUTRANK a perfect -// similarity match. Operator, 2026-09-10: "I was getting a seeming wide variety -// of music from each one when I was hoping to stay in a certain neighborhood." +// A track that sounds nothing like the seed, but which the user liked and has +// not played lately, must not outrank a perfect similarity match: Songs-like +// should stay in the seed's neighbourhood. // // This is the whole point of the songs_like profile, so it is asserted as // BEHAVIOUR — two candidates, which one wins — rather than by checking the @@ -85,7 +84,7 @@ func TestSongsLikeWeights_SimilarityOutrangesEverySeedIndependentTerm(t *testing } // A scope that is seedable but not resettable is the half-wired shape this -// guards: the profile appears in the admin card, the operator turns a knob, +// guards: the profile appears in the admin card, an admin turns a knob, // and Reset then 404s on a scope the rest of the service knows about. func TestShippedWeightsFor_CoversEveryWeightProfile(t *testing.T) { for _, scope := range []string{ScopeRadio, ScopeDailyMix, ScopeSongsLike} { diff --git a/internal/release/agree.go b/internal/release/agree.go new file mode 100644 index 00000000..d220408e --- /dev/null +++ b/internal/release/agree.go @@ -0,0 +1,40 @@ +package release + +import "slices" + +// Comparing a file against a MusicBrainz track: shared by the importer's +// check of a download and the library listing's match of an album. + +// LengthFits allows five seconds or 3%, whichever is more: remasters and +// rips differ by a second or two, a different recording by far more. A +// track MusicBrainz has no length for says nothing against the file. +func LengthFits(fileMs, trackMs int) bool { + if trackMs <= 0 { + return true + } + if fileMs <= 0 { + return false + } + tol := max(5000, trackMs*3/100) + d := fileMs - trackMs + return d <= tol && -d <= tol +} + +// TitleAgrees: the same words, or one title holding all of the other's +// ("Get Lucky (Radio Edit)" holds "Get Lucky"). +func TitleAgrees(a, b string) bool { + ta, tb := Tokens(a), Tokens(b) + if len(ta) == 0 || len(tb) == 0 { + return false + } + return subset(ta, tb) || subset(tb, ta) +} + +func subset(small, big []string) bool { + for _, w := range small { + if !slices.Contains(big, w) { + return false + } + } + return true +} diff --git a/internal/release/agree_test.go b/internal/release/agree_test.go new file mode 100644 index 00000000..50910fe7 --- /dev/null +++ b/internal/release/agree_test.go @@ -0,0 +1,19 @@ +package release + +import "testing" + +func TestLengthFits(t *testing.T) { + for _, c := range []struct { + file, track int + ok bool + }{ + // Short tracks get the 5s floor; longer ones 3% of the track. + {120000, 125000, true}, {120000, 126000, false}, + {240000, 246000, true}, {240000, 248000, false}, {600000, 615000, true}, {600000, 620000, false}, + {0, 240000, false}, {240000, 0, true}, + } { + if got := LengthFits(c.file, c.track); got != c.ok { + t.Errorf("LengthFits(%d, %d) = %v", c.file, c.track, got) + } + } +} diff --git a/internal/release/files.go b/internal/release/files.go new file mode 100644 index 00000000..2bfeca9f --- /dev/null +++ b/internal/release/files.go @@ -0,0 +1,76 @@ +package release + +import ( + "path" + "strconv" + "strings" +) + +// File is one file inside a torrent. +type File struct { + Index int + Path string + Size int64 +} + +// Contents is what a torrent's file list holds. +type Contents struct { + Audio int + Video int + // Format is the format most of the audio files are in, "" when none. + Format string +} + +var videoExt = map[string]bool{ + "mkv": true, "mp4": true, "avi": true, "m2ts": true, "vob": true, + "mov": true, "webm": true, "wmv": true, "m4v": true, "ts": true, +} + +// Inspect counts a file list's audio and video files. +func Inspect(files []File) Contents { + var c Contents + formats := map[string]int{} + for _, f := range files { + ext := strings.TrimPrefix(strings.ToLower(path.Ext(f.Path)), ".") + if fm := formatOf(ext); fm != "" { + c.Audio++ + formats[fm]++ + } else if videoExt[ext] { + c.Video++ + } + } + best := 0 + for fm, n := range formats { + if n > best || (n == best && formatRank[fm] > formatRank[c.Format]) { + c.Format, best = fm, n + } + } + return c +} + +// JudgeFiles says what a torrent's files say about whether it is the album: +// a veto (code and sentence), or a doubt for a person, or nothing. +func JudgeFiles(w Want, files []File) (veto, reason, doubt string) { + if len(files) == 0 { + return "", "", "" + } + c := Inspect(files) + switch { + case c.Audio == 0 && c.Video > 0: + return VetoVideo, "It holds video files and no audio.", "" + case c.Audio == 0: + return VetoNoAudio, "It holds no audio files Minstrel can read.", "" + case w.MinTracks > 0 && c.Audio < w.MinTracks: + return VetoIncomplete, "It holds " + tracks(c.Audio) + "; the shortest edition has " + tracks(w.MinTracks) + ".", "" + case w.MaxTracks > 0 && c.Audio > 2*w.MaxTracks+4: + return "", "", "It holds " + tracks(c.Audio) + ", more than any edition of the album has: probably more than one album." + } + return "", "", "" +} + +func tracks(n int) string { + if n == 1 { + return "1 track" + } + return strconv.Itoa(n) + " tracks" +} diff --git a/internal/release/match.go b/internal/release/match.go new file mode 100644 index 00000000..3fb25006 --- /dev/null +++ b/internal/release/match.go @@ -0,0 +1,210 @@ +package release + +import "strings" + +// Want is the album a search is for, as the catalogue describes it. +type Want struct { + Artist string + Title string + Year int + PrimaryType string // Album, EP, Single, Broadcast, Other + SecondaryTypes []string + // MinTracks and MaxTracks span the group's editions' track counts; zero + // when the catalogue has not read them. + MinTracks int + MaxTracks int +} + +// Veto codes: why a name is not the album. Kept on a candidate as its +// reason's code. +const ( + VetoArtist = "wrong_artist" + VetoTitle = "wrong_title" + VetoVideo = "video" + VetoDiscography = "discography" + VetoVersion = "other_version" + VetoType = "wrong_type" + VetoNoAudio = "no_audio" + VetoIncomplete = "incomplete" +) + +// Match is what a name says about whether it is the album wanted. +type Match struct { + // Veto is a code from above when the name is not the album, with Reason + // saying why in a sentence. + Veto string + Reason string + // Doubt is set when the name may be the album and cannot say for sure: a + // person decides. It is a sentence. + Doubt string + // YearAgrees: the name states the album's year, give or take one. + YearAgrees bool +} + +// versions are words that make a release a different recording of the +// album, with what to call it. A word the wanted title or artist holds does +// not count ("Live Through This", "Instrumental Core"). +var versions = []struct { + words []string + what string + // typ is the MusicBrainz secondary type that wants this version. + typ string +}{ + {[]string{"instrumental", "instrumentals"}, "an instrumental version", ""}, + {[]string{"drumless"}, "a drumless edition", ""}, + {[]string{"karaoke"}, "a karaoke version", ""}, + {[]string{"acapella", "acappella", "accapella", "acapellas"}, "an a cappella version", ""}, + {[]string{"remix", "remixes", "remixed"}, "a remix release", "Remix"}, + {[]string{"live", "bootleg", "bbc", "sessions", "unplugged"}, "a live recording", "Live"}, + {[]string{"demo", "demos"}, "demo recordings", "Demo"}, + // Not "covers": in a release name that is usually the artwork scans. + {[]string{"tribute", "lullaby", "lullabies"}, "a tribute release", ""}, + {[]string{"nightcore", "slowed", "sped"}, "a slowed or sped-up edit", ""}, + {[]string{"commentary", "interview", "interviews"}, "commentary or interviews", "Interview"}, +} + +// collections are words naming more than one album. +var collections = []string{"collection", "anthology", "albums", "boxset", "discography"} + +// Judge says whether a release name is the album wanted. It looks only at +// the name; JudgeFiles looks at what a torrent holds. +func Judge(w Want, r Release) Match { + var m Match + m.YearAgrees = r.Year != 0 && w.Year != 0 && abs(r.Year-w.Year) <= 1 + if r.Video != "" { + m.Veto, m.Reason = VetoVideo, "A video, not audio (“"+r.Video+"”)." + return m + } + artist := Tokens(w.Artist) + title := Tokens(w.Title) + if len(artist) > 0 && !contains(r.Tokens, artist) && !contains(r.Tokens, withoutThe(artist)) { + m.Veto, m.Reason = VetoArtist, "The name doesn’t mention "+w.Artist+"." + return m + } + if len(title) > 0 && !contains(r.Tokens, title) && !contains(r.Tokens, bare(w.Title)) { + m.Veto, m.Reason = VetoTitle, "The name doesn’t contain the title “"+w.Title+"”." + return m + } + own := map[string]bool{} + for _, t := range append(append([]string{}, title...), artist...) { + own[t] = true + } + if r.Discography != "" && !own[r.Discography] { + m.Veto, m.Reason = VetoDiscography, "A discography or collection ("+r.Discography+"), and one album is wanted." + return m + } + for _, c := range collections { + if !own[c] && has(r.Tokens, c) { + m.Veto, m.Reason = VetoDiscography, "A collection of albums (“"+c+"”), and one album is wanted." + return m + } + } + for _, v := range versions { + if v.typ != "" && hasType(w.SecondaryTypes, v.typ) { + continue + } + for _, word := range v.words { + if !own[word] && has(r.Tokens, word) { + m.Veto, m.Reason = VetoVersion, "This is "+v.what+" (“"+word+"”)." + return m + } + } + } + if w.PrimaryType == "Album" || w.PrimaryType == "EP" { + if !own["single"] && has(r.Tokens, "single") { + m.Veto, m.Reason = VetoType, "A single, and the "+strings.ToLower(w.PrimaryType)+" is wanted." + return m + } + } + if w.PrimaryType == "Album" && !own["ep"] && has(r.Tokens, "ep") { + m.Veto, m.Reason = VetoType, "An EP, and the album is wanted." + return m + } + // A self-titled album: every release of the artist names it once. Two + // mentions, or its year, say it is this one. + if len(title) > 0 && equal(title, artist) && count(r.Tokens, title) < 2 && !m.YearAgrees { + m.Doubt = "The album shares the artist’s name, and nothing else in the name says it is this album." + } + return m +} + +func abs(n int) int { + if n < 0 { + return -n + } + return n +} + +func withoutThe(t []string) []string { + if len(t) > 1 && t[0] == "the" { + return t[1:] + } + return t +} + +// bare is a title without a trailing bracketed part ("Album (Remastered)"), +// for names that leave it off. +func bare(title string) []string { + if i := strings.IndexAny(title, "(["); i > 0 { + return Tokens(title[:i]) + } + return nil +} + +// contains says whether want's words appear in name in order and together, +// or, for two or more words, run together ("ACDC" for AC/DC). A single word +// is never looked for run together: "muse" is inside "museum". +func contains(name, want []string) bool { + if len(want) == 0 { + return false + } + if count(name, want) > 0 { + return true + } + joined := strings.Join(want, "") + if len(want) < 2 || len(joined) < 4 { + return false + } + return strings.Contains(strings.Join(name, ""), joined) +} + +// count is how often want's words appear in name in order and together. +func count(name, want []string) int { + n := 0 + for i := 0; i+len(want) <= len(name); i++ { + if equal(name[i:i+len(want)], want) { + n++ + } + } + return n +} + +func equal(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} + +func has(tokens []string, word string) bool { + for _, t := range tokens { + if t == word { + return true + } + } + return false +} + +func hasType(types []string, t string) bool { + for _, x := range types { + if strings.EqualFold(x, t) { + return true + } + } + return false +} diff --git a/internal/release/parse.go b/internal/release/parse.go new file mode 100644 index 00000000..9acce161 --- /dev/null +++ b/internal/release/parse.go @@ -0,0 +1,289 @@ +// Package release reads music release names and judges them against a +// wanted album: what a name says it is (format, bitrate, bit +// depth, source, year, edition), whether it is the album wanted, and which +// quality tier it sits in. Nothing here touches the network or the database, +// so every rule is testable on a string. +package release + +import ( + "regexp" + "strconv" + "strings" +) + +// Formats a name can state. +const ( + FormatFLAC = "FLAC" + FormatALAC = "ALAC" + FormatAPE = "APE" + FormatWV = "WavPack" + FormatWAV = "WAV" + FormatAIFF = "AIFF" + FormatDSD = "DSD" + FormatMP3 = "MP3" + FormatAAC = "AAC" + FormatVorbis = "Vorbis" + FormatOpus = "Opus" +) + +// Sources a name can state. +const ( + SourceWEB = "WEB" + SourceCD = "CD" + SourceVinyl = "Vinyl" + SourceSACD = "SACD" + SourceCassette = "Cassette" +) + +// Release is what a release name says about itself. Zero values mean the +// name does not say. +type Release struct { + Name string + Year int + Format string + // Bitrate is a constant bitrate in kbps; VBR is a LAME preset (V0, V1, V2). + Bitrate int + VBR string + // BitDepth (16, 24) and SampleRate (kHz) for lossless. + BitDepth int + SampleRate float64 + Source string + // Video is the marker that says this is a video ("x264", "1080p"), or "". + Video string + // Discography: a year range or a word naming many albums. + Discography string + // Tokens are the name's words, normalised for matching. + Tokens []string +} + +var ( + formatRe = regexp.MustCompile(`\b(flac|alac|ape|wavpack|wv|wav|aiff|dsd|dsf|dff|mp3|aac|m4a|ogg|vorbis|opus)\b`) + // "24-96", "24/192", "16-44 1" (44.1 after dots become spaces). + depthRateRe = regexp.MustCompile(`\b(16|24|32) ?[-/ ] ?(44 1|44|48|88 2|88|96|176 4|176|192|352 8|384)\b`) + depthRe = regexp.MustCompile(`\b(16|24|32) ?-?bits?\b`) + rateRe = regexp.MustCompile(`\b(44 1|48|88 2|96|176 4|192) ?khz\b`) + hiResRe = regexp.MustCompile(`\bhi ?-?res\b`) + codecRateRe = regexp.MustCompile(`\b(?:mp3|aac|m4a|cbr|ogg|opus)[ \-@]*(320|256|224|192|160|128|112|96|64)\b`) + kbpsRe = regexp.MustCompile(`\b(320|256|224|192|160|128|112|96|64) ?(?:kbps|kbit|kb/s|k)\b`) + vbrRe = regexp.MustCompile(`\bv([0-2])\b`) + webRe = regexp.MustCompile(`\bweb(?: ?-?dl)?\b`) + cdRe = regexp.MustCompile(`\bcd ?(?:rip)?\b`) + vinylRe = regexp.MustCompile(`\b(?:vinyl|vinylrip|lp ?rip|needle ?drop)\b`) + sacdRe = regexp.MustCompile(`\bsacd\b`) + cassetteRe = regexp.MustCompile(`\b(?:cassette|tape ?rip)\b`) + yearRe = regexp.MustCompile(`\b(19[3-9]\d|20[0-4]\d)\b`) + bracketYear = regexp.MustCompile(`[(\[{] ?(19[3-9]\d|20[0-4]\d) ?[)\]}]`) + yearRangeRe = regexp.MustCompile(`\b(19[3-9]\d|20[0-4]\d) ?(?:-|–|to) ?(19[3-9]\d|20[0-4]\d)\b`) + videoRe = regexp.MustCompile(`\b(x264|x265|h ?264|h ?265|hevc|xvid|divx|1080p|1080i|720p|2160p|bluray|blu ?ray|bdrip|brrip|bdremux|dvdrip|dvd9|dvd5|webrip|hdtv|mkv|avi|m2ts)\b`) + discogRe = regexp.MustCompile(`\b(discography|discografia|discographie|diskografie|discografie)\b`) +) + +// Parse reads what a release name says. +func Parse(name string) Release { + r := Release{Name: name} + s := spaced(name) + + r.Format = bestFormat(formatRe.FindAllStringSubmatch(s, -1)) + + if m := depthRateRe.FindStringSubmatch(s); m != nil { + r.BitDepth, _ = strconv.Atoi(m[1]) + r.SampleRate = rate(m[2]) + } + if r.BitDepth == 0 { + if m := depthRe.FindStringSubmatch(s); m != nil { + r.BitDepth, _ = strconv.Atoi(m[1]) + } + } + if r.SampleRate == 0 { + if m := rateRe.FindStringSubmatch(s); m != nil { + r.SampleRate = rate(m[1]) + } + } + if r.BitDepth == 0 && hiResRe.MatchString(s) { + r.BitDepth = 24 + } + + if m := codecRateRe.FindStringSubmatch(s); m != nil { + r.Bitrate, _ = strconv.Atoi(m[1]) + } else if m := kbpsRe.FindStringSubmatch(s); m != nil { + r.Bitrate, _ = strconv.Atoi(m[1]) + } + // A bare "V0" is a LAME preset only beside MP3 or VBR; elsewhere it can + // be part of a title. + if m := vbrRe.FindStringSubmatch(s); m != nil && (r.Format == FormatMP3 || strings.Contains(s, "vbr")) { + r.VBR = "V" + m[1] + } + if r.Format == "" && (r.Bitrate > 0 || r.VBR != "") { + r.Format = FormatMP3 + } + + switch { + case sacdRe.MatchString(s): + r.Source = SourceSACD + case vinylRe.MatchString(s): + r.Source = SourceVinyl + case webRe.MatchString(s): + r.Source = SourceWEB + case cassetteRe.MatchString(s): + r.Source = SourceCassette + case cdRe.MatchString(s): + r.Source = SourceCD + } + + if m := videoRe.FindStringSubmatch(s); m != nil { + r.Video = m[1] + } + if m := discogRe.FindStringSubmatch(s); m != nil { + r.Discography = m[1] + } else if m := yearRangeRe.FindStringSubmatch(s); m != nil && m[2] > m[1] { + r.Discography = m[1] + "–" + m[2] + } + r.Year = year(s) + r.Tokens = Tokens(name) + return r +} + +// spaced lowercases a name and turns its separators into spaces, keeping +// the slash in "24/96" and "kb/s". +func spaced(name string) string { + var b strings.Builder + for _, c := range strings.ToLower(name) { + switch c { + case '_', '.', ',': + b.WriteByte(' ') + default: + b.WriteRune(c) + } + } + return b.String() +} + +// rate reads a sample rate in kHz; "44", "88" and "176" are the short +// spellings of 44.1, 88.2 and 176.4. +func rate(s string) float64 { + switch s { + case "44": + return 44.1 + case "88": + return 88.2 + case "176": + return 176.4 + } + f, _ := strconv.ParseFloat(strings.ReplaceAll(s, " ", "."), 64) + return f +} + +// formatRank orders formats a name might state together ("FLAC + MP3"): +// the best is the one it is described by. +var formatRank = map[string]int{ + FormatDSD: 11, FormatFLAC: 10, FormatALAC: 9, FormatWV: 8, FormatAPE: 7, FormatWAV: 6, FormatAIFF: 5, + FormatOpus: 4, FormatAAC: 3, FormatVorbis: 2, FormatMP3: 1, +} + +func bestFormat(found [][]string) string { + best := "" + for _, m := range found { + f := formatOf(m[1]) + if formatRank[f] > formatRank[best] { + best = f + } + } + return best +} + +// formatOf names the format a word or file extension stands for. +func formatOf(word string) string { + switch word { + case "flac": + return FormatFLAC + case "alac": + return FormatALAC + case "ape": + return FormatAPE + case "wavpack", "wv": + return FormatWV + case "wav": + return FormatWAV + case "aiff", "aif": + return FormatAIFF + case "dsd", "dsf", "dff": + return FormatDSD + case "mp3": + return FormatMP3 + case "aac", "m4a": + return FormatAAC + case "ogg", "vorbis": + return FormatVorbis + case "opus": + return FormatOpus + } + return "" +} + +// year is the release year a name states: one in brackets when there is +// one (so the title "1989" is not read as its year), else the last. +func year(s string) int { + if m := bracketYear.FindStringSubmatch(s); m != nil { + y, _ := strconv.Atoi(m[1]) + return y + } + all := yearRe.FindAllString(s, -1) + if len(all) == 0 { + return 0 + } + y, _ := strconv.Atoi(all[len(all)-1]) + return y +} + +// fold maps the accented letters music names carry to plain ones. +var fold = map[rune]string{ + 'à': "a", 'á': "a", 'â': "a", 'ã': "a", 'ä': "a", 'å': "a", 'ā': "a", 'ą': "a", + 'æ': "ae", 'ç': "c", 'ć': "c", 'č': "c", + 'è': "e", 'é': "e", 'ê': "e", 'ë': "e", 'ē': "e", 'ė': "e", 'ę': "e", 'ě': "e", + 'ì': "i", 'í': "i", 'î': "i", 'ï': "i", 'ī': "i", 'ı': "i", + 'ñ': "n", 'ń': "n", 'ň': "n", + 'ò': "o", 'ó': "o", 'ô': "o", 'õ': "o", 'ö': "o", 'ø': "o", 'ō': "o", 'ő': "o", + 'œ': "oe", 'ù': "u", 'ú': "u", 'û': "u", 'ü': "u", 'ū': "u", 'ů': "u", 'ű': "u", + 'ý': "y", 'ÿ': "y", 'ß': "ss", 'š': "s", 'ś': "s", 'ş': "s", + 'ž': "z", 'ź': "z", 'ż': "z", 'ł': "l", 'đ': "d", 'ð': "d", 'þ': "th", 'ř': "r", +} + +// abbreviations are spelled out on both sides before comparing. +var abbreviations = map[string]string{"vol": "volume", "pt": "part", "n": "and"} + +// Tokens are a name's words for matching: lowercase, accents folded, "&" +// and "+" read as "and", apostrophes dropped (so "Don't" is "dont"), every +// other mark a word break. +func Tokens(s string) []string { + var b strings.Builder + for _, c := range strings.ToLower(s) { + switch { + case c >= 'a' && c <= 'z', c >= '0' && c <= '9': + b.WriteRune(c) + case c == '\'' || c == '’' || c == '`' || c == '´': + case c == '&' || c == '+': + b.WriteString(" and ") + default: + if f, ok := fold[c]; ok { + b.WriteString(f) + } else if c > 127 && isLetter(c) { + b.WriteRune(c) + } else { + b.WriteByte(' ') + } + } + } + words := strings.Fields(b.String()) + for i, w := range words { + if full, ok := abbreviations[w]; ok { + words[i] = full + } + } + return words +} + +// isLetter keeps letters of scripts the fold table does not cover (Cyrillic, +// CJK) as letters rather than breaks. +func isLetter(c rune) bool { + return (c >= 0x0400 && c <= 0x04FF) || (c >= 0x0370 && c <= 0x03FF) || c >= 0x3040 +} diff --git a/internal/release/quality.go b/internal/release/quality.go new file mode 100644 index 00000000..456a3950 --- /dev/null +++ b/internal/release/quality.go @@ -0,0 +1,271 @@ +package release + +import ( + "fmt" + "strconv" + "strings" +) + +// Tier is a rung of the quality ladder. +type Tier string + +// The ladder, best first. +const ( + TierLosslessHiRes Tier = "lossless_hires" + TierLossless Tier = "lossless" + TierLossyHigh Tier = "lossy_high" + TierLossyStandard Tier = "lossy_standard" + TierLossyLow Tier = "lossy_low" + TierLossyPoor Tier = "lossy_poor" + TierLossyTrash Tier = "lossy_trash" + TierUnknown Tier = "unknown" +) + +// losslessCategory is Newznab's Audio/Lossless. +const losslessCategory = 3040 + +// TierInfo is a rung as the settings page shows it. +type TierInfo struct { + Tier Tier `json:"tier"` + Label string `json:"label"` + What string `json:"what"` +} + +// Tiers is the ladder, best first. +var Tiers = []TierInfo{ + {TierLosslessHiRes, "Lossless, hi-res", "24-bit or above 48 kHz: FLAC, ALAC, DSD"}, + {TierLossless, "Lossless", "FLAC, ALAC, APE, WavPack, WAV at CD quality"}, + {TierLossyHigh, "High quality lossy", "MP3 320 or V0, AAC 256 and up"}, + {TierLossyStandard, "Standard quality lossy", "MP3 224–256 or V1/V2, AAC 192, Vorbis, Opus, or a lossy format with no bitrate stated"}, + {TierLossyLow, "Low quality lossy", "160–192 kbps"}, + {TierLossyPoor, "Poor quality lossy", "128 kbps"}, + {TierLossyTrash, "Trash quality lossy", "Under 128 kbps"}, + {TierUnknown, "Unknown", "Nothing says what format it is"}, +} + +// DefaultTiers is what a new install allows: every format from trash lossy +// to lossless. Unknown is left out; a +// release that names no format has its .torrent read to find out. +var DefaultTiers = []Tier{ + TierLosslessHiRes, TierLossless, TierLossyHigh, TierLossyStandard, + TierLossyLow, TierLossyPoor, TierLossyTrash, +} + +// KnownTier says whether t is a rung of the ladder. +func KnownTier(t string) bool { + for _, info := range Tiers { + if string(info.Tier) == t { + return true + } + } + return false +} + +// Rank orders tiers: higher is better, Unknown lowest. +func Rank(t Tier) int { + for i, info := range Tiers { + if info.Tier == t { + return len(Tiers) - i + } + } + return 0 +} + +// Label is a tier's name for a person. +func Label(t Tier) string { + for _, info := range Tiers { + if info.Tier == t { + return info.Label + } + } + return string(t) +} + +// TierOf places a release on the ladder. category is the indexer's; 3040 +// (Lossless) stands in for a name that states no format. +func TierOf(r Release, category int) Tier { + switch r.Format { + case FormatDSD: + return TierLosslessHiRes + case FormatFLAC, FormatALAC, FormatAPE, FormatWV, FormatWAV, FormatAIFF: + if r.BitDepth >= 24 || r.SampleRate > 48 { + return TierLosslessHiRes + } + return TierLossless + case FormatMP3: + switch r.VBR { + case "V0": + return TierLossyHigh + case "V1", "V2": + return TierLossyStandard + } + return byBitrate(r.Bitrate, 320) + case FormatAAC, FormatVorbis, FormatOpus: + return byBitrate(r.Bitrate, 256) + } + if category == losslessCategory { + return TierLossless + } + return TierUnknown +} + +// byBitrate places a lossy bitrate; high is the bitrate from which the codec +// counts as high quality. An unstated bitrate is standard. +func byBitrate(kbps, high int) Tier { + switch { + case kbps == 0: + return TierLossyStandard + case kbps >= high: + return TierLossyHigh + case kbps >= 224 || (high == 256 && kbps >= 192): + return TierLossyStandard + case kbps >= 160: + return TierLossyLow + case kbps >= 128: + return TierLossyPoor + } + return TierLossyTrash +} + +// Describe is a release's format in a few words: "FLAC 24-bit/96 kHz", +// "MP3 V0", "AAC 256". Empty when the name states none. +func Describe(r Release) string { + if r.Format == "" { + return "" + } + parts := []string{r.Format} + switch { + case r.BitDepth > 0 && r.SampleRate > 0: + parts = append(parts, fmt.Sprintf("%d-bit/%s kHz", r.BitDepth, khz(r.SampleRate))) + case r.BitDepth > 0: + parts = append(parts, fmt.Sprintf("%d-bit", r.BitDepth)) + case r.VBR != "": + parts = append(parts, r.VBR) + case r.Bitrate > 0: + parts = append(parts, strconv.Itoa(r.Bitrate)) + } + return strings.Join(parts, " ") +} + +func khz(f float64) string { + return strings.TrimSuffix(strconv.FormatFloat(f, 'f', 1, 64), ".0") +} + +// Profile is which tiers an install takes. +type Profile struct { + Allowed map[Tier]bool +} + +// NewProfile builds a profile from the stored tier list. +func NewProfile(tiers []string) Profile { + p := Profile{Allowed: map[Tier]bool{}} + for _, t := range tiers { + p.Allowed[Tier(t)] = true + } + return p +} + +// Allows says whether the profile takes a tier. +func (p Profile) Allows(t Tier) bool { return p.Allowed[t] } + +// Held is what the library's copy of an album is: its tier, estimated from +// what the scanner records, and the same in words. +type Held struct { + Tier Tier + Words string +} + +// HeldFormat is an album's present tracks of one format, summed. +type HeldFormat struct { + Format string + Bytes, Ms int64 +} + +// HeldCopy judges an album's copy from its tracks grouped by format: the +// lowest tier among them: an album is as good as its worst file. +// Summing a format's tracks first keeps a near-silent VBR interlude from +// dragging the album down. ok is false when no format can be placed. +func HeldCopy(formats []HeldFormat) (Held, bool) { + var out Held + found := false + for _, f := range formats { + kbps := FileKbps(f.Bytes, f.Ms) + t, ok := TierOfFile(f.Format, kbps) + if !ok { + continue + } + if !found || Rank(t) < Rank(out.Tier) { + out = Held{Tier: t, Words: HeldWords(f.Format, t, kbps)} + found = true + } + } + return out, found +} + +// FileKbps is the average bitrate of files totalling bytes over ms of audio: +// the scanner records no bitrate (tracks.bitrate is never filled), so size +// per second is the measure there is. Cover art and tags count too, which +// only ever reads high. +func FileKbps(bytes, ms int64) int { + if bytes <= 0 || ms <= 0 { + return 0 + } + return int(bytes * 8 / ms) +} + +// mp3High is the average bitrate from which an MP3 counts as high quality +// when only its size is known. A V0 encode averages 220 to 260 kbps, so the +// line sits below it: an estimate errs toward the copy held, and an album is +// never fetched again on a guess. +const mp3High = 220 + +// TierOfFile places files of one format on the ladder from their average +// bitrate (FileKbps). format is the scanner's extension, without the dot. +// ok is false for a format the ladder cannot place. +func TierOfFile(format string, kbps int) (Tier, bool) { + switch strings.ToLower(strings.TrimPrefix(format, ".")) { + case "flac", "wav", "aiff", "aif", "ape", "wv", "alac": + // 16-bit/44.1 kHz stereo is 1,411 kbps before compression; past + // 1,600 on average is more bits or a higher rate than CD. + if kbps > 1600 { + return TierLosslessHiRes, true + } + return TierLossless, true + case "dsf", "dff": + return TierLosslessHiRes, true + case "m4a", "mp4": + // ALAC or AAC: the container says nothing, the rate does. No AAC + // encode runs near 500 kbps. + if kbps > 500 { + return TierLossless, true + } + return byBitrate(kbps, 256), kbps > 0 + case "mp3": + if kbps >= mp3High { + return TierLossyHigh, true + } + return byBitrate(kbps, 320), kbps > 0 + case "aac", "ogg", "oga", "opus": + return byBitrate(kbps, 256), kbps > 0 + } + return TierUnknown, false +} + +// HeldWords says what a held copy is: "FLAC", "MP3, about 192 kbps". +func HeldWords(format string, tier Tier, kbps int) string { + f := strings.ToUpper(strings.TrimPrefix(format, ".")) + switch tier { + case TierLosslessHiRes: + return f + ", hi-res" + case TierLossless: + return f + } + if kbps <= 0 { + return f + } + return fmt.Sprintf("%s, about %d kbps", f, roundKbps(kbps)) +} + +// roundKbps rounds an average bitrate to the nearest 8, so a 191 reads as +// the 192 it was encoded at. +func roundKbps(k int) int { return (k + 4) / 8 * 8 } diff --git a/internal/release/release_test.go b/internal/release/release_test.go new file mode 100644 index 00000000..46ab4456 --- /dev/null +++ b/internal/release/release_test.go @@ -0,0 +1,311 @@ +package release + +import ( + "crypto/sha1" + "encoding/hex" + "fmt" + "strings" + "testing" +) + +func TestParse_ReadsWhatTheNameSays(t *testing.T) { + for _, c := range []struct { + name string + format, vbr, src string + bitrate, depth, yr int + rate float64 + tier Tier + }{ + // A bracketed edition with its own year: not the original album. + {"Daft Punk Random Access Memories (Drumless Edition 2023) [Flac 24-88]", FormatFLAC, "", "", 0, 24, 2023, 88.2, TierLosslessHiRes}, + {"Gorillaz - Humanz - (Japanese Edition) - [CD-FLAC] 2017", FormatFLAC, "", SourceCD, 0, 0, 2017, 0, TierLossless}, + {"Radiohead - OK Computer (1997) [MP3 320]", FormatMP3, "", "", 320, 0, 1997, 0, TierLossyHigh}, + {"Artist - Album 2019 MP3 V0", FormatMP3, "V0", "", 0, 0, 2019, 0, TierLossyHigh}, + {"Artist - Album (2020) [WEB] [AAC 256]", FormatAAC, "", SourceWEB, 256, 0, 2020, 0, TierLossyHigh}, + {"Artist_-_Album-(2004)-192kbps", FormatMP3, "", "", 192, 0, 2004, 0, TierLossyLow}, + {"Artist - Album [24bit-96kHz] FLAC Vinyl", FormatFLAC, "", SourceVinyl, 0, 24, 0, 96, TierLosslessHiRes}, + {"Taylor Swift - 1989 (2014) [FLAC]", FormatFLAC, "", "", 0, 0, 2014, 0, TierLossless}, + {"Artist - Album", "", "", "", 0, 0, 0, 0, TierUnknown}, + } { + r := Parse(c.name) + got := fmt.Sprint(r.Format, r.VBR, r.Source, r.Bitrate, r.BitDepth, r.Year, r.SampleRate, TierOf(r, 0)) + want := fmt.Sprint(c.format, c.vbr, c.src, c.bitrate, c.depth, c.yr, c.rate, c.tier) + if got != want { + t.Errorf("Parse(%q)\n got %s\nwant %s", c.name, got, want) + } + } +} + +func TestParse_VideoAndDiscographies(t *testing.T) { + if v := Parse("Artist.Album.2018.1080p.BluRay.x264").Video; v != "1080p" { + t.Errorf("video = %q", v) + } + if d := Parse("Artist Discography 1994-2010 FLAC").Discography; d != "discography" { + t.Errorf("discography = %q", d) + } + if d := Parse("Artist - Studio Albums 1994 - 2010 [MP3]").Discography; d != "1994–2010" { + t.Errorf("year range = %q", d) + } + if d := Parse("Artist - Album (2010) [FLAC]").Discography; d != "" { + t.Errorf("one year read as a range: %q", d) + } +} + +func judge(w Want, name string) Match { return Judge(w, Parse(name)) } + +func TestJudge_TheAlbumAndWhatIsNot(t *testing.T) { + ram := Want{Artist: "Daft Punk", Title: "Random Access Memories", Year: 2013, PrimaryType: "Album"} + for name, veto := range map[string]string{ + "Daft Punk - Random Access Memories (2013) [FLAC]": "", + "Daft_Punk-Random_Access_Memories-WEB-2013": "", + "Daft Punk Random Access Memories (Drumless Edition 2023) [Flac 24-88]": VetoVersion, + "Daft Punk - Random Access Memories (Instrumentals) [MP3]": VetoVersion, + "Daft Punk - Random Access Memories Remixes": VetoVersion, + "Daft Punk - Random Access Memories (Live) FLAC": VetoVersion, + "Daft Punk - Random Access Memories (BBC Sessions) FLAC": VetoVersion, + "Daft Punk - Random Access Memories Unplugged [MP3]": VetoVersion, + "Daft Punk - Discovery (2001) [FLAC]": VetoTitle, + "Random Access Memories (2013) [FLAC]": VetoArtist, + "Daft Punk - Discography 1997-2013 FLAC": VetoTitle, + "Daft Punk - Random Access Memories + Discovery + Homework Collection": VetoDiscography, + "Daft Punk - Random Access Memories (2013) 1080p BluRay": VetoVideo, + "Daft Punk - Random Access Memories (Single) 2013": VetoType, + "Daft Punk - Random Access Memories (Deluxe 10th Anniversary Edition) 2023": "", + } { + if got := judge(ram, name).Veto; got != veto { + t.Errorf("%q: veto %q, want %q", name, got, veto) + } + } + if !judge(ram, "Daft Punk - Random Access Memories (2013) [FLAC]").YearAgrees { + t.Error("the year did not agree") + } +} + +func TestJudge_WordsTheWantedTitleHoldsDoNotVeto(t *testing.T) { + hole := Want{Artist: "Hole", Title: "Live Through This", PrimaryType: "Album"} + if m := judge(hole, "Hole - Live Through This (1994) [FLAC]"); m.Veto != "" { + t.Errorf("Live Through This: %s", m.Reason) + } + alive := Want{Artist: "Daft Punk", Title: "Alive 2007", PrimaryType: "Album", SecondaryTypes: []string{"Live"}} + if m := judge(alive, "Daft Punk - Alive 2007 (Live) [FLAC]"); m.Veto != "" { + t.Errorf("a wanted live album: %s", m.Reason) + } + alive.SecondaryTypes = nil + if m := judge(alive, "Daft Punk - Alive 2007 (Live) [FLAC]"); m.Veto != VetoVersion { + t.Errorf("live without the Live type: veto %q", m.Veto) + } +} + +func TestJudge_SpellingsOfNames(t *testing.T) { + for _, c := range []struct { + w Want + name string + }{ + {Want{Artist: "AC/DC", Title: "Back in Black"}, "ACDC - Back In Black (1980) FLAC"}, + {Want{Artist: "AC/DC", Title: "Back in Black"}, "AC-DC - Back in Black [MP3 320]"}, + {Want{Artist: "Sigur Rós", Title: "Ágætis byrjun"}, "Sigur Ros - Agaetis Byrjun (1999) [FLAC]"}, + {Want{Artist: "Simon & Garfunkel", Title: "Bookends"}, "Simon and Garfunkel - Bookends"}, + {Want{Artist: "The Beatles", Title: "Abbey Road"}, "Beatles - Abbey Road (Remastered 2009)"}, + {Want{Artist: "Florence + the Machine", Title: "Lungs"}, "Florence and The Machine - Lungs (2009)"}, + {Want{Artist: "Guns N' Roses", Title: "Appetite for Destruction"}, "Guns N Roses - Appetite For Destruction"}, + } { + if m := judge(c.w, c.name); m.Veto != "" { + t.Errorf("%q for %s / %s: %s", c.name, c.w.Artist, c.w.Title, m.Reason) + } + } + // One word is never matched run together: Muse is not inside Museum. + if m := judge(Want{Artist: "Muse", Title: "Absolution"}, "Museum - Absolution"); m.Veto != VetoArtist { + t.Errorf("Museum read as Muse: %q", m.Veto) + } +} + +func TestJudge_ASelfTitledAlbumNeedsMoreThanTheName(t *testing.T) { + weezer := Want{Artist: "Weezer", Title: "Weezer", Year: 1994, PrimaryType: "Album"} + if m := judge(weezer, "Weezer - Weezer (Blue Album) [FLAC]"); m.Doubt != "" || m.Veto != "" { + t.Errorf("named twice: %+v", m) + } + if m := judge(weezer, "Weezer (1994) [FLAC]"); m.Doubt != "" { + t.Errorf("the year agrees: %+v", m) + } + if m := judge(weezer, "Weezer (Green Album) FLAC"); m.Doubt == "" { + t.Error("a self-titled name with nothing else was taken as sure") + } +} + +func TestTierOf_TheLadder(t *testing.T) { + for _, c := range []struct { + r Release + cat int + want Tier + }{ + {Release{Format: FormatMP3, Bitrate: 256}, 0, TierLossyStandard}, + {Release{Format: FormatMP3, Bitrate: 128}, 0, TierLossyPoor}, + {Release{Format: FormatMP3, Bitrate: 96}, 0, TierLossyTrash}, + {Release{Format: FormatMP3, VBR: "V2"}, 0, TierLossyStandard}, + {Release{Format: FormatMP3}, 0, TierLossyStandard}, + {Release{Format: FormatAAC, Bitrate: 192}, 0, TierLossyStandard}, + {Release{Format: FormatALAC, SampleRate: 96}, 0, TierLosslessHiRes}, + {Release{Format: FormatDSD}, 0, TierLosslessHiRes}, + {Release{}, 3040, TierLossless}, + {Release{}, 3010, TierUnknown}, + } { + if got := TierOf(c.r, c.cat); got != c.want { + t.Errorf("TierOf(%+v, %d) = %s, want %s", c.r, c.cat, got, c.want) + } + } + if Rank(TierLossless) <= Rank(TierLossyHigh) || Rank(TierUnknown) >= Rank(TierLossyTrash) { + t.Error("the ladder is out of order") + } + if !NewProfile([]string{"lossless"}).Allows(TierLossless) || NewProfile([]string{"lossless"}).Allows(TierUnknown) { + t.Error("profile") + } +} + +func files(paths ...string) []File { + out := make([]File, len(paths)) + for i, p := range paths { + out[i] = File{Index: i, Path: p, Size: 1 << 20} + } + return out +} + +func numbered(n int, ext string) []string { + var out []string + for i := 1; i <= n; i++ { + out = append(out, fmt.Sprintf("Album/%02d - Track.%s", i, ext)) + } + return out +} + +func TestJudgeFiles_WhatTheTorrentHolds(t *testing.T) { + w := Want{MinTracks: 10, MaxTracks: 13} + if veto, _, _ := JudgeFiles(w, files(numbered(13, "flac")...)); veto != "" { + t.Errorf("a whole album: %s", veto) + } + if veto, reason, _ := JudgeFiles(w, files(numbered(3, "flac")...)); veto != VetoIncomplete || !strings.Contains(reason, "3 tracks") { + t.Errorf("three tracks: %s %s", veto, reason) + } + if veto, _, _ := JudgeFiles(w, files("Concert/Concert.mkv", "Concert/cover.jpg")); veto != VetoVideo { + t.Errorf("a video: %s", veto) + } + if veto, _, _ := JudgeFiles(w, files("Album.rar", "Album.r00")); veto != VetoNoAudio { + t.Errorf("an archive: %s", veto) + } + if _, _, doubt := JudgeFiles(w, files(numbered(60, "mp3")...)); doubt == "" { + t.Error("sixty tracks for a thirteen-track album raised no doubt") + } + if c := Inspect(files(append(numbered(12, "flac"), "Album/cover.jpg", "Album/album.cue")...)); c.Format != FormatFLAC || c.Audio != 12 { + t.Errorf("inspect = %+v", c) + } +} + +// bencode writes the few shapes a test torrent needs. +func bencode(v any) string { + switch x := v.(type) { + case string: + return fmt.Sprintf("%d:%s", len(x), x) + case int: + return fmt.Sprintf("i%de", x) + case []any: + var b strings.Builder + b.WriteString("l") + for _, e := range x { + b.WriteString(bencode(e)) + } + return b.String() + "e" + case [][2]any: // an ordered dictionary + var b strings.Builder + b.WriteString("d") + for _, kv := range x { + b.WriteString(bencode(kv[0]) + bencode(kv[1])) + } + return b.String() + "e" + } + panic(fmt.Sprintf("bencode %T", v)) +} + +func TestParseTorrent_FilesAndInfohash(t *testing.T) { + info := bencode([][2]any{ + {"files", []any{ + [][2]any{{"length", 300}, {"path", []any{"01 - One.flac"}}}, + [][2]any{{"attr", "p"}, {"length", 12}, {"path", []any{".pad", "12"}}}, + [][2]any{{"length", 200}, {"path", []any{"Scans", "front.jpg"}}}, + }}, + {"name", "Artist - Album"}, + {"piece length", 16384}, + {"pieces", strings.Repeat("x", 20)}, + }) + body := "d" + bencode("announce") + bencode("http://tracker/") + bencode("info") + info + "e" + meta, err := ParseTorrent([]byte(body)) + if err != nil { + t.Fatal(err) + } + sum := sha1.Sum([]byte(info)) + if meta.InfoHash != hex.EncodeToString(sum[:]) { + t.Errorf("infohash = %s", meta.InfoHash) + } + if len(meta.Files) != 2 || meta.Files[0].Path != "Artist - Album/01 - One.flac" || meta.Files[1].Path != "Artist - Album/Scans/front.jpg" { + t.Errorf("files = %+v", meta.Files) + } + if _, err := ParseTorrent([]byte("not a torrent")); err == nil { + t.Error("HTML parsed as a torrent") + } +} + +func TestTierOfFile(t *testing.T) { + cases := []struct { + format string + kbps int + want Tier + ok bool + }{ + {"flac", 900, TierLossless, true}, + {"FLAC", 2800, TierLosslessHiRes, true}, + {"m4a", 750, TierLossless, true}, + {"m4a", 256, TierLossyHigh, true}, + {"mp3", 320, TierLossyHigh, true}, + {"mp3", 245, TierLossyHigh, true}, // a V0, by size + {"mp3", 192, TierLossyLow, true}, + {"mp3", 128, TierLossyPoor, true}, + {"mp3", 96, TierLossyTrash, true}, + {"opus", 160, TierLossyLow, true}, + {"wma", 192, TierUnknown, false}, + {"mp3", 0, TierLossyStandard, false}, + } + for _, c := range cases { + got, ok := TierOfFile(c.format, c.kbps) + if got != c.want || ok != c.ok { + t.Errorf("TierOfFile(%q, %d) = %s, %v; want %s, %v", c.format, c.kbps, got, ok, c.want, c.ok) + } + } +} + +func TestFileKbpsAndWords(t *testing.T) { + // 5 MB over 3:30 of audio is about 190 kbps. + k := FileKbps(5_000_000, 210_000) + if k != 190 { + t.Fatalf("FileKbps = %d, want 190", k) + } + if got := HeldWords("mp3", TierLossyLow, 191); got != "MP3, about 192 kbps" { + t.Errorf("HeldWords = %q", got) + } + if got := HeldWords("flac", TierLossless, 900); got != "FLAC" { + t.Errorf("HeldWords = %q", got) + } + if FileKbps(0, 1000) != 0 || FileKbps(1000, 0) != 0 { + t.Error("FileKbps with nothing to measure is 0") + } +} + +func TestHeldCopy_TakesTheWorstFormat(t *testing.T) { + h, ok := HeldCopy([]HeldFormat{ + {Format: "flac", Bytes: 300_000_000, Ms: 2_400_000}, + {Format: "mp3", Bytes: 4_800_000, Ms: 200_000}, // 192 kbps + }) + if !ok || h.Tier != TierLossyLow || h.Words != "MP3, about 192 kbps" { + t.Fatalf("HeldCopy = %+v, %v", h, ok) + } + if _, ok := HeldCopy([]HeldFormat{{Format: "wma", Bytes: 1, Ms: 1}}); ok { + t.Error("a format the ladder can't place says nothing") + } +} diff --git a/internal/release/torrent.go b/internal/release/torrent.go new file mode 100644 index 00000000..15c844d4 --- /dev/null +++ b/internal/release/torrent.go @@ -0,0 +1,178 @@ +package release + +import ( + "bytes" + "crypto/sha1" + "encoding/hex" + "errors" + "fmt" + "strconv" + "strings" +) + +// Reading a .torrent file. A .torrent +// is the metadata: its info dictionary names every file, and its SHA-1 is the +// infohash. So a release offered as a .torrent link can be judged on what it +// holds before any of it is downloaded. Bencode is four shapes; walking them +// here costs less than a dependency. + +// TorrentMeta is what a .torrent says about itself. +type TorrentMeta struct { + // InfoHash is the v1 infohash: the SHA-1 of the info dictionary's own + // bytes, lowercase hex. + InfoHash string + Name string + Files []File +} + +// ErrNotTorrent is bytes that are not a .torrent. +var ErrNotTorrent = errors.New("not a .torrent file") + +// maxTorrentBytes bounds what is read, as Exchange bounds what it accepts. +const maxTorrentBytes = 10 << 20 + +// ParseTorrent reads a .torrent's info dictionary. +func ParseTorrent(b []byte) (TorrentMeta, error) { + if len(b) == 0 || len(b) > maxTorrentBytes || b[0] != 'd' { + return TorrentMeta{}, ErrNotTorrent + } + d := &bdecoder{b: b} + root, err := d.value(0) + if err != nil { + return TorrentMeta{}, fmt.Errorf("%w: %v", ErrNotTorrent, err) + } + top, ok := root.(map[string]any) + if !ok || d.infoEnd == 0 { + return TorrentMeta{}, fmt.Errorf("%w: no info dictionary", ErrNotTorrent) + } + info, ok := top["info"].(map[string]any) + if !ok { + return TorrentMeta{}, fmt.Errorf("%w: no info dictionary", ErrNotTorrent) + } + sum := sha1.Sum(b[d.infoStart:d.infoEnd]) + out := TorrentMeta{InfoHash: hex.EncodeToString(sum[:])} + out.Name, _ = info["name"].(string) + + files, multi := info["files"].([]any) + if !multi { + // A single-file torrent: the name IS the file. + length, _ := info["length"].(int64) + out.Files = []File{{Path: out.Name, Size: length}} + return out, nil + } + for index, f := range files { + entry, ok := f.(map[string]any) + if !ok { + continue + } + parts, _ := entry["path"].([]any) + segs := []string{out.Name} + for _, p := range parts { + if s, ok := p.(string); ok { + segs = append(segs, s) + } + } + length, _ := entry["length"].(int64) + // BEP 47 padding files carry no content. + if attr, _ := entry["attr"].(string); strings.Contains(attr, "p") { + continue + } + out.Files = append(out.Files, File{Index: index, Path: strings.Join(segs, "/"), Size: length}) + } + return out, nil +} + +// bdecoder walks bencode, and remembers where the top-level info dictionary +// sits so its raw bytes can be hashed exactly as published. +type bdecoder struct { + b []byte + i int + infoStart, infoEnd int +} + +// maxDepth: a real torrent nests four deep. Anything past this is hostile. +const maxDepth = 32 + +func (d *bdecoder) value(depth int) (any, error) { + if depth > maxDepth { + return nil, errors.New("nested too deep") + } + if d.i >= len(d.b) { + return nil, errors.New("truncated") + } + switch c := d.b[d.i]; { + case c == 'i': + end := bytes.IndexByte(d.b[d.i:], 'e') + if end < 0 { + return nil, errors.New("unterminated integer") + } + n, err := strconv.ParseInt(string(d.b[d.i+1:d.i+end]), 10, 64) + if err != nil { + return nil, err + } + d.i += end + 1 + return n, nil + case c == 'l': + d.i++ + var out []any + for d.i < len(d.b) && d.b[d.i] != 'e' { + v, err := d.value(depth + 1) + if err != nil { + return nil, err + } + out = append(out, v) + } + if d.i >= len(d.b) { + return nil, errors.New("unterminated list") + } + d.i++ + return out, nil + case c == 'd': + d.i++ + out := map[string]any{} + for d.i < len(d.b) && d.b[d.i] != 'e' { + k, err := d.str() + if err != nil { + return nil, err + } + start := d.i + v, err := d.value(depth + 1) + if err != nil { + return nil, err + } + if depth == 0 && k == "info" { + d.infoStart, d.infoEnd = start, d.i + } + // Pieces are 20 bytes a piece and never read: not kept. + if k != "pieces" { + out[k] = v + } + } + if d.i >= len(d.b) { + return nil, errors.New("unterminated dictionary") + } + d.i++ + return out, nil + case c >= '0' && c <= '9': + return d.str() + default: + return nil, fmt.Errorf("unexpected %q", c) + } +} + +func (d *bdecoder) str() (string, error) { + colon := bytes.IndexByte(d.b[d.i:], ':') + if colon < 0 { + return "", errors.New("unterminated string length") + } + n, err := strconv.Atoi(string(d.b[d.i : d.i+colon])) + if err != nil || n < 0 { + return "", errors.New("bad string length") + } + start := d.i + colon + 1 + if start+n > len(d.b) { + return "", errors.New("string runs past the end") + } + d.i = start + n + return string(d.b[start : start+n]), nil +} diff --git a/internal/release/version.go b/internal/release/version.go new file mode 100644 index 00000000..6a849fe2 --- /dev/null +++ b/internal/release/version.go @@ -0,0 +1,79 @@ +package release + +import "strings" + +// TitleVersion is the version of a recording a release group's title names +// in its qualifier: the words in brackets, or after a dash. MusicBrainz +// does not always type such a group: My Chemical Romance's "Common People +// (Recorded For BBC Radio 1)" is a plain Single. Typ is the +// MusicBrainz secondary type the version is, or "" for one no type names +// (an acoustic or instrumental version); Word is the word that said so. +type TitleVersion struct { + Word string + What string + Typ string +} + +// titleVersions are the qualifier words that name a version, in the order +// they are looked for: "rough mix" is a demo before "mix" is a remix. +var titleVersions = []struct { + words []string + what string + typ string +}{ + {[]string{"demo", "demos", "outtake", "outtakes", "rehearsal", "rough mix"}, "demo recordings", "Demo"}, + {[]string{"live", "bootleg", "bbc", "session", "sessions", "unplugged", "recorded for", "in concert"}, "a live recording", "Live"}, + {[]string{"remix", "remixes", "remixed", "rmx", "mix", "mixes"}, "a remix", "Remix"}, + {[]string{"instrumental", "instrumentals", "acoustic", "karaoke", "acapella", "acappella", "a cappella", + "drumless", "nightcore", "slowed", "sped up"}, "another version", ""}, + {[]string{"commentary", "interview", "interviews"}, "commentary or interviews", "Interview"}, +} + +// VersionInTitle reads a release group's title for the version it names, +// if any. Only the qualifier counts, so "Live Through This" and "Live +// Forever" name no version. +func VersionInTitle(title string) (TitleVersion, bool) { + for _, q := range qualifiers(title) { + tokens := Tokens(q) + for _, v := range titleVersions { + for _, w := range v.words { + if contains(tokens, strings.Fields(w)) { + return TitleVersion{Word: w, What: v.what, Typ: v.typ}, true + } + } + } + } + return TitleVersion{}, false +} + +// qualifiers are the parts of a title in brackets, and what follows a +// spaced dash. +func qualifiers(title string) []string { + var out []string + depth, start := 0, 0 + for i, c := range title { + switch c { + case '(', '[', '{': + if depth == 0 { + start = i + 1 + } + depth++ + case ')', ']', '}': + if depth > 0 { + depth-- + if depth == 0 { + out = append(out, title[start:i]) + } + } + } + } + if depth > 0 { + out = append(out, title[start:]) + } + for _, dash := range []string{" - ", " – ", " — "} { + if i := strings.Index(title, dash); i >= 0 { + out = append(out, title[i+len(dash):]) + } + } + return out +} diff --git a/internal/release/version_test.go b/internal/release/version_test.go new file mode 100644 index 00000000..7f9b4fc3 --- /dev/null +++ b/internal/release/version_test.go @@ -0,0 +1,32 @@ +package release + +import "testing" + +func TestVersionInTitle(t *testing.T) { + for title, want := range map[string]string{ + // Groups MusicBrainz types as plain Singles, named by their title. + "Common People (Recorded For BBC Radio 1)": "Live", + "Welcome to the Black Parade (Steve Aoki remix)": "Remix", + "The Five of Us Are Dying (Rough mix)": "Demo", + "I'm Not Okay (I Promise) [outtake version 2]": "Demo", + "Na Na Na (live from Sessions@AOL)": "Live", + "Song - Live at Wembley": "Live", + "Song – Club Mix": "Remix", + "Helena (Acoustic)": "", + "Teenagers (a cappella)": "", + "Hybrid Theory (20th Anniversary Edition)": "none", + "Live Through This": "none", + "Live Forever": "none", + "The Black Parade": "none", + "Warped Tour ’05: Bootleg Series (live at Warped Tour": "Live", // unclosed + } { + v, ok := VersionInTitle(title) + got := v.Typ + if !ok { + got = "none" + } + if got != want { + t.Errorf("VersionInTitle(%q) = %q (%q), want %q", title, got, v.Word, want) + } + } +} diff --git a/internal/requests/reconciler.go b/internal/requests/reconciler.go new file mode 100644 index 00000000..06593805 --- /dev/null +++ b/internal/requests/reconciler.go @@ -0,0 +1,198 @@ +package requests + +import ( + "context" + "errors" + "log/slog" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/eventbus" + "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" +) + +// Reconciler is a background worker that completes approved requests once +// the library holds what they asked for: an artist request when the artist +// is in the library, an album or track request when the want it recorded is +// held (internal/wanted settles wants). Errors per row are logged at WARN and +// do not abort the tick. +type Reconciler struct { + pool *pgxpool.Pool + logger *slog.Logger + bus *eventbus.Bus + notifier *notifications.Notifier // nil: no inbox notifications + tick time.Duration + batch int32 +} + +// NewReconciler constructs a Reconciler with production defaults: 5-minute +// tick, batch size 50. Pass nil for bus when no SSE publishing is desired. +func NewReconciler(pool *pgxpool.Pool, logger *slog.Logger, bus *eventbus.Bus) *Reconciler { + return &Reconciler{pool: pool, logger: logger, bus: bus, tick: 5 * time.Minute, batch: 50} +} + +// SetNotifier makes completions land in the requester's notifications inbox. +// Without one, completions are only broadcast on the bus. +func (r *Reconciler) SetNotifier(n *notifications.Notifier) { r.notifier = n } + +// publishCompleted tells the requester their request arrived: a +// request.status_changed event for open screens, and a notification that +// outlives the connection. +func (r *Reconciler) publishCompleted(ctx context.Context, row dbq.Request) { + p := notifications.Payload{ + RequestID: formatUUIDForBus(row.ID), + RequestKind: string(row.Kind), + Name: DisplayName(row), + Artist: row.ArtistName, + Title: requestedTitle(row), + } + if row.MatchedAlbumID.Valid { + p.AlbumID = formatUUIDForBus(row.MatchedAlbumID) + } else if row.MatchedArtistID.Valid { + p.ArtistID = formatUUIDForBus(row.MatchedArtistID) + } + r.notifier.NotifyLogged(ctx, notifications.KindRequestCompleted, notifications.ToUser(row.UserID), p.Map()) + + if r.bus == nil { + return + } + r.bus.Publish(eventbus.Event{ + Kind: "request.status_changed", + UserID: formatUUIDForBus(row.UserID), + Data: map[string]any{ + "request_id": formatUUIDForBus(row.ID), + "status": "completed", + "kind": string(row.Kind), + }, + }) +} + +// formatUUIDForBus renders a pgtype.UUID as the canonical 8-4-4-4-12 hex +// string. Mirrors helpers in internal/api/convert.go and other packages; +// inlined here to avoid a back-edge dependency from requests onto +// the api layer. +func formatUUIDForBus(u pgtype.UUID) string { + if !u.Valid { + return "" + } + const hex = "0123456789abcdef" + out := make([]byte, 36) + pos := 0 + for i, b := range u.Bytes { + if i == 4 || i == 6 || i == 8 || i == 10 { + out[pos] = '-' + pos++ + } + out[pos] = hex[b>>4] + out[pos+1] = hex[b&0xf] + pos += 2 + } + return string(out) +} + +// Run blocks until ctx is cancelled, ticking every r.tick. Errors from +// tickOnce are logged at WARN and never propagated. +func (r *Reconciler) Run(ctx context.Context) { + t := time.NewTicker(r.tick) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + if err := r.tickOnce(ctx); err != nil { + r.logger.Warn("requests: reconciler tick failed", "err", err) + } + } + } +} + +// tickOnce loads approved requests (oldest first, up to r.batch) and +// completes those the library now holds. +func (r *Reconciler) tickOnce(ctx context.Context) error { + q := dbq.New(r.pool) + rows, err := q.ListApprovedRequestsForReconcile(ctx, r.batch) + if err != nil { + return err + } + for _, row := range rows { + if err := r.reconcileRow(ctx, q, row); err != nil { + r.logger.Warn("requests: reconciler row failed", + "request_id", row.ID, "kind", row.Kind, "err", err) + } + } + return nil +} + +// reconcileRow completes one request if the library holds it. A request +// not yet held is not an error. +func (r *Reconciler) reconcileRow(ctx context.Context, q *dbq.Queries, row dbq.Request) error { + if row.Kind == dbq.RequestKindArtist { + return r.reconcileArtist(ctx, q, row) + } + return r.reconcileAlbum(ctx, q, row) +} + +func (r *Reconciler) reconcileArtist(ctx context.Context, q *dbq.Queries, row dbq.Request) error { + var artistID pgtype.UUID + err := r.pool.QueryRow(ctx, "SELECT id FROM artists WHERE mbid = $1", row.ArtistMbid).Scan(&artistID) + if errors.Is(err, pgx.ErrNoRows) { + return nil // not in library yet + } + if err != nil { + return err + } + completed, err := q.CompleteRequest(ctx, dbq.CompleteRequestParams{ID: row.ID, MatchedArtistID: artistID}) + if err != nil { + return err + } + r.publishCompleted(ctx, completed) + return nil +} + +// reconcileAlbum completes an album or track request once its want is held, +// matched to the album that answered it. A track request is matched to the +// track it named when that recording arrived, else the newest arrival. +func (r *Reconciler) reconcileAlbum(ctx context.Context, q *dbq.Queries, row dbq.Request) error { + if row.AlbumMbid == nil || *row.AlbumMbid == "" { + return nil + } + w, err := q.GetWantByReleaseGroup(ctx, *row.AlbumMbid) + if errors.Is(err, pgx.ErrNoRows) { + // Removed from the wanted list while the request waited. Wanting it + // again keeps the approval meaningful; the admin who removed it can + // reject a request instead. + _, err = wanted.WantWith(ctx, q, wantFor(row)) + return err + } + if err != nil { + return err + } + if !w.FulfilledAt.Valid || !w.AlbumID.Valid { + return nil + } + params := dbq.CompleteRequestParams{ID: row.ID, MatchedAlbumID: w.AlbumID} + if row.Kind == dbq.RequestKindTrack { + var trackID pgtype.UUID + err := r.pool.QueryRow(ctx, ` + SELECT id FROM tracks + WHERE album_id = $1 AND missing_since IS NULL + ORDER BY (mbid IS NOT DISTINCT FROM $2) DESC, added_at DESC, id + LIMIT 1`, w.AlbumID, row.TrackMbid).Scan(&trackID) + if err != nil && !errors.Is(err, pgx.ErrNoRows) { + return err + } + params.MatchedTrackID = trackID + } + completed, err := q.CompleteRequest(ctx, params) + if err != nil { + return err + } + r.publishCompleted(ctx, completed) + return nil +} diff --git a/internal/requests/requests_test.go b/internal/requests/requests_test.go new file mode 100644 index 00000000..5b1b9004 --- /dev/null +++ b/internal/requests/requests_test.go @@ -0,0 +1,310 @@ +package requests + +import ( + "context" + "encoding/json" + "errors" + "io" + "log/slog" + "os" + "testing" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" + "git.fabledsword.com/bvandeusen/minstrel/internal/notifications" +) + +func newPool(t *testing.T) *pgxpool.Pool { + t.Helper() + if testing.Short() { + t.Skip("skipping integration test in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(context.Background(), dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + dbtest.ResetDB(t, pool) + return pool +} + +func seedUser(t *testing.T, pool *pgxpool.Pool, name string) pgtype.UUID { + t.Helper() + u, err := dbq.New(pool).CreateUser(context.Background(), dbq.CreateUserParams{ + Username: dbtest.TestUserPrefix + name, PasswordHash: "x", ApiTokenHash: "x-" + name, + }) + if err != nil { + t.Fatalf("seed user: %v", err) + } + return u.ID +} + +func seedAlbum(t *testing.T, q *dbq.Queries, artistName, artistMBID, title, group string) (dbq.Artist, dbq.Album) { + t.Helper() + ctx := context.Background() + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: artistName, SortName: artistName, Mbid: &artistMBID}) + if err != nil { + t.Fatal(err) + } + release := group + "-release" + album, err := q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: title, SortTitle: title, ArtistID: artist.ID, Mbid: &release}) + if err != nil { + t.Fatal(err) + } + if err := q.SetAlbumReleaseGroupMbidIfNull(ctx, dbq.SetAlbumReleaseGroupMbidIfNullParams{ID: album.ID, ReleaseGroupMbid: &group}); err != nil { + t.Fatal(err) + } + return artist, album +} + +func seedTrack(t *testing.T, q *dbq.Queries, album dbq.Album, artist dbq.Artist, title, path string) dbq.Track { + t.Helper() + tr, err := q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ + Title: title, AlbumID: album.ID, ArtistID: artist.ID, + DurationMs: 180000, FilePath: path, FileSize: 1024, FileFormat: "flac", + }) + if err != nil { + t.Fatal(err) + } + return tr +} + +func quietLogger() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) } + +func TestCreate_ValidatesEachKind(t *testing.T) { + pool := newPool(t) + user := seedUser(t, pool, "creator") + svc := NewService(pool) + ctx := context.Background() + + if r, err := svc.Create(ctx, user, CreateParams{Kind: "artist", ArtistMBID: "a-mbid", ArtistName: "X"}); err != nil || + r.Status != dbq.RequestStatusPending { + t.Fatalf("artist: %+v, %v", r, err) + } + for name, p := range map[string]CreateParams{ + "album without its group": {Kind: "album", ArtistMBID: "a", ArtistName: "X"}, + "track without its album": {Kind: "track", ArtistMBID: "a", ArtistName: "X", TrackMBID: "t", TrackTitle: "Y"}, + "unknown kind": {Kind: "playlist", ArtistMBID: "a", ArtistName: "X"}, + } { + if _, err := svc.Create(ctx, user, p); !errors.Is(err, ErrInvalidKindFields) { + t.Errorf("%s: err = %v, want ErrInvalidKindFields", name, err) + } + } + again, created, err := svc.CreateTracked(ctx, user, CreateParams{Kind: "artist", ArtistMBID: "a-mbid", ArtistName: "X"}) + if err != nil || created || again.Kind != dbq.RequestKindArtist { + t.Errorf("a repeat request dedupes: created=%v, %v", created, err) + } +} + +func TestApprove_AlbumAndTrackRequestsWantTheirReleaseGroup(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + user := seedUser(t, pool, "wanter") + admin := seedUser(t, pool, "approver") + svc := NewService(pool) + ctx := context.Background() + + album, _ := svc.Create(ctx, user, CreateParams{ + Kind: "album", ArtistMBID: "artist-1", ArtistName: "Gorillaz", AlbumMBID: "group-demon-days", AlbumTitle: "Demon Days", + }) + track, _ := svc.Create(ctx, user, CreateParams{ + Kind: "track", ArtistMBID: "artist-1", ArtistName: "Gorillaz", AlbumMBID: "group-humanz", AlbumTitle: "Humanz", + TrackMBID: "rec-1", TrackTitle: "Andromeda", + }) + for _, r := range []dbq.Request{album, track} { + got, err := svc.Approve(ctx, r.ID, admin, pgtype.UUID{}) + if err != nil || got.Status != dbq.RequestStatusApproved || got.DecidedBy != admin { + t.Fatalf("approve %s: %+v, %v", r.Kind, got, err) + } + } + for group, title := range map[string]string{"group-demon-days": "Demon Days", "group-humanz": "Humanz"} { + w, err := q.GetWantByReleaseGroup(ctx, group) + if err != nil { + t.Fatalf("want %s: %v", group, err) + } + if w.Title != title || w.Reason != "request" || w.CreatedBy != user || w.FulfilledAt.Valid { + t.Errorf("want %s = %+v: the requester's open want", group, w) + } + } + if _, err := svc.Approve(ctx, album.ID, admin, pgtype.UUID{}); !errors.Is(err, ErrNotPending) { + t.Errorf("approving twice: err = %v, want ErrNotPending", err) + } + if _, err := svc.Approve(ctx, pgtype.UUID{Bytes: [16]byte{1}, Valid: true}, admin, pgtype.UUID{}); !errors.Is(err, ErrNotFound) { + t.Errorf("no such request: err = %v", err) + } +} + +func TestApprove_ArtistRequestFollowsAndWantsWhatExists(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + user := seedUser(t, pool, "follower") + ctx := context.Background() + svc := NewService(pool) + + albumsOnly, err := q.CreateReleaseTypeProfile(ctx, dbq.CreateReleaseTypeProfileParams{ + Name: "Albums only", PrimaryTypes: []string{"Album"}, SecondaryTypes: []string{}, + }) + if err != nil { + t.Fatal(err) + } + r, _ := svc.Create(ctx, user, CreateParams{Kind: "artist", ArtistMBID: "artist-2", ArtistName: "Moe Shop"}) + if _, err := svc.Approve(ctx, r.ID, user, albumsOnly.ID); err != nil { + t.Fatal(err) + } + f, err := q.GetFollow(ctx, "artist-2") + if err != nil { + t.Fatal(err) + } + if f.ProfileID != albumsOnly.ID || !f.WantExisting || f.BaselineAt.Valid { + t.Errorf("follow = %+v: on the chosen profile, wanting what exists, baseline still to read", f) + } + + r2, _ := svc.Create(ctx, user, CreateParams{Kind: "artist", ArtistMBID: "artist-3", ArtistName: "Kero Kero Bonito"}) + if _, err := svc.Approve(ctx, r2.ID, user, pgtype.UUID{}); err != nil { + t.Fatal(err) + } + if f, _ := q.GetFollow(ctx, "artist-3"); f.ProfileName != "Standard" { + t.Errorf("no profile chosen: on %q, want the default", f.ProfileName) + } +} + +func TestRejectAndCancel(t *testing.T) { + pool := newPool(t) + user := seedUser(t, pool, "rejected") + svc := NewService(pool) + ctx := context.Background() + + r, _ := svc.Create(ctx, user, CreateParams{Kind: "artist", ArtistMBID: "a-1", ArtistName: "X"}) + if got, err := svc.Reject(ctx, r.ID, user, "low quality"); err != nil || got.Status != dbq.RequestStatusRejected { + t.Fatalf("reject: %+v, %v", got, err) + } + if _, err := svc.Reject(ctx, r.ID, user, "again"); !errors.Is(err, ErrNotPending) { + t.Errorf("reject twice: err = %v", err) + } + c, _ := svc.Create(ctx, user, CreateParams{Kind: "artist", ArtistMBID: "a-2", ArtistName: "Y"}) + if _, err := svc.Cancel(ctx, c.ID, user); err != nil { + t.Fatal(err) + } + if _, err := svc.Cancel(ctx, c.ID, user); !errors.Is(err, ErrNotPending) { + t.Errorf("cancel twice: err = %v", err) + } +} + +func TestReconciler_CompletesWhenTheWantIsHeld(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + user := seedUser(t, pool, "reconciled") + svc := NewService(pool) + ctx := context.Background() + + req, _ := svc.Create(ctx, user, CreateParams{ + Kind: "track", ArtistMBID: "artist-4", ArtistName: "Notify Artist", AlbumMBID: "group-notify", AlbumTitle: "Notify Album", + TrackMBID: "rec-wanted", TrackTitle: "Wanted Track", + }) + if _, err := svc.Approve(ctx, req.ID, user, pgtype.UUID{}); err != nil { + t.Fatal(err) + } + rec := NewReconciler(pool, quietLogger(), nil) + rec.SetNotifier(notifications.New(pool, nil, nil)) + if err := rec.tickOnce(ctx); err != nil { + t.Fatal(err) + } + if got, _ := q.GetRequestByID(ctx, req.ID); got.Status != dbq.RequestStatusApproved { + t.Fatalf("status %v before the want is held, want approved", got.Status) + } + + artist, album := seedAlbum(t, q, "Notify Artist", "artist-4", "Notify Album", "group-notify") + _ = seedTrack(t, q, album, artist, "Other Track", "/music/notify/01.flac") + wantedTrack := seedTrack(t, q, album, artist, "Wanted Track", "/music/notify/02.flac") + if _, err := pool.Exec(ctx, "UPDATE tracks SET mbid = 'rec-wanted', mbid_source = 'tag', added_at = added_at - interval '1 hour' WHERE id = $1", wantedTrack.ID); err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, "UPDATE wants SET fulfilled_at = now(), album_id = $1 WHERE release_group_mbid = 'group-notify'", album.ID); err != nil { + t.Fatal(err) + } + for range 2 { // the second tick finds nothing left to complete + if err := rec.tickOnce(ctx); err != nil { + t.Fatal(err) + } + } + got, _ := q.GetRequestByID(ctx, req.ID) + if got.Status != dbq.RequestStatusCompleted || got.MatchedAlbumID != album.ID || got.MatchedTrackID != wantedTrack.ID { + t.Errorf("request = %+v: completed, matched to the album and the track it named", got) + } + inbox, err := q.ListNotifications(ctx, dbq.ListNotificationsParams{UserID: user, PageLimit: 10}) + if err != nil { + t.Fatal(err) + } + if len(inbox) != 1 || inbox[0].Kind != string(notifications.KindRequestCompleted) { + t.Fatalf("inbox = %+v, want one request_completed", inbox) + } + var p notifications.Payload + if err := json.Unmarshal(inbox[0].Payload, &p); err != nil { + t.Fatal(err) + } + if p.Name != "Notify Artist – Wanted Track" || p.AlbumID != formatUUIDForBus(album.ID) { + t.Errorf("payload = %+v", p) + } +} + +func TestReconciler_ArtistRequestCompletesOnceTheArtistIsHere(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + user := seedUser(t, pool, "artistreq") + svc := NewService(pool) + ctx := context.Background() + + req, _ := svc.Create(ctx, user, CreateParams{Kind: "artist", ArtistMBID: "artist-5", ArtistName: "Arrives"}) + if _, err := svc.Approve(ctx, req.ID, user, pgtype.UUID{}); err != nil { + t.Fatal(err) + } + rec := NewReconciler(pool, quietLogger(), nil) + if err := rec.tickOnce(ctx); err != nil { + t.Fatal(err) + } + if got, _ := q.GetRequestByID(ctx, req.ID); got.Status != dbq.RequestStatusApproved { + t.Fatalf("status %v with nothing here", got.Status) + } + artist, _ := seedAlbum(t, q, "Arrives", "artist-5", "First", "group-first") + if err := rec.tickOnce(ctx); err != nil { + t.Fatal(err) + } + if got, _ := q.GetRequestByID(ctx, req.ID); got.Status != dbq.RequestStatusCompleted || got.MatchedArtistID != artist.ID { + t.Errorf("request = %+v", got) + } +} + +func TestReconciler_RewantsWhatWasRemovedWhileTheRequestWaited(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + user := seedUser(t, pool, "rewant") + svc := NewService(pool) + ctx := context.Background() + + req, _ := svc.Create(ctx, user, CreateParams{ + Kind: "album", ArtistMBID: "artist-6", ArtistName: "Gone", AlbumMBID: "group-gone", AlbumTitle: "Gone", + }) + if _, err := svc.Approve(ctx, req.ID, user, pgtype.UUID{}); err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, "DELETE FROM wants"); err != nil { + t.Fatal(err) + } + if err := NewReconciler(pool, quietLogger(), nil).tickOnce(ctx); err != nil { + t.Fatal(err) + } + if _, err := q.GetWantByReleaseGroup(ctx, "group-gone"); err != nil { + t.Errorf("an approved request's want should be recorded again: %v", err) + } +} diff --git a/internal/requests/service.go b/internal/requests/service.go new file mode 100644 index 00000000..203a7f53 --- /dev/null +++ b/internal/requests/service.go @@ -0,0 +1,263 @@ +// Package requests owns the lifecycle of what users ask to have added. The +// synchronous Service handles Create/List/Approve/Reject/Cancel; approving +// records the request on the wanted list: an album or a track +// wants its album's release group, an artist is followed. The async +// Reconciler (separate file) completes approved requests once the library +// holds what they asked for. +package requests + +import ( + "context" + "errors" + "fmt" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/apierror" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/wanted" +) + +// Public errors. Handlers map these to API error codes. +var ( + ErrInvalidKindFields = errors.New("requests: missing required fields for kind") + ErrNotPending = errors.New("requests: request is not pending") + // ErrNotFound aliases apierror.ErrNotFound so handlers can errors.Is + // against the shared sentinel. + ErrNotFound = apierror.ErrNotFound +) + +// CreateParams is the input for a new request from a user. AlbumMBID is the +// album's MusicBrainz release group. +type CreateParams struct { + Kind string // "artist", "album", or "track" + ArtistMBID string + AlbumMBID string // required for kind=album/track + TrackMBID string // required for kind=track + ArtistName string + AlbumTitle string // required for kind=album/track + TrackTitle string // required for kind=track +} + +// Service owns the request lifecycle. +type Service struct { + pool *pgxpool.Pool +} + +// NewService constructs a Service. +func NewService(pool *pgxpool.Pool) *Service { + return &Service{pool: pool} +} + +// Create validates the kind→required-fields invariant and inserts a +// pending row. +func (s *Service) Create(ctx context.Context, userID pgtype.UUID, p CreateParams) (dbq.Request, error) { + row, _, err := s.CreateTracked(ctx, userID, p) + return row, err +} + +// CreateTracked is Create, also reporting whether a row was inserted. A +// request that dedupes into one already in flight is not new, so callers +// that announce new requests say nothing for it. +func (s *Service) CreateTracked(ctx context.Context, userID pgtype.UUID, p CreateParams) (dbq.Request, bool, error) { + if err := validateKindFields(p); err != nil { + return dbq.Request{}, false, err + } + q := dbq.New(s.pool) + // Dedup: if a non-terminal request for this MBID already exists, + // return it instead of inserting a duplicate. The Discover/search UI + // already hides requested candidates; this backstops races and direct + // API callers. Match on the new request's own-kind MBID. + dedupMBID := p.ArtistMBID + switch p.Kind { + case "album": + dedupMBID = p.AlbumMBID + case "track": + dedupMBID = p.TrackMBID + } + if existing, derr := q.GetNonTerminalRequestForMBID(ctx, dedupMBID); derr == nil { + return existing, false, nil + } else if !errors.Is(derr, pgx.ErrNoRows) { + return dbq.Request{}, false, fmt.Errorf("create: dedup check: %w", derr) + } + row, err := q.CreateRequest(ctx, dbq.CreateRequestParams{ + UserID: userID, + Kind: dbq.RequestKind(p.Kind), + ArtistMbid: p.ArtistMBID, + AlbumMbid: strPtr(p.AlbumMBID), + TrackMbid: strPtr(p.TrackMBID), + ArtistName: p.ArtistName, + AlbumTitle: strPtr(p.AlbumTitle), + TrackTitle: strPtr(p.TrackTitle), + }) + if err != nil { + return dbq.Request{}, false, fmt.Errorf("create: %w", err) + } + return row, true, nil +} + +// DisplayName is a request as a person would say it: "Moe Shop" for an +// artist, "Moe Shop – WWW" for an album or a track. +func DisplayName(row dbq.Request) string { + if t := requestedTitle(row); t != "" { + return row.ArtistName + " – " + t + } + return row.ArtistName +} + +// requestedTitle is the album or track a request names, or "" for an +// artist request. DisplayName is the artist and this, joined. +func requestedTitle(row dbq.Request) string { + switch { + case row.Kind == dbq.RequestKindTrack && row.TrackTitle != nil: + return *row.TrackTitle + case row.Kind == dbq.RequestKindAlbum && row.AlbumTitle != nil: + return *row.AlbumTitle + } + return "" +} + +func validateKindFields(p CreateParams) error { + if p.ArtistMBID == "" || p.ArtistName == "" { + return fmt.Errorf("%w: artist_mbid and artist_name are always required", ErrInvalidKindFields) + } + switch p.Kind { + case "artist": + // fine + case "album": + if p.AlbumMBID == "" || p.AlbumTitle == "" { + return fmt.Errorf("%w: album kind requires album_mbid and album_title", ErrInvalidKindFields) + } + case "track": + if p.AlbumMBID == "" || p.AlbumTitle == "" { + return fmt.Errorf("%w: track kind requires album_mbid and album_title (its album is what is wanted)", ErrInvalidKindFields) + } + if p.TrackMBID == "" || p.TrackTitle == "" { + return fmt.Errorf("%w: track kind requires track_mbid and track_title", ErrInvalidKindFields) + } + default: + return fmt.Errorf("%w: unknown kind %q", ErrInvalidKindFields, p.Kind) + } + return nil +} + +// ListByStatus returns requests with the given status, ordered by requested_at DESC. +func (s *Service) ListByStatus(ctx context.Context, status string, limit int32) ([]dbq.Request, error) { + return dbq.New(s.pool).ListRequestsByStatus(ctx, dbq.ListRequestsByStatusParams{ + Status: dbq.RequestStatus(status), Limit: limit, + }) +} + +// ListForUser returns all requests by a user, ordered by requested_at DESC. +func (s *Service) ListForUser(ctx context.Context, userID pgtype.UUID, limit int32) ([]dbq.Request, error) { + return dbq.New(s.pool).ListRequestsForUser(ctx, dbq.ListRequestsForUserParams{ + UserID: userID, Limit: limit, + }) +} + +// Approve puts a pending request on the wanted list and marks it approved, +// in one transaction. An album or track request wants the album's release +// group, recorded as the requester's. An artist request follows the artist +// on profileID (the zero id means the default profile) and also wants what +// they have already released that the profile passes: someone asking for an +// artist wants their music, not only what comes next. +func (s *Service) Approve(ctx context.Context, requestID, approverID, profileID pgtype.UUID) (dbq.Request, error) { + var approved dbq.Request + err := pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + row, err := q.GetRequestByID(ctx, requestID) + if errors.Is(err, pgx.ErrNoRows) { + return ErrNotFound + } + if err != nil { + return fmt.Errorf("approve: get: %w", err) + } + if row.Status != dbq.RequestStatusPending { + return ErrNotPending + } + switch row.Kind { + case dbq.RequestKindArtist: + if _, err := wanted.FollowWith(ctx, q, wanted.FollowParams{ + ArtistMBID: row.ArtistMbid, ArtistName: row.ArtistName, + ProfileID: profileID, By: approverID, WantExisting: true, + }); err != nil { + return err + } + default: + if _, err := wanted.WantWith(ctx, q, wantFor(row)); err != nil { + return err + } + } + approved, err = q.ApproveRequest(ctx, dbq.ApproveRequestParams{ID: requestID, DecidedBy: approverID}) + if err != nil { + return fmt.Errorf("approve: persist: %w", err) + } + return nil + }) + return approved, err +} + +// wantFor is the want an approved album or track request records. +func wantFor(row dbq.Request) wanted.WantParams { + p := wanted.WantParams{ + ArtistMBID: row.ArtistMbid, ArtistName: row.ArtistName, + Reason: wanted.ReasonRequest, CreatedBy: row.UserID, + } + if row.AlbumMbid != nil { + p.GroupMBID = *row.AlbumMbid + } + if row.AlbumTitle != nil { + p.Title = *row.AlbumTitle + } + return p +} + +// Reject transitions a pending request to rejected, recording notes and +// who decided. Returns ErrNotPending if the request is not pending, +// ErrNotFound if no such request exists. +func (s *Service) Reject(ctx context.Context, requestID pgtype.UUID, adminID pgtype.UUID, notes string) (dbq.Request, error) { + row, err := dbq.New(s.pool).RejectRequest(ctx, dbq.RejectRequestParams{ + ID: requestID, Notes: strPtr(notes), DecidedBy: adminID, + }) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + // Either not found OR not pending — check which. + cur, gerr := dbq.New(s.pool).GetRequestByID(ctx, requestID) + if gerr != nil { + return dbq.Request{}, ErrNotFound + } + if cur.Status != dbq.RequestStatusPending { + return dbq.Request{}, ErrNotPending + } + return dbq.Request{}, ErrNotFound + } + return dbq.Request{}, fmt.Errorf("reject: %w", err) + } + return row, nil +} + +// Cancel lets a user withdraw their own pending request. The SQL WHERE +// clause enforces both ownership (user_id = $2) and pending status +// (status = 'pending'). A zero-rows result always means ErrNotPending +// from the caller's perspective. +func (s *Service) Cancel(ctx context.Context, requestID pgtype.UUID, userID pgtype.UUID) (dbq.Request, error) { + row, err := dbq.New(s.pool).CancelRequest(ctx, dbq.CancelRequestParams{ + ID: requestID, DecidedBy: userID, + }) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + return dbq.Request{}, ErrNotPending + } + return dbq.Request{}, fmt.Errorf("cancel: %w", err) + } + return row, nil +} + +func strPtr(s string) *string { + if s == "" { + return nil + } + return &s +} diff --git a/internal/scrobble/threshold.go b/internal/scrobble/threshold.go index a33ecd7e..1b564057 100644 --- a/internal/scrobble/threshold.go +++ b/internal/scrobble/threshold.go @@ -7,9 +7,8 @@ package scrobble // recommended scrobble threshold: ≥240 seconds played OR ≥50% of the // track length played. // -// The two thresholds are deliberately separate from M2's skip-detection -// thresholds (which serve a different purpose — internal engine signal -// for the scoring formula's skip penalty). +// The two thresholds are deliberately separate from the skip-detection +// thresholds, which feed the scoring formula's skip penalty instead. func Qualifies(durationPlayedMs int, completionRatio float64) bool { return durationPlayedMs >= 240_000 || completionRatio >= 0.5 } diff --git a/internal/scrobble/worker.go b/internal/scrobble/worker.go index 3a436158..05c2ad27 100644 --- a/internal/scrobble/worker.go +++ b/internal/scrobble/worker.go @@ -14,7 +14,7 @@ import ( ) // backoffSchedule maps the failure count (1-indexed) to the delay before -// the next attempt. Per spec §M4a: 1m → 5m → 30m → 2h → 6h, then give up. +// the next attempt: 1m → 5m → 30m → 2h → 6h, then give up. var backoffSchedule = []time.Duration{ 1 * time.Minute, 5 * time.Minute, diff --git a/internal/server/hygiene.go b/internal/server/hygiene.go index 649f392f..854e85d3 100644 --- a/internal/server/hygiene.go +++ b/internal/server/hygiene.go @@ -120,8 +120,8 @@ func requireJSONForCookieWrites(next http.Handler) http.Handler { // by the SPA handler, which knows the inline-script hashes. // // HSTS goes out only when the request reached us over HTTPS as the trusted -// proxy reports it (rule 94): sending it over plain HTTP is ignored by -// browsers at best, and the app never forces HTTPS. +// proxy reports it: sent over plain HTTP it is ignored by browsers at best, +// and the app never forces HTTPS. func securityHeaders(hops func() int) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { diff --git a/internal/server/hygiene_test.go b/internal/server/hygiene_test.go index ae9a1c4f..2c31dbf6 100644 --- a/internal/server/hygiene_test.go +++ b/internal/server/hygiene_test.go @@ -115,7 +115,7 @@ func TestSecurityHeaders(t *testing.T) { t.Errorf("Permissions-Policy = %q", base.Get("Permissions-Policy")) } - // Rule 94's three cases for HSTS. + // HSTS with no trusted proxy, and through one saying https or http. if got := serve(0, "https").Get("Strict-Transport-Security"); got != "" { t.Errorf("hops 0 + forwarded https: HSTS = %q, want none", got) } diff --git a/internal/server/release_gate_test.go b/internal/server/release_gate_test.go index 345d84de..c62b1a17 100644 --- a/internal/server/release_gate_test.go +++ b/internal/server/release_gate_test.go @@ -10,7 +10,7 @@ import ( "gopkg.in/yaml.v3" ) -// The publish gate (rule 177, M462 #4984), pinned. release.yml holds every +// The publish gate, pinned. release.yml holds every // verifying lane and every publishing job in one graph so that a publish can // depend on the lanes; these tests keep that dependency from eroding. // diff --git a/internal/server/release_version_test.go b/internal/server/release_version_test.go index fa508d71..ac95415d 100644 --- a/internal/server/release_version_test.go +++ b/internal/server/release_version_test.go @@ -10,17 +10,10 @@ import ( "testing" ) -// Guards the version derivation that stamps every build. -// -// These assertions used to be impossible to run. The derivation lived inline -// in release.yml, which triggers only on `main` and on tags — so a mistake in -// it could not surface until a release was already under way, and its failure -// mode is silence: a version nobody can compare looks exactly like being up to -// date, and nobody reports an update they were never offered. -// -// Moving it to ci/version.sh made it executable, so this runs on every push -// that touches the release machinery. That is the whole point of the file; the -// specific assertions below matter less than the fact that they run at all. +// Guards the version derivation that stamps every build, in ci/version.sh. +// Its failure mode is silence: a version nobody can compare looks exactly +// like being up to date, and nobody reports an update they were never +// offered. Keeping it in a script lets these tests run it on every push. // // The tests EXECUTE the script rather than asserting on its text, so they // break when the behaviour changes rather than when the wording does. @@ -188,10 +181,9 @@ var jobBoundary = regexp.MustCompile(`\n\n [^ \n]`) // // The naive cut — "up to the next `- name:`" — silently returns an EMPTY body // for the last step in a job, and every assertion over it then passes -// vacuously. That is the failure rule 167 names: a check that reads as -// coverage while asserting on nothing. Cutting at a blank line followed by -// job-level indentation handles the last-step case, which is exactly where -// `Build and push` sits. +// vacuously: a check that reads as coverage while asserting on nothing. Cutting +// at a blank line followed by job-level indentation handles the last-step case, +// which is exactly where `Build and push` sits. func stepBody(t *testing.T, yaml, name string) string { t.Helper() i := strings.Index(yaml, "- name: "+name) @@ -274,7 +266,7 @@ func TestDevChannel_PublishesDevAloneAndNeverLatest(t *testing.T) { if strings.Contains(arm, ":latest") { t.Errorf("dev arm moves :latest — that ships dev code to every stable operator\n%s", arm) } - // Rule 145: a rolling channel gets no commit-addressable tag. + // A rolling channel gets no commit-addressable tag. if strings.Contains(arm, "GITHUB_SHA") { t.Errorf("dev arm publishes a per-commit tag; a rolling channel should not\n%s", arm) } @@ -367,21 +359,19 @@ func TestMainChannel_PublishesLatestAndTheRollbackUnit(t *testing.T) { if !strings.Contains(arm, "${IMAGE}:${GITHUB_SHA}") { t.Errorf("main arm publishes no commit-addressable image; there is no rollback target for production commits\n%s", arm) } - // Rule 147: :latest tracks main's tip, and a second name for the same + // :latest tracks main's tip, and a second name for the same // image sends readers looking for a distinction that does not exist. if strings.Contains(arm, "${IMAGE}:main") { t.Errorf("main arm publishes :main — rule 147 says that tag should not exist\n%s", arm) } } -// A release refreshes the CHANNEL and mints nothing else (rules 145 + 146). +// A release refreshes the CHANNEL and mints nothing else. // -// The specific regression: re-adding : here. The tag build rebuilds the -// SAME SOURCE as main's build minutes earlier, differing only in which APK is -// baked in — so a : minted here would overwrite main's immutable rollback -// target with different contents, under the same name. That is the exact thing -// rule 145's immutability clause exists to prevent, and it is the half with -// the incidents behind it. +// The regression to guard: a : tag here. The tag build rebuilds the SAME +// SOURCE as main's build minutes earlier, differing only in which APK is baked +// in — so a : minted here would overwrite main's immutable rollback +// target with different contents, under the same name. func TestReleaseBuild_RefreshesTheChannelAndMintsNothingElse(t *testing.T) { arm, _, _ := imageTagArms(t, releaseYAML(t)) @@ -393,8 +383,8 @@ func TestReleaseBuild_RefreshesTheChannelAndMintsNothingElse(t *testing.T) { } } -// No version-numbered image tags anywhere, on any arm (rule 145, and the -// operator's 2026-09-10 decision to drop them across every project). +// No version-numbered image tags anywhere, on any arm: the build's version is +// self-reported, and the tag map is :latest, : and :dev. // // Asserted across the whole step rather than per-arm because the mistake this // catches is re-adding one ANYWHERE, and the tag arm is only the likeliest @@ -441,9 +431,9 @@ func TestVerifyJob_ChecksTheRollbackImageNotAVersionTag(t *testing.T) { } } -// The server's self-reported version is now the ONLY thing that identifies a +// The server's self-reported version is the ONLY thing that identifies a // build, so a lane that stamps a channel word instead of a version silently -// removes that ability. It used to stamp the literal "main"/"dev". +// removes that ability. func TestImageBuild_StampsADerivedVersionAndAChannel(t *testing.T) { yaml := releaseYAML(t) @@ -465,7 +455,7 @@ func TestImageBuild_StampsADerivedVersionAndAChannel(t *testing.T) { t.Errorf("the main arm stamps the literal string \"main\" as a version; two images months apart would be indistinguishable\n%s", mainArm) } if !strings.Contains(yaml, "ci/version.sh HEAD | sed") { - t.Error("the image job no longer derives its version from ci/version.sh — the version and the APK's version can now drift apart") + t.Error("the image job no longer derives its version from ci/version.sh — the image's version has drifted out of test coverage") } } @@ -518,10 +508,11 @@ func commitFile(t *testing.T, dir, path, epoch string) { } // versionIn runs ci/version.sh inside dir, reading real git rather than the -// pinned-clock override, so the PATHSPEC is what is under test. -func versionIn(t *testing.T, dir string) (string, error) { +// pinned-clock override, so the PATHSPEC is what is under test. flags go +// before the ref, as in `--apk`. +func versionIn(t *testing.T, dir string, flags ...string) (string, error) { t.Helper() - cmd := exec.Command(filepath.Join(repoRoot(t), "ci", "version.sh"), "HEAD") + cmd := exec.Command(filepath.Join(repoRoot(t), "ci", "version.sh"), append(flags, "HEAD")...) cmd.Dir = dir cmd.Env = append(os.Environ(), "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null", @@ -568,11 +559,9 @@ func TestVersionName_IgnoresCommitsThatShipNothing(t *testing.T) { } } -// android/ is deliberately NOT excluded, and that is the subtle half of the -// list. It ships in no server image — but it is the APK's entire source, and -// ONE script derives the version for both artifacts. Excluding it would stop -// an Android-only commit from moving the APK's own version, which is exactly -// the silent downgrade the versioning rework exists to prevent. +// android/ is deliberately NOT excluded from the image's name: the image +// bundles the APK under client/, so an Android-only commit changes what the +// image carries, and an image that changed must not keep its old name. func TestVersionName_AndroidSourcesCount(t *testing.T) { dir := gitRepo(t) commitFile(t, dir, "internal/server/thing.go", "1757443736") @@ -583,7 +572,71 @@ func TestVersionName_AndroidSourcesCount(t *testing.T) { t.Fatalf("version.sh failed: %v\n%s", err, out) } if !strings.Contains(out, "name=2026.09.10.0042") { - t.Errorf("an Android commit did not move the version; the APK would ship new code under its old version name\n%s", out) + t.Errorf("an Android commit did not move the image's version; it carries a new APK under its old name\n%s", out) + } +} + +// The app offers an update whenever the server's APK has a different name +// from its own, so the APK's name must move for Android commits and ONLY for +// them. A name taken from the whole repo would offer every phone a rebuild of +// the app it already has on each server-only deploy. +func TestApkName_MovesOnlyForAndroidCommits(t *testing.T) { + const ( + app = "1757443736" // 2025.09.09.1848 + server = "1789000920" // 2026.09.10.0042, later + again = "1789600000" // 2026.09.16.2306, later still + ) + dir := gitRepo(t) + commitFile(t, dir, "android/app/src/main/Thing.kt", app) + commitFile(t, dir, "internal/server/thing.go", server) + commitFile(t, dir, "web/src/routes/page.svelte", server) + + out, err := versionIn(t, dir, "--apk") + if err != nil { + t.Fatalf("version.sh --apk failed: %v\n%s", err, out) + } + if !strings.Contains(out, "name=2025.09.09.1848") { + t.Errorf("a server-only commit moved the APK's name; every phone would be offered the app it already has\n%s", out) + } + // The image's name still follows the server commit, so the test above + // is not passing because the flag is ignored and everything is stale. + out, err = versionIn(t, dir) + if err != nil { + t.Fatalf("version.sh failed: %v\n%s", err, out) + } + if !strings.Contains(out, "name=2026.09.10.0042") { + t.Errorf("without --apk the name should follow the server commit\n%s", out) + } + + // A test-only Android commit ships nothing either. + commitFile(t, dir, "android/app/src/test/java/A.kt", again) + out, err = versionIn(t, dir, "--apk") + if err != nil { + t.Fatalf("version.sh --apk failed: %v\n%s", err, out) + } + if !strings.Contains(out, "name=2025.09.09.1848") { + t.Errorf("an Android test-only commit moved the APK's name\n%s", out) + } + + // ...and a real Android commit must move it, or new app code would ship + // under the name of the old and never be offered. + commitFile(t, dir, "android/app/src/main/Thing.kt", again) + out, err = versionIn(t, dir, "--apk") + if err != nil { + t.Fatalf("version.sh --apk failed: %v\n%s", err, out) + } + if !strings.Contains(out, "name=2026.09.16.2306") { + t.Errorf("an Android commit did not move the APK's name; the new app would never be offered\n%s", out) + } +} + +// The APK lane must ask for the APK's name. Without the flag it would still +// build, sign and publish, and quietly go back to offering an update on every +// server deploy. +func TestReleaseWorkflow_NamesTheApkFromAndroidAlone(t *testing.T) { + step := stepBody(t, releaseYAML(t), "Compute release version") + if !strings.Contains(step, "ci/version.sh --apk HEAD") { + t.Errorf("the APK lane does not derive its version with --apk\n%s", step) } } diff --git a/internal/server/requestlog.go b/internal/server/requestlog.go index fe597149..b4587626 100644 --- a/internal/server/requestlog.go +++ b/internal/server/requestlog.go @@ -19,18 +19,16 @@ import ( // hits it every 5s and would otherwise drown real signals (~17k lines/day). // // Severity is keyed off the response status so 4xx/5xx surface even when -// the operator's logger level is set above Info. +// the logger level is set above Info. // -// The `remote` attribute holds the address resolved through the operator's -// configured reverse-proxy depth, NOT the raw socket peer (#2453). Behind a -// proxy — the normal deployment for anything public — the socket peer is the -// proxy, so every line would have carried the same useless address, and the -// access log would have disagreed with the Active-sessions surface about who -// connected. The attribute key is unchanged so existing log greps keep -// working; only its accuracy improved. +// The `remote` attribute holds the address resolved through the configured +// reverse-proxy depth, NOT the raw socket peer. Behind a proxy — the normal +// deployment for anything public — the socket peer is the proxy, so every +// line would carry the same useless address, and the access log would +// disagree with the Active-sessions list about who connected. // // trustedHops is a func because this middleware is constructed at boot while -// the value is operator-editable at runtime, and — since Router() registers +// the value is editable at runtime, and — since Router() registers // this before it builds the settings service — because it lets the accessor // be wired before the thing it reads exists. auth.ClientIP tolerates a depth // of 0, which is what a nil service reports. diff --git a/internal/server/requestlog_test.go b/internal/server/requestlog_test.go index b5b552c5..29af63b8 100644 --- a/internal/server/requestlog_test.go +++ b/internal/server/requestlog_test.go @@ -109,10 +109,10 @@ func TestRequestLog_AttributesPresent(t *testing.T) { } } -// The point of routing #2453 through the access log: behind a proxy, `remote` -// must be the client rather than the proxy, and must agree with what the -// Active-sessions surface records for the same request. Logs and UI -// disagreeing about who connected is worse than either being wrong alone. +// Behind a proxy, the access log's `remote` must be the client rather than the +// proxy, and must agree with what the Active-sessions surface records for the +// same request. Logs and UI disagreeing about who connected is worse than +// either being wrong alone. func TestRequestLog_RemoteHonoursTrustedProxyDepth(t *testing.T) { cases := []struct { name string @@ -136,8 +136,8 @@ func TestRequestLog_RemoteHonoursTrustedProxyDepth(t *testing.T) { want: "203.0.113.200", }, { - // The case that motivated the change: proxy on a PUBLIC address, - // which the pre-#2453 heuristic logged as the proxy forever. + // A proxy on a PUBLIC address: a "trust XFF only from private + // peers" heuristic would log the proxy forever. name: "depth 1 through a public-addressed proxy logs the client", hops: func() int { return 1 }, remoteAddr: "203.0.113.200:40000", diff --git a/internal/server/server.go b/internal/server/server.go index 978234bb..47b045ce 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -3,7 +3,6 @@ package server import ( "context" "encoding/json" - "errors" "log/slog" "net/http" "strings" @@ -16,58 +15,25 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/api" "git.fabledsword.com/bvandeusen/minstrel/internal/auth" + "git.fabledsword.com/bvandeusen/minstrel/internal/buildinfo" "git.fabledsword.com/bvandeusen/minstrel/internal/config" "git.fabledsword.com/bvandeusen/minstrel/internal/coverart" "git.fabledsword.com/bvandeusen/minstrel/internal/eventbus" "git.fabledsword.com/bvandeusen/minstrel/internal/library" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrconfig" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrquarantine" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarrrequests" "git.fabledsword.com/bvandeusen/minstrel/internal/mailer" "git.fabledsword.com/bvandeusen/minstrel/internal/netsettings" "git.fabledsword.com/bvandeusen/minstrel/internal/playevents" "git.fabledsword.com/bvandeusen/minstrel/internal/playlists" + "git.fabledsword.com/bvandeusen/minstrel/internal/quarantine" "git.fabledsword.com/bvandeusen/minstrel/internal/reacquisition" "git.fabledsword.com/bvandeusen/minstrel/internal/recsettings" + "git.fabledsword.com/bvandeusen/minstrel/internal/requests" "git.fabledsword.com/bvandeusen/minstrel/internal/subsonic" "git.fabledsword.com/bvandeusen/minstrel/internal/tags" "git.fabledsword.com/bvandeusen/minstrel/internal/tracks" "git.fabledsword.com/bvandeusen/minstrel/web" ) -// lidarrUnmonitorAdapter wires the per-call lidarrClientFn factory pattern -// (used elsewhere in this package so admin Lidarr config edits take effect -// without restart) into the tracks.LidarrUnmonitorer interface that -// internal/tracks expects. When the factory returns nil (Lidarr disabled) -// we surface a sentinel error — tracks.Service treats UnmonitorTrack -// failures as non-fatal, so this collapses to a `lidarr_unmonitor_failed` -// flag in the response and the destructive part still runs. -type lidarrUnmonitorAdapter struct { - fn func() *lidarr.Client -} - -var errLidarrDisabled = errors.New("lidarr disabled") - -func (a lidarrUnmonitorAdapter) UnmonitorTrack(ctx context.Context, trackMbid, albumMbid string) error { - c := a.fn() - if c == nil { - return errLidarrDisabled - } - return c.UnmonitorTrack(ctx, trackMbid, albumMbid) -} - -// ListUnmappedTrackFiles lets the duplicate merge refuse to remove a copy -// Lidarr maps (M498). tracks.ErrLidarrDisabled tells it there is no Lidarr to -// protect. -func (a lidarrUnmonitorAdapter) ListUnmappedTrackFiles(ctx context.Context) ([]lidarr.TrackFile, error) { - c := a.fn() - if c == nil { - return nil, tracks.ErrLidarrDisabled - } - return c.ListUnmappedTrackFiles(ctx) -} - // ScanTrigger is the subset of the scanner the HTTP handler needs. Kept as an // interface so tests can stub it without touching the DB. The progressCb // parameter (added in m7-scan-progress) lets the orchestrator drive partial- @@ -95,38 +61,38 @@ type Server struct { LibraryScanner *library.Scanner ScanCfg library.RunScanConfig // Bus is the live-event bus shared with background workers (the - // lidarr reconciler, scan workers) constructed in cmd/minstrel/main.go. + // request reconciler, scan workers) constructed in cmd/minstrel/main.go. // When nil, Router() constructs a local fallback (test contexts). Bus *eventbus.Bus // PlaylistScheduler fires per-user daily system-playlist builds at - // 03:00 in each user's stored timezone (#392 Half B). Constructed + // 03:00 in each user's stored timezone. Constructed // in cmd/minstrel/main.go and threaded into the API handlers so // PUT /api/me/timezone and POST /api/auth/register can call // Refresh synchronously. PlaylistScheduler *playlists.Scheduler - // RecSettings is the DB-backed recommendation tuning lab (#1250): + // RecSettings is the DB-backed recommendation tuning: // scoring-weight profiles + taste-build knobs. Constructed in // cmd/minstrel/main.go (it pushes daily-mix weights into package // playlists at boot); the API layer reads radio weights per request // and serves the admin tuning endpoints from it. Router() constructs // a fallback when nil (tests). RecSettings *recsettings.Service - // FingerprintSettings is the DB-backed fingerprinting policy (M400 #3913). + // FingerprintSettings is the DB-backed fingerprinting policy. // Constructed in cmd/minstrel/main.go and shared with the scanner and the // fingerprint workers, so a save from the admin card reaches them without a // restart. Router() constructs a fallback when nil (tests). FingerprintSettings *library.FingerprintSettingsService - // LoudnessSettings is the DB-backed loudness analysis policy (M464 #4995), + // LoudnessSettings is the DB-backed loudness analysis policy, // shared with the loudness backfill. Nil makes the router load its own. LoudnessSettings *library.LoudnessSettingsService - // AcoustIDLookup is the AcoustID lookup worker (M401), whose settings, + // AcoustIDLookup is the AcoustID lookup worker, whose settings, // status and "look up now" the admin card drives. Nil in test contexts; // one is built (and never run) so the admin routes still answer. AcoustIDLookup *library.AcoustIDLookupWorker - // ReacqSettings is the DB-backed missing-file re-acquisition policy - // (milestone #290) — the same instance the sweeper in cmd/minstrel/main.go - // reads, so a save from the admin card reaches it without a restart - // (#3936). Router() constructs a fallback when nil (tests). + // ReacqSettings is the DB-backed missing-file re-acquisition policy — the + // same instance the sweeper in cmd/minstrel/main.go reads, so a save from + // the admin card reaches it without a restart. Router() constructs a + // fallback when nil (tests). ReacqSettings *reacquisition.SettingsService // StreamSecret is the HMAC key used by /api/cast/stream-token to // mint signed UPnP / Sonos stream URLs and by /api/tracks/{id}/stream @@ -170,19 +136,7 @@ func (s *Server) Router() http.Handler { s.EventsCfg.SkipMaxCompletionRatio, s.EventsCfg.SkipMaxDurationPlayedMs, ) - lidarrCfg := lidarrconfig.New(s.Pool) - // Per-call client factory: re-reads config so an admin save in - // /admin/integrations takes effect immediately without restart. - // Returns nil when Lidarr is disabled, which the Service-layer - // methods translate to ErrLidarrDisabled. - lidarrClientFn := func() *lidarr.Client { - cfg, err := lidarrCfg.Get(context.Background()) - if err != nil || !cfg.Enabled || cfg.BaseURL == "" || cfg.APIKey == "" { - return nil - } - return lidarr.NewClient(cfg.BaseURL, cfg.APIKey) - } - lidarrReqs := lidarrrequests.NewService(s.Pool, lidarrCfg, lidarrClientFn, nil) + requestSvc := requests.NewService(s.Pool) reacqSettings := s.ReacqSettings if reacqSettings == nil { // Test contexts construct Server without main.go's boot wiring. @@ -196,17 +150,15 @@ func (s *Server) Router() http.Handler { s.Logger.Warn("reacquisition settings unavailable; serving defaults", "err", raErr) } } - lidarrQuar := lidarrquarantine.NewService(s.Pool, lidarrCfg, lidarrClientFn, s.DataDir) - tracksSvc := tracks.NewService(s.Pool, s.Logger, lidarrUnmonitorAdapter{fn: lidarrClientFn}, s.DataDir) + quarantineSvc := quarantine.NewService(s.Pool, s.DataDir) + tracksSvc := tracks.NewService(s.Pool, s.Logger, s.DataDir) playlistsSvc := playlists.NewService(s.Pool, s.Logger, s.DataDir) smtpSender := mailer.NewSMTPSender(s.Pool, s.Logger.With("component", "mailer")) - // Live-event bus for SSE subscribers (#392). Constructed per-process; - // producers in playevents / lidarrrequests / scanner publish into - // the same instance. Background workers (lidarr reconciler, scan - // scheduler) live in cmd/minstrel/main.go where they're constructed - // before Router() runs; they read s.Bus directly so they share this - // process's bus. When Router() runs before main set the bus (tests, - // or future contexts) we fall back to a fresh local instance. + // Live-event bus for SSE subscribers, one per process: producers in + // playevents / requests / scanner publish into the same instance. + // Background workers are built in cmd/minstrel/main.go before Router() + // runs and share s.Bus. With no bus set (tests), a fresh local one is + // used. bus := s.Bus if bus == nil { bus = eventbus.New() @@ -248,7 +200,7 @@ func (s *Server) Router() http.Handler { } acoustIDLookup = library.NewAcoustIDLookupWorker(s.Pool, s.Logger, acoustIDSettings, nil) } - api.Mount(r, s.Pool, s.Logger, writer, s.RecommendationCfg, recSettings, lidarrCfg, lidarrReqs, lidarrQuar, tracksSvc, playlistsSvc, s.CoverEnricher, s.CoverSettings, s.TagSettings, s.LibraryScanner, s.ScanCfg, s.DataDir, smtpSender, bus, s.PlaylistScheduler, s.StreamSecret, netSettings, reacqSettings, fpSettings, loudSettings, acoustIDLookup) + api.Mount(r, s.Pool, s.Logger, writer, s.RecommendationCfg, recSettings, requestSvc, quarantineSvc, tracksSvc, playlistsSvc, s.CoverEnricher, s.CoverSettings, s.TagSettings, s.LibraryScanner, s.ScanCfg, s.DataDir, smtpSender, bus, s.PlaylistScheduler, s.StreamSecret, netSettings, reacqSettings, fpSettings, loudSettings, acoustIDLookup) // /api/admin/scan is the only admin route owned by the server package // (it needs the Scanner). Register it as a single inline-middleware // route — using r.Route("/api/admin", ...) here would create a second @@ -279,15 +231,15 @@ func (s *Server) handleHealthz(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) _ = json.NewEncoder(w).Encode(map[string]string{ "status": "ok", - "version": ServerVersion, - "channel": ServerChannel, + "version": buildinfo.Version, + "channel": buildinfo.Channel, "min_client_version": MinClientVersion, }) } // handleAdminScan runs a scan synchronously and returns the resulting stats. -// Kept synchronous for v1: libraries are small and the operator can tell when -// it's done. Async jobs with status polling are a later-milestone concern. +// Synchronous, so the caller can tell when it's done; the scan's progress is +// also published as scan-run events. func (s *Server) handleAdminScan(w http.ResponseWriter, r *http.Request) { stats, err := s.Scanner.Scan(r.Context(), nil) w.Header().Set("Content-Type", "application/json") diff --git a/internal/server/server_test.go b/internal/server/server_test.go index bc037214..41900cad 100644 --- a/internal/server/server_test.go +++ b/internal/server/server_test.go @@ -150,13 +150,12 @@ func TestRouter_RestPathNotSwallowedBySPA(t *testing.T) { } } -// TestRouter_AdminSubtreeNotShadowed is a regression test for a real bug we -// shipped in 06a1fe1 and didn't catch until M6a: r.Route("/api/admin", ...) -// in server.go was creating a second chi subtree at the same prefix as the -// nested admin Route inside api.Mount. chi.Walk shows both subtrees as -// registered, but runtime routing dispatch only matches one — every admin -// endpoint registered by api.Mount (Lidarr config, quarantine, etc.) -// silently 404'd for authenticated callers in production. +// TestRouter_AdminSubtreeNotShadowed guards the trap of a second +// r.Route("/api/admin", ...) in server.go: it creates a chi subtree at the +// same prefix as the nested admin Route inside api.Mount. chi.Walk shows both +// subtrees as registered, but runtime routing dispatch only matches one, so +// every admin endpoint registered by api.Mount (summary, quarantine, etc.) +// would silently 404 for authenticated callers. // // chi.Walk enumeration would NOT have caught this (both branches are in // the tree). The only reliable check is to make a real authenticated @@ -212,18 +211,17 @@ func TestRouter_AdminSubtreeNotShadowed(t *testing.T) { ts := httptest.NewServer(s.Router()) defer ts.Close() - // Each path is registered by a different package: lidarr/config from + // Each path is registered by a different package: admin/summary from // api.Mount and admin/scan from server.Router. If chi runtime dispatch - // only routes one /api/admin subtree, one of these returns 404 — that - // IS the bug we shipped in 06a1fe1. + // only routes one /api/admin subtree, one of these returns 404. cases := []struct { method string path string owner string }{ - {http.MethodGet, "/api/admin/lidarr/config", "api.Mount"}, + {http.MethodGet, "/api/admin/summary", "api.Mount"}, // /api/admin/scan would actually trigger a real library scan via - // stubScanner; we don't include it. The lidarr/config path is the + // stubScanner; we don't include it. The admin/summary path is the // canonical regression check. } for _, tc := range cases { @@ -244,13 +242,11 @@ func TestRouter_AdminSubtreeNotShadowed(t *testing.T) { } } -// TestRouter_ReacquisitionSettingsSavedThroughTheAPIReachTheSweeper is a -// regression test for #3936. Router() used to construct a -// reacquisition.SettingsService of its own, so a save from the admin card -// refreshed THAT instance's cache while the sweeper in cmd/minstrel/main.go -// kept serving what it had loaded at boot. The card showed the new policy, the -// feature kept running the old one, and only a restart reconciled them — the -// exact thing rule 25 says a setting must not need. +// TestRouter_ReacquisitionSettingsSavedThroughTheAPIReachTheSweeper: a +// Router() that built a reacquisition.SettingsService of its own would refresh +// THAT instance's cache on a save, while the sweeper in cmd/minstrel/main.go +// kept serving what it loaded at boot. The card would show the new policy and +// the feature run the old one until a restart. // // The assertion is made against the instance main.go hands the sweeper: save // through the router, then read that instance. A second service leaves it stale. @@ -316,13 +312,9 @@ func TestRouter_ReacquisitionSettingsSavedThroughTheAPIReachTheSweeper(t *testin defer ts.Close() body, err := json.Marshal(map[string]any{ - "enabled": before.Enabled, - "grace_hours": wantGrace, - "backoff_base_hours": before.BackoffBaseHours, - "backoff_max_hours": before.BackoffMaxHours, - "max_attempts": before.MaxAttempts, - "max_per_pass": before.MaxPerPass, - "auto_approve": before.AutoApprove, + "enabled": before.Enabled, + "grace_hours": wantGrace, + "max_per_pass": before.MaxPerPass, }) if err != nil { t.Fatalf("marshal: %v", err) diff --git a/internal/server/version.go b/internal/server/version.go index efc2399e..2a65d8b6 100644 --- a/internal/server/version.go +++ b/internal/server/version.go @@ -5,27 +5,5 @@ package server // older clients see version_too_old at /healthz and refuse to operate. const MinClientVersion = "0.1.0" -// ServerVersion is the build's own version name — YYYY.MM.DD.HHMM, derived -// from the commit it was built from by ci/version.sh. Defaults to "dev" for -// local builds; overridden at link time via: -// -// -ldflags="-X 'git.fabledsword.com/bvandeusen/minstrel/internal/server.ServerVersion=2026.09.10.1449'" -// -// release.yml passes it through the MINSTREL_VERSION build-arg → Dockerfile -// ldflag. Surfaced at /healthz so operators can verify which image their -// container is running without exec'ing into it. -// -// This carried the literal strings "main" and "dev" until 2026-09-10, which -// made every image on a channel report the same thing forever. It stopped -// being cosmetic when :vYYYY.MM.DD.HHMM image tags were retired (family rule -// 145): this is now the ONLY thing that says which build is running. -var ServerVersion = "dev" - -// ServerChannel is which line this build came off — "stable" or "dev", or -// "local" for a plain `docker build`. -// -// A SIBLING FIELD, never a suffix inside ServerVersion (family rule 149). The -// same commit built on both lanes reports the same version and differs only -// here; folding the two together is what makes a version string stop being -// comparable. -var ServerChannel = "local" +// The build's version and channel live in internal/buildinfo, shared with +// the Subsonic envelope and the admin Overview. diff --git a/internal/similarity/worker.go b/internal/similarity/worker.go index c2d5078a..461a4c79 100644 --- a/internal/similarity/worker.go +++ b/internal/similarity/worker.go @@ -5,14 +5,14 @@ // // For recordings it keeps LB's whole answer in listenbrainz_similar_recordings // and derives the track_similarity edges from it in SQL, against whatever is -// in the library at the time (#5296). Matching once at fetch time missed every -// recording that reached the library afterwards, through Lidarr or an MBID the -// AcoustID lookup filled in. For artists it stores the in-library matches in -// artist_similarity and the rest in artist_similarity_unmatched, as before. +// in the library at the time, so a recording that reaches the library later +// (an import, or an MBID the AcoustID lookup fills in) gains its edges without +// asking LB again. For artists it stores the in-library matches in +// artist_similarity and the rest in artist_similarity_unmatched. // // Every answer is recorded as a fetch, an empty one included, and the queue -// reads those records: freshness used to be read from the edges, so a seed -// that matched nothing was never fresh and held the head of the queue. +// reads those records rather than the edges: a seed that matched nothing is +// still fresh and does not hold the head of the queue. package similarity import ( @@ -85,7 +85,7 @@ func (w *Worker) tickOnce(ctx context.Context) error { return err } // Links recordings that reached the library since their seed was fetched: - // a scan, a Lidarr import, an MBID from the AcoustID lookup. + // a scan, an import, an MBID from the AcoustID lookup. if err := q.ResolveListenBrainzTrackEdges(ctx, nil); err != nil { return fmt.Errorf("resolve similarity edges: %w", err) } @@ -199,6 +199,10 @@ func (w *Worker) upsertArtistSimilar(ctx context.Context, q *dbq.Queries, artist return } sort.Slice(results, func(i, j int) bool { return results[i].Score > results[j].Score }) + // Scores are stored relative to the seed's closest artist, as track edges + // are: LB's raw scores are unbounded counts, and both the candidate pool + // and the suggestions read 0 to 1. + best := results[0].Score mbids := make([]string, 0, len(results)) for _, r := range results { @@ -230,7 +234,7 @@ func (w *Worker) upsertArtistSimilar(ctx context.Context, q *dbq.Queries, artist continue } if uerr := q.UpsertArtistSimilarity(ctx, dbq.UpsertArtistSimilarityParams{ - ArtistAID: artistAID, ArtistBID: localID, Score: r.Score, + ArtistAID: artistAID, ArtistBID: localID, Score: relativeScore(r.Score, best), }); uerr != nil { w.logger.Warn("similarity: UpsertArtistSimilarity", "err", uerr) continue @@ -257,7 +261,7 @@ func (w *Worker) upsertArtistSimilar(ctx context.Context, q *dbq.Queries, artist SeedArtistID: artistAID, CandidateMbid: r.MBID, CandidateName: r.Name, - Score: r.Score, + Score: relativeScore(r.Score, best), Source: "listenbrainz", }); uerr != nil { w.logger.Warn("similarity: UpsertArtistSimilarityUnmatched", "err", uerr) @@ -266,3 +270,12 @@ func (w *Worker) upsertArtistSimilar(ctx context.Context, q *dbq.Queries, artist takenUnmatched++ } } + +// relativeScore is score over the seed's best, in [0, 1]; 0 when the seed has +// no positive score to measure against. +func relativeScore(score, best float64) float64 { + if best <= 0 { + return 0 + } + return score / best +} diff --git a/internal/similarity/worker_integration_test.go b/internal/similarity/worker_integration_test.go index 940bff5c..dea811b2 100644 --- a/internal/similarity/worker_integration_test.go +++ b/internal/similarity/worker_integration_test.go @@ -209,8 +209,40 @@ func TestTickOnce_MapsLBResponseToLocalLibrary(t *testing.T) { } } -// The 20-per-seed cap is gone with the cache (#5296): LB's answer is at most -// 50, and every recording of it that is in the library becomes an edge. +// LB's scores are counts, not 0 to 1. An edge is stored over the seed's best +// score in the whole answer, in the library or not, so the scorer reads it as +// 0 to 1. +func TestTickOnce_EdgeIsRelativeToTheSeedsBest(t *testing.T) { + f := newFixture(t) + mbidA := "11111111-1111-1111-1111-111111111111" + mbidB := "22222222-2222-2222-2222-222222222222" + mbidC := "99999999-9999-9999-9999-999999999999" // NOT in library + trackA := seedTrack(t, f, "A", &mbidA) + trackB := seedTrack(t, f, "B", &mbidB) + markPlayed(t, f, trackA.ID) + body := `[ + {"recording_mbid": "` + mbidC + `", "score": 1445}, + {"recording_mbid": "` + mbidB + `", "score": 289} + ]` + srv := stubLB(body, `[]`, http.StatusOK) + defer srv.Close() + w := newTestWorker(f, srv.URL) + if err := w.tickOnce(context.Background()); err != nil { + t.Fatalf("tickOnce: %v", err) + } + var score float64 + if err := f.pool.QueryRow(context.Background(), + `SELECT score FROM track_similarity WHERE track_a_id = $1 AND track_b_id = $2 AND source = 'listenbrainz'`, + trackA.ID, trackB.ID).Scan(&score); err != nil { + t.Fatalf("edge: %v", err) + } + if score != 0.2 { + t.Errorf("score = %v, want 0.2 (289 over the seed's best, 1445)", score) + } +} + +// LB's answer is at most 50, and every recording of it that is in the library +// becomes an edge: no per-seed cap. func TestTickOnce_KeepsEveryInLibraryMatch(t *testing.T) { f := newFixture(t) mbidSeed := "11111111-1111-1111-1111-111111111111" @@ -279,8 +311,9 @@ func TestTickOnce_RefreshesStaleRow(t *testing.T) { _ = f.pool.QueryRow(context.Background(), `SELECT score, fetched_at FROM track_similarity WHERE track_a_id = $1 AND track_b_id = $2 AND source = 'listenbrainz'`, trackA.ID, other.ID).Scan(&score, &fetchedAt) - if score != 0.95 { - t.Errorf("score = %v, want 0.95 (refreshed)", score) + // The seed's only answer is its best, so the refreshed edge is 1.0. + if score != 1.0 { + t.Errorf("score = %v, want 1.0 (refreshed, relative to the seed's best)", score) } if time.Since(fetchedAt) > time.Minute { t.Errorf("fetched_at not bumped: %v ago", time.Since(fetchedAt)) @@ -419,11 +452,12 @@ func TestUpsertArtistSimilar_PersistsUnmatchedToTable(t *testing.T) { w := newTestWorker(f, "") similars := []listenbrainz.SimilarArtist{ - {MBID: inLibMBID, Name: "InLib Artist", Score: 0.95}, - {MBID: "out-mbid-1", Name: "Outsider One", Score: 0.85}, - {MBID: "out-mbid-2", Name: "Outsider Two", Score: 0.80}, - {MBID: "out-mbid-3", Name: "Outsider Three", Score: 0.70}, - {MBID: "out-mbid-4", Name: "", Score: 0.60}, // empty name — skipped + // LB's scores are counts; rows are stored over the seed's best. + {MBID: inLibMBID, Name: "InLib Artist", Score: 1000}, + {MBID: "out-mbid-1", Name: "Outsider One", Score: 850}, + {MBID: "out-mbid-2", Name: "Outsider Two", Score: 800}, + {MBID: "out-mbid-3", Name: "Outsider Three", Score: 700}, + {MBID: "out-mbid-4", Name: "", Score: 600}, // empty name — skipped } w.upsertArtistSimilar(ctx, f.q, f.artist.ID, similars) @@ -438,6 +472,16 @@ func TestUpsertArtistSimilar_PersistsUnmatchedToTable(t *testing.T) { if matchedCount != 1 { t.Errorf("artist_similarity rows = %d, want 1 (only the in-library match)", matchedCount) } + var matchedScore float64 + if err := f.pool.QueryRow(ctx, + "SELECT score FROM artist_similarity WHERE artist_a_id = $1 AND artist_b_id = $2", + f.artist.ID, inLibArtist.ID, + ).Scan(&matchedScore); err != nil { + t.Fatalf("fetch matched score: %v", err) + } + if matchedScore != 1.0 { + t.Errorf("matched score = %v, want 1.0 (the seed's best)", matchedScore) + } // Unmatched path: 3 rows (out-mbid-1/2/3); the empty-name row is skipped. var unmatchedCount int @@ -464,7 +508,6 @@ func TestUpsertArtistSimilar_PersistsUnmatchedToTable(t *testing.T) { t.Errorf("row = (%q, %v), want (Outsider One, 0.85)", name, score) } - _ = inLibArtist } // recordFetch stamps a seed as answered by ListenBrainz at the given SQL time. @@ -500,9 +543,9 @@ func recordingLB(answer func(mbid string) string) (*httptest.Server, func() []st } } -// The starvation #3879 measured: a seed whose answer matched nothing in the -// library wrote nothing, stayed due, and was asked again every tick ahead of -// everything behind it. Recording the fetch moves the queue on. +// A seed whose answer matches nothing in the library writes no edges, and +// must still leave the head of the queue: recording the fetch moves it on, +// so it is not asked again every tick ahead of everything behind it. func TestTickOnce_UnmatchedAnswerDoesNotHoldTheQueue(t *testing.T) { f := newFixture(t) mbids := []string{ @@ -540,7 +583,7 @@ func TestTickOnce_UnmatchedAnswerDoesNotHoldTheQueue(t *testing.T) { } } -// The point of the cache (#5296): a recording LB named while it was not in the +// The point of the cache: a recording LB named while it was not in the // library becomes an edge once it arrives, on the next tick's resolve, without // asking LB again. One edge per recording when the library holds two copies, // and none to a copy that is missing. diff --git a/internal/subsonic/albumlist_test.go b/internal/subsonic/albumlist_test.go new file mode 100644 index 00000000..da498be3 --- /dev/null +++ b/internal/subsonic/albumlist_test.go @@ -0,0 +1,178 @@ +package subsonic + +import ( + "context" + "encoding/json" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "os" + "slices" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" +) + +// playHistoryFixture is a library of single-track albums and a listener +// whose plays rank them differently by recency and by count. +type playHistoryFixture struct { + pool *pgxpool.Pool + user dbq.User + albums map[string]dbq.Album // keyed by title +} + +// seedPlayHistory builds the fixture: +// +// A: one completed play 1h ago +// B: three completed plays 5h ago +// D: five skipped plays 30m ago (recent counts it, frequent does not) +// C: played 10m ago, but its file is gone (in neither list) +// E: played many times by another user (in neither list) +func seedPlayHistory(t *testing.T) playHistoryFixture { + t.Helper() + if testing.Short() { + t.Skip("skipping in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(context.Background(), dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + dbtest.ResetDB(t, pool) + + ctx := context.Background() + q := dbq.New(pool) + user, err := q.CreateUser(ctx, dbq.CreateUserParams{ + Username: dbtest.TestUserPrefix + "listener", PasswordHash: "x", ApiTokenHash: "x", + }) + if err != nil { + t.Fatalf("create user: %v", err) + } + other, err := q.CreateUser(ctx, dbq.CreateUserParams{ + Username: dbtest.TestUserPrefix + "other", PasswordHash: "x", ApiTokenHash: "y", + }) + if err != nil { + t.Fatalf("create other user: %v", err) + } + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "Artist", SortName: "Artist"}) + if err != nil { + t.Fatalf("artist: %v", err) + } + + f := playHistoryFixture{pool: pool, user: user, albums: map[string]dbq.Album{}} + tracks := map[string]dbq.Track{} + for _, title := range []string{"A", "B", "C", "D", "E"} { + al, err := q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: title, SortTitle: title, ArtistID: artist.ID}) + if err != nil { + t.Fatalf("album %s: %v", title, err) + } + tr, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: title, AlbumID: al.ID, ArtistID: artist.ID, + FilePath: "/tmp/albumlist-" + title + ".flac", DurationMs: 100_000, + }) + if err != nil { + t.Fatalf("track %s: %v", title, err) + } + f.albums[title] = al + tracks[title] = tr + } + if _, err := pool.Exec(ctx, `UPDATE tracks SET missing_since = now() WHERE id = $1`, tracks["C"].ID); err != nil { + t.Fatalf("mark missing: %v", err) + } + + now := time.Now() + play := func(u dbq.User, title string, ago time.Duration, skipped bool, times int) { + t.Helper() + at := now.Add(-ago) + var sessionID pgtype.UUID + if err := pool.QueryRow(ctx, + `INSERT INTO play_sessions (user_id, started_at, last_event_at, client_id) + VALUES ($1, $2, $2, 'albumlist-test') RETURNING id`, + u.ID, at).Scan(&sessionID); err != nil { + t.Fatalf("insert play_session: %v", err) + } + for i := 0; i < times; i++ { + if _, err := pool.Exec(ctx, + `INSERT INTO play_events (user_id, track_id, session_id, started_at, was_skipped) + VALUES ($1, $2, $3, $4, $5)`, + u.ID, tracks[title].ID, sessionID, at, skipped); err != nil { + t.Fatalf("insert play_event: %v", err) + } + } + } + play(user, "A", time.Hour, false, 1) + play(user, "B", 5*time.Hour, false, 3) + play(user, "D", 30*time.Minute, true, 5) + play(user, "C", 10*time.Minute, false, 4) + play(other, "E", time.Minute, false, 9) + return f +} + +// albumListTitles calls getAlbumList2 as the fixture's user and returns the +// album titles in the order served. +func (f playHistoryFixture) albumListTitles(t *testing.T, query string) []string { + t.Helper() + b := &browseHandlers{pool: f.pool} + req := httptest.NewRequest(http.MethodGet, "/rest/getAlbumList2?f=json&"+query, nil) + ctx := context.WithValue(req.Context(), userCtxKey, f.user) + w := httptest.NewRecorder() + b.getAlbumList2(w, req.WithContext(ctx)) + if w.Code != http.StatusOK { + t.Fatalf("status = %d body=%s", w.Code, w.Body.String()) + } + var raw struct { + Response struct { + Status string `json:"status"` + AlbumList2 AlbumList2Container `json:"albumList2"` + } `json:"subsonic-response"` + } + if err := json.Unmarshal(w.Body.Bytes(), &raw); err != nil { + t.Fatalf("decode: %v body=%s", err, w.Body.String()) + } + if raw.Response.Status != "ok" { + t.Fatalf("status %q body=%s", raw.Response.Status, w.Body.String()) + } + titles := make([]string, len(raw.Response.AlbumList2.Albums)) + for i, a := range raw.Response.AlbumList2.Albums { + titles[i] = a.Name + } + return titles +} + +func TestGetAlbumList2_RecentOrdersByLatestPlay(t *testing.T) { + f := seedPlayHistory(t) + got := f.albumListTitles(t, "type=recent") + if want := []string{"D", "A", "B"}; !slices.Equal(got, want) { + t.Errorf("recent = %v, want %v", got, want) + } +} + +func TestGetAlbumList2_FrequentOrdersByCompletedPlays(t *testing.T) { + f := seedPlayHistory(t) + got := f.albumListTitles(t, "type=frequent") + if want := []string{"B", "A"}; !slices.Equal(got, want) { + t.Errorf("frequent = %v, want %v", got, want) + } +} + +func TestGetAlbumList2_RecentPages(t *testing.T) { + f := seedPlayHistory(t) + got := f.albumListTitles(t, "type=recent&size=1&offset=1") + if want := []string{"A"}; !slices.Equal(got, want) { + t.Errorf("recent page = %v, want %v", got, want) + } +} diff --git a/internal/subsonic/browse.go b/internal/subsonic/browse.go index 2afb0ed8..99b18a14 100644 --- a/internal/subsonic/browse.go +++ b/internal/subsonic/browse.go @@ -26,8 +26,8 @@ type browseHandlers struct { // for backwards compatibility with clients that regroup client-side. const ignoredArticles = "The El La Los Las Le Les" -// getMusicFolders always returns a single folder. Minstrel has no multi-library -// concept in M1; clients use the folder id as an opaque cookie so we return 1. +// getMusicFolders always returns a single folder: Minstrel serves one library, +// and clients use the folder id as an opaque cookie, so it is 1. func (b *browseHandlers) getMusicFolders(w http.ResponseWriter, r *http.Request) { Write(w, r, MusicFoldersResponse{ Envelope: NewEnvelope("ok"), @@ -236,9 +236,9 @@ func (b *browseHandlers) getSong(w http.ResponseWriter, r *http.Request) { } // getAlbumList2 returns a paged album list filtered by type. Supported types -// match the Subsonic spec § getAlbumList2. "recent"/"frequent" require play -// history (M2) and currently return an empty list rather than an error so -// clients that poll them don't break. +// match the Subsonic spec § getAlbumList2. "recent" and "frequent" read the +// caller's own play history: most recently played first, and most played +// first. func (b *browseHandlers) getAlbumList2(w http.ResponseWriter, r *http.Request) { params := r.URL.Query() listType := params.Get("type") @@ -285,16 +285,27 @@ func (b *browseHandlers) getAlbumList2(w http.ResponseWriter, r *http.Request) { WriteFail(w, r, ErrMissingParameter, "Missing required parameter: genre") return } - // Genre is a plain string as of #367 — the query now splits - // tracks.genre on [;,] instead of comparing the whole column, so a - // client asking for "Rock" also reaches tracks tagged "Rock;Pop". - // Previously those were unreachable from either of their genres. + // The query splits tracks.genre on [;,] rather than comparing the + // whole column, so a client asking for "Rock" also reaches tracks + // tagged "Rock;Pop". albums, err = q.ListAlbumsByGenre(r.Context(), dbq.ListAlbumsByGenreParams{ Genre: genre, Lim: int32(size), Off: int32(offset), }) case "recent", "frequent": - // Play history lands in M2; return empty to keep clients happy. - albums = nil + user, ok := UserFromContext(r.Context()) + if !ok { + WriteFail(w, r, ErrNotAuthorized, "Not authenticated") + return + } + if listType == "recent" { + albums, err = q.ListAlbumsRecentlyPlayedByUser(r.Context(), dbq.ListAlbumsRecentlyPlayedByUserParams{ + UserID: user.ID, Lim: int32(size), Off: int32(offset), + }) + } else { + albums, err = q.ListAlbumsFrequentlyPlayedByUser(r.Context(), dbq.ListAlbumsFrequentlyPlayedByUserParams{ + UserID: user.ID, Lim: int32(size), Off: int32(offset), + }) + } default: WriteFail(w, r, ErrGeneric, "Invalid list type: "+listType) return diff --git a/internal/subsonic/envelope.go b/internal/subsonic/envelope.go index ffbf4255..4bcd20ed 100644 --- a/internal/subsonic/envelope.go +++ b/internal/subsonic/envelope.go @@ -7,6 +7,8 @@ import ( "io" "net/http" "strings" + + "git.fabledsword.com/bvandeusen/minstrel/internal/buildinfo" ) // APIVersion is the Subsonic REST version we advertise. 1.16.1 is the last @@ -18,9 +20,6 @@ const ( XMLNS = "http://subsonic.org/restapi" ) -// ServerVersion identifies this build. Overwritten by cmd/minstrel at startup. -var ServerVersion = "dev" - // Envelope holds the attributes present on every Subsonic response. Concrete // response types embed it and add endpoint-specific fields with json+xml tags. type Envelope struct { @@ -48,7 +47,7 @@ func NewEnvelope(status string) Envelope { Status: status, Version: APIVersion, Type: ServerType, - ServerVersion: ServerVersion, + ServerVersion: buildinfo.Version, OpenSubsonic: true, } } diff --git a/internal/subsonic/replaygain_test.go b/internal/subsonic/replaygain_test.go index 774093c5..bbc08e81 100644 --- a/internal/subsonic/replaygain_test.go +++ b/internal/subsonic/replaygain_test.go @@ -11,7 +11,7 @@ import ( ) // OpenSubsonic's replayGain is an object in JSON and an element with -// attributes in XML. Clients parse both, so both are pinned (#4997). +// attributes in XML. Clients parse both, so both are pinned. func TestSongRef_ReplayGainEncodesForBothFormats(t *testing.T) { gain, peak := float32(-4.5), float32(0.8913) tr := dbq.Track{Title: "Song", FilePath: "/m/a.flac", FileFormat: "flac"} diff --git a/internal/subsonic/star.go b/internal/subsonic/star.go index 2a9fcc12..0c85d806 100644 --- a/internal/subsonic/star.go +++ b/internal/subsonic/star.go @@ -40,7 +40,7 @@ func (m *mediaHandlers) handleStar(w http.ResponseWriter, r *http.Request) { return } if trackID.Valid { - // Stars the song: every copy of it (M498 #5438). + // Stars the song: every copy of it. liked, err := q.LikeTrack(r.Context(), dbq.LikeTrackParams{UserID: user.ID, TrackID: trackID}) if err != nil { WriteFail(w, r, ErrGeneric, "Could not star track") @@ -182,7 +182,7 @@ func (m *mediaHandlers) handleGetStarred2(w http.ResponseWriter, r *http.Request // loadStarred fetches the user's full starred set in liked_at DESC order, // projects to Subsonic refs (Song / Album / Artist). func loadStarred(ctx context.Context, q *dbq.Queries, userID pgtype.UUID) ([]SongRef, []AlbumRef, []ArtistRef, error) { - const cap = 500 // bounded for safety; M2 doesn't paginate getStarred + const cap = 500 // bounded for safety; getStarred is not paginated songs := []SongRef{} albums := []AlbumRef{} artists := []ArtistRef{} diff --git a/internal/subsonic/subsonic.go b/internal/subsonic/subsonic.go index 3e5e6d95..9b1cd8bb 100644 --- a/internal/subsonic/subsonic.go +++ b/internal/subsonic/subsonic.go @@ -13,6 +13,7 @@ import ( "github.com/jackc/pgx/v5/pgxpool" "git.fabledsword.com/bvandeusen/minstrel/internal/auth" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" "git.fabledsword.com/bvandeusen/minstrel/internal/playevents" ) @@ -26,6 +27,7 @@ import ( func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, cfg Config, events *playevents.Writer, hops func() int) { b := &browseHandlers{pool: pool} m := newMediaHandlers(pool, events, logger) + u := &userHandlers{lookup: dbq.New(pool).GetUserByUsername} r.Route("/rest", func(sub chi.Router) { sub.Use(Middleware(pool, cfg, auth.NewLoginGuard(), hops)) register(sub, "/ping", handlePing) @@ -46,7 +48,7 @@ func Mount(r chi.Router, pool *pgxpool.Pool, logger *slog.Logger, cfg Config, ev register(sub, "/unstar", m.handleUnstar) register(sub, "/getStarred", m.handleGetStarred) register(sub, "/getStarred2", m.handleGetStarred2) - register(sub, "/getUser", handleGetUser) + register(sub, "/getUser", u.getUser) // Subsonic clients expect every /rest/* miss to come back as a // Subsonic-shaped error envelope, not chi's plain-text 404. Without diff --git a/internal/subsonic/types.go b/internal/subsonic/types.go index 2414fe60..8775ff68 100644 --- a/internal/subsonic/types.go +++ b/internal/subsonic/types.go @@ -48,8 +48,8 @@ func parseUUID(s string) (pgtype.UUID, bool) { return u, u.Valid } -// MusicFolder matches Subsonic's /rest/getMusicFolders. v1 always reports a -// single folder because Minstrel has no multi-library concept yet. +// MusicFolder matches Subsonic's /rest/getMusicFolders. Minstrel always +// reports a single folder: it has no multi-library concept. type MusicFolder struct { XMLName xml.Name `json:"-" xml:"musicFolder"` ID int `json:"id" xml:"id,attr"` @@ -181,7 +181,7 @@ type SongRef struct { BitRate int `json:"bitRate,omitempty" xml:"bitRate,attr,omitempty"` IsDir bool `json:"isDir" xml:"isDir,attr"` Type string `json:"type" xml:"type,attr"` - // ReplayGain is OpenSubsonic's replayGain (M464 #4997): Minstrel's own + // ReplayGain is OpenSubsonic's replayGain: Minstrel's own // measurements, so third-party clients that level by ReplayGain do it // from the same numbers the Minstrel apps use. Omitted until measured. ReplayGain *ReplayGain `json:"replayGain,omitempty" xml:"replayGain,omitempty"` diff --git a/internal/subsonic/user.go b/internal/subsonic/user.go index 2f41e6b6..68dc4104 100644 --- a/internal/subsonic/user.go +++ b/internal/subsonic/user.go @@ -1,9 +1,13 @@ package subsonic import ( + "context" "encoding/xml" + "errors" "net/http" + "github.com/jackc/pgx/v5" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) @@ -38,27 +42,41 @@ type User struct { VideoConversionRole bool `json:"videoConversionRole" xml:"videoConversionRole,attr"` } -// handleGetUser returns the authenticated user when no username is provided -// or when the requested username matches; admins may look up any user. Per -// spec a non-admin asking about another user gets ErrNotAuthorized. -func handleGetUser(w http.ResponseWriter, r *http.Request) { +// userHandlers serves getUser. lookup finds a user by exact username; it is +// a field so tests can answer without a database. +type userHandlers struct { + lookup func(ctx context.Context, username string) (dbq.User, error) +} + +// getUser returns the authenticated user when no username is provided or +// when the requested username matches; admins may look up any user, and an +// unknown username is ErrDataNotFound. Per spec a non-admin asking about +// another user gets ErrNotAuthorized. +func (h *userHandlers) getUser(w http.ResponseWriter, r *http.Request) { caller, ok := UserFromContext(r.Context()) if !ok { WriteFail(w, r, ErrNotAuthorized, "Not authenticated") return } requested := r.URL.Query().Get("username") - if requested != "" && requested != caller.Username && !caller.IsAdmin { + if requested == "" || requested == caller.Username { + Write(w, r, UserResponse{Envelope: NewEnvelope("ok"), User: userPayload(caller)}) + return + } + if !caller.IsAdmin { WriteFail(w, r, ErrNotAuthorized, "Not authorized to view other users") return } - // We don't have a per-user lookup wired up yet; until then admins asking - // for another username see the caller's roles. Acceptable while M1 has a - // single user; revisit once user management lands. - Write(w, r, UserResponse{ - Envelope: NewEnvelope("ok"), - User: userPayload(caller), - }) + target, err := h.lookup(r.Context(), requested) + if errors.Is(err, pgx.ErrNoRows) { + WriteFail(w, r, ErrDataNotFound, "User not found") + return + } + if err != nil { + WriteFail(w, r, ErrGeneric, "Failed to look up user") + return + } + Write(w, r, UserResponse{Envelope: NewEnvelope("ok"), User: userPayload(target)}) } func userPayload(u dbq.User) User { diff --git a/internal/subsonic/user_test.go b/internal/subsonic/user_test.go index 3bb3e8c5..c72e3c46 100644 --- a/internal/subsonic/user_test.go +++ b/internal/subsonic/user_test.go @@ -8,6 +8,8 @@ import ( "strings" "testing" + "github.com/jackc/pgx/v5" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" ) @@ -51,6 +53,41 @@ func TestHandleGetUser_NonAdminCannotLookupOthers(t *testing.T) { } } +func TestHandleGetUser_AdminLooksUpAnotherUser(t *testing.T) { + admin := dbq.User{Username: "admin", IsAdmin: true} + bob := dbq.User{Username: "bob", IsAdmin: false} + _, body := callGetUser(t, admin, "bob", bob) + if body.Resp.Status != "ok" { + t.Fatalf("status = %q, want ok (err=%+v)", body.Resp.Status, body.Resp.Error) + } + if body.Resp.User.Username != "bob" { + t.Errorf("username = %q, want bob: the answer must describe the user asked about, not the caller", body.Resp.User.Username) + } + if body.Resp.User.AdminRole || body.Resp.User.SettingsRole { + t.Errorf("bob is not an admin, but the answer carries admin/settings roles %v/%v", + body.Resp.User.AdminRole, body.Resp.User.SettingsRole) + } +} + +func TestHandleGetUser_AdminLooksUpUnknownUser(t *testing.T) { + admin := dbq.User{Username: "admin", IsAdmin: true} + _, body := callGetUser(t, admin, "nobody") + if body.Resp.Status != "failed" || body.Resp.Error == nil || + body.Resp.Error.Code != ErrDataNotFound { + t.Errorf("expected ErrDataNotFound, got status=%q err=%+v", + body.Resp.Status, body.Resp.Error) + } +} + +func TestHandleGetUser_OwnUsernameNeedsNoLookup(t *testing.T) { + // No known users: a lookup would fail, so success proves none was made. + user := dbq.User{Username: "alice", IsAdmin: false} + _, body := callGetUser(t, user, "alice") + if body.Resp.Status != "ok" || body.Resp.User.Username != "alice" { + t.Errorf("status=%q username=%q, want ok/alice", body.Resp.Status, body.Resp.User.Username) + } +} + type getUserPayload struct { Resp struct { Status string `json:"status"` @@ -59,8 +96,18 @@ type getUserPayload struct { } `json:"subsonic-response"` } -func callGetUser(t *testing.T, user dbq.User, requestedUsername string) (*httptest.ResponseRecorder, getUserPayload) { +// callGetUser runs getUser as caller, with the users in known as the only +// accounts a lookup can find. +func callGetUser(t *testing.T, user dbq.User, requestedUsername string, known ...dbq.User) (*httptest.ResponseRecorder, getUserPayload) { t.Helper() + h := &userHandlers{lookup: func(_ context.Context, username string) (dbq.User, error) { + for _, u := range known { + if u.Username == username { + return u, nil + } + } + return dbq.User{}, pgx.ErrNoRows + }} url := "/rest/getUser.view?f=json" if requestedUsername != "" { url += "&username=" + requestedUsername @@ -68,7 +115,7 @@ func callGetUser(t *testing.T, user dbq.User, requestedUsername string) (*httpte req := httptest.NewRequest("GET", url, nil) req = req.WithContext(context.WithValue(req.Context(), userCtxKey, user)) w := httptest.NewRecorder() - handleGetUser(w, req) + h.getUser(w, req) var body getUserPayload if err := json.NewDecoder(strings.NewReader(w.Body.String())).Decode(&body); err != nil { diff --git a/internal/sync/changes.go b/internal/sync/changes.go index a14b617d..ec03db41 100644 --- a/internal/sync/changes.go +++ b/internal/sync/changes.go @@ -38,9 +38,9 @@ func LogChange(ctx context.Context, dbtx dbq.DBTX, entityType EntityType, entity // LogChanges is LogChange for a set of entities of one type sharing one op. // -// Exists because the scan's reconcile pass can mark a quarter of a library -// missing in a single sweep (#2704), and a per-row loop there would be -// thousands of round-trips inside an operation that is already slow. Every +// Exists because the scan's reconcile pass can mark a large part of a library +// missing in a single sweep, and a per-row loop there would be thousands of +// round-trips inside an operation that is already slow. Every // other caller mutates one entity and should keep using LogChange. // // A no-op on an empty set, so callers don't have to guard. diff --git a/internal/sync/compactor.go b/internal/sync/compactor.go index e26943f1..93829b2b 100644 --- a/internal/sync/compactor.go +++ b/internal/sync/compactor.go @@ -1,15 +1,12 @@ -// Periodic compactor for the library_changes log (#357 deferred -// follow-up). Every mutation in the library writes one row, so the -// table grows unbounded without a retention policy. This worker -// trims rows older than CompactorRetention on a daily tick. +// Periodic compactor for the library_changes log. Every mutation in the +// library writes one row, so the table grows unbounded without a retention +// policy. This worker trims rows older than CompactorRetention on a daily +// tick. // -// Why daily / 30 days: Flutter clients track a per-app cursor and -// pull deltas from the last seen id. As long as a client checks in -// at least once per retention window, it never sees the cursor go -// stale. Clients that drop offline longer than that hit the /api/ -// library/sync 410 fallback and do a full resync. 30 days is the -// established norm for similar offline-cache delta logs and matches -// the spec at docs/superpowers/specs/2026-05-09-flutter-offline-mode-design.md. +// Why 30 days: clients keep a cursor and pull deltas from the last seen id. +// A client that checks in at least once per retention window never sees its +// cursor go stale; one offline longer than that gets the /api/library/sync +// 410 and does a full resync. package sync diff --git a/internal/sync/types.go b/internal/sync/types.go index a95baba5..1865caa4 100644 --- a/internal/sync/types.go +++ b/internal/sync/types.go @@ -1,4 +1,4 @@ -// Package sync supports the Flutter delta-sync endpoint by writing an +// Package sync supports the delta-sync endpoint by writing an // append-only library_changes log alongside every mutation on tracked // entities (artists, albums, tracks, likes, playlists, playlist_tracks). // diff --git a/internal/tags/enricher.go b/internal/tags/enricher.go index b41ac4cb..a17ef139 100644 --- a/internal/tags/enricher.go +++ b/internal/tags/enricher.go @@ -93,7 +93,7 @@ func (e *Enricher) EnrichTrack(ctx context.Context, trackID pgtype.UUID, ref Tra // tagFetch pairs a provider ID with a bound fetch call, so the merge-and- // classify loop below is shared between the track chain and the candidate- -// artist chain (#2376) instead of being written twice with one word changed. +// artist chain instead of being written twice with one word changed. type tagFetch struct { providerID string fetch func(context.Context) ([]Tag, error) @@ -241,7 +241,7 @@ func (e *Enricher) EnrichTrackBatch(ctx context.Context, limit int, } // EnrichCandidateArtist runs the artist-tag chain for one out-of-library -// Discover candidate and caches the merged result (#2376). +// Discover candidate and caches the merged result. // // Mirrors EnrichTrack, with one deliberate difference in the transient case: // there is no row to "leave NULL", because eligibility is the ABSENCE of a @@ -381,7 +381,7 @@ func (e *Enricher) logCandidateBatchSummary(eligible, processed, enriched, settl } // logBatchSummary emits one Info line with the category breakdown — the -// enriched/settled/left-null split is the operator's diagnostic for a +// enriched/settled/left-null split is the diagnostic for a // "0 enriched" symptom the collapsed tally can't explain. func (e *Enricher) logBatchSummary(eligible, processed, enriched, settledNone, leftNull, errored int) { e.logger.Info("tags: enrichment batch complete", diff --git a/internal/tags/enricher_chain_test.go b/internal/tags/enricher_chain_test.go index 23b73fb4..171dc9ab 100644 --- a/internal/tags/enricher_chain_test.go +++ b/internal/tags/enricher_chain_test.go @@ -9,7 +9,7 @@ import ( ) // runChain is the merge-and-classify loop shared by the track and -// candidate-artist drains (#2376). It touches no DB, so it is testable with a +// candidate-artist drains. It touches no DB, so it is testable with a // bare Enricher. // // The classification is the part that matters: "every source cleanly had diff --git a/internal/tags/httpclient.go b/internal/tags/httpclient.go index a65f1087..44f3b521 100644 --- a/internal/tags/httpclient.go +++ b/internal/tags/httpclient.go @@ -25,6 +25,10 @@ type httpClientOptions struct { UserAgent string // MinInterval throttles requests; 0 disables rate limiting. MinInterval time.Duration + // Pacer, when set, replaces MinInterval: requests wait on it instead. + // MusicBrainz's provider passes musicbrainz.Shared(), because its limit + // is per address and the release catalogue spends the same budget. + Pacer interface{ Wait(context.Context) error } // MaxRetries is the number of retry attempts on 429 / 5xx. MaxRetries int // BaseBackoff is the base unit for exponential backoff (2^attempt × base ± 20%). @@ -141,6 +145,9 @@ func (c *httpClient) handleStatus(ctx context.Context, resp *http.Response, atte // rateLimitWait blocks until MinInterval has elapsed since the last call, // then claims the slot. Disabled when MinInterval is 0. func (c *httpClient) rateLimitWait(ctx context.Context) error { + if c.opts.Pacer != nil { + return c.opts.Pacer.Wait(ctx) + } if c.opts.MinInterval <= 0 { return nil } diff --git a/internal/tags/provider.go b/internal/tags/provider.go index 900825f3..863d86aa 100644 --- a/internal/tags/provider.go +++ b/internal/tags/provider.go @@ -1,5 +1,5 @@ // Package tags enriches the taste profile's tag facet beyond raw ID3 -// genre (milestone #160, task #1490). It fetches folksonomy style/mood +// genre. It fetches folksonomy style/mood // tags for tracks from a chain of pluggable providers (MusicBrainz, // Last.fm, …), caches them in track_tags, and the taste recompute unions // them alongside tracks.genre. @@ -33,7 +33,7 @@ type Tag struct { // TrackRef is the lookup key passed to a provider. MBID is the track's // recording MBID (preferred by MBID-keyed providers); ArtistName + Title // are the always-populated fallback for name-based providers (Last.fm); -// ArtistMBID (nullable) enables MusicBrainz's artist-tag fallback (#1519) +// ArtistMBID (nullable) enables MusicBrainz's artist-tag fallback // when the recording itself is untagged or MBID-less. type TrackRef struct { MBID string @@ -55,7 +55,7 @@ type Provider interface { // RequiresAPIKey reports whether the provider needs a key to function. // The admin UI shows/hides the key field on this; it does NOT gate the - // enabled toggle (rule #26 — providers with usable defaults ship working). + // enabled toggle: a provider with usable defaults ships working. RequiresAPIKey() bool // DefaultEnabled is whether a freshly-installed Minstrel ships with @@ -63,7 +63,7 @@ type Provider interface { // (keyed) returns false — opt-in once a key is supplied. DefaultEnabled() bool - // Configure applies the operator's settings. Called at boot and on + // Configure applies the admin's settings. Called at boot and on // every admin PATCH; must be idempotent and goroutine-safe (the // enricher may read concurrently). Configure(ProviderSettings) error @@ -96,9 +96,8 @@ type ArtistRef struct { Name string } -// ArtistTagProvider is the artist-level tag capability, added for tag-space -// Discover (#2376). Separate from TrackTagProvider — exactly the split that -// interface's doc comment anticipated — so a source can implement either +// ArtistTagProvider is the artist-level tag capability, used by tag-space +// Discover. Separate from TrackTagProvider, so a source can implement either // without the other, and so the enricher can ask for the capability it needs // rather than checking at the call site. // diff --git a/internal/tags/provider_artist_tags_test.go b/internal/tags/provider_artist_tags_test.go index 489be798..8d99c401 100644 --- a/internal/tags/provider_artist_tags_test.go +++ b/internal/tags/provider_artist_tags_test.go @@ -8,7 +8,7 @@ import ( "testing" ) -// Artist-level tag fetching for out-of-library Discover candidates (#2376). +// Artist-level tag fetching for out-of-library Discover candidates. // Reuses the mbEntityServer / newMBProvider / newLastfmProvider helpers from // the per-provider test files. @@ -16,8 +16,8 @@ func TestMusicBrainzFetchArtistTags_UsesFullWeightNotTheFallbackDiscount(t *test // THE distinction worth a test. artistTagWeightFactor (0.6) exists because // FetchTrackTags uses artist tags as a *proxy* for a track's tags. Here the // artist IS the subject, so weights must land unscaled — otherwise these - // are not comparable with track_tags, which is the exact comparison slice 6 - // is built on. + // are not comparable with track_tags, which is the comparison tag-space + // Discover is built on. srv := mbEntityServer(``, `{"tags":[{"count":4,"name":"shoegaze"},{"count":2,"name":"dream pop"}]}`) defer srv.Close() old := mbBaseURL diff --git a/internal/tags/provider_lastfm.go b/internal/tags/provider_lastfm.go index 12917333..acca1e0f 100644 --- a/internal/tags/provider_lastfm.go +++ b/internal/tags/provider_lastfm.go @@ -122,7 +122,7 @@ func (p *lastfmProvider) FetchTrackTags(ctx context.Context, ref TrackRef) ([]Ta } // FetchArtistTags looks up an artist's top tags by name, with the MBID as an -// extra hint when present (#2376). Name-first is deliberate and the opposite +// extra hint when present. Name-first is deliberate and the opposite // emphasis from MusicBrainz: Last.fm's tag data is keyed on its own artist // pages, and autocorrect resolves most spelling drift from the similarity feed. // diff --git a/internal/tags/provider_musicbrainz.go b/internal/tags/provider_musicbrainz.go index b652a917..d477494d 100644 --- a/internal/tags/provider_musicbrainz.go +++ b/internal/tags/provider_musicbrainz.go @@ -3,23 +3,24 @@ package tags import ( "context" "errors" + "fmt" "net/http" "net/url" "sync/atomic" "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" ) // mbBaseURL is a var (not const) so tests can point it at an httptest server. var mbBaseURL = "https://musicbrainz.org/ws/2" const ( - // mbMinPeriod honors MusicBrainz's 1 req/s anonymous ceiling (with headroom). - mbMinPeriod = 1100 * time.Millisecond // mbUserAgent identifies the app — MusicBrainz rejects requests without one. - mbUserAgent = "Minstrel/1.0 ( https://git.fabledsword.com/bvandeusen/minstrel )" + mbUserAgent = musicbrainz.UserAgent // artistTagWeightFactor down-weights artist-level fallback tags relative // to recording-specific tags — the artist's overall character is a coarser - // signal than a tag on the exact recording (#1519). + // signal than a tag on the exact recording. artistTagWeightFactor = 0.6 ) @@ -27,7 +28,7 @@ const ( // service. Keyless (the public endpoint needs no auth) and on by default — // the always-available baseline source. Prefers recording-level tags by the // track's recording MBID; when the recording is untagged or MBID-less it -// falls back to the artist's tags by artist MBID (#1519), down-weighted as a +// falls back to the artist's tags by artist MBID, down-weighted as a // coarser signal. Name-based lookup is deferred to providers like Last.fm. type musicbrainzProvider struct { enabled atomic.Bool @@ -37,11 +38,13 @@ type musicbrainzProvider struct { func init() { Register(&musicbrainzProvider{ client: newHTTPClient(httpClientOptions{ - Name: "musicbrainz", - UserAgent: mbUserAgent, - HTTPClient: &http.Client{Timeout: 30 * time.Second}, - MinInterval: mbMinPeriod, - MaxRetries: 3, + Name: "musicbrainz", + UserAgent: mbUserAgent, + HTTPClient: &http.Client{Timeout: 30 * time.Second}, + // MusicBrainz's 1 request/second is per address: share it with + // the release catalogue. + Pacer: musicbrainz.Shared(), + MaxRetries: 3, }), }) } @@ -96,7 +99,7 @@ func (p *musicbrainzProvider) FetchTrackTags(ctx context.Context, ref TrackRef) return nil, ErrNotFound } -// FetchArtistTags looks up an artist's own tags by MBID (#2376). MBID-only: +// FetchArtistTags looks up an artist's own tags by MBID. MBID-only: // MusicBrainz has no name-based tag lookup worth trusting for this, and a // wrong-artist match would poison the tag cache silently. // @@ -105,7 +108,7 @@ func (p *musicbrainzProvider) FetchTrackTags(ctx context.Context, ref TrackRef) // the artist's overall character is the coarser signal of the two. Here the // artist IS the subject, so there is nothing to discount relative to — and // applying it would make these weights incomparable with track_tags, which is -// exactly the comparison slice 6 depends on. +// the comparison tag-space Discover depends on. func (p *musicbrainzProvider) FetchArtistTags(ctx context.Context, ref ArtistRef) ([]Tag, error) { if !p.enabled.Load() || ref.MBID == "" { return nil, ErrNotFound @@ -156,8 +159,8 @@ func (p *musicbrainzProvider) TestConnection(ctx context.Context) error { } // ReleaseGroupForRelease names the MusicBrainz release group a release -// belongs to (#5241). Minstrel's albums.mbid is the release id from the tags, -// while Lidarr knows albums only by release group, so re-acquiring a lost +// belongs to. Minstrel's albums.mbid is the release id from the tags, +// while wants know albums only by release group, so re-acquiring a lost // album needs this translation when its files (and their release-group tag) // are already gone. // @@ -170,15 +173,20 @@ func (p *musicbrainzProvider) TestConnection(ctx context.Context) error { func ReleaseGroupForRelease(ctx context.Context, releaseMBID string) (string, error) { p, err := ProviderByID("musicbrainz") if err != nil { - return "", ErrNotFound + return "", ErrSwitchedOff } mb, ok := p.(*musicbrainzProvider) if !ok { - return "", ErrNotFound + return "", ErrSwitchedOff } return mb.releaseGroupForRelease(ctx, releaseMBID) } +// ErrSwitchedOff is ReleaseGroupForRelease's answer while MusicBrainz is +// switched off. It is an ErrNotFound, so callers that only skip still do; a +// caller that would record "MusicBrainz has none" can tell it apart. +var ErrSwitchedOff = fmt.Errorf("%w: MusicBrainz is switched off", ErrNotFound) + // mbReleaseResponse models the subset of /ws/2/release/{mbid}?inc=release-groups // we read. type mbReleaseResponse struct { @@ -188,7 +196,10 @@ type mbReleaseResponse struct { } func (p *musicbrainzProvider) releaseGroupForRelease(ctx context.Context, releaseMBID string) (string, error) { - if !p.enabled.Load() || releaseMBID == "" { + if !p.enabled.Load() { + return "", ErrSwitchedOff + } + if releaseMBID == "" { return "", ErrNotFound } q := url.Values{"inc": {"release-groups"}, "fmt": {"json"}} diff --git a/internal/tags/provider_musicbrainz_test.go b/internal/tags/provider_musicbrainz_test.go index 9de04629..042887a1 100644 --- a/internal/tags/provider_musicbrainz_test.go +++ b/internal/tags/provider_musicbrainz_test.go @@ -177,7 +177,7 @@ func TestMusicBrainzFetch_EmptyTags(t *testing.T) { } } -// #5241: a release id resolves to its release group through +// A release id resolves to its release group through // /release/?inc=release-groups; an unknown id (or one that is already a // group) and a switched-off source both say ErrNotFound. func TestMusicBrainzReleaseGroupForRelease(t *testing.T) { diff --git a/internal/tags/provider_test.go b/internal/tags/provider_test.go index fec124e8..aa6dd2ba 100644 --- a/internal/tags/provider_test.go +++ b/internal/tags/provider_test.go @@ -50,7 +50,7 @@ func TestRegisterDuplicatePanics(t *testing.T) { Register(&fakeProvider{id: "dup"}) } -// TestBuiltinProvidersRegister confirms the two v1 sources register with +// TestBuiltinProvidersRegister confirms the two built-in sources register with // the expected identities/defaults. Runs against a fresh registry so it is // independent of test ordering: re-register constructed instances. func TestBuiltinProvidersRegister(t *testing.T) { diff --git a/internal/tags/worker.go b/internal/tags/worker.go index 35ab883c..0706a7ff 100644 --- a/internal/tags/worker.go +++ b/internal/tags/worker.go @@ -28,7 +28,7 @@ type Worker struct { // // candidateBatch is smaller than the track batch on purpose. Library tracks are // a finite set that drains to completion and then costs nothing; out-of-library -// candidates (#2376) are effectively unbounded — every library artist's +// candidates are effectively unbounded — every library artist's // neighbours — so this arm would otherwise monopolise every tick forever and // starve the track arm. 50/tick at ~1 req/s is roughly a minute of work, and // the query hands back the highest-similarity candidates first so the ones that diff --git a/internal/taste/engagement.go b/internal/taste/engagement.go index 40bcab7d..70613352 100644 --- a/internal/taste/engagement.go +++ b/internal/taste/engagement.go @@ -1,4 +1,4 @@ -// Package taste builds a persistent, decaying per-user taste profile (#796). +// Package taste builds a persistent, decaying per-user taste profile. // engagement.go is the pure core: it turns a single play's completion ratio // into a graded taste signal in [-1, +1], replacing the binary was_skipped // for learning purposes (was_skipped stays as-is for History display). diff --git a/internal/taste/profile.go b/internal/taste/profile.go index 65fd187a..2e43005f 100644 --- a/internal/taste/profile.go +++ b/internal/taste/profile.go @@ -1,7 +1,7 @@ // profile.go assembles the per-user taste profile from graded play // engagement (decayed over time) plus explicit-like bonuses, and -// atomic-replaces the taste_profile_* tables. Phase 1 only builds the -// object; candidate sourcing + scoring consume it in phase 2. +// atomic-replaces the taste_profile_* tables, which candidate sourcing and +// scoring read. package taste import ( @@ -19,9 +19,8 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/mood" ) -// Config holds the operator-tunable knobs. DefaultConfig supplies tuned -// starting values; wiring these into the server RecommendationConfig is a -// later follow-up (phase 1 uses the defaults). +// Config holds the tunable knobs. DefaultConfig supplies the shipped values; +// internal/recsettings overlays what an admin has tuned. type Config struct { Engagement EngagementParams @@ -37,8 +36,8 @@ type Config struct { TrackLikeBonus float64 TagLikeBonus float64 - // EnrichedTagScale weights folksonomy tags (from the track_tags cache, - // #1490) relative to a track's raw ID3 genre when both fold into the tag + // EnrichedTagScale weights folksonomy tags (from the track_tags cache) + // relative to a track's raw ID3 genre when both fold into the tag // facet. Each enriched tag contributes base × tag.weight × this scale, // where base is the play's decayed engagement (or the tag-like bonus for // a liked track) and tag.weight is the normalized folksonomy strength in @@ -47,19 +46,19 @@ type Config struct { // contribution entirely (falls back to genre-only). EnrichedTagScale float64 - // EraScale weights the decade/era facet (#1530): each play adds + // EraScale weights the decade/era facet: each play adds // base × this scale to its album's decade bucket ("1990s"), where base // is the decayed engagement (or the tag-like bonus for a liked track). // Era is a coarse signal, so it's kept ≤ 1 to imprint at most as strongly - // as the raw engagement; 0 disables the era facet entirely. Tunable in the - // lab (mirrors EnrichedTagScale, #1520). + // as the raw engagement; 0 disables the era facet entirely. Tunable in + // Admin › Tuning. EraScale float64 - // MoodScale weights the mood facet (#1534): each play adds base × this + // MoodScale weights the mood facet: each play adds base × this // scale to the canonical mood buckets (melancholic / chill / …) derived // from its enriched folksonomy tags via internal/mood. Coverage is partial // (depends on tag enrichment), so it's a supplement kept ≤ 1; 0 disables - // the mood facet entirely. Tunable in the lab (mirrors EraScale). + // the mood facet entirely. Tunable in Admin › Tuning. MoodScale float64 // ArtistFloor / TagFloor clamp how negative a single entity's weight may @@ -166,7 +165,7 @@ func accumulate( if err != nil { return nil, nil, nil, nil, fmt.Errorf("taste: load play engagement: %w", err) } - // Enriched folksonomy tags keyed by track (#1490) — folded into the tag + // Enriched folksonomy tags keyed by track, folded into the tag // facet alongside raw ID3 genre so a coarse "Rock" gains the cached // "post-punk / shoegaze / melancholic" vocabulary. Window matches the // play window; a track with no cached tags simply contributes genre only. diff --git a/internal/tracks/merge_test.go b/internal/tracks/merge_test.go deleted file mode 100644 index c83d5ba7..00000000 --- a/internal/tracks/merge_test.go +++ /dev/null @@ -1,39 +0,0 @@ -package tracks - -import ( - "testing" - - "git.fabledsword.com/bvandeusen/minstrel/internal/library" -) - -func mbids(track, album string) library.MergedCopy { - c := library.MergedCopy{} - if track != "" { - c.TrackMbid = &track - } - if album != "" { - c.AlbumMbid = &album - } - return c -} - -// Unmonitoring a second file of the kept copy's own album track would stop -// Lidarr managing the kept file too; that case must be recognised. -func TestSameLidarrTrack(t *testing.T) { - for _, tc := range []struct { - name string - kept library.MergedCopy - removed library.MergedCopy - wantSame bool - }{ - {"same recording on the same album", mbids("rec", "alb"), mbids("rec", "alb"), true}, - {"same recording on a compilation", mbids("rec", "alb"), mbids("rec", "comp"), false}, - {"different recordings on one album", mbids("rec", "alb"), mbids("rec-2", "alb"), false}, - {"kept copy has no mbids", mbids("", ""), mbids("rec", "alb"), false}, - {"removed copy has no album mbid", mbids("rec", "alb"), mbids("rec", ""), false}, - } { - if got := sameLidarrTrack(tc.kept, tc.removed); got != tc.wantSame { - t.Errorf("%s: sameLidarrTrack = %v, want %v", tc.name, got, tc.wantSame) - } - } -} diff --git a/internal/tracks/service.go b/internal/tracks/service.go index 1d460677..f1cafd71 100644 --- a/internal/tracks/service.go +++ b/internal/tracks/service.go @@ -1,20 +1,10 @@ // Package tracks owns the track-level admin actions behind DELETE -// /api/admin/tracks/{id}. Today that's RemoveTrack: the destructive part goes -// through library.DeleteTrackFile — the one path that deletes a track file — -// and when the operator opts in via `unmonitor=true` the service also tells -// Lidarr to flip the track's monitored flag off so Lidarr doesn't search for a -// replacement. +// /api/admin/tracks/{id} and the duplicate merge. RemoveTrack's destructive +// part goes through library.DeleteTrackFile — the one path that deletes a +// track file. // -// History: an earlier shape (commit 50a231f, since rewritten) routed -// Lidarr-managed tracks through lidarrquarantine.DeleteViaLidarr — but -// that primitive deletes the entire **album** in Lidarr (Lidarr is -// album-granular and has no per-track delete API), which would silently -// drop sibling tracks the operator didn't ask to remove. The current -// shape per spec revision 723eee9 is "always direct delete; opt-in -// Lidarr unmonitor for replacement-suppression." -// -// The web track-kebab entry that called this was removed in f7278f24; the -// endpoint was kept deliberately for a safer admin surface to rebind. +// No client calls the endpoint today; it is kept for an admin surface to +// rebind. package tracks import ( @@ -31,7 +21,6 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/audit" "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" "git.fabledsword.com/bvandeusen/minstrel/internal/library" - "git.fabledsword.com/bvandeusen/minstrel/internal/lidarr" syncpkg "git.fabledsword.com/bvandeusen/minstrel/internal/sync" ) @@ -41,171 +30,82 @@ import ( // same pointer. var ErrNotFound = apierror.ErrNotFound -// ErrLidarrDisabled is what a LidarrUnmonitorer that also lists Lidarr's -// unmapped files returns when Lidarr is turned off: there is then no Lidarr -// record a removal could break, and a merge goes ahead unguarded. -var ErrLidarrDisabled = errors.New("tracks: lidarr disabled") - -// ErrLidarrUnavailable means Lidarr is on but did not answer, so a merge cannot -// confirm the copies it would remove are ones Lidarr does not need (M498), and -// refuses rather than guess. -var ErrLidarrUnavailable = errors.New("tracks: lidarr did not answer") - -// lidarrFileLister is the optional part of the Lidarr dependency a merge uses to -// learn which files Lidarr holds without mapping to any track. -type lidarrFileLister interface { - ListUnmappedTrackFiles(ctx context.Context) ([]lidarr.TrackFile, error) -} - -// LidarrUnmonitorer is the subset of *lidarr.Client RemoveTrack uses. -// Defined as an interface so tests can stub without spinning up a Lidarr -// httptest server. UnmonitorTrack failures are non-fatal at the service -// layer — the file + DB are already gone — so any error returned here -// surfaces as a `lidarr_unmonitor_failed` flag in the response, not as -// a hard error. -type LidarrUnmonitorer interface { - UnmonitorTrack(ctx context.Context, trackMbid, albumMbid string) error -} - -// Service owns RemoveTrack. lidarr may be nil — when it is, the -// unmonitor branch is skipped entirely (with `lidarrUnmonitorFailed` -// remaining false) regardless of the unmonitor query param. This is -// the right fallback when Lidarr isn't configured: file + DB delete -// still happen. +// Service owns RemoveTrack and MergeDuplicates. type Service struct { pool *pgxpool.Pool logger *slog.Logger - lidarr LidarrUnmonitorer dataDir string } // NewService constructs a Service. logger may be nil (defaults to -// slog.Default). lidarr may be nil to disable the unmonitor branch. -// dataDir is the on-disk root for cached artifacts; used to clean up -// artist-art on artist delete. Empty string disables the cleanup. -func NewService(pool *pgxpool.Pool, logger *slog.Logger, lidarr LidarrUnmonitorer, dataDir string) *Service { +// slog.Default). dataDir is the on-disk root for cached artifacts; used to +// clean up artist-art on artist delete. Empty string disables the cleanup. +func NewService(pool *pgxpool.Pool, logger *slog.Logger, dataDir string) *Service { if logger == nil { logger = slog.Default() } - return &Service{pool: pool, logger: logger, lidarr: lidarr, dataDir: dataDir} + return &Service{pool: pool, logger: logger, dataDir: dataDir} } -// RemoveTrack deletes the track's file and then its row, tidies away an album -// or artist the delete empties, and (when unmonitor is true and the track is -// Lidarr-managed) tells Lidarr to flip the track's monitored flag off so it -// won't search for a replacement. +// RemoveTrack deletes the track's file and then its row, and tidies away an +// album or artist the delete empties. // // Returns: // - deletedAlbumID: non-nil when removing the track left the album // empty and the album row was deleted. // - deletedArtistID: non-nil when both album AND artist were left // empty (only set if deletedAlbumID is also set). -// - lidarrUnmonitorFailed: true when the operator requested unmonitor -// and the Lidarr call failed; the file + DB delete still succeeded. // - err: ErrNotFound, or a failure before anything was deleted. When the // file cannot be removed it is a *library.FileRemoveError and NOTHING was -// deleted — see library.DeleteTrackFile for why that order is the contract -// (#3918). A failed Lidarr unmonitor is reflected in the bool, not here. +// deleted — see library.DeleteTrackFile for why that order is the contract. // -// adminID is currently unused — the cascade audit-log line that would -// reference it isn't wired in this slice. It's threaded through the -// signature so the upcoming admin_tracks handler doesn't have to -// re-plumb when audit logging lands. +// A removal is written to the audit log with adminID as the actor. The +// track's names are read before the delete, since the row, and perhaps its +// album and artist, are gone after it. func (s *Service) RemoveTrack( - ctx context.Context, - trackID, adminID pgtype.UUID, //nolint:revive // adminID reserved for audit-log wiring in a follow-up - unmonitor bool, -) (deletedAlbumID *pgtype.UUID, deletedArtistID *pgtype.UUID, lidarrUnmonitorFailed bool, err error) { + ctx context.Context, trackID, adminID pgtype.UUID, +) (deletedAlbumID *pgtype.UUID, deletedArtistID *pgtype.UUID, err error) { q := dbq.New(s.pool) - track, err := q.GetTrackByID(ctx, trackID) if err != nil { if errors.Is(err, pgx.ErrNoRows) { - return nil, nil, false, ErrNotFound + return nil, nil, ErrNotFound } - return nil, nil, false, fmt.Errorf("get track: %w", err) + return nil, nil, fmt.Errorf("get track: %w", err) } - - // Capture the album's mbid *before* the delete. If removing this track - // empties the album, its row is gone afterwards and the unmonitor walk - // would have no album mbid to name. - var albumMbid string - if track.Mbid != nil && *track.Mbid != "" && unmonitor && s.lidarr != nil { - alb, alerr := q.GetAlbumByID(ctx, track.AlbumID) - if alerr == nil && alb.Mbid != nil { - albumMbid = *alb.Mbid - } - // Lookup failure is tolerated — handled below as - // "no albumMbid → can't unmonitor → flag failure." + meta := map[string]any{ + "track_id": syncpkg.FormatUUID(track.ID), + "title": track.Title, + "file_path": track.FilePath, + } + if artist, err := q.GetArtistByID(ctx, track.ArtistID); err == nil { + meta["artist"] = artist.Name + } + if album, err := q.GetAlbumByID(ctx, track.AlbumID); err == nil { + meta["album"] = album.Title } deleted, err := library.DeleteTrackFile(ctx, s.pool, s.logger, s.dataDir, trackID) if err != nil { if errors.Is(err, library.ErrTrackNotFound) { - return nil, nil, false, ErrNotFound + return nil, nil, ErrNotFound } - return nil, nil, false, fmt.Errorf("delete track: %w", err) + return nil, nil, fmt.Errorf("delete track: %w", err) } - - // Lidarr unmonitor — non-fatal. The destructive part is done; any - // failure here is informational so the operator can retry manually. - if unmonitor && track.Mbid != nil && *track.Mbid != "" && s.lidarr != nil { - if albumMbid == "" { - // Couldn't capture the album mbid (album row had nil mbid - // or was missing somehow). The Lidarr walk needs it; mark - // failure rather than calling with an empty string. - s.logger.Warn("track delete: lidarr unmonitor skipped — no album mbid", - "track_id", trackID, "track_mbid", *track.Mbid) - lidarrUnmonitorFailed = true - } else if uerr := s.lidarr.UnmonitorTrack(ctx, *track.Mbid, albumMbid); uerr != nil { - s.logger.Warn("track delete: lidarr unmonitor failed", - "track_id", trackID, "track_mbid", *track.Mbid, "err", uerr) - lidarrUnmonitorFailed = true - } - } - - return deleted.AlbumID, deleted.ArtistID, lidarrUnmonitorFailed, nil + meta["album_deleted"] = deleted.AlbumID != nil + meta["artist_deleted"] = deleted.ArtistID != nil + audit.WriteOrLog(ctx, s.pool, s.logger, adminID, pgtype.UUID{}, audit.ActionTrackRemove, meta) + return deleted.AlbumID, deleted.ArtistID, nil } // MergeDuplicates merges a duplicate group into the copy to keep -// (library.MergeDuplicateGroupGuarded, refusing to remove a copy Lidarr maps: -// library.ErrCopyTrackedByLidarr, or ErrLidarrUnavailable when Lidarr cannot -// say), then — when asked — tells Lidarr to stop -// monitoring the removed copies so it does not download them again, and records -// the merge in the audit log. -// -// lidarrUnmonitorFailed reports that unmonitoring was asked for and at least one -// removed copy could not be unmonitored. Like RemoveTrack, that never fails the -// merge: the files and rows are already gone. +// (library.MergeDuplicateGroup) and records the merge in the audit log. func (s *Service) MergeDuplicates( - ctx context.Context, groupID, survivorID, actorID pgtype.UUID, unmonitor bool, -) (res library.MergeResult, lidarrUnmonitorFailed bool, err error) { - removable, err := s.lidarrRemovable(ctx) + ctx context.Context, groupID, survivorID, actorID pgtype.UUID, +) (library.MergeResult, error) { + res, err := library.MergeDuplicateGroup(ctx, s.pool, s.logger, s.dataDir, groupID, survivorID) if err != nil { - return res, false, err - } - res, err = library.MergeDuplicateGroupGuarded(ctx, s.pool, s.logger, s.dataDir, groupID, survivorID, removable) - if err != nil { - return res, false, err - } - - if unmonitor && s.lidarr != nil { - for _, removed := range res.Removed { - if sameLidarrTrack(res.Survivor, removed) { - continue - } - if !hasLidarrIdentity(removed) { - s.logger.Warn("duplicate merge: lidarr unmonitor skipped — removed copy has no mbids", - "track_id", syncpkg.FormatUUID(removed.TrackID)) - lidarrUnmonitorFailed = true - continue - } - if uerr := s.lidarr.UnmonitorTrack(ctx, *removed.TrackMbid, *removed.AlbumMbid); uerr != nil { - s.logger.Warn("duplicate merge: lidarr unmonitor failed", - "track_id", syncpkg.FormatUUID(removed.TrackID), "err", uerr) - lidarrUnmonitorFailed = true - } - } + return res, err } removed := make([]map[string]string, 0, len(res.Removed)) @@ -223,42 +123,5 @@ func (s *Service) MergeDuplicates( "likes": res.Likes, "playlist_entries": res.PlaylistEntries, }, }) - return res, lidarrUnmonitorFailed, nil -} - -// lidarrRemovable is the merge's guard (M498): only a copy Lidarr holds without -// mapping it to a track may be removed, because removing a mapped one makes -// Lidarr download it again. Nil, allowing everything, when Lidarr is disabled -// or the dependency cannot list files. -func (s *Service) lidarrRemovable(ctx context.Context) (library.RemovableFunc, error) { - lister, ok := s.lidarr.(lidarrFileLister) - if !ok { - return nil, nil - } - files, err := lister.ListUnmappedTrackFiles(ctx) - if errors.Is(err, ErrLidarrDisabled) { - return nil, nil - } - if err != nil { - return nil, fmt.Errorf("%w: %v", ErrLidarrUnavailable, err) - } - unmapped := make(map[string]bool, len(files)) - for _, f := range files { - unmapped[library.LidarrPathKey(f.Path)] = true - } - return func(p string) bool { return unmapped[library.LidarrPathKey(p)] }, nil -} - -// sameLidarrTrack reports whether two copies are the same Lidarr track: one -// recording on one album. Lidarr monitors per album track, so when the removed -// copy is a second file of the kept copy's own album track — the #3885 case — -// unmonitoring it would also stop Lidarr managing the file the operator chose to -// keep. Those are skipped, and are not a failure. -func sameLidarrTrack(a, b library.MergedCopy) bool { - return hasLidarrIdentity(a) && hasLidarrIdentity(b) && - *a.TrackMbid == *b.TrackMbid && *a.AlbumMbid == *b.AlbumMbid -} - -func hasLidarrIdentity(c library.MergedCopy) bool { - return c.TrackMbid != nil && *c.TrackMbid != "" && c.AlbumMbid != nil && *c.AlbumMbid != "" + return res, nil } diff --git a/internal/tracks/service_test.go b/internal/tracks/service_test.go index 0456619a..06a354f4 100644 --- a/internal/tracks/service_test.go +++ b/internal/tracks/service_test.go @@ -17,7 +17,7 @@ import ( "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" ) -// --- test seed helpers (inlined; mirrors lidarrquarantine patterns) ----- +// --- test seed helpers (inlined; mirrors quarantine patterns) ----------- func newPool(t *testing.T) *pgxpool.Pool { t.Helper() @@ -95,23 +95,6 @@ func seedArtistAlbumTrack(t *testing.T, pool *pgxpool.Pool, prefix, mbid, filePa return artist, album, track } -// --- fake LidarrUnmonitorer -------------------------------------------- - -// fakeLidarrUnmonitorer captures call args + a configurable error so -// tests can assert "called with mbid X" and "RemoveTrack tolerated this -// Lidarr error." -type fakeLidarrUnmonitorer struct { - calls []string // captured trackMbid per call - albumMbid string // last albumMbid received - err error -} - -func (f *fakeLidarrUnmonitorer) UnmonitorTrack(_ context.Context, trackMbid, albumMbid string) error { - f.calls = append(f.calls, trackMbid) - f.albumMbid = albumMbid - return f.err -} - // --- tests -------------------------------------------------------------- func TestRemoveTrack_NotFound(t *testing.T) { @@ -122,8 +105,8 @@ func TestRemoveTrack_NotFound(t *testing.T) { bogus.Bytes = [16]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16} bogus.Valid = true - svc := NewService(pool, nil, nil, "") - _, _, _, err := svc.RemoveTrack(context.Background(), bogus, admin.ID, false) + svc := NewService(pool, nil, "") + _, _, err := svc.RemoveTrack(context.Background(), bogus, admin.ID) if !errors.Is(err, ErrNotFound) { t.Errorf("err = %v, want ErrNotFound", err) } @@ -138,18 +121,13 @@ func TestRemoveTrack_LocalFile_HappyPath(t *testing.T) { if err := os.WriteFile(path, []byte("x"), 0o644); err != nil { t.Fatal(err) } - // mbid empty → no Lidarr branch even with unmonitor=true. _, _, track := seedArtistAlbumTrack(t, pool, "Solo", "", path) - fake := &fakeLidarrUnmonitorer{} - svc := NewService(pool, nil, fake, "") - _, _, lidarrFailed, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID, false) + svc := NewService(pool, nil, "") + _, _, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID) if err != nil { t.Fatalf("RemoveTrack: %v", err) } - if lidarrFailed { - t.Error("lidarrUnmonitorFailed = true; want false on local-file happy path") - } if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) { t.Errorf("file still present: %v", err) } @@ -157,8 +135,46 @@ func TestRemoveTrack_LocalFile_HappyPath(t *testing.T) { if _, err := q.GetTrackByID(context.Background(), track.ID); err == nil { t.Errorf("track row still exists") } - if len(fake.calls) != 0 { - t.Errorf("Lidarr called %d times; want 0 (no mbid, unmonitor=false)", len(fake.calls)) +} + +func TestRemoveTrack_WritesAuditRow(t *testing.T) { + pool := newPool(t) + admin := seedAdmin(t, pool, "alice") + if _, err := pool.Exec(context.Background(), `DELETE FROM audit_log`); err != nil { + t.Fatalf("reset audit_log: %v", err) + } + + dir := t.TempDir() + path := filepath.Join(dir, "track.mp3") + if err := os.WriteFile(path, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + _, _, track := seedArtistAlbumTrack(t, pool, "Audited", "", path) + + svc := NewService(pool, nil, "") + if _, _, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID); err != nil { + t.Fatalf("RemoveTrack: %v", err) + } + + var actor pgtype.UUID + var title, artist, album, filePath string + var albumDeleted bool + err := pool.QueryRow(context.Background(), ` + SELECT actor_id, metadata->>'title', metadata->>'artist', metadata->>'album', + metadata->>'file_path', (metadata->>'album_deleted')::boolean + FROM audit_log WHERE action = 'track_remove'`). + Scan(&actor, &title, &artist, &album, &filePath, &albumDeleted) + if err != nil { + t.Fatalf("read audit row: %v", err) + } + if actor != admin.ID { + t.Errorf("actor = %v, want the admin %v", actor, admin.ID) + } + if title != "Audited Track" || artist != "Audited Artist" || album != "Audited Album" || filePath != path { + t.Errorf("metadata = %q / %q / %q / %q", title, artist, album, filePath) + } + if !albumDeleted { + t.Errorf("album_deleted = false, want true: the track was its album's only one") } } @@ -170,8 +186,8 @@ func TestRemoveTrack_FileAlreadyMissing(t *testing.T) { missing := filepath.Join(t.TempDir(), "does-not-exist.mp3") _, _, track := seedArtistAlbumTrack(t, pool, "Ghost", "", missing) - svc := NewService(pool, nil, nil, "") - _, _, _, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID, false) + svc := NewService(pool, nil, "") + _, _, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID) if err != nil { t.Fatalf("RemoveTrack: %v", err) } @@ -226,8 +242,8 @@ func TestRemoveTrack_Cascade_AlbumEmpty(t *testing.T) { t.Fatalf("trackB: %v", err) } - svc := NewService(pool, nil, nil, "") - delAlbum, delArtist, _, err := svc.RemoveTrack(context.Background(), trackA.ID, admin.ID, false) + svc := NewService(pool, nil, "") + delAlbum, delArtist, err := svc.RemoveTrack(context.Background(), trackA.ID, admin.ID) if err != nil { t.Fatalf("RemoveTrack: %v", err) } @@ -259,8 +275,8 @@ func TestRemoveTrack_Cascade_ArtistEmpty(t *testing.T) { } artist, album, track := seedArtistAlbumTrack(t, pool, "Lone", "", path) - svc := NewService(pool, nil, nil, "") - delAlbum, delArtist, _, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID, false) + svc := NewService(pool, nil, "") + delAlbum, delArtist, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID) if err != nil { t.Fatalf("RemoveTrack: %v", err) } @@ -278,174 +294,3 @@ func TestRemoveTrack_Cascade_ArtistEmpty(t *testing.T) { t.Errorf("artist still exists after cascade") } } - -// unmonitor=true AND mbid set → Lidarr called once; lidarrUnmonitorFailed=false. -// Uses two albums on the same artist so the album's row survives the -// cascade — that lets us assert the album mbid (captured pre-tx) is -// passed through correctly. -func TestRemoveTrack_Unmonitor_HappyPath(t *testing.T) { - pool := newPool(t) - admin := seedAdmin(t, pool, "alice") - - q := dbq.New(pool) - artist, err := q.UpsertArtist(context.Background(), dbq.UpsertArtistParams{ - Name: "Lidarr Artist", SortName: "Lidarr Artist", - }) - if err != nil { - t.Fatalf("artist: %v", err) - } - albumMbid := "album-mbid-xyz" - album, err := q.UpsertAlbum(context.Background(), dbq.UpsertAlbumParams{ - Title: "Lidarr Album", SortTitle: "Lidarr Album", - ArtistID: artist.ID, Mbid: &albumMbid, - }) - if err != nil { - t.Fatalf("album: %v", err) - } - dir := t.TempDir() - pathA := filepath.Join(dir, "a.mp3") - pathB := filepath.Join(dir, "b.mp3") - _ = os.WriteFile(pathA, []byte("x"), 0o644) - _ = os.WriteFile(pathB, []byte("x"), 0o644) - trackMbid := "track-mbid-abc" - track, err := q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ - Title: "T1", AlbumID: album.ID, ArtistID: artist.ID, - DurationMs: 1000, FilePath: pathA, FileSize: 1, FileFormat: "mp3", - Mbid: &trackMbid, - }) - if err != nil { - t.Fatalf("track1: %v", err) - } - // Sibling track keeps the album alive after the delete. - if _, err := q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ - Title: "T2", AlbumID: album.ID, ArtistID: artist.ID, - DurationMs: 1000, FilePath: pathB, FileSize: 1, FileFormat: "mp3", - }); err != nil { - t.Fatalf("track2: %v", err) - } - - fake := &fakeLidarrUnmonitorer{} - svc := NewService(pool, nil, fake, "") - _, _, lidarrFailed, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID, true) - if err != nil { - t.Fatalf("RemoveTrack: %v", err) - } - if lidarrFailed { - t.Error("lidarrUnmonitorFailed = true; want false on happy path") - } - if len(fake.calls) != 1 { - t.Fatalf("Lidarr call count = %d, want 1", len(fake.calls)) - } - if fake.calls[0] != trackMbid { - t.Errorf("Lidarr called with mbid %q, want %q", fake.calls[0], trackMbid) - } - if fake.albumMbid != albumMbid { - t.Errorf("Lidarr called with albumMbid %q, want %q", fake.albumMbid, albumMbid) - } -} - -// unmonitor=true AND Lidarr fails → file + DB still deleted, flag set, -// service returns nil error. -func TestRemoveTrack_Unmonitor_LidarrErrorIsNonFatal(t *testing.T) { - pool := newPool(t) - admin := seedAdmin(t, pool, "alice") - - q := dbq.New(pool) - artist, err := q.UpsertArtist(context.Background(), dbq.UpsertArtistParams{ - Name: "ErrFatal Artist", SortName: "ErrFatal Artist", - }) - if err != nil { - t.Fatalf("artist: %v", err) - } - albumMbid := "err-album-mbid" - album, err := q.UpsertAlbum(context.Background(), dbq.UpsertAlbumParams{ - Title: "ErrFatal Album", SortTitle: "ErrFatal Album", - ArtistID: artist.ID, Mbid: &albumMbid, - }) - if err != nil { - t.Fatalf("album: %v", err) - } - dir := t.TempDir() - path := filepath.Join(dir, "f.mp3") - if err := os.WriteFile(path, []byte("x"), 0o644); err != nil { - t.Fatal(err) - } - trackMbid := "err-track-mbid" - track, err := q.UpsertTrack(context.Background(), dbq.UpsertTrackParams{ - Title: "T", AlbumID: album.ID, ArtistID: artist.ID, - DurationMs: 1000, FilePath: path, FileSize: 1, FileFormat: "mp3", - Mbid: &trackMbid, - }) - if err != nil { - t.Fatalf("track: %v", err) - } - - fake := &fakeLidarrUnmonitorer{err: errors.New("simulated lidarr 5xx")} - svc := NewService(pool, nil, fake, "") - _, _, lidarrFailed, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID, true) - if err != nil { - t.Fatalf("RemoveTrack: %v (want nil — Lidarr failure is non-fatal)", err) - } - if !lidarrFailed { - t.Error("lidarrUnmonitorFailed = false; want true after Lidarr error") - } - // File + DB still gone. - if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) { - t.Errorf("file still present after Lidarr error: %v", err) - } - if _, err := q.GetTrackByID(context.Background(), track.ID); err == nil { - t.Errorf("track row still present after Lidarr error") - } -} - -// unmonitor=true AND mbid empty (non-Lidarr track) → no Lidarr call. -func TestRemoveTrack_Unmonitor_NoMbidSkipsLidarr(t *testing.T) { - pool := newPool(t) - admin := seedAdmin(t, pool, "alice") - - dir := t.TempDir() - path := filepath.Join(dir, "local.mp3") - if err := os.WriteFile(path, []byte("x"), 0o644); err != nil { - t.Fatal(err) - } - _, _, track := seedArtistAlbumTrack(t, pool, "Local", "", path) - - fake := &fakeLidarrUnmonitorer{} - svc := NewService(pool, nil, fake, "") - _, _, lidarrFailed, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID, true) - if err != nil { - t.Fatalf("RemoveTrack: %v", err) - } - if lidarrFailed { - t.Error("lidarrUnmonitorFailed = true; want false (no mbid)") - } - if len(fake.calls) != 0 { - t.Errorf("Lidarr call count = %d, want 0 (track has no mbid)", len(fake.calls)) - } -} - -// unmonitor=false AND mbid set → no Lidarr call. -func TestRemoveTrack_NoUnmonitor_SkipsLidarr(t *testing.T) { - pool := newPool(t) - admin := seedAdmin(t, pool, "alice") - - dir := t.TempDir() - path := filepath.Join(dir, "lidarr.mp3") - if err := os.WriteFile(path, []byte("x"), 0o644); err != nil { - t.Fatal(err) - } - _, _, track := seedArtistAlbumTrack(t, pool, "NoMon", "track-mbid", path) - - fake := &fakeLidarrUnmonitorer{} - svc := NewService(pool, nil, fake, "") - _, _, lidarrFailed, err := svc.RemoveTrack(context.Background(), track.ID, admin.ID, false) - if err != nil { - t.Fatalf("RemoveTrack: %v", err) - } - if lidarrFailed { - t.Error("lidarrUnmonitorFailed = true; want false (unmonitor=false)") - } - if len(fake.calls) != 0 { - t.Errorf("Lidarr call count = %d, want 0 (unmonitor=false)", len(fake.calls)) - } -} diff --git a/internal/wanted/library.go b/internal/wanted/library.go new file mode 100644 index 00000000..ad397b25 --- /dev/null +++ b/internal/wanted/library.go @@ -0,0 +1,174 @@ +package wanted + +import ( + "context" + "errors" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +// ReasonLibrary: on the list because the library held it when the list +// began. +const ReasonLibrary = "library" + +// libraryMatchBudget bounds the time one pass spends matching albums to +// MusicBrainz, at its one request a second, so the pass still settles its +// wants; the rest wait for the next pass. +const libraryMatchBudget = 2 * time.Minute + +// libraryMatchBatch is how many albums one query hands the matching. +const libraryMatchBatch = 25 + +// libraryMatcher is the matching's version. Raise it when the matching +// changes in a way that could match an album it refused: albums an older +// version refused are matched again, even after the listing is done. +// +// 1 tags, release, discography and search, checked by title and length +// 2 titles alone under the album's own name; a refusal names the track +// 3 a second copy of a track counts once; joining words count for nothing +// 4 an album holding every track of an edition, and more, is that group +const libraryMatcher = 4 + +// listLibrary puts the library's albums on the wanted list as held, once, +// so that from then on the list is the record of what the library keeps: an +// album taken off it stays off, and files dropped in by hand are not listed. +// It runs a step each pass until done: +// +// - the first step stamps when the listing began, which is its cutoff: +// an album the library gains afterwards, dropped in by hand, is never +// listed; +// - every album whose tags name its release group and artist goes on the +// list at once; +// - every other album is matched to MusicBrainz, within a time +// budget a pass, and listed as it is matched; +// - once none is left to match, the listing is stamped done, with a count +// of the albums no match was found for. +// +// Each album decided is recorded in library_album_matches, so an album taken +// off the list stays off. Once done the listing runs again only for albums +// an older matcher refused (libraryMatcher), refused albums whose tracks +// have changed since, and library tracks a re-tag filed under a new album. +// A person matches the rest by hand, or leaves them off, in the +// Collection's Unmatched section (MatchAlbumTo, LeaveAlbumOff). +func (s *Service) listLibrary(ctx context.Context, res *PassResult) error { + q := dbq.New(s.pool) + state, err := q.GetLibraryListing(ctx) + if err != nil { + return fmt.Errorf("wanted: read the library listing: %w", err) + } + if state.LibraryListedAt.Valid { + stale, err := q.CountAlbumsToMatch(ctx, dbq.CountAlbumsToMatchParams{ + Cutoff: state.LibraryListingStartedAt, Matcher: libraryMatcher}) + if err != nil { + return fmt.Errorf("wanted: count albums to match again: %w", err) + } + if stale == 0 { + return nil + } + } + cutoff := state.LibraryListingStartedAt + if !cutoff.Valid { + if cutoff, err = q.StartLibraryListing(ctx); err != nil { + return fmt.Errorf("wanted: start the library listing: %w", err) + } + if err := q.RecordListedAlbums(ctx, cutoff); err != nil { + return fmt.Errorf("wanted: record the library: %w", err) + } + n, err := q.ListLibraryAlbums(ctx, dbq.ListLibraryAlbumsParams{Cutoff: cutoff}) + if err != nil { + return fmt.Errorf("wanted: list the library: %w", err) + } + res.Listed += int(n) + s.logger.Info("wanted: listing the library's albums as held", "listed", n) + } + + deadline := s.now().Add(libraryMatchBudget) + for { + albums, err := q.ListAlbumsToMatch(ctx, dbq.ListAlbumsToMatchParams{ + Cutoff: cutoff, Matcher: libraryMatcher, MaxRows: libraryMatchBatch}) + if err != nil { + return fmt.Errorf("wanted: albums to match: %w", err) + } + if len(albums) == 0 { + break + } + for _, a := range albums { + if ctx.Err() != nil { + return ctx.Err() + } + if s.now().After(deadline) { + return nil // the rest next pass + } + n, err := s.listAlbum(ctx, q, a) + if errors.Is(err, errMusicBrainzAway) { + s.logger.Info("wanted: matching the library's albums paused", "err", err) + return nil + } + if err != nil { + return err + } + res.Listed += n + } + } + return s.finishLibrary(ctx, q, res) +} + +// listAlbum matches one album and lists it, or records why it could not. +func (s *Service) listAlbum(ctx context.Context, q *dbq.Queries, a dbq.ListAlbumsToMatchRow) (int, error) { + rows, err := q.ListAlbumTracksToMatch(ctx, a.ID) + if err != nil { + return 0, fmt.Errorf("wanted: read %s's tracks: %w", a.Title, err) + } + m, err := s.matchAlbum(ctx, a, tracksToMatch(rows)) + if err != nil { + return 0, err + } + if !m.OK { + s.logger.Info("wanted: no MusicBrainz match for a library album", "album", a.Title, "artist", a.ArtistName, "why", m.Detail) + return 0, q.RecordAlbumMatch(ctx, dbq.RecordAlbumMatchParams{ + AlbumID: a.ID, Outcome: "unmatched", Detail: m.Detail, Matcher: libraryMatcher}) + } + var listed int64 + err = pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + tq := dbq.New(tx) + if err := tq.SetAlbumReleaseGroup(ctx, dbq.SetAlbumReleaseGroupParams{ID: a.ID, ReleaseGroupMbid: &m.Group}); err != nil { + return err + } + n, err := tq.InsertLibraryWant(ctx, dbq.InsertLibraryWantParams{ + ReleaseGroupMbid: m.Group, ArtistMbid: m.ArtistMBID, ArtistName: m.ArtistName, Title: m.Title, + PrimaryType: m.PrimaryType, FirstReleaseDate: m.FirstReleaseDate, AlbumID: a.ID, + }) + if err != nil { + return err + } + listed = n + return tq.RecordAlbumMatch(ctx, dbq.RecordAlbumMatchParams{ + AlbumID: a.ID, Outcome: "listed", Method: m.Method, ReleaseGroupMbid: m.Group, Matcher: libraryMatcher, + }) + }) + if err != nil { + return 0, fmt.Errorf("wanted: list album %s: %w", a.Title, err) + } + return int(listed), nil +} + +func (s *Service) finishLibrary(ctx context.Context, q *dbq.Queries, res *PassResult) error { + done, err := q.FinishLibraryListing(ctx) + if err != nil { + return fmt.Errorf("wanted: finish the library listing: %w", err) + } + s.logger.Info("wanted: the library's albums are on the list", "listed_this_pass", res.Listed, + "unmatched", done.LibraryUnlisted) + return nil +} + +func deref(p *string) string { + if p == nil { + return "" + } + return *p +} diff --git a/internal/wanted/library_test.go b/internal/wanted/library_test.go new file mode 100644 index 00000000..b153497e --- /dev/null +++ b/internal/wanted/library_test.go @@ -0,0 +1,321 @@ +package wanted + +import ( + "context" + "fmt" + "strings" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/tags" +) + +// startListing clears the reset's "already listed", as a fresh install has it. +func startListing(t *testing.T, pool *pgxpool.Pool) { + t.Helper() + if _, err := pool.Exec(context.Background(), ` + UPDATE acquisition_settings SET library_listing_started_at = NULL, library_listed_at = NULL, library_unlisted = 0`); err != nil { + t.Fatal(err) + } +} + +// seedAlbum puts an album in the library whose tracks are titled titles, +// numbered by numbers (from 1 when none are given). An artistMBID or +// release of "" leaves that id out, as untagged files do. +func seedAlbum(t *testing.T, q *dbq.Queries, artistMBID, title, rel string, titles []string, numbers ...int32) dbq.Album { + t.Helper() + ctx := context.Background() + ap := dbq.UpsertArtistParams{Name: "Blur", SortName: "Blur"} + if artistMBID != "" { + ap.Mbid = &artistMBID + } + artist, err := q.UpsertArtist(ctx, ap) + if err != nil { + t.Fatal(err) + } + params := dbq.UpsertAlbumParams{Title: title, SortTitle: title, ArtistID: artist.ID} + if rel != "" { + params.Mbid = &rel + } + album, err := q.UpsertAlbum(ctx, params) + if err != nil { + t.Fatal(err) + } + for i, name := range titles { + num := int32(i + 1) + if i < len(numbers) { + num = numbers[i] + } + if _, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: name, AlbumID: album.ID, ArtistID: artist.ID, TrackNumber: &num, DurationMs: 180000, + FilePath: fmt.Sprintf("/music/%s/%02d.flac", strings.ReplaceAll(title, " ", "_"), num), FileSize: 1, FileFormat: "flac", + }); err != nil { + t.Fatal(err) + } + } + return album +} + +func matchOf(t *testing.T, pool *pgxpool.Pool, album pgtype.UUID) (outcome, method, group, detail string) { + t.Helper() + if err := pool.QueryRow(context.Background(), ` + SELECT outcome, method, release_group_mbid, detail FROM library_album_matches WHERE album_id = $1`, album, + ).Scan(&outcome, &method, &group, &detail); err != nil { + t.Fatalf("no match recorded: %v", err) + } + return +} + +// The library's albums go on the list as held, once. Tags that name +// the group list it at once; a release id is named through MusicBrainz; +// what nothing matches is recorded with why. Taken off the list, an album +// stays off, and one added to the library afterwards is never listed. +func TestListLibrary_OnceAndThenTheListIsTheTruth_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + s := newService(t, pool, newFakeMB(t)) + s.releaseGroup = func(_ context.Context, release string) (string, error) { + if release == "rel-ballad" { + return theBallad, nil + } + return "", tags.ErrNotFound + } + blurAlbum := seedHeld(t, q, "Blur", "rel-blur", "rg-blur", 3) + ballad := seedAlbum(t, q, blur, "The Ballad of Darren", "rel-ballad", []string{"The Ballad", "St. Charles Square"}) + bootleg := seedAlbum(t, q, blur, "Bootleg", "rel-bootleg", []string{"Intro"}) + // An album with a track gone: held from before the loss, so + // re-acquisition still fetches it back. + lost := seedHeld(t, q, "13", "rel-13", "rg-13", 2) + if _, err := pool.Exec(ctx, `UPDATE tracks SET missing_since = now() - interval '1 day' + WHERE album_id = $1 AND track_number = 1`, lost.ID); err != nil { + t.Fatal(err) + } + // An album row with no tracks is not an album in the library. + if _, err := q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: "Empty", SortTitle: "Empty", ArtistID: blurAlbum.ArtistID}); err != nil { + t.Fatal(err) + } + startListing(t, pool) + + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + for group, album := range map[string]pgtype.UUID{"rg-blur": blurAlbum.ID, theBallad: ballad.ID, "rg-13": lost.ID} { + w, err := q.GetWantByReleaseGroup(ctx, group) + if err != nil { + t.Fatalf("%s not listed: %v", group, err) + } + if w.Reason != ReasonLibrary || !w.FulfilledAt.Valid || w.AlbumID != album || w.ArtistMbid != blur { + t.Errorf("%s = %+v: a held library want about the album", group, w) + } + } + if w, _ := q.GetWantByReleaseGroup(ctx, "rg-13"); !w.FulfilledAt.Time.Before(time.Now().Add(-23 * time.Hour)) { + t.Errorf("an album missing a track is held from %v, not from before the loss", w.FulfilledAt.Time) + } + if outcome, method, _, _ := matchOf(t, pool, ballad.ID); outcome != "listed" || method != matchRelease { + t.Errorf("ballad decided %s by %s", outcome, method) + } + if outcome, _, _, detail := matchOf(t, pool, bootleg.ID); outcome != "unmatched" || !strings.Contains(detail, `no album called "Bootleg"`) { + t.Errorf("bootleg decided %s: %q", outcome, detail) + } + state, err := q.GetLibraryListing(ctx) + if err != nil || !state.LibraryListedAt.Valid || state.LibraryUnlisted != 1 { + t.Fatalf("listing = %+v %v: done, with only the bootleg unmatched", state, err) + } + + // Off the list stays off; an album added later is never listed. + w, _ := q.GetWantByReleaseGroup(ctx, "rg-blur") + if _, err := s.RemoveWant(ctx, w.ID); err != nil { + t.Fatal(err) + } + _ = seedHeld(t, q, "Think Tank", "rel-think", "rg-think", 2) + if _, err := s.Pass(ctx); err != nil { + t.Fatal(err) + } + for _, group := range []string{"rg-blur", "rg-think"} { + if _, err := q.GetWantByReleaseGroup(ctx, group); err == nil { + t.Errorf("%s is on the list", group) + } + } +} + +// An album whose tags name nothing is matched by its title among the +// artist's release groups, and taken only when its tracks agree: a partial +// copy agrees on the tracks it has. One whose title matches but whose +// tracks do not is left unmatched, with why. +func TestListLibrary_MatchesByDiscographyWhenTheTracksAgree_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + s := newService(t, pool, newFakeMB(t)) + // MusicBrainz switched off for tags does not stop matching by name. + s.releaseGroup = func(context.Context, string) (string, error) { return "", tags.ErrSwitchedOff } + partial := seedAlbum(t, q, blur, "The Ballad of Darren (Deluxe)", "rel-ballad", []string{"St Charles Square"}, 2) + wrong := seedAlbum(t, q, blur, "Parklife", "", []string{"Girls & Boys", "Tracy Jacks"}) + startListing(t, pool) + + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + if outcome, method, group, _ := matchOf(t, pool, partial.ID); outcome != "listed" || method != matchDiscography || group != theBallad { + t.Errorf("partial copy decided %s by %s as %s", outcome, method, group) + } + var named *string + if err := pool.QueryRow(ctx, `SELECT release_group_mbid FROM albums WHERE id = $1`, partial.ID).Scan(&named); err != nil || named == nil || *named != theBallad { + t.Errorf("the album's release group = %v %v", named, err) + } + if outcome, _, _, detail := matchOf(t, pool, wrong.ID); outcome != "unmatched" || !strings.Contains(detail, "none has its tracks") { + t.Errorf("Parklife without Parklife's tracks decided %s: %q", outcome, detail) + } + if _, err := q.GetWantByReleaseGroup(ctx, parklife); err == nil { + t.Error("Parklife listed on its title alone") + } +} + +// An artist the tags give no id is found by MusicBrainz's search on artist +// and title, and the want is filed under the artist MusicBrainz credits. +func TestListLibrary_MatchesBySearchWhenTheArtistHasNoID_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + s := newService(t, pool, newFakeMB(t)) + album := seedAlbum(t, q, "", "The Ballad of Darren", "", []string{"The Ballad", "St. Charles Square"}) + startListing(t, pool) + + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + if outcome, method, _, _ := matchOf(t, pool, album.ID); outcome != "listed" || method != matchSearch { + t.Errorf("decided %s by %s", outcome, method) + } + w, err := q.GetWantByReleaseGroup(ctx, theBallad) + if err != nil || w.ArtistMbid != blur || w.Reason != ReasonLibrary || !w.FulfilledAt.Valid { + t.Errorf("want = %+v %v", w, err) + } +} + +// A MusicBrainz that does not answer pauses the matching; the album is not +// decided on half the evidence, and the next pass goes on. +func TestListLibrary_UnreachablePausesTheMatching_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + f := newFakeMB(t) + s := newService(t, pool, f) + album := seedAlbum(t, q, blur, "The Ballad of Darren", "", []string{"The Ballad", "St. Charles Square"}) + startListing(t, pool) + + f.down.Store(true) + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + if state, _ := q.GetLibraryListing(ctx); state.LibraryListedAt.Valid { + t.Fatal("finished while an album was still to be matched") + } + var n int + if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_album_matches WHERE album_id = $1`, album.ID).Scan(&n); err != nil || n != 0 { + t.Fatalf("decided while MusicBrainz was away: %d %v", n, err) + } + f.down.Store(false) + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + if _, err := q.GetWantByReleaseGroup(ctx, theBallad); err != nil { + t.Errorf("not listed once MusicBrainz answered: %v", err) + } + if state, _ := q.GetLibraryListing(ctx); !state.LibraryListedAt.Valid { + t.Error("not finished") + } +} + +func TestTrackAgreement(t *testing.T) { + edition := []editionTrack{ + {Medium: 1, Position: 1, Title: "Intro", LengthMs: 60000, Recording: "rec-1"}, + {Medium: 1, Position: 2, Title: "Song", LengthMs: 200000}, + {Medium: 2, Position: 1, Title: "Encore", LengthMs: 300000}, + } + for _, c := range []struct { + name string + tracks []libraryTrack + want int + }{ + {"all, by position", []libraryTrack{ + {Disc: 1, Number: 1, Title: "Intro", LengthMs: 60000}, {Number: 2, Title: "Song", LengthMs: 201000}, + {Disc: 2, Number: 1, Title: "Encore", LengthMs: 300000}}, 3}, + {"a partial copy", []libraryTrack{{Disc: 2, Number: 1, Title: "Encore", LengthMs: 300500}}, 1}, + {"unnumbered, by title", []libraryTrack{{Title: "Song", LengthMs: 200000}, {Title: "Encore", LengthMs: 299000}}, 2}, + {"the recording, whatever its title", []libraryTrack{{Number: 1, Title: "Untitled", LengthMs: 1, Recording: "REC-1"}}, 1}, + {"the right title at the wrong length", []libraryTrack{{Number: 2, Title: "Song", LengthMs: 400000}}, 0}, + {"a track counted once", []libraryTrack{{Title: "Song", LengthMs: 200000}, {Title: "Song", LengthMs: 200000}}, 1}, + } { + if got, _ := trackAgreement(c.tracks, edition, false); got != c.want { + t.Errorf("%s: %d agree, want %d", c.name, got, c.want) + } + } + // Under the album's own name the titles are enough; the reason a track + // did not agree names both sides. + edit := []libraryTrack{{Number: 2, Title: "Song (Radio Edit)", LengthMs: 150000}} + if n, miss := trackAgreement(edit, edition, false); n != 0 || miss != `"Song (Radio Edit)" runs 2:30 here; MusicBrainz's track 2 is "Song", 3:20` { + t.Errorf("by length: %d, %q", n, miss) + } + if n, _ := trackAgreement(edit, edition, true); n != 1 { + t.Errorf("by title alone: %d agree", n) + } + if _, miss := trackAgreement([]libraryTrack{{Title: "Other", LengthMs: 61000}}, edition, true); miss != `no track of its edition is like "Other" (1:01)` { + t.Errorf("unplaced: %q", miss) + } + if !sameTitle("Brain Stew + Jaded", "Brain Stew / Jaded") || sameTitle("+++", "Grind & Hustle") || sameTitle("Blur", "The Best of Blur") { + t.Error("sameTitle: joining words count for nothing, and a title of none is no title") + } + n := int32(1) + copies := tracksToMatch([]dbq.ListAlbumTracksToMatchRow{ + {Title: "Song", TrackNumber: &n, DurationMs: 1}, {Title: "song", TrackNumber: &n, DurationMs: 2}, {Title: "Other", TrackNumber: &n}}) + if len(copies) != 2 { + t.Errorf("a second copy of a track is read once: %+v", copies) + } + if agrees(3, 4, 0) || !agrees(4, 5, 0) || agrees(0, 0, 0) { + t.Error("agrees: 80% of the album's tracks, and an album has tracks") + } + if !agrees(23, 43, 23) || agrees(2, 10, 2) || agrees(22, 43, 23) { + t.Error("agrees: a whole edition of three or more tracks, with more besides") + } +} + +// A refusal by an older matcher is matched again once the matching improves, +// even after the listing is done; a refusal by this one is not. +func TestListLibrary_RetriesAnOlderMatchersRefusals_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + s := newService(t, pool, newFakeMB(t)) + ballad := seedAlbum(t, q, blur, "The Ballad of Darren", "", []string{"The Ballad", "St. Charles Square"}) + bootleg := seedAlbum(t, q, blur, "Bootleg", "", []string{"Intro"}) + for _, a := range []pgtype.UUID{ballad.ID, bootleg.ID} { + if err := q.RecordAlbumMatch(ctx, dbq.RecordAlbumMatchParams{AlbumID: a, Outcome: "unmatched", Detail: "old", Matcher: 1}); err != nil { + t.Fatal(err) + } + } + if _, err := pool.Exec(ctx, `UPDATE library_album_matches SET matcher = $1 WHERE album_id = $2`, libraryMatcher, bootleg.ID); err != nil { + t.Fatal(err) + } + // A listing done, with these albums from before its cutoff. + if _, err := pool.Exec(ctx, `UPDATE acquisition_settings SET library_listing_started_at = now(), library_listed_at = now()`); err != nil { + t.Fatal(err) + } + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + if outcome, method, _, _ := matchOf(t, pool, ballad.ID); outcome != "listed" || method != matchDiscography { + t.Errorf("the older refusal was decided %s by %s", outcome, method) + } + if _, _, _, detail := matchOf(t, pool, bootleg.ID); detail != "old" { + t.Errorf("this matcher's own refusal was retried: %q", detail) + } + if state, _ := q.GetLibraryListing(ctx); state.LibraryUnlisted != 1 { + t.Errorf("unlisted = %d, want the bootleg alone", state.LibraryUnlisted) + } +} diff --git a/internal/wanted/manual.go b/internal/wanted/manual.go new file mode 100644 index 00000000..a58e8d45 --- /dev/null +++ b/internal/wanted/manual.go @@ -0,0 +1,170 @@ +package wanted + +import ( + "context" + "errors" + "fmt" + "regexp" + "strings" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" + "git.fabledsword.com/bvandeusen/minstrel/internal/tags" +) + +// Matching a library album by hand: the Collection's Unmatched section. A +// person names the MusicBrainz album an album is, by a link or a search, and +// it goes on the list as held; or leaves it off for good, as a local band's +// album MusicBrainz will never have. + +const matchManual = "manual" + +// Manual-match errors, which the API answers as 400, 404 or 409. +var ( + ErrMatchRef = errors.New("wanted: not a MusicBrainz album link or id") + ErrMatchNoGroup = errors.New("wanted: MusicBrainz has no such album") + ErrAlbumNotUnmatched = errors.New("wanted: no such unmatched album") +) + +// ManualMatch is what matching an album by hand did: the group it is now, +// and how many of its tracks agree with the group's closest edition, so the +// person can see whether the link was the right one. +type ManualMatch struct { + Group string + Title string + Artist string + Agree int + Tracks int +} + +var mbidPattern = regexp.MustCompile(`(?i)(?:(release-group|release)/)?([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})`) + +// ParseMusicBrainzRef reads a MusicBrainz link (a release group's or a +// release's page) or a bare id. Kind is "release-group", "release", or "" +// for a bare id, which is tried as a group first. +func ParseMusicBrainzRef(ref string) (kind, id string, ok bool) { + m := mbidPattern.FindStringSubmatch(strings.TrimSpace(ref)) + if m == nil { + return "", "", false + } + return strings.ToLower(m[1]), strings.ToLower(m[2]), true +} + +// MatchAlbumTo puts an unmatched (or left-off) library album on the list as +// held, as the MusicBrainz album ref names. The person's word is taken: the +// tracks are compared only to say how well they agree. +func (s *Service) MatchAlbumTo(ctx context.Context, albumID pgtype.UUID, ref string) (ManualMatch, error) { + kind, id, ok := ParseMusicBrainzRef(ref) + if !ok { + return ManualMatch{}, ErrMatchRef + } + q := dbq.New(s.pool) + a, err := q.GetAlbumToMatch(ctx, albumID) + if errors.Is(err, pgx.ErrNoRows) { + return ManualMatch{}, ErrAlbumNotUnmatched + } + if err != nil { + return ManualMatch{}, fmt.Errorf("wanted: read album: %w", err) + } + group, err := s.groupOf(ctx, kind, id) + if err != nil { + return ManualMatch{}, err + } + g, err := s.cat.MusicBrainz().ReleaseGroup(ctx, group) + if errors.Is(err, musicbrainz.ErrNotFound) { + return ManualMatch{}, ErrMatchNoGroup + } + if err != nil { + return ManualMatch{}, err + } + artist := creditedArtist(g.ArtistMBIDs, deref(a.ArtistMbid)) + if artist == "" { + return ManualMatch{}, fmt.Errorf("%w: it credits no artist", ErrMatchNoGroup) + } + rows, err := q.ListAlbumTracksToMatch(ctx, a.ID) + if err != nil { + return ManualMatch{}, fmt.Errorf("wanted: read %s's tracks: %w", a.Title, err) + } + tracks := tracksToMatch(rows) + n, _, _, err := s.agreementWith(ctx, group, tracks, func(string) bool { return true }) + if err != nil { + return ManualMatch{}, err + } + err = pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + tq := dbq.New(tx) + if err := tq.SetAlbumReleaseGroup(ctx, dbq.SetAlbumReleaseGroupParams{ID: a.ID, ReleaseGroupMbid: &group}); err != nil { + return err + } + if _, err := tq.InsertLibraryWant(ctx, dbq.InsertLibraryWantParams{ + ReleaseGroupMbid: group, ArtistMbid: artist, ArtistName: g.ArtistCredit, Title: g.Title, + PrimaryType: g.PrimaryType, FirstReleaseDate: g.FirstReleaseDate, AlbumID: a.ID, + }); err != nil { + return err + } + if err := tq.RecordAlbumMatch(ctx, dbq.RecordAlbumMatchParams{ + AlbumID: a.ID, Outcome: "listed", Method: matchManual, ReleaseGroupMbid: group, Matcher: libraryMatcher, + }); err != nil { + return err + } + return tq.RecountLibraryUnlisted(ctx) + }) + if err != nil { + return ManualMatch{}, fmt.Errorf("wanted: match %s by hand: %w", a.Title, err) + } + s.logger.Info("wanted: a library album matched by hand", "album", a.Title, "group", group, "agree", max(n, 0), "tracks", len(tracks)) + return ManualMatch{Group: group, Title: g.Title, Artist: g.ArtistCredit, Agree: max(n, 0), Tracks: len(tracks)}, nil +} + +// groupOf names the release group a MusicBrainz id is, or belongs to. +func (s *Service) groupOf(ctx context.Context, kind, id string) (string, error) { + if kind != "release" { + _, err := s.cat.MusicBrainz().ReleaseGroup(ctx, id) + if err == nil { + return id, nil + } + if !errors.Is(err, musicbrainz.ErrNotFound) { + return "", err + } + if kind == "release-group" { + return "", ErrMatchNoGroup + } + } + group, err := s.releaseGroup(ctx, id) + switch { + case errors.Is(err, tags.ErrNotFound): + return "", ErrMatchNoGroup + case err != nil: + return "", err + case group == "": + return "", ErrMatchNoGroup + } + return group, nil +} + +// LeaveAlbumOff leaves an unmatched library album off the list for good: +// matching never tries it again, and it is no longer counted as unmatched. +func (s *Service) LeaveAlbumOff(ctx context.Context, albumID pgtype.UUID) error { + return s.setMatchOutcome(ctx, albumID, "unmatched", "dismissed") +} + +// PutAlbumBack undoes LeaveAlbumOff: the album is unmatched again. +func (s *Service) PutAlbumBack(ctx context.Context, albumID pgtype.UUID) error { + return s.setMatchOutcome(ctx, albumID, "dismissed", "unmatched") +} + +func (s *Service) setMatchOutcome(ctx context.Context, albumID pgtype.UUID, was, outcome string) error { + return pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + tq := dbq.New(tx) + n, err := tq.SetAlbumMatchOutcome(ctx, dbq.SetAlbumMatchOutcomeParams{AlbumID: albumID, Was: was, Outcome: outcome}) + if err != nil { + return err + } + if n == 0 { + return ErrAlbumNotUnmatched + } + return tq.RecountLibraryUnlisted(ctx) + }) +} diff --git a/internal/wanted/manual_test.go b/internal/wanted/manual_test.go new file mode 100644 index 00000000..c76fe0d1 --- /dev/null +++ b/internal/wanted/manual_test.go @@ -0,0 +1,166 @@ +package wanted + +import ( + "context" + "errors" + "testing" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/tags" +) + +func TestParseMusicBrainzRef(t *testing.T) { + const id = "11111111-0000-4000-8000-000000000003" + for ref, want := range map[string]string{ + "https://musicbrainz.org/release-group/" + id: "release-group " + id, + "https://musicbrainz.org/release/" + id + "?tab=x": "release " + id, + " " + id + " ": " " + id, + "https://MusicBrainz.org/Release-Group/" + "11111111-0000-4000-8000-00000000000A": "release-group 11111111-0000-4000-8000-00000000000a", + } { + kind, got, ok := ParseMusicBrainzRef(ref) + if !ok || kind+" "+got != want { + t.Errorf("ParseMusicBrainzRef(%q) = %q %q %v, want %q", ref, kind, got, ok, want) + } + } + for _, bad := range []string{"", "Dial Tone", "https://musicbrainz.org/artist/nope"} { + if _, _, ok := ParseMusicBrainzRef(bad); ok { + t.Errorf("ParseMusicBrainzRef(%q) read an id", bad) + } + } +} + +// A person names the MusicBrainz album an unmatched album is, and it goes +// on the list as held; or leaves it off, and matching never tries it again. +func TestMatchAlbumTo_ByLinkOrLeaveItOff_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + s := newService(t, pool, newFakeMB(t)) + s.releaseGroup = func(context.Context, string) (string, error) { return "", tags.ErrNotFound } + split := seedAlbum(t, q, blur, "Blur + Friends", "", []string{"The Ballad", "St. Charles Square"}) + local := seedAlbum(t, q, blur, "Dial Tone", "", []string{"Intro"}) + for _, a := range []pgtype.UUID{split.ID, local.ID} { + if err := q.RecordAlbumMatch(ctx, dbq.RecordAlbumMatchParams{ + AlbumID: a, Outcome: "unmatched", Detail: "no match", Matcher: libraryMatcher}); err != nil { + t.Fatal(err) + } + } + if _, err := pool.Exec(ctx, `UPDATE acquisition_settings + SET library_listing_started_at = now(), library_listed_at = now(), library_unlisted = 2`); err != nil { + t.Fatal(err) + } + + for ref, want := range map[string]error{ + "not a link": ErrMatchRef, + "https://musicbrainz.org/release-group/99999999-0000-4000-8000-000000000000": ErrMatchNoGroup, + "99999999-0000-4000-8000-000000000000": ErrMatchNoGroup, + } { + if _, err := s.MatchAlbumTo(ctx, split.ID, ref); !errors.Is(err, want) { + t.Errorf("MatchAlbumTo(%q) = %v, want %v", ref, err, want) + } + } + + m, err := s.MatchAlbumTo(ctx, split.ID, "https://musicbrainz.org/release-group/"+theBallad) + if err != nil { + t.Fatal(err) + } + if m.Group != theBallad || m.Title != "The Ballad of Darren" || m.Agree != 2 || m.Tracks != 2 { + t.Errorf("match = %+v", m) + } + w, err := q.GetWantByReleaseGroup(ctx, theBallad) + if err != nil || w.Reason != ReasonLibrary || !w.FulfilledAt.Valid || w.AlbumID != split.ID || w.ArtistMbid != blur { + t.Fatalf("want = %+v, %v: held, about the album", w, err) + } + if outcome, method, group, _ := matchOf(t, pool, split.ID); outcome != "listed" || method != matchManual || group != theBallad { + t.Errorf("decided %s by %s as %s", outcome, method, group) + } + if state, _ := q.GetLibraryListing(ctx); state.LibraryUnlisted != 1 { + t.Errorf("unlisted = %d, want the local album alone", state.LibraryUnlisted) + } + + // The local band's album: left off, it is not counted, and a newer + // matcher does not try it again. + if err := s.LeaveAlbumOff(ctx, local.ID); err != nil { + t.Fatal(err) + } + if outcome, _, _, _ := matchOf(t, pool, local.ID); outcome != "dismissed" { + t.Errorf("left off = %s", outcome) + } + if state, _ := q.GetLibraryListing(ctx); state.LibraryUnlisted != 0 { + t.Errorf("unlisted = %d after leaving it off", state.LibraryUnlisted) + } + if _, err := pool.Exec(ctx, `UPDATE library_album_matches SET matcher = 0 WHERE album_id = $1`, local.ID); err != nil { + t.Fatal(err) + } + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + if outcome, _, _, _ := matchOf(t, pool, local.ID); outcome != "dismissed" { + t.Errorf("an album left off was matched again: %s", outcome) + } + rows, err := q.ListUnmatchedLibraryAlbums(ctx, 10) + if err != nil || len(rows) != 1 || rows[0].Outcome != "dismissed" || rows[0].Tracks != 1 { + t.Errorf("unmatched section = %+v, %v: the left-off album alone", rows, err) + } + + // Put back, it is unmatched again; putting it back twice is not. + if err := s.PutAlbumBack(ctx, local.ID); err != nil { + t.Fatal(err) + } + if outcome, _, _, _ := matchOf(t, pool, local.ID); outcome != "unmatched" { + t.Errorf("put back = %s", outcome) + } + if err := s.PutAlbumBack(ctx, local.ID); !errors.Is(err, ErrAlbumNotUnmatched) { + t.Errorf("put back twice: %v", err) + } +} + +// A re-tag files a library track under a new album row. That album is the +// library's, not one dropped in by hand, so the listing matches it; the +// old row, its tracks gone, is no longer unmatched. +func TestListLibrary_MatchesARetaggedAlbum_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + q := dbq.New(pool) + s := newService(t, pool, newFakeMB(t)) + untagged := seedAlbum(t, q, blur, "Unknown Album", "", []string{"The Ballad", "St. Charles Square"}) + if err := q.RecordAlbumMatch(ctx, dbq.RecordAlbumMatchParams{ + AlbumID: untagged.ID, Outcome: "unmatched", Detail: "no match", Matcher: libraryMatcher}); err != nil { + t.Fatal(err) + } + for _, stmt := range []string{ + `UPDATE albums SET created_at = now() - interval '2 hours' WHERE id = $1`, + `UPDATE tracks SET added_at = now() - interval '2 hours' WHERE album_id = $1`, + } { + if _, err := pool.Exec(ctx, stmt, untagged.ID); err != nil { + t.Fatal(err) + } + } + if _, err := pool.Exec(ctx, `UPDATE acquisition_settings SET library_listing_started_at = now() - interval '1 hour', + library_listed_at = now() - interval '1 hour', library_unlisted = 1`); err != nil { + t.Fatal(err) + } + // The re-tag: the same tracks, under the album their new tags name. + retagged, err := q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: "The Ballad of Darren", SortTitle: "The Ballad of Darren", ArtistID: untagged.ArtistID}) + if err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, `UPDATE tracks SET album_id = $1, updated_at = now() WHERE album_id = $2`, retagged.ID, untagged.ID); err != nil { + t.Fatal(err) + } + + if err := s.listLibrary(ctx, &PassResult{}); err != nil { + t.Fatal(err) + } + if outcome, method, group, _ := matchOf(t, pool, retagged.ID); outcome != "listed" || method != matchDiscography || group != theBallad { + t.Errorf("the re-tagged album was decided %s by %s as %s", outcome, method, group) + } + if rows, _ := q.ListUnmatchedLibraryAlbums(ctx, 10); len(rows) != 0 { + t.Errorf("unmatched = %+v: the old row has no tracks left", rows) + } + if state, _ := q.GetLibraryListing(ctx); state.LibraryUnlisted != 0 { + t.Errorf("unlisted = %d", state.LibraryUnlisted) + } +} diff --git a/internal/wanted/match.go b/internal/wanted/match.go new file mode 100644 index 00000000..c1fe3fa8 --- /dev/null +++ b/internal/wanted/match.go @@ -0,0 +1,404 @@ +package wanted + +import ( + "context" + "errors" + "fmt" + "slices" + "strings" + "time" + + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" + "git.fabledsword.com/bvandeusen/minstrel/internal/tags" + "git.fabledsword.com/bvandeusen/minstrel/internal/words" +) + +// Matching a library album to its MusicBrainz release group. +// +// Music from any mainstream source was matched against MusicBrainz on its way +// in, so nearly every album has a group. The ids its tags carry are taken as +// they are. Without them, the groups called what the album is called, from the +// artist's discography and then from a search, are checked against the album's +// own tracks, and the one its tracks agree with is taken. + +// Match methods, as library_album_matches records them. +const ( + matchTags = "tags" + matchRelease = "release" + matchDiscography = "discography" + matchSearch = "search" +) + +// matchAgreement is the share of an album's tracks, in tenths, an edition +// must agree with: most, not all, since a rip's track may carry a typo or a +// hidden track's length. +const matchAgreement = 8 + +// matchCandidates bounds how many groups one album is checked against: each +// costs a MusicBrainz read the first time. +const matchCandidates = 8 + +// matchDiscographyAge is how old a held discography may be and still be +// read for matching: a name does not change, and a new release is not in +// the library. +const matchDiscographyAge = 90 * 24 * time.Hour + +// errMusicBrainzAway: MusicBrainz did not answer, so the album is left for +// the next pass rather than decided on half the evidence. +var errMusicBrainzAway = errors.New("wanted: MusicBrainz did not answer") + +// libraryTrack is what one of an album's tracks says about itself. +type libraryTrack struct { + Disc, Number int + Title string + LengthMs int + Recording string +} + +// editionTrack is one track of a MusicBrainz edition. +type editionTrack struct { + Medium, Position int + Title string + LengthMs int + Recording string +} + +// candidate is a release group an album may be. +type candidate struct { + Group string + Title string + PrimaryType string + FirstReleaseDate string + ArtistMBID string + ArtistName string + Method string +} + +// albumMatch is what matching decided: a group, or why there is none. +type albumMatch struct { + candidate + OK bool + Detail string +} + +// tracksToMatch reads an album's tracks, each once: a second copy of a +// track (the same disc, number and title, as a duplicate file in another +// format is) would otherwise count against the match as a track no edition +// has. +func tracksToMatch(rows []dbq.ListAlbumTracksToMatchRow) []libraryTrack { + out := make([]libraryTrack, 0, len(rows)) + seen := make(map[string]bool, len(rows)) + for _, r := range rows { + t := libraryTrack{Title: r.Title, LengthMs: int(r.DurationMs), Recording: deref(r.Mbid)} + if r.DiscNumber != nil { + t.Disc = int(*r.DiscNumber) + } + if r.TrackNumber != nil { + t.Number = int(*r.TrackNumber) + } + key := fmt.Sprintf("%d/%d/%s", max(t.Disc, 1), t.Number, strings.Join(release.Tokens(t.Title), " ")) + if seen[key] { + continue + } + seen[key] = true + out = append(out, t) + } + return out +} + +// trackAgreement counts the album's tracks an edition agrees with: the same +// recording, or a title that agrees at a length that fits. When the group is +// called just what the album is called, the titles alone are enough: the +// lengths are there to tell apart albums whose names only overlap, and a +// rip's single may be the edit MusicBrainz lists at another length. A track +// is paired by its disc and number first, then by title, so an album missing +// some tracks still pairs the ones it has. miss says how the first track +// that did not agree differs, for the reason a match was refused. +func trackAgreement(tracks []libraryTrack, edition []editionTrack, titlesEnough bool) (n int, miss string) { + byPos := make(map[[2]int]int, len(edition)) + for i, e := range edition { + byPos[[2]int{max(e.Medium, 1), e.Position}] = i + } + agree := func(t libraryTrack, e editionTrack) bool { + if t.Recording != "" && strings.EqualFold(t.Recording, e.Recording) { + return true + } + return release.TitleAgrees(t.Title, e.Title) && (titlesEnough || release.LengthFits(t.LengthMs, e.LengthMs)) + } + used := make([]bool, len(edition)) + for _, t := range tracks { + at, placed := -1, false + if t.Number > 0 { + if i, ok := byPos[[2]int{max(t.Disc, 1), t.Number}]; ok { + at = i + placed = !used[i] && agree(t, edition[i]) + if placed { + used[i] = true + } + } + } + for i, e := range edition { + if !placed && !used[i] && agree(t, e) { + used[i], placed = true, true + } + } + switch { + case placed: + n++ + case miss != "": + case at >= 0: + e := edition[at] + miss = fmt.Sprintf("%q runs %s here; MusicBrainz's track %d is %q, %s", + t.Title, minutes(t.LengthMs), e.Position, e.Title, minutes(e.LengthMs)) + default: + miss = fmt.Sprintf("no track of its edition is like %q (%s)", t.Title, minutes(t.LengthMs)) + } + } + return n, miss +} + +// minutes writes a length as m:ss, or "no length". +func minutes(ms int) string { + if ms <= 0 { + return "no length" + } + sec := (ms + 500) / 1000 + return fmt.Sprintf("%d:%02d", sec/60, sec%60) +} + +// agrees: enough of the album's tracks agree for the edition to be it, or +// the album holds every track of the edition and more besides (a folder +// with the deluxe edition and its demos). A single or a two-track release +// is not taken on containment: too many albums hold its songs. +func agrees(n, tracks, edition int) bool { + return tracks > 0 && (n*10 >= tracks*matchAgreement || (edition >= 3 && n == edition)) +} + +// sameTitle: the same words, not merely one holding the other. "And", +// "&" and "+" join two titles however a release spells it ("Brain Stew + +// Jaded", "Brain Stew / Jaded"), so they count for nothing. +func sameTitle(a, b string) bool { + words := func(s string) []string { + return slices.DeleteFunc(release.Tokens(s), func(w string) bool { return w == "and" }) + } + wa, wb := words(a), words(b) + return len(wa) > 0 && slices.Equal(wa, wb) +} + +// matchAlbum finds the release group a library album is. A MusicBrainz +// that does not answer is errMusicBrainzAway; anything else MusicBrainz +// says is evidence, and an album with none is an albumMatch that is not OK. +func (s *Service) matchAlbum(ctx context.Context, a dbq.ListAlbumsToMatchRow, tracks []libraryTrack) (albumMatch, error) { + // The ids the tags carry name it. + if group := deref(a.ReleaseGroupMbid); group != "" { + if c, err := s.groupCandidate(ctx, group, matchTags, a); err != nil || c.Group != "" { + return albumMatch{candidate: c, OK: err == nil}, err + } + } + if rel := deref(a.Mbid); rel != "" { + group, err := s.releaseGroup(ctx, rel) + switch { + case errors.Is(err, tags.ErrNotFound): // includes MusicBrainz switched off for tags + case err != nil: + return albumMatch{}, fmt.Errorf("%w: %v", errMusicBrainzAway, err) + default: + if c, err := s.groupCandidate(ctx, group, matchRelease, a); err != nil || c.Group != "" { + return albumMatch{candidate: c, OK: err == nil}, err + } + } + } + + // Otherwise its name, checked against its tracks: the artist's own + // discography first, then a search. + tried := map[string]bool{} + var best albumMatch + bestN, bestEdition, considered, bestMiss := -1, 0, 0, "" + check := func(cands []candidate) error { + for _, c := range cands { + if tried[c.Group] || len(tried) >= matchCandidates { + continue + } + tried[c.Group] = true + n, size, miss, err := s.agreementWith(ctx, c.Group, tracks, func(edition string) bool { + return sameTitle(c.Title, a.Title) || sameTitle(edition, a.Title) + }) + if err != nil { + return err + } + considered++ + // More tracks agreeing wins; then the same title over a longer + // one; then the edition nearest the album's length. + st, bt := sameTitle(c.Title, a.Title), sameTitle(best.Title, a.Title) + if n > bestN || (n == bestN && ((st && !bt) || (st == bt && abs(size-len(tracks)) < abs(bestEdition-len(tracks))))) { + best, bestN, bestEdition, bestMiss = albumMatch{candidate: c}, n, size, miss + } + } + return nil + } + discography, err := s.discographyCandidates(ctx, a) + if err != nil { + return albumMatch{}, err + } + if err := check(discography); err != nil { + return albumMatch{}, err + } + if !agrees(bestN, len(tracks), bestEdition) { + found, err := s.searchCandidates(ctx, a) + if err != nil { + return albumMatch{}, err + } + if err := check(found); err != nil { + return albumMatch{}, err + } + } + if agrees(bestN, len(tracks), bestEdition) { + best.OK = true + return best, nil + } + if considered == 0 { + return albumMatch{Detail: fmt.Sprintf("MusicBrainz has no album called %q by %s.", a.Title, a.ArtistName)}, nil + } + detail := fmt.Sprintf("MusicBrainz has %s called %q by %s, but none has its tracks: the closest, %q, agrees on %d of its %d", + words.Count(considered, "album", "albums"), a.Title, a.ArtistName, best.Title, max(bestN, 0), len(tracks)) + switch { + case bestN < 0: + detail += " (MusicBrainz lists no edition of it)" + case bestMiss != "": + detail += ": " + bestMiss + } + return albumMatch{Detail: detail + "."}, nil +} + +// groupCandidate reads a group the tags name. A group MusicBrainz does not +// have is a zero candidate, so the album is matched by name instead. +func (s *Service) groupCandidate(ctx context.Context, group, method string, a dbq.ListAlbumsToMatchRow) (candidate, error) { + g, err := s.cat.MusicBrainz().ReleaseGroup(ctx, group) + switch { + case errors.Is(err, musicbrainz.ErrNotFound): + return candidate{}, nil + case err != nil: + return candidate{}, fmt.Errorf("%w: %v", errMusicBrainzAway, err) + } + artist := creditedArtist(g.ArtistMBIDs, deref(a.ArtistMbid)) + if artist == "" { + return candidate{}, nil + } + return candidate{Group: g.MBID, Title: g.Title, PrimaryType: g.PrimaryType, FirstReleaseDate: g.FirstReleaseDate, + ArtistMBID: artist, ArtistName: g.ArtistCredit, Method: method}, nil +} + +// discographyCandidates are the artist's release groups called what the +// album is called. +func (s *Service) discographyCandidates(ctx context.Context, a dbq.ListAlbumsToMatchRow) ([]candidate, error) { + artist := deref(a.ArtistMbid) + if !Followable(artist, a.ArtistName) { + return nil, nil // no artist, or Various Artists: thousands of groups + } + d, err := s.cat.DiscographyWithin(ctx, artist, matchDiscographyAge) + switch { + case errors.Is(err, catalogue.ErrNotFound): + return nil, nil + case err != nil: + return nil, fmt.Errorf("%w: %v", errMusicBrainzAway, err) + } + var out []candidate + for _, g := range d.Groups { + if release.TitleAgrees(g.Title, a.Title) { + out = append(out, candidate{Group: g.Mbid, Title: g.Title, PrimaryType: g.PrimaryType, + FirstReleaseDate: g.FirstReleaseDate, ArtistMBID: artist, ArtistName: d.Artist.Name, Method: matchDiscography}) + } + } + sortSameTitleFirst(out, a.Title) + return out, nil +} + +// searchCandidates are the groups MusicBrainz's search finds by the album's +// artist and title, whose title and artist both agree. +func (s *Service) searchCandidates(ctx context.Context, a dbq.ListAlbumsToMatchRow) ([]candidate, error) { + found, err := s.cat.MusicBrainz().SearchReleaseGroupsBy(ctx, a.ArtistName, a.Title, 10) + switch { + case errors.Is(err, musicbrainz.ErrNotFound): + return nil, nil + case errors.Is(err, musicbrainz.ErrBadResponse): + // A query MusicBrainz will not read is not a reason to wait. + s.logger.Warn("wanted: MusicBrainz refused a search", "album", a.Title, "artist", a.ArtistName, "err", err) + return nil, nil + case err != nil: + return nil, fmt.Errorf("%w: %v", errMusicBrainzAway, err) + } + var out []candidate + for _, g := range found { + artist := creditedArtist(g.ArtistMBIDs, deref(a.ArtistMbid)) + if artist == "" || !release.TitleAgrees(g.Title, a.Title) { + continue + } + if !slices.Contains(g.ArtistMBIDs, deref(a.ArtistMbid)) && !release.TitleAgrees(g.ArtistCredit, a.ArtistName) { + continue + } + out = append(out, candidate{Group: g.MBID, Title: g.Title, PrimaryType: g.PrimaryType, + FirstReleaseDate: g.FirstReleaseDate, ArtistMBID: artist, ArtistName: g.ArtistCredit, Method: matchSearch}) + } + sortSameTitleFirst(out, a.Title) + return out, nil +} + +// agreementWith is how many of the album's tracks the group's best edition +// agrees with, that edition's length, and how it differs. A group +// MusicBrainz no longer has, or lists no edition of, agrees with none (-1). +// ownName says whether an edition, by its title, is called what the album +// is, so that its track titles are enough. +func (s *Service) agreementWith(ctx context.Context, group string, tracks []libraryTrack, ownName func(edition string) bool) (n, size int, miss string, err error) { + eds, err := s.cat.Releases(ctx, group) + switch { + case errors.Is(err, catalogue.ErrNotFound): + return -1, 0, "", nil + case err != nil: + return 0, 0, "", fmt.Errorf("%w: %v", errMusicBrainzAway, err) + } + n = -1 + for _, e := range eds.Editions { + edition := make([]editionTrack, 0, len(e.Tracks)) + for _, t := range e.Tracks { + edition = append(edition, editionTrack{Medium: int(t.MediumPosition), Position: int(t.Position), + Title: t.Title, LengthMs: int(t.LengthMs), Recording: t.RecordingMbid}) + } + if got, m := trackAgreement(tracks, edition, ownName(e.Release.Title)); got > n || (got == n && abs(len(edition)-len(tracks)) < abs(size-len(tracks))) { + n, size, miss = got, len(edition), m + } + } + return n, size, miss, nil +} + +// creditedArtist is the artist a want is filed under: the library's own +// artist when MusicBrainz credits them, else the first credited. +func creditedArtist(credited []string, own string) string { + if own != "" && (len(credited) == 0 || slices.Contains(credited, own)) { + return own + } + if len(credited) > 0 { + return credited[0] + } + return "" +} + +func sortSameTitleFirst(cs []candidate, title string) { + slices.SortStableFunc(cs, func(x, y candidate) int { + switch sx, sy := sameTitle(x.Title, title), sameTitle(y.Title, title); { + case sx && !sy: + return -1 + case sy && !sx: + return 1 + } + return 0 + }) +} + +func abs(n int) int { + if n < 0 { + return -n + } + return n +} diff --git a/internal/wanted/profiles.go b/internal/wanted/profiles.go new file mode 100644 index 00000000..d1e86764 --- /dev/null +++ b/internal/wanted/profiles.go @@ -0,0 +1,228 @@ +package wanted + +import ( + "context" + "errors" + "fmt" + "slices" + "strings" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgconn" + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// PrimaryTypes and SecondaryTypes are MusicBrainz's release-group types, in +// the order the profile editor lists them. +var ( + PrimaryTypes = []string{"Album", "EP", "Single", "Broadcast", "Other"} + SecondaryTypes = []string{ + "Compilation", "Live", "Remix", "Soundtrack", "DJ-mix", "Mixtape/Street", + "Demo", "Spokenword", "Interview", "Audiobook", "Audio drama", "Field recording", + } +) + +// Profile errors, which the API answers as 400 or 409. +var ( + ErrProfileInvalid = errors.New("wanted: profile is not valid") + ErrProfileNotFound = errors.New("wanted: no such profile") + ErrProfileInUse = errors.New("wanted: profile is in use") + ErrProfileDefault = errors.New("wanted: the default profile cannot be deleted") + ErrProfileNameUsed = errors.New("wanted: a profile already has that name") +) + +// Passes reports whether a release group of the given types is one a follow +// on profile p wants: its primary type is listed, and every secondary type +// it carries is allowed. A group with no secondary type is a studio release, +// which every profile passes on its primary type alone. +// +// A version its title names counts as a secondary type MusicBrainz left +// off: "Common People (Recorded For BBC Radio 1)" is live whatever +// its types say, and a version no type names, an acoustic one, no profile +// passes. +func Passes(p dbq.ReleaseTypeProfile, title, primary string, secondary []string) bool { + if !containsFold(p.PrimaryTypes, primary) { + return false + } + for _, s := range secondary { + if !containsFold(p.SecondaryTypes, s) { + return false + } + } + if v, ok := release.VersionInTitle(title); ok { + return v.Typ != "" && containsFold(p.SecondaryTypes, v.Typ) + } + return true +} + +func containsFold(list []string, s string) bool { + return slices.ContainsFunc(list, func(v string) bool { return strings.EqualFold(v, s) }) +} + +// ProfileInput is a profile as the editor sends it. +type ProfileInput struct { + Name string + PrimaryTypes []string + SecondaryTypes []string + IsDefault bool +} + +// normalize checks the input against MusicBrainz's types and returns it with +// each type in MusicBrainz's spelling and the editor's order. +func (in ProfileInput) normalize() (ProfileInput, error) { + in.Name = strings.TrimSpace(in.Name) + if in.Name == "" || len(in.Name) > 60 { + return in, fmt.Errorf("%w: a name of 1 to 60 characters is required", ErrProfileInvalid) + } + primary, err := pick(PrimaryTypes, in.PrimaryTypes) + if err != nil { + return in, err + } + if len(primary) == 0 { + return in, fmt.Errorf("%w: choose at least one kind of release", ErrProfileInvalid) + } + secondary, err := pick(SecondaryTypes, in.SecondaryTypes) + if err != nil { + return in, err + } + in.PrimaryTypes, in.SecondaryTypes = primary, secondary + return in, nil +} + +func pick(known, chosen []string) ([]string, error) { + for _, c := range chosen { + if !containsFold(known, c) { + return nil, fmt.Errorf("%w: %q is not a MusicBrainz release type", ErrProfileInvalid, c) + } + } + out := []string{} + for _, k := range known { + if containsFold(chosen, k) { + out = append(out, k) + } + } + return out, nil +} + +// CreateProfile adds a profile, making it the default when asked. +func (s *Service) CreateProfile(ctx context.Context, in ProfileInput) (dbq.ReleaseTypeProfile, error) { + in, err := in.normalize() + if err != nil { + return dbq.ReleaseTypeProfile{}, err + } + var out dbq.ReleaseTypeProfile + err = pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + p, err := q.CreateReleaseTypeProfile(ctx, dbq.CreateReleaseTypeProfileParams{ + Name: in.Name, PrimaryTypes: in.PrimaryTypes, SecondaryTypes: in.SecondaryTypes, + }) + if err != nil { + return nameErr(err) + } + if in.IsDefault { + if err := makeDefault(ctx, q, p.ID); err != nil { + return err + } + p.IsDefault = true + } + out = p + return nil + }) + return out, err +} + +// UpdateProfile replaces a profile's name and types. IsDefault only ever +// moves the default here; to unset it, make another profile the default. +func (s *Service) UpdateProfile(ctx context.Context, id pgtype.UUID, in ProfileInput) (dbq.ReleaseTypeProfile, error) { + in, err := in.normalize() + if err != nil { + return dbq.ReleaseTypeProfile{}, err + } + var out dbq.ReleaseTypeProfile + err = pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + q := dbq.New(tx) + p, err := q.UpdateReleaseTypeProfile(ctx, dbq.UpdateReleaseTypeProfileParams{ + ID: id, Name: in.Name, PrimaryTypes: in.PrimaryTypes, SecondaryTypes: in.SecondaryTypes, + }) + if errors.Is(err, pgx.ErrNoRows) { + return ErrProfileNotFound + } + if err != nil { + return nameErr(err) + } + if in.IsDefault && !p.IsDefault { + if err := makeDefault(ctx, q, p.ID); err != nil { + return err + } + p.IsDefault = true + } + out = p + return nil + }) + return out, err +} + +func makeDefault(ctx context.Context, q *dbq.Queries, id pgtype.UUID) error { + if err := q.ClearDefaultReleaseTypeProfile(ctx); err != nil { + return fmt.Errorf("wanted: clear default: %w", err) + } + if _, err := q.SetDefaultReleaseTypeProfile(ctx, id); err != nil { + return fmt.Errorf("wanted: set default: %w", err) + } + return nil +} + +// DeleteProfile removes a profile no follow uses. The default stays. +func (s *Service) DeleteProfile(ctx context.Context, id pgtype.UUID) error { + q := dbq.New(s.pool) + p, err := q.GetReleaseTypeProfile(ctx, id) + if errors.Is(err, pgx.ErrNoRows) { + return ErrProfileNotFound + } + if err != nil { + return fmt.Errorf("wanted: read profile: %w", err) + } + if p.IsDefault { + return ErrProfileDefault + } + if _, err := q.DeleteReleaseTypeProfile(ctx, id); err != nil { + var pgErr *pgconn.PgError + if errors.As(err, &pgErr) && pgErr.Code == "23503" { + return ErrProfileInUse + } + return fmt.Errorf("wanted: delete profile: %w", err) + } + return nil +} + +// profileOrDefault reads the profile a follow names; the zero id means the +// default. +func profileOrDefault(ctx context.Context, q *dbq.Queries, id pgtype.UUID) (dbq.ReleaseTypeProfile, error) { + var ( + p dbq.ReleaseTypeProfile + err error + ) + if id.Valid { + p, err = q.GetReleaseTypeProfile(ctx, id) + } else { + p, err = q.GetDefaultReleaseTypeProfile(ctx) + } + if errors.Is(err, pgx.ErrNoRows) { + return p, ErrProfileNotFound + } + if err != nil { + return p, fmt.Errorf("wanted: read profile: %w", err) + } + return p, nil +} + +func nameErr(err error) error { + var pgErr *pgconn.PgError + if errors.As(err, &pgErr) && pgErr.Code == "23505" { + return ErrProfileNameUsed + } + return fmt.Errorf("wanted: save profile: %w", err) +} diff --git a/internal/wanted/search.go b/internal/wanted/search.go new file mode 100644 index 00000000..5c8f0102 --- /dev/null +++ b/internal/wanted/search.go @@ -0,0 +1,360 @@ +package wanted + +import ( + "context" + "encoding/json" + "errors" + "log/slog" + "strconv" + "time" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/indexer" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// Searching for wants on a schedule. The schedule decides which wants and when; +// internal/acquire judges what a search finds and grabs. + +// Search outcomes, matching wants_search_outcome_check. +const ( + OutcomeGrabbed = "grabbed" + OutcomeWaiting = "waiting" + OutcomeNothing = "nothing" + OutcomeError = "error" +) + +const ( + // searchTick is how often the schedule looks for wants due; the + // search interval setting is how long a want waits between searches. + searchTick = 10 * time.Minute + // searchWindow is how long one pass searches: most of a tick, so a pass + // ends before the next begins. + searchWindow = 9 * time.Minute + // duePage bounds what one pass reads. The sources' pace is the real + // limit; this caps a pass whose sources are all unpaced. + duePage = 200 + // grabTries is how many grabbable releases one search tries in turn + // when the first is refused for its files. + grabTries = 3 + // passTimeout backs up the window for a call none of the clients' + // own timeouts cover, such as a wedged database read. + passTimeout = 15 * time.Minute +) + +// ErrWantHeld: the want is already answered by the library, and is not +// wanted for a better copy. +var ErrWantHeld = errors.New("wanted: the library already holds this album") + +// Searcher searches for wants and grabs what they need. +type Searcher struct { + pool *pgxpool.Pool + cat *catalogue.Service + logger *slog.Logger +} + +// NewSearcher returns a Searcher reading editions through cat (nil reads +// only what the catalogue already holds). +func NewSearcher(pool *pgxpool.Pool, cat *catalogue.Service, logger *slog.Logger) *Searcher { + if logger == nil { + logger = slog.Default() + } + return &Searcher{pool: pool, cat: cat, logger: logger} +} + +// Report is one pass. +type Report struct { + Looked int `json:"looked"` + Grabbed int `json:"grabbed"` + Waiting int `json:"waiting"` + Nothing int `json:"nothing"` + Failed int `json:"failed"` + // Paced: the pass stopped because the next search would wait past its + // window for a source's pace. + Paced bool `json:"paced"` + // Off: the schedule is switched off. Unconfigured: nothing to search + // with. Both are states, not faults. + Off bool `json:"off"` + Unconfigured bool `json:"unconfigured"` + AutoGrab bool `json:"auto_grab"` + Error string `json:"error,omitempty"` +} + +// Run searches every searchTick until ctx ends. +func (s *Searcher) Run(ctx context.Context) { + ticker := time.NewTicker(searchTick) + defer ticker.Stop() + said := "" + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + rep := s.Pass(ctx) + // A pass that searched is always logged; an idle one only when the + // schedule's state changes, so the log says once why it is quiet. + if state := rep.state(); rep.Looked > 0 || rep.Error != "" || state != said { + s.logger.Info("wanted: search pass", "state", state, "looked", rep.Looked, + "grabbed", rep.Grabbed, "waiting", rep.Waiting, "nothing", rep.Nothing, + "failed", rep.Failed, "paced", rep.Paced, "error", rep.Error) + said = state + } + } +} + +// state names why a pass did what it did: off, unconfigured, failed, +// idle (nothing was due) or searched. +func (r Report) state() string { + switch { + case r.Off: + return "off" + case r.Unconfigured: + return "unconfigured" + case r.Error != "": + return "failed" + case r.Looked == 0: + return "idle" + } + return "searched" +} + +// Pass searches for the wants due, paced by the sources, until the window +// closes, and records what it did. +func (s *Searcher) Pass(ctx context.Context) Report { + rep := s.pass(ctx) + if body, err := json.Marshal(rep); err == nil { + if err := dbq.New(s.pool).RecordSearchPass(ctx, body); err != nil && ctx.Err() == nil { + s.logger.Warn("wanted: record search pass", "err", err) + } + } + return rep +} + +func (s *Searcher) pass(ctx context.Context) Report { + var rep Report + ctx, cancel := context.WithTimeout(ctx, passTimeout) + defer cancel() + svc, settings, err := acquire.Build(ctx, s.pool) + if err != nil { + rep.Error = indexer.FirstLine(err.Error()) + return rep + } + rep.AutoGrab = settings.AutoGrab + switch { + case settings.SearchIntervalHours <= 0: + rep.Off = true + return rep + case svc.Search == nil: + rep.Unconfigured = true + return rep + } + deadline := time.Now().Add(searchWindow) + due, err := dbq.New(s.pool).ListWantsDueForSearch(ctx, dbq.ListWantsDueForSearchParams{ + Hours: settings.SearchIntervalHours, MaxRows: duePage, + }) + if err != nil { + rep.Error = indexer.FirstLine(err.Error()) + return rep + } + for _, row := range due { + if !svc.Ready(ctx, deadline) { + rep.Paced = ctx.Err() == nil + return rep + } + want := s.wantFor(ctx, searchRow{id: row.ID, artist: row.ArtistName, title: row.Title, + primary: row.PrimaryType, secondary: row.SecondaryTypes, date: row.FirstReleaseDate, + group: row.ReleaseGroupMbid, upgrade: row.Upgrade, heldTier: row.HeldTier, held: row.HeldQuality}) + out := s.one(ctx, svc, want, settings.AutoGrab) + rep.Looked++ + switch out.Outcome { + case OutcomeGrabbed: + rep.Grabbed++ + case OutcomeWaiting: + rep.Waiting++ + case OutcomeNothing: + rep.Nothing++ + default: + rep.Failed++ + s.logger.Warn("wanted: search failed", "want", want.Artist+" - "+want.Title, "err", out.Error) + } + } + return rep +} + +// searchRow is what a search needs of a wants row. +type searchRow struct { + id pgtype.UUID + artist, title, primary, date, group string + secondary []string + // upgrade, heldTier and held: a held album wanted for a better copy + // and the copy it has. + upgrade, heldTier, held string +} + +// wantFor describes a want for the matcher, with its editions' track counts +// from the catalogue (read from MusicBrainz the first time when it can be). +// A held album wanted for better quality takes its copy's tier as a floor. +func (s *Searcher) wantFor(ctx context.Context, r searchRow) acquire.Want { + w := acquire.Want{ID: r.id, Want: release.Want{ + Artist: r.artist, Title: r.title, Year: yearOf(r.date), PrimaryType: r.primary, SecondaryTypes: r.secondary, + }} + if r.upgrade == UpgradeQuality && release.KnownTier(r.heldTier) { + w.Floor, w.Held = release.Tier(r.heldTier), r.held + } + q := dbq.New(s.pool) + span, err := q.EditionTrackSpan(ctx, r.group) + if err == nil && span.MinTracks == 0 && s.cat != nil { + if _, rerr := s.cat.Releases(ctx, r.group); rerr == nil { + span, err = q.EditionTrackSpan(ctx, r.group) + } + } + if err == nil { + w.MinTracks, w.MaxTracks = int(span.MinTracks), int(span.MaxTracks) + } + return w +} + +func yearOf(date string) int { + if len(date) < 4 { + return 0 + } + y, _ := strconv.Atoi(date[:4]) + return y +} + +// Searched is what one want's search came to. +type Searched struct { + Outcome string `json:"outcome"` + Considered int `json:"considered"` + Best string `json:"best_release"` + Verdict string `json:"best_verdict"` + Grabbed string `json:"grabbed,omitempty"` + Narrowed []acquire.Narrowing `json:"narrowed"` + Error string `json:"error,omitempty"` +} + +// one searches for a want, takes the best release when auto-grab is on, and +// records what came of it on the want. +func (s *Searcher) one(ctx context.Context, svc *acquire.Service, want acquire.Want, autoGrab bool) Searched { + found := svc.For(ctx, want) + if found.KeepError != "" { + s.logger.Warn("wanted: searched, but could not keep the candidates", "want", want.Title, "err", found.KeepError) + } + out := Searched{Considered: found.Considered, Narrowed: found.Narrowed} + if out.Narrowed == nil { + out.Narrowed = []acquire.Narrowing{} + } + if len(found.Candidates) > 0 { + out.Best, out.Verdict = found.Candidates[0].Name, string(found.Candidates[0].Verdict) + } + switch { + case found.Unconfigured: + out.Outcome, out.Error = OutcomeError, "No indexer is set up, so there is nothing to search with." + case found.Error != "": + out.Outcome, out.Error = OutcomeError, found.Error + case !found.ForAPerson(): + out.Outcome = OutcomeNothing + case len(found.Grabbable()) == 0 || !autoGrab: + out.Outcome = OutcomeWaiting + default: + out.Outcome = OutcomeWaiting + s.grabBest(ctx, svc, want, found, &out) + } + s.record(ctx, want.ID, out) + return out +} + +// grabBest takes the best grabbable release, trying the next when one is +// refused for what its files hold. +func (s *Searcher) grabBest(ctx context.Context, svc *acquire.Service, want acquire.Want, found acquire.Result, out *Searched) { + for i, c := range found.Grabbable() { + if i == grabTries { + return + } + took := svc.Grab(ctx, want, &c, found.Candidates, false, acquire.Origin{By: acquire.OriginSchedule}) + switch { + case took.Added: + out.Outcome, out.Grabbed, out.Best, out.Verdict = OutcomeGrabbed, c.Name, c.Name, string(acquire.Grab) + if took.Error != "" { + s.logger.Warn("wanted: took a release but could not record it", "want", want.Title, "err", took.Error) + } + s.logger.Info("wanted: took a release", "want", want.Title, "artist", want.Artist, + "release", c.Name, "quality", c.Tier, "save_path", took.SavePath) + return + case took.Refused: + continue + case took.Unconfigured: + out.Outcome, out.Error = OutcomeError, "No download client is set up, so nothing was taken." + return + case took.Contest != nil: + return + case took.Error != "": + out.Outcome, out.Error = OutcomeError, took.Error + return + } + } +} + +func (s *Searcher) record(ctx context.Context, id pgtype.UUID, out Searched) { + if err := dbq.New(s.pool).RecordWantSearch(ctx, dbq.RecordWantSearchParams{ + ID: id, Outcome: out.Outcome, Considered: int32(out.Considered), + BestRelease: out.Best, BestVerdict: out.Verdict, SearchError: out.Error, + }); err != nil && ctx.Err() == nil { + s.logger.Warn("wanted: record search", "err", err) + } +} + +// SearchNow searches for one want at once, as the schedule would, for a +// person pressing Search: it spends the sources' pace but never waits on it. +func (s *Searcher) SearchNow(ctx context.Context, id pgtype.UUID) (Searched, error) { + want, err := s.load(ctx, id) + if err != nil { + return Searched{}, err + } + svc, settings, err := acquire.Build(ctx, s.pool) + if err != nil { + return Searched{}, err + } + return s.one(ctx, svc, want, settings.AutoGrab), nil +} + +// Take grabs the release named by key for a person, searching again first. +func (s *Searcher) Take(ctx context.Context, id pgtype.UUID, key string, force bool, by pgtype.UUID) (acquire.Taken, error) { + want, err := s.load(ctx, id) + if err != nil { + return acquire.Taken{}, err + } + svc, _, err := acquire.Build(ctx, s.pool) + if err != nil { + return acquire.Taken{}, err + } + took := svc.Take(ctx, want, key, force, acquire.Origin{By: acquire.OriginOperator, User: by}) + if took.Added && took.Chosen != nil { + s.record(ctx, id, Searched{ + Outcome: OutcomeGrabbed, Best: took.Chosen.Name, Verdict: string(took.Chosen.Verdict), + Considered: 0, Grabbed: took.Chosen.Name, + }) + } + return took, nil +} + +// load reads a want for a search: an open one, or a held album wanted for a +// better copy. +func (s *Searcher) load(ctx context.Context, id pgtype.UUID) (acquire.Want, error) { + row, err := dbq.New(s.pool).GetWantForSearch(ctx, id) + if err != nil { + return acquire.Want{}, err + } + if row.FulfilledAt.Valid && row.Upgrade == "" { + return acquire.Want{}, ErrWantHeld + } + return s.wantFor(ctx, searchRow{id: row.ID, artist: row.ArtistName, title: row.Title, + primary: row.PrimaryType, secondary: row.SecondaryTypes, date: row.FirstReleaseDate, + group: row.ReleaseGroupMbid, upgrade: row.Upgrade, heldTier: row.HeldTier, held: row.HeldQuality}), nil +} diff --git a/internal/wanted/search_test.go b/internal/wanted/search_test.go new file mode 100644 index 00000000..407dbd22 --- /dev/null +++ b/internal/wanted/search_test.go @@ -0,0 +1,213 @@ +package wanted + +import ( + "context" + "encoding/json" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + + "git.fabledsword.com/bvandeusen/minstrel/internal/acquire" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" +) + +const ramHash = "0123456789abcdef0123456789abcdef01234567" + +const ramFeed = ` + +Daft Punk - Random Access Memories (2013) [FLAC] + magnet:?xt=urn:btih:` + ramHash + `400000000 + +Daft Punk Random Access Memories (Drumless Edition 2023) [Flac 24-88] + magnet:?xt=urn:btih:fedcba9876543210fedcba9876543210fedcba98900000000 + +` + +// searchStack is a fake indexer feed and a fake FabledExchange download +// client, configured as an admin would in Admin › Acquisition. +func searchStack(t *testing.T, ctx context.Context, store *acquire.Store) *atomic.Int32 { + t.Helper() + var added atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.URL.Path == "/feed": + if q := r.URL.Query().Get("q"); q != "Daft Punk Random Access Memories" { + t.Errorf("searched for %q", q) + } + _, _ = io.WriteString(w, ramFeed) + case r.URL.Path == "/api/downloads" && r.Method == http.MethodPost: + var body struct { + Magnet string `json:"magnet"` + SavePath string `json:"save_path"` + } + _ = json.NewDecoder(r.Body).Decode(&body) + if !strings.Contains(body.Magnet, ramHash) || body.SavePath != "/downloads/"+ramHash { + t.Errorf("added %+v", body) + } + added.Add(1) + _, _ = io.WriteString(w, "{}") + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + + feed, err := store.Create(ctx, acquire.ProtocolTorznab) + if err != nil { + t.Fatal(err) + } + if _, err := store.Save(ctx, feed.Code, acquire.Update{ + Label: "Feed", Enabled: true, BaseURL: srv.URL + "/feed", TimeoutMs: 5000, LocalPath: "/downloads", + }); err != nil { + t.Fatal(err) + } + ex, err := store.Create(ctx, acquire.ProtocolExchange) + if err != nil { + t.Fatal(err) + } + if _, err := store.Save(ctx, ex.Code, acquire.Update{ + Label: "Exchange", Enabled: true, BaseURL: srv.URL, TimeoutMs: 5000, ServesDownloads: true, LocalPath: "/downloads", + }); err != nil { + t.Fatal(err) + } + return &added +} + +func TestSearcher_FindsAndTakesAWant_Integration(t *testing.T) { + ctx := context.Background() + pool := newPool(t) + q := dbq.New(pool) + added := searchStack(t, ctx, acquire.NewStore(pool)) + w, err := WantWith(ctx, q, WantParams{ + GroupMBID: "rg-ram", ArtistMBID: "mb-daft-punk", ArtistName: "Daft Punk", Title: "Random Access Memories", + PrimaryType: "Album", FirstReleaseDate: "2013-05-17", Reason: ReasonRequest, + }) + if err != nil { + t.Fatal(err) + } + s := NewSearcher(pool, nil, slog.New(slog.NewTextHandler(io.Discard, nil))) + + rep := s.Pass(ctx) + if rep.Looked != 1 || rep.Grabbed != 1 || !rep.AutoGrab || added.Load() != 1 { + t.Fatalf("pass = %+v, added %d", rep, added.Load()) + } + got, err := q.GetWantByReleaseGroup(ctx, "rg-ram") + if err != nil { + t.Fatal(err) + } + if got.SearchOutcome != OutcomeGrabbed || got.SearchConsidered != 2 || !strings.Contains(got.BestRelease, "(2013) [FLAC]") { + t.Errorf("want after the pass: %s %d %q", got.SearchOutcome, got.SearchConsidered, got.BestRelease) + } + cands, err := q.ListWantCandidates(ctx, w.ID) + if err != nil { + t.Fatal(err) + } + if len(cands) != 2 || cands[0].Verdict != "grab" || cands[1].Verdict != "rejected" || !strings.Contains(cands[1].Reason, "drumless") { + t.Errorf("candidates = %+v", cands) + } + g, err := q.GetLiveGrab(ctx, w.ID) + if err != nil { + t.Fatal(err) + } + if g.InfoHash != ramHash || g.Client != "exchange" || g.Origin != acquire.OriginSchedule || g.Quality != "lossless" { + t.Errorf("grab = %+v", g) + } + settings, _ := q.GetAcquisitionSettings(ctx) + var last Report + if err := json.Unmarshal(settings.LastPass, &last); err != nil || !settings.LastPassAt.Valid || last.Grabbed != 1 { + t.Errorf("last pass = %s (%v)", settings.LastPass, err) + } + + // A want with a download running is not searched again. + if rep := s.Pass(ctx); rep.Looked != 0 { + t.Errorf("searched a want already downloading: %+v", rep) + } + + // Setting the drumless edition aside marks it on every kept candidate; + // allowing it again leaves it for a person until the next search. + mem := acquire.NewMemory(pool) + if err := mem.Block(ctx, dbq.BlockReleaseParams{ + InfoHash: cands[1].InfoHash, Name: cands[1].Name, Reason: acquire.BlockOperator, WantID: w.ID, + }); err != nil { + t.Fatal(err) + } + c, _ := q.GetWantCandidate(ctx, dbq.GetWantCandidateParams{WantID: w.ID, Key: cands[1].Key}) + if c.Verdict != "blocked" { + t.Errorf("blocked candidate = %s", c.Verdict) + } + if ok, err := mem.Unblock(ctx, cands[1].InfoHash); err != nil || !ok { + t.Fatalf("unblock: %v %v", ok, err) + } + c, _ = q.GetWantCandidate(ctx, dbq.GetWantCandidateParams{WantID: w.ID, Key: cands[1].Key}) + if c.Verdict != "review" { + t.Errorf("unblocked candidate = %s", c.Verdict) + } +} + +func TestSearcher_OffAndUnconfigured_Integration(t *testing.T) { + ctx := context.Background() + pool := newPool(t) + s := NewSearcher(pool, nil, slog.New(slog.NewTextHandler(io.Discard, nil))) + if rep := s.Pass(ctx); !rep.Unconfigured { + t.Errorf("no indexer: %+v", rep) + } + if _, err := dbq.New(pool).UpdateAcquisitionSettings(ctx, dbq.UpdateAcquisitionSettingsParams{ + AutoGrab: true, SearchIntervalHours: 0, AllowedQualities: []string{"lossless"}, + }); err != nil { + t.Fatal(err) + } + if rep := s.Pass(ctx); !rep.Off { + t.Errorf("interval 0: %+v", rep) + } +} + +func TestGrabs_OneLivePerWant_Integration(t *testing.T) { + ctx := context.Background() + pool := newPool(t) + q := dbq.New(pool) + w, err := WantWith(ctx, q, WantParams{GroupMBID: "rg-x", ArtistMBID: "mb-x", ArtistName: "X", Title: "Y", Reason: ReasonFollow}) + if err != nil { + t.Fatal(err) + } + mem := acquire.NewMemory(pool) + grab := func(hash string) dbq.InsertGrabParams { + return dbq.InsertGrabParams{WantID: w.ID, InfoHash: hash, Name: hash, Key: hash, Client: "qbittorrent", + SavePath: "/downloads/" + hash, Origin: acquire.OriginSchedule} + } + first, err := mem.Grabbed(ctx, grab("aaaa"), dbq.Grab{}.ID, "") + if err != nil { + t.Fatal(err) + } + if _, err := mem.Grabbed(ctx, grab("bbbb"), dbq.Grab{}.ID, ""); err == nil { + t.Error("a second live grab for one want was recorded") + } + second, err := mem.Grabbed(ctx, grab("bbbb"), first.ID, "Replaced by bbbb.") + if err != nil { + t.Fatalf("replacing: %v", err) + } + live, ok, err := mem.LiveGrab(ctx, w.ID) + if err != nil || !ok || live.ID != second.ID { + t.Errorf("live = %+v %v %v", live, ok, err) + } +} + +func TestReport_StateNamesWhyAPassWasQuiet(t *testing.T) { + for _, c := range []struct { + rep Report + want string + }{ + {Report{Off: true}, "off"}, + {Report{Unconfigured: true}, "unconfigured"}, + {Report{Error: "boom"}, "failed"}, + {Report{}, "idle"}, + {Report{Looked: 2, Grabbed: 1}, "searched"}, + } { + if got := c.rep.state(); got != c.want { + t.Errorf("%+v: state = %q, want %q", c.rep, got, c.want) + } + } +} diff --git a/internal/wanted/upgrade.go b/internal/wanted/upgrade.go new file mode 100644 index 00000000..5b620286 --- /dev/null +++ b/internal/wanted/upgrade.go @@ -0,0 +1,125 @@ +package wanted + +import ( + "context" + "fmt" + "time" + + "github.com/jackc/pgx/v5/pgtype" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +// Upgrades: a held album stays wanted when the copy the library has is below +// the quality the install settles for (acquisition_settings. +// upgrade_until), or when one of its tracks is a video rip. The search +// schedule then looks for a better release weekly, and the import replaces +// the old tracks with the new ones, carrying their history across. + +// Why a held album is wanted again, matching wants_upgrade_check. +const ( + UpgradeQuality = "quality" + UpgradeRip = "rip" +) + +// upgradeCheckEvery is how often held albums are judged. Their files change +// only by an import or a scan, and the check reads every held album. +const upgradeCheckEvery = time.Hour + +// UpgradeFor says why a held album is wanted again, if it is: a video rip +// among its tracks always (a rip is no copy of the album, whatever the +// setting), else a copy below until. until "" switches quality upgrades off. +func UpgradeFor(held release.Held, judged bool, rips int, until release.Tier) string { + switch { + case rips > 0: + return UpgradeRip + case until == "" || !judged: + return "" + case release.Rank(held.Tier) < release.Rank(until): + return UpgradeQuality + } + return "" +} + +// heldVerdict is one held want as the check judged it. +type heldVerdict struct { + id pgtype.UUID + upgrade, tier, words string + oldUpgrade, oldTier, old string +} + +func (v heldVerdict) changed() bool { + return v.upgrade != v.oldUpgrade || v.tier != v.oldTier || v.words != v.old +} + +// checkUpgrades judges every held album, at most once an hour, and records +// which are wanted for a better copy. +func (s *Service) checkUpgrades(ctx context.Context, res *PassResult) error { + if !s.upgradesChecked.IsZero() && s.now().Sub(s.upgradesChecked) < upgradeCheckEvery { + return nil + } + q := dbq.New(s.pool) + settings, err := q.GetAcquisitionSettings(ctx) + if err != nil { + return fmt.Errorf("read settings: %w", err) + } + rows, err := q.ListHeldWantFormats(ctx) + if err != nil { + return fmt.Errorf("read held albums: %w", err) + } + until := release.Tier(settings.UpgradeUntil) + var out []heldVerdict + flush := func(id pgtype.UUID, group []dbq.ListHeldWantFormatsRow) { + if len(group) == 0 { + return + } + formats := make([]release.HeldFormat, 0, len(group)) + rips := 0 + for _, r := range group { + formats = append(formats, release.HeldFormat{Format: r.FileFormat, Bytes: r.Bytes, Ms: r.DurationMs}) + rips += int(r.Rips) + } + held, judged := release.HeldCopy(formats) + v := heldVerdict{id: id, upgrade: UpgradeFor(held, judged, rips, until), + oldUpgrade: group[0].Upgrade, oldTier: group[0].HeldTier, old: group[0].HeldQuality} + if judged { + v.tier, v.words = string(held.Tier), held.Words + } + if v.upgrade != "" { + res.Upgrades++ + } + if v.changed() { + out = append(out, v) + } + } + var group []dbq.ListHeldWantFormatsRow + for _, r := range rows { + if len(group) > 0 && group[0].WantID != r.WantID { + flush(group[0].WantID, group) + group = group[:0] + } + group = append(group, r) + } + if len(group) > 0 { + flush(group[0].WantID, group) + } + + if len(out) > 0 { + p := dbq.SetWantUpgradesParams{} + for _, v := range out { + p.Ids = append(p.Ids, v.id) + p.Upgrades = append(p.Upgrades, v.upgrade) + p.HeldTiers = append(p.HeldTiers, v.tier) + p.HeldQualities = append(p.HeldQualities, v.words) + } + if err := q.SetWantUpgrades(ctx, p); err != nil { + return fmt.Errorf("record upgrades: %w", err) + } + } + if _, err := q.ClearHeldUpgrades(ctx); err != nil { + return fmt.Errorf("clear upgrades: %w", err) + } + s.upgradesChecked = s.now() + return nil +} diff --git a/internal/wanted/upgrade_test.go b/internal/wanted/upgrade_test.go new file mode 100644 index 00000000..d9837bba --- /dev/null +++ b/internal/wanted/upgrade_test.go @@ -0,0 +1,173 @@ +package wanted + +import ( + "context" + "fmt" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" +) + +func TestUpgradeFor(t *testing.T) { + low := release.Held{Tier: release.TierLossyLow} + flac := release.Held{Tier: release.TierLossless} + high := release.Held{Tier: release.TierLossyHigh} + cases := []struct { + name string + held release.Held + judged bool + rips int + until release.Tier + want string + }{ + {"below the cutoff", low, true, 0, release.TierLossyHigh, UpgradeQuality}, + {"at the cutoff", high, true, 0, release.TierLossyHigh, ""}, + {"above the cutoff", flac, true, 0, release.TierLossyHigh, ""}, + {"upgrades off", low, true, 0, "", ""}, + {"a rip, upgrades off", flac, true, 1, "", UpgradeRip}, + {"nothing to judge", release.Held{}, false, 0, release.TierLossless, ""}, + } + for _, c := range cases { + if got := UpgradeFor(c.held, c.judged, c.rips, c.until); got != c.want { + t.Errorf("%s: %q, want %q", c.name, got, c.want) + } + } +} + +// seedMP3s puts Blur's The Ballad of Darren in the library as MP3s at about +// kbps, two tracks, the first held back as a video rip when rip is set. +func seedMP3s(t *testing.T, pool *pgxpool.Pool, kbps int64, rip bool) dbq.Album { + t.Helper() + ctx := context.Background() + q := dbq.New(pool) + mbid, group, rel := blur, theBallad, "r-ballad-mp3" + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "Blur", SortName: "Blur", Mbid: &mbid}) + if err != nil { + t.Fatal(err) + } + album, err := q.UpsertAlbum(ctx, dbq.UpsertAlbumParams{Title: "The Ballad of Darren", SortTitle: "Ballad of Darren", + ArtistID: artist.ID, Mbid: &rel, ReleaseGroupMbid: &group}) + if err != nil { + t.Fatal(err) + } + for i := range 2 { + num := int32(i + 1) + tr, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: fmt.Sprintf("Track %d", i+1), AlbumID: album.ID, ArtistID: artist.ID, TrackNumber: &num, + DurationMs: 200000, FilePath: fmt.Sprintf("/music/ballad/%02d.mp3", i+1), + FileSize: kbps * 200000 / 8, FileFormat: "mp3", + }) + if err != nil { + t.Fatal(err) + } + if rip && i == 0 { + if _, err := pool.Exec(ctx, `UPDATE tracks SET source_verdict = 'suspect' WHERE id = $1`, tr.ID); err != nil { + t.Fatal(err) + } + } + } + return album +} + +func TestCheckUpgrades_AHeldAlbumBelowTheCutoffIsWantedAgain_Integration(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + ctx := context.Background() + s := newService(t, pool, newFakeMB(t)) + + if _, err := WantWith(ctx, q, WantParams{ + GroupMBID: theBallad, ArtistMBID: blur, ArtistName: "Blur", Title: "The Ballad of Darren", Reason: ReasonRequest, + }); err != nil { + t.Fatal(err) + } + seedMP3s(t, pool, 192, false) + res, err := s.Pass(ctx) + if err != nil || res.Held != 1 || res.Upgrades != 1 { + t.Fatalf("pass: %+v, %v", res, err) + } + w, _ := q.GetWantByReleaseGroup(ctx, theBallad) + if w.Upgrade != UpgradeQuality || w.HeldTier != string(release.TierLossyLow) || w.HeldQuality != "MP3, about 192 kbps" { + t.Fatalf("want = upgrade %q, tier %q, %q", w.Upgrade, w.HeldTier, w.HeldQuality) + } + due, err := q.ListWantsDueForSearch(ctx, dbq.ListWantsDueForSearchParams{Hours: 24, MaxRows: 10}) + if err != nil || len(due) != 1 || due[0].Upgrade != UpgradeQuality { + t.Fatalf("due for search: %+v, %v", due, err) + } + + // Upgrades switched off: the album is not wanted again. + if _, err := pool.Exec(ctx, `UPDATE acquisition_settings SET upgrade_until = ''`); err != nil { + t.Fatal(err) + } + s.upgradesChecked = time.Time{} + if _, err := s.Pass(ctx); err != nil { + t.Fatal(err) + } + w, _ = q.GetWantByReleaseGroup(ctx, theBallad) + if w.Upgrade != "" { + t.Errorf("upgrades off: upgrade %q", w.Upgrade) + } +} + +func TestCheckUpgrades_ARipIsReplacedWhateverTheSetting_Integration(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + ctx := context.Background() + s := newService(t, pool, newFakeMB(t)) + + if _, err := WantWith(ctx, q, WantParams{ + GroupMBID: theBallad, ArtistMBID: blur, ArtistName: "Blur", Title: "The Ballad of Darren", Reason: ReasonRequest, + }); err != nil { + t.Fatal(err) + } + seedMP3s(t, pool, 320, true) + if _, err := pool.Exec(ctx, `UPDATE acquisition_settings SET upgrade_until = ''`); err != nil { + t.Fatal(err) + } + if _, err := s.Pass(ctx); err != nil { + t.Fatal(err) + } + w, _ := q.GetWantByReleaseGroup(ctx, theBallad) + if w.Upgrade != UpgradeRip { + t.Errorf("a rip: upgrade %q", w.Upgrade) + } +} + +// A missing track with no disc number is on disc 1, so a present 1/2 stands in +// for a missing "2", and the album is not downloaded again for nothing. A +// missing track no present track covers still counts. +func TestWantHoldings_AMissingTrackWithNoDiscIsOnDiscOne_Integration(t *testing.T) { + pool := newPool(t) + ctx := context.Background() + album := seedMP3s(t, pool, 320, false) + if _, err := pool.Exec(ctx, `UPDATE tracks SET disc_number = 1 WHERE album_id = $1`, album.ID); err != nil { + t.Fatal(err) + } + q := dbq.New(pool) + for _, n := range []int32{2, 3} { + num := n + tr, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: fmt.Sprintf("Stale %d", n), AlbumID: album.ID, ArtistID: album.ArtistID, TrackNumber: &num, + DurationMs: 200000, FilePath: fmt.Sprintf("/music/ballad/stale-%d.mp3", n), FileSize: 1, FileFormat: "mp3", + }) + if err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, `UPDATE tracks SET missing_since = now() - interval '2 days' WHERE id = $1`, tr.ID); err != nil { + t.Fatal(err) + } + } + rows, err := q.WantHoldings(ctx, dbq.WantHoldingsParams{ + Since: pgtype.Timestamptz{Time: time.Now(), Valid: true}, GroupMbid: theBallad, AlbumID: album.ID, + }) + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 || rows[0].UncoveredMissing != 1 { + t.Fatalf("holdings = %+v, want one album with 1 uncovered missing track (track 3 only)", rows) + } +} diff --git a/internal/wanted/wanted.go b/internal/wanted/wanted.go new file mode 100644 index 00000000..afbbec71 --- /dev/null +++ b/internal/wanted/wanted.go @@ -0,0 +1,542 @@ +// Package wanted is what Minstrel is looking for: followed artists, each on +// a release-type profile, and wanted release groups. A want comes from an +// approved request, a followed artist's new release, or an album whose files +// went missing; it is held once the library has a full copy. The library's +// own albums are put on the list as held once, so the list is the record of +// what the library keeps. Searching for open wants is the Searcher's. +package wanted + +import ( + "context" + "errors" + "fmt" + "log/slog" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" + "git.fabledsword.com/bvandeusen/minstrel/internal/release" + "git.fabledsword.com/bvandeusen/minstrel/internal/tags" +) + +// Reasons a release group is wanted, matching wants_reason_check. +const ( + ReasonRequest = "request" + ReasonFollow = "follow" + ReasonMissing = "missing" +) + +// variousArtists is MusicBrainz's Various Artists. Its "discography" is every +// compilation ever catalogued; nobody follows it. +const variousArtists = "89ad4ac3-39f7-470e-963a-56509c546377" + +// ErrNotFollowable: an artist nobody follows (Various Artists, or one of +// MusicBrainz's bracketed placeholders such as [unknown]). +var ErrNotFollowable = errors.New("wanted: not an artist anyone follows") + +// Service owns follows, wants and the worker that keeps them current. +type Service struct { + pool *pgxpool.Pool + cat *catalogue.Service + logger *slog.Logger + now func() time.Time + // releaseGroup names the group of a MusicBrainz release id, to move a + // want recorded under a release id to its group. A field so tests + // stand in for MusicBrainz. + releaseGroup func(ctx context.Context, releaseMBID string) (string, error) + // upgradesChecked is when held albums were last judged for an upgrade + // the worker alone touches it. + upgradesChecked time.Time +} + +// New makes a Service reading discographies and editions through cat. +func New(pool *pgxpool.Pool, cat *catalogue.Service, logger *slog.Logger) *Service { + if logger == nil { + logger = slog.Default() + } + return &Service{pool: pool, cat: cat, logger: logger, now: time.Now, releaseGroup: tags.ReleaseGroupForRelease} +} + +// Followable reports whether an artist is one anyone would follow. +func Followable(mbid, name string) bool { + if mbid == "" || mbid == variousArtists { + return false + } + return len(name) < 2 || name[0] != '[' || name[len(name)-1] != ']' +} + +// FollowParams is a follow as the caller asks for it. A zero ProfileID means +// the default profile. WantExisting also wants what the artist has already +// released that the profile passes and the library lacks. +type FollowParams struct { + ArtistMBID string + ArtistName string + ProfileID pgtype.UUID + By pgtype.UUID + WantExisting bool +} + +// Follow starts or changes a follow. The worker reads the discography. +func (s *Service) Follow(ctx context.Context, p FollowParams) (dbq.Follow, error) { + return FollowWith(ctx, dbq.New(s.pool), p) +} + +// FollowWith is Follow inside the caller's transaction (request approval). +func FollowWith(ctx context.Context, q *dbq.Queries, p FollowParams) (dbq.Follow, error) { + if !Followable(p.ArtistMBID, p.ArtistName) { + return dbq.Follow{}, ErrNotFollowable + } + profile, err := profileOrDefault(ctx, q, p.ProfileID) + if err != nil { + return dbq.Follow{}, err + } + f, err := q.UpsertFollow(ctx, dbq.UpsertFollowParams{ + ArtistMbid: p.ArtistMBID, ArtistName: p.ArtistName, ProfileID: profile.ID, + FollowedBy: p.By, WantExisting: p.WantExisting, + }) + if err != nil { + return dbq.Follow{}, fmt.Errorf("wanted: follow: %w", err) + } + return f, nil +} + +// Unfollow stops following. Wants already recorded stay. +func (s *Service) Unfollow(ctx context.Context, artistMBID string) (bool, error) { + n, err := dbq.New(s.pool).DeleteFollow(ctx, artistMBID) + if err != nil { + return false, fmt.Errorf("wanted: unfollow: %w", err) + } + return n > 0, nil +} + +// WantParams is a release group to want. +type WantParams struct { + GroupMBID string + ArtistMBID string + ArtistName string + Title string + PrimaryType string + FirstReleaseDate string + Reason string + CreatedBy pgtype.UUID +} + +// WantWith records a want inside the caller's transaction, or returns the +// want already recorded for the group. +func WantWith(ctx context.Context, q *dbq.Queries, p WantParams) (dbq.Want, error) { + if p.GroupMBID == "" || p.ArtistMBID == "" { + return dbq.Want{}, errors.New("wanted: a release group and its artist are required") + } + w, err := q.WantReleaseGroup(ctx, dbq.WantReleaseGroupParams{ + ReleaseGroupMbid: p.GroupMBID, ArtistMbid: p.ArtistMBID, ArtistName: p.ArtistName, + Title: p.Title, PrimaryType: p.PrimaryType, FirstReleaseDate: p.FirstReleaseDate, + Reason: p.Reason, CreatedBy: p.CreatedBy, + }) + if err != nil { + return dbq.Want{}, fmt.Errorf("wanted: want: %w", err) + } + return w, nil +} + +// RemoveWant drops a want, open or held. +func (s *Service) RemoveWant(ctx context.Context, id pgtype.UUID) (bool, error) { + n, err := dbq.New(s.pool).DeleteWant(ctx, id) + if err != nil { + return false, fmt.Errorf("wanted: remove want: %w", err) + } + return n > 0, nil +} + +const ( + // FollowRefreshAge is how often a followed artist's discography is read + // for new releases: daily. + FollowRefreshAge = 24 * time.Hour + // NewReleaseGrace: a group first seen after the baseline is wanted only + // if it came out no earlier than this before the follow began. MusicBrainz + // gains old releases all the time; a single from 1998 catalogued today is + // not a new release. + NewReleaseGrace = 30 * 24 * time.Hour + // Interval is how often the worker looks for work; followsPerPass and + // wantsPerPass bound one pass so neither starves the other. + Interval = 5 * time.Minute + followsPerPass = 60 + wantsPerPass = 500 +) + +// Run applies follows and settles wants every Interval until ctx ends. +func (s *Service) Run(ctx context.Context) { + ticker := time.NewTicker(Interval) + defer ticker.Stop() + for { + res, err := s.Pass(ctx) + switch { + case err != nil && ctx.Err() == nil: + s.logger.Warn("wanted: pass stopped", "err", err, "follows", res.Follows, "wanted", res.Wanted, "held", res.Held) + case res.Wanted > 0 || res.Held > 0 || res.Baselined > 0 || res.Listed > 0 || res.Upgrades > 0: + s.logger.Info("wanted: pass complete", "follows", res.Follows, "baselined", res.Baselined, + "wanted", res.Wanted, "held", res.Held, "listed", res.Listed, "upgrades", res.Upgrades) + } + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + } +} + +// PassResult is what one pass did. +type PassResult struct { + Follows int // follows whose discography was applied + Baselined int // of those, first reads + Wanted int // wants recorded from follows + Held int // wants the library now answers + Listed int // library albums put on the list as held + // Upgrades: held albums wanted for a better copy, when the hourly check + // ran this pass. + Upgrades int +} + +// Pass applies the follows due and settles open wants. An unreachable +// MusicBrainz stops the follows (they need it) but not the settling, which +// falls back to what is held. +func (s *Service) Pass(ctx context.Context) (PassResult, error) { + var res PassResult + ferr := s.applyFollows(ctx, &res) + if ctx.Err() != nil { + return res, ctx.Err() + } + // The one-time listing of the library; a failure waits for the + // next pass and does not hold back settling. + if err := s.listLibrary(ctx, &res); err != nil && ctx.Err() == nil { + s.logger.Warn("wanted: list the library", "err", err) + } + if err := s.settleWants(ctx, &res); err != nil { + return res, err + } + if err := s.checkUpgrades(ctx, &res); err != nil && ctx.Err() == nil { + s.logger.Warn("wanted: judge held albums for upgrades", "err", err) + } + return res, ferr +} + +func (s *Service) applyFollows(ctx context.Context, res *PassResult) error { + q := dbq.New(s.pool) + due, err := q.ListFollowsDue(ctx, dbq.ListFollowsDueParams{ + Before: pgtype.Timestamptz{Time: s.now().Add(-FollowRefreshAge), Valid: true}, + MaxRows: followsPerPass, + }) + if err != nil { + return fmt.Errorf("wanted: list follows due: %w", err) + } + for _, f := range due { + n, err := s.applyFollow(ctx, q, f) + if err != nil { + if errors.Is(err, musicbrainz.ErrUnreachable) || ctx.Err() != nil { + return err + } + s.logger.Warn("wanted: apply follow", "artist", f.ArtistName, "mbid", f.ArtistMbid, "err", err) + continue + } + res.Follows++ + res.Wanted += n + if !f.BaselineAt.Valid { + res.Baselined++ + } + } + return nil +} + +// applyFollow reads the artist's discography and judges each release group +// the follow has not judged before. On the first read (the baseline) they +// are only marked seen, unless the follow asked to want what already exists. +// After it, a group is wanted when the profile passes it, it is recent, and +// the library does not hold it. +func (s *Service) applyFollow(ctx context.Context, q *dbq.Queries, f dbq.Follow) (int, error) { + d, err := s.cat.DiscographyWithin(ctx, f.ArtistMbid, FollowRefreshAge) + if err == nil && d.Stale { + // The held copy, served because MusicBrainz could not be asked: + // applying it would mark the follow checked without having looked. + return 0, musicbrainz.ErrUnreachable + } + if err != nil { + if !errors.Is(err, catalogue.ErrNotFound) { + return 0, err + } + // Merged away or removed from MusicBrainz: nothing to read. Marked + // checked so it is asked again tomorrow, not every pass. + s.logger.Info("wanted: followed artist not in MusicBrainz", "artist", f.ArtistName, "mbid", f.ArtistMbid) + return 0, q.MarkFollowChecked(ctx, f.ArtistMbid) + } + profile, err := q.GetReleaseTypeProfile(ctx, f.ProfileID) + if err != nil { + return 0, fmt.Errorf("read profile: %w", err) + } + groups, err := q.ListFollowGroups(ctx, f.ArtistMbid) + if err != nil { + return 0, fmt.Errorf("read discography: %w", err) + } + baseline := !f.BaselineAt.Valid + wanted := 0 + for _, g := range groups { + judge := (baseline && f.WantExisting) || (!baseline && !g.Seen && s.recent(g.FirstReleaseDate, f.BaselineAt.Time)) + passes := Passes(profile, g.Title, g.PrimaryType, g.SecondaryTypes) + if !passes { + if err := s.dropRefusedVersion(ctx, q, g.Mbid, g.Title); err != nil { + return wanted, err + } + } + if judge && passes { + held, err := q.LibraryHoldsReleaseGroup(ctx, dbq.LibraryHoldsReleaseGroupParams{ + GroupMbid: g.Mbid, Title: g.Title, ArtistMbid: f.ArtistMbid, + }) + if err != nil { + return wanted, fmt.Errorf("library check: %w", err) + } + if !held { + w, err := WantWith(ctx, q, WantParams{ + GroupMBID: g.Mbid, ArtistMBID: f.ArtistMbid, ArtistName: f.ArtistName, Title: g.Title, + PrimaryType: g.PrimaryType, FirstReleaseDate: g.FirstReleaseDate, + Reason: ReasonFollow, CreatedBy: f.FollowedBy, + }) + if err != nil { + return wanted, err + } + if !w.FulfilledAt.Valid { + wanted++ + } + } + } + if !g.Seen { + if err := q.MarkGroupSeen(ctx, dbq.MarkGroupSeenParams{ArtistMbid: f.ArtistMbid, ReleaseGroupMbid: g.Mbid}); err != nil { + return wanted, fmt.Errorf("mark seen: %w", err) + } + } + } + return wanted, q.MarkFollowChecked(ctx, f.ArtistMbid) +} + +// dropRefusedVersion takes off the list a group a follow wants that Passes +// now refuses for the version its title names, while it is still only +// wanted: one the library holds, or wanted for another reason, stays. +func (s *Service) dropRefusedVersion(ctx context.Context, q *dbq.Queries, group, title string) error { + v, ok := release.VersionInTitle(title) + if !ok { + return nil + } + w, err := q.GetWantByReleaseGroup(ctx, group) + if errors.Is(err, pgx.ErrNoRows) { + return nil + } + if err != nil { + return fmt.Errorf("read want %s: %w", title, err) + } + if w.Reason != ReasonFollow || w.FulfilledAt.Valid { + return nil + } + if _, err := q.DeleteWant(ctx, w.ID); err != nil { + return fmt.Errorf("drop want %s: %w", title, err) + } + s.logger.Info("wanted: a followed artist's release is another version, taken off the list", + "want", title, "version", v.What, "word", v.Word) + return nil +} + +// recent reports whether a release dated date (MusicBrainz's partial date: +// "2026", "2026-10", "2026-10-09", or empty when not yet known) came out no +// earlier than NewReleaseGrace before the follow's baseline. A partial date +// is read as the last day it could mean, so a year-only date is not +// dismissed for being "January". +func (s *Service) recent(date string, baseline time.Time) bool { + t, ok := latestDate(date) + if !ok { + return true + } + return !t.Before(baseline.Add(-NewReleaseGrace)) +} + +func latestDate(date string) (time.Time, bool) { + if t, err := time.Parse("2006-01-02", date); err == nil { + return t, true + } + if t, err := time.Parse("2006-01", date); err == nil { + return t.AddDate(0, 1, -1), true + } + if t, err := time.Parse("2006", date); err == nil { + return t.AddDate(1, 0, -1), true + } + return time.Time{}, false +} + +// settleWants marks open wants held when the library answers them. Each +// want's editions come from the catalogue, read from MusicBrainz the first +// time; once MusicBrainz is unreachable the rest use only what is held. +func (s *Service) settleWants(ctx context.Context, res *PassResult) error { + q := dbq.New(s.pool) + open, err := q.ListOpenWants(ctx, wantsPerPass) + if err != nil { + return fmt.Errorf("wanted: list open wants: %w", err) + } + mbDown := false + for _, w := range open { + if ctx.Err() != nil { + return ctx.Err() + } + shortest, moved, err := s.shortestEdition(ctx, q, w, &mbDown) + if err != nil { + s.logger.Warn("wanted: read editions", "want", w.Title, "group", w.ReleaseGroupMbid, "err", err) + } + if moved { + continue // settled next pass, under its group + } + album, ok, err := s.answeredBy(ctx, q, w, shortest) + if err != nil { + return err + } + if !ok { + continue + } + if err := q.FulfillWant(ctx, dbq.FulfillWantParams{ID: w.ID, AlbumID: album}); err != nil { + return fmt.Errorf("wanted: fulfil: %w", err) + } + res.Held++ + } + return nil +} + +// shortestEdition is the track count of the group's shortest official +// edition (any edition when none is official), or 0 when the catalogue +// cannot say. moved reports that the want was recorded under a release id +// and has been moved to its group. +func (s *Service) shortestEdition(ctx context.Context, q *dbq.Queries, w dbq.Want, mbDown *bool) (int, bool, error) { + var releases []dbq.MbRelease + if *mbDown { + held, err := q.ListMBReleasesForGroup(ctx, w.ReleaseGroupMbid) + if err != nil { + return 0, false, err + } + releases = held + } else { + ed, err := s.cat.Releases(ctx, w.ReleaseGroupMbid) + switch { + case err == nil: + for _, e := range ed.Editions { + releases = append(releases, e.Release) + } + case errors.Is(err, catalogue.ErrNotFound): + moved, merr := s.moveToGroup(ctx, w) + return 0, moved, merr + case errors.Is(err, musicbrainz.ErrUnreachable): + *mbDown = true + return s.shortestEdition(ctx, q, w, mbDown) + default: + return 0, false, err + } + } + return Shortest(releases), false, nil +} + +// Shortest is the track count of the shortest official release with tracks, +// or of the shortest release with tracks when none is official, or 0. +func Shortest(releases []dbq.MbRelease) int { + best, bestAny := 0, 0 + for _, r := range releases { + n := int(r.TrackCount) + if n <= 0 { + continue + } + if bestAny == 0 || n < bestAny { + bestAny = n + } + if r.Status == "Official" && (best == 0 || n < best) { + best = n + } + } + if best > 0 { + return best + } + return bestAny +} + +// moveToGroup handles a want MusicBrainz has no release group under: one +// recorded under a release id. It is moved to the release's +// group, with the requests that name it; when the group is already wanted, +// this copy is dropped. +func (s *Service) moveToGroup(ctx context.Context, w dbq.Want) (bool, error) { + group, err := s.releaseGroup(ctx, w.ReleaseGroupMbid) + if err != nil { + if errors.Is(err, tags.ErrNotFound) { + return false, nil + } + return false, fmt.Errorf("release group of %s: %w", w.ReleaseGroupMbid, err) + } + if group == "" || group == w.ReleaseGroupMbid { + return false, nil + } + err = pgx.BeginFunc(ctx, s.pool, func(tx pgx.Tx) error { + tq := dbq.New(tx) + if _, gerr := tq.GetWantByReleaseGroup(ctx, group); gerr == nil { + if _, derr := tq.DeleteWant(ctx, w.ID); derr != nil { + return derr + } + } else if errors.Is(gerr, pgx.ErrNoRows) { + if rerr := tq.RepointWant(ctx, dbq.RepointWantParams{ID: w.ID, ReleaseGroupMbid: group}); rerr != nil { + return rerr + } + } else { + return gerr + } + return tq.RepointRequestsAlbumMbid(ctx, dbq.RepointRequestsAlbumMbidParams{ + NewMbid: &group, OldMbid: &w.ReleaseGroupMbid, + }) + }) + if err != nil { + return false, fmt.Errorf("move want to its group: %w", err) + } + s.logger.Info("wanted: want moved from a release id to its group", + "want", w.Title, "release_mbid", w.ReleaseGroupMbid, "release_group_mbid", group) + return true, nil +} + +// answeredBy finds the library album that answers a want, if any: +// +// - for a want about an album whose files went missing, that album once no +// missing track is left without a track on disk in its place; +// - otherwise an album of the group with at least as many tracks on disk as +// the shortest official edition; +// - and when the catalogue cannot say how long the group is: a track +// arrived since the want, or nothing missing before it is still missing. +// +// Only albums with a track on disk count. +func (s *Service) answeredBy(ctx context.Context, q *dbq.Queries, w dbq.Want, shortest int) (pgtype.UUID, bool, error) { + holdings, err := q.WantHoldings(ctx, dbq.WantHoldingsParams{ + Since: w.WantedAt, GroupMbid: w.ReleaseGroupMbid, AlbumID: w.AlbumID, + }) + if err != nil { + return pgtype.UUID{}, false, fmt.Errorf("wanted: read holdings: %w", err) + } + for _, h := range holdings { + if Answers(w, h, shortest) { + return h.AlbumID, true, nil + } + } + return pgtype.UUID{}, false, nil +} + +// Answers is answeredBy's verdict on one album. +func Answers(w dbq.Want, h dbq.WantHoldingsRow, shortest int) bool { + if h.Present == 0 { + return false + } + if w.Reason == ReasonMissing && w.AlbumID.Valid && h.AlbumID == w.AlbumID { + return h.UncoveredMissing == 0 + } + if shortest > 0 { + return int(h.Present) >= shortest + } + return h.ArrivedSince || !h.MissingBefore +} diff --git a/internal/wanted/wanted_test.go b/internal/wanted/wanted_test.go new file mode 100644 index 00000000..e4512c31 --- /dev/null +++ b/internal/wanted/wanted_test.go @@ -0,0 +1,481 @@ +package wanted + +import ( + "context" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "os" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgtype" + "github.com/jackc/pgx/v5/pgxpool" + + "git.fabledsword.com/bvandeusen/minstrel/internal/catalogue" + "git.fabledsword.com/bvandeusen/minstrel/internal/db" + "git.fabledsword.com/bvandeusen/minstrel/internal/db/dbq" + "git.fabledsword.com/bvandeusen/minstrel/internal/dbtest" + "git.fabledsword.com/bvandeusen/minstrel/internal/musicbrainz" + "git.fabledsword.com/bvandeusen/minstrel/internal/tags" +) + +func TestPasses_PrimaryListedAndEverySecondaryAllowed(t *testing.T) { + standard := dbq.ReleaseTypeProfile{PrimaryTypes: []string{"Album", "EP", "Single"}, SecondaryTypes: []string{}} + withLive := dbq.ReleaseTypeProfile{PrimaryTypes: []string{"Album"}, SecondaryTypes: []string{"Live"}} + for _, c := range []struct { + profile dbq.ReleaseTypeProfile + title string + primary string + secondary []string + want bool + }{ + {standard, "Revenge", "Album", nil, true}, + {standard, "Helena", "single", nil, true}, // MusicBrainz's case is not relied on + {standard, "Live at Koko", "Album", []string{"Live"}, false}, + {standard, "Radio Show", "Broadcast", nil, false}, + {withLive, "Live at Koko", "Album", []string{"Live"}, true}, + {withLive, "Live & Rare", "Album", []string{"Live", "Compilation"}, false}, + {withLive, "Like Phantoms, Forever", "EP", nil, false}, + // MusicBrainz types these plain Singles; their titles say otherwise. + {standard, "Common People (Recorded For BBC Radio 1)", "Single", nil, false}, + {standard, "Welcome to the Black Parade (Steve Aoki remix)", "Single", nil, false}, + {standard, "The Five of Us Are Dying (Rough mix)", "Single", nil, false}, + {withLive, "Desolation Row (live at BBC Maida Vale)", "Album", nil, true}, + {withLive, "Helena (acoustic)", "Album", nil, false}, + {standard, "Live Through This", "Album", nil, true}, + } { + if got := Passes(c.profile, c.title, c.primary, c.secondary); got != c.want { + t.Errorf("Passes(%v/%v, %q %s %v) = %v, want %v", c.profile.PrimaryTypes, c.profile.SecondaryTypes, + c.title, c.primary, c.secondary, got, c.want) + } + } +} + +func TestProfileInput_NormalizesToMusicBrainzTypes(t *testing.T) { + in, err := ProfileInput{Name: " Live too ", PrimaryTypes: []string{"single", "Album"}, SecondaryTypes: []string{"live"}}.normalize() + if err != nil { + t.Fatal(err) + } + if in.Name != "Live too" || fmt.Sprint(in.PrimaryTypes) != "[Album Single]" || fmt.Sprint(in.SecondaryTypes) != "[Live]" { + t.Errorf("normalized = %+v", in) + } + for name, bad := range map[string]ProfileInput{ + "no name": {PrimaryTypes: []string{"Album"}}, + "no primary type": {Name: "x"}, + "unknown type": {Name: "x", PrimaryTypes: []string{"Mixtape"}}, + } { + if _, err := bad.normalize(); !errors.Is(err, ErrProfileInvalid) { + t.Errorf("%s: err = %v", name, err) + } + } +} + +func TestRecent_ReadsAPartialDateAsItsLastDay(t *testing.T) { + s := &Service{} + baseline := time.Date(2026, 10, 10, 0, 0, 0, 0, time.UTC) + for date, want := range map[string]bool{ + "": true, // not yet known: an announced release + "2026-11-01": true, + "2026-09-15": true, // inside the grace before the follow + "2026-08-01": false, + "2026": true, // could be December + "2025": false, + "2026-09": true, + "1998-04-01": false, + } { + if got := s.recent(date, baseline); got != want { + t.Errorf("recent(%q) = %v, want %v", date, got, want) + } + } +} + +func TestShortest_PrefersOfficialEditions(t *testing.T) { + rel := func(status string, n int32) dbq.MbRelease { return dbq.MbRelease{Status: status, TrackCount: n} } + for _, c := range []struct { + releases []dbq.MbRelease + want int + }{ + {nil, 0}, + {[]dbq.MbRelease{rel("Official", 14), rel("Official", 12), rel("Promotion", 3)}, 12}, + {[]dbq.MbRelease{rel("Bootleg", 9), rel("", 11)}, 9}, + {[]dbq.MbRelease{rel("Official", 0), rel("Bootleg", 9)}, 9}, + } { + if got := Shortest(c.releases); got != c.want { + t.Errorf("Shortest(%v) = %d, want %d", c.releases, got, c.want) + } + } +} + +func TestAnswers(t *testing.T) { + album := pgtype.UUID{Bytes: [16]byte{1}, Valid: true} + other := pgtype.UUID{Bytes: [16]byte{2}, Valid: true} + missing := dbq.Want{Reason: ReasonMissing, AlbumID: album} + request := dbq.Want{Reason: ReasonRequest} + for name, c := range map[string]struct { + w dbq.Want + h dbq.WantHoldingsRow + shortest int + want bool + }{ + "nothing on disk": {request, dbq.WantHoldingsRow{AlbumID: other, ArrivedSince: true}, 10, false}, + "a full edition": {request, dbq.WantHoldingsRow{AlbumID: other, Present: 10}, 10, true}, + "short of every edition": {request, dbq.WantHoldingsRow{AlbumID: other, Present: 9, ArrivedSince: true}, 10, false}, + "no editions known, an arrival": {request, dbq.WantHoldingsRow{AlbumID: other, Present: 1, ArrivedSince: true, MissingBefore: true}, 0, true}, + "no editions known, still missing": {request, dbq.WantHoldingsRow{AlbumID: other, Present: 5, MissingBefore: true}, 0, false}, + "the lost album, a gap remains": {missing, dbq.WantHoldingsRow{AlbumID: album, Present: 12, UncoveredMissing: 1}, 10, false}, + "the lost album, every gap filled": {missing, dbq.WantHoldingsRow{AlbumID: album, Present: 9, UncoveredMissing: 0}, 10, true}, + "another copy of the lost album": {missing, dbq.WantHoldingsRow{AlbumID: other, Present: 10}, 10, true}, + } { + if got := Answers(c.w, c.h, c.shortest); got != c.want { + t.Errorf("%s: Answers = %v, want %v", name, got, c.want) + } + } +} + +func TestFollowable(t *testing.T) { + for _, c := range []struct { + mbid, name string + want bool + }{ + {"a", "Gorillaz", true}, + {variousArtists, "Various Artists", false}, + {"b", "[unknown]", false}, + {"", "Nameless", false}, + {"c", "[ ]Bracketed Band", true}, + } { + if got := Followable(c.mbid, c.name); got != c.want { + t.Errorf("Followable(%q, %q) = %v", c.mbid, c.name, got) + } + } +} + +// --- integration --- + +func newPool(t *testing.T) *pgxpool.Pool { + t.Helper() + if testing.Short() { + t.Skip("skipping integration test in -short mode") + } + dsn := os.Getenv("MINSTREL_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("MINSTREL_TEST_DATABASE_URL not set") + } + if err := db.Migrate(dsn, slog.New(slog.NewTextHandler(io.Discard, nil))); err != nil { + t.Fatalf("migrate: %v", err) + } + pool, err := pgxpool.New(context.Background(), dsn) + if err != nil { + t.Fatalf("pool: %v", err) + } + t.Cleanup(pool.Close) + dbtest.ResetDB(t, pool) + return pool +} + +const ( + blur = "ba853904-ae25-4ebb-89d6-c44cfbd71bd2" + parklife = "11111111-0000-4000-8000-000000000001" // 1994 album, held + liveAtWemb = "11111111-0000-4000-8000-000000000002" // 2009 live album + theBallad = "11111111-0000-4000-8000-000000000003" // 2023 album, not held + newSingle = "11111111-0000-4000-8000-000000000004" // appears after the baseline + oldSingle = "11111111-0000-4000-8000-000000000005" // 1997 single, catalogued late + bbcSingle = "11111111-0000-4000-8000-000000000006" // live, typed a plain Single + parkRelease = "22222222-0000-4000-8000-000000000001" +) + +// fakeMB is a MusicBrainz holding Blur's discography; added groups appear in +// it after the baseline, and down makes every request a 503. +type fakeMB struct { + srv *httptest.Server + added atomic.Bool + down atomic.Bool +} + +func newFakeMB(t *testing.T) *fakeMB { + t.Helper() + f := &fakeMB{} + group := func(id, title, primary, secondary, date string) string { + return fmt.Sprintf(`{"id": %q, "title": %q, "primary-type": %q, "secondary-types": [%s], "first-release-date": %q, + "artist-credit": [{"name": "Blur", "artist": {"id": %q}}]}`, id, title, primary, secondary, date, blur) + } + f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if f.down.Load() { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + switch { + case r.URL.Path == "/artist/"+blur: + _, _ = fmt.Fprintf(w, `{"id": %q, "name": "Blur", "type": "Group", "country": "GB"}`, blur) + case r.URL.Path == "/release-group" && r.URL.Query().Get("artist") == blur: + groups := group(parklife, "Parklife", "Album", "", "1994-04-25") + "," + + group(liveAtWemb, "Live at Wembley Stadium", "Album", `"Live"`, "2009-07-02") + "," + + group(theBallad, "The Ballad of Darren", "Album", "", "2023-07-21") + "," + + group(bbcSingle, "Song 2 (Recorded For BBC Radio 1)", "Single", "", "2023-08-01") + n := 4 + if f.added.Load() { + groups += "," + group(newSingle, "New Single", "Single", "", "") + + "," + group(oldSingle, "Old Single", "Single", "", "1997-02-03") + n = 6 + } + _, _ = fmt.Fprintf(w, `{"release-group-count": %d, "release-groups": [%s]}`, n, groups) + case r.URL.Path == "/release-group/"+theBallad: + _, _ = fmt.Fprint(w, group(theBallad, "The Ballad of Darren", "Album", "", "2023-07-21")) + case r.URL.Path == "/release" && r.URL.Query().Get("release-group") == theBallad: + _, _ = fmt.Fprint(w, `{"release-count": 2, "releases": [ + {"id": "r-ballad-1", "title": "The Ballad of Darren", "status": "Official", "media": [{"position": 1, + "tracks": [{"position": 1, "number": "1", "title": "The Ballad", "recording": {"id": "rec-1"}}, + {"position": 2, "number": "2", "title": "St. Charles Square", "recording": {"id": "rec-2"}}]}]}, + {"id": "r-ballad-2", "title": "The Ballad of Darren", "status": "Official", "media": [{"position": 1, + "tracks": [{"position": 1, "number": "1", "title": "The Ballad", "recording": {"id": "rec-1"}}, + {"position": 2, "number": "2", "title": "St. Charles Square", "recording": {"id": "rec-2"}}, + {"position": 3, "number": "3", "title": "Bonus", "recording": {"id": "rec-3"}}]}]}]}`) + case r.URL.Path == "/release": + _, _ = fmt.Fprint(w, `{"release-count": 0, "releases": []}`) + case r.URL.Path == "/release-group" && strings.Contains(r.URL.Query().Get("query"), "releasegroup:"): + // The library listing's search by artist and title. + found := "" + if strings.Contains(r.URL.Query().Get("query"), "Ballad") { + found = group(theBallad, "The Ballad of Darren", "Album", "", "2023-07-21") + } + _, _ = fmt.Fprintf(w, `{"release-groups": [%s]}`, found) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(f.srv.Close) + return f +} + +func newService(t *testing.T, pool *pgxpool.Pool, f *fakeMB) *Service { + t.Helper() + mb := &musicbrainz.Client{BaseURL: f.srv.URL, Pace: &musicbrainz.Pacer{}, Retries: -1} + s := New(pool, catalogue.New(pool, mb, nil), slog.New(slog.NewTextHandler(io.Discard, nil))) + // Never the real MusicBrainz from a test. + s.releaseGroup = func(context.Context, string) (string, error) { return "", tags.ErrNotFound } + return s +} + +// seedHeld puts an album of Blur's in the library with n tracks on disk. +func seedHeld(t *testing.T, q *dbq.Queries, title, release, group string, n int) dbq.Album { + t.Helper() + ctx := context.Background() + mbid := blur + artist, err := q.UpsertArtist(ctx, dbq.UpsertArtistParams{Name: "Blur", SortName: "Blur", Mbid: &mbid}) + if err != nil { + t.Fatal(err) + } + params := dbq.UpsertAlbumParams{Title: title, SortTitle: title, ArtistID: artist.ID, Mbid: &release} + if group != "" { + params.ReleaseGroupMbid = &group + } + album, err := q.UpsertAlbum(ctx, params) + if err != nil { + t.Fatal(err) + } + for i := range n { + num := int32(i + 1) + if _, err := q.UpsertTrack(ctx, dbq.UpsertTrackParams{ + Title: fmt.Sprintf("Track %d", i+1), AlbumID: album.ID, ArtistID: artist.ID, TrackNumber: &num, + DurationMs: 180000, FilePath: fmt.Sprintf("/music/%s/%02d.flac", release, i+1), FileSize: 1, FileFormat: "flac", + }); err != nil { + t.Fatal(err) + } + } + return album +} + +func TestFollow_BaselineThenOnlyNewReleasesThatPass_Integration(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + ctx := context.Background() + f := newFakeMB(t) + s := newService(t, pool, f) + _ = seedHeld(t, q, "Parklife", parkRelease, parklife, 2) + + if _, err := s.Follow(ctx, FollowParams{ArtistMBID: blur, ArtistName: "Blur"}); err != nil { + t.Fatal(err) + } + res, err := s.Pass(ctx) + if err != nil { + t.Fatal(err) + } + if res.Baselined != 1 || res.Wanted != 0 { + t.Fatalf("baseline pass = %+v: a follow wants nothing it already lists", res) + } + + // MusicBrainz gains two singles: one new, one from 1997. A day later the + // follow reads the discography again. + f.added.Store(true) + if _, err := pool.Exec(ctx, ` + UPDATE mb_artists SET discography_refreshed_at = now() - interval '2 days'; + UPDATE follows SET checked_at = now() - interval '2 days'`); err != nil { + t.Fatal(err) + } + if res, err = s.Pass(ctx); err != nil { + t.Fatal(err) + } + if res.Wanted != 1 { + t.Errorf("second pass wanted %d, want 1 (the new single)", res.Wanted) + } + if w, err := q.GetWantByReleaseGroup(ctx, newSingle); err != nil || w.Reason != ReasonFollow || w.PrimaryType != "Single" { + t.Errorf("new single: %+v, %v", w, err) + } + if _, err := q.GetWantByReleaseGroup(ctx, oldSingle); err == nil { + t.Error("a 1997 single catalogued late is not a new release") + } +} + +func TestFollow_WantExistingWantsWhatPassesAndIsNotHeld_Integration(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + ctx := context.Background() + f := newFakeMB(t) + s := newService(t, pool, f) + _ = seedHeld(t, q, "Parklife", parkRelease, parklife, 2) + // Wanted before the follow refused the version its title names. + if _, err := WantWith(ctx, q, WantParams{GroupMBID: bbcSingle, ArtistMBID: blur, ArtistName: "Blur", + Title: "Song 2 (Recorded For BBC Radio 1)", PrimaryType: "Single", Reason: ReasonFollow}); err != nil { + t.Fatal(err) + } + + if _, err := s.Follow(ctx, FollowParams{ArtistMBID: blur, ArtistName: "Blur", WantExisting: true}); err != nil { + t.Fatal(err) + } + if _, err := s.Pass(ctx); err != nil { + t.Fatal(err) + } + if _, err := q.GetWantByReleaseGroup(ctx, theBallad); err != nil { + t.Errorf("the studio album not held should be wanted: %v", err) + } + for group, why := range map[string]string{parklife: "held", liveAtWemb: "live, and Standard is studio only", + bbcSingle: "recorded for the BBC, which its title says, and Standard is studio only"} { + if _, err := q.GetWantByReleaseGroup(ctx, group); err == nil { + t.Errorf("%s wanted, but it is %s", group, why) + } + } + if fol, _ := q.GetFollow(ctx, blur); fol.WantExisting || !fol.BaselineAt.Valid { + t.Errorf("follow = %+v: want_existing applied once, then cleared", fol) + } + + // A profile that allows live releases, applied with "want what exists". + live, err := s.CreateProfile(ctx, ProfileInput{Name: "With live", PrimaryTypes: []string{"Album"}, SecondaryTypes: []string{"Live"}}) + if err != nil { + t.Fatal(err) + } + if _, err := s.Follow(ctx, FollowParams{ArtistMBID: blur, ArtistName: "Blur", ProfileID: live.ID, WantExisting: true}); err != nil { + t.Fatal(err) + } + if _, err := s.Pass(ctx); err != nil { + t.Fatal(err) + } + if _, err := q.GetWantByReleaseGroup(ctx, liveAtWemb); err != nil { + t.Errorf("with live allowed, the live album is wanted: %v", err) + } + if err := s.DeleteProfile(ctx, live.ID); !errors.Is(err, ErrProfileInUse) { + t.Errorf("deleting a profile in use: err = %v", err) + } +} + +func TestSettle_HeldOnceAFullEditionIsOnDisk_Integration(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + ctx := context.Background() + s := newService(t, pool, newFakeMB(t)) + + if _, err := WantWith(ctx, q, WantParams{ + GroupMBID: theBallad, ArtistMBID: blur, ArtistName: "Blur", Title: "The Ballad of Darren", Reason: ReasonRequest, + }); err != nil { + t.Fatal(err) + } + album := seedHeld(t, q, "The Ballad of Darren", "r-ballad-1", theBallad, 1) + if res, err := s.Pass(ctx); err != nil || res.Held != 0 { + t.Fatalf("one track of a two-track edition: %+v, %v", res, err) + } + _ = seedHeld(t, q, "The Ballad of Darren", "r-ballad-1", theBallad, 2) + if res, err := s.Pass(ctx); err != nil || res.Held != 1 { + t.Fatalf("the shortest edition complete: %+v, %v", res, err) + } + w, _ := q.GetWantByReleaseGroup(ctx, theBallad) + if !w.FulfilledAt.Valid || w.AlbumID != album.ID { + t.Errorf("want = %+v: held by the album", w) + } +} + +func TestSettle_MovesAWantNamedByReleaseIdToItsGroup_Integration(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + ctx := context.Background() + s := newService(t, pool, newFakeMB(t)) + s.releaseGroup = func(_ context.Context, release string) (string, error) { + if release == "r-ballad-1" { + return theBallad, nil + } + return "", tags.ErrNotFound + } + if _, err := WantWith(ctx, q, WantParams{ + GroupMBID: "r-ballad-1", ArtistMBID: blur, ArtistName: "Blur", Title: "The Ballad of Darren", Reason: ReasonRequest, + }); err != nil { + t.Fatal(err) + } + group := "r-ballad-1" + if _, err := q.CreateRequest(ctx, dbq.CreateRequestParams{ + UserID: seedUser(t, pool), Kind: dbq.RequestKindAlbum, ArtistMbid: blur, AlbumMbid: &group, ArtistName: "Blur", + }); err != nil { + t.Fatal(err) + } + if _, err := s.Pass(ctx); err != nil { + t.Fatal(err) + } + if _, err := q.GetWantByReleaseGroup(ctx, theBallad); err != nil { + t.Errorf("the want should be under its group now: %v", err) + } + var moved string + if err := pool.QueryRow(ctx, "SELECT album_mbid FROM requests").Scan(&moved); err != nil || moved != theBallad { + t.Errorf("the request should follow its want: %q, %v", moved, err) + } +} + +func TestPass_MusicBrainzDownStopsFollowsButStillSettles_Integration(t *testing.T) { + pool := newPool(t) + q := dbq.New(pool) + ctx := context.Background() + f := newFakeMB(t) + s := newService(t, pool, f) + if _, err := s.Follow(ctx, FollowParams{ArtistMBID: blur, ArtistName: "Blur"}); err != nil { + t.Fatal(err) + } + if _, err := WantWith(ctx, q, WantParams{ + GroupMBID: "some-group", ArtistMBID: blur, ArtistName: "Blur", Title: "Some", Reason: ReasonRequest, + }); err != nil { + t.Fatal(err) + } + _ = seedHeld(t, q, "Some", "some-release", "some-group", 3) + f.down.Store(true) + res, err := s.Pass(ctx) + if !errors.Is(err, musicbrainz.ErrUnreachable) { + t.Errorf("err = %v, want ErrUnreachable", err) + } + if res.Held != 1 { + t.Errorf("held = %d: with no editions known, an album that arrived since the want answers it", res.Held) + } + if fol, _ := q.GetFollow(ctx, blur); fol.BaselineAt.Valid { + t.Error("a follow is not baselined without reading its discography") + } +} + +func seedUser(t *testing.T, pool *pgxpool.Pool) pgtype.UUID { + t.Helper() + u, err := dbq.New(pool).CreateUser(context.Background(), dbq.CreateUserParams{ + Username: dbtest.TestUserPrefix + "wanter", PasswordHash: "x", ApiTokenHash: "x", + }) + if err != nil { + t.Fatal(err) + } + return u.ID +} diff --git a/internal/words/words.go b/internal/words/words.go new file mode 100644 index 00000000..d4bbbd96 --- /dev/null +++ b/internal/words/words.go @@ -0,0 +1,15 @@ +// Package words says counts in English for what the server writes to people: +// notifications, emails, the reasons a scan or a match gives. +package words + +import "strconv" + +// Count says n with the noun that agrees with it: "1 track", "3 tracks". The +// noun may carry its verb ("track went missing" / "tracks went missing"). +// The web app's plural() is the same thing on the other side. +func Count[N ~int | ~int32 | ~int64](n N, one, many string) string { + if n == 1 { + return "1 " + one + } + return strconv.FormatInt(int64(n), 10) + " " + many +} diff --git a/internal/words/words_test.go b/internal/words/words_test.go new file mode 100644 index 00000000..411bb673 --- /dev/null +++ b/internal/words/words_test.go @@ -0,0 +1,19 @@ +package words + +import "testing" + +func TestCount(t *testing.T) { + cases := []struct { + got, want string + }{ + {Count(1, "track", "tracks"), "1 track"}, + {Count(0, "track", "tracks"), "0 tracks"}, + {Count(int64(3), "track went missing", "tracks went missing"), "3 tracks went missing"}, + {Count(int32(-1), "copy", "copies"), "-1 copies"}, + } + for _, c := range cases { + if c.got != c.want { + t.Errorf("got %q, want %q", c.got, c.want) + } + } +} diff --git a/shared/README.md b/shared/README.md index e6f862e4..f8c720c4 100644 --- a/shared/README.md +++ b/shared/README.md @@ -3,12 +3,8 @@ `fabledsword.tokens.json` is the canonical statement of the FabledSword palette — the dark, light and flat cohorts in one machine-readable place. -It lived under `flutter_client/shared/` until that client was deleted -(2026-08-16). It was never Flutter-specific: the Android theme -(`FabledSwordTokens.kt`) names it as its source of truth, and the web -client's CSS custom properties carry the same values. It moved here rather -than going with the client, because losing the one place all three cohorts -are written down together would have been collateral damage. +The Android theme (`FabledSwordTokens.kt`) names it as its source of +truth, and the web client's CSS custom properties carry the same values. Nothing generates from it today — both clients hold their own transcription, so this is the reference they must agree with, not a build input. If the diff --git a/tools/gen-brand-assets.py b/tools/gen-brand-assets.py index 8a5c3c3b..8badb6d8 100644 --- a/tools/gen-brand-assets.py +++ b/tools/gen-brand-assets.py @@ -180,11 +180,10 @@ def main(): " Literal colours, for any consumer that cannot inline the SVG and\n" " therefore cannot supply a currentColor. -->\n")) - # The tab icon is the full mark too (#5267). A hand-drawn reduced hat used - # to stand in at tab size because the art loses detail at 16px, but a - # redrawing reads as a different logo. The full mark keeps the logo's own - # shape, and a tab on a high-resolution screen draws it from 32px, where it - # holds. Do not swap a redrawn form back in here. + # The tab icon is the full mark too. A hand-drawn reduced form would lose + # less detail at 16px, but a redrawing reads as a different logo; the full + # mark keeps the logo's own shape, and a tab on a high-resolution screen + # draws it from 32px, where it holds. Do not swap a redrawn form in here. (ROOT / "web/static/brand/favicon.svg").write_text(svg_doc( vb, tr, "currentColor", ACCENT, hat_d, notes_d, head=" +

+ {#if !canFetch} + No download client is switched on, so nothing can be downloaded. + {:else if !canSearch} + No indexer is switched on, so nothing new can be found. + {:else} + Music can be searched for and downloaded. + {/if} +

+ +
+ + {#each indexers as service (service.code)} + + {:else} +

No indexer yet. Prowlarr is the usual choice.

+ {/each} + {#if searchingExchanges.length > 0} +

+ Also searched: {searchingExchanges.map((s) => s.label).join(', ')}, under download clients. +

+ {/if} + {@render addMenu('search', 'Add indexer')} + +
+ +
+ + {#each clients as service (service.code)} + + {:else} +

No download client yet.

+ {/each} + {@render addMenu('download', 'Add download client')} + +
+ {/if} + diff --git a/web/src/lib/components/AcquisitionCard.test.ts b/web/src/lib/components/AcquisitionCard.test.ts new file mode 100644 index 00000000..3db6448f --- /dev/null +++ b/web/src/lib/components/AcquisitionCard.test.ts @@ -0,0 +1,168 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor, within } from '@testing-library/svelte'; +import type { AcquisitionService } from '#lib/api/admin.js'; + +vi.mock('#lib/api/admin.js', () => ({ + listAcquisitionServices: vi.fn(), + addAcquisitionService: vi.fn(), + updateAcquisitionService: vi.fn(), + removeAcquisitionService: vi.fn(), + testAcquisitionService: vi.fn(), + setAcquisitionSecret: vi.fn(), + clearAcquisitionSecret: vi.fn() +})); + +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import AcquisitionCard from './AcquisitionCard.svelte'; +import { + addAcquisitionService, + listAcquisitionServices, + setAcquisitionSecret, + testAcquisitionService, + updateAcquisitionService +} from '#lib/api/admin.js'; + +function service(over: Partial = {}): AcquisitionService { + return { + code: 'prowlarr', + kind: 'search', + protocol: 'prowlarr', + label: 'Prowlarr', + enabled: false, + base_url: '', + has_api_key: false, + has_password: false, + username: '', + category: '', + timeout_ms: 60000, + serves_search: true, + serves_downloads: true, + serves_enrichment: true, + local_path: '/downloads', + remote_path: '', + searches_per_hour: 120, + usable: false, + ...over + }; +} + +const exchange = (over: Partial = {}) => + service({ + code: 'exchange', + kind: 'download', + protocol: 'exchange', + label: 'FabledExchange', + category: 'minstrel', + searches_per_hour: 0, + ...over + }); + +afterEach(() => vi.clearAllMocks()); + +async function renderCard(services: AcquisitionService[]) { + vi.mocked(listAcquisitionServices).mockResolvedValue(services); + render(AcquisitionCard); + await screen.findByTestId('acquisition-health'); +} + +describe('AcquisitionCard', () => { + test('starts empty: nothing is seeded', async () => { + await renderCard([]); + expect(screen.getByText(/no indexer yet/i)).toBeInTheDocument(); + expect(screen.getByText(/no download client yet/i)).toBeInTheDocument(); + expect(screen.getByTestId('acquisition-health')).toHaveTextContent(/nothing can be downloaded/i); + }); + + test('adding offers the types each list takes', async () => { + await renderCard([]); + await fireEvent.click(screen.getByRole('button', { name: 'Add indexer' })); + const indexerChoices = within(screen.getByTestId('add-search')); + expect(indexerChoices.getByRole('button', { name: /prowlarr/i })).toBeInTheDocument(); + expect(indexerChoices.getByRole('button', { name: /torznab feed/i })).toBeInTheDocument(); + expect(indexerChoices.queryByRole('button', { name: /qbittorrent/i })).toBeNull(); + + vi.mocked(addAcquisitionService).mockResolvedValue(service()); + await fireEvent.click(indexerChoices.getByRole('button', { name: /prowlarr/i })); + expect(addAcquisitionService).toHaveBeenCalledWith('prowlarr'); + + await fireEvent.click(screen.getByRole('button', { name: 'Add download client' })); + const clientChoices = within(screen.getByTestId('add-download')); + expect(clientChoices.getByRole('button', { name: /qbittorrent/i })).toBeInTheDocument(); + expect(clientChoices.getByRole('button', { name: /fabledexchange/i })).toBeInTheDocument(); + }); + + test('an exchange sits under download clients with a switch per job; others have none', async () => { + await renderCard([ + service({ code: 'qbittorrent', kind: 'download', protocol: 'qbittorrent', label: 'qBittorrent' }), + exchange() + ]); + const clients = within(screen.getByTestId('download-clients')); + expect(clients.getAllByTestId('acquisition-service')).toHaveLength(2); + // Only the exchange carries the job switches. + expect(clients.getAllByTestId('exchange-jobs')).toHaveLength(1); + expect(clients.getByRole('checkbox', { name: /downloads what is grabbed/i })).toBeChecked(); + expect(clients.getByRole('checkbox', { name: /searches its dht index/i })).toBeChecked(); + expect(clients.getByRole('checkbox', { name: /checks a release/i })).toBeChecked(); + // qBittorrent asks for a username and a password; the exchange for an optional key. + expect(clients.getByText('Username')).toBeInTheDocument(); + expect(clients.getByText(/save password/i)).toBeInTheDocument(); + expect(clients.getByText(/save key/i)).toBeInTheDocument(); + }); + + test('an exchange searching is named under indexers too', async () => { + await renderCard([exchange({ usable: true, enabled: true, base_url: 'http://ex' })]); + expect(within(screen.getByTestId('indexers')).getByText(/also searched: fabledexchange/i)).toBeInTheDocument(); + expect(screen.getByTestId('acquisition-health')).toHaveTextContent(/searched for and downloaded/i); + }); + + test('switching a job off and saving sends the switches', async () => { + const ex = exchange({ base_url: 'http://ex' }); + await renderCard([ex]); + await fireEvent.click(screen.getByRole('checkbox', { name: /checks a release/i })); + vi.mocked(updateAcquisitionService).mockResolvedValue({ ...ex, serves_enrichment: false }); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + await waitFor(() => expect(updateAcquisitionService).toHaveBeenCalled()); + const [code, sent] = vi.mocked(updateAcquisitionService).mock.calls[0]; + expect(code).toBe('exchange'); + expect(sent).toMatchObject({ serves_enrichment: false, serves_search: true, serves_downloads: true }); + }); + + test('a refused setting is shown beside the form in the server’s words', async () => { + await renderCard([service({ base_url: 'http://p' })]); + await fireEvent.input(screen.getByRole('textbox', { name: /^name$/i }), { + target: { value: 'Mine' } + }); + vi.mocked(updateAcquisitionService).mockRejectedValue({ + code: 'invalid_setting', + message: 'Address: start with http:// or https://.', + status: 400 + }); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + expect(await screen.findByTestId('acquisition-problem')).toHaveTextContent( + 'Address: start with http:// or https://.' + ); + }); + + test('Test shows what the server found, and needs an address', async () => { + await renderCard([service({ code: 'a' }), service({ code: 'b', label: 'Second', base_url: 'http://p' })]); + const [noAddress, addressed] = screen.getAllByRole('button', { name: 'Test' }); + expect(noAddress).toBeDisabled(); + vi.mocked(testAcquisitionService).mockResolvedValue({ + ok: false, + detail: 'the API key was refused: HTTP 401. Check it on the service’s own settings page.' + }); + await fireEvent.click(addressed); + expect(testAcquisitionService).toHaveBeenCalledWith('b'); + expect(await screen.findByTestId('acquisition-test-result')).toHaveTextContent(/key was refused/i); + }); + + test('a key is sent write-only and never shown back', async () => { + await renderCard([service({ base_url: 'http://p', has_api_key: true })]); + expect(screen.getByLabelText(/replace the api key/i)).toHaveValue(''); + await fireEvent.input(screen.getByLabelText(/replace the api key/i), { target: { value: 'k2' } }); + vi.mocked(setAcquisitionSecret).mockResolvedValue(undefined); + await fireEvent.click(screen.getByRole('button', { name: /save key/i })); + await waitFor(() => expect(setAcquisitionSecret).toHaveBeenCalledWith('prowlarr', 'api-key', 'k2')); + }); +}); diff --git a/web/src/lib/components/AcquisitionServiceRow.svelte b/web/src/lib/components/AcquisitionServiceRow.svelte new file mode 100644 index 00000000..48f7df91 --- /dev/null +++ b/web/src/lib/components/AcquisitionServiceRow.svelte @@ -0,0 +1,394 @@ + + +
+
+ +
+ {#each jobs as job (job)} + {job} + {/each} + {#if service.usable} + On + {:else if service.enabled} + On, but no address + {:else} + Off + {/if} +
+
+ +

{blurb[service.protocol]}

+ +
+ Connection +
+ + + + + {#if isExchange} +
+ What it does here + + + +
+ {/if} + + {#if isQbit} + + {/if} + + {#if fetches} + + + + {/if} + + {#if searches} + + {/if} + + +
+ +
+ +
+ + + {#if secretSet} + + {/if} +
+ + {#if service.protocol === 'prowlarr'} + Prowlarr shows it under Settings → General. + {:else if isExchange} + Optional: only if your FabledExchange asks for one. Give Minstrel a key of its own there, + so it can be revoked on its own. + {:else if service.protocol === 'torznab'} + Optional: only if the feed asks for one. + {:else} + Empty if your WebUI does not ask for one. + {/if} + {secretSet ? 'One is saved.' : ''} Never shown again once saved. + +
+
+ + {#if problem} +

{problem}

+ {/if} + {#if result} +

+ {result.detail} +

+ {/if} + +
+ {#if confirmRemove} + + Remove {service.label}? Minstrel forgets its address and credentials. + Nothing on the service itself changes. + + + + {:else} + + + + {/if} +
+
diff --git a/web/src/lib/components/ActiveSessions.svelte b/web/src/lib/components/ActiveSessions.svelte index f1aaa0b0..1b033f0f 100644 --- a/web/src/lib/components/ActiveSessions.svelte +++ b/web/src/lib/components/ActiveSessions.svelte @@ -1,4 +1,5 @@ -
-

Active sessions

-

- Every device signed in to your account. If you see one you don't recognise — especially - one marked as having moved — sign it out and change your password. -

+ +
+
+

Signed-in devices

+

+ One you don't recognise? Sign it out and change your password. +

+
{#if loadError}

@@ -190,7 +193,7 @@ {#if confirmingLogoutOthers}

- Sign out {others === 1 ? '1 other device' : `${others} other devices`}? You'll stay + Sign out {plural(others, 'other device', 'other devices')}? You'll stay signed in here.
- {#if state === 'snoozed'} + {#if cardState === 'snoozed'} - {:else if state === 'requestable'} + {:else if cardState === 'requestable'}
- {:else if state === 'kept'} + {:else if cardState === 'kept'} +

+{:else if current === null} +

Loading…

+{:else if !editing} + + + {#each current.folders as f (f.path)}{f.path}{:else}—{/each} + + + + {#each current.folders.filter((f) => !f.present) as f (f.path)} +

+ The server can't see {f.path}{f.tracks > 0 ? `, which holds ${plural(f.tracks, 'track', 'tracks')}` : ''}. + If its files are now somewhere else, change the folder to where they are. +

+ {/each} +{:else} +
+ Folders +

+ Change a folder's path when its files have moved; the tracks move with it and keep their plays, likes and + playlists. Each path is as the server sees it, inside its container. +

+ {#each rows as row, i (i)} +
+ + +
+ {/each} +
+ +
+
+ {#each moved as m (m.was)} +

{m.was} → {m.path.trim()}: its tracks move with it.

+ {/each} + {#each removed as f (f.path)} +

+ {f.path} will no longer be read{f.tracks > 0 + ? `; its ${plural(f.tracks, 'track is', 'tracks are')} marked missing at the next scan` + : ''}. +

+ {/each} +
+
+ {#if blank} +

Fill in or remove the empty folder.

+ {:else if rows.length === 0} +

The library needs at least one folder.

+ {/if} + + +
+
+{/if} diff --git a/web/src/lib/components/LibraryFoldersSetting.test.ts b/web/src/lib/components/LibraryFoldersSetting.test.ts new file mode 100644 index 00000000..c3134026 --- /dev/null +++ b/web/src/lib/components/LibraryFoldersSetting.test.ts @@ -0,0 +1,82 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor } from '@testing-library/svelte'; +import type { LibraryFolders } from '#lib/api/admin.js'; + +vi.mock('#lib/api/admin.js', () => ({ + getLibraryFolders: vi.fn(), + updateLibraryFolders: vi.fn() +})); + +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import LibraryFoldersSetting from './LibraryFoldersSetting.svelte'; +import { getLibraryFolders, updateLibraryFolders } from '#lib/api/admin.js'; +import { pushToast } from '#lib/stores/toast.svelte.js'; + +// The mount moved: the server can no longer see /music. +const before: LibraryFolders = { + saved: false, + folders: [{ path: '/music', present: false, tracks: 12000 }] +}; + +afterEach(() => vi.clearAllMocks()); + +async function renderSetting(f: LibraryFolders = before) { + vi.mocked(getLibraryFolders).mockResolvedValue(f); + const onSaved = vi.fn(); + render(LibraryFoldersSetting, { onSaved }); + await screen.findByTestId('library-folders'); + return onSaved; +} + +describe('LibraryFoldersSetting', () => { + test('lists the folders, says where they come from, and flags one the server cannot see', async () => { + await renderSetting(); + expect(screen.getByTestId('library-folders').textContent).toBe('/music'); + expect(screen.getByText(/MINSTREL_LIBRARY_SCAN_PATHS/)).toBeInTheDocument(); + expect(screen.getByTestId('folder-missing').textContent).toMatch(/can't see \/music, which holds 12,000 tracks/); + }); + + test('changing a folder moves its tracks there', async () => { + const onSaved = await renderSetting(); + vi.mocked(updateLibraryFolders).mockResolvedValue({ + saved: true, + moved: 12000, + folders: [{ path: '/data/music', present: true, tracks: 12000 }] + }); + await fireEvent.click(screen.getByRole('button', { name: 'Change' })); + await fireEvent.input(screen.getByRole('textbox', { name: 'Folder 1' }), { target: { value: '/data/music' } }); + expect(screen.getByTestId('folder-changes').textContent).toMatch(/\/music → \/data\/music: its tracks move with it/); + + await fireEvent.click(screen.getByRole('button', { name: 'Save' })); + await waitFor(() => + expect(updateLibraryFolders).toHaveBeenCalledWith([{ path: '/data/music', was: '/music' }]) + ); + await waitFor(() => expect(onSaved).toHaveBeenCalled()); + expect(pushToast).toHaveBeenCalledWith(expect.stringMatching(/12,000 tracks moved/)); + expect((await screen.findByTestId('library-folders')).textContent).toBe('/data/music'); + }); + + test('a folder removed warns that its tracks will be marked missing', async () => { + await renderSetting({ + saved: true, + folders: [ + { path: '/music', present: true, tracks: 5 }, + { path: '/more', present: true, tracks: 1 } + ] + }); + await fireEvent.click(screen.getByRole('button', { name: 'Change' })); + await fireEvent.click(screen.getByRole('button', { name: 'Remove folder 2' })); + expect(screen.getByTestId('folder-changes').textContent).toMatch( + /\/more will no longer be read; its 1 track is marked missing/ + ); + }); + + test('a blank folder cannot be saved', async () => { + await renderSetting(); + await fireEvent.click(screen.getByRole('button', { name: 'Change' })); + await fireEvent.click(screen.getByRole('button', { name: 'Add folder' })); + expect(screen.getByRole('button', { name: 'Save' })).toBeDisabled(); + expect(screen.getByText(/Fill in or remove the empty folder/)).toBeInTheDocument(); + }); +}); diff --git a/web/src/lib/components/LikeButton.svelte b/web/src/lib/components/LikeButton.svelte index 34fffcf7..b4b1df5c 100644 --- a/web/src/lib/components/LikeButton.svelte +++ b/web/src/lib/components/LikeButton.svelte @@ -1,4 +1,5 @@ -
-
-

Loudness analysis

-

- Measures how loud each track is, so playback can even out the volume between tracks. It runs - in the background; until a track is measured, it plays at its own volume. -

-
- + {#if coverage && coverage.total > 0} -
- - {coverage.measured.toLocaleString()} of {coverage.total.toLocaleString()} tracks measured - - {#if coverage.pending > 0} - · - {coverage.pending.toLocaleString()} pending - {/if} - {#if !coverage.enabled && coverage.pending > 0} - · - paused while analysis is off - {/if} - {#if coverage.silent > 0} - · - - {coverage.silent.toLocaleString()} silent - - {/if} - {#if coverage.unreadable > 0} - · - - {coverage.unreadable.toLocaleString()} unreadable - - {/if} + {@const c = coverage} +
+ + {#snippet hint()} + {c.measured.toLocaleString()} of {c.total.toLocaleString()} tracks measured + {#if c.pending > 0}{` · ${c.pending.toLocaleString()} pending`}{/if} + {#if !c.enabled && c.pending > 0}{' · paused while analysis is off'}{/if} + {#if c.silent > 0} + {` · ${c.silent.toLocaleString()} silent`} + {/if} + {#if c.unreadable > 0} + {` · ${c.unreadable.toLocaleString()} unreadable`} + {/if} + {/snippet} +
{/if} {#if loadError} -

+

Couldn't load loudness analysis settings.

{:else if form === null} -

Loading…

+

Loading…

{:else} - - - - - - - {#if !valid} -
- {#if !concurrencyOk}

Files analyzed at once must be a whole number from 1 to 8.

{/if} - {#if !cacheOk}

Speaker cache must be a whole number of MB from 256 to 65536.

{/if} -
- {/if} - -
-
{/if} -
+ diff --git a/web/src/lib/components/MobileAppDownload.svelte b/web/src/lib/components/MobileAppDownload.svelte index 519b3219..f0c0a4a5 100644 --- a/web/src/lib/components/MobileAppDownload.svelte +++ b/web/src/lib/components/MobileAppDownload.svelte @@ -1,29 +1,26 @@ {#if info} -
-

Mobile app

-

- Install the Android app paired to this server. Sign in with the same account. -

- + - - - Get the Android app - - {info.version}{info.sizeBytes > 0 ? ` · ${formatSize(info.sizeBytes)}` : ''} - - - -
+
+ diff --git a/web/src/lib/components/NotificationSettings.svelte b/web/src/lib/components/NotificationSettings.svelte index c068d6fd..7f5b4a15 100644 --- a/web/src/lib/components/NotificationSettings.svelte +++ b/web/src/lib/components/NotificationSettings.svelte @@ -13,7 +13,7 @@ import { user } from '#lib/auth/store.svelte.js'; import { pushToast } from '#lib/stores/toast.svelte.js'; - // Per-user notification settings (M489): one row per kind, a toggle per + // Per-user notification settings: one row per kind, a toggle per // channel. Toggles are optimistic; a failed save puts the switch back. const client = useQueryClient(); @@ -30,7 +30,7 @@ // Bumped by every local change, so a save resolving after a newer toggle // does not overwrite it, and a failed save only reverts when nothing newer - // happened (snippet #5106's generation counter). + // happened. let generation = 0; function commit(next: NotificationSettings) { @@ -80,53 +80,62 @@ } -{#snippet rows(list: NotificationKindSetting[])} -
- - {#each channels as c (c.key)} - {c.label} - {/each} - {#each list as row (row.kind)} - {label(row.kind)} - {#each channels as c (c.key)} - - toggle(row, c.key)} - /> - - {/each} - {/each} -
+{#snippet row(r: NotificationKindSetting)} + {label(r.kind)} + {#each channels as c (c.key)} + + toggle(r, c.key)} + /> + + {/each} {/snippet} -
-

Notifications

+ +
+
+

Notifications

+

What reaches you, and where. Phone and email follow the inbox.

+
- {#if !settings} -

Loading…

- {:else} - {@render rows(forEveryone)} - - {#if forAdmins.length > 0} -

Library health

- {@render rows(forAdmins)} - {/if} - - {#if !settings.email_available} -

- {#if settings.email_unavailable_reason === 'no_address'} - Email is off: add an email address in your profile. - {:else if user.value?.is_admin} - Email is off: SMTP isn't set up. - {:else} - Email is off: this server doesn't send email. +

+ {#if !settings} +

Loading…

+ {:else} +
+ + {#each channels as c (c.key)} + {c.label} + {/each} + {#each forEveryone as r (r.kind)} + {@render row(r)} + {/each} + {#if forAdmins.length > 0} +

Library health

+ {#each forAdmins as r (r.kind)} + {@render row(r)} + {/each} {/if} -

+
+ + {#if !settings.email_available} +

+ {#if settings.email_unavailable_reason === 'no_address'} + Email is off: add an email address in your profile. + {:else if user.value?.is_admin} + Email is off: SMTP isn't set up. + {:else} + Email is off: this server doesn't send email. + {/if} +

+ {/if} {/if} - {/if} +
diff --git a/web/src/lib/components/NotificationSettings.test.ts b/web/src/lib/components/NotificationSettings.test.ts index f644de0e..ee61f279 100644 --- a/web/src/lib/components/NotificationSettings.test.ts +++ b/web/src/lib/components/NotificationSettings.test.ts @@ -108,7 +108,7 @@ describe('NotificationSettings', () => { expect(screen.getByRole('checkbox', { name: 'Request approved: Email' })).toBeDisabled(); expect(screen.getByRole('link', { name: 'add an email address in your profile' })).toHaveAttribute( 'href', - '#profile' + '/settings/account#profile' ); }); @@ -123,7 +123,7 @@ describe('NotificationSettings', () => { render(NotificationSettings); expect(screen.getByRole('link', { name: "SMTP isn't set up" })).toHaveAttribute( 'href', - '/admin/integrations' + '/admin/email' ); }); }); diff --git a/web/src/lib/components/PageColumn.svelte b/web/src/lib/components/PageColumn.svelte index 907080f0..faab08b3 100644 --- a/web/src/lib/components/PageColumn.svelte +++ b/web/src/lib/components/PageColumn.svelte @@ -1,9 +1,8 @@ -
-
-

Public address

-

- The address people use to reach Minstrel. Password-reset emails link here. -

-
- + {#if loadError} -

- Couldn't load network settings. +

+ Couldn't load the public address.

{:else if saved === null} -

Loading…

+

Loading…

{:else} -
- - {#if here && value.trim() !== here} - - {/if} - -
- + + {#if here && value.trim() !== here} + + + + {/if} {#if !saved} -

-

+ Not set, so password-reset emails are not being sent.

{/if} {/if} -
+ diff --git a/web/src/lib/components/QueueList.svelte b/web/src/lib/components/QueueList.svelte index 7c18da87..2fbe9b9f 100644 --- a/web/src/lib/components/QueueList.svelte +++ b/web/src/lib/components/QueueList.svelte @@ -1,4 +1,5 @@ + +
+

Recommendation metrics

+

+ How plays launched from each recommendation surface land, over the last + {$metrics.data?.window_days ?? 30} days — compared against the baseline of + music you picked yourself. Deltas are percentage points vs that baseline. +

+ {#if $metrics.isPending} +

Loading…

+ {:else if $metrics.isError} +

Couldn't load metrics.

+ {:else if $metrics.data && ($metrics.data.groups.length > 0 || $metrics.data.baseline)} + {@const baseline = $metrics.data.baseline} + {#if baseline} +

+ Baseline — manual plays: + + {baseline.plays} plays · {pct(baseline.skip_rate)} skip · + {pct(baseline.avg_completion)} completion + +

+ {:else} +

+ No manual plays in this window yet, so deltas are hidden — raw rates only. +

+ {/if} + {#each $metrics.data.groups as group (group.intent)} +
+

{group.label}

+

{intentHints[group.intent]}

+ + + + + + + + + + + {#each group.surfaces as m (m.key)} + + + + + + + + {#if expandedBreakdowns.has(m.key)} + {#each m.breakdown ?? [] as b (b.key)} + + + + + + + {/each} + {/if} + {/each} + +
SurfacePlaysSkip rateAvg completion
+ {#if (m.breakdown ?? []).length > 0} + + {:else} + {m.label} + {/if}{#if m.low_confidence}· low data{/if} + {m.plays} + {pct(m.skip_rate)} + {#if m.skip_delta} + {deltaText(m.skip_delta)} + {/if} + + {pct(m.avg_completion)} + {#if m.completion_delta} + {deltaText(m.completion_delta)} + {/if} +
+ ↳ {b.label}{#if b.low_confidence}· low data{/if} + {b.plays} + {pct(b.skip_rate)} + {#if b.skip_delta} + {deltaText(b.skip_delta)} + {/if} + + {pct(b.avg_completion)} + {#if b.completion_delta} + {deltaText(b.completion_delta)} + {/if} +
+
+ {/each} +

+ Deltas compare each surface with your manual plays. A delta marked + ≈ is smaller than its own margin of error at this + sample size — it can't be told apart from no difference, however big it looks. + Hover any delta for its range. +

+ {:else} +

+ No plays recorded yet. Play something from For You, Discover, or a mix. +

+ {/if} +
diff --git a/web/src/lib/components/RecommendationMetrics.test.ts b/web/src/lib/components/RecommendationMetrics.test.ts new file mode 100644 index 00000000..d3a7d465 --- /dev/null +++ b/web/src/lib/components/RecommendationMetrics.test.ts @@ -0,0 +1,163 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor } from '@testing-library/svelte'; + +// The card lives on Library › Your listening. + +// Mutable holder so individual tests can inject populated metrics; +// vi.mock is hoisted, hence vi.hoisted for the shared reference. +const metricsMock = vi.hoisted(() => ({ + data: { window_days: 30, baseline: null, groups: [] } as unknown +})); + +vi.mock('#lib/api/metrics.js', () => ({ + createRecommendationMetricsQuery: () => ({ + subscribe: (run: (v: unknown) => void) => { + run({ isPending: false, isError: false, data: metricsMock.data }); + return () => {}; + } + }) +})); + +import RecommendationMetrics from './RecommendationMetrics.svelte'; + +afterEach(() => { + metricsMock.data = { window_days: 30, baseline: null, groups: [] }; +}); + +describe('RecommendationMetrics (Library › Your listening)', () => { + const metric = (key: string, label: string, over: Record = {}) => ({ + key, + label, + plays: 30, + skips: 3, + skip_rate: 0.1, + avg_completion: 0.9, + low_confidence: false, + ...over + }); + + test('breakdown rows are collapsed by default and expand on toggle (#1249/#1270)', async () => { + metricsMock.data = { + window_days: 30, + baseline: metric('manual', 'Manual library plays', { plays: 100 }), + groups: [ + { + intent: 'go_to', + label: 'Go-to surfaces', + surfaces: [ + metric('for_you', 'For You', { + plays: 45, + breakdown: [ + metric('for_you_taste', 'Taste picks'), + metric('for_you_fresh', 'Fresh picks', { + plays: 10, + skip_rate: 0.4, + low_confidence: true + }), + metric('for_you_unattributed', 'Earlier plays', { plays: 5 }) + ] + }) + ] + } + ] + }; + render(RecommendationMetrics); + const toggle = await screen.findByRole('button', { name: /for you/i }); + // Collapsed by default: every stamping mix now carries a breakdown, + // so always-open sub-rows would swamp the surface-level view. + expect(toggle).toHaveAttribute('aria-expanded', 'false'); + expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument(); + + await fireEvent.click(toggle); + expect(toggle).toHaveAttribute('aria-expanded', 'true'); + expect(screen.getByText(/Taste picks/)).toBeInTheDocument(); + expect(screen.getByText(/Fresh picks/)).toBeInTheDocument(); + expect(screen.getByText(/Earlier plays/)).toBeInTheDocument(); + + await fireEvent.click(toggle); + expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument(); + }); + + // Deltas arrive from the server with a margin, and an indistinguishable one + // is marked with "≈" and dimmed rather than coloured, so a -12 on 59 plays + // does not look as solid as a -6 on 400. + test('a delta smaller than its margin is marked as indistinguishable', async () => { + metricsMock.data = { + window_days: 30, + baseline: metric('manual', 'Manual library plays', { plays: 400, skip_rate: 0.27 }), + groups: [ + { + intent: 'discovery', + label: 'Discovery mixes', + surfaces: [ + metric('discover', 'Discover', { + plays: 59, + skip_rate: 0.153, + // 13.3pp gap, but the margin at n=59 is wider than the gap. + skip_delta: { delta_pp: -13.3, margin_pp: 14.5, distinguishable: false }, + completion_delta: { delta_pp: 28.0, margin_pp: 12.1, distinguishable: true } + }) + ] + } + ] + }; + render(RecommendationMetrics); + await waitFor(() => expect(screen.getByText('Discover')).toBeInTheDocument()); + + // Targeted by test id rather than text: the legend below the table also + // contains a "≈", so matching on the glyph finds the explanation instead of + // the delta. (It did, on the first attempt at this test.) + const skip = screen.getByTestId('skip-delta-discover'); + expect(skip).toHaveTextContent('≈-13'); + expect(skip).toHaveAttribute('title', expect.stringContaining('not distinguishable from zero')); + // It must NOT be coloured as a real regression/improvement. + expect(skip.className).toContain('opacity-60'); + + // The completion delta clears its margin, so it renders plainly. + const completion = screen.getByTestId('completion-delta-discover'); + expect(completion).toHaveTextContent('+28'); + expect(completion.className).not.toContain('opacity-60'); + + // And the legend explains the glyph rather than leaving it a mystery. + expect(screen.getByText(/smaller than its own margin of error/i)).toBeInTheDocument(); + }); + + // A delta is omitted entirely when the samples are too thin for a margin to + // mean anything — the server decides that, and the cell must simply show the + // rate rather than a bare "0". + test('a surface with no delta shows its rate and nothing else', async () => { + metricsMock.data = { + window_days: 30, + baseline: metric('manual', 'Manual library plays', { plays: 400 }), + groups: [ + { + intent: 'go_to', + label: 'Go-to surfaces', + surfaces: [metric('radio', 'Radio', { plays: 1, skip_rate: 0 })] + } + ] + }; + render(RecommendationMetrics); + await waitFor(() => expect(screen.getByText('Radio')).toBeInTheDocument()); + expect(screen.queryByTestId('skip-delta-radio')).not.toBeInTheDocument(); + expect(screen.queryByTestId('completion-delta-radio')).not.toBeInTheDocument(); + }); + + test('surfaces without a breakdown render no toggle and no sub-rows', async () => { + metricsMock.data = { + window_days: 30, + baseline: null, + groups: [ + { + intent: 'go_to', + label: 'Go-to surfaces', + surfaces: [metric('radio', 'Radio')] + } + ] + }; + render(RecommendationMetrics); + await waitFor(() => expect(screen.getByText('Radio')).toBeInTheDocument()); + expect(screen.queryByRole('button', { name: /radio/i })).not.toBeInTheDocument(); + expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument(); + }); +}); diff --git a/web/src/lib/components/ReleaseProfilesCard.svelte b/web/src/lib/components/ReleaseProfilesCard.svelte new file mode 100644 index 00000000..9bb6fdd6 --- /dev/null +++ b/web/src/lib/components/ReleaseProfilesCard.svelte @@ -0,0 +1,233 @@ + + +{#snippet editor()} +
+ + +
+ Kinds of release +
+ {#each primaryTypes as t (t)} + + {/each} +
+
+ +
+ Also allow + + Leave these unticked for studio releases only. A live album is wanted only when Live is + ticked here. + +
+ {#each secondaryTypes as t (t)} + + {/each} +
+
+ + + +
+ + +
+
+{/snippet} + +
+
+

Release-type profiles

+

+ Each followed artist is on one profile, which says which of their new releases are + wanted. Requests for a specific album aren't filtered. +

+
+ + {#if query.isPending} +

Loading…

+ {:else if query.isError} +

Couldn't load the profiles.

+ {:else} +
    + {#each profiles as p (p.id)} +
  • +
    +
    +
    + {p.name} + {#if p.is_default} + Default + {/if} +
    +
    + {describe(p)} · {plural(p.follow_count, 'artist', 'artists')} +
    +
    + + {#if !p.is_default} + + {/if} +
    + {#if editing === p.id} + {@render editor()} + {/if} +
    • + {/each} +
+ + {#if editing === 'new'} + {@render editor()} + {:else} + + {/if} + {/if} +
diff --git a/web/src/lib/components/ReleaseProfilesCard.test.ts b/web/src/lib/components/ReleaseProfilesCard.test.ts new file mode 100644 index 00000000..f36d5e9d --- /dev/null +++ b/web/src/lib/components/ReleaseProfilesCard.test.ts @@ -0,0 +1,91 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor, within } from '@testing-library/svelte'; +import { mockQuery } from '../../test-utils/query'; +import type { ReleaseProfilesResponse } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', () => ({ + createReleaseProfilesQuery: vi.fn(), + createReleaseProfile: vi.fn().mockResolvedValue({}), + updateReleaseProfile: vi.fn().mockResolvedValue({}), + deleteReleaseProfile: vi.fn().mockResolvedValue(undefined) +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import ReleaseProfilesCard from './ReleaseProfilesCard.svelte'; +import { + createReleaseProfile, + createReleaseProfilesQuery, + deleteReleaseProfile, + updateReleaseProfile +} from '#lib/api/wanted.js'; + +const data: ReleaseProfilesResponse = { + profiles: [ + { id: 'p-std', name: 'Standard', primary_types: ['Album', 'EP', 'Single'], secondary_types: [], is_default: true, follow_count: 3 }, + { id: 'p-live', name: 'With live', primary_types: ['Album'], secondary_types: ['Live'], is_default: false, follow_count: 0 } + ], + primary_types: ['Album', 'EP', 'Single', 'Broadcast', 'Other'], + secondary_types: ['Compilation', 'Live', 'Remix'] +}; + +afterEach(() => vi.clearAllMocks()); + +function setup() { + vi.mocked(createReleaseProfilesQuery).mockReturnValue( + mockQuery({ data }) as unknown as ReturnType + ); + return render(ReleaseProfilesCard); +} + +describe('ReleaseProfilesCard', () => { + test('says what each profile wants', () => { + setup(); + const rows = screen.getAllByTestId('profile-row'); + expect(rows[0].textContent).toMatch(/studio only/i); + expect(rows[0].textContent).toMatch(/3 artists/i); + expect(rows[1].textContent).toMatch(/including live/i); + }); + + // The default can't be deleted; the server refuses it, so the card doesn't offer it. + test('the default profile has no delete button', () => { + setup(); + expect(screen.queryByRole('button', { name: /delete standard/i })).toBeNull(); + expect(screen.getByRole('button', { name: /delete with live/i })).toBeTruthy(); + }); + + test('editing ticks a secondary type and saves it', async () => { + setup(); + await fireEvent.click(screen.getByRole('button', { name: /edit standard/i })); + const editor = screen.getByTestId('profile-editor'); + await fireEvent.click(within(editor).getByLabelText('Live')); + await fireEvent.click(within(editor).getByRole('button', { name: /save profile/i })); + await waitFor(() => + expect(updateReleaseProfile).toHaveBeenCalledWith('p-std', { + name: 'Standard', + primary_types: ['Album', 'EP', 'Single'], + secondary_types: ['Live'], + is_default: true + }) + ); + }); + + test('a new profile is created', async () => { + setup(); + await fireEvent.click(screen.getByRole('button', { name: /new profile/i })); + const editor = screen.getByTestId('profile-editor'); + await fireEvent.input(within(editor).getByLabelText(/name/i), { target: { value: 'Everything live' } }); + await fireEvent.click(within(editor).getByLabelText('Live')); + await fireEvent.click(within(editor).getByRole('button', { name: /save profile/i })); + await waitFor(() => + expect(createReleaseProfile).toHaveBeenCalledWith( + expect.objectContaining({ name: 'Everything live', secondary_types: ['Live'] }) + ) + ); + }); + + test('delete removes a profile', async () => { + setup(); + await fireEvent.click(screen.getByRole('button', { name: /delete with live/i })); + await waitFor(() => expect(deleteReleaseProfile).toHaveBeenCalledWith('p-live')); + }); +}); diff --git a/web/src/lib/components/RequestRow.svelte b/web/src/lib/components/RequestRow.svelte new file mode 100644 index 00000000..9807734c --- /dev/null +++ b/web/src/lib/components/RequestRow.svelte @@ -0,0 +1,74 @@ + + + + +
  • + + +
    +
    + {r.kind[0].toUpperCase() + r.kind.slice(1)} + +
    +
    {title}
    +
    {meta}
    + {#if r.imported_album_count > 0 || r.imported_track_count > 0} +
    + {#if r.kind === 'artist'} + {plural(r.imported_album_count, 'album', 'albums')} + · {plural(r.imported_track_count, 'track', 'tracks')} ingested + {:else if r.kind === 'album'} + {plural(r.imported_track_count, 'track', 'tracks')} ingested + {:else} + Track ingested + {/if} +
    + {/if} + {@render details?.()} +
    + + {#if actions} +
    {@render actions()}
    + {/if} +
    • diff --git a/web/src/lib/components/RequestRow.test.ts b/web/src/lib/components/RequestRow.test.ts new file mode 100644 index 00000000..3da44fa4 --- /dev/null +++ b/web/src/lib/components/RequestRow.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, test } from 'vitest'; +import { render, screen } from '@testing-library/svelte'; +import type { MusicRequest } from '#lib/api/types.js'; +import RequestRow, { requestName } from './RequestRow.svelte'; + +function req(over: Partial = {}): MusicRequest { + return { + id: 'r1', + user_id: 'u1', + status: 'approved', + kind: 'album', + artist_mbid: 'art-mbid', + artist_name: 'Boards of Canada', + album_title: 'Geogaddi', + track_title: null, + requested_at: '2026-04-28T12:00:00Z', + updated_at: '2026-04-28T12:00:00Z', + imported_album_count: 0, + imported_track_count: 0, + ...over + }; +} + +const text = (el: HTMLElement) => (el.textContent ?? '').replace(/\s+/g, ' ').trim(); + +// The row both Requests pages draw: the admin queue and the +// requester's own list read the same request the same way. +describe('RequestRow', () => { + test('says what was asked for, where it stands, and who or when', () => { + render(RequestRow, { request: req(), meta: 'by Boards of Canada · 4/28/2026', testid: 'request-row' }); + const row = screen.getByTestId('request-row'); + expect(row.dataset.status).toBe('approved'); + expect(text(row)).toMatch(/^Album Approved · downloading Geogaddi by Boards of Canada · 4\/28\/2026$/); + }); + + test('says how much has arrived, in the words for each kind', () => { + const { unmount } = render(RequestRow, { + request: req({ kind: 'artist', imported_album_count: 3, imported_track_count: 41 }), + meta: '', + testid: 'r' + }); + expect(text(screen.getByTestId('ingest-progress'))).toBe('3 albums · 41 tracks ingested'); + unmount(); + + render(RequestRow, { request: req({ kind: 'track', track_title: 'Roygbiv', imported_track_count: 1 }), meta: '', testid: 'r' }); + expect(text(screen.getByTestId('ingest-progress'))).toBe('Track ingested'); + }); + + test('nothing arrived yet says nothing about it', () => { + render(RequestRow, { request: req(), meta: '', testid: 'r' }); + expect(screen.queryByTestId('ingest-progress')).toBeNull(); + }); + + test('a request with no title still has a name a screen reader can say', () => { + render(RequestRow, { request: req({ album_title: null }), meta: '', testid: 'r' }); + expect(text(screen.getByTestId('r'))).toMatch(/—/); + expect(requestName(req({ album_title: null }))).toBe('this album by Boards of Canada'); + expect(requestName(req({ kind: 'artist' }))).toBe('Boards of Canada'); + }); +}); diff --git a/web/src/lib/components/SearchSettingsCard.svelte b/web/src/lib/components/SearchSettingsCard.svelte new file mode 100644 index 00000000..f23bb25c --- /dev/null +++ b/web/src/lib/components/SearchSettingsCard.svelte @@ -0,0 +1,240 @@ + + + + {#if query.isPending} +

    Loading…

    + {:else if query.isError || !form || !query.data} +

    + Couldn't load the search settings. + +

    + {:else} + {@const data = query.data} + + {#snippet hint()}{lastPass(data)}{/snippet} + Wanted + + + edit({ auto_grab: e.currentTarget.checked })} + class="h-4 w-4 cursor-pointer accent-accent" + /> + + + edit({ search_interval_hours: Number(e.currentTarget.value) })} + class={field} + /> + hours + + + {#each data.tiers as t (t.tier)} +
    + + toggle(t.tier)} + class="h-4 w-4 cursor-pointer accent-accent" + /> + +
    + {/each} + + + Upgrades + + {#if data.scan_paths.length > 1} + + + + {:else if data.scan_paths.length === 1} + + {data.scan_paths[0]} + + {/if} + + edit({ seed_ratio: Number(e.currentTarget.value) })} + class={field} + /> + + + edit({ seed_days: Number(e.currentTarget.value) })} + class={field} + /> + days + + + edit({ stall_hours: Number(e.currentTarget.value) })} + class={field} + /> + hours + +
    + +
    + {/if} +     diff --git a/web/src/lib/components/SearchSettingsCard.test.ts b/web/src/lib/components/SearchSettingsCard.test.ts new file mode 100644 index 00000000..9d14858c --- /dev/null +++ b/web/src/lib/components/SearchSettingsCard.test.ts @@ -0,0 +1,127 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockQuery } from '../../test-utils/query'; +import type { SearchSettings } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', () => ({ + createSearchSettingsQuery: vi.fn(), + putSearchSettings: vi.fn() +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import SearchSettingsCard from './SearchSettingsCard.svelte'; +import { createSearchSettingsQuery, putSearchSettings } from '#lib/api/wanted.js'; + +const settings: SearchSettings = { + auto_grab: true, + search_interval_hours: 24, + allowed_qualities: ['lossless', 'lossy_high'], + tiers: [ + { tier: 'lossless', label: 'Lossless', what: 'FLAC, ALAC' }, + { tier: 'lossy_high', label: 'High quality lossy', what: 'MP3 320 or V0' }, + { tier: 'unknown', label: 'Unknown', what: 'Nothing says what format it is' } + ], + last_pass_at: new Date(Date.now() - 5 * 60_000).toISOString(), + last_pass: { looked: 12, grabbed: 3, waiting: 2, nothing: 7, failed: 0, paced: true }, + import_root: '', + scan_paths: ['/music', '/music2'], + seed_ratio: 2, + seed_days: 14, + stall_hours: 6, + upgrade_until: 'lossy_high' +}; + +afterEach(() => vi.clearAllMocks()); + +function setup(s: SearchSettings = settings) { + vi.mocked(createSearchSettingsQuery).mockReturnValue( + mockQuery({ data: s }) as unknown as ReturnType + ); + vi.mocked(putSearchSettings).mockImplementation(async (body) => ({ ...s, ...body })); + return render(SearchSettingsCard); +} + +describe('SearchSettingsCard', () => { + test('shows the settings and what the last pass did', () => { + setup(); + expect((screen.getByLabelText(/take the best release without asking/i) as HTMLInputElement).checked).toBe(true); + expect((screen.getByLabelText(/search again after/i) as HTMLInputElement).value).toBe('24'); + expect((screen.getByLabelText(/^unknown/i) as HTMLInputElement).checked).toBe(false); + const last = screen.getByTestId('last-pass').textContent ?? ''; + expect(last).toMatch(/searched 12, took 3, 2 waiting for you, 7 with nothing found\./); + expect(last).toMatch(/searches per hour/); + }); + + test('saves what changed', async () => { + setup(); + const save = screen.getByRole('button', { name: /^save$/i }) as HTMLButtonElement; + expect(save.disabled).toBe(true); + await fireEvent.click(screen.getByLabelText(/take the best release without asking/i)); + await fireEvent.click(screen.getByLabelText(/^unknown/i)); + await fireEvent.click(save); + await waitFor(() => + expect(putSearchSettings).toHaveBeenCalledWith({ + auto_grab: false, + search_interval_hours: 24, + allowed_qualities: ['lossless', 'lossy_high', 'unknown'], + import_root: '', + seed_ratio: 2, + seed_days: 14, + stall_hours: 6, + upgrade_until: 'lossy_high' + }) + ); + }); + + test('new music is filed into the chosen library folder, the first by default', async () => { + setup(); + const folder = screen.getByLabelText(/file new music into/i) as HTMLSelectElement; + expect(folder.value).toBe('/music'); + await fireEvent.change(folder, { target: { value: '/music2' } }); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + await waitFor(() => + expect(putSearchSettings).toHaveBeenCalledWith(expect.objectContaining({ import_root: '/music2' })) + ); + }); + + test('with one library folder there is nothing to choose', () => { + setup({ ...settings, scan_paths: ['/music'] }); + expect(screen.queryByLabelText(/file new music into/i)).toBeNull(); + expect(screen.getByText('/music')).toBeTruthy(); + }); + + test('seeding and giving up on a stalled download are set here', async () => { + setup(); + expect((screen.getByLabelText(/seed ratio/i) as HTMLInputElement).value).toBe('2'); + await fireEvent.input(screen.getByLabelText(/seed ratio/i), { target: { value: '1.5' } }); + await fireEvent.input(screen.getByLabelText(/seed days/i), { target: { value: '7' } }); + await fireEvent.input(screen.getByLabelText(/give up on a stalled download/i), { target: { value: '0' } }); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + await waitFor(() => + expect(putSearchSettings).toHaveBeenCalledWith( + expect.objectContaining({ seed_ratio: 1.5, seed_days: 7, stall_hours: 0 }) + ) + ); + }); + + test('upgrades go up to the chosen quality, or are switched off', async () => { + setup(); + const until = screen.getByLabelText(/upgrade until/i) as HTMLSelectElement; + expect(until.value).toBe('lossy_high'); + expect([...until.options].map((o) => o.textContent)).toEqual([ + "Don't upgrade", + 'Lossless', + 'High quality lossy' + ]); + await fireEvent.change(until, { target: { value: '' } }); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + await waitFor(() => + expect(putSearchSettings).toHaveBeenCalledWith(expect.objectContaining({ upgrade_until: '' })) + ); + }); + + test('before the first pass it says so', () => { + setup({ ...settings, last_pass_at: null, last_pass: {} }); + expect(screen.getByTestId('last-pass').textContent).toMatch(/hasn’t run yet/); + }); +}); diff --git a/web/src/lib/components/SectionRail.svelte b/web/src/lib/components/SectionRail.svelte new file mode 100644 index 00000000..e08cb471 --- /dev/null +++ b/web/src/lib/components/SectionRail.svelte @@ -0,0 +1,61 @@ + + + + + diff --git a/web/src/lib/components/ServerVersion.svelte b/web/src/lib/components/ServerVersion.svelte index b855a50f..9cb4ecfc 100644 --- a/web/src/lib/components/ServerVersion.svelte +++ b/web/src/lib/components/ServerVersion.svelte @@ -2,14 +2,13 @@ import { onMount } from 'svelte'; // Reads /healthz on mount and surfaces the deployed server version - // as understated text. Useful when verifying which image a container - // is actually running (came up debugging the in-app update flow when - // it wasn't obvious whether v2026.05.10.0 or .1 was deployed). + // as understated text, for verifying which image a container is + // actually running. // - // This is now the ONLY place an operator can see which build they are on. - // Image tags stopped carrying the version on 2026-09-10 — :latest and :dev - // are rolling names and : answers "which commit", not "which build" — - // so the server's self-report is the answer. + // This is the ONLY place an admin can see which build they are on: image + // tags do not carry the version — :latest and :dev are rolling names and + // : answers "which commit", not "which build" — so the server's + // self-report is the answer. // // The channel is shown BESIDE the version, never spliced into it: the same // commit built on both lanes reports an identical version and differs only @@ -42,8 +41,11 @@ }); + {#if version} -

    - Server {version}{#if channel} · {channel}{/if} -

    + + {version}{#if channel} · {channel}{/if} + +{:else} + — {/if} diff --git a/web/src/lib/components/Shell.svelte b/web/src/lib/components/Shell.svelte index f534c562..1e8557d9 100644 --- a/web/src/lib/components/Shell.svelte +++ b/web/src/lib/components/Shell.svelte @@ -1,7 +1,7 @@ e.key === 'Escape' && (menuOpen = false)} /> diff --git a/web/src/lib/components/Shell.test.ts b/web/src/lib/components/Shell.test.ts index f7b658df..c4ce25ec 100644 --- a/web/src/lib/components/Shell.test.ts +++ b/web/src/lib/components/Shell.test.ts @@ -33,6 +33,7 @@ vi.mock('#lib/api/notifications.js', async () => { createNotificationsQuery: () => mockQuery({ data: { items: [], unread_count: 0 } }), markNotificationRead: vi.fn(), markAllNotificationsRead: vi.fn(), + readAllCutoff: () => undefined, badgeLabel: () => '' }; }); @@ -67,6 +68,21 @@ describe('Shell', () => { expect(screen.getByRole('link', { name: 'Discover' })).toHaveAttribute('href', '/discover'); }); + test('admins get Collection after Discover; others do not', () => { + const primary = () => + screen + .getAllByRole('link') + .filter((el) => el.closest('nav[aria-label="Primary"]') !== null) + .map((el) => el.textContent?.trim()); + const { unmount } = render(Shell); + expect(primary()).not.toContain('Collection'); + unmount(); + userState.current = { id: '1', username: 'alice', is_admin: true }; + render(Shell); + expect(primary()).toEqual(['Home', 'Library', 'Discover', 'Collection']); + expect(screen.getByRole('link', { name: 'Collection' })).toHaveAttribute('href', '/collection'); + }); + test('main nav omits Artists/Albums/Liked/History/Playlists/Search/Settings/Admin', () => { userState.current = { id: '1', username: 'alice', is_admin: true }; render(Shell); diff --git a/web/src/lib/components/StatusPill.svelte b/web/src/lib/components/StatusPill.svelte index e5636a5c..6a890fb7 100644 --- a/web/src/lib/components/StatusPill.svelte +++ b/web/src/lib/components/StatusPill.svelte @@ -1,14 +1,14 @@ diff --git a/web/src/lib/components/StatusPill.test.ts b/web/src/lib/components/StatusPill.test.ts index 6d8d1e08..7f4d7632 100644 --- a/web/src/lib/components/StatusPill.test.ts +++ b/web/src/lib/components/StatusPill.test.ts @@ -2,7 +2,7 @@ import { describe, expect, test } from 'vitest'; import { render, screen } from '@testing-library/svelte'; import StatusPill from './StatusPill.svelte'; -// Voice-rule labels and semantic tones per spec §6 / project_design_system.md. +// Labels in Minstrel's voice, and their semantic tones. // Each row is the contract: input status -> rendered label + data-tone attr. const CASES = [ { status: 'pending', label: 'Awaiting review', tone: 'warning' }, diff --git a/web/src/lib/components/SuggestionFeed.svelte b/web/src/lib/components/SuggestionFeed.svelte index 7ed71ee4..5f00ee18 100644 --- a/web/src/lib/components/SuggestionFeed.svelte +++ b/web/src/lib/components/SuggestionFeed.svelte @@ -45,11 +45,11 @@ } // "Matches your taste in shoegaze and dream pop." — the reason line when the - // candidate's own tags overlap the taste profile (#2377). Preferred over seed + // candidate's own tags overlap the taste profile. Preferred over seed // attribution because it describes the MUSIC ("sounds like what you like") - // rather than the graph ("adjacent to something you played"), which is the - // whole point of slice 6. Falls back when there are no matched tags, which is - // the common case: coverage is partial by nature (#2376). + // rather than the graph ("adjacent to something you played"). Falls back + // when there are no matched tags, which is the common case: coverage is + // partial by nature. function reasonText(s: ArtistSuggestion): string { const tags = s.matched_tags ?? []; if (tags.length === 0) return attributionText(s.attribution); @@ -94,14 +94,14 @@ try { await createRequest({ kind: 'artist', - lidarr_artist_mbid: s.mbid, + artist_mbid: s.mbid, artist_name: s.name }); optimisticRequested = withMbid(optimisticRequested, s.mbid, true); // The server-side filter hides this candidate on next refetch. await client.invalidateQueries({ queryKey: qk.suggestions() }); } catch { - // Swallow for v1; the SPA will refetch on next mount and the card + // Swallowed: the SPA will refetch on next mount and the card // stays requestable so the user can retry. } } @@ -148,9 +148,9 @@ {#if !query.isPending && suggestions.length === 0}

    {snoozes.length === 0 diff --git a/web/src/lib/components/SuggestionFeed.test.ts b/web/src/lib/components/SuggestionFeed.test.ts index b7e39e5e..f22e6c4f 100644 --- a/web/src/lib/components/SuggestionFeed.test.ts +++ b/web/src/lib/components/SuggestionFeed.test.ts @@ -112,7 +112,7 @@ describe('SuggestionFeed', () => { await fireEvent.click(screen.getByRole('button', { name: /request outsider/i })); expect(createRequest).toHaveBeenCalledWith({ kind: 'artist', - lidarr_artist_mbid: 'mb1', + artist_mbid: 'mb1', artist_name: 'Outsider' }); expect(invalidateMock).toHaveBeenCalled(); @@ -138,7 +138,7 @@ describe('SuggestionFeed snooze (#2375)', () => { render(SuggestionFeed); await fireEvent.click(screen.getByRole('button', { name: /not right now/i })); // Still on screen — the disappearance happens on refetch, not under the - // cursor (rule #24). + // cursor. expect(screen.getByText('Outsider')).toBeInTheDocument(); expect(screen.getByRole('button', { name: /bring outsider back/i })).toBeInTheDocument(); expect(screen.getByRole('status')).toHaveTextContent('Not right now'); @@ -210,10 +210,10 @@ describe('SuggestionFeed snooze (#2375)', () => { expect(screen.getByText(/back in 14 days/i)).toBeInTheDocument(); }); - // Pins the days→months seam. The singular branch was originally dead code - // here too: the threshold (45) sat above the divisor (30), so no day count - // could round to one month without hitting the days branch first. Kept in - // lockstep with SuggestionSnoozeRefTest on Android. + // Pins the days→months seam: with a threshold above the divisor (45 vs 30), + // no day count could round to one month without hitting the days branch + // first, and the singular branch would be dead code. Kept in lockstep with + // SuggestionSnoozeRefTest on Android. test('the days-to-months boundary sits at 30 days, so "a month" is reachable', () => { setSuggestions([]); setSnoozes([ diff --git a/web/src/lib/components/TabStrip.svelte b/web/src/lib/components/TabStrip.svelte index 5f329e03..6317e303 100644 --- a/web/src/lib/components/TabStrip.svelte +++ b/web/src/lib/components/TabStrip.svelte @@ -1,7 +1,7 @@ + +

    + {#if query.isPending} +

    Loading what the last search found…

    + {:else if query.isError} +

    Couldn't load the releases.

    + {:else if items.length === 0} +

    + Nothing kept from a search yet. Search now, or wait for the schedule. +

    + {:else} +

    + {plural(items.length, 'release', 'releases')} found {relativeTime(items[0].searched_at)}, best first. +

    +
      + {#each items as c (c.key)} +
    • +
      +
      {c.name}
      +
      {meta(c)}
      +
      + + {c.downloading ? 'Downloading' : verdictLabel[c.verdict]} + + {c.reason} +
      +
      +
      + {#if c.verdict === 'blocked'} + + {:else if !c.downloading} + + {#if c.info_hash} + + {/if} + {/if} +
      +
      • + {/each} +
    + {/if} +
    diff --git a/web/src/lib/components/WantCandidates.test.ts b/web/src/lib/components/WantCandidates.test.ts new file mode 100644 index 00000000..3c7fcecf --- /dev/null +++ b/web/src/lib/components/WantCandidates.test.ts @@ -0,0 +1,90 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockQuery } from '../../test-utils/query'; +import type { Candidate } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', () => ({ + createCandidatesQuery: vi.fn(), + grabCandidate: vi.fn().mockResolvedValue({ added: true, refused: false, gone: false, unconfigured: false, kept: false, replaced: false }), + blockCandidate: vi.fn().mockResolvedValue(undefined), + unblockRelease: vi.fn().mockResolvedValue(undefined) +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import WantCandidates from './WantCandidates.svelte'; +import { blockCandidate, createCandidatesQuery, grabCandidate, unblockRelease } from '#lib/api/wanted.js'; +import { pushToast } from '#lib/stores/toast.svelte.js'; + +function cand(key: string, extra: Partial = {}): Candidate { + return { + key, + name: `Daft Punk - Random Access Memories (2013) [${key}]`, + info_hash: key, + indexer: 'Knaben', + size_bytes: 450_000_000, + seeders: 12, + files_count: 13, + format: 'FLAC', + quality: 'lossless', + year: 2013, + verdict: 'grab', + reason: 'The album, FLAC.', + reasons: [], + folded_from: [], + searched_at: new Date().toISOString(), + downloading: false, + ...extra + }; +} + +afterEach(() => vi.clearAllMocks()); + +function setup(items: Candidate[]) { + vi.mocked(createCandidatesQuery).mockReturnValue( + mockQuery({ data: items }) as unknown as ReturnType + ); + return render(WantCandidates, { wantId: 'w1', title: 'Random Access Memories' }); +} + +describe('WantCandidates', () => { + test('every release shows its verdict and why', () => { + setup([ + cand('aaaa'), + cand('bbbb', { verdict: 'rejected', reason: 'This is a drumless edition (“drumless”).' }) + ]); + expect(screen.getAllByTestId('candidate-row')).toHaveLength(2); + expect(screen.getByText('Would take')).toBeTruthy(); + expect(screen.getByText('Not the album')).toBeTruthy(); + expect(screen.getByText(/drumless edition/)).toBeTruthy(); + expect(screen.getAllByText(/FLAC · 450 MB · 12 seeders · Knaben/)).toHaveLength(2); + }); + + test('a release the verdict would take is grabbed; any other is grabbed anyway', async () => { + setup([cand('aaaa'), cand('bbbb', { verdict: 'review', reason: 'Nobody is sharing it right now.' })]); + await fireEvent.click(screen.getByRole('button', { name: /^grab daft punk.*\[aaaa\]/i })); + await waitFor(() => expect(grabCandidate).toHaveBeenCalledWith('w1', 'aaaa', false)); + await waitFor(() => expect(pushToast).toHaveBeenCalledWith('Taken for Random Access Memories.', 'info')); + await fireEvent.click(screen.getByRole('button', { name: /^grab anyway daft punk.*\[bbbb\]/i })); + await waitFor(() => expect(grabCandidate).toHaveBeenCalledWith('w1', 'bbbb', true)); + }); + + test('a release can be set aside, and allowed again', async () => { + setup([cand('aaaa'), cand('bbbb', { verdict: 'blocked', reason: 'You set this release aside.' })]); + await fireEvent.click(screen.getByRole('button', { name: /never take .*\[aaaa\]/i })); + await waitFor(() => expect(blockCandidate).toHaveBeenCalledWith('w1', 'aaaa')); + await fireEvent.click(screen.getByRole('button', { name: /allow .*\[bbbb\] again/i })); + await waitFor(() => expect(unblockRelease).toHaveBeenCalledWith('bbbb')); + expect(screen.queryByRole('button', { name: /grab.*\[bbbb\]/i })).toBeNull(); + }); + + test('the release downloading says so and offers nothing', () => { + setup([cand('aaaa', { downloading: true })]); + expect(screen.getByText('Downloading')).toBeTruthy(); + expect(screen.queryByRole('button', { name: /grab/i })).toBeNull(); + }); + + test('before any search, it says how to get one', () => { + setup([]); + expect(screen.getByText(/nothing kept from a search yet/i)).toBeTruthy(); + }); +}); diff --git a/web/src/lib/components/admin/ProviderGroup.svelte b/web/src/lib/components/admin/ProviderGroup.svelte new file mode 100644 index 00000000..405fe536 --- /dev/null +++ b/web/src/lib/components/admin/ProviderGroup.svelte @@ -0,0 +1,180 @@ + + + + {#if query.data} + {#each query.data.providers as p (p.id)} + {@const l = local[p.id]} + {@const r = results[p.id]} +
    + + {#snippet hint()} + {supplies(p)} + {#if note}{@render note(p)}{/if} + {/snippet} + {#if l} + + {/if} + + {#if l} +
    + {#if p.requires_api_key} + + {/if} + + {#if p.testable} + + {/if} + {#if r} + + {#if r.ok} + Works{r.duration_ms ? ` · answered in ${r.duration_ms} ms` : ''} + {:else} + Didn't work: {r.error} + {/if} + + {/if} +
    + {/if} +
    + {/each} + {:else if query.isPending} +

    Loading…

    + {:else if query.isError} +

    + Couldn't load the {noun}. + +

    + {:else} +

    None available.

    + {/if} +     diff --git a/web/src/lib/components/admin/SmtpSettings.svelte b/web/src/lib/components/admin/SmtpSettings.svelte new file mode 100644 index 00000000..5e99a36f --- /dev/null +++ b/web/src/lib/components/admin/SmtpSettings.svelte @@ -0,0 +1,146 @@ + + + + {#if smtpQ.isError && !loaded} +

    + Couldn't load the email settings, so they can't be saved yet. + +

    + {/if} + + + + + + + + + + + + + + + + + + + + + + + + +
    + + +
    +     diff --git a/web/src/lib/components/admin/SmtpSettings.test.ts b/web/src/lib/components/admin/SmtpSettings.test.ts new file mode 100644 index 00000000..ab52310b --- /dev/null +++ b/web/src/lib/components/admin/SmtpSettings.test.ts @@ -0,0 +1,69 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor } from '@testing-library/svelte'; +import { mockQuery } from '../../../test-utils/query'; +import type { SMTPConfig } from '#lib/api/admin.js'; + +vi.mock('@tanstack/svelte-query', async (orig) => { + const actual = (await orig()) as Record; + return { ...actual, useQueryClient: () => ({ invalidateQueries: vi.fn() }) }; +}); + +vi.mock('#lib/api/admin.js', () => ({ + createSMTPConfigQuery: vi.fn(), + updateSMTPConfig: vi.fn(), + testSMTPConfig: vi.fn() +})); + +import SmtpSettings from './SmtpSettings.svelte'; +import { createSMTPConfigQuery, updateSMTPConfig, testSMTPConfig } from '#lib/api/admin.js'; + +const configured: SMTPConfig = { + enabled: true, + host: 'smtp.example.com', + port: 587, + username: 'user@example.com', + password: '***', + from_address: 'noreply@example.com', + from_name: 'Minstrel', + use_tls: true +}; + +afterEach(() => vi.clearAllMocks()); + +function setup(q = mockQuery({ data: configured })) { + vi.mocked(createSMTPConfigQuery).mockReturnValue(q as never); + return render(SmtpSettings); +} + +describe('SmtpSettings', () => { + test('the saved settings fill the rows, and a saved password is kept unless one is typed', async () => { + setup(); + expect((screen.getByLabelText(/^server$/i) as HTMLInputElement).value).toBe('smtp.example.com'); + expect(screen.getByPlaceholderText(/saved · leave empty to keep it/i)).toBeInTheDocument(); + vi.mocked(updateSMTPConfig).mockResolvedValue(undefined as never); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + await waitFor(() => expect(updateSMTPConfig).toHaveBeenCalledWith(expect.objectContaining({ host: 'smtp.example.com', password: '' }))); + }); + + test('a typed password is sent', async () => { + setup(); + vi.mocked(updateSMTPConfig).mockResolvedValue(undefined as never); + await fireEvent.input(screen.getByLabelText(/^password$/i), { target: { value: 'newsecret' } }); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + await waitFor(() => expect(updateSMTPConfig).toHaveBeenCalledWith(expect.objectContaining({ password: 'newsecret' }))); + }); + + test('Send test email sends one, and with no address on file says where to set it', async () => { + setup(); + vi.mocked(testSMTPConfig).mockRejectedValueOnce({ code: 'no_email_on_file' }); + await fireEvent.click(screen.getByRole('button', { name: /send test email/i })); + await waitFor(() => expect(testSMTPConfig).toHaveBeenCalled()); + await waitFor(() => expect(screen.getByTestId('toast').textContent).toMatch(/set your email in settings › account/i)); + }); + + test('when the settings fail to load, Save is off, so defaults are never written over them', () => { + setup(mockQuery({ isError: true, error: { code: 'server_error' } })); + expect(screen.getByTestId('smtp-error')).toBeInTheDocument(); + expect((screen.getByRole('button', { name: /^save$/i }) as HTMLButtonElement).disabled).toBe(true); + }); +}); diff --git a/web/src/lib/components/reorder-drag.ts b/web/src/lib/components/reorder-drag.ts index bbb168e1..128eae21 100644 --- a/web/src/lib/components/reorder-drag.ts +++ b/web/src/lib/components/reorder-drag.ts @@ -1,5 +1,5 @@ // Drag-to-reorder for a row in a vertical list, shared by the playlist and -// queue rows (#5393). Wraps @neodrag/svelte with the three things a list +// queue rows. Wraps @neodrag/svelte with the three things a list // needs that neodrag does not do on its own: // // - The row moves in whole-row steps, so it visibly snaps into the slot it diff --git a/web/src/lib/components/settings/Segmented.svelte b/web/src/lib/components/settings/Segmented.svelte new file mode 100644 index 00000000..8e8999f3 --- /dev/null +++ b/web/src/lib/components/settings/Segmented.svelte @@ -0,0 +1,39 @@ + + +
    + {#each options as o, i (o.value)} + + {/each} +
    diff --git a/web/src/lib/components/settings/SettingRow.svelte b/web/src/lib/components/settings/SettingRow.svelte new file mode 100644 index 00000000..ef1b3f52 --- /dev/null +++ b/web/src/lib/components/settings/SettingRow.svelte @@ -0,0 +1,36 @@ + + +
    +
    + {#if forId} + + {:else} + {label} + {/if} + {#if typeof hint === 'function'} +

    {@render hint()}

    + {:else if hint} +

    {hint}

    + {/if} +
    + {#if children} +
    + {@render children()} +
    + {/if} +
    diff --git a/web/src/lib/components/settings/SettingsColumns.svelte b/web/src/lib/components/settings/SettingsColumns.svelte new file mode 100644 index 00000000..bc5f5be3 --- /dev/null +++ b/web/src/lib/components/settings/SettingsColumns.svelte @@ -0,0 +1,24 @@ + + +
    +
    + {@render main()} +
    + {#if side} +
    + {@render side()} +
    + {/if} +
    diff --git a/web/src/lib/components/settings/SettingsColumns.test.ts b/web/src/lib/components/settings/SettingsColumns.test.ts new file mode 100644 index 00000000..1c0187ca --- /dev/null +++ b/web/src/lib/components/settings/SettingsColumns.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, test } from 'vitest'; +import { render, screen } from '@testing-library/svelte'; +import { createRawSnippet } from 'svelte'; +import SettingsColumns from './SettingsColumns.svelte'; + +const snip = (id: string) => createRawSnippet(() => ({ render: () => `
    ` })); + +describe('SettingsColumns', () => { + test('main and side are two columns that sit side by side on a wide screen', () => { + render(SettingsColumns, { main: snip('main'), side: snip('side') }); + const grid = screen.getByTestId('main').parentElement!.parentElement!; + expect(grid.className).toMatch(/xl:grid-cols-2/); + expect(grid.children).toHaveLength(2); + expect(screen.getByTestId('side').parentElement!.parentElement).toBe(grid); + }); + + test('with only main, the groups keep to one column', () => { + render(SettingsColumns, { main: snip('only') }); + expect(screen.getByTestId('only').parentElement!.parentElement!.children).toHaveLength(1); + }); +}); diff --git a/web/src/lib/components/settings/SettingsGroup.svelte b/web/src/lib/components/settings/SettingsGroup.svelte new file mode 100644 index 00000000..3a7cdcfb --- /dev/null +++ b/web/src/lib/components/settings/SettingsGroup.svelte @@ -0,0 +1,26 @@ + + +
    +
    +

    {title}

    + {#if hint} +

    {hint}

    + {/if} +
    +
    + {@render children()} +
    +
    diff --git a/web/src/lib/components/wanted/BlockList.svelte b/web/src/lib/components/wanted/BlockList.svelte new file mode 100644 index 00000000..c30a43a4 --- /dev/null +++ b/web/src/lib/components/wanted/BlockList.svelte @@ -0,0 +1,95 @@ + + +{#if blocks.isPending} +

    Reading the blocked releases…

    +{:else if blocks.isError && !blocks.isFetchNextPageError} +

    Couldn't load the blocked releases.

    +{:else if items.length === 0} +
    + +

    No releases are blocked.

    +

    + A release you set aside, or one an import found wasn't the album, lands here so it's never + taken again. +

    +
    +{:else} +
      + {#each items as b (b.info_hash)} +
    • +
      +
      {b.name || b.info_hash}
      + {#if b.want_title} +
      For {b.want_artist} – {b.want_title}
      + {/if} +
      {by(b)} {relativeTime(b.blocked_at)}{until(b)}
      + {#if b.detail} +

      {b.detail}

      + {/if} +
      + +
      • + {/each} +
    + + blocks.fetchNextPage()} + /> +{/if} diff --git a/web/src/lib/components/wanted/BlockList.test.ts b/web/src/lib/components/wanted/BlockList.test.ts new file mode 100644 index 00000000..ae4817e0 --- /dev/null +++ b/web/src/lib/components/wanted/BlockList.test.ts @@ -0,0 +1,57 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockInfiniteQuery } from '../../../test-utils/query'; +import type { ReleaseBlock } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', () => ({ + createReleaseBlocksQuery: vi.fn(), + unblockRelease: vi.fn().mockResolvedValue(undefined) +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import BlockList from './BlockList.svelte'; +import { createReleaseBlocksQuery, unblockRelease } from '#lib/api/wanted.js'; + +const block: ReleaseBlock = { + info_hash: 'abc123', + name: 'Daft Punk - Random Access Memories (Drumless Edition)', + reason: 'import', + detail: 'The album tag says another version.', + blocked_at: new Date(Date.now() - 3_600_000).toISOString(), + expires_at: null, + want_title: 'Random Access Memories', + want_artist: 'Daft Punk' +}; + +afterEach(() => vi.clearAllMocks()); + +function setup(items: ReleaseBlock[]) { + vi.mocked(createReleaseBlocksQuery).mockReturnValue( + mockInfiniteQuery({ pages: [{ items, total: items.length, limit: 50, offset: 0 }] }) as unknown as ReturnType< + typeof createReleaseBlocksQuery + > + ); + return render(BlockList); +} + +describe('BlockList', () => { + test('a blocked release says why and can be allowed again', async () => { + setup([block]); + expect(screen.getByText(/refused on import/i)).toBeTruthy(); + expect(screen.getByText(/another version/)).toBeTruthy(); + expect(screen.getByText(/for daft punk – random access memories/i)).toBeTruthy(); + await fireEvent.click(screen.getByRole('button', { name: /allow .* again/i })); + await waitFor(() => expect(unblockRelease).toHaveBeenCalledWith('abc123')); + }); + + test('a release given up on says until when it is held back', () => { + setup([{ ...block, reason: 'stalled', detail: 'Nobody was sharing it.', expires_at: '2026-10-17T12:00:00Z' }]); + const line = screen.getByText(/given up on: it would not finish/i).textContent ?? ''; + expect(line).toMatch(/held back until/); + }); + + test('with nothing blocked it says so', () => { + setup([]); + expect(screen.getByText(/no releases are blocked/i)).toBeTruthy(); + }); +}); diff --git a/web/src/lib/components/wanted/CatalogueArtistCard.svelte b/web/src/lib/components/wanted/CatalogueArtistCard.svelte new file mode 100644 index 00000000..9384c0d0 --- /dev/null +++ b/web/src/lib/components/wanted/CatalogueArtistCard.svelte @@ -0,0 +1,46 @@ + + + +
    + +
    +
    +
    +
    +
    +
    {a.artist_name}
    +
    {a.held} / {a.albums} held
    + {#if notes.length} +
    {notes.join(' · ')}
    + {/if} +
    +     diff --git a/web/src/lib/components/wanted/CatalogueGrid.svelte b/web/src/lib/components/wanted/CatalogueGrid.svelte new file mode 100644 index 00000000..e95e09b6 --- /dev/null +++ b/web/src/lib/components/wanted/CatalogueGrid.svelte @@ -0,0 +1,72 @@ + + +{#if library && !library.listed_at} +

    + Your library's albums are being added to the catalogue. Albums whose tags don't name them are + matched on MusicBrainz first, by their artist, title and tracks, about one a second. +

    +{/if} + +{#if !artists.isPending && (items.length > 0 || q)} +
    + +
    +{/if} + +{#if artists.isPending} +

    Reading the catalogue…

    +{:else if artists.isError && !artists.isFetchNextPageError} +

    Couldn't load the catalogue.

    +{:else if items.length === 0 && q} +

    + No artist in the catalogue, or album of theirs, is named like '{q}'. +

    +{:else if items.length === 0} +
    + +

    The catalogue is empty.

    +

    + Your library's albums land here once it's been scanned, as do the albums users request and + the releases of artists Minstrel follows. +

    +
    +{:else} + a.sort_name || a.artist_name} + hasMore={!!artists.hasNextPage} + onLoadMore={() => artists.fetchNextPage()} + > + {#snippet item(a: WantArtist)} + + {/snippet} + + + artists.fetchNextPage()} + /> +{/if} diff --git a/web/src/lib/components/wanted/CatalogueGrid.test.ts b/web/src/lib/components/wanted/CatalogueGrid.test.ts new file mode 100644 index 00000000..5c49cf87 --- /dev/null +++ b/web/src/lib/components/wanted/CatalogueGrid.test.ts @@ -0,0 +1,86 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockInfiniteQuery } from '../../../test-utils/query'; +import type { LibraryListing, WantArtist, WantArtistsPage } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', () => ({ createWantArtistsQuery: vi.fn() })); + +import CatalogueGrid from './CatalogueGrid.svelte'; +import { createWantArtistsQuery } from '#lib/api/wanted.js'; + +const DAY = 24 * 3_600_000; +const ago = (ms: number) => new Date(Date.now() - ms).toISOString(); + +function artist(mbid: string, name: string, extra: Partial = {}): WantArtist { + return { + artist_mbid: mbid, + artist_name: name, + sort_name: name, + image_url: `/api/catalogue/art/artist/${mbid}`, + albums: 4, + held: 3, + wanted: 1, + missing: 0, + downloading: 0, + following: false, + ...extra + }; +} + +function setup( + items: WantArtist[] = [ + artist('blur', 'Blur', { following: true, downloading: 1 }), + artist('beatles', 'The Beatles', { sort_name: 'Beatles, The', albums: 13, held: 13, wanted: 0 }) + ], + library: LibraryListing = { listed_at: ago(DAY), unlisted: 0 } +) { + const page: WantArtistsPage = { + items, + total: items.length, + limit: 96, + offset: 0, + counts: { artists: items.length, wanted: 1, missing: 0, held: 16, upgrades: 0, following: 1, activity: 1, attention: 0, blocked: 0, unmatched: 0 }, + library + }; + vi.mocked(createWantArtistsQuery).mockReturnValue( + mockInfiniteQuery({ pages: [page] }) as unknown as ReturnType + ); + return render(CatalogueGrid); +} + +afterEach(() => vi.clearAllMocks()); + +describe('CatalogueGrid', () => { + test('each artist is a card: how many albums are held, and whether they are followed', () => { + setup(); + const cards = screen.getAllByTestId('catalogue-artist'); + expect(cards.map((c) => c.getAttribute('href'))).toEqual(['/collection/artist/blur', '/collection/artist/beatles']); + expect(cards[0].textContent?.replace(/\s+/g, ' ')).toMatch(/Blur 3 \/ 4 held Following · 1 downloading/); + const bar = screen.getByRole('progressbar', { name: '13 of 13 albums held' }); + expect(bar.getAttribute('aria-valuenow')).toBe('13'); + expect((bar.firstElementChild as HTMLElement).style.width).toBe('100%'); + }); + + test('the alphabet rail files an artist under their sort name', () => { + setup(); + expect(document.getElementById('alpha-B')?.textContent).toMatch(/Blur/); + }); + + test('the filter asks the server for artists by name or album', async () => { + setup(); + await fireEvent.input(screen.getByRole('searchbox', { name: /filter artists and albums/i }), { + target: { value: 'parklife' } + }); + await waitFor(() => expect(createWantArtistsQuery).toHaveBeenLastCalledWith('parklife')); + }); + + test('an empty catalogue says what lands there', () => { + setup([]); + expect(screen.getByText(/the catalogue is empty/i)).toBeTruthy(); + }); + + test('says while the library is being listed', () => { + setup([], { listed_at: null, unlisted: 0 }); + expect(screen.getByTestId('library-listing').textContent).toMatch(/being added to the catalogue/i); + }); +}); diff --git a/web/src/lib/components/wanted/CollectionRail.svelte b/web/src/lib/components/wanted/CollectionRail.svelte new file mode 100644 index 00000000..56a12be5 --- /dev/null +++ b/web/src/lib/components/wanted/CollectionRail.svelte @@ -0,0 +1,33 @@ + + + diff --git a/web/src/lib/components/wanted/DownloadList.svelte b/web/src/lib/components/wanted/DownloadList.svelte new file mode 100644 index 00000000..ee3222d8 --- /dev/null +++ b/web/src/lib/components/wanted/DownloadList.svelte @@ -0,0 +1,264 @@ + + +{#each clientsDown as c (c.client)} +

    + Can't reach the download client {c.client} since {relativeTime(c.since)}: {c.detail}. Its + downloads are left as they are until it answers. +

    +{/each} + +{#if downloads.isPending} +

    Reading the downloads…

    +{:else if downloads.isError && !downloads.isFetchNextPageError} +

    Couldn't load the downloads.

    +{:else if items.length === 0} +
    + +

    Nothing has been downloaded yet.

    +

    + A release the search takes, or one you take from a want's Releases, shows here while it + downloads and once it's imported. +

    +
    +{:else} +
      + {#each items as d (d.id)} +
    • +
      +
      +
      + {d.want_artist ? `${d.want_artist} – ${d.want_title}` : d.name} +
      +
      + {d.name}{#if d.quality} · {d.quality}{/if} +
      +
      + + {label(d)} + +
      + + {#if !d.ended_at && (d.import_state === 'downloading' || d.import_state === '')} +
      +
      +
      + {/if} + + {#if running(d) && d.condition} +

      {condition(d.condition)}

      + {/if} + {#if ending(d)} +

      {ending(d)}

      + {/if} + {#if d.import_reason} +

      + {d.import_reason} +

      + {/if} + {#if d.evidence} +

      {counted(d.evidence)}

      + {/if} + {#if filed(d)} +

      {filed(d)}

      + {/if} + {#if seeded(d)} +

      {seeded(d)}

      + {/if} + {#if d.rehomed_from} +

      Taken for {d.rehomed_from}; it turned out to be this album.

      + {/if} +

      {when(d)}

      + + {#if running(d)} +
      + +
      + {:else if needsYou(d)} +
      + {#if d.import_state === 'review'} + + {:else} + + {/if} + +
      + {/if} +
      • + {/each} +
    + + downloads.fetchNextPage()} + /> +{/if} diff --git a/web/src/lib/components/wanted/DownloadList.test.ts b/web/src/lib/components/wanted/DownloadList.test.ts new file mode 100644 index 00000000..605b42e2 --- /dev/null +++ b/web/src/lib/components/wanted/DownloadList.test.ts @@ -0,0 +1,160 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockInfiniteQuery } from '../../../test-utils/query'; +import type { ClientDown, Download } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', () => ({ + createDownloadsQuery: vi.fn(), + importAnyway: vi.fn().mockResolvedValue(undefined), + retryImport: vi.fn().mockResolvedValue(undefined), + refuseDownload: vi.fn().mockResolvedValue(undefined), + abandonDownload: vi.fn().mockResolvedValue(undefined) +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import DownloadList from './DownloadList.svelte'; +import { abandonDownload, createDownloadsQuery, importAnyway, refuseDownload, retryImport } from '#lib/api/wanted.js'; + +const ago = (ms: number) => new Date(Date.now() - ms).toISOString(); + +function download(id: string, extra: Partial = {}): Download { + return { + id, + want_id: 'w1', + want_title: 'Random Access Memories', + want_artist: 'Daft Punk', + name: 'Daft Punk - Random Access Memories (2013) [FLAC]', + indexer: 'Prowlarr', + client: 'qbittorrent', + quality: 'lossless', + origin: 'schedule', + grabbed_at: ago(3_600_000), + ended_at: null, + end_reason: '', + progress: 0, + import_state: 'downloading', + import_reason: '', + imported_at: null, + imported_files: 0, + destination: '', + files: [], + skipped: 0, + end_detail: '', + seeding: false, + ratio: 0, + seed_released_at: null, + condition: null, + ...extra + }; +} + +afterEach(() => vi.clearAllMocks()); + +function setup(items: Download[], clientsDown: ClientDown[] = []) { + // The Activity page carries the clients not answering beside its items. + const page = { items, total: items.length, limit: 50, offset: 0, clients_down: clientsDown }; + vi.mocked(createDownloadsQuery).mockReturnValue( + mockInfiniteQuery({ pages: [page] }) as unknown as ReturnType< + typeof createDownloadsQuery + > + ); + return render(DownloadList); +} + +describe('DownloadList', () => { + test('a running download shows how far it has got, and can be given up', async () => { + setup([download('d1', { progress: 0.375 })]); + expect(screen.getByTestId('download-state').textContent).toMatch(/downloading/i); + expect(screen.getByRole('progressbar', { name: /downloaded/i }).getAttribute('aria-valuenow')).toBe('37'); + expect(screen.queryByTestId('download-condition')).toBeNull(); + await fireEvent.click(screen.getByRole('button', { name: /give up/i })); + await waitFor(() => expect(abandonDownload).toHaveBeenCalledWith('d1')); + }); + + test('a stalled download says what is wrong and how often it was asked for again', () => { + setup([ + download('d1', { + progress: 0.4, + condition: { trigger: 'no_peers', detail: 'Nobody has shared it in 7 hours; 40% downloaded.', retries: 2, since: ago(1_200_000) } + }) + ]); + expect(screen.getByTestId('download-condition').textContent).toMatch( + /nobody has shared it in 7 hours; 40% downloaded\. asked for again 2 of 3 times/i + ); + }); + + test('a download given up on says why it ended', () => { + setup([ + download('d1', { + ended_at: ago(60_000), + end_reason: 'stalled', + end_detail: 'Taken 7 hours ago and never finished: nobody was sharing it. Held back until 17 Oct.' + }) + ]); + expect(screen.getByTestId('download-state').textContent).toMatch(/given up/i); + expect(screen.getByTestId('download-ending').textContent).toMatch(/nobody was sharing it/); + expect(screen.queryByRole('button')).toBeNull(); + }); + + test('a download client that does not answer is named above the list', () => { + setup([download('d1')], [{ client: 'qbittorrent', detail: 'connection refused', since: ago(600_000) }]); + expect(screen.getByTestId('client-down').textContent).toMatch(/can't reach the download client qbittorrent .*connection refused/i); + }); + + test('a doubtful download says why and can be imported anyway or refused', async () => { + setup([ + download('d1', { + import_state: 'review', + import_reason: 'Only the lengths agree. A person decides.', + evidence: { files: 13, tracks: 13, identified: 0, recordings: 0, lengths: 13, titles: 0, fingerprinted: false } + }) + ]); + expect(screen.getByTestId('download-state').textContent).toMatch(/needs you/i); + expect(screen.getByTestId('download-reason').textContent).toMatch(/a person decides/i); + expect(screen.getByText(/13 files against 13 tracks · no AcoustID · 13 lengths agree/)).toBeTruthy(); + await fireEvent.click(screen.getByRole('button', { name: /import anyway/i })); + await waitFor(() => expect(importAnyway).toHaveBeenCalledWith('d1')); + await fireEvent.click(screen.getByRole('button', { name: /refuse/i })); + await waitFor(() => expect(refuseDownload).toHaveBeenCalledWith('d1')); + }); + + test('a failed import can be tried again', async () => { + setup([download('d1', { import_state: 'failed', import_reason: 'The library folder is not writable.' })]); + expect(screen.queryByRole('button', { name: /import anyway/i })).toBeNull(); + await fireEvent.click(screen.getByRole('button', { name: /try again/i })); + await waitFor(() => expect(retryImport).toHaveBeenCalledWith('d1')); + }); + + test('an imported download says what was filed where', () => { + setup([ + download('d1', { + ended_at: ago(60_000), + end_reason: 'imported', + import_state: 'imported', + imported_at: ago(60_000), + imported_files: 12, + skipped: 1, + destination: '/music/Daft Punk/Random Access Memories (2013)' + }) + ]); + expect(screen.getByTestId('download-state').textContent).toMatch(/imported/i); + expect(screen.getByText(/12 tracks filed, 1 already held into \/music\/Daft Punk/)).toBeTruthy(); + expect(screen.queryByRole('button')).toBeNull(); + }); + + test('an imported download says how its seeding goes', () => { + const imported = { ended_at: ago(60_000), end_reason: 'imported', import_state: 'imported' as const }; + setup([ + download('d1', { ...imported, seeding: true, ratio: 0.8 }), + download('d2', { ...imported, seeding: true, ratio: 2.1, seed_released_at: ago(60_000) }) + ]); + const lines = screen.getAllByTestId('download-seeding').map((e) => e.textContent); + expect(lines[0]).toMatch(/seeding: share ratio 0\.80/i); + expect(lines[1]).toMatch(/seeded, then removed/i); + }); + + test('with nothing downloaded it says what lands here', () => { + setup([]); + expect(screen.getByText(/nothing has been downloaded yet/i)).toBeTruthy(); + }); +}); diff --git a/web/src/lib/components/wanted/FollowList.svelte b/web/src/lib/components/wanted/FollowList.svelte new file mode 100644 index 00000000..f5702487 --- /dev/null +++ b/web/src/lib/components/wanted/FollowList.svelte @@ -0,0 +1,118 @@ + + +{#if follows.isPending} +

    Reading who's followed…

    +{:else if follows.isError} +

    Couldn't load the followed artists.

    +{:else if (follows.data ?? []).length === 0} +

    No artists are followed. Follow one from its artist page.

    +{:else} +
      + {#each follows.data ?? [] as f (f.artist_mbid)} +
    • + +
      +
      + {#if f.artist_id} + {f.artist_name} + {:else} + {f.artist_name} + {/if} +
      +
      + {plural(f.open_wants, 'album wanted', 'albums wanted')} · + {f.checked_at ? `checked ${relativeTime(f.checked_at)}` : 'not checked yet'} +
      +
      + + +
      • + {/each} +
    +{/if} diff --git a/web/src/lib/components/wanted/FollowList.test.ts b/web/src/lib/components/wanted/FollowList.test.ts new file mode 100644 index 00000000..7ae23c05 --- /dev/null +++ b/web/src/lib/components/wanted/FollowList.test.ts @@ -0,0 +1,73 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockQuery } from '../../../test-utils/query'; +import type { Follow, ReleaseProfilesResponse } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', () => ({ + createFollowsQuery: vi.fn(), + createReleaseProfilesQuery: vi.fn(), + putFollow: vi.fn().mockResolvedValue({ following: true }), + deleteFollow: vi.fn().mockResolvedValue(undefined) +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import FollowList from './FollowList.svelte'; +import { createFollowsQuery, createReleaseProfilesQuery, deleteFollow, putFollow } from '#lib/api/wanted.js'; + +const ago = (ms: number) => new Date(Date.now() - ms).toISOString(); + +const follow: Follow = { + artist_mbid: 'blur', + artist_name: 'Blur', + artist_id: 'art-1', + profile_id: 'p-std', + profile_name: 'Standard', + followed_at: ago(10 * 86_400_000), + checked_at: ago(3 * 3_600_000), + open_wants: 2, + image_url: '' +}; + +const profiles: ReleaseProfilesResponse = { + profiles: [ + { id: 'p-std', name: 'Standard', primary_types: ['Album'], secondary_types: [], is_default: true, follow_count: 1 }, + { id: 'p-live', name: 'With live', primary_types: ['Album'], secondary_types: ['Live'], is_default: false, follow_count: 0 } + ], + primary_types: ['Album'], + secondary_types: ['Live'] +}; + +afterEach(() => vi.clearAllMocks()); + +function setup(follows: Follow[] = [follow]) { + vi.mocked(createFollowsQuery).mockReturnValue( + mockQuery({ data: follows }) as unknown as ReturnType + ); + vi.mocked(createReleaseProfilesQuery).mockReturnValue( + mockQuery({ data: profiles }) as unknown as ReturnType + ); + return render(FollowList); +} + +describe('FollowList', () => { + test('lists artists; the profile can change and the artist be unfollowed', async () => { + setup(); + expect(screen.getAllByTestId('follow-row')).toHaveLength(1); + expect(screen.getByText(/2 albums wanted/)).toBeTruthy(); + + const select = screen.getByLabelText(/profile for blur/i) as HTMLSelectElement; + expect(select.value).toBe('p-std'); + await fireEvent.change(select, { target: { value: 'p-live' } }); + await waitFor(() => + expect(putFollow).toHaveBeenCalledWith('blur', { artist_name: 'Blur', profile_id: 'p-live' }) + ); + + await fireEvent.click(screen.getByRole('button', { name: /unfollow blur/i })); + await waitFor(() => expect(deleteFollow).toHaveBeenCalledWith('blur')); + }); + + test('with no follows it says where to follow from', () => { + setup([]); + expect(screen.getByText(/follow one from its artist page/i)).toBeTruthy(); + }); +}); diff --git a/web/src/lib/components/wanted/UnmatchedList.svelte b/web/src/lib/components/wanted/UnmatchedList.svelte new file mode 100644 index 00000000..aaaeb93d --- /dev/null +++ b/web/src/lib/components/wanted/UnmatchedList.svelte @@ -0,0 +1,150 @@ + + +{#if unmatched.isPending} +

    Reading the unmatched albums…

    +{:else if unmatched.isError} +

    Couldn't load the unmatched albums.

    +{:else} + {#if toMatch.length === 0} +
    + +

    Every album in your library is matched.

    +

    + An album no MusicBrainz album matches by its artist, title and tracks lands here, to be + matched by hand or left off. +

    +
    + {:else} +

    + No MusicBrainz album matched {toMatch.length === 1 ? 'this album' : `these ${toMatch.length} albums`} + by artist, title and tracks, so {toMatch.length === 1 ? "it isn't" : "they aren't"} fetched back if + {toMatch.length === 1 ? 'its' : 'their'} files go missing. Match each to its MusicBrainz album, or + leave it off if MusicBrainz doesn't have it. +

    +
      + {#each toMatch as u (u.album_id)} +
    • +
      +
      + {u.title} +
      {u.artist} · {plural(u.tracks, 'track', 'tracks')}
      +

      {u.detail}

      +
      +
      + + +
      +
      + {#if open === u.album_id} + onMatch(u, ref)} /> + {/if} +
      • + {/each} +
    + {/if} + + {#if leftOff.length > 0} +
    +

    Left off

    +

    + Kept in the library, never matched again, and not fetched back if their files go missing. +

    +
      + {#each leftOff as u (u.album_id)} +
    • +
      + {u.title} +
      + {u.artist} · {plural(u.tracks, 'track', 'tracks')} · left off {relativeTime(u.decided_at)} +
      +
      + +
      • + {/each} +
    +
    + {/if} +{/if} diff --git a/web/src/lib/components/wanted/UnmatchedList.test.ts b/web/src/lib/components/wanted/UnmatchedList.test.ts new file mode 100644 index 00000000..2245246c --- /dev/null +++ b/web/src/lib/components/wanted/UnmatchedList.test.ts @@ -0,0 +1,109 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import type { CatalogueSearchResult, UnmatchedAlbum } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', async () => { + const { mockQuery } = await import('../../../test-utils/query'); + return { + createUnmatchedQuery: vi.fn(() => mockQuery({ data: [] })), + matchUnmatched: vi.fn().mockResolvedValue({ + release_group_mbid: 'rg-1', + title: 'Songs for the Deaf', + artist: 'Queens of the Stone Age', + agree: 9, + tracks: 10 + }), + leaveUnmatchedOff: vi.fn().mockResolvedValue(undefined), + putUnmatchedBack: vi.fn().mockResolvedValue(undefined) + }; +}); +vi.mock('#lib/api/catalogue.js', () => ({ createCatalogueSearchQuery: vi.fn() })); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import UnmatchedList from './UnmatchedList.svelte'; +import { mockQuery } from '../../../test-utils/query'; +import { createUnmatchedQuery, leaveUnmatchedOff, matchUnmatched, putUnmatchedBack } from '#lib/api/wanted.js'; +import { createCatalogueSearchQuery } from '#lib/api/catalogue.js'; +import { pushToast } from '#lib/stores/toast.svelte.js'; + +const DAY = 24 * 3_600_000; + +function album(id: string, title: string, artist: string, extra: Partial = {}): UnmatchedAlbum { + return { + album_id: id, + title, + artist, + detail: `MusicBrainz has no album called "${title}" by ${artist}.`, + tracks: 10, + left_off: false, + decided_at: new Date(Date.now() - 2 * DAY).toISOString(), + ...extra + }; +} + +const result: CatalogueSearchResult = { + mbid: 'rg-1', + name: 'Songs for the Deaf', + secondary_text: 'Queens of the Stone Age · 2002 · Album', + image_url: '', + artist_mbid: 'qotsa', + album_mbid: '', + in_library: false, + requested: false +}; + +afterEach(() => vi.clearAllMocks()); + +function setup(items: UnmatchedAlbum[], results: CatalogueSearchResult[] = [result]) { + vi.mocked(createUnmatchedQuery).mockReturnValue( + mockQuery({ data: items }) as unknown as ReturnType + ); + vi.mocked(createCatalogueSearchQuery).mockReturnValue( + mockQuery({ data: results }) as unknown as ReturnType + ); + return render(UnmatchedList); +} + +describe('UnmatchedList', () => { + test('each unmatched album says why, with its tracks, and links to the album', () => { + setup([album('al-1', 'Dial Tone', 'Annadale', { tracks: 1 })]); + const row = screen.getByTestId('unmatched-row'); + expect(row.textContent).toMatch(/Annadale · 1 track/); + expect(row.textContent).toMatch(/no album called "Dial Tone"/); + expect(screen.getByRole('link', { name: 'Dial Tone' }).getAttribute('href')).toBe('/albums/al-1'); + }); + + test('Match searches MusicBrainz for the album by its own title and artist, and matches a result', async () => { + setup([album('al-2', 'Songs for the Deaf', 'Queens of the Stone Age')]); + await fireEvent.click(screen.getByRole('button', { name: /match…/i })); + expect(createCatalogueSearchQuery).toHaveBeenLastCalledWith('Songs for the Deaf Queens of the Stone Age', 'album'); + await fireEvent.click(screen.getByRole('button', { name: /match to songs for the deaf/i })); + await waitFor(() => expect(matchUnmatched).toHaveBeenCalledWith('al-2', 'rg-1')); + await waitFor(() => expect(pushToast).toHaveBeenCalledWith(expect.stringMatching(/9 of 10 tracks agree/))); + }); + + test('a pasted MusicBrainz link matches the album', async () => { + setup([album('al-3', '+++', 'DROELOE')]); + await fireEvent.click(screen.getByRole('button', { name: /match…/i })); + const link = 'https://musicbrainz.org/release/11111111-0000-4000-8000-000000000003'; + await fireEvent.input(screen.getByRole('textbox', { name: /musicbrainz album link/i }), { target: { value: link } }); + await fireEvent.click(screen.getByRole('button', { name: /^match$/i })); + await waitFor(() => expect(matchUnmatched).toHaveBeenCalledWith('al-3', link)); + }); + + test("a local band's album is left off, and one left off can be put back", async () => { + setup([album('al-1', 'Dial Tone', 'Annadale'), album('al-4', 'Demo', 'Annadale', { left_off: true })]); + await fireEvent.click(screen.getByRole('button', { name: /leave it off/i })); + await waitFor(() => expect(leaveUnmatchedOff).toHaveBeenCalledWith('al-1')); + const leftOff = screen.getByTestId('left-off-row'); + expect(leftOff.textContent).toMatch(/left off 2d ago/); + await fireEvent.click(screen.getByRole('button', { name: /put back/i })); + await waitFor(() => expect(putUnmatchedBack).toHaveBeenCalledWith('al-4')); + }); + + test('with every album matched it says so, and still lists those left off', () => { + setup([album('al-4', 'Demo', 'Annadale', { left_off: true })]); + expect(screen.getByText(/every album in your library is matched/i)).toBeTruthy(); + expect(screen.getByTestId('left-off')).toBeTruthy(); + }); +}); diff --git a/web/src/lib/components/wanted/UnmatchedMatch.svelte b/web/src/lib/components/wanted/UnmatchedMatch.svelte new file mode 100644 index 00000000..b09059fc --- /dev/null +++ b/web/src/lib/components/wanted/UnmatchedMatch.svelte @@ -0,0 +1,106 @@ + + +
    +
    { + e.preventDefault(); + asked = text.trim(); + }} + > + + +
    + + {#if search.isPending && asked} +

    Asking MusicBrainz…

    + {:else if search.isError} +

    Couldn't reach MusicBrainz. A pasted link works the same.

    + {:else if asked && results.length === 0} +

    MusicBrainz has no album like that.

    + {:else if results.length > 0} +
      + {#each results as r (r.mbid)} +
    • +
      +
      {r.name}
      +
      {r.secondary_text}
      +
      + MusicBrainz + +
      • + {/each} +
    + {/if} + +
    { + e.preventDefault(); + if (link.trim()) onMatch(link.trim()); + }} + > + + +
    +
    diff --git a/web/src/lib/components/wanted/WantList.svelte b/web/src/lib/components/wanted/WantList.svelte new file mode 100644 index 00000000..65838a73 --- /dev/null +++ b/web/src/lib/components/wanted/WantList.svelte @@ -0,0 +1,308 @@ + + +{#if artist && head} +
    +
    + +
    +
    + ← Catalogue +

    {head.artist_name}

    +

    + {head.held} of {plural(head.albums, 'album', 'albums')} held{head.wanted ? ` · ${head.wanted} wanted` : ''}{head.missing + ? ` · ${head.missing} with files missing` + : ''}{head.downloading ? ` · ${head.downloading} downloading` : ''} +

    +
    + + {#if head.artist_id} + Open in the library + {/if} +
    +
    +
    +{/if} + +{#if !artist && !wants.isPending && (items.length > 0 || q)} +
    + +
    +{/if} + +{#if wants.isPending} +

    Reading the wanted list…

    +{:else if wants.isError && !wants.isFetchNextPageError} +

    Couldn't load {artist ? "this artist's albums" : 'the wanted list'}.

    +{:else if items.length === 0 && q} +

    + Nothing on this list has an artist or album named like '{q}'. +

    +{:else if items.length === 0} +
    + +

    + {#if artist}None of this artist's albums are in the catalogue. + {:else if list === 'upgrade'}No album is wanted for a better copy. + {:else}Nothing is wanted.{/if} +

    +

    + {#if list === 'upgrade'} + An album you have below the quality set under + Acquisition, or with a video + rip among its tracks, lands here. + {:else} + A request you approve, a new release from a followed artist, or an album whose files went + missing lands here. + {/if} +

    +
    +{:else} +
    + {#each sections as a, i (a.key + ':' + i)} +
    + {#if !artist}

    + + {#if a.mbid} + {a.name} + {:else} + {a.name} + {/if} + + {plural(a.wants.length, 'album', 'albums')} +

    {/if} +
      + {#each a.wants as w (w.id)} +
    • +
      + +
      +
      + {#if w.album_id && w.held_at} + {w.title} + {:else} + {w.title} + {/if} + {#if kind(w)} · {kind(w)}{/if} +
      +
      + {why(w)} · {w.held_at ? `held ${relativeTime(w.held_at)}` : `wanted ${relativeTime(w.wanted_at)}`} +
      + {#if w.held_at && w.upgrade} +
      {upgradeLine(w)}
      + {/if} + {#if searchable(w)} +
      + {wantStatus(w)} + {#if needsYou(w)} + Decide on Activity + {/if} +
      + {:else if arrived(w)} +
      {arrived(w)}
      + {/if} +
      +
      + + {#if searchable(w)} + + + {/if} + +
      +
      + {#if shown === w.id} +
      + +
      + {/if} + {#if open === w.id && searchable(w)} +
      + +
      + {/if} +
      • + {/each} +
    +
    + {/each} +
    + + wants.fetchNextPage()} + /> +{/if} diff --git a/web/src/lib/components/wanted/WantList.test.ts b/web/src/lib/components/wanted/WantList.test.ts new file mode 100644 index 00000000..662f2ed0 --- /dev/null +++ b/web/src/lib/components/wanted/WantList.test.ts @@ -0,0 +1,345 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockInfiniteQuery, mockQuery } from '../../../test-utils/query'; +import type { LibraryListing, Want, WantGrab, WantsPage } from '#lib/api/types.js'; + +vi.mock('#lib/api/wanted.js', async () => { + const { mockQuery } = await import('../../../test-utils/query'); + return { + // The artist page's FollowControl. + createFollowQuery: vi.fn(() => mockQuery({ data: { following: true } })), + createReleaseProfilesQuery: vi.fn(() => mockQuery({ data: { profiles: [], primary_types: [], secondary_types: [] } })), + putFollow: vi.fn(), + deleteFollow: vi.fn(), + createWantsQuery: vi.fn(), + removeWant: vi.fn().mockResolvedValue(undefined), + searchWant: vi.fn().mockResolvedValue({ + outcome: 'nothing', considered: 0, best_release: '', best_verdict: '', narrowed: [] + }), + createCandidatesQuery: vi.fn(), + createWantTracksQuery: vi.fn(), + grabCandidate: vi.fn(), + blockCandidate: vi.fn(), + unblockRelease: vi.fn() + }; +}); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import WantList from './WantList.svelte'; +import { pushToast } from '#lib/stores/toast.svelte.js'; +import { createCandidatesQuery, createWantTracksQuery, createWantsQuery, removeWant, searchWant } from '#lib/api/wanted.js'; +import type { WantState, WantTracks } from '#lib/api/types.js'; + +const DAY = 24 * 3_600_000; +const ago = (ms: number) => new Date(Date.now() - ms).toISOString(); + +function want(id: string, extra: Partial = {}): Want { + return { + id, + release_group_mbid: `rg-${id}`, + title: 'The Ballad of Darren', + artist_mbid: 'blur', + artist_name: 'Blur', + artist_id: 'art-1', + primary_type: 'Album', + secondary_types: [], + year: '2023', + reason: 'follow', + wanted_at: ago(2 * DAY), + held_at: null, + image_url: '', + ...extra + }; +} + +function page( + items: Want[], + library: LibraryListing = { listed_at: ago(DAY), unlisted: 0 }, + artist?: WantsPage['artist'] +): WantsPage { + return { + artist, + items, + total: items.length, + limit: 50, + offset: 0, + counts: { artists: 2, wanted: 2, missing: 1, held: 4, upgrades: 0, following: 1, activity: 1, attention: 0, blocked: 0, unmatched: 0 }, + library + }; +} + +function grab(extra: Partial = {}): WantGrab { + return { + id: 'g1', + name: 'Blur - Blur (1997) [FLAC]', + grabbed_at: ago(60_000), + origin: 'schedule', + quality: 'lossless', + live: true, + end_reason: '', + progress: 0, + import_state: '', + import_reason: '', + end_detail: '', + imported_at: null, + imported_files: 0, + ...extra + }; +} + +afterEach(() => vi.clearAllMocks()); + +function setup( + items: Want[] = [want('w1'), want('w2', { reason: 'request', requested_by: 'sam', title: 'Parklife' })], + state: WantState = 'wanted', + library?: LibraryListing, + tracks: WantTracks = { + source: 'musicbrainz', + edition: '2023 · CD · GB', + tracks: [ + { disc: 1, number: 1, title: 'The Ballad', length_ms: 214000, missing: false }, + { disc: 1, number: 2, title: 'St. Charles Square', length_ms: 3_725_000, missing: false } + ] + } +) { + vi.mocked(createWantsQuery).mockReturnValue( + mockInfiniteQuery({ pages: [page(items, library)] }) as unknown as ReturnType + ); + vi.mocked(createCandidatesQuery).mockReturnValue( + mockQuery({ data: [] }) as unknown as ReturnType + ); + vi.mocked(createWantTracksQuery).mockReturnValue( + mockQuery({ data: tracks }) as unknown as ReturnType + ); + return render(WantList, { list: state }); +} + +describe('WantList', () => { + test('asks for its own state', () => { + setup([], 'upgrade'); + expect(createWantsQuery).toHaveBeenCalledWith('upgrade', ''); + expect(screen.getByText(/no album is wanted for a better copy/i)).toBeTruthy(); + }); + + test('a want says why it is wanted', () => { + setup(); + const why = screen.getAllByTestId('want-why').map((e) => e.textContent ?? ''); + expect(why[0]).toMatch(/new from a followed artist/i); + expect(why[1]).toMatch(/requested by sam/i); + }); + + test('remove takes a want off the list', async () => { + setup(); + await fireEvent.click(screen.getByRole('button', { name: /stop wanting parklife/i })); + await waitFor(() => expect(removeWant).toHaveBeenCalledWith('w2')); + }); + + test('a held album can be taken off the list, and is not searched for', async () => { + setup([want('w1', { held_at: ago(DAY), album_id: 'al-1', reason: 'library' })], 'held'); + expect(screen.queryByRole('button', { name: /stop wanting|search for/i })).toBeNull(); + expect(screen.getByRole('link', { name: /the ballad of darren/i }).getAttribute('href')).toBe('/albums/al-1'); + expect(screen.getByTestId('want-why').textContent).toMatch(/in your library when the list began/i); + await fireEvent.click(screen.getByRole('button', { name: /take the ballad of darren off the list/i })); + await waitFor(() => expect(removeWant).toHaveBeenCalledWith('w1')); + }); + + test('an empty list explains what lands there', () => { + setup([]); + expect(screen.getByText(/nothing is wanted/i)).toBeTruthy(); + }); + + test('each open want says what its last search found', () => { + setup([ + want('w1'), + want('w2', { + title: 'Parklife', + search: { searched_at: ago(3_600_000), outcome: 'nothing', considered: 4, best_release: 'x', best_verdict: 'rejected', error: '' } + }), + want('w3', { title: 'Blur', grab: grab() }), + want('w4', { title: '13', grab: grab({ import_state: 'downloading', progress: 0.42 }) }) + ]); + const lines = screen.getAllByTestId('want-search').map((e) => e.textContent ?? ''); + expect(lines[0]).toMatch(/not searched yet/i); + expect(lines[1]).toMatch(/4 found, none of them this album/i); + expect(lines[2]).toMatch(/downloading blur - blur \(1997\) \[flac\], taken by the schedule/i); + expect(lines[3]).toMatch(/downloading 42%/i); + }); + + test('a download waiting on a person says why and points to Activity', () => { + setup([ + want('w1', { grab: grab({ import_state: 'review', import_reason: 'Only the lengths agree. A person decides.' }) }) + ]); + const line = screen.getByTestId('want-search'); + expect(line.textContent).toMatch(/needs you: only the lengths agree/i); + expect(screen.getByRole('link', { name: /decide on activity/i }).getAttribute('href')).toBe('/collection/activity'); + // A live download can't be searched for again. + expect((screen.getByRole('button', { name: /search for the ballad of darren now/i }) as HTMLButtonElement).disabled).toBe(true); + }); + + test('a refused download is named beside the search that follows it', () => { + setup([ + want('w1', { + search: { searched_at: ago(3_600_000), outcome: 'nothing', considered: 0, best_release: '', best_verdict: '', error: '' }, + grab: grab({ live: false, end_reason: 'refused', import_state: 'refused', import_reason: 'Another album.' }) + }) + ]); + expect(screen.getByTestId('want-search').textContent).toMatch(/last download refused: another album/i); + }); + + test('a download given up on is named beside the search that follows it', () => { + setup([ + want('w1', { + search: { searched_at: ago(3_600_000), outcome: 'waiting', considered: 4, best_release: '', best_verdict: '', error: '' }, + grab: grab({ live: false, end_reason: 'stalled', import_state: 'downloading', end_detail: 'Nobody was sharing it.' }) + }) + ]); + expect(screen.getByTestId('want-search').textContent).toMatch(/last download given up: nobody was sharing it/i); + }); + + test('a held album says how it arrived', () => { + setup( + [ + want('w1', { + held_at: ago(DAY), + album_id: 'al-1', + grab: grab({ live: false, end_reason: 'imported', import_state: 'imported', imported_at: ago(DAY), imported_files: 11 }) + }) + ], + 'held' + ); + expect(screen.getByTestId('want-arrived').textContent).toMatch(/imported 11 tracks .* from blur - blur/i); + }); + + test('Search runs one now and says what came of it', async () => { + setup(); + await fireEvent.click(screen.getByRole('button', { name: /search for parklife now/i })); + await waitFor(() => expect(searchWant).toHaveBeenCalledWith('w2')); + await waitFor(() => expect(pushToast).toHaveBeenCalledWith('Nothing found for Parklife.', 'info')); + }); + + test('Releases opens what the last search found', async () => { + setup(); + expect(screen.queryByTestId('want-candidates')).toBeNull(); + const toggle = screen.getByRole('button', { name: /releases found for parklife/i }); + await fireEvent.click(toggle); + expect(toggle.getAttribute('aria-expanded')).toBe('true'); + expect(screen.getByTestId('want-candidates')).toBeTruthy(); + expect(createCandidatesQuery).toHaveBeenCalledWith('w2'); + }); + + test('albums sit under their artist, as in the library', () => { + setup([ + want('w1', { artist_mbid: 'blur', artist_name: 'Blur', artist_id: 'art-1', title: 'Parklife' }), + want('w2', { artist_mbid: 'blur', artist_name: 'Blur', artist_id: 'art-1', title: '13' }), + want('w3', { artist_mbid: 'camel', artist_name: 'Camel', artist_id: undefined, title: 'Moonmadness' }) + ]); + const sections = screen.getAllByTestId('want-artist'); + expect(sections.map((s) => s.getAttribute('aria-label'))).toEqual(['Blur', 'Camel']); + expect(sections[0].querySelectorAll('[data-testid="want-row"]').length).toBe(2); + expect(sections[0].textContent).toMatch(/2 albums/); + expect(sections[1].textContent).toMatch(/1 album/); + // Each artist opens on their page in the catalogue. + expect(screen.getByRole('link', { name: 'Blur' }).getAttribute('href')).toBe('/collection/artist/blur'); + expect(screen.getByRole('link', { name: 'Camel' }).getAttribute('href')).toBe('/collection/artist/camel'); + }); + + test('Tracks shows the album\'s tracks, with the edition they come from', async () => { + setup(); + expect(screen.queryByTestId('want-tracks')).toBeNull(); + const toggle = screen.getByRole('button', { name: /tracks of parklife/i }); + await fireEvent.click(toggle); + expect(toggle.getAttribute('aria-expanded')).toBe('true'); + expect(createWantTracksQuery).toHaveBeenCalledWith('w2'); + const panel = screen.getByTestId('want-tracks'); + expect(panel.textContent).toMatch(/MusicBrainz's standard edition\s+·\s+2023 · CD · GB/); + const rows = screen.getAllByTestId('want-track').map((e) => e.textContent?.replace(/\s+/g, ' ').trim()); + expect(rows).toEqual(['1 The Ballad 3:34', '2 St. Charles Square 1:02:05']); + }); + + test('a held album\'s tracks are its own, a lost file marked', async () => { + setup([want('w1', { held_at: ago(DAY), album_id: 'al-1' })], 'held', undefined, { + source: 'library', + tracks: [ + { disc: 1, number: 1, title: 'The Ballad', length_ms: 214000, missing: true }, + { disc: 2, number: 1, title: 'Bonus', length_ms: 0, missing: false } + ] + }); + await fireEvent.click(screen.getByRole('button', { name: /tracks of the ballad of darren/i })); + const panel = screen.getByTestId('want-tracks'); + expect(panel.textContent).not.toMatch(/MusicBrainz/); + const rows = screen.getAllByTestId('want-track').map((e) => e.textContent?.replace(/\s+/g, ' ').trim()); + expect(rows).toEqual(['1-1 The Ballad file missing 3:34', '2-1 Bonus']); + }); + + test('the filter asks the server for the artists and albums it names', async () => { + setup(); + await fireEvent.input(screen.getByRole('searchbox', { name: /filter artists and albums/i }), { + target: { value: ' park ' } + }); + await waitFor(() => expect(createWantsQuery).toHaveBeenLastCalledWith('wanted', 'park')); + }); + + test("an artist's page lists every album of theirs under a header", () => { + vi.mocked(createWantsQuery).mockReturnValue( + mockInfiniteQuery({ + pages: [ + page([want('w1', { held_at: ago(DAY), album_id: 'al-1', title: 'Parklife' }), want('w2')], undefined, { + artist_mbid: 'blur', + artist_name: 'Blur', + sort_name: 'Blur', + artist_id: 'art-1', + image_url: '/art/blur', + albums: 2, + held: 1, + wanted: 1, + missing: 0, + downloading: 0, + following: true + }) + ] + }) as unknown as ReturnType + ); + vi.mocked(createCandidatesQuery).mockReturnValue( + mockQuery({ data: [] }) as unknown as ReturnType + ); + render(WantList, { artist: 'blur' }); + expect(createWantsQuery).toHaveBeenCalledWith('all', '', 'blur'); + const head = screen.getByTestId('catalogue-artist-head'); + expect(head.textContent).toMatch(/Blur/); + expect(screen.getByTestId('catalogue-artist-counts').textContent).toMatch(/1 of 2 albums held\s+·\s+1 wanted/); + expect(screen.getByRole('link', { name: /open in the library/i }).getAttribute('href')).toBe('/artists/art-1'); + expect(screen.getByRole('link', { name: /catalogue/i }).getAttribute('href')).toBe('/collection'); + // One artist: no filter, no artist headings, both albums. + expect(screen.queryByRole('searchbox')).toBeNull(); + expect(screen.getAllByTestId('want-row')).toHaveLength(2); + expect(screen.queryByRole('link', { name: 'Blur' })).toBeNull(); + }); + + test('an upgrade is a held album searched for a better copy', async () => { + setup( + [ + want('u1', { held_at: ago(DAY), reason: 'library', upgrade: 'quality', held_quality: 'MP3, about 192 kbps' }), + want('u2', { held_at: ago(DAY), reason: 'library', upgrade: 'rip', title: 'Humanz' }) + ], + 'upgrade' + ); + expect(createWantsQuery).toHaveBeenCalledWith('upgrade', ''); + const lines = screen.getAllByTestId('want-upgrade').map((e) => e.textContent ?? ''); + expect(lines[0]).toMatch(/you have mp3, about 192 kbps; looking for better/i); + expect(lines[1]).toMatch(/video rip; looking for a clean copy/i); + expect(screen.getAllByTestId('want-search')[0].textContent).toMatch(/not searched yet/i); + await fireEvent.click(screen.getByRole('button', { name: /search for humanz now/i })); + await waitFor(() => expect(searchWant).toHaveBeenCalledWith('u2')); + }); + + test('a held album not wanted for a better copy has no search', () => { + setup([want('h1', { held_at: ago(DAY), reason: 'library' })], 'held'); + expect(screen.queryByTestId('want-upgrade')).toBeNull(); + expect(screen.queryByRole('button', { name: /search for/i })).toBeNull(); + }); + + test('with no upgrades it says what would land here', () => { + setup([], 'upgrade'); + expect(screen.getByText(/no album is wanted for a better copy/i)).toBeTruthy(); + }); +}); diff --git a/web/src/lib/components/wanted/WantTracks.svelte b/web/src/lib/components/wanted/WantTracks.svelte new file mode 100644 index 00000000..f57595cb --- /dev/null +++ b/web/src/lib/components/wanted/WantTracks.svelte @@ -0,0 +1,52 @@ + + +
    + {#if tracks.isPending} +

    Reading the tracks…

    + {:else if tracks.isError} +

    + {away ? "MusicBrainz can't be reached, so the tracks can't be shown." : "Couldn't load the tracks."} +

    + {:else if tracks.data.source === 'none' || list.length === 0} +

    MusicBrainz lists no edition of this album, so there are no tracks to show.

    + {:else} + {#if tracks.data.source === 'musicbrainz'} +

    + MusicBrainz's standard edition{tracks.data.edition ? ` · ${tracks.data.edition}` : ''} +

    + {/if} +
      + {#each list as t, i (i)} +
    1. + {position(t)} + + {t.title} + + {#if t.missing}file missing{/if} + {#if t.length_ms > 0} + {formatDuration(t.length_ms / 1000)} + {/if} +
      2. + {/each} +
    + {/if} +
    diff --git a/web/src/lib/media/covers.ts b/web/src/lib/media/covers.ts index 5304a6fb..de26a198 100644 --- a/web/src/lib/media/covers.ts +++ b/web/src/lib/media/covers.ts @@ -1,6 +1,6 @@ // Fallback cover used when the real cover fails to load (e.g. scanner // didn't pick up embedded art and there's no sidecar). Served as a static -// asset from web/static/placeholders/ so the operator can swap the artwork +// asset from web/static/placeholders/ so an admin can swap the artwork // without touching code. SvelteKit + adapter-static publishes everything // under web/static/ at the URL root, and the Go binary embeds the build // output, so this resolves identically in dev and production. diff --git a/web/src/lib/media/dominantColor.test.ts b/web/src/lib/media/dominantColor.test.ts new file mode 100644 index 00000000..e450d18a --- /dev/null +++ b/web/src/lib/media/dominantColor.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, test } from 'vitest'; +import { dominantOf } from './dominantColor'; + +// Pixels as RGBA, n of each colour. +function pixels(...runs: [number, number, number, number][]): Uint8ClampedArray { + const out: number[] = []; + for (const [r, g, b, n] of runs) for (let i = 0; i < n; i++) out.push(r, g, b, 255); + return new Uint8ClampedArray(out); +} + +const near = (got: { r: number; g: number; b: number } | null, r: number, g: number, b: number) => + !!got && Math.abs(got.r - r) <= 8 && Math.abs(got.g - g) <= 8 && Math.abs(got.b - b) <= 8; + +describe('dominantOf', () => { + test('the colour covering the most of the cover wins, not a small bright accent', () => { + const got = dominantOf(pixels([40, 70, 140, 700], [250, 20, 60, 100], [30, 160, 60, 200])); + expect(near(got, 40, 70, 140)).toBe(true); + }); + + test('is not the mean: two halves of different colours give one of them', () => { + const got = dominantOf(pixels([200, 40, 40, 520], [40, 40, 200, 480])); + expect(near(got, 200, 40, 40)).toBe(true); + }); + + test('near black and near white are passed over, as Palette does', () => { + const got = dominantOf(pixels([0, 0, 0, 800], [255, 255, 255, 600], [60, 120, 90, 100])); + expect(near(got, 60, 120, 90)).toBe(true); + }); + + test('a cover of nothing but black and white has no colour', () => { + expect(dominantOf(pixels([0, 0, 0, 500], [255, 255, 255, 500]))).toBeNull(); + }); + + test('transparent pixels are not counted', () => { + const data = pixels([40, 70, 140, 100], [200, 40, 40, 300]); + for (let i = 100 * 4; i < data.length; i += 4) data[i + 3] = 0; + expect(near(dominantOf(data), 40, 70, 140)).toBe(true); + }); +}); diff --git a/web/src/lib/media/dominantColor.ts b/web/src/lib/media/dominantColor.ts index 16b76a13..74657a19 100644 --- a/web/src/lib/media/dominantColor.ts +++ b/web/src/lib/media/dominantColor.ts @@ -1,19 +1,26 @@ -// Cheap dominant-color extractor: loads an image, draws it scaled -// to a 1x1 canvas, reads the resulting pixel. The browser's bilinear -// downsample produces an arithmetic-mean color, which approximates -// the "dominant" tone close enough for an ambient accent strip. -// Not aiming for ColorThief-grade clustering — that adds ~5KB and -// noticeable extraction latency. The user-visible feature is just -// "the page subtly takes on the current track's hue" and a mean -// color delivers that. +// The colour that covers the most of a cover, as Android takes it +// (player/ui/DominantColor.kt): androidx.palette's dominant swatch, not +// the mean and not the most vibrant. A mean turns a busy cover to mud; +// a vibrant pick lets a small bright accent tint the whole page. The +// dominant swatch matches what the eye sees most of. // -// Cover URLs are same-origin (/api/albums//cover), so no CORS -// dance is needed. Returns null on any failure (load error, decode -// error, opaque-canvas getImageData reject) so the caller can fall -// back to the static accent. +// Palette's method, kept close so the web and Android agree: shrink the +// image, quantise each pixel to 5 bits a channel, drop the colours its +// default filter drops (near black, near white, and the skin-tone band +// on the red "I line"), split what's left by median cut into at most 16 +// boxes, and take the box holding the most pixels. +// +// Cover URLs are same-origin (/api/albums//cover), so the canvas is +// readable. Returns null on any failure (load or decode error, no 2D +// context, a cover with nothing but black and white) and the caller +// keeps what it showed before. export type Rgb = { r: number; g: number; b: number }; +// Palette shrinks a bitmap to about 112×112 before quantising. +const SAMPLE = 112; +const MAX_COLORS = 16; + const cache = new Map>(); export function dominantColorFromUrl(url: string): Promise { @@ -32,13 +39,12 @@ function sample(url: string): Promise { img.onload = () => { try { const canvas = document.createElement('canvas'); - canvas.width = 1; - canvas.height = 1; + canvas.width = SAMPLE; + canvas.height = SAMPLE; const ctx = canvas.getContext('2d'); if (!ctx) return resolve(null); - ctx.drawImage(img, 0, 0, 1, 1); - const data = ctx.getImageData(0, 0, 1, 1).data; - resolve({ r: data[0], g: data[1], b: data[2] }); + ctx.drawImage(img, 0, 0, SAMPLE, SAMPLE); + resolve(dominantOf(ctx.getImageData(0, 0, SAMPLE, SAMPLE).data)); } catch { resolve(null); } @@ -48,6 +54,119 @@ function sample(url: string): Promise { }); } +type Color5 = { r: number; g: number; b: number; n: number }; + +// dominantOf picks the dominant colour of RGBA pixel data. +export function dominantOf(data: Uint8ClampedArray): Rgb | null { + const hist = new Map(); + for (let i = 0; i < data.length; i += 4) { + if (data[i + 3] < 128) continue; + const key = ((data[i] >> 3) << 10) | ((data[i + 1] >> 3) << 5) | (data[i + 2] >> 3); + hist.set(key, (hist.get(key) ?? 0) + 1); + } + const colors: Color5[] = []; + for (const [key, n] of hist) { + const c = { r: (key >> 10) & 31, g: (key >> 5) & 31, b: key & 31, n }; + if (!ignored(to8(c))) colors.push(c); + } + if (colors.length === 0) return null; + + let best: Rgb | null = null; + let bestN = 0; + for (const box of medianCut(colors, MAX_COLORS)) { + let n = 0, r = 0, g = 0, b = 0; + for (const c of box) { + n += c.n; + r += c.r * c.n; + g += c.g * c.n; + b += c.b * c.n; + } + const avg = to8({ r: r / n, g: g / n, b: b / n }); + if (n > bestN && !ignored(avg)) { + best = avg; + bestN = n; + } + } + return best; +} + +// medianCut splits the colours into at most max boxes, always splitting +// the box with the largest volume, at the population median of its +// longest side. +function medianCut(colors: Color5[], max: number): Color5[][] { + const boxes: Color5[][] = [colors]; + while (boxes.length < max) { + let pick = -1; + let pickVolume = 0; + boxes.forEach((box, i) => { + if (box.length < 2) return; + const v = volume(box); + if (v > pickVolume) { + pick = i; + pickVolume = v; + } + }); + if (pick < 0) break; + const box = boxes[pick]; + const side = longestSide(box); + box.sort((a, b) => a[side] - b[side]); + const total = box.reduce((s, c) => s + c.n, 0); + let run = 0; + let at = box.length - 1; + for (let i = 0; i < box.length - 1; i++) { + run += box[i].n; + if (run >= total / 2) { + at = i + 1; + break; + } + } + boxes.splice(pick, 1, box.slice(0, at), box.slice(at)); + } + return boxes; +} + +function range(box: Color5[], side: 'r' | 'g' | 'b'): number { + let lo = 31, hi = 0; + for (const c of box) { + lo = Math.min(lo, c[side]); + hi = Math.max(hi, c[side]); + } + return hi - lo + 1; +} + +function volume(box: Color5[]): number { + return range(box, 'r') * range(box, 'g') * range(box, 'b'); +} + +function longestSide(box: Color5[]): 'r' | 'g' | 'b' { + const r = range(box, 'r'), g = range(box, 'g'), b = range(box, 'b'); + return r >= g && r >= b ? 'r' : g >= b ? 'g' : 'b'; +} + +function to8(c: { r: number; g: number; b: number }): Rgb { + return { r: Math.round((c.r * 255) / 31), g: Math.round((c.g * 255) / 31), b: Math.round((c.b * 255) / 31) }; +} + +// Palette's default filter: near black, near white, and the band of +// hues around skin tones (10°–37°, not fully saturated). +function ignored({ r, g, b }: Rgb): boolean { + const [h, s, l] = hsl(r, g, b); + return l <= 0.05 || l >= 0.95 || (h >= 10 && h <= 37 && s <= 0.82); +} + +function hsl(r8: number, g8: number, b8: number): [number, number, number] { + const r = r8 / 255, g = g8 / 255, b = b8 / 255; + const max = Math.max(r, g, b), min = Math.min(r, g, b); + const l = (max + min) / 2; + const d = max - min; + if (d === 0) return [0, 0, l]; + const s = d / (1 - Math.abs(2 * l - 1)); + let h = max === r ? ((g - b) / d) % 6 : max === g ? (b - r) / d + 2 : (r - g) / d + 4; + h *= 60; + if (h < 0) h += 360; + return [h, s, l]; +} + export function rgbToCssString({ r, g, b }: Rgb): string { return `rgb(${r} ${g} ${b})`; } diff --git a/web/src/lib/player/audioLoader.ts b/web/src/lib/player/audioLoader.ts index 50d1aa97..aaaf2739 100644 --- a/web/src/lib/player/audioLoader.ts +++ b/web/src/lib/player/audioLoader.ts @@ -1,13 +1,12 @@ -// Audio loader seam. Today: returns the server-relative stream URL -// directly. In the future Tauri shell, the loader can be swapped to -// return blob URLs backed by a Rust-side cache (mirroring the Flutter -// drift+LockCachingAudioSource pattern), without touching the player UI. +// Audio loader seam: returns the server-relative stream URL directly. A +// native shell can swap the loader to return blob URLs backed by its own +// cache, without touching the player UI. // // Two operations: // - resolve(track) — returns the URL the
    -
    diff --git a/web/src/routes/admin/+layout.svelte b/web/src/routes/admin/+layout.svelte index cf0baf47..84c02180 100644 --- a/web/src/routes/admin/+layout.svelte +++ b/web/src/routes/admin/+layout.svelte @@ -1,10 +1,54 @@ -
    - - {@render children?.()} +
    +

    Admin

    +
    +
    +
    + +
    +
    +
    + {@render children?.()} +
    +
    diff --git a/web/src/routes/admin/+page.svelte b/web/src/routes/admin/+page.svelte index 68dd3836..4e2c8a89 100644 --- a/web/src/routes/admin/+page.svelte +++ b/web/src/routes/admin/+page.svelte @@ -1,594 +1,268 @@ {pageTitle('Admin · Overview')}
    -
    -

    Overview

    -
    - - -
    - -
    Pending requests
    -
    {pendingCount}
    -     - -
    Quarantined tracks
    -
    {quarantineCount}
    -     - -
    Lidarr
    -
    - {lidarrConnected ? 'Connected' : 'Unset'} -
    -     -
    - - -
    -
    -

    Pending requests

    - {#if pendingCount > PREVIEW_LIMIT} - View all {pendingCount} → - {/if} -
    - - {#if requests.isPending} -

    Loading…

    - {:else if requestPreview.length === 0} -

    No pending requests.

    + +
    +

    What needs you

    + {#if summaryQ.isPending} +

    Reading the queues…

    + {:else if summaryQ.isError} +

    + Couldn't read the queues, so this page can't say whether anything needs you. + +

    {:else} -
      - {#each requestPreview as r (r.id)} - {@const Icon = fallbackIcon(r.kind)} -
    • - -
      -
      {rowTitle(r)}
      -
      {rowSubtitle(r)}
      -
      - - -
      • - {/each} -
    - {/if} -
    - - -
    -
    -
    -

    Library scan

    -

    - Walks the music tree, fills missing MusicBrainz IDs from tags, and refetches album covers. -

    -
    - -
    - - {#if !scan || !scan.id} -

    No scan has run yet.

    - {:else} -
    - -
    -
    -
    Library walk
    - -
    - {#if scan.library} -
    -
    Scanned
    {scan.library.scanned}
    -
    Added
    {scan.library.added}
    -
    Updated
    {scan.library.updated}
    -
    Skipped
    {scan.library.skipped}
    -
    Errored
    {scan.library.errored}
    -
    - {:else if stageState(scan, 'library') === 'pending'} -

    Waiting for previous stage…

    - {/if} -
    - - -
    -
    -
    MBID backfill
    - -
    - {#if scan.mbid_backfill} -
    -
    Processed
    {scan.mbid_backfill.processed}
    -
    Healed
    {scan.mbid_backfill.healed}
    -
    Skipped
    {scan.mbid_backfill.skipped}
    -
    Duplicates
    {scan.mbid_backfill.duplicates ?? 0}
    -
    - {:else if stageState(scan, 'mbid_backfill') === 'pending'} -

    Waiting for previous stage…

    - {/if} -
    - - -
    -
    -
    Cover enrichment
    - -
    - {#if scan.cover_enrich} -
    -
    Processed
    {scan.cover_enrich.processed}
    -
    Succeeded
    {scan.cover_enrich.succeeded}
    -
    Failed
    {scan.cover_enrich.failed}
    -
    - {:else if stageState(scan, 'cover_enrich') === 'pending'} -

    Waiting for previous stage…

    - {/if} -
    - - -
    -
    -
    Artist art
    - -
    - {#if scan.artist_art_enrich} -
    -
    Processed
    {scan.artist_art_enrich.processed}
    -
    Succeeded
    {scan.artist_art_enrich.succeeded}
    -
    Failed
    {scan.artist_art_enrich.failed}
    -
    - {:else if stageState(scan, 'artist_art_enrich') === 'pending'} -

    Waiting for previous stage…

    - {/if} -
    -
    - -
    - Started {formatTime(scan.started_at)} - {#if scan.finished_at}· Finished {formatTime(scan.finished_at)}{/if} - {#if scan.error_message}· Error: {scan.error_message}{/if} -
    - {/if} - - {#if triggerResult} -

    {triggerResult}

    - {/if} - - - {#if fingerprints && fingerprints.total > 0} -
    - Fingerprints - {fingerprints.fingerprinted.toLocaleString()} of {fingerprints.total.toLocaleString()} tracks - {#if fingerprints.pending > 0} - · - {fingerprints.pending.toLocaleString()} pending - {/if} - {#if fingerprints.enabled === false} - - · - fingerprinting is off - {/if} - {#if fingerprints.rejected > 0} - · - - {fingerprints.rejected.toLocaleString()} unreadable - - {/if} -
    - {/if} -
    - - - - - -
    -

    Cover art

    -

    - Refetch cover art for albums that are missing one or where the previous attempt failed. -

    -
    - - - - {#if coverage} - {#if coverage.total === 0} - Library is empty. + {@const problems = waiting.length + (scan?.error_message ? 1 : 0) + statusProblems.length} +

    + {#if problems === 0} + Nothing needs you. {:else} -

    - {coverage.with_art.toLocaleString()} with art - · - 0 ? 'cursor-help' : ''} - title={coverage.pending_no_mbid > 0 - ? `${coverage.pending_no_mbid.toLocaleString()} of these have no MBID and won't be helped by another scan — fix tags or merge duplicate albums.` - : undefined} - > - {coverage.pending.toLocaleString()} pending - - · - {coverage.settled.toLocaleString()} settled + {plural(problems, 'thing needs you.', 'things need you.')} + {/if} +

    + {#if problems > 0} + + {/if} + {#if clear.length > 0} +

    + No {clear.map((d) => d.none).join(', ')}. +

    + {/if} + {/if} +
    + + {#if statusQ.isError} +

    + Couldn't read the server's status. + +

    + {:else if status} + {@const l = status.listening} + {@const dl = status.downloads} + {@const lib = status.library} + {@const srv = status.server} +
    +

    Listening now

    + {#if l.now.length === 0} +

    Nobody is listening.

    + {:else} +
      + {#each l.now as p (p.username + p.started_at)} +
    • + {p.username} + {p.title} · {p.artist_name} + {p.client || p.device_class || 'a client'} · started {relativeTime(p.started_at)} +
      • + {/each} +
    + {/if} +

    + {#if l.plays_day > 0} + {plural(l.plays_day, 'play', 'plays')} by {plural(l.listeners_day, 'person', 'people')} in the last day{#if l.last_play_at}, + the latest {relativeTime(l.last_play_at)}{/if}. + {:else} + No plays in the last day. + {/if} +

    +
    + +
    +

    Downloads

    +

    + {dl.running === 0 ? 'Nothing downloading' : `${dl.running} downloading`}. + In the last day, {dl.imported_day} imported{dl.given_up_day ? ` and ${dl.given_up_day} given up` : ''}. + Activity +

    + {#if dl.live.length > 0} +
      + {#each dl.live as d (d.id)} +
    • + + {d.want_title ? `${d.want_title} · ${d.want_artist}` : d.name} + + {downloadLine(d)} · {relativeTime(d.grabbed_at)} +
      • + {/each} +
    + {/if} +
    + +
    +

    Library

    +

    + {plural(lib.tracks, 'track', 'tracks')} on {plural(lib.albums, 'album', 'albums')} by {plural(lib.artists, 'artist', 'artists')}, + {formatBytes(lib.bytes)}. + {#if scan?.in_flight} + Scanning now. + {:else if scan?.finished_at} + Last scanned {relativeTime(scan.finished_at)}. + {/if} + Library +

    + {#each lib.disks as d (d.path)} + {#if !d.error} + {@const used = 1 - freeShare(d)} +
    +
    + {#if d.data}Minstrel's data · {/if}{d.path} + {formatBytes(d.free_bytes)} free of {formatBytes(d.total_bytes)} +
    +
    {/if} - {/if} -
    - {#if bulkResult} -

    {bulkResult}

    - {/if} -
    + {/each} +
    - -
    -
    -

    Quarantine

    - {#if quarantineCount > PREVIEW_LIMIT} - View all {quarantineCount} → - {/if} -
    - - {#if quarantine.isPending} -

    Loading…

    - {:else if quarantinePreview.length === 0} -

    No quarantined tracks.

    - {:else} -
      - {#each quarantinePreview as row (row.track_id)} - {@const fileKey = confirmKey(row.track_id, 'delete-file')} - {@const lidarrKey = confirmKey(row.track_id, 'delete-lidarr')} - {@const fileArmed = confirmingKey === fileKey} - {@const lidarrArmed = confirmingKey === lidarrKey} -
    • - -
      -
      {row.track_title}
      -
      - {row.artist_name} · {row.album_title} - · {row.report_count} {row.report_count === 1 ? 'report' : 'reports'} - {#if row.reports[0]} - · {REASON_LABELS[row.reports[0].reason] ?? row.reports[0].reason} - {/if} +
      +

      Server

      +

      + Version {srv.version} ({srv.channel}), database schema {srv.migration}, started {relativeTime(srv.started_at)}. + {#if srv.error_count === 0} + No errors since. + {:else} + {plural(srv.error_count, 'error', 'errors')} logged since{srv.error_count > srv.errors.length + ? `; the latest ${srv.errors.length}:` + : ':'} + {/if} +

      + {#if srv.errors.length > 0} +
        + {#each srv.errors as e, i (i)} +
      • +
        + {e.message} + {relativeTime(e.at)}
        -
      - - - - - - -
      • - {/each} -
    - {/if} -
    + {#if e.attrs} +

    {e.attrs}

    + {/if} + + {/each} + + {/if} +
    + {/if} diff --git a/web/src/routes/admin/acquisition/+page.svelte b/web/src/routes/admin/acquisition/+page.svelte new file mode 100644 index 00000000..1f306a69 --- /dev/null +++ b/web/src/routes/admin/acquisition/+page.svelte @@ -0,0 +1,25 @@ + + +{pageTitle('Admin · Acquisition')} + +

    Acquisition

    + + + {#snippet main()} + + + {/snippet} + {#snippet side()} +
    + {/snippet} +     diff --git a/web/src/routes/admin/admin.test.ts b/web/src/routes/admin/admin.test.ts index 2ec14f48..5daebdc4 100644 --- a/web/src/routes/admin/admin.test.ts +++ b/web/src/routes/admin/admin.test.ts @@ -19,44 +19,15 @@ vi.mock('$app/state', () => ({ vi.mock('#lib/api/admin.js', async () => { const { readable } = await import('svelte/store'); return { - createAdminRequestsQuery: vi.fn(), - createLidarrConfigQuery: vi.fn(), - createAdminQuarantineQuery: vi.fn(), + createAdminSummaryQuery: vi.fn(), + createAdminStatusQuery: vi.fn(), createScanStatusQuery: vi.fn(() => readable({ data: { id: '', started_at: '', finished_at: null, in_flight: false }, isPending: false, isError: false }) - ), - createCoverageQuery: () => - readable({ - data: undefined, - isPending: false, - isError: false - }), - createFingerprintCoverageQuery: () => - readable({ - data: undefined, - isPending: false, - isError: false - }), - approveRequest: vi.fn().mockResolvedValue({}), - rejectRequest: vi.fn().mockResolvedValue({}), - resolveQuarantine: vi.fn().mockResolvedValue({}), - deleteQuarantineFile: vi.fn().mockResolvedValue({}), - deleteQuarantineViaLidarr: vi.fn().mockResolvedValue({}), - triggerScan: vi.fn().mockResolvedValue({}), - refetchMissingCovers: vi.fn().mockResolvedValue({ started: true }), - researchMissingArt: vi.fn().mockResolvedValue({ version: 1 }), - // LoudnessSettingsCard, rendered by the page and tested on its own. - getLoudnessSettings: vi - .fn() - .mockResolvedValue({ enabled: true, backfill_concurrency: 2, leveled_cache_mb: 2048 }), - updateLoudnessSettings: vi.fn(), - getLoudnessCoverage: vi.fn().mockResolvedValue({ - total: 0, measured: 0, silent: 0, unreadable: 0, pending: 0, enabled: true - }) + ) }; }); @@ -95,188 +66,235 @@ describe('admin/+layout.ts load gate', () => { // where the page picked up empty data despite `mockReturnValue` being set. // Importing at module load means the mocks attach to the same handles the // page binds against. -import { - createAdminRequestsQuery, - createLidarrConfigQuery, - createAdminQuarantineQuery, - createScanStatusQuery, - approveRequest, - rejectRequest, - resolveQuarantine, - deleteQuarantineFile, - deleteQuarantineViaLidarr, - researchMissingArt -} from '#lib/api/admin.js'; +import { createAdminStatusQuery, createAdminSummaryQuery, createScanStatusQuery } from '#lib/api/admin.js'; +import type { AdminStatus, AdminSummary, Download } from '#lib/api/types.js'; import OverviewPage from './+page.svelte'; +import AdminLayout from './+layout.svelte'; +import { createRawSnippet } from 'svelte'; -function setupOverview(opts: { - requests?: unknown[]; - quarantine?: unknown[]; - lidarr?: { enabled: boolean }; -} = {}) { - (createAdminRequestsQuery as ReturnType).mockReturnValue( - mockQuery({ data: opts.requests ?? [] }) +const DAY = 24 * 3_600_000; +const ago = (days: number) => new Date(Date.now() - days * DAY).toISOString(); +const empty = { count: 0, since: null }; + +function summary(over: Partial = {}): AdminSummary { + return { + requests: empty, + reports: empty, + duplicates: empty, + suspect: empty, + missing: empty, + ...over + }; +} + +function setupOverview(data: AdminSummary | null = summary(), refetch = vi.fn(), st?: AdminStatus) { + (createAdminSummaryQuery as ReturnType).mockReturnValue( + data + ? mockQuery({ data, refetch }) + : mockQuery({ isError: true, error: new Error('boom'), refetch }) ); - (createAdminQuarantineQuery as ReturnType).mockReturnValue( - mockQuery({ data: opts.quarantine ?? [] }) - ); - (createLidarrConfigQuery as ReturnType).mockReturnValue( - mockQuery({ data: opts.lidarr ?? { enabled: false } }) + (createAdminStatusQuery as ReturnType).mockReturnValue( + st ? mockQuery({ data: st }) : mockQuery({ isPending: true }) ); } +const minutesAgo = (m: number) => new Date(Date.now() - m * 60_000).toISOString(); + +function status(over: Partial = {}): AdminStatus { + return { + listening: { now: [], plays_day: 0, listeners_day: 0, last_play_at: null }, + downloads: { + running: 0, needs_you: 0, needs_you_since: null, imported_day: 0, given_up_day: 0, live: [], clients_down: [] + }, + library: { + tracks: 41_000, albums: 3_200, artists: 900, bytes: 1_400_000_000_000, + disks: [{ path: '/music', total_bytes: 4_000_000_000_000, free_bytes: 2_000_000_000_000 }] + }, + server: { + version: '2026.10.11.1840', channel: 'dev', started_at: minutesAgo(180), migration: 92, dirty: false, + errors: [], error_count: 0 + }, + ...over + }; +} + +// The text of a testid'd element with its whitespace folded. +const textOf = (id: string) => (screen.getByTestId(id).textContent ?? '').replace(/\s+/g, ' ').trim(); + describe('admin Overview page', () => { - test('renders top stat cards (requests, quarantine, lidarr)', async () => { - setupOverview({ - requests: [{ id: '1' }, { id: '2' }], - quarantine: [{ track_id: 't1', report_count: 2, reports: [] }], - lidarr: { enabled: true } - }); - render(OverviewPage); - // "Pending requests" appears as both stat-card label and section - // header — assert via the link role to disambiguate to the stat card. - const reqLink = screen.getByRole('link', { name: /pending requests/i }); - expect(reqLink).toHaveTextContent('2'); - const quarLink = screen.getByRole('link', { name: /quarantined tracks/i }); - expect(quarLink).toHaveTextContent('1'); - expect(screen.getByText(/connected/i)).toBeInTheDocument(); - }); - - test('shows empty-state copy when both queues are empty', async () => { - setupOverview({ - requests: [], - quarantine: [], - lidarr: { enabled: false } - }); - render(OverviewPage); - expect(screen.getByText(/no pending requests/i)).toBeInTheDocument(); - expect(screen.getByText(/no quarantined tracks/i)).toBeInTheDocument(); - expect(screen.getByText(/unset/i)).toBeInTheDocument(); - }); - - test('top stat cards link to their sub-pages', async () => { + test('with every queue empty, the verdict says nothing needs you, and names what is clear', () => { setupOverview(); render(OverviewPage); - expect(screen.getByRole('link', { name: /pending requests/i })) - .toHaveAttribute('href', '/admin/requests'); - expect(screen.getByRole('link', { name: /quarantined tracks/i })) - .toHaveAttribute('href', '/admin/quarantine'); - expect(screen.getByRole('link', { name: /^lidarr/i })) - .toHaveAttribute('href', '/admin/integrations'); + expect(screen.getByTestId('verdict').textContent).toMatch(/nothing needs you/i); + expect(screen.queryAllByTestId('needs-you')).toHaveLength(0); + expect(screen.getByTestId('all-clear').textContent).toMatch(/no requests, reports, duplicates, suspect sources, missing files/i); }); - test('clicking Approve on a request row calls approveRequest', async () => { - setupOverview({ - requests: [ - { id: 'r1', kind: 'artist', artist_name: 'Boards of Canada', status: 'pending' } - ] - }); - render(OverviewPage); - await fireEvent.click(screen.getByRole('button', { name: /approve boards of canada/i })); - expect(approveRequest).toHaveBeenCalledWith('r1'); - }); - - test('clicking Reject on a request row calls rejectRequest', async () => { - setupOverview({ - requests: [ - { id: 'r1', kind: 'artist', artist_name: 'Boards of Canada', status: 'pending' } - ] - }); - render(OverviewPage); - await fireEvent.click(screen.getByRole('button', { name: /reject boards of canada/i })); - expect(rejectRequest).toHaveBeenCalledWith('r1'); - }); - - test('Resolve fires immediately on a quarantine row', async () => { - setupOverview({ - quarantine: [ - { track_id: 't1', track_title: 'Roygbiv', artist_name: 'BoC', - album_title: 'Geogaddi', report_count: 1, reports: [] } - ] - }); - render(OverviewPage); - await fireEvent.click(screen.getByRole('button', { name: /resolve roygbiv/i })); - expect(resolveQuarantine).toHaveBeenCalledWith('t1'); - }); - - test('Delete file requires two clicks (inline confirm)', async () => { - setupOverview({ - quarantine: [ - { track_id: 't1', track_title: 'Roygbiv', artist_name: 'BoC', - album_title: 'Geogaddi', report_count: 1, reports: [] } - ] - }); - render(OverviewPage); - // First click: arms the confirm; no API call yet. - await fireEvent.click(screen.getByRole('button', { name: /^delete file roygbiv$/i })); - expect(deleteQuarantineFile).not.toHaveBeenCalled(); - // Second click on the now-armed button fires the action. - await fireEvent.click( - screen.getByRole('button', { name: /confirm delete file roygbiv/i }) - ); - expect(deleteQuarantineFile).toHaveBeenCalledWith('t1'); - }); - - test('Delete via Lidarr requires two clicks (inline confirm)', async () => { - setupOverview({ - quarantine: [ - { track_id: 't1', track_title: 'Roygbiv', artist_name: 'BoC', - album_title: 'Geogaddi', report_count: 1, reports: [] } - ] - }); - render(OverviewPage); - await fireEvent.click( - screen.getByRole('button', { name: /^delete via lidarr roygbiv$/i }) - ); - expect(deleteQuarantineViaLidarr).not.toHaveBeenCalled(); - await fireEvent.click( - screen.getByRole('button', { name: /confirm delete via lidarr roygbiv/i }) - ); - expect(deleteQuarantineViaLidarr).toHaveBeenCalledWith('t1'); - }); - - test('"View all" link surfaces only when count exceeds the preview limit', async () => { - // 6 requests > limit of 5; quarantine empty. - const six = Array.from({ length: 6 }, (_, i) => ({ - id: `r${i}`, - kind: 'artist' as const, - artist_name: `Artist ${i}`, - status: 'pending' as const + test('each queue that waits on you is listed with how long its oldest has waited, and opens its page', () => { + setupOverview(summary({ + requests: { count: 2, since: ago(3) }, + missing: { count: 1, since: ago(10) } })); - setupOverview({ requests: six }); render(OverviewPage); - expect(screen.getByRole('link', { name: /view all 6/i })).toHaveAttribute( - 'href', '/admin/requests' + expect(screen.getByTestId('verdict').textContent).toMatch(/2 things need you/); + const rows = screen.getAllByTestId('needs-you').map((r) => r.textContent?.replace(/\s+/g, ' ').trim()); + expect(rows).toEqual(['2 requests to decide oldest 3d ago Open', '1 missing file oldest 10d ago Open']); + expect(screen.getByRole('link', { name: /2 requests to decide/i }).getAttribute('href')).toBe('/admin/requests'); + expect(screen.getByTestId('all-clear').textContent).not.toMatch(/requests/); + }); + + // The library's health queues live under Collection; the Overview + // still lists them, linking there. + test('a library queue that waits on you opens its Collection page', () => { + setupOverview(summary({ duplicates: { count: 5, since: ago(4) } })); + render(OverviewPage); + expect(screen.getByRole('link', { name: /5 sets of duplicates to review/i }).getAttribute('href')).toBe( + '/collection/duplicates' ); }); - test('renders Re-search missing art button and invokes on click', async () => { - setupOverview(); - render(OverviewPage); - const btn = screen.getByRole('button', { name: /re-search missing art/i }); - expect(btn).toBeInTheDocument(); - await fireEvent.click(btn); - expect(researchMissingArt).toHaveBeenCalled(); - }); - - test('Library scan card shows Processing badge when library tally is populated and in-flight', async () => { + test('a failed scan needs you too', async () => { const { readable } = await import('svelte/store'); (createScanStatusQuery as ReturnType).mockReturnValueOnce( readable({ - data: { - id: 'in-flight', - started_at: '2026-05-06T11:00:00Z', - finished_at: null, - in_flight: true, - library: { scanned: 100, added: 1, updated: 0, skipped: 99, errored: 0 } - }, + data: { id: 's1', started_at: ago(1), finished_at: ago(1), in_flight: false, error_message: 'library root not mounted' }, isPending: false, isError: false }) ); setupOverview(); - userState.current = { id: 'admin', username: 'admin', is_admin: true }; render(OverviewPage); - expect(await screen.findByText('Processing')).toBeInTheDocument(); + expect(screen.getByTestId('verdict').textContent).toMatch(/1 thing needs you/); + expect(screen.getByText(/the last library scan failed: library root not mounted/i).closest('a')?.getAttribute('href')).toBe( + '/admin/library' + ); + }); + + test('when the queues fail to load it says so, with Try again, rather than reading as all-clear', async () => { + const refetch = vi.fn(); + setupOverview(null, refetch); + render(OverviewPage); + expect(screen.queryByTestId('verdict')).toBeNull(); + expect(screen.getByTestId('summary-error').textContent).toMatch(/can't say whether anything needs you/i); + await fireEvent.click(screen.getByRole('button', { name: /try again/i })); + expect(refetch).toHaveBeenCalled(); + }); +}); + +// Below the verdict, the server's state. +describe('admin Overview status', () => { + test('who is listening, and the last day of plays', () => { + setupOverview(summary(), vi.fn(), status({ + listening: { + now: [{ + username: 'bryan', track_id: 't1', title: 'Avril 14th', artist_name: 'Aphex Twin', album_id: 'al1', + album_title: 'Drukqs', duration_ms: 125_000, started_at: minutesAgo(1), client: 'android', device_class: 'phone' + }], + plays_day: 37, listeners_day: 2, last_play_at: minutesAgo(1) + } + })); + render(OverviewPage); + const rows = screen.getAllByTestId('listening-row'); + expect(rows).toHaveLength(1); + expect(rows[0].textContent?.replace(/\s+/g, ' ')).toMatch(/bryan Avril 14th · Aphex Twin android · started 1m ago/); + expect(textOf('listening-day')).toBe('37 plays by 2 people in the last day, the latest 1m ago.'); + }); + + test('nobody listening says so', () => { + setupOverview(summary(), vi.fn(), status()); + render(OverviewPage); + expect(screen.getByText('Nobody is listening.')).toBeTruthy(); + expect(textOf('listening-day')).toBe('No plays in the last day.'); + }); + + test('the library, its size, and the room left on its disk', () => { + setupOverview(summary(), vi.fn(), status()); + render(OverviewPage); + expect(textOf('library-line')).toMatch(/^41,000 tracks on 3,200 albums by 900 artists, 1.4 TB\./); + expect(textOf('disk')).toBe('/music 2.0 TB free of 4.0 TB'); + }); + + test('the server says its version, when it started, and its recent errors', () => { + setupOverview(summary(), vi.fn(), status({ + server: { + version: '2026.10.11.1840', channel: 'dev', started_at: minutesAgo(180), migration: 92, dirty: false, error_count: 3, + errors: [{ at: minutesAgo(5), message: 'admin: summary', attrs: 'err=boom' }] + } + })); + render(OverviewPage); + expect(textOf('server-line')).toBe('Version 2026.10.11.1840 (dev), database schema 92, started 3h ago. 3 errors logged since; the latest 1:'); + expect(screen.getAllByTestId('server-error')[0].textContent).toMatch(/admin: summary[\s\S]*err=boom/); + }); + + test('a download waiting on you, a client not answering and a nearly full disk count in the verdict', () => { + setupOverview(summary(), vi.fn(), status({ + downloads: { + running: 1, needs_you: 2, needs_you_since: minutesAgo(90), imported_day: 4, given_up_day: 0, live: [], + clients_down: [{ client: 'qbittorrent', detail: 'connection refused', since: minutesAgo(30) }] + }, + library: { + tracks: 1, albums: 1, artists: 1, bytes: 1, + disks: [{ path: '/music', total_bytes: 1_000, free_bytes: 10 }] + } + })); + render(OverviewPage); + expect(screen.getByTestId('verdict').textContent).toMatch(/3 things need you/); + const needs = screen.getAllByTestId('needs-you').map((li) => li.textContent?.replace(/\s+/g, ' ').trim()); + expect(needs).toEqual([ + '2 downloads need you since 1h ago Open', + "qbittorrent isn't answering: connection refused since 30m ago Open", + '/music is nearly full: 10 B free Open' + ]); + expect(textOf('downloads-line')).toBe('1 downloading. In the last day, 4 imported. Activity'); + }); + + // Worded as the wanted list words a download. + test('each live download says what it is doing, as the wanted list says it', () => { + const d = (id: string, import_state: Download['import_state'], import_reason = ''): Download => ({ + id, want_id: null, want_title: `Album ${id}`, want_artist: 'Blur', name: id, indexer: '', client: 'qbittorrent', + quality: 'FLAC', origin: 'schedule', grabbed_at: minutesAgo(5), ended_at: null, end_reason: '', progress: 0.42, + import_state, import_reason, imported_at: null, imported_files: 0, destination: '', files: [], skipped: 0, + end_detail: '', seeding: false, ratio: 0, seed_released_at: null, condition: null + }); + setupOverview(summary(), vi.fn(), status({ + downloads: { + running: 3, needs_you: 1, needs_you_since: minutesAgo(5), imported_day: 0, given_up_day: 0, clients_down: [], + live: [d('a', 'review', '9 files against 12 tracks'), d('b', 'downloading'), d('c', '')] + } + })); + render(OverviewPage); + const rows = screen.getAllByTestId('download-row').map((li) => li.textContent?.replace(/\s+/g, ' ').trim()); + expect(rows).toEqual([ + 'Album a · Blur Needs you: 9 files against 12 tracks · 5m ago', + 'Album b · Blur Downloading 42% · 5m ago', + 'Album c · Blur Downloading · 5m ago' + ]); + }); +}); + +describe('admin layout', () => { + test('the sections run down a rail, each queue carrying how many wait on you', () => { + setupOverview(summary({ requests: { count: 2, since: ago(1) }, duplicates: { count: 5, since: ago(4) } })); + const children = createRawSnippet(() => ({ render: () => '

    body

    ' })); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + render(AdminLayout, { children } as any); + const nav = screen.getByRole('navigation', { name: /admin sections/i }); + const links = Array.from(nav.querySelectorAll('a')).map((a) => [ + a.getAttribute('href'), + a.textContent?.replace(/\s+/g, ' ').trim() + ]); + expect(links).toEqual([ + ['/admin', 'Overview'], + ['/admin/requests', 'Requests 2'], + ['/admin/library', 'Library'], + ['/admin/acquisition', 'Acquisition'], + ['/admin/metadata', 'Metadata sources'], + ['/admin/email', 'Email & network'], + ['/admin/users', 'Users'], + ['/admin/diagnostics', 'Diagnostics'], + ['/admin/tuning', 'Tuning'] + ]); + expect(screen.getByRole('link', { name: /^overview/i })).toHaveAttribute('aria-current', 'page'); + expect(screen.getByTestId('admin-body')).toBeInTheDocument(); }); }); diff --git a/web/src/routes/admin/diagnostics/+page.svelte b/web/src/routes/admin/diagnostics/+page.svelte index 3a3319db..783052cd 100644 --- a/web/src/routes/admin/diagnostics/+page.svelte +++ b/web/src/routes/admin/diagnostics/+page.svelte @@ -1,5 +1,7 @@ + +{pageTitle('Admin · Email & network')} + +

    Email and network

    + + + {#snippet main()} + + + {/snippet} + {#snippet side()} + + + {/snippet} + diff --git a/web/src/routes/admin/integrations/+page.svelte b/web/src/routes/admin/integrations/+page.svelte deleted file mode 100644 index 81d06060..00000000 --- a/web/src/routes/admin/integrations/+page.svelte +++ /dev/null @@ -1,862 +0,0 @@ - - -{pageTitle('Admin · Integrations')} - -
    -
    -

    Integrations

    - {#if config.data?.enabled} - - - Lidarr · connected - - {:else} - - - Lidarr · unset - - {/if} -
    - - -
    -
    -

    Lidarr

    -

    - Search Lidarr from /discover and - route approved requests to it. -

    -
    - - - - - - - - - - - - {#if testResult} - {#if testResult.ok} -

    - Connected — Lidarr {testResult.version} -

    - {:else} -

    Connection failed — {testResult.error}

    - {/if} - {/if} - {#if saveError} -

    Save failed — {saveError}

    - {/if} - -
    - - - -
    -
    - - - - - -
    -
    -

    Cover art providers

    -

    - External sources for album covers and artist images. Providers - are tried in registration order; first hit wins. -

    -
    - - {#if coverProvidersQ.data} - {#each coverProvidersQ.data.providers as provider (provider.id)} -
    -
    -
    -

    {provider.display_name}

    - -
    -
    - {#each provider.supports as cap} - {cap.replace('_', ' ')} - {/each} -
    -
    - - {#if coverLocalState[provider.id]} -
    - -
    - -
    - - {#if provider.requires_api_key} - -
    - - {#if provider.api_key_set} - ✓ Set - {/if} -
    - {/if} -
    - -
    - - {#if provider.testable} - - {/if} - {#if coverTestResults[provider.id]} - {#if coverTestResults[provider.id].ok} -

    - OK{coverTestResults[provider.id].duration_ms ? ` (${coverTestResults[provider.id].duration_ms}ms)` : ''} -

    - {:else} -

    - Failed — {coverTestResults[provider.id].error} -

    - {/if} - {/if} -
    - {/if} - - {#if provider.id === 'theaudiodb'} -

    - Public test key works out of the box. A free personal key gives separate quota - accounting — apply at theaudiodb.com/api_apply.php. -

    - {/if} -
    - {/each} - {:else if coverProvidersQ.isPending} -

    Loading…

    - {:else} -

    No providers registered.

    - {/if} -
    - - -
    -
    -

    Tag enrichment sources

    -

    - Folksonomy style/mood tags fetched per track and folded into each - listener's taste profile alongside the raw file genre — so "Rock" - gains "post-punk / shoegaze / melancholic". All enabled sources are - merged. Enabling or disabling a source re-opens previously-processed - tracks for a fresh pass. -

    -
    - - {#if tagProvidersQ.data} - {#each tagProvidersQ.data.providers as provider (provider.id)} -
    -
    -
    -

    {provider.display_name}

    - -
    -
    - {#each provider.supports as cap} - {cap.replace('_', ' ')} - {/each} -
    -
    - - {#if tagLocalState[provider.id]} -
    - -
    - -
    - - {#if provider.requires_api_key} - -
    - - {#if provider.api_key_set} - ✓ Set - {/if} -
    - {/if} -
    - -
    - - {#if provider.testable} - - {/if} - {#if tagTestResults[provider.id]} - {#if tagTestResults[provider.id].ok} -

    - OK{tagTestResults[provider.id].duration_ms ? ` (${tagTestResults[provider.id].duration_ms}ms)` : ''} -

    - {:else} -

    - Failed — {tagTestResults[provider.id].error} -

    - {/if} - {/if} -
    - {/if} - - {#if provider.id === 'musicbrainz'} -

    - Keyless and on by default — the always-available baseline. Matches tags by - recording MBID, so coverage tracks how well your library is MusicBrainz-tagged. -

    - {:else if provider.id === 'lastfm'} -

    - Name-based (artist + title), so it covers tracks without an MBID. Requires a free - API key — get one at last.fm/api. -

    - {/if} -
    - {/each} - {:else if tagProvidersQ.isPending} -

    Loading…

    - {:else} -

    No providers registered.

    - {/if} -
    - -
    -
    -

    Email (SMTP)

    -

    - Configure outgoing email for password resets and notifications. The "Send test email" - button verifies your config end-to-end. -

    -
    - - - -
    - - - - - - - - - - - - - - - - - - - TLS - -
    - -
    - - -
    -
    - - - - - - -
    - - -

    - This clears the saved configuration. Type - DISCONNECT to remove the - Lidarr connection. -

    - - {#if disconnectError} -

    Disconnect failed — {disconnectError}

    - {/if} -
    - - -
    -     diff --git a/web/src/routes/admin/integrations/integrations.test.ts b/web/src/routes/admin/integrations/integrations.test.ts deleted file mode 100644 index cfb148e9..00000000 --- a/web/src/routes/admin/integrations/integrations.test.ts +++ /dev/null @@ -1,745 +0,0 @@ -import { afterEach, describe, expect, test, vi } from 'vitest'; -import { render, screen, fireEvent, waitFor, within } from '@testing-library/svelte'; -import { mockQuery } from '../../../test-utils/query'; -import type { LidarrConfig } from '#lib/api/types.js'; -import type { CoverProvider, TagProvider, SMTPConfig } from '#lib/api/admin.js'; - -// useQueryClient is wrapped so the page can call invalidateQueries() without -// a real QueryClient context. Per-file mock OVERRIDES the vitest.setup default -// so assertions see this named spy rather than the setup's anonymous fn. -const invalidateQueries = vi.fn(); -vi.mock('@tanstack/svelte-query', async (orig) => { - const actual = (await orig()) as Record; - return { ...actual, useQueryClient: () => ({ invalidateQueries }) }; -}); - -vi.mock('#lib/api/admin.js', () => ({ - createLidarrConfigQuery: vi.fn(), - createQualityProfilesQuery: vi.fn(), - createMetadataProfilesQuery: vi.fn(), - createRootFoldersQuery: vi.fn(), - putLidarrConfig: vi.fn(), - testLidarrConnection: vi.fn(), - createCoverProvidersQuery: vi.fn(), - updateCoverProvider: vi.fn(), - testCoverProvider: vi.fn(), - createTagProvidersQuery: vi.fn(), - updateTagProvider: vi.fn(), - testTagProvider: vi.fn(), - createSMTPConfigQuery: vi.fn(), - updateSMTPConfig: vi.fn(), - testSMTPConfig: vi.fn(), - // AcoustIDCard loads on mount; a never-settling read leaves it on its own - // "Loading AcoustID settings…", which the page's /loading…/i queries skip. - getAcoustID: vi.fn(() => new Promise(() => {})), - updateAcoustIDSettings: vi.fn(), - runAcoustIDLookup: vi.fn(), - listUnsettledAcoustID: vi.fn(), - // NotificationEmailCard loads on mount; left pending like AcoustIDCard. - getNotificationEmailSettings: vi.fn(() => new Promise(() => {})), - updateNotificationEmailSettings: vi.fn() -})); - -import IntegrationsPage from './+page.svelte'; -import { - createLidarrConfigQuery, - createQualityProfilesQuery, - createMetadataProfilesQuery, - createRootFoldersQuery, - putLidarrConfig, - testLidarrConnection, - createCoverProvidersQuery, - updateCoverProvider, - testCoverProvider, - createTagProvidersQuery, - updateTagProvider, - testTagProvider, - createSMTPConfigQuery, - updateSMTPConfig, - testSMTPConfig -} from '#lib/api/admin.js'; - -const cfgConnected: LidarrConfig = { - enabled: true, - base_url: 'http://lidarr.local', - api_key: '***', - default_quality_profile_id: 1, - default_metadata_profile_id: 1, - default_root_folder_path: '/music' -}; - -const cfgUnset: LidarrConfig = { - enabled: false, - base_url: '', - api_key: '', - default_quality_profile_id: 0, - default_metadata_profile_id: 0, - default_root_folder_path: '' -}; - -// Fixture cover providers -const providerMbcaa: CoverProvider = { - id: 'mbcaa', - display_name: 'MusicBrainz CAA', - requires_api_key: false, - supports: ['album_cover'], - enabled: true, - api_key_set: false, - display_order: 0, - testable: true -}; - -const providerTheAudioDB: CoverProvider = { - id: 'theaudiodb', - display_name: 'TheAudioDB', - requires_api_key: true, - supports: ['album_cover', 'artist_thumb', 'artist_fanart'], - enabled: true, - api_key_set: false, - display_order: 1, - testable: true -}; - -const providerTheAudioDBKeySet: CoverProvider = { - ...providerTheAudioDB, - api_key_set: true -}; - -const defaultCoverProviders = { - providers: [providerMbcaa, providerTheAudioDB], - sources_version: 1 -}; - -// Fixture tag providers -const providerMusicbrainz: TagProvider = { - id: 'musicbrainz', - display_name: 'MusicBrainz', - requires_api_key: false, - supports: ['track_tags'], - enabled: true, - api_key_set: false, - display_order: 0, - testable: true -}; - -const providerLastfm: TagProvider = { - id: 'lastfm', - display_name: 'Last.fm', - requires_api_key: true, - supports: ['track_tags'], - enabled: false, - api_key_set: false, - display_order: 1, - testable: true -}; - -const defaultTagProviders = { - providers: [providerMusicbrainz, providerLastfm], - sources_version: 1 -}; - -const testResponseOk = { - ok: true as const, - version: '2.0.5', - quality_profiles: [ - { id: 1, name: 'FLAC' }, - { id: 2, name: 'MP3-320' } - ], - metadata_profiles: [{ id: 1, name: 'Standard' }], - root_folders: [{ path: '/music', accessible: true, free_space: 12345 }] -}; - -const smtpUnset: SMTPConfig = { - enabled: false, - host: '', - port: 587, - username: '', - password: '', - from_address: '', - from_name: 'Minstrel', - use_tls: true -}; - -const smtpConfigured: SMTPConfig = { - enabled: true, - host: 'smtp.example.com', - port: 587, - username: 'user@example.com', - password: '***', - from_address: 'noreply@example.com', - from_name: 'Minstrel', - use_tls: true -}; - -afterEach(() => { - vi.clearAllMocks(); - invalidateQueries.mockReset(); -}); - -function setup(opts: { config?: LidarrConfig; coverProviders?: typeof defaultCoverProviders | null; tagProviders?: typeof defaultTagProviders | null; smtp?: SMTPConfig } = {}) { - (createLidarrConfigQuery as ReturnType).mockReturnValue( - mockQuery({ data: opts.config ?? cfgConnected }) - ); - (createQualityProfilesQuery as ReturnType).mockReturnValue( - mockQuery({ - data: [ - { id: 1, name: 'Standard' }, - { id: 2, name: 'Lossless' } - ] - }) - ); - (createMetadataProfilesQuery as ReturnType).mockReturnValue( - mockQuery({ - data: [ - { id: 1, name: 'Standard' }, - { id: 2, name: 'Singles only' } - ] - }) - ); - (createRootFoldersQuery as ReturnType).mockReturnValue( - mockQuery({ data: [{ path: '/music', accessible: true, free_space: 0 }] }) - ); - const cpData = opts.coverProviders === null ? undefined : (opts.coverProviders ?? defaultCoverProviders); - (createCoverProvidersQuery as ReturnType).mockReturnValue( - mockQuery({ data: cpData, isPending: opts.coverProviders === null }) - ); - const tpData = opts.tagProviders === null ? undefined : (opts.tagProviders ?? defaultTagProviders); - (createTagProvidersQuery as ReturnType).mockReturnValue( - mockQuery({ data: tpData, isPending: opts.tagProviders === null }) - ); - (createSMTPConfigQuery as ReturnType).mockReturnValue( - mockQuery({ data: opts.smtp ?? smtpUnset }) - ); - return render(IntegrationsPage); -} - -// Helper for finding the modal's Disconnect confirm button. Both the panel -// trigger and the modal confirm share the visible name "Disconnect"; the -// modal one is the descendant of role="dialog". -function modalConfirmButton(): HTMLButtonElement { - const matches = screen.getAllByRole('button', { name: /^disconnect$/i }); - const inDialog = matches.find((b) => b.closest('[role="dialog"]') !== null); - if (!inDialog) throw new Error('modal Disconnect button not found'); - return inDialog as HTMLButtonElement; -} - -// Section-scoping helpers. Both Lidarr and the cover-art-providers panel render -// "Save changes", "Test connection", and "API key" controls, so unscoped queries -// match multiple elements. These helpers narrow every query to its own
    . - -function lidarrSection(): HTMLElement { - return screen.getByRole('heading', { name: /^lidarr$/i, level: 3 }).closest('section') as HTMLElement; -} - -function coverProvidersSection(): HTMLElement { - return screen.getByRole('heading', { name: /cover art providers/i, level: 3 }).closest('section') as HTMLElement; -} - -// Each provider is rendered inside an
    . Provider headings are h4. -function providerCard(providerName: string): HTMLElement { - return within(coverProvidersSection()) - .getByRole('heading', { name: providerName, level: 4 }) - .closest('article') as HTMLElement; -} - -function tagProvidersSection(): HTMLElement { - return screen.getByRole('heading', { name: /tag enrichment sources/i, level: 3 }).closest('section') as HTMLElement; -} - -function tagProviderCard(providerName: string): HTMLElement { - return within(tagProvidersSection()) - .getByRole('heading', { name: providerName, level: 4 }) - .closest('article') as HTMLElement; -} - -describe('/admin/integrations', () => { - test('Save calls putLidarrConfig with empty api_key when input is blank', async () => { - setup(); - // Save now runs Test first (commit bca8622); mock both endpoints. The - // test response also has to match enough of the existing config that - // the Save handler treats it as a valid passthrough. - (testLidarrConnection as ReturnType).mockResolvedValueOnce(testResponseOk); - (putLidarrConfig as ReturnType).mockResolvedValueOnce(cfgConnected); - await fireEvent.click(within(lidarrSection()).getByRole('button', { name: /save changes/i })); - await waitFor(() => { - expect(putLidarrConfig).toHaveBeenCalledWith( - expect.objectContaining({ - enabled: true, - base_url: 'http://lidarr.local', - api_key: '', - default_quality_profile_id: 1, - default_root_folder_path: '/music' - }) - ); - }); - }); - - test('Save with new api_key sends the typed value', async () => { - setup(); - const apiInput = screen.getByPlaceholderText(/saved — leave empty to keep/i); - await fireEvent.input(apiInput, { target: { value: 'newkey' } }); - (testLidarrConnection as ReturnType).mockResolvedValueOnce(testResponseOk); - (putLidarrConfig as ReturnType).mockResolvedValueOnce(cfgConnected); - await fireEvent.click(within(lidarrSection()).getByRole('button', { name: /save changes/i })); - await waitFor(() => { - expect(putLidarrConfig).toHaveBeenCalledWith( - expect.objectContaining({ api_key: 'newkey' }) - ); - }); - }); - - test('Test connection renders Lidarr version on success', async () => { - setup(); - (testLidarrConnection as ReturnType).mockResolvedValueOnce({ - ok: true, - version: '2.0.0' - }); - await fireEvent.click( - within(lidarrSection()).getByRole('button', { name: /test connection/i }) - ); - await waitFor(() => - expect(screen.getByText(/lidarr 2\.0\.0/i)).toBeInTheDocument() - ); - }); - - test('Test connection renders error code on failure', async () => { - setup(); - (testLidarrConnection as ReturnType).mockResolvedValueOnce({ - ok: false, - error: 'lidarr_unreachable' - }); - await fireEvent.click( - within(lidarrSection()).getByRole('button', { name: /test connection/i }) - ); - await waitFor(() => - expect(screen.getByText(/lidarr_unreachable/i)).toBeInTheDocument() - ); - }); - - test('Disconnect requires typed confirm; cancelling does not call API', async () => { - setup(); - // Open modal via the panel-level Disconnect button. - await fireEvent.click( - screen.getByRole('button', { name: /^disconnect$/i }) - ); - expect(screen.getByRole('dialog')).toBeInTheDocument(); - // Confirm button is disabled until "DISCONNECT" is typed. - expect(modalConfirmButton().disabled).toBe(true); - await fireEvent.click(screen.getByRole('button', { name: /cancel/i })); - expect(putLidarrConfig).not.toHaveBeenCalled(); - expect(screen.queryByRole('dialog')).not.toBeInTheDocument(); - }); - - test('Disconnect with typed confirmation calls API with cleared config', async () => { - setup(); - await fireEvent.click( - screen.getByRole('button', { name: /^disconnect$/i }) - ); - const confirmInput = screen.getByPlaceholderText('DISCONNECT'); - await fireEvent.input(confirmInput, { target: { value: 'DISCONNECT' } }); - (putLidarrConfig as ReturnType).mockResolvedValueOnce(cfgUnset); - await fireEvent.click(modalConfirmButton()); - expect(putLidarrConfig).toHaveBeenCalledWith( - expect.objectContaining({ - enabled: false, - base_url: '', - api_key: '', - default_quality_profile_id: 0, - default_root_folder_path: '' - }) - ); - }); - - test('Quality / metadata / root folder dropdowns populate from API', () => { - setup(); - // 'Standard' appears in both quality + metadata fixtures, so use *AllBy*. - expect(screen.getAllByText('Standard').length).toBeGreaterThanOrEqual(2); - expect(screen.getByText('Lossless')).toBeInTheDocument(); - expect(screen.getByText('Singles only')).toBeInTheDocument(); - expect(screen.getByText('/music')).toBeInTheDocument(); - }); - - test('header shows "Lidarr · connected" when configured', () => { - setup(); - expect(screen.getByText(/lidarr · connected/i)).toBeInTheDocument(); - }); - - test('header shows "Lidarr · unset" when not configured', () => { - setup({ config: cfgUnset }); - expect(screen.getByText(/lidarr · unset/i)).toBeInTheDocument(); - }); -}); - -describe('/admin/integrations — cover art providers', () => { - test('renders one card per provider in the response', () => { - setup(); - expect(screen.getByText('MusicBrainz CAA')).toBeInTheDocument(); - expect(screen.getByText('TheAudioDB')).toBeInTheDocument(); - }); - - test('capability badges render for each capability in supports[]', () => { - setup(); - // TheAudioDB has album_cover, artist_thumb, artist_fanart - expect(screen.getAllByText('album cover').length).toBeGreaterThanOrEqual(1); - expect(screen.getByText('artist thumb')).toBeInTheDocument(); - expect(screen.getByText('artist fanart')).toBeInTheDocument(); - }); - - test('"✓ Set" indicator renders when api_key_set is true', () => { - setup({ - coverProviders: { - providers: [providerTheAudioDBKeySet], - sources_version: 1 - } - }); - expect(screen.getByText(/✓ set/i)).toBeInTheDocument(); - }); - - test('"✓ Set" indicator does not render when api_key_set is false', () => { - setup(); - expect(screen.queryByText(/✓ set/i)).not.toBeInTheDocument(); - }); - - test('Save button is disabled initially (no changes)', () => { - setup(); - const cpSection = coverProvidersSection(); - const saveButtons = within(cpSection).getAllByRole('button', { name: /save changes/i }); - // Both provider cards render Save buttons, both should be disabled initially - for (const btn of saveButtons) { - expect((btn as HTMLButtonElement).disabled).toBe(true); - } - }); - - test('Save button enables after typing in API key field', async () => { - setup(); - const theAudioDbCard = providerCard('TheAudioDB'); - const keyInput = within(theAudioDbCard).getByLabelText(/api key/i); - await fireEvent.input(keyInput, { target: { value: 'myapikey' } }); - // The TheAudioDB save button should now be enabled - const theAudioDbSave = within(theAudioDbCard).getByRole('button', { name: /save changes/i }); - expect(theAudioDbSave).toBeDefined(); - expect((theAudioDbSave as HTMLButtonElement).disabled).toBe(false); - }); - - test('clicking Save calls updateCoverProvider with api_key patch body', async () => { - setup(); - (updateCoverProvider as ReturnType).mockResolvedValueOnce({ - ...providerTheAudioDB, - version_bumped: false - }); - const theAudioDbCard = providerCard('TheAudioDB'); - const keyInput = within(theAudioDbCard).getByLabelText(/api key/i); - await fireEvent.input(keyInput, { target: { value: 'newkey123' } }); - const btn = within(theAudioDbCard).getByRole('button', { name: /save changes/i }); - await fireEvent.click(btn); - expect(updateCoverProvider).toHaveBeenCalledWith( - 'theaudiodb', - expect.objectContaining({ api_key: 'newkey123' }) - ); - }); - - test('when version_bumped: true, invalidateQueries is called with qk.coverage()', async () => { - setup(); - (updateCoverProvider as ReturnType).mockResolvedValueOnce({ - ...providerMbcaa, - enabled: false, - version_bumped: true - }); - // Toggle enabled on MusicBrainz CAA card - const mbcaaCard = screen.getByText('MusicBrainz CAA').closest('article')!; - const enabledCheckbox = mbcaaCard.querySelector('input[type="checkbox"]') as HTMLInputElement; - await fireEvent.click(enabledCheckbox); - const saveButtons = screen.getAllByRole('button', { name: /save changes/i }); - const mbcaaSave = saveButtons.find((b) => b.closest('article')?.contains(enabledCheckbox)); - await fireEvent.click(mbcaaSave!); - await waitFor(() => { - expect(invalidateQueries).toHaveBeenCalledWith( - expect.objectContaining({ queryKey: ['coverage'] }) - ); - }); - }); - - test('when version_bumped: false, invalidateQueries is NOT called with qk.coverage()', async () => { - setup(); - (updateCoverProvider as ReturnType).mockResolvedValueOnce({ - ...providerMbcaa, - enabled: false, - version_bumped: false - }); - const mbcaaCard = screen.getByText('MusicBrainz CAA').closest('article')!; - const enabledCheckbox = mbcaaCard.querySelector('input[type="checkbox"]') as HTMLInputElement; - await fireEvent.click(enabledCheckbox); - const saveButtons = screen.getAllByRole('button', { name: /save changes/i }); - const mbcaaSave = saveButtons.find((b) => b.closest('article')?.contains(enabledCheckbox)); - await fireEvent.click(mbcaaSave!); - await waitFor(() => { - expect(invalidateQueries).toHaveBeenCalledWith( - expect.objectContaining({ queryKey: ['coverProviders'] }) - ); - }); - // coverage invalidation should NOT have been called - const coverageCalls = invalidateQueries.mock.calls.filter( - (call: any[]) => JSON.stringify(call[0]?.queryKey) === JSON.stringify(['coverage']) - ); - expect(coverageCalls.length).toBe(0); - }); - - test('Test connection button calls testCoverProvider and shows OK result', async () => { - setup(); - (testCoverProvider as ReturnType).mockResolvedValueOnce({ - ok: true, - duration_ms: 120 - }); - const mbcaaCard = providerCard('MusicBrainz CAA'); - await fireEvent.click(within(mbcaaCard).getByRole('button', { name: /test connection/i })); - await waitFor(() => - expect(screen.getByText(/ok.*120ms/i)).toBeInTheDocument() - ); - }); - - test('Test connection button shows failure message on error response', async () => { - setup(); - (testCoverProvider as ReturnType).mockResolvedValueOnce({ - ok: false, - error: 'mbcaa_unreachable' - }); - const mbcaaCard = providerCard('MusicBrainz CAA'); - await fireEvent.click(within(mbcaaCard).getByRole('button', { name: /test connection/i })); - await waitFor(() => - expect(screen.getByText(/mbcaa_unreachable/i)).toBeInTheDocument() - ); - }); - - test('TheAudioDB help text renders with link', () => { - setup(); - expect(screen.getByText(/public test key works out of the box/i)).toBeInTheDocument(); - expect(screen.getByRole('link', { name: /theaudiodb\.com\/api_apply\.php/i })).toBeInTheDocument(); - }); - - test('shows loading state when cover providers query is pending', () => { - setup({ coverProviders: null }); - expect(screen.getByText(/loading…/i)).toBeInTheDocument(); - }); -}); - -function smtpSection(): HTMLElement { - return screen.getByRole('heading', { name: /email \(smtp\)/i, level: 3 }).closest('section') as HTMLElement; -} - -describe('/admin/integrations — SMTP card', () => { - test('renders "Email (SMTP)" heading', () => { - setup(); - expect(screen.getByRole('heading', { name: /email \(smtp\)/i, level: 3 })).toBeInTheDocument(); - }); - - test('Save calls updateSMTPConfig with form values; empty password preserves stored value', async () => { - setup({ smtp: smtpConfigured }); - (updateSMTPConfig as ReturnType).mockResolvedValue(undefined); - // password input starts empty — should send '' which tells backend to preserve - const section = smtpSection(); - await fireEvent.click(within(section).getByRole('button', { name: /^save$/i })); - await waitFor(() => - expect(updateSMTPConfig).toHaveBeenCalledWith( - expect.objectContaining({ password: '' }) - ) - ); - }); - - test('Save with a typed password sends the typed value', async () => { - setup({ smtp: smtpConfigured }); - (updateSMTPConfig as ReturnType).mockResolvedValue(undefined); - const section = smtpSection(); - const pwInput = within(section).getByLabelText(/password/i); - await fireEvent.input(pwInput, { target: { value: 'newsecret' } }); - await fireEvent.click(within(section).getByRole('button', { name: /^save$/i })); - await waitFor(() => - expect(updateSMTPConfig).toHaveBeenCalledWith( - expect.objectContaining({ password: 'newsecret' }) - ) - ); - }); - - test('Test button calls testSMTPConfig', async () => { - setup(); - (testSMTPConfig as ReturnType).mockResolvedValue(undefined); - const section = smtpSection(); - await fireEvent.click(within(section).getByRole('button', { name: /send test email/i })); - await waitFor(() => expect(testSMTPConfig).toHaveBeenCalled()); - }); - - test('no_email_on_file error from testSMTPConfig surfaces actionable toast', async () => { - setup(); - (testSMTPConfig as ReturnType).mockRejectedValueOnce({ code: 'no_email_on_file' }); - const section = smtpSection(); - await fireEvent.click(within(section).getByRole('button', { name: /send test email/i })); - await waitFor(() => - expect(screen.getByTestId('toast').textContent).toMatch(/set your email in \/settings/i) - ); - }); - - test('password input placeholder shows hint when existing password is masked', async () => { - setup({ smtp: smtpConfigured }); - // smtpConfigured has password: '***' — placeholder should show "Leave blank to keep current" - const section = smtpSection(); - await waitFor(() => - expect(within(section).getByPlaceholderText(/leave blank to keep current/i)).toBeInTheDocument() - ); - }); -}); - -describe('Lidarr first-time setup (Test-then-Save)', () => { - // Fresh-install variant of setup() — the three list queries return - // empty arrays so we can verify the dropdowns get populated from the - // test response, not from the (empty) cached query data. - function setupFresh() { - (createLidarrConfigQuery as ReturnType).mockReturnValue( - mockQuery({ data: cfgUnset }) - ); - (createQualityProfilesQuery as ReturnType).mockReturnValue(mockQuery({ data: [] })); - (createMetadataProfilesQuery as ReturnType).mockReturnValue(mockQuery({ data: [] })); - (createRootFoldersQuery as ReturnType).mockReturnValue(mockQuery({ data: [] })); - (createCoverProvidersQuery as ReturnType).mockReturnValue( - mockQuery({ data: defaultCoverProviders }) - ); - (createSMTPConfigQuery as ReturnType).mockReturnValue( - mockQuery({ data: smtpUnset }) - ); - return render(IntegrationsPage); - } - - test('Save calls test first, fills defaults from response, then put-config', async () => { - setupFresh(); - (testLidarrConnection as ReturnType).mockResolvedValueOnce(testResponseOk); - (putLidarrConfig as ReturnType).mockResolvedValueOnce(cfgConnected); - - const section = lidarrSection(); - await fireEvent.input(within(section).getByLabelText(/Base URL/i), { - target: { value: 'http://lidarr.local' } - }); - await fireEvent.input(within(section).getByLabelText(/API key/i), { - target: { value: 'secret-key' } - }); - await fireEvent.click(within(section).getByRole('button', { name: /Save changes/i })); - - await waitFor(() => { - expect(testLidarrConnection).toHaveBeenCalledWith({ - base_url: 'http://lidarr.local', - api_key: 'secret-key' - }); - }); - await waitFor(() => { - expect(putLidarrConfig).toHaveBeenCalledWith({ - enabled: true, - base_url: 'http://lidarr.local', - api_key: 'secret-key', - default_quality_profile_id: 1, - default_metadata_profile_id: 1, - default_root_folder_path: '/music' - }); - }); - }); - - test('Save aborts when test fails and surfaces the error', async () => { - setupFresh(); - (testLidarrConnection as ReturnType).mockResolvedValueOnce({ - ok: false, - error: 'lidarr_unreachable' - }); - - const section = lidarrSection(); - await fireEvent.input(within(section).getByLabelText(/Base URL/i), { - target: { value: 'http://wrong.local' } - }); - await fireEvent.input(within(section).getByLabelText(/API key/i), { - target: { value: 'k' } - }); - await fireEvent.click(within(section).getByRole('button', { name: /Save changes/i })); - - await waitFor(() => { - expect(testLidarrConnection).toHaveBeenCalled(); - }); - expect(putLidarrConfig).not.toHaveBeenCalled(); - await waitFor(() => { - expect(within(lidarrSection()).getByText(/save failed — lidarr_unreachable/i)).toBeInTheDocument(); - }); - }); - - test('Save preserves operator-picked defaults over auto-defaults', async () => { - setupFresh(); - (testLidarrConnection as ReturnType).mockResolvedValue(testResponseOk); - (putLidarrConfig as ReturnType).mockResolvedValueOnce(cfgConnected); - - const section = lidarrSection(); - await fireEvent.input(within(section).getByLabelText(/Base URL/i), { - target: { value: 'http://lidarr.local' } - }); - await fireEvent.input(within(section).getByLabelText(/API key/i), { - target: { value: 'secret-key' } - }); - // Click Test first to populate the dropdowns, then change the - // quality profile selection before clicking Save. - await fireEvent.click(within(section).getByRole('button', { name: /Test connection/i })); - await waitFor(() => { - expect(testLidarrConnection).toHaveBeenCalledTimes(1); - }); - const qualitySelect = within(section).getByLabelText(/Default quality profile/i); - await fireEvent.change(qualitySelect, { target: { value: '2' } }); - - await fireEvent.click(within(section).getByRole('button', { name: /Save changes/i })); - - await waitFor(() => { - expect(putLidarrConfig).toHaveBeenCalledWith( - expect.objectContaining({ - default_quality_profile_id: 2, - default_metadata_profile_id: 1, - default_root_folder_path: '/music' - }) - ); - }); - }); -}); - -describe('/admin/integrations tag sources', () => { - test('renders tag enrichment providers', () => { - setup(); - const section = tagProvidersSection(); - expect(within(section).getByRole('heading', { name: 'MusicBrainz', level: 4 })).toBeInTheDocument(); - expect(within(section).getByRole('heading', { name: 'Last.fm', level: 4 })).toBeInTheDocument(); - expect(within(section).getAllByText('track tags').length).toBeGreaterThanOrEqual(1); - }); - - test('enabling Last.fm and typing a key calls updateTagProvider', async () => { - setup(); - (updateTagProvider as ReturnType).mockResolvedValueOnce({ - ...providerLastfm, - enabled: true, - api_key_set: true, - version_bumped: true - }); - const card = tagProviderCard('Last.fm'); - const enabledCheckbox = card.querySelector('input[type="checkbox"]') as HTMLInputElement; - await fireEvent.click(enabledCheckbox); - const keyInput = within(card).getByLabelText(/api key/i); - await fireEvent.input(keyInput, { target: { value: 'lfmkey' } }); - await fireEvent.click(within(card).getByRole('button', { name: /save changes/i })); - - expect(updateTagProvider).toHaveBeenCalledWith( - 'lastfm', - expect.objectContaining({ enabled: true, api_key: 'lfmkey' }) - ); - await waitFor(() => - expect(invalidateQueries).toHaveBeenCalledWith( - expect.objectContaining({ queryKey: ['tagProviders'] }) - ) - ); - }); - - test('Test connection on a tag provider calls testTagProvider and shows OK', async () => { - setup(); - (testTagProvider as ReturnType).mockResolvedValueOnce({ ok: true, duration_ms: 42 }); - const card = tagProviderCard('MusicBrainz'); - await fireEvent.click(within(card).getByRole('button', { name: /test connection/i })); - await waitFor(() => expect(testTagProvider).toHaveBeenCalledWith('musicbrainz')); - await waitFor(() => expect(within(card).getByText(/ok.*42ms/i)).toBeInTheDocument()); - }); -}); diff --git a/web/src/routes/admin/library/+page.svelte b/web/src/routes/admin/library/+page.svelte new file mode 100644 index 00000000..657e0dc8 --- /dev/null +++ b/web/src/routes/admin/library/+page.svelte @@ -0,0 +1,213 @@ + + +{pageTitle('Admin · Library')} + +

    Library

    + + + {#snippet main()} + + + { + client.invalidateQueries({ queryKey: qk.scanStatus() }); + client.invalidateQueries({ queryKey: qk.searchSettings() }); + }} + /> + + + + {#if scan?.id} + {#each jobs as job (job.key)} +
    + + + +
    + {/each} + {/if} +     + + + + + + {#if coverage && coverage.pending_no_mbid > 0} + + {/if} + + + + + {/snippet} + {#snippet side()} + {#if prints && prints.total > 0} + + + + {/if} +
    + client.invalidateQueries({ queryKey: qk.fingerprintCoverage() })} + /> +
    + + {/snippet} +     diff --git a/web/src/routes/admin/library/library.test.ts b/web/src/routes/admin/library/library.test.ts new file mode 100644 index 00000000..f1563f8c --- /dev/null +++ b/web/src/routes/admin/library/library.test.ts @@ -0,0 +1,114 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor } from '@testing-library/svelte'; +import { mockQuery } from '../../../test-utils/query'; +import type { ScanStatus } from '#lib/api/admin.js'; + +vi.mock('@tanstack/svelte-query', async (orig) => { + const actual = (await orig()) as Record; + return { ...actual, useQueryClient: () => ({ invalidateQueries: vi.fn() }) }; +}); + +vi.mock('#lib/api/admin.js', () => ({ + createScanStatusQuery: vi.fn(), + createCoverageQuery: vi.fn(), + createFingerprintCoverageQuery: vi.fn(), + triggerScan: vi.fn().mockResolvedValue({}), + refetchMissingCovers: vi.fn().mockResolvedValue({ started: true }), + researchMissingArt: vi.fn().mockResolvedValue({ version: 1 }), + // The loudness and fingerprint settings load on mount and have their own + // suites; left pending. + getLoudnessSettings: vi.fn(() => new Promise(() => {})), + updateLoudnessSettings: vi.fn(), + getLoudnessCoverage: vi.fn(() => new Promise(() => {})), + getFingerprintSettings: vi.fn(() => new Promise(() => {})), + updateFingerprintSettings: vi.fn(), + getLibraryFolders: vi.fn().mockResolvedValue({ + saved: false, + folders: [ + { path: '/music', present: true, tracks: 1200 }, + { path: '/more-music', present: true, tracks: 3 } + ] + }), + updateLibraryFolders: vi.fn() +})); + +import LibraryPage from './+page.svelte'; +import { + createScanStatusQuery, + createCoverageQuery, + createFingerprintCoverageQuery, + triggerScan, + researchMissingArt, + refetchMissingCovers +} from '#lib/api/admin.js'; + +const hoursAgo = (h: number) => new Date(Date.now() - h * 3_600_000).toISOString(); + +afterEach(() => vi.clearAllMocks()); + +function setup(scan: Partial = { id: '', started_at: '', finished_at: null, in_flight: false }) { + vi.mocked(createScanStatusQuery).mockReturnValue(mockQuery({ data: scan }) as never); + vi.mocked(createCoverageQuery).mockReturnValue( + mockQuery({ data: { total: 100, with_art: 90, pending: 6, settled: 4, pending_no_mbid: 0 } }) as never + ); + vi.mocked(createFingerprintCoverageQuery).mockReturnValue( + mockQuery({ data: { total: 100, fingerprinted: 80, rejected: 1, pending: 19, enabled: true } }) as never + ); + return render(LibraryPage); +} + +const squash = (el: Element) => el.textContent?.replace(/\s+/g, ' ').trim(); + +describe('Admin › Library', () => { + test('lists the folders Minstrel reads', async () => { + setup(); + const folders = [...(await screen.findByTestId('library-folders')).children].map((el) => el.textContent); + expect(folders).toEqual(['/music', '/more-music']); + }); + + test('before any scan it says so, with no job rows', () => { + setup(); + expect(screen.getByText("Hasn't run yet.")).toBeInTheDocument(); + expect(screen.queryAllByTestId('scan-job')).toHaveLength(0); + }); + + test('a running scan has a row per job, each with what it did and where it is', () => { + setup({ + id: 's1', + started_at: hoursAgo(0.1), + finished_at: null, + in_flight: true, + library: { scanned: 1200, added: 3, updated: 2, skipped: 1195, errored: 0 } + }); + const jobs = screen.getAllByTestId('scan-job').map(squash); + expect(jobs[0]).toBe('Reading files 1,200 read · 3 new · 2 changed Processing'); + expect(jobs[1]).toMatch(/^MusicBrainz ids Pending$/); + expect((screen.getByRole('button', { name: /scanning…/i }) as HTMLButtonElement).disabled).toBe(true); + }); + + test('a failed scan says so on the scan row', () => { + setup({ id: 's1', started_at: hoursAgo(2), finished_at: hoursAgo(1), in_flight: false, error_message: 'library root not mounted' }); + expect(screen.getByText(/failed 1h ago: library root not mounted/i)).toBeInTheDocument(); + }); + + test('Run scan starts one', async () => { + setup(); + await fireEvent.click(screen.getByRole('button', { name: /run scan/i })); + await waitFor(() => expect(triggerScan).toHaveBeenCalled()); + await waitFor(() => expect(screen.getByTestId('toast').textContent).toMatch(/scan started/i)); + }); + + test('cover art says how many have art, and can look for the missing and look again', async () => { + setup(); + expect(screen.getByText('90 of 100 albums have art · 6 still to look for · 4 no source has')).toBeInTheDocument(); + await fireEvent.click(screen.getByRole('button', { name: /look for missing covers/i })); + await waitFor(() => expect(refetchMissingCovers).toHaveBeenCalled()); + await fireEvent.click(screen.getByRole('button', { name: /^look again$/i })); + await waitFor(() => expect(researchMissingArt).toHaveBeenCalled()); + }); + + test('fingerprint progress is one line', () => { + setup(); + expect(screen.getByText('80 of 100 tracks · 19 to go · 1 unreadable')).toBeInTheDocument(); + }); +}); diff --git a/web/src/routes/admin/metadata/+page.svelte b/web/src/routes/admin/metadata/+page.svelte new file mode 100644 index 00000000..8c004fa1 --- /dev/null +++ b/web/src/routes/admin/metadata/+page.svelte @@ -0,0 +1,92 @@ + + +{pageTitle('Admin · Metadata sources')} + +

    Metadata sources

    + +{#snippet coverNote(p: { id: string })} + {#if p.id === 'theaudiodb'} + Works with the public key; + a free key of your own gets its own quota. + {/if} +{/snippet} + +{#snippet tagNote(p: { id: string })} + {#if p.id === 'musicbrainz'} + Needs no key; matches by MusicBrainz id. + {:else if p.id === 'lastfm'} + Matches by artist and title. + Get a free key. + {/if} +{/snippet} + + + {#snippet main()} + + {/snippet} + {#snippet side()} + + + {/snippet} + diff --git a/web/src/routes/admin/metadata/metadata.test.ts b/web/src/routes/admin/metadata/metadata.test.ts new file mode 100644 index 00000000..464deb4c --- /dev/null +++ b/web/src/routes/admin/metadata/metadata.test.ts @@ -0,0 +1,174 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor, within } from '@testing-library/svelte'; +import { mockQuery } from '../../../test-utils/query'; +import type { CoverProvider, TagProvider } from '#lib/api/admin.js'; + +const invalidateQueries = vi.fn(); +vi.mock('@tanstack/svelte-query', async (orig) => { + const actual = (await orig()) as Record; + return { ...actual, useQueryClient: () => ({ invalidateQueries }) }; +}); + +vi.mock('#lib/api/admin.js', () => ({ + createCoverProvidersQuery: vi.fn(), + updateCoverProvider: vi.fn(), + testCoverProvider: vi.fn(), + createTagProvidersQuery: vi.fn(), + updateTagProvider: vi.fn(), + testTagProvider: vi.fn(), + // AcoustIDCard loads on mount and has its own suite; left pending. + getAcoustID: vi.fn(() => new Promise(() => {})), + updateAcoustIDSettings: vi.fn(), + runAcoustIDLookup: vi.fn(), + listUnsettledAcoustID: vi.fn() +})); + +import MetadataPage from './+page.svelte'; +import { + createCoverProvidersQuery, + updateCoverProvider, + testCoverProvider, + createTagProvidersQuery, + updateTagProvider, + testTagProvider +} from '#lib/api/admin.js'; + +const caa: CoverProvider = { + id: 'mbcaa', + display_name: 'MusicBrainz CAA', + requires_api_key: false, + supports: ['album_cover'], + enabled: true, + api_key_set: false, + display_order: 0, + testable: true +}; +const audiodb: CoverProvider = { + id: 'theaudiodb', + display_name: 'TheAudioDB', + requires_api_key: true, + supports: ['album_cover', 'artist_thumb', 'artist_fanart'], + enabled: true, + api_key_set: false, + display_order: 1, + testable: true +}; +const mb: TagProvider = { + id: 'musicbrainz', + display_name: 'MusicBrainz', + requires_api_key: false, + supports: ['track_tags'], + enabled: true, + api_key_set: false, + display_order: 0, + testable: true +}; +const lastfm: TagProvider = { + id: 'lastfm', + display_name: 'Last.fm', + requires_api_key: true, + supports: ['track_tags'], + enabled: false, + api_key_set: false, + display_order: 1, + testable: true +}; + +afterEach(() => { + vi.clearAllMocks(); + invalidateQueries.mockReset(); +}); + +function setup(opts: { cover?: ReturnType; covers?: CoverProvider[] } = {}) { + vi.mocked(createCoverProvidersQuery).mockReturnValue( + (opts.cover ?? mockQuery({ data: { providers: opts.covers ?? [caa, audiodb], sources_version: 1 } })) as never + ); + vi.mocked(createTagProvidersQuery).mockReturnValue( + mockQuery({ data: { providers: [mb, lastfm], sources_version: 1 } }) as never + ); + return render(MetadataPage); +} + +const source = (id: string) => screen.getAllByTestId('provider').find((el) => el.dataset.provider === id)!; +const group = (title: string) => screen.getByRole('heading', { name: title, level: 2 }).closest('section') as HTMLElement; + +describe('Admin › Metadata sources', () => { + test('each source is a row saying what it supplies in words, not its codes', () => { + setup(); + const row = source('theaudiodb'); + expect(row.textContent).toMatch(/Album covers, artist photos, artist backdrops/); + expect(row.textContent).not.toMatch(/artist_thumb|album_cover/); + expect(within(group('Tags')).getAllByTestId('provider')).toHaveLength(2); + }); + + test('a saved key says so, and an unsaved one invites a key', () => { + setup({ covers: [{ ...audiodb, api_key_set: true }] }); + expect(within(source('theaudiodb')).getByPlaceholderText(/key saved/i)).toBeInTheDocument(); + }); + + test('Save is off until something changes, then sends only the change', async () => { + setup(); + vi.mocked(updateCoverProvider).mockResolvedValueOnce({ ...audiodb, version_bumped: false }); + const row = source('theaudiodb'); + const save = within(row).getByRole('button', { name: /^save$/i }) as HTMLButtonElement; + expect(save.disabled).toBe(true); + await fireEvent.input(within(row).getByLabelText(/theaudiodb api key/i), { target: { value: 'newkey123' } }); + expect(save.disabled).toBe(false); + await fireEvent.click(save); + expect(updateCoverProvider).toHaveBeenCalledWith('theaudiodb', { api_key: 'newkey123' }); + }); + + test('switching a cover source off reopens covers no source had', async () => { + setup(); + vi.mocked(updateCoverProvider).mockResolvedValueOnce({ ...caa, enabled: false, version_bumped: true }); + const row = source('mbcaa'); + await fireEvent.click(within(row).getByRole('switch')); + await fireEvent.click(within(row).getByRole('button', { name: /^save$/i })); + await waitFor(() => expect(invalidateQueries).toHaveBeenCalledWith(expect.objectContaining({ queryKey: ['coverage'] }))); + }); + + test('a save that fails says so', async () => { + setup(); + vi.mocked(updateCoverProvider).mockRejectedValueOnce({ code: 'server_error' }); + const row = source('theaudiodb'); + await fireEvent.input(within(row).getByLabelText(/api key/i), { target: { value: 'k' } }); + await fireEvent.click(within(row).getByRole('button', { name: /^save$/i })); + await waitFor(() => expect(screen.getByTestId('toast').textContent).toMatch(/couldn't save theaudiodb/i)); + }); + + test('Test says whether the source answered', async () => { + setup(); + vi.mocked(testCoverProvider).mockResolvedValueOnce({ ok: true, duration_ms: 120 }); + await fireEvent.click(within(source('mbcaa')).getByRole('button', { name: /^test$/i })); + await waitFor(() => expect(within(source('mbcaa')).getByTestId('provider-test').textContent).toMatch(/works · answered in 120 ms/i)); + }); + + test('sources that fail to load say so, with Try again', () => { + setup({ cover: mockQuery({ isError: true, error: { code: 'server_error' } }) }); + expect(within(group('Cover art')).getByTestId('providers-error').textContent).toMatch(/couldn't load the cover art sources/i); + }); + + test('a tag source is switched on with its key in one save', async () => { + setup(); + vi.mocked(updateTagProvider).mockResolvedValueOnce({ ...lastfm, enabled: true, api_key_set: true, version_bumped: true }); + const row = source('lastfm'); + await fireEvent.click(within(row).getByRole('switch')); + await fireEvent.input(within(row).getByLabelText(/api key/i), { target: { value: 'lfmkey' } }); + await fireEvent.click(within(row).getByRole('button', { name: /^save$/i })); + expect(updateTagProvider).toHaveBeenCalledWith('lastfm', { enabled: true, api_key: 'lfmkey' }); + await waitFor(() => expect(invalidateQueries).toHaveBeenCalledWith(expect.objectContaining({ queryKey: ['tagProviders'] }))); + }); + + test('Test on a tag source asks that source', async () => { + setup(); + vi.mocked(testTagProvider).mockResolvedValueOnce({ ok: true, duration_ms: 42 }); + await fireEvent.click(within(source('musicbrainz')).getByRole('button', { name: /^test$/i })); + await waitFor(() => expect(testTagProvider).toHaveBeenCalledWith('musicbrainz')); + }); + + test('where to get a key is linked from the source that needs one', () => { + setup(); + expect(within(source('lastfm')).getByRole('link', { name: /get a free key/i })).toBeInTheDocument(); + expect(within(source('theaudiodb')).getByRole('link', { name: /a free key of your own/i })).toBeInTheDocument(); + }); +}); diff --git a/web/src/routes/admin/missing-files/+page.svelte b/web/src/routes/admin/missing-files/+page.svelte deleted file mode 100644 index 683c7851..00000000 --- a/web/src/routes/admin/missing-files/+page.svelte +++ /dev/null @@ -1,200 +0,0 @@ - - -{pageTitle('Admin · Missing files')} - -
    -
    -
    -

    Missing files

    - {#if total > 0} - - {total} - - {/if} -
    -

    - Tracks whose audio file the last scan couldn't find. They keep their play - history and likes, and stop being offered anywhere until the file returns. -

    -
    - - - - - {#if query.isPending} -

    Checking what's missing…

    - {:else if query.isError} -

    Couldn't load the missing-files list.

    - {:else if groups.length === 0} - -
    - -

    Every track's file is where it should be.

    -

    - A track lands here when a library scan can't find its file — moved outside - Minstrel, deleted, or on a drive that didn't mount. It leaves on its own - when the file comes back. -

    -
    - {:else} -
      - {#each groups as group (group.directory)} -
    • - -
      -

      - {group.directory} -

      - - {trackCountLabel(group.tracks.length)} · gone {relativeTime(group.missing_since)} - -
      - - - {#if group.reacquisition} - {@const r = group.reacquisition} -

      - {#if r.gave_up_at} - Gave up - after {attemptLabel(r.attempts)} — last tried - {relativeTime(r.last_attempt_at ?? r.gave_up_at)}. It'll be tried again - if the files come back and go missing later. - {:else if r.last_attempt_at} - Asked Lidarr {attemptLabel(r.attempts)}, last - {relativeTime(r.last_attempt_at)}{#if r.next_attempt_at}, next - {timeUntil(r.next_attempt_at)}{/if}. - {/if} -

      - {/if} - -
        - {#each group.tracks as t (t.track_id)} -
      • - - -
        -
        {t.title}
        -
        - {t.artist_name} · {t.album_title} -
        -
        - - - - {#if t.last_played_at} - last played {relativeTime(t.last_played_at)} - {:else} - never played - {/if} - - {durationLabel(t.duration_sec)} -
        • - {/each} -
      -
      • - {/each} -
    - - {#if hasMore || offset > 0} - - {/if} - {/if} -
    diff --git a/web/src/routes/admin/missing-files/missing-files.test.ts b/web/src/routes/admin/missing-files/missing-files.test.ts deleted file mode 100644 index 673a01df..00000000 --- a/web/src/routes/admin/missing-files/missing-files.test.ts +++ /dev/null @@ -1,183 +0,0 @@ -import { afterEach, describe, expect, test, vi } from 'vitest'; -import { render, screen } from '@testing-library/svelte'; -import { mockQuery } from '../../../test-utils/query'; -import type { AdminMissingResponse } from '#lib/api/types.js'; - -// The page now embeds ReacquisitionSettingsCard, which loads its own settings -// from this same module on mount. Stubbing both keeps these tests about the -// missing-files list — the card has its own suite. -vi.mock('#lib/api/admin.js', () => ({ - createMissingFilesQuery: vi.fn(), - getReacquisitionSettings: vi.fn().mockResolvedValue({ - enabled: true, - grace_hours: 24, - backoff_base_hours: 6, - backoff_max_hours: 168, - max_attempts: 3, - max_per_pass: 20, - auto_approve: true, - unnameable_albums: 0 - }), - updateReacquisitionSettings: vi.fn() -})); - -import AdminMissingFilesPage from './+page.svelte'; -import { createMissingFilesQuery } from '#lib/api/admin.js'; - -const DAY = 24 * 3_600_000; - -function track(title: string, opts: { lastPlayed?: string | null } = {}) { - return { - track_id: `t-${title}`, - title, - artist_id: 'ar-1', - artist_name: 'Linkin Park', - album_id: 'al-1', - album_title: 'Minutes to Midnight', - file_path: `/music/Linkin Park/Minutes to Midnight/${title}.flac`, - duration_sec: 185, - missing_since: new Date(Date.now() - 3 * DAY).toISOString(), - last_played_at: opts.lastPlayed === undefined ? null : opts.lastPlayed - }; -} - -const response: AdminMissingResponse = { - total: 3, - limit: 50, - offset: 0, - groups: [ - { - directory: '/music/Linkin Park/Minutes to Midnight', - missing_since: new Date(Date.now() - 3 * DAY).toISOString(), - reacquisition: null, - tracks: [ - track('Given Up', { lastPlayed: new Date(Date.now() - 2 * DAY).toISOString() }), - track('Bleed It Out') - ] - }, - { - directory: '/music/Boards of Canada/Geogaddi', - missing_since: new Date(Date.now() - 9 * DAY).toISOString(), - reacquisition: null, - tracks: [track('1969')] - } - ] -}; - -afterEach(() => vi.clearAllMocks()); - -// Whitespace-normalised text of a testid'd element. Svelte templates wrap -// mid-sentence, so raw textContent has newlines where the rendered page has -// single spaces — asserting on it directly makes tests fail on reflow. -function text(testId: string): string { - return (screen.getByTestId(testId).textContent ?? '').replace(/\s+/g, ' ').trim(); -} - -function renderWith(data: AdminMissingResponse | undefined, extra = {}) { - vi.mocked(createMissingFilesQuery).mockReturnValue( - mockQuery({ data, ...extra }) as ReturnType - ); - return render(AdminMissingFilesPage); -} - -describe('admin missing files', () => { - test('groups rows under their directory', () => { - renderWith(response); - expect(screen.getByText('/music/Linkin Park/Minutes to Midnight')).toBeTruthy(); - expect(screen.getByText('/music/Boards of Canada/Geogaddi')).toBeTruthy(); - expect(screen.getAllByTestId('missing-track-row')).toHaveLength(3); - }); - - test('the count pill reports the server total, not the page size', () => { - renderWith(response); - expect(screen.getByTestId('missing-count-pill').textContent?.trim()).toBe('3'); - }); - - test('a group states how many tracks and how long gone', () => { - renderWith(response); - expect(screen.getByText(/2 tracks · gone 3d ago/)).toBeTruthy(); - expect(screen.getByText(/1 track · gone 9d ago/)).toBeTruthy(); - }); - - // The deciding fact for whether a missing file is worth chasing. - test('distinguishes a played track from one never played', () => { - renderWith(response); - expect(screen.getByText(/last played 2d ago/)).toBeTruthy(); - expect(screen.getAllByText('never played')).toHaveLength(2); - }); - - // An operator who has never seen this page should not have to guess what - // would put a row here. - test('the empty state explains what missing means', () => { - renderWith({ total: 0, limit: 50, offset: 0, groups: [] }); - expect(screen.getByText(/every track's file is where it should be/i)).toBeTruthy(); - expect(screen.getByText(/can't find its file/i)).toBeTruthy(); - expect(screen.queryByTestId('missing-count-pill')).toBeNull(); - }); - - test('shows a loading line while pending', () => { - renderWith(undefined, { isPending: true }); - expect(screen.getByText(/checking what's missing/i)).toBeTruthy(); - }); - - test('shows an error line when the query fails', () => { - renderWith(undefined, { isError: true }); - expect(screen.getByText(/couldn't load the missing-files list/i)).toBeTruthy(); - }); - - // Nothing attempted yet is the common case for a folder that just went - // missing; a line saying so would be noise on every row. - test('no re-acquisition line before anything has been attempted', () => { - renderWith(response); - expect(screen.queryByTestId('reacquisition-state')).toBeNull(); - }); - - test('an in-flight re-acquisition says how often and when next', () => { - const withState = structuredClone(response); - withState.groups[0].reacquisition = { - attempts: 2, - last_attempt_at: new Date(Date.now() - 2 * DAY).toISOString(), - next_attempt_at: new Date(Date.now() + 4 * 3_600_000).toISOString(), - gave_up_at: null - }; - renderWith(withState); - // Collapsed: the markup wraps mid-sentence, so textContent carries the - // template's newlines and indentation between the words. - const line = text('reacquisition-state'); - expect(line).toMatch(/asked lidarr twice/i); - expect(line).toMatch(/last 2d ago/i); - // Forward-looking, not relativeTime — which would say "just now" for a - // future timestamp and read as nonsense. - expect(line).toMatch(/next in 4h/i); - }); - - test('a given-up album says so and says it can come back', () => { - const withState = structuredClone(response); - withState.groups[0].reacquisition = { - attempts: 3, - last_attempt_at: new Date(Date.now() - 5 * DAY).toISOString(), - next_attempt_at: null, - gave_up_at: new Date(Date.now() - 5 * DAY).toISOString() - }; - renderWith(withState); - const line = text('reacquisition-state'); - expect(line).toMatch(/gave up/i); - expect(line).toMatch(/after 3 times/i); - expect(line).toMatch(/tried again/i); - }); - - // Paging only appears when it can do something: a single page of results - // should not render dead Previous/Next buttons. - test('no pager when everything fits on one page', () => { - renderWith(response); - expect(screen.queryByLabelText('Missing files pages')).toBeNull(); - }); - - test('pager appears when the server reports more than this page holds', () => { - renderWith({ ...response, total: 120 }); - expect(screen.getByLabelText('Missing files pages')).toBeTruthy(); - expect(screen.getByText('1–3 of 120')).toBeTruthy(); - expect(screen.getByRole('button', { name: 'Previous' })).toHaveProperty('disabled', true); - expect(screen.getByRole('button', { name: 'Next' })).toHaveProperty('disabled', false); - }); -}); diff --git a/web/src/routes/admin/playback-errors/+page.svelte b/web/src/routes/admin/playback-errors/+page.svelte deleted file mode 100644 index 66be479a..00000000 --- a/web/src/routes/admin/playback-errors/+page.svelte +++ /dev/null @@ -1,295 +0,0 @@ - - -{pageTitle('Admin · Playback errors')} - -
    -
    -
    -

    Playback errors

    -

    - Tracks that failed to play on a client. Reported automatically by - the Android player when a track loads with zero duration or - decode-fails — the player skips fast and logs here for triage. -

    -
    - {#if !query.isPending && !query.isError && !resolved} - - {rows.length} unresolved - - {/if} -
    - - - (resolved ? 'resolved' : 'unresolved'), (v) => (resolved = v === 'resolved')} - items={[ - { id: 'unresolved', label: 'Unresolved' }, - { id: 'resolved', label: 'Resolved' } - ]} - /> - - {#if query.isError} -

    Couldn't load: {errMessage(query.error)}

    - {:else if query.isPending} -

    Loading…

    - {:else if rows.length === 0} -

    - {resolved ? 'No resolved errors yet.' : 'No unresolved errors.'} -

    - {:else} -
      - {#each rows as r (r.id)} -
    • -
      -
      - - {r.track_title} - - - {r.artist_name} · {r.album_title} - - - {kindLabel(r.kind)} - - {#if resolved && r.resolution} - - {resolutionLabel(r.resolution)} - - {/if} -
      -
      - by {r.username} · {relativeTime(r.occurred_at)} - {#if r.detail} · {r.detail}{/if} -
      -
      - {r.track_file_path} -
      -
      - {#if !resolved} - {@const a = actionsFor(r)} - - {/if} -
      • - {/each} -
    - {/if} -
    - - { deleteFileOpen = null; }} -> - {#if deleteFileOpen} -

    - This removes {deleteFileOpen.track_title} - from the library and deletes the underlying file. The error row - will be marked resolved. -

    -

    {deleteFileOpen.track_file_path}

    -
    - - -
    - {/if} -     - - { resolveOpen = null; }} -> - {#if resolveOpen} -

    - Marking {resolveOpen.track_title} resolved. - Pick how you handled it: -

    - -
    - - -
    - {/if} -     diff --git a/web/src/routes/admin/quarantine/+page.svelte b/web/src/routes/admin/quarantine/+page.svelte deleted file mode 100644 index 0c13ed86..00000000 --- a/web/src/routes/admin/quarantine/+page.svelte +++ /dev/null @@ -1,397 +0,0 @@ - - -{pageTitle('Admin · Quarantine')} - -
    -
    -
    -

    Quarantine

    - {#if totalReports > 0} - - {totalReports} - - {/if} -
    -

    User-reported broken tracks waiting for triage.

    -
    - - {#if query.isPending} -

    Reading the queue…

    - {:else if query.isError} -

    Couldn't load the quarantine queue.

    - {:else if rows.length === 0} -

    Nothing to triage right now.

    - {:else} -
      - {#each rows as r (r.track_id)} - {@const isExpanded = !!expanded[r.track_id]} - {@const lidarrDisabled = !r.lidarr_album_mbid} - {@const primary: RowAction = { - icon: RotateCcw, - label: 'Resolve', - ariaLabel: `Resolve ${r.track_title}`, - onclick: () => onResolve(r) - }} - {@const secondary: RowAction[] = [ - { - icon: Play, - label: 'Play', - ariaLabel: `Play ${r.track_title}`, - onclick: () => onPlay(r) - }, - { - icon: Trash2, - label: 'Delete file', - ariaLabel: `Delete file for ${r.track_title}`, - onclick: () => openDeleteFile(r), - danger: true - } - ]} -
    • - - - -
      -
      - {r.track_title} -
      -
      - {r.artist_name} · {r.album_title} · {r.report_count} reports — latest {relativeTime(r.latest_at)} -
      - - -
      - {#each Object.entries(r.reason_counts) as [reason, count] (reason)} - - {count}× {REASON_LABELS[reason as LidarrQuarantineReason] ?? reason} - - {/each} -
      - - - - - {#if isExpanded} -
        - {#each r.reports as rep, idx (rep.user_id + ':' + rep.created_at + ':' + idx)} -
      • - {rep.username} - · {REASON_LABELS[rep.reason] ?? rep.reason} - {#if rep.notes} - · {rep.notes} - {/if} - · {relativeTime(rep.created_at)} -
        • - {/each} -
      - {/if} -
      - - -
      - - - {#if lidarrDisabled} - - {:else} - - {/if} -
      -
      • - {/each} -
    - {/if} -
    - - - {#if deleteFileRow} -

    - Remove {deleteFileRow.track_title} - from disk and clear {deleteFileRow.report_count} reports? Lidarr may auto-redownload depending on its monitor settings. -

    -
    - - -
    - {/if} -     - - - {#if deleteLidarrRow} -

    - This will tell Lidarr to remove {deleteLidarrRow.album_title} - (artist {deleteLidarrRow.artist_name}) - and add it to the import-list exclusion. Affects all tracks on the album. Type DELETE to confirm. -

    - - {#if deleteLidarrError} -

    - {deleteLidarrError} -

    - {/if} -
    - - -
    - {/if} -     - - diff --git a/web/src/routes/admin/quarantine/quarantine.test.ts b/web/src/routes/admin/quarantine/quarantine.test.ts deleted file mode 100644 index d18d4de5..00000000 --- a/web/src/routes/admin/quarantine/quarantine.test.ts +++ /dev/null @@ -1,176 +0,0 @@ -import { afterEach, describe, expect, test, vi } from 'vitest'; -import { render, screen, fireEvent, waitFor } from '@testing-library/svelte'; -import { mockQuery } from '../../../test-utils/query'; -import type { AdminQuarantineRow } from '#lib/api/types.js'; - -// Wrap useQueryClient so the page can call invalidateQueries() without a real -// QueryClient context. Everything else from svelte-query passes through. -vi.mock('#lib/api/admin.js', () => ({ - createAdminQuarantineQuery: vi.fn(), - resolveQuarantine: vi.fn(), - deleteQuarantineFile: vi.fn(), - deleteQuarantineViaLidarr: vi.fn() -})); - -vi.mock('#lib/player/store.svelte.js', () => ({ - playRadio: vi.fn() -})); - -import AdminQuarantinePage from './+page.svelte'; -import { - createAdminQuarantineQuery, - resolveQuarantine, - deleteQuarantineFile, - deleteQuarantineViaLidarr -} from '#lib/api/admin.js'; -import { playRadio } from '#lib/player/store.svelte.js'; - -const baseRow: AdminQuarantineRow = { - track_id: 't-001', - track_title: 'Roygbiv', - artist_name: 'Boards of Canada', - album_title: 'Geogaddi', - album_id: 'al-001', - lidarr_album_mbid: 'al-mbid-1', - report_count: 3, - latest_at: new Date(Date.now() - 2 * 3_600_000).toISOString(), - reason_counts: { bad_rip: 2, wrong_tags: 1 }, - reports: [ - { - user_id: 'u1', - username: 'alice', - reason: 'bad_rip', - notes: 'crackles at 1:24', - created_at: new Date(Date.now() - 2 * 3_600_000).toISOString() - }, - { - user_id: 'u2', - username: 'bob', - reason: 'bad_rip', - notes: null, - created_at: new Date(Date.now() - 5 * 3_600_000).toISOString() - }, - { - user_id: 'u3', - username: 'carol', - reason: 'wrong_tags', - notes: 'wrong year', - created_at: new Date(Date.now() - 7 * 3_600_000).toISOString() - } - ] -}; - -afterEach(() => vi.clearAllMocks()); - -function setup(rows: AdminQuarantineRow[] = [baseRow]) { - (createAdminQuarantineQuery as ReturnType).mockReturnValue( - mockQuery({ data: rows }) - ); - return render(AdminQuarantinePage); -} - -describe('/admin/quarantine', () => { - test('Empty state shows the spec copy', () => { - setup([]); - expect(screen.getByText('Nothing to triage right now.')).toBeInTheDocument(); - }); - - test('Aggregated row renders with reason distribution + report count', () => { - setup(); - expect(screen.getByText('Roygbiv')).toBeInTheDocument(); - expect( - screen.getByText(/Boards of Canada · Geogaddi · 3 reports — latest/i) - ).toBeInTheDocument(); - const pills = screen.getAllByTestId('reason-pill').map((el) => el.textContent?.trim()); - expect(pills).toContain('2× Bad rip'); - expect(pills).toContain('1× Wrong tags'); - expect(screen.getByTestId('report-count-pill')).toHaveTextContent('3'); - }); - - test('Resolve fires resolveQuarantine and invalidates the query', async () => { - setup(); - (resolveQuarantine as ReturnType).mockResolvedValueOnce({ - action_id: 'a1', - affected_users: 3 - }); - await fireEvent.click(screen.getByRole('button', { name: /resolve roygbiv/i })); - expect(resolveQuarantine).toHaveBeenCalledWith('t-001'); - }); - - test('Delete file → modal-confirm → fires deleteQuarantineFile', async () => { - setup(); - (deleteQuarantineFile as ReturnType).mockResolvedValueOnce({ - action_id: 'a2', - affected_users: 3 - }); - await fireEvent.click( - screen.getByRole('button', { name: /delete file for roygbiv/i }) - ); - const dialog = await screen.findByRole('dialog'); - expect(dialog).toHaveTextContent(/Remove/i); - expect(dialog).toHaveTextContent(/clear 3 reports/i); - await fireEvent.click( - screen.getByRole('button', { name: /confirm delete file/i }) - ); - expect(deleteQuarantineFile).toHaveBeenCalledWith('t-001'); - }); - - test('Delete via Lidarr → typed-confirm "DELETE" → fires deleteQuarantineViaLidarr', async () => { - setup(); - (deleteQuarantineViaLidarr as ReturnType).mockResolvedValueOnce({ - action_id: 'a3', - affected_users: 3, - deleted_track_count: 12 - }); - await fireEvent.click( - screen.getByRole('button', { name: /delete via lidarr for roygbiv/i }) - ); - await screen.findByRole('dialog'); - const confirmBtn = screen.getByRole('button', { - name: /confirm delete via lidarr/i - }) as HTMLButtonElement; - expect(confirmBtn).toBeDisabled(); - const input = screen.getByLabelText(/type delete to confirm/i) as HTMLInputElement; - await fireEvent.input(input, { target: { value: 'DELETE' } }); - expect(confirmBtn).not.toBeDisabled(); - await fireEvent.click(confirmBtn); - expect(deleteQuarantineViaLidarr).toHaveBeenCalledWith('t-001'); - }); - - test('Lidarr-unreachable error shows toast with spec copy', async () => { - setup(); - (resolveQuarantine as ReturnType).mockRejectedValueOnce({ - code: 'lidarr_unreachable', - message: 'unreachable', - status: 503 - }); - await fireEvent.click(screen.getByRole('button', { name: /resolve roygbiv/i })); - await waitFor(() => - expect( - screen.getByText( - 'Lidarr is unreachable right now. Try again, or check Admin → Integrations.' - ) - ).toBeInTheDocument() - ); - }); - - test('Dimmed Delete-via-Lidarr when lidarr_album_mbid is null', () => { - setup([{ ...baseRow, lidarr_album_mbid: null }]); - const disabled = screen.getByTestId('delete-lidarr-disabled'); - expect(disabled).toHaveClass('cursor-not-allowed'); - expect(disabled).toHaveClass('opacity-50'); - expect(disabled).toHaveAttribute( - 'title', - 'Local-only track — no Lidarr album to remove.' - ); - expect( - screen.queryByRole('button', { name: /delete via lidarr for roygbiv/i }) - ).not.toBeInTheDocument(); - }); - - test('Inline play button calls playRadio with the track id', async () => { - setup(); - await fireEvent.click(screen.getByRole('button', { name: /play roygbiv/i })); - expect(playRadio).toHaveBeenCalledWith('t-001'); - }); -}); diff --git a/web/src/routes/admin/requests/+page.svelte b/web/src/routes/admin/requests/+page.svelte index 55ea2f7a..25f42e5b 100644 --- a/web/src/routes/admin/requests/+page.svelte +++ b/web/src/routes/admin/requests/+page.svelte @@ -1,134 +1,95 @@ - - {pageTitle('Tuning')} - +{pageTitle('Admin · Tuning')} -
    -
    -

    Recommendation tuning

    -

    - The defaults-discovery lab. Changes apply live — radio on the next request, daily mixes on - the next rebuild — and every change is recorded so the metrics page can tie outcome shifts - to knob turns. Found-good values get baked into shipped defaults; a dot marks knobs that - currently deviate from them. -

    -
    +

    Tuning

    + +{#snippet knobs(scope: TuningScope, title: string, hint: string, fields: { key: string; label: string; hint: string }[], step: (key: string) => string, shipped: (key: string) => number)} + + {#each fields as f (f.key)} + + {#if deviates(scope, f.key)} + Changed from the default, {shipped(f.key)} + {/if} + + + {/each} +
    + + +
    +     +{/snippet} + +
    +

    + Scoring knobs that apply live, each change recorded against the trends below. A dot marks one + changed from the shipped default. +

    {#if loadFailed} -

    Couldn't load tuning settings.

    +

    + Couldn't load the tuning settings. + +

    {:else if !snapshot}

    Loading…

    {:else} -
    + {@const snap = snapshot} +
    {#each profileScopes as p (p.scope)} -
    -
    -

    {p.label}

    -

    {p.blurb}

    -
    -
    - {#each weightFields as f (f.key)} -
    - - -
    - {/each} -
    -
    - - -
    -
    + {@render knobs(p.scope, p.label, p.blurb, weightFields, () => '0.1', (k) => snap.shipped.profiles[p.scope][k as keyof WeightProfile])} {/each} -
    - -
    -
    -

    Taste profile build

    -

    - How plays become the learned taste profile: the influence half-life and the - completion→engagement curve (must stay ordered: hard-skip < neutral < full). -

    -
    -
    - {#each tasteFields as f (f.key)} -
    - - -
    - {/each} -
    -
    - - -
    -
    - - -
    -
    -

    Discover requests

    -

    - How the Discover suggestion deck ranks out-of-library artists. Tag coverage for - artists you don't own is partial by nature, so an untagged candidate keeps its - similarity score rather than being pushed down. -

    -
    -
    - {#each discoverFields as f (f.key)} -
    - - -
    - {/each} -
    -
    - - -
    -
    + {@render knobs( + 'taste', + 'Taste profile build', + 'How plays become the taste profile. Keep dislike < neither < like.', + tasteFields, + (k) => (k === 'half_life_days' ? '1' : '0.05'), + (k) => snap.shipped.taste[k as keyof TasteTuning] + )} + {@render knobs( + 'discover', + 'Discover requests', + "How Discover ranks artists you don't have.", + discoverFields, + (k) => (k === 'snooze_days' ? '1' : '0.1'), + (k) => snap.shipped.discover[k as keyof DiscoverTuning] + )} +
    {/if} - -
    -
    -

    Weekly trends

    -

    - Skip rate per surface over the last {trends?.weeks ?? 12} weeks (lower is better; all - users aggregated, rates only). Dashed ticks mark tuning changes. Taste hit is the share - of plays whose artist fits the current taste profile. - The skip and completion columns show the most recent week - alone, not the whole window — the figure after the skip rate is that week's - play count, so a rate drawn from a handful of listens reads as what it is. -

    -
    - {#if trendsFailed} -

    Couldn't load trends.

    - {:else if !trends} -

    Loading…

    - {:else if trends.series.length === 0} -

    - No plays recorded yet — trends appear once listening accumulates. -

    - {:else} - - - - - - - - - - - - - {#each trends.series as s (s.key)} - - - - - - - + + +
    + {#if trendsFailed} +

    + Couldn't load trends. + +

    + {:else if !trends} +

    Loading…

    + {:else if trends.series.length === 0} +

    No plays recorded yet; trends appear once listening accumulates.

    + {:else} + {@const t = trends} +
    SurfaceSkip rate by weekPlays (window)Skip (last wk)Completion (last wk)Taste hit (window)
    {s.label} - - - {#each trends.markers as m, i (i)} - - {markerSummary(m)} - - {/each} - {#if s.points.length > 1} - - {:else if s.points.length === 1} - - {/if} - - {s.plays} - {pct(latest(s).skip)} - /{latest(s).plays} - {pct(latest(s).completion)}{pct(windowTasteHitRate(s))}
    + + + + + + + + - {/each} - -
    SurfaceSkip rate, last {t.weeks} weeksPlays (window)Skip (last wk)Completion (last wk)Taste hit (window)
    - {#if trends.markers.length > 0} -
    -

    Tuning changes in this window

    -
      - {#each trends.markers as m, i (i)} -
    • {markerSummary(m)}
      • + + + {#each t.series as s (s.key)} + + {s.label} + + + 100% + 0% + + + + {#each t.markers as m, i (i)} + + {markerSummary(m)} + + {/each} + {#if s.points.length > 1} + + {:else if s.points.length === 1} + + {/if} + + + + {s.plays} + + {pct(latest(s).skip)} + /{latest(s).plays} + + {pct(latest(s).completion)} + {pct(windowTasteHitRate(s))} + {/each} -
    -
    + + + {#if t.markers.length > 0} +
    +

    Tuning changes in this window

    +
      + {#each t.markers as m, i (i)} +
    • {markerSummary(m)}
      • + {/each} +
    +
    + {/if} {/if} - {/if} -
    +
    + diff --git a/web/src/routes/admin/tuning/tuning.test.ts b/web/src/routes/admin/tuning/tuning.test.ts index b2d8671c..f4ca8908 100644 --- a/web/src/routes/admin/tuning/tuning.test.ts +++ b/web/src/routes/admin/tuning/tuning.test.ts @@ -99,10 +99,9 @@ describe('Admin tuning page', () => { }); // Songs-like is a separate weight profile precisely so it can be tuned - // apart from For-You (#3881). If its card stopped rendering its OWN values - // — or quietly fell back to daily_mix's — the split would exist in the - // backend and be unreachable in the UI, which is the same as not shipping - // it (rule 27). + // apart from For-You. If its card stopped rendering its OWN values — or + // quietly fell back to daily_mix's — the split would exist in the backend + // and be unreachable in the UI, which is the same as not shipping it. test('the songs-like card carries its own weights, not daily_mix\'s', async () => { (getTuning as ReturnType).mockResolvedValue(snapshot()); render(TuningPage); @@ -141,9 +140,9 @@ describe('Admin tuning page', () => { await waitFor(() => expect(pushToast).toHaveBeenCalledWith('Nothing changed.')); }); - // Targets the button by its accessible name, not its position. This used to - // click the LAST reset button and assume that meant taste — which silently - // became the wrong scope the moment a card was added below it. + // Targets the button by its accessible name, not its position: clicking the + // LAST reset button and assuming it meant taste would silently pick the + // wrong scope the moment a card was added below it. test('reset calls resetTuning for the scope', async () => { (getTuning as ReturnType).mockResolvedValue(snapshot()); (resetTuning as ReturnType).mockResolvedValue(snapshot()); @@ -226,7 +225,7 @@ describe('Admin tuning page', () => { expect(screen.getByTestId('sparkline-radio')).toBeInTheDocument(); expect(screen.getByTestId('sparkline-discover')).toBeInTheDocument(); // The skip column is the LAST WEEK's rate, now carrying that week's play - // count so a rate off a handful of plays reads as what it is (#2495). + // count so a rate off a handful of plays reads as what it is. // Radio's latest week: 40% skip over 15 plays; Discover's: 60% over 5. expect(screen.getByText('/15')).toBeInTheDocument(); expect(screen.getByText('/5')).toBeInTheDocument(); @@ -238,8 +237,7 @@ describe('Admin tuning page', () => { // play-count span. Assert the count rather than pretending it's unique. expect(screen.getAllByText('40%')).toHaveLength(2); // The window/last-week distinction has to be visible in the headers, or the - // Plays total reads as the denominator of the skip rate. That misreading is - // what #2495 was filed over. + // Plays total reads as the denominator of the skip rate. // // Queried as column headers rather than by text: the caption below also // mentions "Plays", and matching on the word finds the prose too. @@ -264,7 +262,7 @@ describe('Admin tuning page', () => { expect(screen.getByText(/trends appear once listening accumulates/i)).toBeInTheDocument() ); }); - // --- Discover scope (#2377) --- + // --- Discover scope --- test('renders the Discover card with its two knobs', async () => { (getTuning as ReturnType).mockResolvedValue(snapshot()); @@ -288,7 +286,7 @@ describe('Admin tuning page', () => { ); }); - // 0 is the operator's off switch for the whole tag term, so the form must be + // 0 is an admin's off switch for the whole tag term, so the form must be // able to send it — a falsy-value bug here would silently make the feature // impossible to disable. test('a Discover weight of 0 is sent, not dropped as falsy', async () => { diff --git a/web/src/routes/admin/users/+page.svelte b/web/src/routes/admin/users/+page.svelte index f1271e6b..59f0b213 100644 --- a/web/src/routes/admin/users/+page.svelte +++ b/web/src/routes/admin/users/+page.svelte @@ -1,4 +1,5 @@ + +
    +
    +

    Collection

    +

    + Every artist and album in your collection: what the library holds, and what Minstrel is + getting for it, from user requests, new releases of the artists it follows, and albums whose + files went missing. It searches your indexers on a schedule, checks each download against + MusicBrainz, and files what's right into the library. Searching and quality are set under + Acquisition. +

    + {#if verdict} +

    + {verdict} + {#if counts?.attention} + See Activity + {:else if counts?.unmatched} + See Unmatched + {/if} +

    + {/if} + {#if health.length > 0} +

    + To look at: + {#each health as d, i (d.key)}{queueLabel(d, countOf(summary, d.key))}{i < health.length - 1 ? ', ' : '.'} + {/each} +

    + {/if} +
    + +
    +
    +
    + +
    +
    +
    + {@render children?.()} +
    +
    +
    diff --git a/web/src/routes/collection/+layout.ts b/web/src/routes/collection/+layout.ts new file mode 100644 index 00000000..067439a6 --- /dev/null +++ b/web/src/routes/collection/+layout.ts @@ -0,0 +1,13 @@ +import { redirect } from '@sveltejs/kit'; +import { user } from '#lib/auth/store.svelte.js'; +import type { LayoutLoad } from './$types'; + +// Hard route gate: the Wanted view is an admin's. Runs before the +// layout renders; auth is bootstrapped by the root +layout.ts, so user.value is +// either a settled User object or null. +export const load: LayoutLoad = () => { + if (!user.value || !user.value.is_admin) { + throw redirect(302, '/'); + } + return {}; +}; diff --git a/web/src/routes/collection/+page.svelte b/web/src/routes/collection/+page.svelte new file mode 100644 index 00000000..5a07e2e2 --- /dev/null +++ b/web/src/routes/collection/+page.svelte @@ -0,0 +1,8 @@ + + +{pageTitle('Collection')} + + diff --git a/web/src/routes/collection/activity/+page.svelte b/web/src/routes/collection/activity/+page.svelte new file mode 100644 index 00000000..d36f1008 --- /dev/null +++ b/web/src/routes/collection/activity/+page.svelte @@ -0,0 +1,8 @@ + + +{pageTitle('Collection · Activity')} + + diff --git a/web/src/routes/collection/artist/[mbid]/+page.svelte b/web/src/routes/collection/artist/[mbid]/+page.svelte new file mode 100644 index 00000000..53b4ada0 --- /dev/null +++ b/web/src/routes/collection/artist/[mbid]/+page.svelte @@ -0,0 +1,15 @@ + + +{pageTitle('Collection · Artist')} + +{#key mbid} + +{/key} diff --git a/web/src/routes/collection/blocked/+page.svelte b/web/src/routes/collection/blocked/+page.svelte new file mode 100644 index 00000000..07c7b02d --- /dev/null +++ b/web/src/routes/collection/blocked/+page.svelte @@ -0,0 +1,8 @@ + + +{pageTitle('Collection · Blocked')} + + diff --git a/web/src/routes/collection/collection.test.ts b/web/src/routes/collection/collection.test.ts new file mode 100644 index 00000000..19132d70 --- /dev/null +++ b/web/src/routes/collection/collection.test.ts @@ -0,0 +1,123 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen } from '@testing-library/svelte'; +import { createRawSnippet } from 'svelte'; +import { mockInfiniteQuery } from '../../test-utils/query'; +import { pageUrlModule } from '#test-utils/mocks/appState.js'; +import { mockQuery } from '../../test-utils/query'; +import type { AdminSummary, WantArtistsPage, WantCounts } from '#lib/api/types.js'; + +const state = vi.hoisted(() => ({ pageUrl: new URL('http://localhost/collection') })); +vi.mock('$app/state', () => pageUrlModule(state)); +vi.mock('#lib/api/wanted.js', () => ({ createWantArtistsQuery: vi.fn() })); +vi.mock('#lib/api/admin.js', () => ({ createAdminSummaryQuery: vi.fn() })); + +import WantedLayout from './+layout.svelte'; +import { createWantArtistsQuery } from '#lib/api/wanted.js'; +import { createAdminSummaryQuery } from '#lib/api/admin.js'; + +const counts: WantCounts = { artists: 7, wanted: 5, missing: 1, held: 9, upgrades: 6, following: 2, activity: 3, attention: 0, blocked: 4, unmatched: 3 }; +const children = createRawSnippet(() => ({ render: () => `

    body

    ` })); + +afterEach(() => vi.clearAllMocks()); + +const none = { count: 0, since: null }; +const clear: AdminSummary = { requests: none, reports: none, duplicates: none, suspect: none, missing: none }; + +function setup(c: WantCounts = counts, path = '/collection', summary: AdminSummary = clear) { + state.pageUrl = new URL(`http://localhost${path}`); + vi.mocked(createAdminSummaryQuery).mockReturnValue( + mockQuery({ data: summary }) as unknown as ReturnType + ); + const page: WantArtistsPage = { + items: [], + total: 0, + limit: 50, + offset: 0, + counts: c, + library: { listed_at: null, unlisted: 0 } + }; + vi.mocked(createWantArtistsQuery).mockReturnValue( + mockInfiniteQuery({ pages: [page] }) as unknown as ReturnType + ); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + return render(WantedLayout, { children } as any); +} + +describe('/collection layout', () => { + test('the sections down the left carry their counts, the current one marked', () => { + setup(counts, '/collection/activity'); + const nav = screen.getByRole('navigation', { name: /collection sections/i }); + const links = Array.from(nav.querySelectorAll('a')).map((a) => [a.getAttribute('href'), a.textContent?.replace(/\s+/g, ' ').trim()]); + expect(links).toEqual([ + ['/collection', 'Catalogue 7'], + ['/collection/wanted', 'Wanted 5'], + ['/collection/upgrades', 'Upgrades 6'], + ['/collection/activity', 'Activity 3'], + ['/collection/following', 'Following 2'], + ['/collection/blocked', 'Blocked 4'], + ['/collection/unmatched', 'Unmatched 3'], + ['/collection/reports', 'Reports'], + ['/collection/duplicates', 'Duplicates'], + ['/collection/suspect-sources', 'Suspect sources'], + ['/collection/missing', 'Missing files'] + ]); + expect(screen.getByRole('link', { name: /activity/i })).toHaveAttribute('aria-current', 'page'); + expect(screen.getByRole('link', { name: /^catalogue/i })).not.toHaveAttribute('aria-current'); + expect(screen.getByTestId('tab-body')).toBeTruthy(); + }); + + test('when nothing needs a person the verdict says so', () => { + setup({ ...counts, unmatched: 0 }); + expect(screen.getByTestId('collection-verdict').textContent).toMatch(/5 albums wanted, 3 downloading\. Nothing needs you\./); + expect(screen.queryByTestId('tab-attention')).toBeNull(); + }); + + test('downloads waiting on a person lead the page and mark Activity', () => { + setup({ ...counts, attention: 2 }); + const verdict = screen.getByTestId('collection-verdict'); + expect(verdict.textContent).toMatch(/2 downloads need you/); + expect(screen.getByRole('link', { name: /see activity/i }).getAttribute('href')).toBe('/collection/activity'); + expect(screen.getByTestId('tab-attention').textContent).toBe('2'); + }); + + test('library albums waiting to be matched need a person, and link to Unmatched', () => { + setup(); + expect(screen.getByTestId('collection-verdict').textContent).toMatch( + /5 albums wanted, 3 downloading\. 3 albums in your library need matching\./ + ); + expect(screen.getByRole('link', { name: /see unmatched/i }).getAttribute('href')).toBe('/collection/unmatched'); + }); + + // The library's health queues moved here from Admin: their counts + // come from the admin summary, accented, and a line names the ones waiting. + test('the library health queues carry what waits on you, and a line links to each', () => { + setup(counts, '/collection', { + ...clear, + reports: { count: 2, since: null }, + duplicates: { count: 5, since: null }, + missing: { count: 9, since: null } + }); + const nav = screen.getByRole('navigation', { name: /collection sections/i }); + const dup = Array.from(nav.querySelectorAll('a')).find((a) => a.getAttribute('href') === '/collection/duplicates')!; + expect(dup.textContent?.replace(/\s+/g, ' ').trim()).toBe('Duplicates 5'); + const health = screen.getByTestId('collection-health'); + expect(health.textContent?.replace(/\s+/g, ' ').trim()).toBe( + 'To look at: 2 broken tracks reported, 5 sets of duplicates to review, 9 missing files.' + ); + expect(Array.from(health.querySelectorAll('a')).map((a) => a.getAttribute('href'))).toEqual([ + '/collection/reports', + '/collection/duplicates', + '/collection/missing' + ]); + }); + + test('with nothing waiting there is no health line', () => { + setup(); + expect(screen.queryByTestId('collection-health')).toBeNull(); + }); + + test("an artist's page sits under Catalogue", () => { + setup(counts, '/collection/artist/blur'); + expect(screen.getByRole('link', { name: /^catalogue/i })).toHaveAttribute('aria-current', 'page'); + }); +}); diff --git a/web/src/routes/admin/duplicates/+page.svelte b/web/src/routes/collection/duplicates/+page.svelte similarity index 51% rename from web/src/routes/admin/duplicates/+page.svelte rename to web/src/routes/collection/duplicates/+page.svelte index 7a1dca40..6a3484cb 100644 --- a/web/src/routes/admin/duplicates/+page.svelte +++ b/web/src/routes/collection/duplicates/+page.svelte @@ -1,39 +1,52 @@ -{pageTitle('Admin · Duplicates')} +{pageTitle('Collection · Duplicates')}
    @@ -223,8 +281,10 @@

    - Tracks that hold the same recording more than once. Copies Lidarr doesn't use are merged on - their own, and a copy Lidarr uses is never removed. What's left here needs you. + Tracks that hold the same recording more than once. Most settle on their own: a song held twice + on one album is merged, copies on different releases count as one song, two versions of a song + are left alone, and a file filling another song's track slot is removed so the album is fetched + again. What's left here needs you.

    @@ -259,41 +319,51 @@ {#if resolved.isPending}

    Loading…

    {:else if resolved.isError && !resolved.isFetchNextPageError} -

    Couldn't load what was resolved.

    +

    + Couldn't load what was resolved. + +

    {:else if activity.length === 0}

    Nothing resolved automatically yet.

    - Merges and Lidarr release changes made on their own are listed here. + What Minstrel settled on its own is listed here.

    {:else}
      {#each activity as item (item.id)} + {@const removed = item.details.removed ?? []}
    • - {#if item.action === 'duplicate_merge'} - {@const removed = item.details.removed ?? []} + {#if item.action === 'duplicate_dismiss'} +

      + {item.details.artist}: {(item.details.titles ?? []).join(' and ')} are different versions, + not duplicates. Both kept. +

      + {:else if item.action === 'duplicate_wrong_file' && !item.details.kept_title} +

      + {item.details.artist}: one recording was filed as {removed.map((r) => r.title).join(' and ')}, + and it is none of them. Removed; the album will be fetched again. +

      + {:else if item.action === 'duplicate_wrong_file'} +

      + {item.details.artist}: the recording of {item.details.kept_title} was also filed as + {removed.map((r) => r.title).join(', ')}. Removed from there; the album will be fetched again. +

      + {:else}

      Kept {basename(item.details.survivor_path ?? '')} {#if item.details.survivor_reason} ({item.details.survivor_reason}) {/if} - and removed {fileCountLabel(removed.length)} Lidarr doesn't use. + and removed {plural(removed.length, 'file', 'files')}.

      + {/if} + {#if item.action !== 'duplicate_dismiss'} {#each removed as r (r.track_id)}

      {r.file_path}

      {/each} - {:else} -

      - Lidarr now monitors {item.details.to_release ?? 'another release'} of - {item.details.album_title ?? 'an album'}{item.details.artist_name - ? ` by ${item.details.artist_name}` - : ''}, which lists each song once. -

      - {#if item.details.from_release} -

      Was {item.details.from_release}.

      - {/if} {/if}

      {relativeTime(item.created_at)}

      • @@ -309,7 +379,10 @@ {:else if query.isPending}

      Loading duplicates…

      {:else if query.isError && !query.isFetchNextPageError} -

      Couldn't load the duplicates report.

      +

      + Couldn't load the duplicates report. + +

      {:else if groups.length === 0} @@ -319,8 +392,8 @@

      Nothing to compare yet.

      {#if prints.enabled === false} - Fingerprinting is off, so no track has a fingerprint to compare. Turn it on in the - settings below. + Fingerprinting is off, so no track has a fingerprint to compare. Turn it on under + Library. {:else} The library is still being fingerprinted — {prints.pending.toLocaleString()} tracks to go. Duplicates appear here as the sweep finds them. @@ -354,7 +427,6 @@

        {#each groups as group (group.id)} {@const titles = differingTitles(group)} - {@const blocked = trackedLosers(group)}
      • @@ -365,27 +437,49 @@ - + {#if wrongSlot(group)} + {@const picked = pickedOf(group)} + + {:else} + + {/if}
        @@ -399,6 +493,10 @@

        {/if} +

        + {evidence(group)} +

        + {#if titles.length > 0}

        {/if} + {#if confirmingRefetch === group.id} + {@const picked = pickedOf(group)} +
        +

        + {#if picked} + The copy titled “{picked.title}” stays, and {plural(refetchRemoves(group).length, 'copy', 'copies')} + under other titles will be deleted from disk. + {:else} + All {plural(group.members.length, 'copy', 'copies')} will be deleted from disk. + {/if} + Their tracks stay, marked missing, with their likes and plays. The release the files came + from is set aside, and the album is fetched again from another one. +

        + +
        + {/if} + {#if confirmingMerge === group.id}
        - {#if blocked.length > 0} -

        - Lidarr uses {blocked.length === 1 ? 'a copy' : 'copies'} this would remove, and would - download {blocked.length === 1 ? 'it' : 'them'} again. Keep the copy Lidarr uses, or - leave this group as it is. -

        - {:else} -

        - The copy marked Keep stays. The other {fileCountLabel(group.members.length - 1)} will be - deleted from disk, and their likes, plays and playlist entries move to the copy kept. -

        - - {/if} +

        + The copy marked Keep stays. The other {plural(group.members.length - 1, 'file', 'files')} will be + deleted from disk, and their likes, plays and playlist entries move to the copy kept. +

        +
        -
        {m.file_format.toUpperCase()} · {sizeLabel(m.file_size)} · {durationLabel(m.duration_sec)}
        +
        {m.file_format.toUpperCase()} · {formatBytes(m.file_size)} · {formatDuration(m.duration_sec)}
        {historyLabel(m)}
        • {/each}
      + {#if wrongSlot(group) && pickedOf(group)} + + {/if} {/each}
    @@ -516,7 +648,10 @@ /> {/if} - - + +

    + Fingerprinting is {prints?.enabled === false ? 'off' : 'on'}. It's set under + Library. +

    diff --git a/web/src/routes/admin/duplicates/duplicates.test.ts b/web/src/routes/collection/duplicates/duplicates.test.ts similarity index 58% rename from web/src/routes/admin/duplicates/duplicates.test.ts rename to web/src/routes/collection/duplicates/duplicates.test.ts index 4f452295..3fb657b7 100644 --- a/web/src/routes/admin/duplicates/duplicates.test.ts +++ b/web/src/routes/collection/duplicates/duplicates.test.ts @@ -16,25 +16,22 @@ vi.mock('#lib/api/admin.js', () => ({ survivor_track_id: 'www-01', removed_paths: ['/music/Moe Shop/WWW (2020)/www-02.mp3'] }), - // The page embeds FingerprintSettingsCard, which loads its own settings from - // this module; the card has its own suite. - getFingerprintSettings: vi.fn().mockResolvedValue({ - enabled: true, - chromaprint_length_sec: 120, - acoustic_max_bit_error_rate: 0.15, - backfill_concurrency: 2, - sweep_interval_hours: 1, - auto_resolve: true - }), - updateFingerprintSettings: vi.fn() + refetchDuplicateGroup: vi.fn().mockResolvedValue({ + kept_track_id: null, + removed_paths: ['/music/Moe Shop/WWW (2020)/www-01.mp3', '/music/Moe Shop/WWW (2020)/www-02.mp3'], + blocked: ['Moe Shop - WWW [WEB]'] + }) })); +vi.mock('#lib/player/store.svelte.js', () => ({ playQueue: vi.fn() })); import AdminDuplicatesPage from './+page.svelte'; +import { playQueue } from '#lib/player/store.svelte.js'; import { createDuplicatesQuery, createResolvedDuplicatesQuery, dismissDuplicateGroup, mergeDuplicateGroup, + refetchDuplicateGroup, runDuplicateSweep } from '#lib/api/admin.js'; @@ -56,6 +53,7 @@ function member(id: string, extra: Partial { expect(history).toEqual(['2 likes · 14 plays', 'no likes or plays']); }); + // Size and length come from the shared formatters. + test('each copy shows its format, size and length', () => { + const group = response().groups[0]; + group.members = [member('www-01'), member('www-02', { file_size: 812_000, duration_sec: 3665 })]; + renderWith(response({ groups: [group] })); + expect(screen.getAllByTestId('member-file').map(text)).toEqual([ + 'MP3 · 6.9 MB · 3:35', + 'MP3 · 812 KB · 1:01:05' + ]); + }); + // The badge must sit on the proposed survivor, not merely appear somewhere. test('the keep badge is on the survivor row', () => { renderWith(response()); @@ -152,14 +161,46 @@ describe('admin duplicates', () => { expect(rows[1].querySelector('[data-testid="survivor-badge"]')).toBeNull(); }); + test('picking another copy moves the keep badge to it', async () => { + renderWith(response()); + const rows = screen.getAllByTestId('duplicate-member'); + await fireEvent.click(rows[1].querySelector('input[type="radio"]') as HTMLInputElement); + const after = screen.getAllByTestId('duplicate-member'); + expect(after[0].querySelector('[data-testid="survivor-badge"]')).toBeNull(); + expect(text(after[1].querySelector('[data-testid="survivor-badge"]') as HTMLElement)).toBe('Keep · your pick'); + }); + test('an acoustic group reads as a match percentage', () => { const r = response(); r.groups[0] = { ...r.groups[0], tier: 'acoustic', worst_bit_error_rate: 0.04 }; renderWith(r); - expect(text(screen.getByTestId('duplicate-tier'))).toBe('Same recording · 96% match'); + expect(text(screen.getByTestId('duplicate-tier'))).toBe('Same music · 96% match'); }); - // #3885: identical audio titled "WWW" and "WWW (instrumental)". One file + // Read alone, size and length make the match look like a guess. + test('each group says what the match rests on', () => { + const r = response(); + r.groups.push({ ...r.groups[0], id: 'g-2', tier: 'acoustic', worst_bit_error_rate: 0.04 }); + renderWith(r); + const lines = screen.getAllByTestId('match-evidence').map(text); + expect(lines[0]).toContain('the same audio data, byte for byte'); + expect(lines[1]).toBe( + 'Matched by listening, not by size or length: 96% of their audio fingerprints agree, so these files play the same music.' + ); + }); + + test('each copy can be played', async () => { + renderWith(response()); + const plays = screen.getAllByTestId('member-play'); + expect(plays).toHaveLength(2); + await fireEvent.click(plays[1]); + expect(playQueue).toHaveBeenLastCalledWith( + [expect.objectContaining({ id: 'www-02', artist_id: 'ar-1', stream_url: '/api/tracks/www-02/stream' })], + 0 + ); + }); + + // Identical audio titled "WWW" and "WWW (instrumental)". One file // claimed to be the vocal and wasn't, and nothing said the vocal was missing. describe('identical audio under different titles', () => { function www() { @@ -212,6 +253,74 @@ describe('admin duplicates', () => { renderWith(r); expect(screen.queryByTestId('title-mismatch')).toBeNull(); }); + + // Unless the resolver found different songs on one album: Savant's Mortals + // held one recording as 16 tracks, each cut to its own song's length. + test('an acoustic group the resolver found to be different songs says so', () => { + const r = www(); + r.groups[0] = { ...r.groups[0], tier: 'acoustic', worst_bit_error_rate: 0, class: 'mismatch' }; + renderWith(r); + const notice = text(screen.getByTestId('title-mismatch')); + expect(notice).toContain('The music is the same, but the copies are titled “WWW” and “WWW (instrumental)”'); + expect(notice).toContain("If you can't tell, get the album again"); + }); + }); + + // One recording in several of an album's slots. A merge would keep + // one title whether or not it is the audio's and fetch nothing, so it is not + // offered: an admin says which song it is, or that they can't tell. + describe('one recording in several slots', () => { + function slots() { + const r = response(); + r.groups[0] = { + ...r.groups[0], + tier: 'acoustic', + worst_bit_error_rate: 0.04, + class: 'mismatch', + members: [member('www-01'), member('www-02', { title: 'WWW (instrumental)' })] + }; + return r; + } + + test('offers Get it again instead of Merge, with no copy proposed', () => { + renderWith(slots()); + expect(screen.queryByRole('button', { name: 'Merge…' })).toBeNull(); + expect(screen.getByRole('button', { name: 'Get it again…' })).toBeTruthy(); + expect(screen.getAllByRole('radio').some((r) => (r as HTMLInputElement).checked)).toBe(false); + expect(screen.queryByTestId('survivor-badge')).toBeNull(); + }); + + test("can't tell: every copy goes, after a second click", async () => { + renderWith(slots()); + await fireEvent.click(screen.getByRole('button', { name: 'Get it again…' })); + expect(refetchDuplicateGroup).not.toHaveBeenCalled(); + const confirm = text(screen.getByTestId('refetch-confirm')); + expect(confirm).toContain('All 2 copies will be deleted from disk'); + expect(confirm).toContain('the album is fetched again from another one'); + + await fireEvent.click(screen.getByRole('button', { name: 'Remove 2 files and get it again' })); + expect(refetchDuplicateGroup).toHaveBeenCalledWith('g-1', { keep_track_id: '' }); + }); + + test('naming the song keeps that copy and removes the others', async () => { + renderWith(slots()); + await fireEvent.click(screen.getByRole('radio', { name: /This is WWW \(instrumental\)/ })); + expect(screen.getByTestId('song-badge')).toBeTruthy(); + await fireEvent.click(screen.getByRole('button', { name: 'Keep “WWW (instrumental)”…' })); + expect(text(screen.getByTestId('refetch-confirm'))).toContain( + 'The copy titled “WWW (instrumental)” stays, and 1 copy under other titles will be deleted from disk' + ); + await fireEvent.click(screen.getByRole('button', { name: 'Remove 1 file and get it again' })); + expect(refetchDuplicateGroup).toHaveBeenCalledWith('g-1', { keep_track_id: 'www-02' }); + }); + + test("I can't tell takes the pick back", async () => { + renderWith(slots()); + await fireEvent.click(screen.getAllByRole('radio')[0]); + await fireEvent.click(screen.getByRole('button', { name: "I can't tell which song it is" })); + expect(screen.getByRole('button', { name: 'Get it again…' })).toBeTruthy(); + expect(screen.queryByTestId('song-badge')).toBeNull(); + }); }); // Three states all show zero groups. Each must say which it is. @@ -227,7 +336,7 @@ describe('admin duplicates', () => { }); // Switched off, the backlog never shrinks. "Still being fingerprinted" would - // promise work nothing is doing (#3913). + // promise work nothing is doing. test('with fingerprinting off the page says so instead of promising progress', () => { renderWith( response({ @@ -242,9 +351,11 @@ describe('admin duplicates', () => { expect(text(screen.getByTestId('sweep-status'))).toContain('Fingerprinting is off'); }); - test('the fingerprinting settings are on this page', async () => { + test('the fingerprinting settings are a link to Library, saying whether it is on', () => { renderWith(response()); - expect(await screen.findByRole('spinbutton', { name: /seconds of audio/i })).toBeTruthy(); + const line = screen.getByTestId('fingerprint-settings-link'); + expect(text(line)).toContain('Fingerprinting is on'); + expect(line.querySelector('a')?.getAttribute('href')).toBe('/admin/library#fingerprints'); }); test('empty before any sweep says the sweep has not run', () => { @@ -283,10 +394,7 @@ describe('admin duplicates', () => { expect(text(screen.getByTestId('merge-confirm'))).toContain('The other 1 file will be deleted from disk'); await fireEvent.click(screen.getByRole('button', { name: 'Remove 1 file and merge' })); - expect(mergeDuplicateGroup).toHaveBeenCalledWith('g-1', { - survivor_track_id: 'www-01', - unmonitor: false - }); + expect(mergeDuplicateGroup).toHaveBeenCalledWith('g-1', { survivor_track_id: 'www-01' }); }); test('choosing another copy to keep sends that copy', async () => { @@ -297,10 +405,7 @@ describe('admin duplicates', () => { await fireEvent.click(screen.getByRole('button', { name: 'Merge…' })); await fireEvent.click(screen.getByRole('button', { name: 'Remove 1 file and merge' })); - expect(mergeDuplicateGroup).toHaveBeenCalledWith('g-1', { - survivor_track_id: 'www-02', - unmonitor: false - }); + expect(mergeDuplicateGroup).toHaveBeenCalledWith('g-1', { survivor_track_id: 'www-02' }); }); test('Cancel disarms the merge', async () => { @@ -311,7 +416,7 @@ describe('admin duplicates', () => { expect(screen.getByRole('button', { name: 'Merge…' })).toBeTruthy(); }); - // M498: the resolver's three outcomes are three tabs, each with its size. + // The resolver's three outcomes are three tabs, each with its size. describe('tabs', () => { test('each tab shows its count, and Across releases asks for that view', async () => { renderWith(response()); @@ -332,7 +437,7 @@ describe('admin duplicates', () => { expect(dismissDuplicateGroup).toHaveBeenCalledWith('g-1'); }); - test('Resolved automatically lists merges and release changes', async () => { + test('Resolved automatically lists the merges the resolver made', async () => { renderWith(response(), [ { id: 'a-1', @@ -340,65 +445,75 @@ describe('admin duplicates', () => { created_at: ago(HOUR), details: { survivor_path: '/music/Gorillaz/Humanz (2017)/03 - Saturnz Barz.mp3', - survivor_reason: 'the copy Lidarr tracks', + survivor_reason: 'the copy that is not a video rip', removed: [{ track_id: 't-2', file_path: '/music/Gorillaz/Humanz (2017)/03 - Saturnz Barz (Official Video).mp3' }] } - }, - { - id: 'a-2', - action: 'lidarr_release_change', - created_at: ago(2 * HOUR), - details: { - album_title: 'Humanz', - artist_name: 'Gorillaz', - from_release: '14x12" Vinyl, Digital Media (68 tracks)', - to_release: 'Digital Media (40 tracks)' - } } ]); await fireEvent.click(screen.getByRole('tab', { name: /resolved automatically/i })); const items = screen.getAllByTestId('resolved-item').map(text); - expect(items[0]).toContain('Kept 03 - Saturnz Barz.mp3 (the copy Lidarr tracks) and removed 1 file'); - expect(items[1]).toContain('Lidarr now monitors Digital Media (40 tracks) of Humanz by Gorillaz'); + expect(items[0]).toContain('Kept 03 - Saturnz Barz.mp3 (the copy that is not a video rip) and removed 1 file.'); expect(screen.queryByTestId('duplicate-group')).toBeNull(); }); + + test('Resolved automatically says what was dismissed and which slots were emptied', async () => { + renderWith(response(), [ + { + id: 'a-2', + action: 'duplicate_dismiss', + created_at: ago(HOUR), + details: { artist: 'Moe Shop', titles: ['Lovesick', 'Lovesick (Instrumental)'] } + }, + { + id: 'a-3', + action: 'duplicate_wrong_file', + created_at: ago(HOUR), + details: { + artist: 'Savant', + kept_title: 'Void', + removed: [{ track_id: 't-9', file_path: '/data/Music/Savant/Void/17 - Reach.m4a', title: 'Reach' }] + } + }, + { + id: 'a-4', + action: 'duplicate_wrong_file', + created_at: ago(HOUR), + details: { + artist: 'Shnabubula', + removed: [ + { track_id: 't-5', file_path: '/data/Music/Shnabubula/05 - Fading Light.mp3', title: 'Fading Light' }, + { track_id: 't-10', file_path: '/data/Music/Shnabubula/10 - Looking for Answers.mp3', title: 'Looking for Answers' } + ] + } + } + ]); + await fireEvent.click(screen.getByRole('tab', { name: /resolved automatically/i })); + const items = screen.getAllByTestId('resolved-item').map(text); + expect(items[0]).toContain('Moe Shop: Lovesick and Lovesick (Instrumental) are different versions, not duplicates. Both kept.'); + expect(items[1]).toContain( + 'Savant: the recording of Void was also filed as Reach. Removed from there; the album will be fetched again.' + ); + expect(items[1]).toContain('/data/Music/Savant/Void/17 - Reach.m4a'); + expect(items[2]).toContain( + 'Shnabubula: one recording was filed as Fading Light and Looking for Answers, and it is none of them. Removed; the album will be fetched again.' + ); + }); }); - describe("Lidarr's view of each copy", () => { - function tracked() { + describe("the resolver's note", () => { + test('shows above the group it explains', () => { const r = response(); - r.groups[0] = { - ...r.groups[0], - resolve_note: 'Lidarr tracks every copy as its own track.', - members: [member('www-01', { lidarr_state: 'tracked' }), member('www-02', { lidarr_state: 'unmapped' })] - }; - return r; - } - - test('each copy says whether Lidarr uses it, and the resolver note shows', () => { - renderWith(tracked()); - expect(screen.getAllByTestId('lidarr-state').map(text)).toEqual([ - 'Lidarr uses this', - "Lidarr doesn't use this" - ]); - expect(text(screen.getByTestId('resolve-note'))).toBe('Lidarr tracks every copy as its own track.'); + r.groups[0] = { ...r.groups[0], resolve_note: 'Identical audio filed under different titles.' }; + renderWith(r); + expect(text(screen.getByTestId('resolve-note'))).toBe('Identical audio filed under different titles.'); }); - // Keeping the unmapped copy would remove the one Lidarr uses. The server - // refuses that; the confirm says so first and offers no button to press. - test('a merge that would remove the copy Lidarr uses is blocked before it is sent', async () => { - renderWith(tracked()); + test('a merge confirms and sends the copy chosen to keep', async () => { + renderWith(response()); await fireEvent.click(screen.getAllByRole('radio')[1]); await fireEvent.click(screen.getByRole('button', { name: 'Merge…' })); - expect(text(screen.getByTestId('merge-blocked'))).toContain('Lidarr uses a copy this would remove'); - const confirm = screen.getByRole('button', { name: 'Remove 1 file and merge' }) as HTMLButtonElement; - expect(confirm.disabled).toBe(true); - - // Choosing the copy Lidarr uses clears it. - await fireEvent.click(screen.getAllByRole('radio')[0]); - expect(screen.queryByTestId('merge-blocked')).toBeNull(); await fireEvent.click(screen.getByRole('button', { name: 'Remove 1 file and merge' })); - expect(mergeDuplicateGroup).toHaveBeenCalledWith('g-1', { survivor_track_id: 'www-01', unmonitor: false }); + expect(mergeDuplicateGroup).toHaveBeenCalledWith('g-1', { survivor_track_id: 'www-02' }); }); }); }); diff --git a/web/src/routes/collection/following/+page.svelte b/web/src/routes/collection/following/+page.svelte new file mode 100644 index 00000000..b891a091 --- /dev/null +++ b/web/src/routes/collection/following/+page.svelte @@ -0,0 +1,12 @@ + + +{pageTitle('Collection · Following')} + +
    + + +
    diff --git a/web/src/routes/collection/missing/+page.svelte b/web/src/routes/collection/missing/+page.svelte new file mode 100644 index 00000000..92c2af49 --- /dev/null +++ b/web/src/routes/collection/missing/+page.svelte @@ -0,0 +1,299 @@ + + +{pageTitle('Collection · Missing files')} + +
    +
    +
    +

    Missing files

    + {#if total > 0} + + {total} + + {/if} +
    +

    + Tracks whose audio file the last scan couldn't find. They keep their play + history and likes, and stop being offered anywhere until the file returns. +

    +
    + +

    + {#if policy} + {policy.enabled + ? `Albums missing for ${policy.grace_hours} hours go back on the wanted list by themselves.` + : 'Missing albums are not wanted back automatically.'} + {/if} + + {policy ? 'Change' : 'Set what happens to missing files'} + +

    + + {#if query.isPending} +

    Checking what's missing…

    + {:else if query.isError} +

    + Couldn't load the missing files. + +

    + {:else if groups.length === 0} + +
    + +

    Every track's file is where it should be.

    +

    + A track lands here when a library scan can't find its file — moved outside + Minstrel, deleted, or on a drive that didn't mount. It leaves on its own + when the file comes back. +

    +
    + {:else} +
      + {#each groups as group (group.directory)} +
    • + +
      +

      + {group.directory} +

      + + {plural(group.tracks.length, 'track', 'tracks')} · gone {relativeTime(group.missing_since)} + +
      + + + {#if group.reacquisition} + {@const r = group.reacquisition} + {@const w = r.want} +
      +
      +
      +

      + {#if r.held_at} + Held again {relativeTime(r.held_at)}. The rows here clear on the next scan. + {:else if r.wanted_at} + Wanted back since {relativeTime(r.wanted_at)}. + {/if} +

      + {#if w && !r.held_at} +

      + {wantStatus(w)} + {#if needsYou(w)} + Decide on Activity + {/if} +

      + {/if} +
      + {#if w && !r.held_at} +
      + + + +
      + {/if} +
      + {#if w && open === w.id} + + {/if} +
      + {/if} + +
        + {#each group.tracks as t (t.track_id)} +
      • + + +
        +
        {t.title}
        +
        + {t.artist_name} · {t.album_title} +
        +
        + + + + {#if t.last_played_at} + last played {relativeTime(t.last_played_at)} + {:else} + never played + {/if} + + {formatDuration(t.duration_sec)} +
        • + {/each} +
      +
      • + {/each} +
    + + {#if hasMore || offset > 0} + + {/if} + {/if} +
    diff --git a/web/src/routes/collection/missing/missing.test.ts b/web/src/routes/collection/missing/missing.test.ts new file mode 100644 index 00000000..7494a2b1 --- /dev/null +++ b/web/src/routes/collection/missing/missing.test.ts @@ -0,0 +1,259 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/svelte'; +import { mockQuery } from '../../../test-utils/query'; +import type { AdminMissingResponse, Want } from '#lib/api/types.js'; + +// The page reads the re-acquisition policy to say what happens to these; +// the settings themselves are on Admin › Acquisition. +vi.mock('#lib/api/admin.js', () => ({ + createMissingFilesQuery: vi.fn(), + getReacquisitionSettings: vi.fn().mockResolvedValue({ + enabled: true, + grace_hours: 24, + max_per_pass: 20, + unnameable_albums: 0 + }) +})); +// A folder's want is acted on here as on the wanted list. +vi.mock('#lib/api/wanted.js', () => ({ + removeWant: vi.fn().mockResolvedValue(undefined), + searchWant: vi.fn().mockResolvedValue({ + outcome: 'nothing', considered: 0, best_release: '', best_verdict: '', narrowed: [] + }), + createCandidatesQuery: vi.fn(), + grabCandidate: vi.fn(), + blockCandidate: vi.fn(), + unblockRelease: vi.fn() +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import MissingPage from './+page.svelte'; +import { createMissingFilesQuery } from '#lib/api/admin.js'; +import { removeWant, searchWant } from '#lib/api/wanted.js'; +import { pushToast } from '#lib/stores/toast.svelte.js'; + +const DAY = 24 * 3_600_000; + +function track(title: string, opts: { lastPlayed?: string | null } = {}) { + return { + track_id: `t-${title}`, + title, + artist_id: 'ar-1', + artist_name: 'Linkin Park', + album_id: 'al-1', + album_title: 'Minutes to Midnight', + file_path: `/music/Linkin Park/Minutes to Midnight/${title}.flac`, + duration_sec: 185, + missing_since: new Date(Date.now() - 3 * DAY).toISOString(), + last_played_at: opts.lastPlayed === undefined ? null : opts.lastPlayed + }; +} + +const response: AdminMissingResponse = { + total: 3, + limit: 50, + offset: 0, + groups: [ + { + directory: '/music/Linkin Park/Minutes to Midnight', + missing_since: new Date(Date.now() - 3 * DAY).toISOString(), + reacquisition: null, + tracks: [ + track('Given Up', { lastPlayed: new Date(Date.now() - 2 * DAY).toISOString() }), + track('Bleed It Out') + ] + }, + { + directory: '/music/Boards of Canada/Geogaddi', + missing_since: new Date(Date.now() - 9 * DAY).toISOString(), + reacquisition: null, + tracks: [track('1969')] + } + ] +}; + +afterEach(() => vi.clearAllMocks()); + +// Whitespace-normalised text of a testid'd element. Svelte templates wrap +// mid-sentence, so raw textContent has newlines where the rendered page has +// single spaces — asserting on it directly makes tests fail on reflow. +function text(testId: string): string { + return (screen.getByTestId(testId).textContent ?? '').replace(/\s+/g, ' ').trim(); +} + +function renderWith(data: AdminMissingResponse | undefined, extra = {}) { + vi.mocked(createMissingFilesQuery).mockReturnValue( + mockQuery({ data, ...extra }) as ReturnType + ); + return render(MissingPage); +} + +function want(extra: Partial = {}): Want { + return { + id: 'w-1', + release_group_mbid: 'rg-1', + title: 'Minutes to Midnight', + artist_mbid: 'lp', + artist_name: 'Linkin Park', + primary_type: 'Album', + secondary_types: [], + year: '2007', + reason: 'missing', + wanted_at: new Date(Date.now() - 2 * DAY).toISOString(), + held_at: null, + image_url: '', + search: { + searched_at: new Date(Date.now() - 3_600_000).toISOString(), + outcome: 'nothing', + considered: 0, + best_release: '', + best_verdict: '', + error: '' + }, + ...extra + }; +} + +function wantedBack(w: Want | null = want()): AdminMissingResponse { + const withState = structuredClone(response); + withState.groups[0].reacquisition = { wanted_at: new Date(Date.now() - 2 * DAY).toISOString(), held_at: null, want: w }; + return withState; +} + +describe('/collection/missing', () => { + test('says what happens to missing files, and links to where it is set', async () => { + renderWith(response); + const line = screen.getByTestId('reacquisition-policy'); + await screen.findByText(/missing for 24 hours go back on the wanted list/i); + expect(line.querySelector('a')?.getAttribute('href')).toBe('/admin/acquisition#reacquisition'); + }); + + test('groups rows under their directory', () => { + renderWith(response); + expect(screen.getByText('/music/Linkin Park/Minutes to Midnight')).toBeTruthy(); + expect(screen.getByText('/music/Boards of Canada/Geogaddi')).toBeTruthy(); + expect(screen.getAllByTestId('missing-track-row')).toHaveLength(3); + }); + + test('the count pill reports the server total, not the page size', () => { + renderWith(response); + expect(screen.getByTestId('missing-count-pill').textContent?.trim()).toBe('3'); + }); + + test('a group states how many tracks and how long gone', () => { + renderWith(response); + expect(screen.getByText(/2 tracks · gone 3d ago/)).toBeTruthy(); + expect(screen.getByText(/1 track · gone 9d ago/)).toBeTruthy(); + }); + + // The deciding fact for whether a missing file is worth chasing. + // The page's lengths come from the shared formatter: an hour-long track + // reads 1:01:05, not 61:05. + test('each track shows its length, hours and all', () => { + const long = structuredClone(response); + long.groups[1].tracks[0].duration_sec = 3665; + renderWith(long); + expect(screen.getAllByTestId('missing-track-length').map((el) => el.textContent?.trim())).toEqual([ + '3:05', + '3:05', + '1:01:05' + ]); + }); + + test('distinguishes a played track from one never played', () => { + renderWith(response); + expect(screen.getByText(/last played 2d ago/)).toBeTruthy(); + expect(screen.getAllByText('never played')).toHaveLength(2); + }); + + // An operator who has never seen this page should not have to guess what + // would put a row here. + test('the empty state explains what missing means', () => { + renderWith({ total: 0, limit: 50, offset: 0, groups: [] }); + expect(screen.getByText(/every track's file is where it should be/i)).toBeTruthy(); + expect(screen.getByText(/can't find its file/i)).toBeTruthy(); + expect(screen.queryByTestId('missing-count-pill')).toBeNull(); + }); + + test('shows a loading line while pending', () => { + renderWith(undefined, { isPending: true }); + expect(screen.getByText(/checking what's missing/i)).toBeTruthy(); + }); + + test('shows an error line when the query fails', () => { + renderWith(undefined, { isError: true }); + expect(screen.getByText(/couldn't load the missing files/i)).toBeTruthy(); + }); + + // Not wanted yet is the common case for a folder that just went missing; + // a line saying so would be noise on every row. + test('no re-acquisition line before the album is wanted', () => { + renderWith(response); + expect(screen.queryByTestId('reacquisition-state')).toBeNull(); + }); + + test('an album wanted back says since when, and what its last search found', () => { + renderWith(wantedBack()); + expect(text('reacquisition-state')).toMatch(/wanted back since 2d ago/i); + expect(text('missing-want-status')).toBe('Searched 1h ago: nothing found'); + }); + + test('its download, when one is running, is the line', () => { + renderWith( + wantedBack( + want({ + grab: { + id: 'g', name: 'LP - MtM [FLAC]', grabbed_at: new Date().toISOString(), origin: 'schedule', quality: 'FLAC', + live: true, end_reason: '', progress: 0.42, import_state: 'downloading', import_reason: '', end_detail: '', + imported_at: null, imported_files: 0 + } + }) + ) + ); + expect(text('missing-want-status')).toBe('Downloading 42% · LP - MtM [FLAC]'); + expect(screen.getByRole('button', { name: /search for minutes to midnight now/i })).toHaveProperty('disabled', true); + }); + + test('Search runs one now and says what it found', async () => { + renderWith(wantedBack()); + await fireEvent.click(screen.getByRole('button', { name: /search for minutes to midnight now/i })); + await waitFor(() => expect(searchWant).toHaveBeenCalledWith('w-1')); + await waitFor(() => expect(pushToast).toHaveBeenCalledWith('Nothing found for Minutes to Midnight.', 'info')); + }); + + test('Remove takes it off the wanted list', async () => { + renderWith(wantedBack()); + await fireEvent.click(screen.getByRole('button', { name: /stop wanting minutes to midnight/i })); + await waitFor(() => expect(removeWant).toHaveBeenCalledWith('w-1')); + expect(pushToast).toHaveBeenCalledWith("Minutes to Midnight is off the wanted list, and won't be fetched back."); + }); + + test('an album held again says so', () => { + const withState = structuredClone(response); + withState.groups[0].reacquisition = { + wanted_at: new Date(Date.now() - 5 * DAY).toISOString(), + held_at: new Date(Date.now() - DAY).toISOString(), + want: want({ held_at: new Date(Date.now() - DAY).toISOString() }) + }; + renderWith(withState); + expect(text('reacquisition-state')).toMatch(/held again 1d ago/i); + // Nothing left to search for. + expect(screen.queryByTestId('missing-want-status')).toBeNull(); + expect(screen.queryByRole('button', { name: /search for/i })).toBeNull(); + }); + + // Paging only appears when it can do something: a single page of results + // should not render dead Previous/Next buttons. + test('no pager when everything fits on one page', () => { + renderWith(response); + expect(screen.queryByLabelText('Missing files pages')).toBeNull(); + }); + + test('pager appears when the server reports more than this page holds', () => { + renderWith({ ...response, total: 120 }); + expect(screen.getByLabelText('Missing files pages')).toBeTruthy(); + expect(screen.getByText('1–3 of 120')).toBeTruthy(); + expect(screen.getByRole('button', { name: 'Previous' })).toHaveProperty('disabled', true); + expect(screen.getByRole('button', { name: 'Next' })).toHaveProperty('disabled', false); + }); +}); diff --git a/web/src/routes/collection/reports/+page.svelte b/web/src/routes/collection/reports/+page.svelte new file mode 100644 index 00000000..31c20752 --- /dev/null +++ b/web/src/routes/collection/reports/+page.svelte @@ -0,0 +1,302 @@ + + +{pageTitle('Collection · Reports')} + +
    +
    +

    Reports

    +

    Tracks a listener flagged, or that failed to play.

    +
    + + + + {#if view === 'open'} + {#if loading} +

    Reading the reports…

    + {:else if failed} +

    + Couldn't load the reports. + +

    + {:else if rows.length === 0} +

    No reports. Nothing is waiting on you here.

    + {:else} +

    + {plural(rows.length, 'track needs', 'tracks need')} a look. +

    +
      + {#each rows as r (r.track_id)} + {@const a = actionsFor(r)} + {@const open = !!expanded[r.track_id]} +
    • + +
      +
      {r.title}
      +
      + {r.artist} · {r.album} · latest {relativeTime(r.latest)} +
      +
      + {#each Object.entries(r.reason_counts) as [reason, n] (reason)} + {n}× {REASON_LABELS[reason as QuarantineReason] ?? reason} + {/each} + {#each errorCounts(r) as [kind, n] (kind)} + {n}× {ERROR_LABELS[kind] ?? kind} + {/each} +
      + + {#if open} +
        + {#each r.reports as rep, i (rep.user_id + i)} +
      • + {rep.username} flagged it · + {REASON_LABELS[rep.reason] ?? rep.reason}{#if rep.notes}{' · '}{rep.notes}{/if} + · {relativeTime(rep.created_at)} +
        • + {/each} + {#each r.errors as e (e.id)} +
      • + Failed to play for {e.username} · + {ERROR_LABELS[e.kind] ?? e.kind}{#if e.detail}{` · ${e.detail}`}{/if} · {relativeTime(e.occurred_at)} +
        • + {/each} + {#if r.file_path} +
      • {r.file_path}
        • + {/if} +
      + {/if} +
      + +
      • + {/each} +
    + {/if} + {:else if actions?.isPending || resolved?.isPending} +

    Reading what was done…

    + {:else if actions?.isError || resolved?.isError} +

    + Couldn't load what was done. + +

    + {:else if handled.length === 0} +

    Nothing has been handled yet.

    + {:else} +
      + {#each handled as h (h.key)} +
    • + {h.title} + {h.artist} + {h.what} · {relativeTime(h.at)} +
      • + {/each} +
    + {/if} +
    + + (deleteFileId = null)}> + {#if deleteFileRow} +

    + Remove {deleteFileRow.title} from the library and delete its + file. Its reports and playback errors go with it. +

    +
    + + +
    + {/if} +     + + diff --git a/web/src/routes/collection/reports/reports.test.ts b/web/src/routes/collection/reports/reports.test.ts new file mode 100644 index 00000000..6cc9c4cb --- /dev/null +++ b/web/src/routes/collection/reports/reports.test.ts @@ -0,0 +1,125 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { fireEvent, render, screen, waitFor, within } from '@testing-library/svelte'; +import { mockQuery } from '../../../test-utils/query'; +import type { AdminPlaybackError, AdminQuarantineRow } from '#lib/api/types.js'; + +vi.mock('#lib/api/admin.js', () => ({ + createAdminQuarantineQuery: vi.fn(), + createAdminPlaybackErrorsQuery: vi.fn(), + createQuarantineActionsQuery: vi.fn(), + resolveQuarantine: vi.fn().mockResolvedValue({}), + resolvePlaybackError: vi.fn().mockResolvedValue({}), + deleteQuarantineFile: vi.fn().mockResolvedValue({}) +})); +vi.mock('#lib/player/store.svelte.js', () => ({ playRadio: vi.fn() })); + +import Page from './+page.svelte'; +import { + createAdminPlaybackErrorsQuery, + createAdminQuarantineQuery, + createQuarantineActionsQuery, + deleteQuarantineFile, + resolvePlaybackError, + resolveQuarantine +} from '#lib/api/admin.js'; + +const hoursAgo = (h: number) => new Date(Date.now() - h * 3_600_000).toISOString(); + +const flagged: AdminQuarantineRow = { + track_id: 't1', + track_title: 'Roygbiv', + artist_name: 'Boards of Canada', + album_title: 'Geogaddi', + album_id: 'al1', + report_count: 1, + latest_at: hoursAgo(5), + reason_counts: { bad_rip: 1 }, + reports: [{ user_id: 'u1', username: 'alice', reason: 'bad_rip', notes: 'crackles', created_at: hoursAgo(5) }] +}; + +const failure = (id: string, track: string, title: string, h: number): AdminPlaybackError => ({ + id, + track_id: track, + user_id: 'u2', + username: 'bob', + client_id: 'android', + kind: 'load_failed', + occurred_at: hoursAgo(h), + track_title: title, + track_file_path: `/music/${title}.flac`, + artist_name: 'Annadale', + album_title: 'Dial Tone', + album_id: 'al2' +}); + +afterEach(() => vi.clearAllMocks()); + +function setup(q: AdminQuarantineRow[], e: AdminPlaybackError[], opts: { error?: boolean } = {}) { + vi.mocked(createAdminQuarantineQuery).mockReturnValue( + (opts.error ? mockQuery({ isError: true, error: new Error('x') }) : mockQuery({ data: q })) as never + ); + vi.mocked(createAdminPlaybackErrorsQuery).mockReturnValue(mockQuery({ data: e }) as never); + vi.mocked(createQuarantineActionsQuery).mockReturnValue(mockQuery({ data: [] }) as never); + return render(Page); +} + +describe('/collection/reports', () => { + test('a flagged track and a track that failed to play are one list, one row per track, newest first', () => { + setup([flagged], [failure('e1', 't1', 'Roygbiv', 1), failure('e2', 't2', 'Intro', 9)]); + const rows = screen.getAllByTestId('report-row'); + expect(rows).toHaveLength(2); + expect(screen.getByTestId('reports-verdict').textContent).toMatch(/2 tracks need a look/); + const first = within(rows[0]).getByTestId('report-kinds').textContent?.replace(/\s+/g, ' '); + expect(first).toMatch(/1× Bad rip/); + expect(first).toMatch(/1× Wouldn't load/); + expect(rows[1].textContent).toMatch(/Intro/); + }); + + test('Resolve clears the listeners\' reports and marks the playback errors fixed', async () => { + setup([flagged], [failure('e1', 't1', 'Roygbiv', 1)]); + await fireEvent.click(screen.getByRole('button', { name: /resolve roygbiv/i })); + await waitFor(() => expect(resolveQuarantine).toHaveBeenCalledWith('t1')); + await waitFor(() => expect(resolvePlaybackError).toHaveBeenCalledWith('e1', 'fixed')); + }); + + test('a track only failing to play is resolved without touching listener reports', async () => { + setup([], [failure('e2', 't2', 'Intro', 2)]); + await fireEvent.click(screen.getByRole('button', { name: /resolve intro/i })); + await waitFor(() => expect(resolvePlaybackError).toHaveBeenCalledWith('e2', 'fixed')); + expect(resolveQuarantine).not.toHaveBeenCalled(); + }); + + test('Delete file is one path, whatever the report came from', async () => { + setup([], [failure('e2', 't2', 'Intro', 2)]); + await fireEvent.click(screen.getByRole('button', { name: /delete file for intro/i })); + await fireEvent.click(within(screen.getByRole('dialog')).getByRole('button', { name: /confirm delete file/i })); + await waitFor(() => expect(deleteQuarantineFile).toHaveBeenCalledWith('t2')); + expect(resolvePlaybackError).not.toHaveBeenCalled(); + }); + + test("on a phone the row's other actions are in its menu", async () => { + setup([flagged], [failure('e2', 't2', 'Intro', 2)]); + const roygbiv = screen.getAllByTestId('report-row').find((r) => r.textContent?.includes('Roygbiv'))!; + await fireEvent.click(within(roygbiv).getByRole('button', { name: /more actions/i })); + expect(within(screen.getByRole('menu')).getByText('Delete file')).toBeInTheDocument(); + }); + + test('details show who flagged it, who it failed for, and the file', async () => { + setup([flagged], [failure('e1', 't1', 'Roygbiv', 1)]); + await fireEvent.click(screen.getByRole('button', { name: /^details$/i })); + const details = screen.getByTestId('report-details').textContent?.replace(/\s+/g, ' '); + expect(details).toMatch(/alice flagged it · Bad rip · crackles/); + expect(details).toMatch(/Failed to play for bob · Wouldn't load/); + expect(details).toMatch(/\/music\/Roygbiv\.flac/); + }); + + test('a load failure says so with Try again, and an empty queue says nothing waits', () => { + setup([], [], { error: true }); + expect(screen.getByTestId('reports-error').textContent).toMatch(/couldn't load the reports/i); + }); + + test('with nothing reported it says so', () => { + setup([], []); + expect(screen.getByTestId('reports-empty').textContent).toMatch(/no reports/i); + }); +}); diff --git a/web/src/routes/admin/suspect-sources/+page.svelte b/web/src/routes/collection/suspect-sources/+page.svelte similarity index 83% rename from web/src/routes/admin/suspect-sources/+page.svelte rename to web/src/routes/collection/suspect-sources/+page.svelte index 101362fb..b01b0e6d 100644 --- a/web/src/routes/admin/suspect-sources/+page.svelte +++ b/web/src/routes/collection/suspect-sources/+page.svelte @@ -1,4 +1,5 @@ -{pageTitle('Admin · Suspect sources')} +{pageTitle('Collection · Suspect sources')}
    @@ -100,15 +98,20 @@ such as "(Official Video)", "[Audio]" or "Visualizer". These usually came from a video site, and can be the wrong audio: a live take, a music-video edit, or something else entirely. Each one is kept out of radio and the mixes, though it still - plays when you choose it, and it is removed where Lidarr keeps a clean copy of the same - song. If one is fine, say so and it goes back in. + plays when you choose it, and it is merged away where the library has a clean copy of the + same song. Where it is the only copy, its album is searched for a clean copy once a week, + under Upgrades. If one + is fine, say so and it goes back in.

    {#if query.isPending}

    Reading file names…

    {:else if query.isError && !query.isFetchNextPageError} -

    Couldn't load the suspect-sources list.

    +

    + Couldn't load the suspect sources. + +

    {:else if groups.length === 0}
    @@ -127,7 +130,7 @@ {group.directory} - {trackCountLabel(group.tracks.length)} + {plural(group.tracks.length, 'track', 'tracks')}
    @@ -178,6 +181,13 @@ Held back{#if t.verdict_at && !changed[t.track_id]} {relativeTime(t.verdict_at)}{/if} + {#if t.replacing} + + {t.searched_at + ? `Looked for a clean copy ${relativeTime(t.searched_at)}` + : 'A clean copy is wanted'} + + {/if} -
    - {:else} -
    - - -
    - {/if} - - - - - {#if $status.data.last_scrobbled_at} -

    - Last scrobbled: {new Date($status.data.last_scrobbled_at).toLocaleString()} -

    - {/if} - -

    - Get a token at listenbrainz.org/profile. - Tokens are stored unencrypted in this server's database — treat as sensitive. -

    - - {/if} -
    - - -
    -

    Recommendation metrics

    -

    - How plays launched from each recommendation surface land, over the last - {$metrics.data?.window_days ?? 30} days — compared against the baseline of - music you picked yourself. Deltas are percentage points vs that baseline. -

    - {#if $metrics.isPending} -

    Loading…

    - {:else if $metrics.isError} -

    Couldn't load metrics.

    - {:else if $metrics.data && ($metrics.data.groups.length > 0 || $metrics.data.baseline)} - {@const baseline = $metrics.data.baseline} - {#if baseline} -

    - Baseline — manual plays: - - {baseline.plays} plays · {pct(baseline.skip_rate)} skip · - {pct(baseline.avg_completion)} completion - -

    - {:else} -

    - No manual plays in this window yet, so deltas are hidden — raw rates only. -

    - {/if} - {#each $metrics.data.groups as group (group.intent)} -
    -

    {group.label}

    -

    {intentHints[group.intent]}

    - - - - - - - - - - - {#each group.surfaces as m (m.key)} - - - - - - - - {#if expandedBreakdowns.has(m.key)} - {#each m.breakdown ?? [] as b (b.key)} - - - - - - - {/each} - {/if} - {/each} - -
    SurfacePlaysSkip rateAvg completion
    - {#if (m.breakdown ?? []).length > 0} - - {:else} - {m.label} - {/if}{#if m.low_confidence}· low data{/if} - {m.plays} - {pct(m.skip_rate)} - {#if m.skip_delta} - {deltaText(m.skip_delta)} - {/if} - - {pct(m.avg_completion)} - {#if m.completion_delta} - {deltaText(m.completion_delta)} - {/if} -
    - ↳ {b.label}{#if b.low_confidence}· low data{/if} - {b.plays} - {pct(b.skip_rate)} - {#if b.skip_delta} - {deltaText(b.skip_delta)} - {/if} - - {pct(b.avg_completion)} - {#if b.completion_delta} - {deltaText(b.completion_delta)} - {/if} -
    -
    - {/each} -

    - Deltas compare each surface with your manual plays. A delta marked - ≈ is smaller than its own margin of error at this - sample size — it can't be told apart from no difference, however big it looks. - Hover any delta for its range. -

    - {:else} -

    - No plays recorded yet. Play something from For You, Discover, or a mix. -

    - {/if} -
    - - -
    -

    Profile

    -

    - Your display name and recovery email. -

    -
    - - - - - -
    - -
    - - -
    -

    Password

    -
    -
    - - - - - - - - -
    - -
    -
    - - -
    -

    API Token

    -

    - Used by Subsonic clients that sign in with an API key (OpenSubsonic apiKey). - Minstrel keeps only a fingerprint of the token, so it can't show you the current one. - Regenerate to get a new token; clients using the old one will need the new one. -

    - {#if apiToken} - - {apiToken} - -

    Copy this now. It won't be shown again.

    - {/if} -
    - {#if apiToken} - - {/if} - -
    -
    - - -
    -

    Subsonic password

    -

    - For Subsonic apps that can't use an API token and ask for a username and password instead. - Sign those apps in with your username and this password, not your login password. - Minstrel generates it, and has to store it readable for these apps to work, so it is never - your login password. Use the API token where the app supports it. -

    - {#if subsonicEnabled !== null} -

    - {subsonicEnabled ? 'A Subsonic password is set.' : "No Subsonic password is set; these apps can't sign in."} -

    - {/if} - {#if subsonicPassword} - - {subsonicPassword} - -

    Copy this now. It won't be shown again.

    - {/if} -
    - {#if subsonicPassword} - - {/if} - - {#if subsonicEnabled} - - {/if} -
    -
    - - - - -
    -

    Library

    -
      -
    • - Hidden tracks - — tracks you've flagged as bad rips, wrong tags, or otherwise unplayable. -
      • -
    -
    - - - -
    - -
    - diff --git a/web/src/routes/settings/+page.ts b/web/src/routes/settings/+page.ts new file mode 100644 index 00000000..2127d7df --- /dev/null +++ b/web/src/routes/settings/+page.ts @@ -0,0 +1,7 @@ +import { redirect } from '@sveltejs/kit'; + +// /settings has no page of its own: it opens on Account, the first section. +// A universal load, as the app is adapter-static with ssr off. +export const load = () => { + throw redirect(308, '/settings/account'); +}; diff --git a/web/src/routes/settings/Appearance.test.ts b/web/src/routes/settings/Appearance.test.ts deleted file mode 100644 index 38288712..00000000 --- a/web/src/routes/settings/Appearance.test.ts +++ /dev/null @@ -1,91 +0,0 @@ -import { describe, expect, test, vi, beforeEach } from 'vitest'; -import { render, screen, fireEvent } from '@testing-library/svelte'; - -vi.mock('#lib/api/listenbrainz.js', () => { - // Mocks must look like a Svelte readable store. The page reads e.g. - // $status.isPending, which is shorthand for "subscribe to status, - // then read .isPending on the emitted value." Returning an object - // that ALSO carries isPending/etc. as bare properties (without a - // working subscribe) makes $status resolve to undefined. - const stub = >(value: T) => ({ - subscribe: (run: (v: T) => void) => { - run(value); - return () => {}; - } - }); - return { - createLBStatusQuery: () => - stub({ - isPending: false, - isError: false, - data: { token_set: false, enabled: false } - }), - createTokenMutation: () => stub({ mutate: vi.fn(), isPending: false }), - createEnabledMutation: () => stub({ mutate: vi.fn(), isPending: false }) - }; -}); - -// The Notifications section has its own tests (NotificationSettings.test.ts); -// here it only needs a settled query so the page renders. -vi.mock('#lib/api/notifications.js', async (importOriginal) => { - const mod = await importOriginal(); - const { mockQuery } = await import('../../test-utils/query'); - return { - ...mod, - createNotificationSettingsQuery: () => - mockQuery({ data: { kinds: [], email_available: true } }), - putNotificationSettings: vi.fn() - }; -}); - -// The Volume leveling setting loads on mount; keep it off the network. -vi.mock('#lib/api/normalization.js', async (importOriginal) => { - const mod = await importOriginal(); - return { - ...mod, - getNormalization: vi.fn().mockResolvedValue(mod.DEFAULT_NORMALIZATION), - putNormalization: vi.fn() - }; -}); - -vi.mock('#lib/api/metrics.js', () => ({ - createRecommendationMetricsQuery: () => ({ - subscribe: (run: (v: unknown) => void) => { - run({ isPending: false, isError: false, data: { window_days: 30, baseline: null, groups: [] } }); - return () => {}; - } - }) -})); - -beforeEach(() => { - globalThis.localStorage.clear(); - document.documentElement.removeAttribute('data-theme'); -}); - -import SettingsPage from './+page.svelte'; - -describe('Settings → Appearance', () => { - test('renders three theme options', () => { - render(SettingsPage); - expect(screen.getByText(/^Appearance$/)).toBeInTheDocument(); - expect(screen.getByLabelText('dark', { exact: false })).toBeInTheDocument(); - expect(screen.getByLabelText('light', { exact: false })).toBeInTheDocument(); - expect(screen.getByLabelText('system', { exact: false })).toBeInTheDocument(); - }); - - test('clicking light writes localStorage and sets data-theme', async () => { - render(SettingsPage); - const lightRadio = screen.getByLabelText('light', { exact: false }) as HTMLInputElement; - await fireEvent.click(lightRadio); - expect(localStorage.getItem('minstrel:theme')).toBe('light'); - expect(document.documentElement.getAttribute('data-theme')).toBe('light'); - }); - - test('clicking dark writes localStorage and sets data-theme', async () => { - render(SettingsPage); - const darkRadio = screen.getByLabelText('dark', { exact: false }) as HTMLInputElement; - await fireEvent.click(darkRadio); - expect(localStorage.getItem('minstrel:theme')).toBe('dark'); - expect(document.documentElement.getAttribute('data-theme')).toBe('dark'); - }); -}); diff --git a/web/src/routes/settings/about/+page.svelte b/web/src/routes/settings/about/+page.svelte new file mode 100644 index 00000000..9bcd2d83 --- /dev/null +++ b/web/src/routes/settings/about/+page.svelte @@ -0,0 +1,25 @@ + + +{pageTitle('Settings · About')} + + + {#snippet main()} + + + + + {#if user.value} + + {user.value.username} + + {/if} + + {/snippet} + diff --git a/web/src/routes/settings/account/+page.svelte b/web/src/routes/settings/account/+page.svelte new file mode 100644 index 00000000..faba9ce1 --- /dev/null +++ b/web/src/routes/settings/account/+page.svelte @@ -0,0 +1,197 @@ + + +{pageTitle('Settings · Account')} + + + {#snippet main()} + + +
    + + {#if loadFailed} +
    + Couldn't load your profile. + +
    + {:else} + + {stored?.username ?? '…'} + + + + + + + +
    + +
    + {/if} +     +
    + +
    + + + + + + + + + + +
    + +
    +     +
    + {/snippet} + {#snippet side()} + + {/snippet} +     diff --git a/web/src/routes/settings/account/account.test.ts b/web/src/routes/settings/account/account.test.ts new file mode 100644 index 00000000..4ada7c47 --- /dev/null +++ b/web/src/routes/settings/account/account.test.ts @@ -0,0 +1,94 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor } from '@testing-library/svelte'; + +vi.mock('#lib/api/me.js', () => ({ + getMyProfile: vi.fn(), + updateProfile: vi.fn(), + changePassword: vi.fn(), + // ActiveSessions, the page's last group. + listSessions: vi.fn().mockResolvedValue([]), + revokeSession: vi.fn(), + revokeOtherSessions: vi.fn() +})); +vi.mock('#lib/api/client.js', async (importOriginal) => ({ + ...(await importOriginal()), + api: { get: vi.fn() } +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import AccountPage from './+page.svelte'; +import { getMyProfile, updateProfile, changePassword } from '#lib/api/me.js'; +import { api } from '#lib/api/client.js'; +import { pushToast } from '#lib/stores/toast.svelte.js'; + +const alice = { id: '1', username: 'alice', display_name: 'Alice', email: 'alice@example.com', is_admin: false }; + +afterEach(() => vi.clearAllMocks()); + +function setup(profile: typeof alice | Error = alice) { + if (profile instanceof Error) vi.mocked(getMyProfile).mockRejectedValue(profile); + else vi.mocked(getMyProfile).mockResolvedValue(profile); + vi.mocked(api.get).mockRejectedValue(new Error('no apk')); + return render(AccountPage); +} + +describe('Settings › Account', () => { + test('the form holds what is stored, and Save is off until something changes', async () => { + setup(); + await waitFor(() => expect(screen.getByLabelText(/display name/i)).toHaveValue('Alice')); + expect(screen.getByLabelText(/^email/i)).toHaveValue('alice@example.com'); + expect(screen.getByText('alice')).toBeInTheDocument(); + expect(screen.getByRole('button', { name: /save profile/i })).toBeDisabled(); + }); + + // A blank field reads as "clear" on the server, so a form that sent blanks + // would wipe the name and the email in one click. + test('changing the display name sends the display name alone, never a blank email', async () => { + setup(); + vi.mocked(updateProfile).mockResolvedValue({ ...alice, display_name: 'Al' }); + const name = screen.getByLabelText(/display name/i); + await waitFor(() => expect(name).toHaveValue('Alice')); + await fireEvent.input(name, { target: { value: 'Al' } }); + await fireEvent.click(screen.getByRole('button', { name: /save profile/i })); + await waitFor(() => expect(updateProfile).toHaveBeenCalledWith({ display_name: 'Al' })); + expect(pushToast).toHaveBeenCalledWith('Profile saved.'); + }); + + test('a profile that would not load offers a retry and no Save', async () => { + setup(new Error('down')); + expect(await screen.findByText(/couldn't load your profile/i)).toBeInTheDocument(); + expect(screen.queryByRole('button', { name: /save profile/i })).toBeNull(); + expect(screen.getByRole('button', { name: /try again/i })).toBeInTheDocument(); + }); + + test('mismatched passwords are refused before the server is asked', async () => { + setup(); + await fireEvent.input(screen.getByLabelText(/current password/i), { target: { value: 'oldpw1234' } }); + await fireEvent.input(screen.getByLabelText(/^new password/i), { target: { value: 'newpw1234' } }); + await fireEvent.input(screen.getByLabelText(/confirm new password/i), { target: { value: 'different1' } }); + await fireEvent.click(screen.getByRole('button', { name: /change password/i })); + expect(changePassword).not.toHaveBeenCalled(); + expect(pushToast).toHaveBeenCalledWith('New passwords do not match.', 'error'); + }); + + test('matched passwords call changePassword(current, new)', async () => { + setup(); + vi.mocked(changePassword).mockResolvedValue(undefined); + await fireEvent.input(screen.getByLabelText(/current password/i), { target: { value: 'oldpw1234' } }); + await fireEvent.input(screen.getByLabelText(/^new password/i), { target: { value: 'newpw1234' } }); + await fireEvent.input(screen.getByLabelText(/confirm new password/i), { target: { value: 'newpw1234' } }); + await fireEvent.click(screen.getByRole('button', { name: /change password/i })); + await waitFor(() => expect(changePassword).toHaveBeenCalledWith('oldpw1234', 'newpw1234')); + }); + + test('the Android app leads the page when the server bundles it', async () => { + vi.mocked(getMyProfile).mockResolvedValue(alice); + vi.mocked(api.get).mockResolvedValue({ version: '2026.10.10.2300', apk_url: '/api/client/apk', size_bytes: 31_457_280 }); + const { container } = render(AccountPage); + const download = await screen.findByRole('link', { name: /download/i }); + expect(download).toHaveAttribute('href', '/api/client/apk'); + expect(screen.getByText(/2026\.10\.10\.2300 · 31\.5 MB/)).toBeInTheDocument(); + const titles = Array.from(container.querySelectorAll('h2')).map((h) => h.textContent?.trim()); + expect(titles[0]).toBe('Android app'); + }); +}); diff --git a/web/src/routes/settings/appearance/+page.svelte b/web/src/routes/settings/appearance/+page.svelte new file mode 100644 index 00000000..9725a7be --- /dev/null +++ b/web/src/routes/settings/appearance/+page.svelte @@ -0,0 +1,26 @@ + + +{pageTitle('Settings · Appearance')} + + + {#snippet main()} + + + + + + {/snippet} + diff --git a/web/src/routes/settings/appearance/appearance.test.ts b/web/src/routes/settings/appearance/appearance.test.ts new file mode 100644 index 00000000..ef1b07c2 --- /dev/null +++ b/web/src/routes/settings/appearance/appearance.test.ts @@ -0,0 +1,31 @@ +import { beforeEach, describe, expect, test } from 'vitest'; +import { render, screen, fireEvent } from '@testing-library/svelte'; + +import AppearancePage from './+page.svelte'; + +beforeEach(() => { + globalThis.localStorage.clear(); + document.documentElement.removeAttribute('data-theme'); +}); + +describe('Settings › Appearance', () => { + test('the theme is one choice of three', () => { + render(AppearancePage); + expect(screen.getByRole('radiogroup', { name: 'Theme' })).toBeInTheDocument(); + expect(screen.getAllByRole('radio').map((r) => r.parentElement?.textContent?.trim())).toEqual(['Dark', 'Light', 'System']); + }); + + test('choosing light stores it and sets data-theme', async () => { + render(AppearancePage); + await fireEvent.click(screen.getByRole('radio', { name: 'Light' })); + expect(localStorage.getItem('minstrel:theme')).toBe('light'); + expect(document.documentElement.getAttribute('data-theme')).toBe('light'); + }); + + test('choosing dark stores it and sets data-theme', async () => { + render(AppearancePage); + await fireEvent.click(screen.getByRole('radio', { name: 'Dark' })); + expect(localStorage.getItem('minstrel:theme')).toBe('dark'); + expect(document.documentElement.getAttribute('data-theme')).toBe('dark'); + }); +}); diff --git a/web/src/routes/settings/connections/+page.svelte b/web/src/routes/settings/connections/+page.svelte new file mode 100644 index 00000000..50a1260d --- /dev/null +++ b/web/src/routes/settings/connections/+page.svelte @@ -0,0 +1,266 @@ + + +{pageTitle('Settings · Connections')} + +{#snippet lbHint()} + Get one at listenbrainz.org/profile. It's stored on this server unencrypted. +{/snippet} + +{#snippet secret(value: string, label: string)} +
    + {value} + +

    Copy it now. It won't be shown again.

    +
    +{/snippet} + + + {#snippet main()} + + {#if $status.isPending} +
    Loading…
    + {:else if $status.isError} +
    Couldn't load ListenBrainz status.
    + {:else if $status.data} + {@const s = $status.data} + + {lbState(s)} + + + {#if s.token_set} + Saved + + {:else} + + + {/if} + + + $enabledMutation.mutate(!s.enabled)} + class="h-4 w-4 cursor-pointer accent-accent disabled:cursor-not-allowed" + /> + + {/if} +     + {/snippet} + {#snippet side()} + + + + + {#if apiToken} + {@render secret(apiToken, 'API token')} + {/if} + + + {#if subsonicEnabled !== null} + {subsonicEnabled ? 'On' : 'Off'} + {/if} + + + {#if subsonicEnabled} + + {/if} + + + {#if subsonicPassword} + {@render secret(subsonicPassword, 'Subsonic password')} + {/if} + + {/snippet} +     diff --git a/web/src/routes/settings/connections/connections.test.ts b/web/src/routes/settings/connections/connections.test.ts new file mode 100644 index 00000000..0b35c43e --- /dev/null +++ b/web/src/routes/settings/connections/connections.test.ts @@ -0,0 +1,118 @@ +import { afterEach, describe, expect, test, vi } from 'vitest'; +import { render, screen, fireEvent, waitFor, within } from '@testing-library/svelte'; +import { readable } from 'svelte/store'; +import type { LBStatus } from '#lib/api/listenbrainz.js'; + +vi.mock('#lib/api/listenbrainz.js', () => ({ + createLBStatusQuery: vi.fn(), + createTokenMutation: vi.fn(), + createEnabledMutation: vi.fn() +})); +vi.mock('#lib/api/me.js', () => ({ + regenerateAPIToken: vi.fn(), + getSubsonicPasswordStatus: vi.fn().mockResolvedValue({ enabled: false }), + generateSubsonicPassword: vi.fn(), + clearSubsonicPassword: vi.fn() +})); +vi.mock('#lib/stores/toast.svelte.js', () => ({ pushToast: vi.fn() })); + +import ConnectionsPage from './+page.svelte'; +import { createLBStatusQuery, createTokenMutation, createEnabledMutation } from '#lib/api/listenbrainz.js'; +import { + regenerateAPIToken, + getSubsonicPasswordStatus, + generateSubsonicPassword, + clearSubsonicPassword +} from '#lib/api/me.js'; + +const HOUR = 3_600_000; + +function setup(lb: LBStatus = { enabled: false, token_set: false, last_scrobbled_at: null }, mutate = vi.fn()) { + vi.mocked(createLBStatusQuery).mockReturnValue( + readable({ data: lb, isPending: false, isError: false }) as unknown as ReturnType + ); + const mutation = () => + readable({ isPending: false, mutate, mutateAsync: vi.fn() }) as unknown as ReturnType; + vi.mocked(createTokenMutation).mockReturnValue(mutation()); + vi.mocked(createEnabledMutation).mockReturnValue(mutation() as unknown as ReturnType); + return render(ConnectionsPage); +} + +afterEach(() => vi.clearAllMocks()); + +describe('Settings › Connections — ListenBrainz', () => { + test('with no token: says not connected, asks for one, and sending is off', () => { + setup(); + expect(screen.getByTestId('lb-state')).toHaveTextContent('Not connected'); + expect(screen.getByPlaceholderText(/paste your token/i)).toBeInTheDocument(); + expect(screen.getByRole('link', { name: 'listenbrainz.org/profile' })).toBeInTheDocument(); + expect(screen.getByRole('switch', { name: /send my plays/i })).toBeDisabled(); + }); + + test('Save sends the typed token', async () => { + const mutate = vi.fn(); + setup(undefined, mutate); + await fireEvent.input(screen.getByPlaceholderText(/paste your token/i), { target: { value: 'mytoken' } }); + await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); + expect(mutate).toHaveBeenCalledWith('mytoken', expect.anything()); + }); + + test('a saved token can be removed', async () => { + const mutate = vi.fn(); + setup({ enabled: false, token_set: true, last_scrobbled_at: null }, mutate); + expect(screen.getByTestId('lb-state')).toHaveTextContent('Connected · not sending plays'); + expect(screen.getByText('Saved')).toBeInTheDocument(); + await fireEvent.click(screen.getByRole('button', { name: /remove/i })); + expect(mutate).toHaveBeenCalledWith(''); + }); + + test('sending plays says when the last one went', () => { + setup({ enabled: true, token_set: true, last_scrobbled_at: new Date(Date.now() - 3 * HOUR).toISOString() }); + expect(screen.getByTestId('lb-state')).toHaveTextContent('Sending plays · last one 3h ago'); + expect(screen.getByRole('switch', { name: /send my plays/i })).toBeChecked(); + }); +}); + +describe('Settings › Connections — Subsonic apps', () => { + test('Regenerate asks for a second click, then shows the new token once', async () => { + setup(); + vi.mocked(regenerateAPIToken).mockResolvedValue({ api_token: 'new_tok_xyz' }); + expect(screen.queryByTestId('secret')).toBeNull(); + await fireEvent.click(screen.getByRole('button', { name: /^regenerate$/i })); + expect(regenerateAPIToken).not.toHaveBeenCalled(); + await fireEvent.click(screen.getByRole('button', { name: /click again to confirm/i })); + await waitFor(() => expect(screen.getByTestId('secret')).toHaveTextContent('new_tok_xyz')); + expect(screen.getByRole('button', { name: 'Copy API token' })).toBeInTheDocument(); + }); + + test('with no Subsonic password, Generate makes one on the first click and shows it once', async () => { + vi.mocked(getSubsonicPasswordStatus).mockResolvedValue({ enabled: false }); + vi.mocked(generateSubsonicPassword).mockResolvedValue({ password: 'gen_pw_123' }); + setup(); + await waitFor(() => expect(screen.getByTestId('subsonic-state')).toHaveTextContent('Off')); + expect(screen.queryByRole('button', { name: /turn off/i })).toBeNull(); + await fireEvent.click(screen.getByRole('button', { name: /^generate$/i })); + await waitFor(() => expect(generateSubsonicPassword).toHaveBeenCalledTimes(1)); + await waitFor(() => expect(screen.getByTestId('secret')).toHaveTextContent('gen_pw_123')); + expect(screen.getByTestId('subsonic-state')).toHaveTextContent('On'); + }); + + test('with one set, Regenerate and Turn off each need a second click', async () => { + vi.mocked(getSubsonicPasswordStatus).mockResolvedValue({ enabled: true }); + vi.mocked(clearSubsonicPassword).mockResolvedValue(undefined); + setup(); + const actions = await screen.findByTestId('subsonic-actions'); + const turnOff = await within(actions).findByRole('button', { name: /turn off/i }); + const regen = within(actions).getByRole('button', { name: /^regenerate$/i }); + + await fireEvent.click(regen); + expect(generateSubsonicPassword).not.toHaveBeenCalled(); + expect(regen).toHaveTextContent(/click again to confirm/i); + + await fireEvent.click(turnOff); + expect(clearSubsonicPassword).not.toHaveBeenCalled(); + await fireEvent.click(turnOff); + await waitFor(() => expect(clearSubsonicPassword).toHaveBeenCalledTimes(1)); + await waitFor(() => expect(screen.getByTestId('subsonic-state')).toHaveTextContent('Off')); + }); +}); diff --git a/web/src/routes/settings/notifications/+page.svelte b/web/src/routes/settings/notifications/+page.svelte new file mode 100644 index 00000000..4cd58450 --- /dev/null +++ b/web/src/routes/settings/notifications/+page.svelte @@ -0,0 +1,13 @@ + + +{pageTitle('Settings · Notifications')} + + + {#snippet main()} + + {/snippet} + diff --git a/web/src/routes/settings/playback/+page.svelte b/web/src/routes/settings/playback/+page.svelte new file mode 100644 index 00000000..fadc038a --- /dev/null +++ b/web/src/routes/settings/playback/+page.svelte @@ -0,0 +1,41 @@ + + +{pageTitle('Settings · Playback')} + + + {#snippet main()} + + + setCrossfade(Number((e.currentTarget as HTMLInputElement).value))} + class="w-40 accent-accent" + /> + + {player.crossfadeSec === 0 ? 'Off' : `${player.crossfadeSec}s`} + + + + {/snippet} + {#snippet side()} + + + + {/snippet} + diff --git a/web/src/routes/settings/settings.test.ts b/web/src/routes/settings/settings.test.ts index ef3dfcf5..64d06670 100644 --- a/web/src/routes/settings/settings.test.ts +++ b/web/src/routes/settings/settings.test.ts @@ -1,437 +1,49 @@ import { afterEach, describe, expect, test, vi } from 'vitest'; -import { render, screen, fireEvent, waitFor, within } from '@testing-library/svelte'; -import { readable, writable } from 'svelte/store'; -import type { LBStatus } from '#lib/api/listenbrainz.js'; +import { render, screen } from '@testing-library/svelte'; +import { createRawSnippet } from 'svelte'; +import { pageUrlModule } from '#test-utils/mocks/appState.js'; -vi.mock('#lib/api/listenbrainz.js', () => ({ - createLBStatusQuery: vi.fn(), - createTokenMutation: vi.fn(), - createEnabledMutation: vi.fn(), - setListenBrainzToken: vi.fn(), - setListenBrainzEnabled: vi.fn() -})); +const state = vi.hoisted(() => ({ pageUrl: new URL('http://localhost/settings/account') })); +vi.mock('$app/state', () => pageUrlModule(state)); -// The Notifications section has its own tests (NotificationSettings.test.ts); -// here it only needs a settled query so the page renders. -vi.mock('#lib/api/notifications.js', async (importOriginal) => { - const mod = await importOriginal(); - const { mockQuery } = await import('../../test-utils/query'); - return { - ...mod, - createNotificationSettingsQuery: () => - mockQuery({ data: { kinds: [], email_available: true } }), - putNotificationSettings: vi.fn() - }; -}); +import SettingsLayout from './+layout.svelte'; +import { load } from './+page'; -// The Volume leveling setting loads on mount; keep it off the network. -vi.mock('#lib/api/normalization.js', async (importOriginal) => { - const mod = await importOriginal(); - return { - ...mod, - getNormalization: vi.fn().mockResolvedValue(mod.DEFAULT_NORMALIZATION), - putNormalization: vi.fn() - }; -}); +const children = createRawSnippet(() => ({ render: () => `

    body

    ` })); -vi.mock('#lib/api/me.js', () => ({ - updateProfile: vi.fn(), - changePassword: vi.fn(), - // Default to a resolved value so the page's $effect doesn't crash - // on `.then()` of undefined when individual tests don't override. - regenerateAPIToken: vi.fn(), - getSubsonicPasswordStatus: vi.fn(), - generateSubsonicPassword: vi.fn(), - clearSubsonicPassword: vi.fn() -})); +afterEach(() => vi.clearAllMocks()); -// Mutable holder so individual tests can inject populated metrics; -// vi.mock is hoisted, hence vi.hoisted for the shared reference. -const metricsMock = vi.hoisted(() => ({ - data: { window_days: 30, baseline: null, groups: [] } as unknown -})); +function setup(path: string) { + state.pageUrl = new URL(`http://localhost${path}`); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + return render(SettingsLayout, { children } as any); +} -vi.mock('#lib/api/metrics.js', () => ({ - createRecommendationMetricsQuery: () => ({ - subscribe: (run: (v: unknown) => void) => { - run({ isPending: false, isError: false, data: metricsMock.data }); - return () => {}; +describe('/settings', () => { + test('the sections run down the left, Account first, the current one marked', () => { + setup('/settings/connections'); + const nav = screen.getByRole('navigation', { name: /settings sections/i }); + const links = Array.from(nav.querySelectorAll('a')).map((a) => [a.getAttribute('href'), a.textContent?.trim()]); + expect(links).toEqual([ + ['/settings/account', 'Account'], + ['/settings/playback', 'Playback'], + ['/settings/appearance', 'Appearance'], + ['/settings/notifications', 'Notifications'], + ['/settings/connections', 'Connections'], + ['/settings/about', 'About'] + ]); + expect(screen.getByRole('link', { name: 'Connections' })).toHaveAttribute('aria-current', 'page'); + expect(screen.getByRole('link', { name: 'Account' })).not.toHaveAttribute('aria-current'); + expect(screen.getByTestId('section-body')).toBeTruthy(); + }); + + test('a bare /settings lands on Account', () => { + let thrown: unknown; + try { + load(); + } catch (e) { + thrown = e; } - }) -})); - -import SettingsPage from './+page.svelte'; -import { - createLBStatusQuery, - createTokenMutation, - createEnabledMutation, - setListenBrainzToken, - setListenBrainzEnabled -} from '#lib/api/listenbrainz.js'; -import { - updateProfile, - changePassword, - regenerateAPIToken, - getSubsonicPasswordStatus, - generateSubsonicPassword, - clearSubsonicPassword -} from '#lib/api/me.js'; - -function mockStatusStore(data: LBStatus) { - return readable({ data, isPending: false, isError: false }); -} - -function mockMutationStore(mutateFn: (vars: unknown) => void = vi.fn()) { - return readable({ isPending: false, mutate: mutateFn, mutateAsync: vi.fn() }); -} - -afterEach(() => { - vi.clearAllMocks(); - metricsMock.data = { window_days: 30, baseline: null, groups: [] }; -}); - -describe('Settings page — ListenBrainz', () => { - test('token-not-set state renders input + Save button', async () => { - (createLBStatusQuery as ReturnType).mockReturnValue( - mockStatusStore({ enabled: false, token_set: false, last_scrobbled_at: null }) - ); - (createTokenMutation as ReturnType).mockReturnValue(mockMutationStore()); - (createEnabledMutation as ReturnType).mockReturnValue(mockMutationStore()); - render(SettingsPage); - await waitFor(() => - expect(screen.getByPlaceholderText(/paste your lb token/i)).toBeInTheDocument() - ); - expect(screen.getByRole('button', { name: /^save$/i })).toBeInTheDocument(); - }); - - test('token-set state renders masked + Clear button', async () => { - (createLBStatusQuery as ReturnType).mockReturnValue( - mockStatusStore({ enabled: false, token_set: true, last_scrobbled_at: null }) - ); - (createTokenMutation as ReturnType).mockReturnValue(mockMutationStore()); - (createEnabledMutation as ReturnType).mockReturnValue(mockMutationStore()); - render(SettingsPage); - await waitFor(() => expect(screen.getByText(/\(set\)/)).toBeInTheDocument()); - expect(screen.getByText('clear')).toBeInTheDocument(); - }); - - test('Save button calls setListenBrainzToken with typed value', async () => { - (createLBStatusQuery as ReturnType).mockReturnValue( - mockStatusStore({ enabled: false, token_set: false, last_scrobbled_at: null }) - ); - const mutateFn = vi.fn(); - (createTokenMutation as ReturnType).mockReturnValue(mockMutationStore(mutateFn)); - (createEnabledMutation as ReturnType).mockReturnValue(mockMutationStore()); - render(SettingsPage); - const input = await screen.findByPlaceholderText(/paste your lb token/i); - await fireEvent.input(input, { target: { value: 'mytoken' } }); - await fireEvent.click(screen.getByRole('button', { name: /^save$/i })); - await waitFor(() => expect(mutateFn).toHaveBeenCalledWith('mytoken', expect.anything())); - }); - - test('Enabled checkbox is disabled when no token', async () => { - (createLBStatusQuery as ReturnType).mockReturnValue( - mockStatusStore({ enabled: false, token_set: false, last_scrobbled_at: null }) - ); - (createTokenMutation as ReturnType).mockReturnValue(mockMutationStore()); - (createEnabledMutation as ReturnType).mockReturnValue(mockMutationStore()); - render(SettingsPage); - const checkbox = await screen.findByRole('checkbox'); - expect(checkbox).toBeDisabled(); - }); - - test('Last-scrobbled-at renders when present', async () => { - (createLBStatusQuery as ReturnType).mockReturnValue( - mockStatusStore({ enabled: true, token_set: true, last_scrobbled_at: '2026-04-28T03:00:00Z' }) - ); - (createTokenMutation as ReturnType).mockReturnValue(mockMutationStore()); - (createEnabledMutation as ReturnType).mockReturnValue(mockMutationStore()); - render(SettingsPage); - await waitFor(() => - expect(screen.getByText(/last scrobbled:/i)).toBeInTheDocument() - ); - }); -}); - -// Shared setup for cards that don't need LB state -function setupPage() { - (createLBStatusQuery as ReturnType).mockReturnValue( - mockStatusStore({ enabled: false, token_set: false, last_scrobbled_at: null }) - ); - (createTokenMutation as ReturnType).mockReturnValue(mockMutationStore()); - (createEnabledMutation as ReturnType).mockReturnValue(mockMutationStore()); -} - -describe('Settings page — Profile card', () => { - test('Save profile calls updateProfile with form values', async () => { - setupPage(); - (updateProfile as ReturnType).mockResolvedValue({}); - render(SettingsPage); - await fireEvent.input(screen.getByLabelText(/display name/i), { target: { value: 'Alice' } }); - await fireEvent.input(screen.getByLabelText(/email/i), { target: { value: 'alice@example.com' } }); - await fireEvent.click(screen.getByRole('button', { name: /save profile/i })); - await waitFor(() => - expect(updateProfile).toHaveBeenCalledWith( - expect.objectContaining({ display_name: 'Alice', email: 'alice@example.com' }) - ) - ); - }); - - test('Save profile shows toast on success', async () => { - setupPage(); - (updateProfile as ReturnType).mockResolvedValue({}); - render(SettingsPage); - await fireEvent.click(screen.getByRole('button', { name: /save profile/i })); - await waitFor(() => expect(screen.getByTestId('toast').textContent).toMatch(/profile saved/i)); - }); -}); - -describe('Settings page — Password card', () => { - test('mismatched passwords show toast and do not call changePassword', async () => { - setupPage(); - render(SettingsPage); - await fireEvent.input(screen.getByLabelText(/current password/i), { target: { value: 'oldpw1234' } }); - await fireEvent.input(screen.getByLabelText(/new password/i), { target: { value: 'newpw1234' } }); - await fireEvent.input(screen.getByLabelText(/confirm/i), { target: { value: 'different1' } }); - await fireEvent.click(screen.getByRole('button', { name: /change password/i })); - expect(changePassword).not.toHaveBeenCalled(); - await waitFor(() => expect(screen.getByTestId('toast').textContent).toMatch(/do not match/i)); - }); - - test('matched passwords call changePassword(current, new)', async () => { - setupPage(); - (changePassword as ReturnType).mockResolvedValue(undefined); - render(SettingsPage); - await fireEvent.input(screen.getByLabelText(/current password/i), { target: { value: 'oldpw1234' } }); - await fireEvent.input(screen.getByLabelText(/new password/i), { target: { value: 'newpw1234' } }); - await fireEvent.input(screen.getByLabelText(/confirm/i), { target: { value: 'newpw1234' } }); - await fireEvent.click(screen.getByRole('button', { name: /change password/i })); - await waitFor(() => - expect(changePassword).toHaveBeenCalledWith('oldpw1234', 'newpw1234') - ); - }); -}); - -describe('Settings page — Recommendation metrics card', () => { - const metric = (key: string, label: string, over: Record = {}) => ({ - key, - label, - plays: 30, - skips: 3, - skip_rate: 0.1, - avg_completion: 0.9, - low_confidence: false, - ...over - }); - - test('breakdown rows are collapsed by default and expand on toggle (#1249/#1270)', async () => { - setupPage(); - metricsMock.data = { - window_days: 30, - baseline: metric('manual', 'Manual library plays', { plays: 100 }), - groups: [ - { - intent: 'go_to', - label: 'Go-to surfaces', - surfaces: [ - metric('for_you', 'For You', { - plays: 45, - breakdown: [ - metric('for_you_taste', 'Taste picks'), - metric('for_you_fresh', 'Fresh picks', { - plays: 10, - skip_rate: 0.4, - low_confidence: true - }), - metric('for_you_unattributed', 'Earlier plays', { plays: 5 }) - ] - }) - ] - } - ] - }; - render(SettingsPage); - const toggle = await screen.findByRole('button', { name: /for you/i }); - // Collapsed by default: every stamping mix now carries a breakdown, - // so always-open sub-rows would swamp the surface-level view. - expect(toggle).toHaveAttribute('aria-expanded', 'false'); - expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument(); - - await fireEvent.click(toggle); - expect(toggle).toHaveAttribute('aria-expanded', 'true'); - expect(screen.getByText(/Taste picks/)).toBeInTheDocument(); - expect(screen.getByText(/Fresh picks/)).toBeInTheDocument(); - expect(screen.getByText(/Earlier plays/)).toBeInTheDocument(); - - await fireEvent.click(toggle); - expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument(); - }); - - // #2495: the card used to render a delta computed client-side with no notion - // of uncertainty, so a -12 on 59 plays looked exactly as solid as a -6 on 400. - // Deltas now arrive from the server with a margin, and an indistinguishable - // one is marked with "≈" and dimmed rather than coloured. - test('a delta smaller than its margin is marked as indistinguishable', async () => { - setupPage(); - metricsMock.data = { - window_days: 30, - baseline: metric('manual', 'Manual library plays', { plays: 400, skip_rate: 0.27 }), - groups: [ - { - intent: 'discovery', - label: 'Discovery mixes', - surfaces: [ - metric('discover', 'Discover', { - plays: 59, - skip_rate: 0.153, - // 13.3pp gap, but the margin at n=59 is wider than the gap. - skip_delta: { delta_pp: -13.3, margin_pp: 14.5, distinguishable: false }, - completion_delta: { delta_pp: 28.0, margin_pp: 12.1, distinguishable: true } - }) - ] - } - ] - }; - render(SettingsPage); - await waitFor(() => expect(screen.getByText('Discover')).toBeInTheDocument()); - - // Targeted by test id rather than text: the legend below the table also - // contains a "≈", so matching on the glyph finds the explanation instead of - // the delta. (It did, on the first attempt at this test.) - const skip = screen.getByTestId('skip-delta-discover'); - expect(skip).toHaveTextContent('≈-13'); - expect(skip).toHaveAttribute('title', expect.stringContaining('not distinguishable from zero')); - // It must NOT be coloured as a real regression/improvement. - expect(skip.className).toContain('opacity-60'); - - // The completion delta clears its margin, so it renders plainly. - const completion = screen.getByTestId('completion-delta-discover'); - expect(completion).toHaveTextContent('+28'); - expect(completion.className).not.toContain('opacity-60'); - - // And the legend explains the glyph rather than leaving it a mystery. - expect(screen.getByText(/smaller than its own margin of error/i)).toBeInTheDocument(); - }); - - // A delta is omitted entirely when the samples are too thin for a margin to - // mean anything — the server decides that, and the cell must simply show the - // rate rather than a bare "0". - test('a surface with no delta shows its rate and nothing else', async () => { - setupPage(); - metricsMock.data = { - window_days: 30, - baseline: metric('manual', 'Manual library plays', { plays: 400 }), - groups: [ - { - intent: 'go_to', - label: 'Go-to surfaces', - surfaces: [metric('radio', 'Radio', { plays: 1, skip_rate: 0 })] - } - ] - }; - render(SettingsPage); - await waitFor(() => expect(screen.getByText('Radio')).toBeInTheDocument()); - expect(screen.queryByTestId('skip-delta-radio')).not.toBeInTheDocument(); - expect(screen.queryByTestId('completion-delta-radio')).not.toBeInTheDocument(); - }); - - test('surfaces without a breakdown render no toggle and no sub-rows', async () => { - setupPage(); - metricsMock.data = { - window_days: 30, - baseline: null, - groups: [ - { - intent: 'go_to', - label: 'Go-to surfaces', - surfaces: [metric('radio', 'Radio')] - } - ] - }; - render(SettingsPage); - await waitFor(() => expect(screen.getByText('Radio')).toBeInTheDocument()); - expect(screen.queryByRole('button', { name: /radio/i })).not.toBeInTheDocument(); - expect(screen.queryByText(/Taste picks/)).not.toBeInTheDocument(); - }); -}); - -describe('Settings page — API Token card', () => { - test('first click on Regenerate shows "Click again to confirm"', async () => { - setupPage(); - render(SettingsPage); - await fireEvent.click(screen.getByRole('button', { name: /regenerate/i })); - await waitFor(() => - expect(screen.getByRole('button', { name: /click again to confirm/i })).toBeInTheDocument() - ); - expect(regenerateAPIToken).not.toHaveBeenCalled(); - }); - - test('second click within 5s calls regenerateAPIToken', async () => { - setupPage(); - (regenerateAPIToken as ReturnType).mockResolvedValue({ api_token: 'new_tok_xyz' }); - render(SettingsPage); - await fireEvent.click(screen.getByRole('button', { name: /regenerate/i })); - await waitFor(() => - expect(screen.getByRole('button', { name: /click again to confirm/i })).toBeInTheDocument() - ); - await fireEvent.click(screen.getByRole('button', { name: /click again to confirm/i })); - await waitFor(() => expect(regenerateAPIToken).toHaveBeenCalled()); - // The new key is shown once, with a way to copy it. - expect(await screen.findByText('new_tok_xyz')).toBeInTheDocument(); - expect(screen.getByRole('button', { name: /^copy$/i })).toBeInTheDocument(); - }); - - test('no token is shown or fetched before Regenerate', async () => { - setupPage(); - render(SettingsPage); - await waitFor(() => - expect(screen.getByRole('button', { name: /regenerate/i })).toBeInTheDocument() - ); - expect(screen.queryByRole('button', { name: /^copy$/i })).toBeNull(); - }); -}); - -describe('Settings page — Subsonic password card', () => { - function mockLB() { - (createLBStatusQuery as ReturnType).mockReturnValue( - mockStatusStore({ enabled: false, token_set: false, last_scrobbled_at: null }) - ); - (createTokenMutation as ReturnType).mockReturnValue(mockMutationStore()); - (createEnabledMutation as ReturnType).mockReturnValue(mockMutationStore()); - } - - test('with none set, Generate creates one on the first click and shows it once', async () => { - mockLB(); - (getSubsonicPasswordStatus as ReturnType).mockResolvedValue({ enabled: false }); - (generateSubsonicPassword as ReturnType).mockResolvedValue({ password: 'gen_pw_123' }); - render(SettingsPage); - await waitFor(() => expect(screen.getByText(/no subsonic password is set/i)).toBeInTheDocument()); - expect(screen.queryByRole('button', { name: /turn off/i })).not.toBeInTheDocument(); - - await fireEvent.click(screen.getByRole('button', { name: /^generate$/i })); - await waitFor(() => expect(generateSubsonicPassword).toHaveBeenCalledTimes(1)); - await waitFor(() => expect(screen.getByText('gen_pw_123')).toBeInTheDocument()); - expect(screen.getByText(/a subsonic password is set/i)).toBeInTheDocument(); - }); - - test('with one set, Regenerate and Turn off each need a second click', async () => { - mockLB(); - (getSubsonicPasswordStatus as ReturnType).mockResolvedValue({ enabled: true }); - (clearSubsonicPassword as ReturnType).mockResolvedValue(undefined); - render(SettingsPage); - // The API token card has a Regenerate button too; the Subsonic card's is - // the one beside Turn off. - const turnOff = await screen.findByRole('button', { name: /turn off/i }); - const subsonicRegen = within(turnOff.parentElement!).getByRole('button', { name: /^regenerate$/i }); - - await fireEvent.click(subsonicRegen); - expect(generateSubsonicPassword).not.toHaveBeenCalled(); - expect(subsonicRegen).toHaveTextContent(/click again to confirm/i); - - await fireEvent.click(turnOff); - expect(clearSubsonicPassword).not.toHaveBeenCalled(); - await fireEvent.click(turnOff); - await waitFor(() => expect(clearSubsonicPassword).toHaveBeenCalledTimes(1)); - await waitFor(() => expect(screen.getByText(/no subsonic password is set/i)).toBeInTheDocument()); + expect(thrown).toMatchObject({ status: 308, location: '/settings/account' }); }); }); diff --git a/web/src/test-utils/appModuleMocks.test.ts b/web/src/test-utils/appModuleMocks.test.ts index 1359f45c..0564fdf1 100644 --- a/web/src/test-utils/appModuleMocks.test.ts +++ b/web/src/test-utils/appModuleMocks.test.ts @@ -2,8 +2,8 @@ import { describe, expect, test, vi } from 'vitest'; import { goto } from '$app/navigation'; import { page } from '$app/state'; -// Guards the suite-wide $app mocks in vitest.setup.ts (#3943). If they are -// removed, tests that never mock $app themselves go back to loading +// Guards the suite-wide $app mocks in vitest.setup.ts. If they are +// removed, tests that never mock $app themselves load // SvelteKit's client runtime, which fails at module load on some CI runs. describe('suite-wide SvelteKit app-module mocks', () => { test('$app/navigation is the mock, not the client runtime', () => { diff --git a/web/vitest.setup.ts b/web/vitest.setup.ts index 9584fb78..1ac5f700 100644 --- a/web/vitest.setup.ts +++ b/web/vitest.setup.ts @@ -15,17 +15,16 @@ vi.mock('@tanstack/svelte-query', async (orig) => { return { ...actual, useQueryClient: () => ({ invalidateQueries: vi.fn() }) }; }); -// Default SvelteKit app-module mocks for the whole suite (#3943). Without -// them, any page or component that reaches $app/navigation or $app/state — -// the home page does, through AlbumCard → AlbumMenu — loads SvelteKit's -// client runtime, whose $app/paths reads __SVELTEKIT_PAYLOAD__ at module -// load. That global only exists when the kit plugin's define reaches the -// module, which it does not reliably under vitest: page.test.ts failed to -// load on one CI run and passed on the re-run of the same commit (#374 -// was the same class). Stubbing the modules keeps every test off the -// client bootstrap. Tests that assert on goto or need a particular URL -// still mock per file (pageUrlModule in test-utils/mocks/appState.ts), -// and per-file mocks win over these. +// Default SvelteKit app-module mocks for the whole suite. Without them, any +// page or component that reaches $app/navigation or $app/state — the home page +// does, through AlbumCard → AlbumMenu — loads SvelteKit's client runtime, whose +// $app/paths reads __SVELTEKIT_PAYLOAD__ at module load. That global only +// exists when the kit plugin's define reaches the module, which it does not +// reliably under vitest, so a page test can fail to load on one run and pass on +// the next. Stubbing the modules keeps every test off the client bootstrap. +// Tests that assert on goto or need a particular URL still mock per file +// (pageUrlModule in test-utils/mocks/appState.ts), and per-file mocks win over +// these. vi.mock('$app/navigation', () => ({ goto: vi.fn(() => Promise.resolve()), invalidate: vi.fn(() => Promise.resolve()),