feat: the extension adds Discord channels to an artist you pick, and its tests gate the XPI (milestone 429)

Server (#4420) - extension_service gains a Discord pattern (server or channel, jump links, ptb/canary; not DMs or threads), mirrored in platforms.js and pinned by the shared artist-url-samples.json. - probe on a Discord URL matches the source by ids under any artist, reports a whole-server source as covering the channel, suggests the artist who owns another source on the same server, and names server/channel via the stored token (best-effort, bounded, no rate-limit waits). - quick-add takes artist_id / artist_name; Discord URLs are stored canonical. Extension (#4421, #4422) - Content script on discord.com; SPA navigation by URL polling (the old pushState patch ran in the isolated world and never fired); stale probes are dropped. - Discord chip opens an Add panel: this channel or the whole server, and the suggested artist / a search / a new name. - Popup: sources show artist, platform and state; a Discord token export is verified by FC and the result shown. Token capture covers ptb/canary. - Pure logic in lib/chip.js and lib/popup-format.js, with specs. CI (#4423) - extension.yml's lane (web-ext lint, vitest, XPI contents) moves into build.yml as extension-test and joins the needs of sign-extension, build-web and build-agent. As a separate workflow it gated nothing: a red extension suite still signed and shipped the XPI (rule 177). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
fix: a never-checked source still reports no next check while its first backfill is armed
2026-09-24 23:31:38 -04:00 · 2026-09-24 23:19:23 -04:00 · 2026-09-24 23:14:48 -04:00 · 2026-09-24 19:42:00 -04:00 · 2026-09-24 19:26:48 -04:00 · 2026-09-24 19:09:07 -04:00
261 changed files with 21236 additions and 4532 deletions
@@ -79,7 +79,7 @@ jobs:
      - name: Resolve the Postgres service and install deps
        run: |
          set -eux
-          # Same service-IP dance as ci.yml's integration job; see the long
+          # Same service-IP dance as build.yml's integration job; see the long
          # comment there for why the job name must stay separator-free.
          PG=$(docker ps --filter "name=compare" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
          test -n "$PG"
@@ -89,7 +89,7 @@ jobs:
          echo "DB_HOST=$PG_IP" >> "$GITHUB_ENV"
          # Socket probe in python, not bash's /dev/tcp — these steps run under
          # `sh -e`, where that path does not exist. Same fix and same reasoning
-          # as ci.yml's integration job; see the comment there.
+          # as build.yml's integration job; see the comment there.
          pg_ready=""
          for i in $(seq 1 60); do
            if python -c "import socket,sys; s=socket.socket(); s.settimeout(2); sys.exit(0 if s.connect_ex(('$PG_IP', 5432)) == 0 else 1)"; then
@@ -1,294 +0,0 @@
 name: CI
 # CI lanes per FabledRulebook/forgejo.md "CI philosophy":
 #   - lint: ruff only, no dep install — fast-fail for the common lint bounce.
 #   - extension-version: the derived version resolves and is a shape AMO takes.
 #   - backend-lint-and-test: `pytest -m "not integration"`, no service containers.
 #   - frontend-build: vitest unit + vite build.
 #   - integration: pgvector + redis service containers; alembic + `pytest -m integration`.
 on:
  push:
    branches: [dev, main]
  # Renovate opens PRs from `renovate/*` branches into `dev`. Those branches
  # never push to dev/main, so the push trigger above gives them NO pre-merge
  # CI — a bump could only be validated after it was already merged. This
  # pull_request trigger (base `dev` only) validates Renovate PRs before merge.
  # It deliberately does NOT fire on dev→main PRs (base `main`), which still
  # rely on the dev push run — so no duplicate runs. FC has no fork PRs
  # (single-operator Forgejo repo), so secrets-on-PR is not a concern.
  pull_request:
    branches: [dev]
 jobs:
  # Fast-fail lint lane. ruff is pre-installed in the ci-python image, so
  # this runs with NO dependency install and surfaces the most common bounce
  # class (lint: I001 / UP037 / ASYNC109 / W293 …) in seconds — instead of
  # after the backend job's ~30-60s wheel install. ruff is static analysis,
  # so no DB/secret env is needed.
  lint:
    runs-on: python-ci
    container:
      image: git.fabledsword.com/bvandeusen/ci-python:3.14
    steps:
      - uses: actions/checkout@v4
      - name: Ruff lint
        # agent/ included so the GPU-agent is linted before its image is built
        # (build.yml only `docker build`s it — this is where it gets checked).
        # scripts/ likewise: release_notes.py runs only on a tag push, so a
        # syntax or import error there would otherwise surface at the one
        # moment nobody wants to debug a workflow.
        run: ruff check backend/ tests/ alembic/ agent/ scripts/
      - name: Agent syntax check
        # The agent's runtime deps (torch/transformers/ultralytics) aren't in the
        # CI image, so we can't import it — but compileall parses every module,
        # catching syntax errors before the image build.
        run: python -m compileall -q agent/fc_agent
  # The extension version is DERIVED, not hand-maintained (milestone 271 step
  # 4): build.yml computes it from the commit TIME of the newest packaged
  # extension change and stamps it into manifest.json / package.json at build
  # time. The guard that used to live here — "packaged files changed but nobody
  # bumped the version" — was therefore checking a fact that had stopped
  # existing. Worse than useless: it would have failed this lane on every real
  # extension change, demanding a bump that decides nothing. Retired 2026-08-27
  # rather than left running beside the new mechanism (rule 22).
  #
  # Two things are still worth asserting, and this is the only lane that can:
  # the extension.yml suite runs on node:24-slim, which is exactly why
  # version.spec.js sticks to packaging.sh's git-free subcommands.
  #   1. the derivation actually resolves on this commit
  #   2. the derived string is one AMO will accept, checked against Mozilla's
  #      own published grammar rather than a loose "digits and dots"
  #
  # The MAJOR.MINOR-agreement check that used to be (2) is gone with milestone
  # 318 step 8: the committed version no longer seeds anything, so there is no
  # hand-set part left for the two files to disagree about.
  #
  # Deliberately NOT checked here: that the derived value beats what has already
  # been signed. That guard belongs in build.yml, where it compares against the
  # real ext-* releases. Comparing against origin/main here would be wrong —
  # dev legitimately derives a LOWER value whenever main is ahead on the
  # extension, and a lane that fails for being behind is a lane people learn to
  # ignore.
  extension-version:
    runs-on: python-ci
    container:
      image: git.fabledsword.com/bvandeusen/ci-python:3.14
    steps:
      - uses: actions/checkout@v4
        with:
          # The derivation needs real history: a depth-1 clone sees one commit
          # and produces a wrong, too-low value RATHER THAN FAILING. Checking
          # that here is half the point of the lane.
          fetch-depth: 0
      - name: Extension version derives cleanly
        run: |
          set -eu
          # busybox sh on the act_runner — no bashisms (family rule).
          VERSION=$(sh extension/scripts/packaging.sh version)
          echo "derived: $VERSION"
          # Mozilla's published grammar for AMO, transcribed verbatim from
          # MDN's manifest.json/version page:
          #
          #     ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$
          #
          # Not the looser `^[0-9]+(\.[0-9]+)*$` this lane used to carry. That
          # one passes `2026.08.29.0201`, which AMO REJECTS — a segment must be
          # the single digit 0 or start 1-9 — and it also passes five segments,
          # where AMO allows four. Both would surface as a failed sign with the
          # version already burned: AMO 409s on re-signing, so a rejected value
          # cannot be reclaimed and cannot be reused. This lane is the cheap
          # place to find out. (#3138, milestone 318 step 8.)
          if ! echo "$VERSION" | grep -qE '^(0|[1-9][0-9]{0,8})(\.(0|[1-9][0-9]{0,8})){0,3}$'; then
            echo "ERROR: derived version '$VERSION' is not a version AMO accepts."
            echo "AMO's grammar: ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$"
            echo "Most likely cause: a zero-padded segment (08, 0201). The rest"
            echo "of the family pads; the extension must not — see packaging.sh."
            exit 1
          fi
          # ...and the shape this project actually derives. AMO would happily
          # take `1.0.3500147` too, so the grammar check alone would not notice
          # a regression to the pre-318 shape — which orders BELOW everything
          # signed since, and is unrecoverable once Firefox has the higher one.
          if ! echo "$VERSION" | grep -qE '^20[0-9][0-9]\.[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,4}$'; then
            echo "ERROR: derived version '$VERSION' is not YYYY.M.D.HHMM."
            echo "Rule 148's CalVer is what build.yml signs; the old"
            echo "1.0.<minutes> shape would order below every ext-2026.* release."
            exit 1
          fi
          echo "OK: derived version $VERSION"
  backend-lint-and-test:
    runs-on: python-ci
    container:
      image: git.fabledsword.com/bvandeusen/ci-python:3.14
    env:
      # DB_PASSWORD and SECRET_KEY are required by config.py at import time
      # even though unit tests don't actually touch the DB or use the secret.
      DB_PASSWORD: ci_unit_test_placeholder
      SECRET_KEY: ci_unit_test_placeholder
    steps:
      - uses: actions/checkout@v4
        with:
          # Full history for tests/test_artifact_identity.py, which derives
          # each artifact's revision to check the identity scheme. On a
          # depth-1 clone that derivation either fails or returns the tip sha
          # — so the lane would go green while asserting nothing, which is
          # the one outcome worse than a red one.
          fetch-depth: 0
      # Cache step removed 2026-05-26: act_runner's cache backend has been
      # broken on this homelab runner since 2026-05-15 (first as request-
      # timeout warnings, then as hard "Cannot find module .../dist/restore/
      # index.js" failures that tank the whole job). The cache step targeted
      # ~/.cache/pip but the install below uses `uv pip install` primarily,
      # whose own cache lives at ~/.cache/uv — so the cache step's real
      # benefit was marginal even when working. Cost of removal: ~30s of
      # wheel downloads per job. Future re-enable: mount ~/.cache/uv as a
      # docker volume at the runner level (skips actions/cache entirely),
      # or fix the runner-side cache backend (clear /var/run/act/actions/*,
      # pin act_runner version, etc.).
      - name: Install Python deps
        # ruff is pre-installed in the ci-python image (see CI-Runner/CI-python/
        # Dockerfile's RUFF_VERSION). Per FabledRulebook ci-runners.md, toolchain
        # versions live on the runner image, not here.
        # uv: 5-10x faster wheel resolve than pip for cold caches.
        # Falls back to pip install on uv-missing runners (older images).
        run: |
          if command -v uv >/dev/null 2>&1; then
            uv pip install --system -r requirements.txt pytest pytest-asyncio
          else
            pip install -r requirements.txt pytest pytest-asyncio
          fi
      # Ruff moved to the dedicated fast `lint` job above (fails in seconds,
      # no dep install). This job is now unit tests only.
      - name: Pytest (unit only — integration runs in the integration job)
        run: pytest tests/ -v -m "not integration"
  frontend-build:
    runs-on: python-ci
    container:
      image: git.fabledsword.com/bvandeusen/ci-python:3.14
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v4
      # No package-lock.json is tracked yet (we don't run npm locally per
      # feedback-no-local-runs). Using `npm install` instead of `npm ci`.
      # If we want strict lockfile-based reproducibility later, commit a
      # package-lock.json and flip this back to `npm ci`.
      - run: npm install --no-audit --no-fund
      # No type-check step: the frontend is pure JS (no .ts files, no JSDoc),
      # so a type-checker has nothing to do. The vue-tsc devDep + its `check`
      # script were dropped 2026-07-11 rather than bumped to v3. If we add
      # TS/JSDoc later, re-add a tsconfig.json + vue-tsc + a type-check step.
      - run: npm run test:unit
      - run: npm run build
  # Single integration job — collapsed from a 3-way shard split on 2026-06-04.
  # The shards existed to parallelize ~8.5min of integration tests; once the
  # throwaway Postgres runs with fsync OFF (the durability step below) the whole
  # suite runs in ~45s, so the split only triplicated the ~2min fixed overhead
  # (container + `uv pip install` + `alembic upgrade head`) and burned 3 of 6
  # runner slots for no wall-clock gain. One job now: spin up once, install
  # once, migrate once, run every integration test.
  #
  # The docker-ps filter scopes to THIS job's own Postgres/Redis service
  # containers by job name. act_runner strips underscores from job names when
  # labelling containers (`int_api` matched nothing on 2026-05-25), so the name
  # stays separator-free (`integration`). The step prints `docker ps -a` first
  # so a future naming-convention shift surfaces in the log without a
  # guess-and-push cycle.
  #
  # Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT done —
  # per ci-requirements.md, FC is the only Python consumer of that image and the
  # CI-Runner "add deps to image when used by >1 project" rule keeps it per-job.
  integration:
    runs-on: python-ci
    container:
      image: git.fabledsword.com/bvandeusen/ci-python:3.14
    env:
      DB_USER: fabledcurator
      DB_PASSWORD: ci_integration
      DB_PORT: "5432"
      DB_NAME: fabledcurator_test
      SECRET_KEY: ci_integration_placeholder
    services:
      postgres:
        image: pgvector/pgvector:pg16
        env:
          POSTGRES_USER: fabledcurator
          POSTGRES_PASSWORD: ci_integration
          POSTGRES_DB: fabledcurator_test
        options: >-
          --health-cmd "pg_isready -U fabledcurator"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 10
      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 10
    steps:
      - uses: actions/checkout@v4
      - name: Integration suite (resolve service IPs, migrate, test)
        run: |
          set -eux
          echo "=== container landscape (diagnostic for filter scoping) ==="
          docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
          echo "=== end landscape ==="
          PG=$(docker ps --filter "name=integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
          RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:7-alpine" -q | head -n1)
          test -n "$PG" && test -n "$RD"
          PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
          RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
          test -n "$PG_IP" && test -n "$RD_IP"
          export DB_HOST="$PG_IP"
          export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
          export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
          # These steps run under `sh -e`, not bash, so bash's /dev/tcp magic
          # path does not exist here — the probe this loop used to run could
          # never succeed and simply burned the full 120s on every run, green
          # or red, then continued without having established anything. Python
          # is in the image and needs no installed package for a socket
          # connect, so it is the probe. Exhausting the budget is now a named
          # failure rather than a silent fall-through (rule 156): if Postgres
          # is genuinely not up, that is what the log should say, instead of
          # whatever the first query happens to raise two minutes later.
          pg_ready=""
          for i in $(seq 1 60); do
            if python -c "import socket,sys; s=socket.socket(); s.settimeout(2); sys.exit(0 if s.connect_ex(('$PG_IP', 5432)) == 0 else 1)"; then
              pg_ready=1
              break
            fi
            sleep 2
          done
          if [ -z "$pg_ready" ]; then
            echo "postgres at $PG_IP:5432 did not accept a connection within 120s"
            exit 1
          fi
          if command -v uv >/dev/null 2>&1; then
            uv pip install --system -r requirements.txt pytest pytest-asyncio
          else
            pip install -r requirements.txt pytest pytest-asyncio
          fi
          # Relax durability on the throwaway CI Postgres so the per-test
          # TRUNCATE's commit-fsync — the integration teardown's dominant cost
          # (~1.5-2s/test, which collapsed the suite from ~13min to ~45s) — is
          # skipped. fsync/full_page_writes are sighup GUCs and synchronous_commit
          # is user-context, so ALTER SYSTEM + pg_reload_conf() applies them with
          # NO restart. Ephemeral DB ⇒ fsync-off is safe. Non-fatal so a perms
          # surprise can't red the job; fabledcurator is the postgres image's
          # bootstrap superuser.
          python -c "import os,psycopg; c=psycopg.connect(host=os.environ['DB_HOST'],port=5432,user=os.environ['DB_USER'],password=os.environ['DB_PASSWORD'],dbname=os.environ['DB_NAME'],autocommit=True); [c.execute(q) for q in ('ALTER SYSTEM SET fsync=off','ALTER SYSTEM SET synchronous_commit=off','ALTER SYSTEM SET full_page_writes=off','SELECT pg_reload_conf()')]; c.close()" || echo 'WARN: durability GUC relax failed (continuing)'
          alembic upgrade head
          pytest tests/ -v -m integration --durations=15
@@ -1,87 +0,0 @@
 name: extension
 # Lint + unit tests. The sign-and-publish dance moved into build.yml's
 # `sign-extension` job (2026-05-25) — `:latest` now always bundles the XPI
 # because sign-extension runs as a build-web dependency in the SAME workflow,
 # eliminating the prior race between build.yml and a separate extension.yml.
 # Signed XPIs are cached in Forgejo Release Assets named `ext-<version>`.
 on:
  push:
    branches: [dev, main]
    paths:
      - 'extension/**'
      - '.forgejo/workflows/extension.yml'
      # test/version.spec.js asserts things ABOUT the other two workflows —
      # that neither inlines the packaged-file set, and that build.yml derives
      # the shipped version rather than reading it out of the repo. A
      # workflow-only edit can therefore break this suite, so it has to trigger
      # it. build.yml joined the list at milestone 271 step 5, when the spec
      # started asserting against it.
      - '.forgejo/workflows/ci.yml'
      - '.forgejo/workflows/build.yml'
  pull_request:
    branches: [main]
    paths:
      - 'extension/**'
      - '.forgejo/workflows/ci.yml'
      - '.forgejo/workflows/build.yml'
  workflow_dispatch:
 jobs:
  lint:
    runs-on: python-ci
    container:
      image: node:24-bookworm-slim
    steps:
      - uses: actions/checkout@v4
      # Not --no-save: vitest and web-ext are both real devDependencies now,
      # and the suite needs vitest resolvable from node_modules.
      - name: Install dev dependencies
        run: cd extension && npm install --no-audit --no-fund
      - name: Lint
        run: cd extension && npm run lint
      # Pure-logic specs over lib/url.js and lib/platforms.js plus manifest /
      # package version-consistency checks. No browser, no network.
      - name: Unit tests
        run: cd extension && npm run test:unit
      # Everything else about packaging is asserted against our own declaration
      # of what ships. This is the only check that asks web-ext what it ACTUALLY
      # put in the archive. Until now that was an unverified assumption about
      # glob semantics — and a fragile one: `test/**` reaches web-ext intact
      # only because callers `set -f` first, so losing that quoting would
      # silently start shipping dev files with no other signal.
      - name: Verify XPI contents
        run: |
          set -eu
          command -v unzip >/dev/null 2>&1 || { apt-get update -qq && apt-get install -y -qq unzip; }
          cd extension
          npm run build
          ZIP=$(ls web-ext-artifacts/*.zip | head -1)
          echo "=== packaged entries in $ZIP ==="
          unzip -Z1 "$ZIP" | sort
          echo "=== end ==="
          ENTRIES=$(unzip -Z1 "$ZIP")
          fail=0
          # Must NOT ship: repo infrastructure with no business in a user's browser.
          for pat in 'test/' 'scripts/' 'vitest.config.js' 'package.json' 'package-lock.json' 'README.md' 'node_modules/' 'web-ext-artifacts/'; do
            if echo "$ENTRIES" | grep -q "^$pat"; then
              echo "ERROR: '$pat' was packaged into the XPI but must not be"
              fail=1
            fi
          done
          # Must ship: if an exclusion pattern ever over-matches, the extension
          # breaks at runtime rather than at build time, so assert presence too.
          for req in 'manifest.json' 'lib/url.js' 'lib/api.js' 'lib/platforms.js' 'lib/cookies.js'; do
            if ! echo "$ENTRIES" | grep -q "^$req$"; then
              echo "ERROR: '$req' is missing from the XPI"
              fail=1
            fi
          done
          for dir in 'background/' 'popup/' 'options/' 'content/' 'icons/'; do
            if ! echo "$ENTRIES" | grep -q "^$dir"; then
              echo "ERROR: nothing from '$dir' was packaged"
              fail=1
            fi
          done
          [ "$fail" -eq 0 ] || exit 1
          echo "XPI contents verified."
@@ -28,6 +28,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
        postgresql-client \
        zstd \
        megatools \
    # PID 1 for every role. See the ENTRYPOINT note at the foot of this file:
    # without it the image needs `init: true` in whatever runs it, which is a
    # deployment remembering a flag for the image to behave correctly.
        tini \
        libjpeg62-turbo \
        libwebp7 \
        libpng16-16 \
@@ -36,9 +40,59 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 WORKDIR /app
-COPY requirements.txt ./
+COPY requirements.txt requirements-ml.txt ./
 RUN pip install -r requirements.txt
 # --- ML, merged from Dockerfile.ml (milestone 422 step 6) --------------------
 #
 # ONE image now serves every lane. It was two because the ML lane ran in its
 # own container; with the single-container layout (step 5) running every lane
 # in one process tree, a second image would mean the `ml` lane could never be
 # enabled from the UI — there would be no worker in this container to enable.
 #
 # THE COST, MEASURED from run 7273 rather than guessed — and it is far
 # smaller than the estimate this comment first carried, which said "everyone
 # pulls ~4GB":
 #
 #   torch 2.12.1+cpu wheel     192.3 MB
 #   torchvision 0.27.1+cpu       1.8 MB
 #   transformers / onnxruntime / opencv / sklearn and friends (opencv and
 #   onnxruntime since dropped, #1451 — nothing here imported them)
 #                               62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB
 #   largest newly-pushed layer 222.07 MB
 #
 # So the ML code adds a few hundred MB to the pull, not gigabytes. The CPU
 # index is what makes that true: the default PyPI torch wheel bundles the
 # NVIDIA CUDA runtime and is ~2GB on its own.
 #
 # The GIGABYTES are in the MODEL — ~3.5GB of SigLIP weights — and those are
 # NOT in this image. They arrive only when the operator enables the lane,
 # which is what lets rule 164 permit a runtime fetch at all ("optional and
 # clearly off"). That also settles the trade this step was asked to weigh:
 # baking the weights in would add ~3.5GB to every pull for a feature many
 # adopters never enable, against ~350MB for the code that makes the switch
 # available. Off-by-default wins by an order of magnitude, which was NOT
 # obvious before measuring — the estimate had the two costs within 15% of
 # each other.
 #
 # `--index-url`, not `--extra-index-url`: the latter would let pip resolve a
 # +cu wheel anyway, and the whole saving above depends on it not doing that.
 #
 # CPU-only torch from the PyTorch CPU index. Nothing here uses a GPU — the
 # GPU agent is a separate service with its own image.
 RUN pip install --index-url https://download.pytorch.org/whl/cpu \
        "torch>=2.14" "torchvision>=0.29"
 RUN pip install -r requirements-ml.txt
 # Where the model lands. Deliberately NOT a VOLUME instruction: that mints an
 # anonymous volume when nobody mounts one, which survives `docker rm` and
 # accumulates 3.5GB copies nobody can find. The compose files mount it
 # explicitly instead, so an unmounted run simply re-downloads — visible, and
 # recoverable.
 ENV HF_HOME=/models/.huggingface \
    TRANSFORMERS_CACHE=/models/.huggingface \
    ML_MODEL_DIR=/models
 COPY backend/ ./backend/
 COPY alembic/ ./alembic/
 COPY alembic.ini ./
@@ -72,5 +126,51 @@ ENV FC_VERSION=${FC_VERSION}
 EXPOSE 8080
-ENTRYPOINT ["./entrypoint.sh"]
+# ONE healthcheck for every role, because the image knows which role it is
-CMD ["web"]
+# running and a deployment should not have to repeat it. `healthcheck` reads
 # the role entrypoint.sh recorded and asks the right question: HTTP for web,
 # a self-addressed celery ping for a worker lane, both-for-every-lane for the
 # consolidated `all`.
 #
 # start-period covers the SLOWEST role, which is `all`: alembic, then
 # hypercorn, then four celery workers registering with the broker. A web-only
 # container is ready long before this; the cost of the shared number is that
 # a broken one takes a little longer to be called broken.
 #
 # A service may still declare its own healthcheck and docker will prefer it —
 # the escape hatch for a deployment that wants something different.
 HEALTHCHECK --interval=30s --timeout=15s --start-period=90s --retries=3 \
  CMD ["python", "-m", "backend.app.scripts.healthcheck"]
 # tini is PID 1, and the image brings its own rather than asking the
 # deployment for one.
 #
 # PID 1 carries a duty no other process has: every orphaned process in the
 # container reparents to it and must be reaped, or it stays a zombie holding
 # a PID slot. This app makes orphans in normal operation — six service
 # modules shell out (gallery-dl, ffmpeg, pg_dump, the external fetchers) and
 # celery's prefork pool forks children that spawn them.
 #
 # Whatever the role, something that is not an init ends up as PID 1:
 # supervisord for `all`, hypercorn for `web`, celery for a worker. The fix
 # was `init: true` in the compose/stack file, which is out of the norm and
 # put correct process handling in the hands of whoever deploys the image —
 # the same mistake as declaring the healthcheck per service. A flag that is
 # silently dropped (an older Swarm, a `docker run` without it) costs reaping
 # with no signal at all.
 #
 # So the image owns it. `docker run <image>` is correct on its own, and
 # nothing downstream has to know. The smoke asserts /proc/1/comm is tini.
 ENTRYPOINT ["/usr/bin/tini", "--", "./entrypoint.sh"]
 # The DEFAULT is the whole application, not one lane of it.
 #
 # `docker run fabledcurator` with no command starts hypercorn plus every
 # worker lane under supervisord — the shape an adopter wants and the shape the
 # consolidated stack runs. It was `web`, which meant the single-container
 # layout only worked if you knew to ask for it by name, and a compose file
 # that forgot `command:` got a web server with nothing processing its queues:
 # a gallery that loads, accepts an import, and never finishes one.
 #
 # The multi-service stack is unaffected — every service there names its role
 # explicitly, which is exactly what makes it the multi-service stack.
 CMD ["all"]
@@ -1,43 +0,0 @@
 # syntax=docker/dockerfile:1.25
 FROM python:3.14-slim
 ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    HF_HOME=/models/.huggingface \
    TRANSFORMERS_CACHE=/models/.huggingface \
    ML_MODEL_DIR=/models
 RUN apt-get update && apt-get install -y --no-install-recommends \
        ffmpeg \
        libpq5 \
        libjpeg62-turbo \
        libwebp7 \
        libpng16-16 \
        libgl1 \
        libglib2.0-0 \
        ca-certificates \
    && rm -rf /var/lib/apt/lists/*
 WORKDIR /app
 COPY requirements-ml.txt requirements.txt ./
 # CPU-only torch: the default PyPI wheel bundles the CUDA runtime (~5.6GB
 # layer); this pipeline never uses a GPU. --index-url (not --extra-index-url)
 # guarantees only +cpu wheels are considered, so no nvidia-*-cu12 deps.
 RUN pip install --index-url https://download.pytorch.org/whl/cpu \
        "torch>=2.12,<3.0" "torchvision>=0.27,<0.28"
 RUN pip install -r requirements-ml.txt
 COPY backend/ ./backend/
 COPY alembic/ ./alembic/
 COPY alembic.ini ./
 COPY entrypoint.sh ./
 RUN chmod +x entrypoint.sh
 # Models self-heal into /models on first start (FC-2 implements this)
 VOLUME ["/models"]
 ENTRYPOINT ["./entrypoint.sh"]
 CMD ["ml-worker"]
@@ -245,9 +245,9 @@ reasoning is note #3127 §5). Rolling back is `docker pull …:c-<sha>`.
 Each artifact still has a version, derived rather than chosen: the commit time
 of the newest change to that artifact's *own* shipped files, as
-`YYYY.MM.DD.HHMM` UTC (rule 148). Four artifacts, four independent versions —
+`YYYY.MM.DD.HHMM` UTC (rule 148). Three artifacts, three independent versions
-a push touching only `agent/` re-versions the agent and leaves web and ml
+— a push touching only `agent/` re-versions the agent and leaves web and the
-alone, and CI skips the builds whose content did not move.
+extension alone, and CI skips the builds whose content did not move.
 Because no registry name carries it, the running instance's own report is the
 only answer to "which build is this?". The foot of Settings shows
@@ -260,22 +260,32 @@ commits since the previous tag; it builds no image.
 ## What's in here
-Five deployable pieces, built by `.forgejo/workflows/build.yml`:
+Four deployable pieces, built by `.forgejo/workflows/build.yml`:
 | Piece | Built from | Image | Role |
 | --- | --- | --- | --- |
-| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`. The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
+| **Web / workers** | `Dockerfile` | `fabledcurator` | Quart API + the built Vue SPA in one image. `entrypoint.sh` picks the role: `web`, `worker`, `scheduler`, `ml-worker`, or `all` (every lane under supervisord, the single-container layout). The `maintenance-long` service is a second `worker` pinned to the long-running maintenance queue. |
 | **ML worker** | `Dockerfile.ml` | `fabledcurator-ml` | Same app, plus `requirements-ml.txt` — tagging and embedding models that run in-container. |
 | **GPU agent** | `agent/Dockerfile` | `fabledcurator-agent` | Optional desktop-GPU worker (`agent/`). Leases jobs over **HTTP only** — never touches the database or Redis. See `agent/README.md`. |
 | **Firefox extension** | `extension/` | signed XPI | MV3 extension: pushes platform session cookies into FC and adds a creator as a Source in one click. AMO-signed on both `dev` and `main` (one signature per extension change, shared by the two channels), bundled into that channel's web image and served from Settings → Maintenance. See `extension/README.md`. |
 | **Data** | — | `pgvector/pgvector:pg16`, `redis:7-alpine` | Postgres with pgvector for embeddings; Redis as the Celery broker. |
 ## CI / Forgejo setup
-Four workflows: `ci.yml` (lint, extension-version check, backend unit tests,
+Two workflows that matter here: `build.yml` (the six verification lanes — lint,
-frontend build, integration), `extension.yml` (extension lint, vitest, XPI
+extension-version check, backend unit tests, frontend build, extension lint +
-content verification), `build.yml` (sign + publish), and `release.yml`, which
+vitest + XPI content check, integration — and then sign + publish), and
-runs only on a `v*` tag and publishes a changelog without building anything.
+`release.yml`, which runs only on a `v*` tag and publishes a changelog without
 building anything. The extension lane was its own `extension.yml` until
 milestone 429, which let a red extension suite sign and ship the XPI anyway.
 **The lanes and the publish are one workflow on purpose.** They were two
 (`ci.yml` and `build.yml`) until 2026-09-23, on the same push trigger, which
 meant the build could not see the tests' verdict and published whatever it
 built — a red unit lane and a fresh `:dev` image, in the same minute. A
 `needs:` edge only exists inside one workflow graph, so the two are one graph
 and the gate is that edge: a lane that fails, **or that merely skips**, leaves
 the publishing jobs unrun. Pull-request runs (Renovate bumps into `dev`) are
 the lanes and nothing else.
 **The toolchain each job runs in is its `container.image`, not its `runs-on`
 label.** `runs-on: python-ci` only schedules the job onto a runner; every job
@@ -1,10 +1,21 @@
 # FabledCurator GPU agent — runs on the desktop with the GPU.
-# CUDA 12.9 + cuDNN 9 runtime so onnxruntime-gpu can use the card (it needs
+#
-# cuDNN 9 — the plain -runtime image lacks it: "libcudnn.so.9: cannot open
+# The `base` flavour, not `cudnn-runtime`: CUDA and cuDNN arrive as the
-# shared object file"); ffmpeg for video frames. Ubuntu 24.04 → Python 3.12.
+# `nvidia-*` pip packages torch and onnxruntime-gpu depend on, so the base only
-# Stays on the CUDA-12 / cuDNN-9 line the default onnxruntime-gpu + torch are
+# has to hand the container the driver (it sets NVIDIA_VISIBLE_DEVICES /
-# built against (CUDA 13 has only nascent ONNX Runtime support).
+# NVIDIA_DRIVER_CAPABILITIES for the Container Toolkit). Until #1451 this was
-FROM nvidia/cuda:12.9.2-cudnn-runtime-ubuntu24.04
+# `12.9.2-cudnn-runtime` under a `torch==2.6.0+cu124` — and requirements.txt then
 # REPLACED that torch with PyPI's CUDA-13 build (ultralytics pulls torchvision,
 # which pulls its matching torch), beside a CUDA-13 onnxruntime-gpu. The image
 # ran CUDA 13 on a CUDA-12 base, carrying ~3 GB of base libraries and a ~3 GB
 # torch nothing loaded: 10 GB compressed.
 #
 # 13.0 because that is the line both wheels are built for (torch's cu130 index,
 # onnxruntime-gpu's `nvidia-cuda-runtime~=13.0`). Needs an NVIDIA driver that
 # supports CUDA 13 (580+); fc_agent/accel.py logs at startup whether torch and
 # onnxruntime actually got the GPU, since both fall back to the CPU silently.
 # ffmpeg for video frames. Ubuntu 24.04 → Python 3.12.
 FROM nvidia/cuda:13.0.3-base-ubuntu24.04
 # PIP_BREAK_SYSTEM_PACKAGES: Ubuntu 24.04 marks its system Python as externally
 # managed (PEP 668), so a global `pip install` errors without this. It's a
@@ -16,10 +27,12 @@ RUN apt-get update \
    && rm -rf /var/lib/apt/lists/*
 WORKDIR /app
-# torch from the CUDA-12.4 wheel index; its wheels bundle their own CUDA + cuDNN
+# torch AND torchvision from the cu130 index, together and first. Installing
-# so they run on the 12.9 base and coexist with onnxruntime-gpu. Installed first
+# torch alone is what let the next step swap it out: ultralytics needs
-# + separately so the GPU build of torch is deterministic and layer-cached.
+# torchvision, PyPI's torchvision pins its own torch, and pip replaced ours to
-RUN pip3 install --no-cache-dir torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
+# match. With both present, requirements.txt finds them satisfied.
 RUN pip3 install --no-cache-dir --index-url https://download.pytorch.org/whl/cu130 \
        torch torchvision
 COPY requirements.txt .
 RUN pip3 install --no-cache-dir -r requirements.txt
 COPY fc_agent ./fc_agent
@@ -27,6 +40,23 @@ COPY fc_agent ./fc_agent
 # imgutils ONNX models + the transformers SigLIP weights both cache here; mount
 # a volume to persist them across restarts (the SigLIP download is ~3.5 GB once).
 ENV HF_HOME=/models
 # Declared LAST on purpose, exactly as the web Dockerfile does: an ARG/ENV
 # invalidates every layer below it, and these are the only values that differ
 # between builds of otherwise identical source. Any earlier and the ~6.3 GB
 # CUDA + torch layers could never be shared between the dev and main builds of
 # one commit — which is the cost #3114 measured at 9m26s cold.
 #
 # Three values, never folded together (rule 149) — the NAME a person reads, the
 # CHANNEL it came from, and the REVISION that identifies the content. See
 # fc_agent/build_info.py; CI derives all three from scripts/artifacts.sh.
 ARG FC_CHANNEL=""
 ENV FC_CHANNEL=${FC_CHANNEL}
 ARG FC_VERSION=""
 ENV FC_VERSION=${FC_VERSION}
 ARG FC_REVISION=""
 ENV FC_REVISION=${FC_REVISION}
 EXPOSE 8770
 # The control UI; the worker is started from it (or POST /start).
@@ -15,13 +15,28 @@ sudo pacman -S nvidia-container-toolkit
 sudo nvidia-ctk runtime configure --runtime=docker
 sudo systemctl restart docker
 # verify:
-docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
+docker run --rm --gpus all nvidia/cuda:13.0.3-base-ubuntu24.04 nvidia-smi
 # the header's CUDA version must be 13.0 or later (driver 580+)
 ```
 ### After a driver update: regenerate the CDI spec
 If the agent's first log lines say `accel: torch is NOT on the GPU` or report
 `cudaGetDeviceCount: unknown error (999)` while `nvidia-smi` still works, the
 toolkit's saved device list (`/etc/cdi/nvidia.yaml`) is out of date. The
 `nvidia-uvm` device number changes between driver versions, and a spec
 generated before the update hands the container a device node that no longer
 exists (2026-09-24: host `511,0`, container `235,0`). Compare
 `ls -l /dev/nvidia-uvm` on the host with the same inside the container, then:
 ```sh
 sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
 # if your toolkit ships it, this keeps it current on every driver update:
 sudo systemctl enable --now nvidia-cdi-refresh.path
 ```
 ## 1. Get a token
 In FC: **Settings → Tagging → GPU agent → Generate token** (or Rotate). Copy it.
-## 2. Pull (CI publishes it alongside the web/ml images)
+## 2. Pull (CI publishes it alongside the web image)
 ```sh
 docker pull git.fabledsword.com/bvandeusen/fabledcurator-agent:latest
 ```
@@ -0,0 +1,124 @@
 """Which accelerator each runtime actually got — reported once, at startup.
 The agent has two GPU runtimes and both fall back to the CPU without raising:
 torch when the driver is too old for its CUDA build, and onnxruntime (the imgutils
 detector + CCIP models) when its CUDA provider cannot load its libraries. A
 fallback shows up only as slower work, and nothing reported it. On 2026-09-24 the
 image turned out to be running a CUDA-13 torch and onnxruntime on a CUDA-12 base
 (#1451), and whether the ONNX half was on the GPU could not be answered from
 anything the agent had ever logged.
 Also the fix for the likeliest way the ONNX half misses: onnxruntime-gpu's CUDA
 provider finds libcudart/cuBLAS/cuDNN only on the loader path, and in this image
 they live in the `nvidia-*` pip packages torch installs. `preload_dlls()` (ORT
 1.21+) loads them from there, so the provider resolves them by soname.
 Stdlib-only at import, so the unit suite can import it — torch and onnxruntime
 are imported inside the functions.
 """
 from __future__ import annotations
 import ctypes
 import importlib
 import logging
 from pathlib import Path
 log = logging.getLogger("fc_agent.accel")
 # Filled by report(); /status carries it so the page can show it too.
 LAST: dict = {}
 def torch_status(imp=importlib.import_module) -> dict:
    try:
        torch = imp("torch")
    except Exception as e:
        return {"device": "unavailable", "error": str(e)}
    out = {"version": torch.__version__, "cuda_build": torch.version.cuda}
    if torch.cuda.is_available():
        out["device"] = "cuda"
        out["gpu"] = torch.cuda.get_device_name(0)
    else:
        out["device"] = "cpu"
    return out
 def onnx_status(imp=importlib.import_module, load=ctypes.CDLL) -> dict:
    try:
        ort = imp("onnxruntime")
    except Exception as e:
        return {"device": "unavailable", "error": str(e)}
    out = {"version": ort.__version__, "providers": list(ort.get_available_providers())}
    if "CUDAExecutionProvider" not in out["providers"]:
        out["device"] = "cpu"
        return out
    preload = getattr(ort, "preload_dlls", None)
    if preload is not None:
        try:
            preload()
        except Exception as e:
            out["preload_error"] = str(e)
    # "Available" only means the build HAS the provider. Loading its library is
    # what resolves libcudart/cuBLAS/cuDNN — the step that fails when they are
    # missing, and the one a session would otherwise fail silently on.
    capi = Path(ort.__file__).parent / "capi"
    try:
        load(str(capi / "libonnxruntime_providers_shared.so"), mode=ctypes.RTLD_GLOBAL)
        load(str(capi / "libonnxruntime_providers_cuda.so"))
    except OSError as e:
        out["device"] = "cpu"
        out["error"] = str(e)
        return out
    # Loading proves the libraries resolve, NOT that a GPU can be used: on
    # 2026-09-24 this reported "onnx on GPU" beside torch failing cuInit with
    # "CUDA unknown error" (a driver update awaiting a reboot). Asking the CUDA
    # runtime for a device initialises the driver the provider would use.
    error = _cuda_device_error(load)
    out["device"] = "cpu" if error else "cuda"
    if error:
        out["error"] = error
    return out
 def _cuda_device_error(load=ctypes.CDLL) -> str | None:
    """None when the CUDA runtime can reach a device, else why it cannot."""
    try:
        cudart = load("libcudart.so.13")
    except OSError as e:
        return str(e)
    count = ctypes.c_int(0)
    rc = cudart.cudaGetDeviceCount(ctypes.byref(count))
    if rc != 0:
        cudart.cudaGetErrorString.restype = ctypes.c_char_p
        return f"cudaGetDeviceCount: {cudart.cudaGetErrorString(rc).decode()} ({rc})"
    return None if count.value > 0 else "no CUDA device visible"
 def summary() -> dict | None:
    """The report as FabledCurator stores it: each runtime's device, and why
    when it is not the GPU. Sent on every lease and heartbeat, so the System
    view can call a running agent that fell back to the CPU "degraded" rather
    than "running" — the 2026-09-24 fallback went unseen for weeks because
    only this agent's own log said so. None before report() has run."""
    if not LAST:
        return None
    out = {}
    for name, s in LAST.items():
        entry = {"device": s.get("device")}
        if s.get("error"):
            entry["error"] = str(s["error"])[:200]
        out[name] = entry
    return out
 def report() -> dict:
    """Check both runtimes, log the result, and keep it for /status."""
    LAST.clear()
    LAST.update(torch=torch_status(), onnx=onnx_status())
    for name, s in LAST.items():
        if s.get("device") == "cuda":
            log.info("accel: %s on GPU (%s)", name, s)
        else:
            log.warning("accel: %s is NOT on the GPU — work runs on the CPU (%s)", name, s)
    return dict(LAST)
@@ -11,17 +11,22 @@ import logging
 from fastapi import FastAPI, Request
 from fastapi.responses import HTMLResponse, JSONResponse
-from . import logbuf
+from . import accel, logbuf
 from .build_info import FC_CHANNEL, FC_REVISION, FC_VERSION, build_id, display_version
 from .config import Config
 from .gpu import read_gpu
 from .worker import Worker
 log = logging.getLogger("fc_agent.app")
-# Bump on every agent change. The page embeds this and /status reports it; the UI
+# DERIVED at image build time, not hand-maintained — see build_info. This was a
-# warns to reload when they differ — so a stale browser-cached page can't be
+# literal an author was asked to bump on every agent change, and the September
-# mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
+# image printed the same "2026-07-17.1" as the July one, so the surface meant to
-VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle the GPU models release their VRAM and reload on the next job (env IDLE_UNLOAD_SECONDS, 0=off) · sleep mode sheds to one downloader"
+# answer "did my pull work?" answered the same either way.
 #
 # Two values with two jobs, kept apart (rule 149): the page SHOWS the version
 # and COMPARES the build id. /status reports both, plus the raw fields, so a
 # reader never has to take a formatted string apart to get at one of them.
 logbuf.install()
 cfg = Config.from_env()
@@ -42,6 +47,9 @@ async def _no_store(request, call_next):
@app.on_event("startup")
 def _maybe_autostart() -> None:
    # Before the worker: the report also preloads the CUDA libraries the ONNX
    # models need, and it says in the log which runtimes landed on the GPU.
    accel.report()
    # With AUTO_START set, a container restart (host reboot, or `restart:
    # unless-stopped` after a crash) resumes the worker on its own — the slots
    # then ride out a still-down curator via lease backoff. Lets the agent
@@ -52,7 +60,14 @@ def _maybe_autostart() -> None:
@app.get("/", response_class=HTMLResponse)
 def index() -> str:
-    return _PAGE.replace("__BUILD__", VERSION)
+    # Two substitutions, not one: `__VERSION__` is what a person reads in the
    # meta line, `__BUILD_ID__` is what the script compares against /status to
    # notice the page is a cached copy from a previous build.
    return (
        _PAGE
        .replace("__VERSION__", display_version())
        .replace("__BUILD_ID__", build_id())
    )
@app.post("/start")
@@ -117,7 +132,15 @@ def status():
    s["fc_url"] = cfg.fc_url
    s["configured"] = bool(cfg.token)
    s["queue"] = worker.latest_queue()
-    s["build"] = VERSION
+    # `build` is the comparison token the page checks — see build_info.
    # `version`/`channel`/`revision` ride BESIDE it rather than inside it, so a
    # reader wanting the version never has to parse it back out of something
    # else. Absent rather than empty when the image carries no stamp.
    s["build"] = build_id()
    s["version"] = FC_VERSION or None
    s["channel"] = FC_CHANNEL or None
    s["revision"] = FC_REVISION or None
    s["accel"] = accel.LAST or None
    return JSONResponse(s)
@@ -169,7 +192,11 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
  width:30px;height:32px;font:700 16px system-ui;cursor:pointer}
 .step:hover{border-color:var(--acc)}
 #conc,#bw{width:3.4rem;height:32px;text-align:center;font:700 16px system-ui;background:#11151a;
-  color:var(--fg);border:1px solid var(--bd);border-radius:8px}
+  color:var(--fg);border:1px solid var(--bd);border-radius:8px;appearance:textfield;-moz-appearance:textfield}
 /* The browser's own spin arrows, hidden: the − / + beside each field are the
    control, styled like the rest of the page (operator, 2026-09-24). */
 #conc::-webkit-inner-spin-button,#conc::-webkit-outer-spin-button,
 #bw::-webkit-inner-spin-button,#bw::-webkit-outer-spin-button{-webkit-appearance:none;margin:0}
 .unit{color:var(--mut);font-size:12px;font-weight:600}
 .hint{color:var(--mut);font-size:12px;margin-top:12px}
 .tiles{display:grid;grid-template-columns:repeat(6,1fr);gap:8px;margin-bottom:16px}
@@ -203,11 +230,12 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
  <div class=brand><span class=logo>◆</span> FabledCurator <span class=sub>GPU agent</span></div>
  <div class=conn><span class="dot" id=dot></span><span id=connlbl>—</span></div>
 </header>
- <p class=meta>Server <code id=fc>—</code> · token <code id=cfg>—</code> · build <code id=build>__BUILD__</code></p>
+ <p class=meta>Server <code id=fc>—</code> · token <code id=cfg>—</code> · build <code id=build>__VERSION__</code></p>
 <div id=verbanner class=banner style="display:none;background:#3a1212;border-color:#5a1717;color:#ffb3b3">
  a newer agent version is running — reload this page (Ctrl+Shift+R) to update the controls
 </div>
 <div id=accelbanner class=banner style="display:none;background:#3a1212;border-color:#5a1717;color:#ffb3b3"></div>
 <div id=banner class=banner style=display:none>
  curator unreachable — holding work + retrying, resumes on its own (no restart needed)
 </div>
@@ -225,7 +253,9 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
    <button class=step onclick=setc(1)>+</button>
   </div>
   <div class=stepper title="aggregate download cap, downloads + video streams combined — 0 = unlimited">
    <button class=step onclick=stepbw(-1)>−</button>
    <input id=bw type=number min=0 step=1 value=8 onchange="setbw(this.value)">
    <button class=step onclick=stepbw(1)>+</button>
    <span class=unit>MB/s</span>
   </div>
  </div>
@@ -262,7 +292,7 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
 </section>
 </div>
 <script>
- const PAGE_BUILD="__BUILD__"
+ const PAGE_BUILD="__BUILD_ID__"
 let CAP=8
 // Optimistic transitional state on click, then apply the POST's own status
 // response (it returns worker.status()) for instant feedback — don't wait on the
@@ -293,6 +323,14 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
  await fetch('/auto',{method:'POST',headers:{'Content-Type':'application/json'},
   body:JSON.stringify({value:on})});refresh()
 }
 function stepbw(d){ setbw((parseFloat(bw.value)||0)+d) }
 // Runtimes that did NOT get the GPU, from the startup report. Both fall back
 // to the CPU without raising, so this banner and the pill are the only place
 // on this page a slow, CPU-bound agent announces itself.
 function cpuOnly(s){
  const a=s.accel||{}
  return Object.keys(a).filter(k=>a[k] && a[k].device!=='cuda')
 }
 async function setbw(v){
  v=Math.max(0,parseFloat(v)||0); bw.value=v
  await fetch('/bandwidth',{method:'POST',headers:{'Content-Type':'application/json'},
@@ -363,11 +401,17 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
   // unreachable curator; grey when stopped; red with no token.
   let dc='dot', lbl='stopped'
   if(!ok){ dc='dot red'; lbl='no token' }
-   else if(st==='running'){ dc='dot '+(s.queue?'green':'amber'); lbl=s.queue?'running':'running · curator unreachable' }
+   else if(st==='running'){ dc='dot '+(s.queue?'green':'amber'); lbl=s.queue?'running':'running · curator unreachable'
    if(s.queue && cpuOnly(s).length){ dc='dot amber'; lbl='running · CPU only (degraded)' } }
   else if(st==='starting'){ dc='dot amber'; lbl='starting…' }
   else if(st==='stopping'){ dc='dot amber'; lbl='stopping…' }
   dot.className=dc; connlbl.textContent=lbl
   banner.style.display=(st==='running' && !s.queue)?'block':'none'
   const slow=cpuOnly(s)
   accelbanner.style.display=slow.length?'block':'none'
   accelbanner.textContent=slow.length?('degraded — '+slow.join(' + ')+' not on the GPU, so that work runs on the CPU: '
    +slow.map(k=>k+': '+(s.accel[k].error||s.accel[k].device)).join(' · ')
    +'. After a driver update, regenerate the CDI spec (agent README).'):''
   queue.textContent=s.queue?('queue · pending '+s.queue.pending+' · in flight '+s.queue.leased+' · done '+s.queue.done+' · errored '+s.queue.error):'queue · unreachable'
  }
 }
@@ -0,0 +1,79 @@
 """What this agent build IS — stamped at image build time, not configurable.
 The mirror of `backend/app/build_info.py`, for the same reasons and with the
 same posture. Kept as its own module rather than as constants in `app.py`
 because it is stdlib-only and therefore importable by the test suite, which
 cannot import `app` (torch, transformers and ultralytics are not in the CI
 image — see build.yml's "Agent syntax check").
 ## Why this replaced a hand-written string
 `app.VERSION` used to be a literal an author was asked to bump, carrying a
 version AND a changelog in one string:
    VERSION = "2026-07-17.1 · idle model-unload: after ~5 min idle ..."
 Nobody bumped it. The September image printed the identical string to the July
 one, so the one surface that was supposed to answer *"did my pull work?"*
 answered *"2026-07-17.1"* either way. An artifact that cannot identify itself
 is worse than one that says nothing, because the stale value reads as an
 answer.
 The values are now derived by `scripts/artifacts.sh` from the commit its
 shipped files last changed in — the same derivation the web image has used
 since milestone 313, and the same one the reuse check already ran for the
 agent and discarded.
 ## Three values, never folded together (rule 149)
 * `FC_VERSION` — the NAME, `YYYY.MM.DD.HHMM` UTC, derived from COMMIT time.
  For people to read and quote. Identical on `dev` and `main` for the same
  source, which is the property that makes "am I running the same code as
  production?" answerable at a glance.
 * `FC_CHANNEL` — a SIBLING field, never a suffix inside the name.
 * `FC_REVISION` — the 12-char commit sha, the artifact's IDENTITY. This is
  what the reuse check already keys on as the `fc.revision` image label.
 **Absent rather than empty when unknown.** A locally built image has no
 stamp, and neither does any image predating this module. One spelling of
 "cannot say" instead of two.
 """
 import os
 FC_VERSION = os.environ.get("FC_VERSION", "").strip()
 FC_CHANNEL = os.environ.get("FC_CHANNEL", "").strip()
 FC_REVISION = os.environ.get("FC_REVISION", "").strip()
 def display_version() -> str:
    """The build, as a line for a human: `2026.09.24.1052 (dev)`.
    `unknown` rather than a blank when unstamped — an empty slot in the meta
    line reads as "no version", which is a different and false claim from "this
    build does not carry one".
    """
    if not FC_VERSION:
        return "unknown"
    return f"{FC_VERSION} ({FC_CHANNEL})" if FC_CHANNEL else FC_VERSION
 def build_id() -> str:
    """The token the control page compares against `/status` to notice it is
    showing a CACHED page from a previous build.
    Deliberately NOT `display_version()`. That is the value for reading; this
    is the value for deciding, and folding the two is what rule 149 is about.
    The revision is the better discriminator of the two — two builds of the
    same commit ARE the same agent and should not prompt a reload, and two
    different commits always differ here even when they land in the same
    minute and derive one version name.
    A locally built image falls through to a constant, so the reload banner
    cannot fire for it. That is honest rather than a gap: nothing in an
    unstamped image knows what source it was built from, and a per-process
    nonce would make every ordinary container RESTART claim a new version had
    arrived — a false positive on the exact surface the banner exists to keep
    trustworthy.
    """
    return FC_REVISION or FC_VERSION or "local"
@@ -7,6 +7,8 @@ import requests
 from requests.adapters import HTTPAdapter
 from urllib3.util.retry import Retry
 from . import accel
 class FcClient:
    def __init__(self, base_url: str, token: str, agent_id: str):
@@ -72,7 +74,10 @@ class FcClient:
    def lease(self, batch_size: int) -> list[dict]:
        r = self.s.post(
            f"{self.base}/api/gpu/jobs/lease",
-            json={"agent_id": self.agent_id, "batch_size": batch_size},
+            json={
                "agent_id": self.agent_id, "batch_size": batch_size,
                "accel": accel.summary(),
            },
            timeout=30,
        )
        r.raise_for_status()
@@ -90,7 +95,9 @@ class FcClient:
        })
    def heartbeat(self, job_ids: list[int]) -> None:
-        self._post_quiet("/api/gpu/jobs/heartbeat", {"job_ids": job_ids})
+        self._post_quiet(
            "/api/gpu/jobs/heartbeat", {"job_ids": job_ids, "accel": accel.summary()},
        )
    def fail(self, job_id: int, error: str) -> None:
        self._post_quiet("/api/gpu/jobs/fail", {"job_id": job_id, "error": error})
@@ -342,15 +342,44 @@ class Worker:
    # --- background loops ---------------------------------------------------
    def _heartbeat_loop(self) -> None:
-        """Keep every held lease alive so buffered jobs waiting on the GPU aren't
+        """Keep every held lease alive, and say we are here even when holding none.
-        reclaimed by curator's 180s TTL. Errors are swallowed by client.heartbeat;
+
-        a reclaimed lease just re-leases elsewhere — never fatal."""
+        Leases: buffered jobs waiting on the GPU would otherwise be reclaimed by
        curator's 180s TTL. Errors are swallowed by client.heartbeat; a reclaimed
        lease just re-leases elsewhere — never fatal.
        ## Why this sends with an EMPTY list rather than skipping
        Curator's roster records a check-in on this call (and on `lease`), and
        calls an agent stopped after 300s of silence. This loop used to be
        gated on `if ids:` — so an agent holding no leases sent nothing at all,
        and the only check-in left was the lease poll, which sleep mode backs
        off exponentially to a 900s ceiling (see IDLE_POLL_MAX_SECONDS).
        900 against 300: an IDLE agent was structurally guaranteed to read as
        stopped. Operator, 2026-09-23: *"I'm running the gpu agent on my device
        and it currently reads as 'offline' but it's running and has checked in
        recently."* It had — twelve minutes ago, partway up the backoff ladder.
        The two halves were written ten weeks apart and never reconciled: sleep
        mode landed 2026-07-02, and the roster adopted the lease as its
        check-in on 2026-09-02 without noticing the call it was piggybacking on
        had been deliberately slowed.
        An empty heartbeat extends nothing (`id.in_([])` matches no rows) and
        costs one small POST every 45s — against the 6/min lease poll sleep
        mode exists to avoid, that is not a cadence worth protecting, and it is
        what makes "is the agent alive" answerable at all.
        Still gated on `self._running`: a worker that has been stopped is not
        checking in for work, and reporting it as present would be a different
        lie.
        """
        while True:
            if self._running:
                with self._held_lock:
                    ids = list(self._held)
-                if ids:
+                self.client.heartbeat(ids)
                    self.client.heartbeat(ids)
            time.sleep(HEARTBEAT_INTERVAL)
    def _queue_poll_loop(self):
@@ -1,10 +1,12 @@
 # CCIP + figure detection (ONNX models, auto-downloaded from HuggingFace).
 dghs-imgutils>=0.4
 # GPU inference for the ONNX models. Swap to onnxruntime (CPU) for a slow
-# server-side fallback run.
+# server-side fallback run. The extras declare the CUDA/cuDNN pip packages its
-onnxruntime-gpu
+# CUDA provider loads (fc_agent/accel.py preloads them) rather than relying on
-# The crop EMBEDDER (concept bag). torch is installed separately in the
+# torch happening to install the same ones.
-# Dockerfile from the CUDA-12.4 wheel index so the GPU build is deterministic;
+onnxruntime-gpu[cuda,cudnn]
 # The crop EMBEDDER (concept bag). torch + torchvision are installed separately
 # in the Dockerfile from the cu130 wheel index, so pip never swaps them out;
 # transformers loads whatever SigLIP-family model the server announces.
 transformers>=4.45
 # Crop PROPOSERS — small YOLO detectors (booru_yolo anatomy, COCO person, comic
@@ -0,0 +1,90 @@
 """Widen image_record.phash to 256-bit and re-hash the library (issue #4223).
 The operator reported a 15-image variant pack landing as 3 records, and then
 that variants were STILL being dropped with `phash_threshold` at 0. Zero was
 already the floor of the dial, so no setting could have fixed it: at
 `hash_size=8` a pHash is 64 bits of coarse light/dark layout, and variant
 artwork sharing a composition produces the SAME 64 bits. Distance 0 meant
 "identical hash", never "identical image".
 `utils/phash.py` moves to `hash_size=16` (256 bits, what ImageRepo always
 used) and adds an aspect-ratio gate plus a pixel-level confirm, so a merge is
 accepted on the files rather than on the hash.
 ## Why this NULLs every phash
 Widening the column does not correct the values already in it. Every stored
 hash is a 64-bit hash of an image the app will now hash at 256 bits, and the
 two cannot be compared — `find_similar` skips a mismatched-length candidate
 rather than guessing, so leaving them would silently mean "no dedup, forever,
 for everything imported before today". NULL is the state `backfill_phash`
 already knows how to repair: it is NULL-only, keyset-paginated and
 restart-safe, and the beat schedule runs it daily.
 Until that backfill finishes, image dedup degrades to sha256 only —
 duplicates may be kept. That is the safe direction, and the only one
 available: the alternative is comparing hashes of different widths, which
 would drop artwork. NOTHING here deletes or supersedes a file.
 ## Why the threshold is reset rather than carried over
 `phash_threshold` counts bits, and the denominator went from 64 to 256. The
 stored number would keep its value while meaning something four times
 tighter. There is no honest carry-over, so every row goes to the new default
 of 24 — including the operator's 0, which was a workaround for the bug this
 revision fixes.
 Revision ID: 0098
 Revises: 0097
 Create Date: 2026-09-21
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0098"
 down_revision: Union[str, None] = "0097"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    # varchar(32) -> varchar(64): widening a length limit is a catalog-only
    # change in Postgres, so this does not rewrite the table or its index.
    op.alter_column(
        "image_record", "phash",
        existing_type=sa.String(32),
        type_=sa.String(64),
        existing_nullable=True,
    )
    op.execute("UPDATE image_record SET phash = NULL WHERE phash IS NOT NULL")
    op.alter_column(
        "import_settings", "phash_threshold",
        existing_type=sa.Integer(),
        server_default="24",
        existing_nullable=False,
    )
    op.execute("UPDATE import_settings SET phash_threshold = 24")
 def downgrade() -> None:
    # The 64-bit hashes this replaced are gone, and a 64-char value does not
    # fit back into varchar(32) — so the column is cleared again on the way
    # down and left for backfill_phash to refill at whatever HASH_SIZE the
    # code is running. Rule #22: no legacy to preserve.
    op.execute("UPDATE image_record SET phash = NULL WHERE phash IS NOT NULL")
    op.alter_column(
        "image_record", "phash",
        existing_type=sa.String(64),
        type_=sa.String(32),
        existing_nullable=True,
    )
    op.alter_column(
        "import_settings", "phash_threshold",
        existing_type=sa.Integer(),
        server_default="10",
        existing_nullable=False,
    )
    op.execute("UPDATE import_settings SET phash_threshold = 10")
@@ -0,0 +1,99 @@
 """library_placement_run — the placement reconciler's plan/apply/undo ledger.
 Milestone #421 step 3. The survey (#4245) measured 33,789 ImageRecord rows
 sitting outside their artist's canonical directory, across 56 artists. This
 table holds one run of the sweep that trues them up: the plan, what it did,
 and where every file came from.
 ## Why the moves live in a table rather than a log line
 `ImageRecord.path` is the only pointer at the bytes, so a move rewrites the
 row. Once that write lands, the previous location exists nowhere — unless it
 was recorded first. `moves` is that record, which is what makes a 33,789-file
 operation something the operator can undo per artist after looking at the
 result, rather than a one-way door.
 An `applied` row is therefore HISTORY, not state (lesson #4226). Any future
 retention on this table may prune `ready`, `cancelled` and `error` runs; an
 `applied` one is only disposable once someone decides undo is no longer
 wanted. That is deliberately not a timer's decision, and no pruning is added
 here.
 Revision ID: 0099
 Revises: 0098
 Create Date: 2026-09-21
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 from sqlalchemy.dialects import postgresql
 revision: str = "0099"
 down_revision: Union[str, None] = "0098"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    op.create_table(
        "library_placement_run",
        sa.Column("id", sa.Integer(), nullable=False),
        sa.Column(
            "status", sa.String(length=16), server_default="running",
            nullable=False,
        ),
        # SET NULL, not CASCADE: deleting an artist must not destroy the
        # record of where their files were moved.
        sa.Column("artist_id", sa.Integer(), nullable=True),
        sa.Column(
            "started_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
        sa.Column(
            "planned_count", sa.Integer(), server_default="0", nullable=False,
        ),
        sa.Column(
            "moved_count", sa.Integer(), server_default="0", nullable=False,
        ),
        sa.Column(
            "refused_count", sa.Integer(), server_default="0", nullable=False,
        ),
        sa.Column(
            "moves", postgresql.JSONB(astext_type=sa.Text()),
            server_default=sa.text("'[]'::jsonb"), nullable=False,
        ),
        sa.Column(
            "refusals", postgresql.JSONB(astext_type=sa.Text()),
            server_default=sa.text("'[]'::jsonb"), nullable=False,
        ),
        sa.Column("error", sa.Text(), nullable=True),
        sa.ForeignKeyConstraint(
            ["artist_id"], ["artist.id"],
            name="fk_library_placement_run_artist_id", ondelete="SET NULL",
        ),
        sa.PrimaryKeyConstraint("id"),
    )
    op.create_index(
        "ix_library_placement_run_status", "library_placement_run", ["status"],
    )
    op.create_index(
        "ix_library_placement_run_artist_id", "library_placement_run",
        ["artist_id"],
    )
 def downgrade() -> None:
    # Dropping this table destroys the only record of where moved files came
    # from. That is correct for a downgrade — the code that reads it is going
    # away too — but it is worth saying out loud rather than discovering.
    op.drop_index(
        "ix_library_placement_run_artist_id",
        table_name="library_placement_run",
    )
    op.drop_index(
        "ix_library_placement_run_status", table_name="library_placement_run",
    )
    op.drop_table("library_placement_run")
@@ -0,0 +1,109 @@
 """Drop library_placement_run — the placement reconciler is removed.
 Milestone #421 built a sweep that compared each image's `artist_id` to the
 name of the directory its file sat in, and called every mismatch a misplaced
 file. On the operator's library that reported 33,789 of 63,605 images as
 wrongly filed.
 That number was an artefact of the comparison, not a fact about the library:
 - **97.1%** of it was one artist's own folder spelled differently —
  `Telepurte/` versus `telepurte/`. Same artist, same art, nothing wrong.
 - Of the 1% that sat in a differently-named folder, querying `ImageProvenance`
  — which records the post and source each file was actually downloaded from —
  showed 87 where provenance agreed with the FOLDER and not the record, and 40
  genuinely posted by several creators. The sweep would have misfiled or
  arbitrarily picked for roughly 41% of that set.
 The system already knows where every file came from. The reconciler inferred
 it from a column and a directory name instead, and manufactured work out of a
 naming convention. Operator's call, 2026-09-21: *"the current system
 consistently records where items are and where they came from this is just
 complicating something works and doesn't need fixing."*
 Rule #22 — no legacy to preserve. The table goes with the code.
 ## What is deliberately kept
 `utils.paths.canonical_subdir` stays: new filesystem imports derive their
 directory from the artist's slug, matching what the downloader has always
 done. It is not part of this tool and removing it would be churn for no fix.
 Run 1's 327 moved files (`InsoUwu/` -> `insouwu/`) also stay where they are —
 same artist either way, and the gallery renders them correctly.
 Revision ID: 0100
 Revises: 0099
 Create Date: 2026-09-21
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 from sqlalchemy.dialects import postgresql
 revision: str = "0100"
 down_revision: Union[str, None] = "0099"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    op.drop_index(
        "ix_library_placement_run_artist_id",
        table_name="library_placement_run",
    )
    op.drop_index(
        "ix_library_placement_run_status", table_name="library_placement_run",
    )
    op.drop_table("library_placement_run")
 def downgrade() -> None:
    # Recreates the table only. The three runs it held (one applied, two
    # planned-and-never-run) are not restored and are not worth restoring —
    # the code that reads them is gone.
    op.create_table(
        "library_placement_run",
        sa.Column("id", sa.Integer(), nullable=False),
        sa.Column(
            "status", sa.String(length=16), server_default="running",
            nullable=False,
        ),
        sa.Column("artist_id", sa.Integer(), nullable=True),
        sa.Column(
            "started_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
        sa.Column(
            "planned_count", sa.Integer(), server_default="0", nullable=False,
        ),
        sa.Column(
            "moved_count", sa.Integer(), server_default="0", nullable=False,
        ),
        sa.Column(
            "refused_count", sa.Integer(), server_default="0", nullable=False,
        ),
        sa.Column(
            "moves", postgresql.JSONB(astext_type=sa.Text()),
            server_default=sa.text("'[]'::jsonb"), nullable=False,
        ),
        sa.Column(
            "refusals", postgresql.JSONB(astext_type=sa.Text()),
            server_default=sa.text("'[]'::jsonb"), nullable=False,
        ),
        sa.Column("error", sa.Text(), nullable=True),
        sa.ForeignKeyConstraint(
            ["artist_id"], ["artist.id"],
            name="fk_library_placement_run_artist_id", ondelete="SET NULL",
        ),
        sa.PrimaryKeyConstraint("id"),
    )
    op.create_index(
        "ix_library_placement_run_status", "library_placement_run", ["status"],
    )
    op.create_index(
        "ix_library_placement_run_artist_id", "library_placement_run",
        ["artist_id"],
    )
@@ -0,0 +1,66 @@
 """Clear failure state on sources that are disabled (#4279).
 `failing_sources_clause()` now means "enabled AND erroring", so a disabled
 source no longer counts as failing. That fixes what the surfaces REPORT; it
 does not touch what the rows already CARRY, and the rows are the reason the
 operator saw a banner for six days with no way to act on it (lesson #4202 —
 a guard does not undo the value already stored).
 ## The row this exists for
 Ebi77 (source 19): the membership sweep stopped it as `former_patron` at
 02:50 on 2026-09-15 and correctly cleared its failure state. A deep scan was
 armed twenty minutes later — `/backfill` had no `enabled` guard, which this
 release also fixes — and could not complete without access, so the recovery
 sweep stranded it:
    consecutive_failures = 1
    last_error = "stranded by recovery sweep (no terminal status after time_limit)"
 Nothing could clear that. A disabled source is never scheduled, so no
 successful run resets the counter; `SourceService.update` clears failure
 state only on an explicit disable, and the source was already disabled; and
 the card's Retry routes to `/check`, which refuses a disabled source.
 ## Why every disabled source, not just that one
 The clear matches what `SourceService.update` already does when a source is
 disabled through the app — "disable the subs you're not paying for without
 them lingering as failing" — so this brings rows disabled by any OTHER path
 (the membership sweep, a retired platform in 0097) into line with the rows
 disabled by hand. Same shape as 0097: a repair migration reaches the live
 instance on deploy rather than waiting for someone to find the row.
 Enabled sources are untouched — a real failure on a live source must keep
 showing.
 Revision ID: 0101
 Revises: 0100
 Create Date: 2026-09-21
 """
 from typing import Sequence, Union
 from alembic import op
 revision: str = "0101"
 down_revision: Union[str, None] = "0100"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    op.execute(
        "UPDATE source SET last_error = NULL, error_type = NULL, "
        "consecutive_failures = 0 "
        "WHERE NOT enabled "
        "AND (last_error IS NOT NULL OR error_type IS NOT NULL "
        "     OR consecutive_failures <> 0)"
    )
 def downgrade() -> None:
    # Irreversible by design: the cleared strings and counts are not recorded
    # anywhere, and restoring a failure state nobody can act on would only
    # re-create the banner this removes. Rule #22 owes no story backwards.
    pass
@@ -0,0 +1,138 @@
 """Drop pixiv's ledgers, and delete credentials for platforms FC no longer has.
 Milestone #406 step 6 (with issue #3980 folded in). Phase 1 unregistered pixiv
 and the commit alongside this one deleted its client, downloader, ingester and
 models. This removes the data those models described, and the stored secrets of
 every platform that has been retired.
 ## The two ledger tables
 `pixiv_seen_media` and `pixiv_failed_media` are the per-source seen / dead-letter
 ledgers for a downloader that no longer exists. They were created in
 `0089_baseline.py`, so dropping them needs a new revision rather than an edit
 there.
 ## The credentials
 Written as *delete every credential whose platform is not registered* rather
 than as `platform = 'pixiv'`, at the explicit ask in this step's plan. That is
 what makes one migration cover two retirements:
 - **pixiv** — a live OAuth refresh token for a service FC no longer talks to.
 - **deviantart** — issue #3980. #3069 retired DeviantArt in code on 2026-08-27
  and left its stored session behind; seven weeks later it was still there.
 And it is the only way either row can go. The credentials UI
 (`subscriptions/SettingsTab.vue`) renders one card per platform returned by
 `/api/platforms`, then looks the credential up by key — so a row whose platform
 is unregistered has no card, no Remove button, and no way for the operator to
 reach it. `CredentialService.list()` would return it; nothing asks.
 The registered set is written out literally instead of importing
 `known_platform_keys()`. A migration is a statement about one moment in the
 schema's history: if it imported the live registry, retiring a fifth platform
 in 2027 would silently change what this 2026 revision did on a fresh database.
 The list below is the registry as of 2026-09-21.
 ## What is deliberately kept
 **Every pixiv `Source` row.** The original plan deleted them; the operator's
 call on 2026-09-21 was to keep them, and the reason is that `platform` is
 stored ONLY on `Source` — neither `Post` nor `ImageRecord` carries it. Both
 FKs are `ON DELETE SET NULL`, so a delete would not lose the art, but it would
 drop every pixiv image into the gallery's `__unsourced__` bucket and strip the
 platform chip off every pixiv post. The rows stay disabled (0097) and their
 platform is unregistered, so nothing schedules them, nothing downloads through
 them, and `POST /api/sources` will not make another. Keeping them costs
 nothing and keeps the attribution the milestone's goal — *"the art already
 downloaded from pixiv stays"* — is actually about.
 Every pixiv `Post` and `ImageRecord` is likewise untouched.
 Revision ID: 0102
 Revises: 0101
 Create Date: 2026-09-21
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0102"
 down_revision: Union[str, None] = "0101"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 # services/platforms/__init__.py's PLATFORMS as of this revision. See the
 # docstring for why this is a literal and not an import.
 _REGISTERED_PLATFORMS = ("patreon", "subscribestar", "hentaifoundry", "discord")
 def upgrade() -> None:
    op.execute(
        sa.text(
            "DELETE FROM credential WHERE platform NOT IN :registered"
        ).bindparams(
            sa.bindparam("registered", value=_REGISTERED_PLATFORMS, expanding=True)
        )
    )
    op.drop_index("ix_pixiv_failed_media_source_id", table_name="pixiv_failed_media")
    op.drop_table("pixiv_failed_media")
    op.drop_index("ix_pixiv_seen_media_source_id", table_name="pixiv_seen_media")
    op.drop_table("pixiv_seen_media")
 def downgrade() -> None:
    # The tables come back empty, and the credentials do not come back at all:
    # they were encrypted blobs, this migration does not copy them anywhere,
    # and restoring a live token for a platform FC cannot talk to would only
    # re-create the liability. Rule #22 owes no story backwards.
    op.create_table(
        "pixiv_seen_media",
        sa.Column("id", sa.Integer(), nullable=False),
        sa.Column("source_id", sa.Integer(), nullable=False),
        sa.Column("filehash", sa.String(length=128), nullable=False),
        sa.Column("url", sa.Text(), nullable=True),
        sa.Column(
            "created_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.ForeignKeyConstraint(
            ["source_id"], ["source.id"],
            name=op.f("fk_pixiv_seen_media_source_id_source"), ondelete="CASCADE",
        ),
        sa.PrimaryKeyConstraint("id", name=op.f("pk_pixiv_seen_media")),
        sa.UniqueConstraint(
            "source_id", "filehash", name="uq_pixiv_seen_media_source_id",
        ),
    )
    op.create_index(
        op.f("ix_pixiv_seen_media_source_id"), "pixiv_seen_media", ["source_id"],
        unique=False,
    )
    op.create_table(
        "pixiv_failed_media",
        sa.Column("id", sa.Integer(), nullable=False),
        sa.Column("source_id", sa.Integer(), nullable=False),
        sa.Column("filehash", sa.String(length=128), nullable=False),
        sa.Column("url", sa.Text(), nullable=True),
        sa.Column("error", sa.Text(), nullable=True),
        sa.Column("attempts", sa.Integer(), server_default="1", nullable=False),
        sa.Column(
            "created_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.ForeignKeyConstraint(
            ["source_id"], ["source.id"],
            name=op.f("fk_pixiv_failed_media_source_id_source"), ondelete="CASCADE",
        ),
        sa.PrimaryKeyConstraint("id", name=op.f("pk_pixiv_failed_media")),
        sa.UniqueConstraint(
            "source_id", "filehash", name="uq_pixiv_failed_media_source_id",
        ),
    )
    op.create_index(
        op.f("ix_pixiv_failed_media_source_id"), "pixiv_failed_media", ["source_id"],
        unique=False,
    )
@@ -0,0 +1,121 @@
 """worker_lane — settings-backed slots for each celery lane.
 Milestone 422 step 1. One row per lane, holding only what an operator can
 change: how many slots it runs, the ceiling they have set for themselves, and
 whether it consumes its queues at all.
 ## What is deliberately not a column
 **The queues.** They are decided by `celery_app.py`'s `task_routes`, not by
 preference, so a stored copy could contradict the routing table with nothing
 to notice until a queue had no consumer. They live in
 `services/worker_lanes.py`.
 **The derived ceiling.** Computed from the container's cgroup limits on every
 read. A row written on a 32GB host and later run in a 4GB container must be
 bounded by the 4GB; a stored ceiling would quietly authorise what the box can
 no longer hold.
 ## The seeded values
 Written out literally rather than imported from `worker_lanes.LANES`. A
 migration is a statement about one moment in the schema's history — if it
 imported the live defaults, changing them in 2027 would silently change what
 this 2026 revision does on a fresh database. The two are allowed to diverge
 afterwards, and that is correct: `LANES` supplies defaults for a lane added
 later, this file records what was seeded today.
    lane              slots  cap  enabled
    worker                1    4  yes
    scheduler             1    2  yes
    maintenance_long      1    2  yes
    ml                    0    1  NO
 One of each, per the operator (2026-09-22: *"that starting value should be one
 of each"*), and far below their own production numbers — worker 8 and ml 2 are
 tuned for their hardware and are not a sane first boot for a stranger.
 **ml ships at zero and disabled**, which is milestone 422 step 6's requirement
 arriving early: enabling the lane is what triggers the SigLIP download, and
 rule 164 permits a runtime fetch only for a feature that is "optional and
 clearly off". Seeding it on would make every fresh install reach HuggingFace.
 The caps start low on purpose. A cap that begins at the ceiling is a rubber
 stamp; starting at 4/2/2/1 means raising slots within the cap is ordinary and
 raising the cap is a deliberate act.
 ## Existing installs
 Nothing is migrated FROM. The `CELERY_QUEUES` / `CELERY_CONCURRENCY` env vars
 stay exactly as they are and remain the baseline each lane boots at; these
 rows are the adjustment applied on top (step 3). So this migration changes no
 behaviour on a running stack — it only makes the numbers storable.
 Revision ID: 0103
 Revises: 0102
 Create Date: 2026-09-22
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0103"
 down_revision: Union[str, None] = "0102"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 # (name, slots, slots_cap, enabled) — see the docstring for why these are
 # literals and not an import.
 _SEED = (
    ("worker", 1, 4, True),
    ("scheduler", 1, 2, True),
    ("maintenance_long", 1, 2, True),
    ("ml", 0, 1, False),
 )
 def upgrade() -> None:
    worker_lane = op.create_table(
        "worker_lane",
        sa.Column("name", sa.String(length=32), nullable=False),
        sa.Column("slots", sa.Integer(), nullable=False),
        sa.Column("slots_cap", sa.Integer(), nullable=False),
        sa.Column("enabled", sa.Boolean(), nullable=False),
        sa.Column(
            "updated_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.PrimaryKeyConstraint("name", name=op.f("pk_worker_lane")),
        # Bare constraint names: Base.metadata's naming convention prepends
        # ck_worker_lane_, and pre-prefixing doubles it — the defect alembic
        # 0088 had to rename four constraints for (#3275). op.f() marks these
        # as already-final so autogenerate does not propose renaming them.
        sa.CheckConstraint("slots >= 0", name=op.f("ck_worker_lane_slots_non_negative")),
        sa.CheckConstraint("slots_cap >= 0", name=op.f("ck_worker_lane_cap_non_negative")),
        # The invariant that makes the cap mean anything, in the database
        # rather than only in the service: a row violating it is not a
        # rejected request, it is a lane that step 3's reconcile will drive UP
        # to a number the operator capped.
        sa.CheckConstraint("slots <= slots_cap", name=op.f("ck_worker_lane_slots_within_cap")),
    )
    # No index beyond the primary key, deliberately — four rows, forever. Same
    # reasoning as service_seen, and the lesson of #3301, which removed seven
    # indexes that were write cost buying nothing.
    op.bulk_insert(
        worker_lane,
        [
            {"name": name, "slots": slots, "slots_cap": cap, "enabled": enabled}
            for name, slots, cap, enabled in _SEED
        ],
    )
 def downgrade() -> None:
    # The rows go with the table. They are settings with shipped defaults, not
    # operator data that predates this revision — a downgrade returns the stack
    # to reading its concurrency from env, which is where it reads it from
    # today anyway.
    op.drop_table("worker_lane")
@@ -0,0 +1,51 @@
 """worker_lane.autoscale — may this lane grow itself?
 Milestone 422 step 7. One boolean, defaulting FALSE on every existing row and
 on every new one.
 ## Why the default is false and not "sensible"
 This is the only part of the milestone that acts without anyone watching. The
 manual dial (step 4) and the reconcile (step 3) both do exactly what someone
 asked for; this one decides. Shipping it on would mean every install starts
 with a process that changes its own resource usage based on a heuristic tuned
 against nobody's workload.
 Off also makes the failure mode benign: if the signal is wrong, nothing
 happens until an operator opts a lane in, and they opted in while watching.
 ## Why per lane and not one global switch
 The lanes are not alike in what a slot costs. A `worker` slot is a process;
 an `ml` slot is another copy of a ~3.5GB model. A global switch would enable
 growth on a lane whose behaviour under load nobody has observed, and the one
 it would hurt most is the one whose cost is least visible.
 Revision ID: 0104
 Revises: 0103
 Create Date: 2026-09-22
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0104"
 down_revision: Union[str, None] = "0103"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    op.add_column(
        "worker_lane",
        sa.Column(
            "autoscale", sa.Boolean(),
            server_default=sa.text("false"), nullable=False,
        ),
    )
 def downgrade() -> None:
    op.drop_column("worker_lane", "autoscale")
@@ -0,0 +1,133 @@
 """worker_lane — one number: the cap. `slots`, `enabled` and `autoscale` go.
 Milestone 422, reshaped by the operator 2026-09-23:
    "auto should be always on, not a setting, so that idle instances quiet
    down when not running. the number that is visible and something the user
    can tweak and manage should be the cap itself the number of running
    workers is handled by the autoscaling function which is always on."
 ## What each dropped column was, and why it is not needed
 **`slots`** — how many workers the lane should run. That is a MEASUREMENT,
 not a preference: the autoscaler moves the live pool between one and the cap
 according to the backlog, and reads it back from the worker every minute.
 Storing it made it look like something to keep in agreement with the cap,
 which is exactly what the operator had to do.
 **`autoscale`** — whether the lane was allowed to size itself. It gated the
 mechanism behind a per-lane opt-in, so a lane nobody enabled simply never
 gave its slots back. Always on now, which is the only way "idle instances
 quiet down" can be true of an install nobody has configured.
 **`enabled`** — whether the lane consumes its queues. Derived from `cap > 0`.
 It and `slots = 0` were two spellings of one fact and were free to disagree;
 this migration picks the one an operator can see.
 ## Why the caps are rewritten rather than preserved
 The old defaults were 4 / 2 / 2 / 1, chosen when the number meant "the most
 you may raise SLOTS to" — a bound on a manual control, deliberately loose
 because moving within it was the ordinary act. The number now means "the most
 workers this lane may actually use", which is a different promise, and
 carrying the old figure over would silently quadruple the worker lane on
 every existing install at the moment this deploys.
 So every row is reset to the new defaults: **one for each required lane, zero
 for ML.** That loses whatever an operator had set — which is the honest
 trade, because what they set was an answer to a different question. The UI
 now tells a busy lane's operator to raise its cap, which is how the number
 gets back up on an install that needs it.
 ML at zero also keeps rule 164's carve-out intact: no consumers, so no model
 download until someone raises the cap.
 Revision ID: 0105
 Revises: 0104
 Create Date: 2026-09-23
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0105"
 down_revision: Union[str, None] = "0104"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 # (name, cap) — the same values `services/worker_lanes.LANES` declares. Seeded
 # here as literals rather than imported: a migration must describe the schema
 # at ITS point in history, and importing the live table would make this file
 # change meaning every time that table does.
 _CAPS = (
    ("worker", 1),
    ("scheduler", 1),
    ("maintenance_long", 1),
    ("ml", 0),
 )
 def upgrade() -> None:
    # The constraints go first: they name `slots`, so dropping the column out
    # from under them fails on Postgres.
    #
    # `op.f()` around each name, and it is load-bearing. Without it alembic
    # runs the name through Base.metadata's naming convention, which prepends
    # `ck_worker_lane_` to a string that already carries it — and the DROP
    # goes looking for `ck_worker_lane_ck_worker_lane_slots_within_cap`, which
    # no database has. That is #3275 exactly, from the other direction:
    # alembic 0088 had to RENAME four constraints created with the same
    # doubling. Caught here by the integration lane, run 7365.
    op.drop_constraint(
        op.f("ck_worker_lane_slots_within_cap"), "worker_lane", type_="check",
    )
    op.drop_constraint(
        op.f("ck_worker_lane_slots_non_negative"), "worker_lane", type_="check",
    )
    op.drop_column("worker_lane", "slots")
    op.drop_column("worker_lane", "enabled")
    op.drop_column("worker_lane", "autoscale")
    # Reset to the new meaning. See the docstring: the old value answered a
    # different question, and carrying it over would raise every lane.
    for name, cap in _CAPS:
        op.execute(
            sa.text("UPDATE worker_lane SET slots_cap = :cap WHERE name = :name")
            .bindparams(cap=cap, name=name)
        )
    # A lane the old seed never wrote — or one an operator added by hand — is
    # left alone rather than guessed at. `_rows_by_name` creates any missing
    # row at the lane's default on first read.
 def downgrade() -> None:
    op.add_column(
        "worker_lane",
        sa.Column("slots", sa.Integer(), nullable=False, server_default="1"),
    )
    op.add_column(
        "worker_lane",
        sa.Column(
            "enabled", sa.Boolean(), nullable=False, server_default=sa.text("true"),
        ),
    )
    op.add_column(
        "worker_lane",
        sa.Column(
            "autoscale", sa.Boolean(), nullable=False, server_default=sa.text("false"),
        ),
    )
    # Restore the pre-0105 invariants. `slots` comes back as 1 everywhere and
    # the caps are 1/1/1/0, so a lane at cap 0 would violate `slots <= cap` —
    # hence the clamp before the constraint is added.
    op.execute(sa.text("UPDATE worker_lane SET slots = 0 WHERE slots_cap = 0"))
    op.execute(sa.text("UPDATE worker_lane SET enabled = (slots_cap > 0)"))
    op.create_check_constraint(
        op.f("ck_worker_lane_slots_non_negative"), "worker_lane", "slots >= 0",
    )
    op.create_check_constraint(
        op.f("ck_worker_lane_slots_within_cap"), "worker_lane", "slots <= slots_cap",
    )
@@ -0,0 +1,70 @@
 """service_seen — delete the roster rows the fixed code can no longer write.
 Operator, 2026-09-23: *"clean up the stale service_seen rows"*. They were not
 stale. They were PHANTOMS, written on purpose by code that identified a celery
 worker from the queues it was consuming.
 A lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
 with an empty list. The roster grouped on that empty set, wrote it under the
 key `celery:` and rendered `role_display_name(())` as the display name — a row
 called **`Worker ()`**, reported as running, beside the real lane's row going
 stale because nothing updated it any more.
 `worker_lanes.lane_for_node` fixes the cause: a worker is attributed by its
 NODE NAME, which survives having no consumers. Nothing will write `celery:`
 again.
 ## Why a migration and not a retention sweep
 Lesson #4202: a guard that refuses to produce a bad value does not undo the
 bad value already stored. The row is the thing that has to change.
 And it must be deleted rather than aged out, because the roster deliberately
 NEVER forgets — *"anything that has run at least once stays listed, that is
 what lets a stopped one be noticed rather than simply vanishing"*. A row that
 merely goes quiet is exactly what the roster is for. Only a row that cannot
 correspond to anything real is safe to remove, and `celery:` is precisely
 that: the empty queue set, which no correctly-attributed worker can produce.
 ## What is deliberately NOT deleted
 **Celery rows with a real but unmatched queue set.** A deployment slicing
 `CELERY_QUEUES` differently is supported and its rows are true. It is not this
 migration's business to decide that somebody else's worker is obsolete.
 **Agent rows, including a possible `agent:agent` from a build that omitted
 `agent_id`.** Nothing here can tell an abandoned agent id from a second agent
 that is currently down, and deleting a real one would hide a genuinely dead
 GPU agent — the one thing the roster exists to show. If such a row is present
 it needs a person to look at it, not a migration guessing.
 Revision ID: 0106
 Revises: 0105
 Create Date: 2026-09-23
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0106"
 down_revision: Union[str, None] = "0105"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    # Exactly the one key the empty queue set produced. Matched literally
    # rather than by a LIKE or a prefix: `celery:` with nothing after it is
    # the phantom, and `celery:ml` is a real lane.
    op.execute(
        sa.text("DELETE FROM service_seen WHERE key = :key").bindparams(key="celery:")
    )
 def downgrade() -> None:
    # Nothing. The row carried no information — an empty queue set and a
    # timestamp — and the roster re-learns anything real on its next refresh.
    # Re-creating it would put a phantom back.
    pass
@@ -0,0 +1,63 @@
 """worker_lane_sample — where the sizing sweep leaves what it measured.
 Operator, 2026-09-23, on the System tab: *"there is a repull every time this
 page loads — is there a reason this info isn't being tracked in the
 background and stored in some way?"*
 `/api/system/workers` ran a full celery inspect on every call — four
 broadcasts on an eleven-second budget — and the page polls it every fifteen
 seconds. `size_worker_lanes` was already inspecting on a timer to decide pool
 sizes, computing exactly these numbers and discarding them. This table is
 where they land instead, and the endpoint becomes a plain read.
 ## Why a new table rather than columns on `worker_lane`
 `worker_lane` holds the one number an operator sets. Putting a measurement
 beside it is the mistake alembic 0105 undid: `slots` sat next to `slots_cap`,
 and a measurement next to a preference reads as a second preference.
 No backfill. A row appears when the sweep first runs (within its period), and
 until then the lane reads as not-yet-measured, which is true.
 Revision ID: 0107
 Revises: 0106
 Create Date: 2026-09-23
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0107"
 down_revision: Union[str, None] = "0106"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    op.create_table(
        "worker_lane_sample",
        sa.Column("lane", sa.String(length=32), primary_key=True),
        # Nullable=False with no server_default: the sweep writes every column
        # on every upsert, so a row only ever exists complete.
        sa.Column("present", sa.Boolean(), nullable=False),
        sa.Column("replicas", sa.Integer(), nullable=False),
        # Nullable on purpose — unknown, never zero. A worker that answered
        # without reporting its pool, and a queue the broker did not answer
        # for, must not be summed as empty.
        sa.Column("pool", sa.Integer(), nullable=True),
        sa.Column("active", sa.Integer(), nullable=False),
        sa.Column("reserved", sa.Integer(), nullable=False),
        sa.Column("queue_depth", sa.Integer(), nullable=True),
        sa.Column(
            "measured_at",
            sa.DateTime(timezone=True),
            nullable=False,
            server_default=sa.func.now(),
        ),
    )
 def downgrade() -> None:
    op.drop_table("worker_lane_sample")
@@ -0,0 +1,54 @@
 """download_revisit_days — how far back a tick keeps looking for EDITED posts.
 Operator, 2026-09-23, pointing at a Floppystack post: *"this post has been
 updated as he implements hot fixes — any chance we have a way to scan for or
 see updated posts so we can update ours to match and pull the new attachments
 and pictures etc."*
 A tick stopped after 20 contiguous already-have-it items. That is the right
 instinct and the wrong unit: a post edited three days after publication sits
 well below twenty seen items, so the walk turned around before reaching it. The
 walk now needs BOTH a run of seen items and a post older than this many days
 before it stops.
 A settings row rather than a constant (rule 25) because the right window is a
 property of the CREATOR, not of FabledCurator — one artist appends hotfix
 builds for a fortnight, another never touches a post again. 0 turns the revisit
 off entirely and restores the pure count early-out.
 30 days is the operator's own number, 2026-09-23.
 Revision ID: 0108
 Revises: 0107
 Create Date: 2026-09-23
 """
 from typing import Sequence, Union
 import sqlalchemy as sa
 from alembic import op
 revision: str = "0108"
 down_revision: Union[str, None] = "0107"
 branch_labels: Union[str, Sequence[str], None] = None
 depends_on: Union[str, Sequence[str], None] = None
 def upgrade() -> None:
    # server_default so the existing single settings row gets the window without
    # a data migration — and so an install that predates this column reads 30
    # rather than 0. 0 is a real, meaningful value here (revisit off), so the
    # column must never be allowed to arrive at it by omission.
    op.add_column(
        "import_settings",
        sa.Column(
            "download_revisit_days",
            sa.Integer(),
            nullable=False,
            server_default="30",
        ),
    )
 def downgrade() -> None:
    op.drop_column("import_settings", "download_revisit_days")
@@ -0,0 +1,44 @@
 """discord_link_auto — whether FC links a conclusive pair without asking.
 Operator, 2026-09-24: *"I don't want this to be manual that defeats the
 convenience that I'm going for."*
 Confirm-only was right while every signal was circumstantial. Time proximity
 and a body that mentions Discord can never be more than suggestive, so asking
 was the honest response. A shared working name is different in kind: when the
 creator's own name for a piece appears in exactly these two posts and nowhere
 else in their library, there is nothing left for the operator to adjudicate,
 and asking is just a chore FC invented for them.
 Defaults ON, which is a real change of posture and deliberate. It only governs
 the conclusive band — weaker evidence still queues — and a link is a row the
 operator can dismiss, so the reversal is a click rather than a migration.
 Revision ID: 0109
 Revises: 0108
 Create Date: 2026-09-24
 """
 import sqlalchemy as sa
 from alembic import op
 revision = "0109"
 down_revision = "0108"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.add_column(
        "import_settings",
        sa.Column(
            "discord_link_auto",
            sa.Boolean(),
            nullable=False,
            server_default=sa.text("true"),
        ),
    )
 def downgrade():
    op.drop_column("import_settings", "discord_link_auto")
@@ -0,0 +1,59 @@
 """The unified post card — fold window, family window, and who linked a pair.
 Milestone 388, #4402 and #4401. A Patreon teaser's card shows the Discord drop
 it announced, and the rest of that piece's variants, by REFERENCE: nothing is
 absorbed, nothing changes owner, and every Discord post keeps its own place.
 Three columns:
 * `import_settings.discord_link_fold_hours` — a linked drop leaves the feed
  only when it is this close to its teaser (the same release, shown twice).
 * `import_settings.discord_family_window_days` — how far from the teaser the
  card reaches for variants. 60 is measured: named families spread up to 44
  days on artist 8, every collision found over 500.
 * `post_association.linked_by` — "fc" or "operator", so a link FC made by
  itself can say so on the card and offer the undo the operator asked for.
 Revision ID: 0110
 Revises: 0109
 Create Date: 2026-09-24
 """
 import sqlalchemy as sa
 from alembic import op
 revision = "0110"
 down_revision = "0109"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.add_column(
        "import_settings",
        sa.Column(
            "discord_link_fold_hours",
            sa.Float(),
            nullable=False,
            server_default=sa.text("24"),
        ),
    )
    op.add_column(
        "import_settings",
        sa.Column(
            "discord_family_window_days",
            sa.Float(),
            nullable=False,
            server_default=sa.text("60"),
        ),
    )
    op.add_column(
        "post_association",
        sa.Column("linked_by", sa.String(length=16), nullable=True),
    )
 def downgrade():
    op.drop_column("post_association", "linked_by")
    op.drop_column("import_settings", "discord_family_window_days")
    op.drop_column("import_settings", "discord_link_fold_hours")
@@ -0,0 +1,75 @@
 """Discord native ingester ledgers — seen and dead-letter, per source.
 Milestone 428, #4415. Discord moves off gallery-dl onto the native core, which
 keeps its memory of what a source has already fetched in these two tables
 instead of gallery-dl's archive. Same shape as the SubscribeStar pair.
 Revision ID: 0111
 Revises: 0110
 Create Date: 2026-09-24
 """
 import sqlalchemy as sa
 from alembic import op
 revision = "0111"
 down_revision = "0110"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.create_table(
        "discord_seen_media",
        sa.Column("id", sa.Integer(), nullable=False),
        sa.Column("source_id", sa.Integer(), nullable=False),
        sa.Column("filehash", sa.String(length=128), nullable=False),
        sa.Column("post_id", sa.String(length=64), nullable=True),
        sa.Column(
            "seen_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.ForeignKeyConstraint(
            ["source_id"], ["source.id"],
            name=op.f("fk_discord_seen_media_source_id_source"), ondelete="CASCADE",
        ),
        sa.PrimaryKeyConstraint("id", name=op.f("pk_discord_seen_media")),
        sa.UniqueConstraint("source_id", "filehash", name="uq_discord_seen_media_source_id"),
    )
    op.create_index(
        op.f("ix_discord_seen_media_source_id"), "discord_seen_media", ["source_id"],
    )
    op.create_table(
        "discord_failed_media",
        sa.Column("id", sa.Integer(), nullable=False),
        sa.Column("source_id", sa.Integer(), nullable=False),
        sa.Column("filehash", sa.String(length=128), nullable=False),
        sa.Column("attempts", sa.Integer(), server_default="1", nullable=False),
        sa.Column("last_error", sa.Text(), nullable=True),
        sa.Column(
            "first_failed_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.Column(
            "last_failed_at", sa.DateTime(timezone=True),
            server_default=sa.text("now()"), nullable=False,
        ),
        sa.ForeignKeyConstraint(
            ["source_id"], ["source.id"],
            name=op.f("fk_discord_failed_media_source_id_source"), ondelete="CASCADE",
        ),
        sa.PrimaryKeyConstraint("id", name=op.f("pk_discord_failed_media")),
        sa.UniqueConstraint(
            "source_id", "filehash", name="uq_discord_failed_media_source_id",
        ),
    )
    op.create_index(
        op.f("ix_discord_failed_media_source_id"), "discord_failed_media", ["source_id"],
    )
 def downgrade():
    op.drop_index(op.f("ix_discord_failed_media_source_id"), table_name="discord_failed_media")
    op.drop_table("discord_failed_media")
    op.drop_index(op.f("ix_discord_seen_media_source_id"), table_name="discord_seen_media")
    op.drop_table("discord_seen_media")
@@ -41,6 +41,7 @@ def all_blueprints() -> list[Blueprint]:
    from .system_health import system_health_bp
    from .tags import tags_bp
    from .thumbnails import thumbnails_bp
    from .workers import workers_bp
    return [
        api_bp,
        attachments_bp,
@@ -52,6 +53,7 @@ def all_blueprints() -> list[Blueprint]:
        showcase_bp,
        settings_bp,
        system_activity_bp,
        workers_bp,
        system_health_bp,
        system_backup_bp,
        admin_bp,
@@ -475,6 +475,20 @@ async def trigger_reclaim_attachments():
    return _queued(async_result)
@admin_bp.route("/maintenance/repair-discord-downloads", methods=["POST"])
 async def trigger_repair_discord_downloads():
    """Clean re-download of the Discord files broken by the `None` naming
    (#3999). Body {"dry_run": bool}; dry_run is the DEFAULT, because the apply
    deletes files and makes gallery-dl forget every Discord download. Returns the
    Celery task id — poll /maintenance/task-result/<id> for the summary."""
    from ..tasks.admin import repair_discord_downloads_task
    body = await request.get_json(silent=True) or {}
    dry_run = bool(body.get("dry_run", True))
    async_result = repair_discord_downloads_task.delay(dry_run=dry_run)
    return _queued(async_result)
@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
 async def trigger_dedup_videos():
    """Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
@@ -65,6 +65,16 @@ async def autocomplete():
    ])
@artists_bp.route("/names", methods=["GET"])
 async def names():
    """Every artist, id + name + slug, alphabetical. For filter pickers that
    list artists before anything is typed; `autocomplete` deliberately returns
    nothing for an empty query."""
    async with get_session() as session:
        rows = await ArtistService(session).all_names()
    return jsonify([{"id": i, "name": n, "slug": s} for i, n, s in rows])
@artists_bp.route("/directory", methods=["GET"])
 async def directory():
    """FC-3f: cursor-paginated artists directory.
@@ -19,6 +19,7 @@ from ..models import AppSetting
 from ..services.extension_service import (
    ExtensionService,
    InvalidUrlError,
    UnknownArtistError,
    UnknownPlatformError,
 )
 from ..services.source_service import KNOWN_PLATFORMS
@@ -87,10 +88,14 @@ async def probe_source():
    url = (request.args.get("url") or "").strip()
    if not url:
        return _bad("invalid_body", detail="url query parameter is required")
    from .credentials import _get_crypto
    async with get_session() as session:
        if not await _ext_key_required(session):
            return _bad("unauthorized", status=401)
-        result = await ExtensionService(session).probe(url)
+        # crypto lets a Discord probe name the server and channel with the
        # stored token; every other platform ignores it.
        result = await ExtensionService(session, _get_crypto()).probe(url)
    return jsonify(result)
@@ -102,6 +107,15 @@ async def quick_add_source():
    url = body.get("url")
    if not isinstance(url, str) or not url.strip():
        return _bad("invalid_body", detail="url is required")
    # Optional: connect the new source to an existing artist (artist_id) or to
    # the artist of that name (artist_name). A Discord channel names no
    # creator, so the extension's Add panel always sends one of them.
    artist_id = body.get("artist_id")
    if artist_id is not None and (isinstance(artist_id, bool) or not isinstance(artist_id, int)):
        return _bad("invalid_body", detail="artist_id must be an integer")
    artist_name = body.get("artist_name")
    if artist_name is not None and not isinstance(artist_name, str):
        return _bad("invalid_body", detail="artist_name must be a string")
    from .credentials import _get_crypto
@@ -109,9 +123,13 @@ async def quick_add_source():
        if not await _ext_key_required(session):
            return _bad("unauthorized", status=401)
        try:
-            # crypto lets a pixiv add resolve the artist's display name via the
+            # crypto lets an add resolve the artist's display name via the
-            # stored OAuth token (else it falls back to the numeric id). #130.
+            # stored credential (else it falls back to the URL handle). #130.
-            result = await ExtensionService(session, _get_crypto()).quick_add_source(url)
+            result = await ExtensionService(session, _get_crypto()).quick_add_source(
                url, artist_id=artist_id, artist_name=artist_name,
            )
        except UnknownArtistError as exc:
            return _bad("not_found", detail=str(exc), status=404)
        except UnknownPlatformError as exc:
            return _bad(
                "unknown_platform",
@@ -245,6 +245,29 @@ async def errors_recover(image_id: int):
 # --- Agent (bearer token): lease / submit / heartbeat / fail ------------
 def _accel_detail(body: dict) -> dict:
    """The agent's own report of which runtime got the GPU, kept on its roster
    row so the System view can call a CPU-bound agent degraded (#4410).
    Only a dict of {runtime: {device, error?}} is kept, and each value is
    reduced to those two short strings: this is written on every lease, by a
    client the server does not control. An agent that sends nothing (an
    older build) simply has no `accel`, which reads as not-yet-reported.
    """
    raw = body.get("accel")
    if not isinstance(raw, dict):
        return {}
    accel = {}
    for name, entry in list(raw.items())[:4]:
        if not isinstance(entry, dict):
            continue
        clean = {"device": str(entry.get("device") or "")[:16]}
        if entry.get("error"):
            clean["error"] = str(entry["error"])[:200]
        accel[str(name)[:16]] = clean
    return {"accel": accel} if accel else {}
@gpu_bp.route("/jobs/lease", methods=["POST"])
 async def lease():
    body = await request.get_json(silent=True) or {}
@@ -267,7 +290,10 @@ async def lease():
            key=f"agent:{agent_id}",
            kind="agent",
            display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})",
-            details={"agent_id": agent_id, "last_call": "lease", "leased": len(jobs)},
+            details={
                "agent_id": agent_id, "last_call": "lease", "leased": len(jobs),
                **_accel_detail(body),
            },
        )
        ml = await MLSettings.load(session)
        # image rows for url/mime in one shot
@@ -347,7 +373,10 @@ async def heartbeat():
            key=f"agent:{agent_id}",
            kind="agent",
            display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})",
-            details={"agent_id": agent_id, "last_call": "heartbeat", "extended": n},
+            details={
                "agent_id": agent_id, "last_call": "heartbeat", "extended": n,
                **_accel_detail(body),
            },
        )
        await session.commit()
    return jsonify({"extended": n})
@@ -207,6 +207,8 @@ async def rescan_associations():
    be within the window to exist at all); this is the button for a first run
    over a library that predates the feature."""
    async with get_session() as session:
-        result = await association_rescan(session)
+        # full=True: the button reaches the whole history, which the hourly
        # sweep's 48-hour horizon never does.
        result = await association_rescan(session, full=True)
        await session.commit()
    return jsonify(result)
@@ -36,6 +36,7 @@ _EDITABLE_FIELDS = (
    "download_validate_files",
    "download_schedule_default_seconds",
    "download_event_retention_days",
    "download_revisit_days",
    "download_failure_warning_threshold",
    "series_suggest_enabled",
    "series_suggest_threshold",
@@ -43,6 +44,9 @@ _EDITABLE_FIELDS = (
    "discord_link_enabled",
    "discord_link_threshold",
    "discord_link_window_hours",
    "discord_link_auto",
    "discord_link_fold_hours",
    "discord_family_window_days",
    "extdl_mega_enabled",
    "extdl_gdrive_enabled",
    "extdl_mediafire_enabled",
@@ -113,6 +117,12 @@ async def update_import_settings():
        v = body["download_schedule_default_seconds"]
        if not isinstance(v, int) or isinstance(v, bool) or v < 60 or v > 86400:
            return _bad_int("download_schedule_default_seconds", 60, 86400)
    # 0 is a real value (revisit off), so the floor is 0, not 1 — and the
    # ceiling is a year, past which a "tick" is a backfill wearing a hat.
    if "download_revisit_days" in body:
        v = body["download_revisit_days"]
        if not isinstance(v, int) or isinstance(v, bool) or v < 0 or v > 365:
            return _bad_int("download_revisit_days", 0, 365)
    if "download_event_retention_days" in body:
        v = body["download_event_retention_days"]
        if not isinstance(v, int) or isinstance(v, bool) or v < 1 or v > 3650:
@@ -158,6 +168,10 @@ async def update_import_settings():
        body["discord_link_enabled"], bool
    ):
        return jsonify({"error": "discord_link_enabled must be a boolean"}), 400
    if "discord_link_auto" in body and not isinstance(
        body["discord_link_auto"], bool
    ):
        return jsonify({"error": "discord_link_auto must be a boolean"}), 400
    if "discord_link_threshold" in body:
        v = body["discord_link_threshold"]
        if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:
@@ -170,6 +184,12 @@ async def update_import_settings():
            return jsonify(
                {"error": "discord_link_window_hours must be a positive number"}
            ), 400
    # Zero is meaningful for both: fold nothing, or reference no variants.
    for key in ("discord_link_fold_hours", "discord_family_window_days"):
        if key in body:
            v = body[key]
            if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0:
                return jsonify({"error": f"{key} must be a number >= 0"}), 400
    if "wip_title_tagging_enabled" in body and not isinstance(
        body["wip_title_tagging_enabled"], bool
    ):
@@ -201,6 +201,16 @@ async def set_backfill(source_id: int):
            rec = await SourceService(session).get(source_id)
            if rec is None:
                return _bad("not_found", status=404)
            # A disabled source must not be armable for a deep walk — the same
            # rule /check has carried all along (see `source_disabled` below).
            # Arming one anyway is how #4279 happened: the membership sweep had
            # stopped Ebi77 as `former_patron`, a deep scan was armed twenty
            # minutes later, the walk could not complete without access, and
            # the recovery sweep stranded it with a failure count no surface
            # could clear — a disabled source is never scheduled again, and
            # Retry routes to /check, which refuses it.
            if not rec.enabled:
                return _bad("source_disabled", detail="enable the source first")
            native = uses_native_ingester(rec.platform)
            if native:
                cred = CredentialService(session, _get_crypto())
@@ -21,18 +21,22 @@ from ..config import get_config
 from ..extensions import get_session
 from ..models import TaskRun
 from ..services.scheduler_service import scheduler_status
 from ..services.worker_lanes import LANES
 system_activity_bp = Blueprint(
    "system_activity", __name__, url_prefix="/api/system/activity",
 )
-# Canonical queue order — must match celery_app.task_routes. UI renders
+# Every queue, grouped by the lane that consumes it. DERIVED from
-# in this order; queues with no LLEN response show as null rather than
+# `worker_lanes.LANES` (milestone 422 step 1) rather than written out:
-# absent.
+# this was a hand-kept third copy of "which queues exist", alongside
-_QUEUE_NAMES = (
+# celery_app.task_routes and service_roster.ROLE_NAMES, and its own comment
-    "default", "import", "thumbnail", "ml",
+# admitted the coupling — "must match celery_app.task_routes".
-    "download", "scan", "maintenance", "maintenance_long",
+#
-)
+# The rendered ORDER changes with this: lane order rather than the previous
 # hand-chosen one. That is the better grouping for a lane-oriented UI, and
 # queues with no LLEN response still show as null rather than absent.
 _QUEUE_NAMES = tuple(q for lane in LANES for q in lane.queues)
 # Cache module-level so all requests share the cache between polls.
 # Tests can reset via direct dict mutation if needed.
@@ -26,7 +26,6 @@ a true statement.
 from __future__ import annotations
 import asyncio
 import logging
 import time
 from datetime import UTC, datetime
@@ -36,9 +35,7 @@ from sqlalchemy import select, text
 from ..config import get_config
 from ..extensions import get_session
 from ..models import ServiceSeen
-from ..services.service_roster import refresh_if_stale
+from ..services.worker_lanes import SWEEP_PERIOD_SECONDS
 log = logging.getLogger(__name__)
 system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system")
@@ -53,14 +50,36 @@ system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system"
 STALE_AFTER_SECONDS = 90
 DOWN_AFTER_SECONDS = 300
 # The celery roster is written by `size_worker_lanes` and by nothing else, so
 # these thresholds are only meaningful against ITS cadence. Asserted at import
 # rather than left to a reader, because this is precisely the comparison that
 # was never made for the GPU agent: its lease poll backed off to 900s while
 # the roster called it stopped at 300s, and both numbers were individually
 # correct, in different directions, in different files (lesson #4355).
 #
 # Two clear sweeps before a part is even called STALE. One missed tick is
 # routine — the sweep rides the maintenance queue and does an inspect that can
 # take eleven seconds — and must not turn the page yellow.
 _SWEEPS_BEFORE_STALE = 2
 assert STALE_AFTER_SECONDS >= SWEEP_PERIOD_SECONDS * _SWEEPS_BEFORE_STALE, (
    f"a {SWEEP_PERIOD_SECONDS}s sweep cannot keep a roster fresh against a "
    f"{STALE_AFTER_SECONDS}s stale threshold: raise the threshold or shorten "
    f"the sweep"
 )
 # Probes cross a process boundary, so they carry deadlines. A hung Postgres
 # must make this endpoint say "postgres: down", not hang alongside it.
 PROBE_TIMEOUT_SECONDS = 2.0
 _OK, _STALE, _DOWN, _UNKNOWN = "ok", "stale", "down", "unknown"
 # Checking in, but working at a fraction of its speed: a GPU agent whose
 # runtimes fell back to the CPU (#4410). Below stale — a part that may have
 # stopped is the more urgent question — and above unknown, because this one
 # IS known to be wrong.
 _DEGRADED = "degraded"
 # Worst-first, so an overall verdict is just the max.
-_SEVERITY = {_OK: 0, _UNKNOWN: 1, _STALE: 2, _DOWN: 3}
+_SEVERITY = {_OK: 0, _UNKNOWN: 1, _DEGRADED: 2, _STALE: 3, _DOWN: 4}
 def _age_state(age_seconds: float) -> str:
@@ -86,6 +105,39 @@ def _describe_learned(name: str, state: str, age: float, details: dict) -> str:
    return f"{name} has not checked in for {ago} — treat it as stopped"
 def _cpu_runtimes(details: dict) -> list[str]:
    """The runtimes an agent reported as NOT on the GPU, with why.
    Both torch and onnxruntime fall back to the CPU without raising, so an
    agent in that state leases, works and checks in exactly like a healthy
    one. On 2026-09-24 one had been doing so since a driver update left a
    stale CDI spec; the only sign was a line in the agent's own log.
    """
    accel = details.get("accel")
    if not isinstance(accel, dict):
        return []
    out = []
    for name, entry in sorted(accel.items()):
        if not isinstance(entry, dict) or entry.get("device") == "cuda":
            continue
        why = entry.get("error") or entry.get("device") or "unknown"
        out.append(f"{name} ({why})")
    return out
 def _learned_state(name: str, state: str, age: float, details: dict) -> tuple[str, str]:
    """A roster row's state and its sentence, degraded included."""
    if state == _OK:
        cpu = _cpu_runtimes(details)
        if cpu:
            return _DEGRADED, (
                f"{name} is running on the CPU — not on the GPU: {'; '.join(cpu)}. "
                "After a driver update, regenerate the agent host's CDI spec "
                "(agent README)."
            )
    return state, _describe_learned(name, state, age, details)
 async def _probe_postgres(session) -> dict:
    started = time.monotonic()
    try:
@@ -151,26 +203,25 @@ async def system_health():
        parts.append(pg)
        if pg["state"] == _OK:
-            # Rate-limited inside; see service_roster on why the web process
+            # A PURE READ since 2026-09-23. This used to refresh the celery
-            # is the right observer.
+            # roster here, rate-limited to once per 20s — so the roster only
-            try:
+            # advanced while somebody had a browser open, and a broadcast rode
-                await refresh_if_stale(session)
+            # on a request. `size_worker_lanes` writes it now, on a timer, and
-                await session.commit()
+            # the assertion below is what keeps that cadence honest.
            except Exception:  # noqa: BLE001
                log.warning("system health: roster refresh failed", exc_info=True)
            rows = (
                await session.execute(select(ServiceSeen).order_by(ServiceSeen.display_name))
            ).scalars().all()
            for row in rows:
                age = (now - row.last_seen_at).total_seconds()
-                state = _age_state(age)
+                state, detail = _learned_state(
                    row.display_name, _age_state(age), age, row.details or {},
                )
                parts.append({
                    "key": row.key,
                    "kind": row.kind,
                    "name": row.display_name,
                    "state": state,
-                    "detail": _describe_learned(row.display_name, state, age, row.details or {}),
+                    "detail": detail,
                    "last_seen_at": row.last_seen_at.isoformat(),
                    "first_seen_at": row.first_seen_at.isoformat(),
                    **{k: v for k, v in (row.details or {}).items() if k != "agent_id"},
@@ -0,0 +1,170 @@
 """Worker lanes: what each is doing, and the dial that changes it.
 Milestone 422 step 2. The write half of a surface `api/system_activity.py`
 only reads.
 ## Why this is a separate blueprint
 `system_activity` says in its own first line that it is read-only, and it
 answers a different question: its `/workers` is keyed on celery HOSTNAME and
 reports which nodes answered. That stays as it is — the existing
 SystemActivityTab consumes it.
 This is keyed on LANE, joins the stored cap to the live pool, and accepts
 writes. Two endpoints answering "which celery processes exist" and "how much
 work is each lane allowed to do" are not the same endpoint, and folding the
 second into the first would make a read-only module a write one.
 """
 from __future__ import annotations
 from datetime import UTC, datetime
 from functools import partial
 from quart import Blueprint, current_app, jsonify, request
 from ..extensions import get_session
 from ..services.worker_control import (
    LaneUpdateRefused,
    lane_settings,
    lane_view,
    push_lane_cap,
    store_lane_cap,
 )
 from ..services.worker_lanes import (
    LANES_BY_NAME,
    SWEEP_PERIOD_SECONDS,
    Lane,
    derived_ceiling,
 )
 from ._responses import error_response as _bad
 workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers")
@workers_bp.route("", methods=["GET"])
 async def list_lanes():
    """Every lane: its cap, the ceiling above it, and what is live.
    Response: {lanes: [...], fetched_at: iso8601}
    One database read, and NO broker call. Operator, 2026-09-23: *"there is a
    repull every time this page loads — is there a reason this info isn't
    being tracked in the background and stored in some way?"*
    It used to inspect the broker here, four broadcasts on an eleven-second
    budget, four times a minute per open tab — while `size_worker_lanes` was
    already inspecting on a timer and discarding the same numbers. The sweep
    stores them now (`worker_lane_sample`) and this reads them.
    So the live figures are up to `SWEEP_PERIOD_SECONDS` old, and each lane
    carries the `measured_at` that says so. `sweep_period_seconds` is returned
    alongside, so the UI can explain the age without hard-coding the cadence
    in a second place.
    """
    async with get_session() as session:
        settings = await lane_settings(session)
    return jsonify({
        "lanes": lane_view(settings),
        "fetched_at": datetime.now(UTC).isoformat(),
        "sweep_period_seconds": SWEEP_PERIOD_SECONDS,
    })
@workers_bp.route("/<name>", methods=["POST"])
 async def update_lane(name: str):
    """Set a lane's cap. Stores it, answers, and makes the lane follow after.
    ONE field, since 2026-09-23. It used to take `slots`, `slots_cap`,
    `enabled` and `autoscale`; how many workers are running is now a
    measurement the sizing pass owns, and `enabled` is `cap > 0`.
    ## The reply does not wait for the lane
    Operator, 2026-09-23: *"when the number is changed the change should be
    queued so that it isn't blocking of the webui or the system itself. we
    shouldn't have to wait for the validation live."*
    So the request does exactly one thing that can be slow — a row update —
    and hands the broker work to a background task. Turning a lane off is
    four `cancel_consumer` messages and a resize; lowering a cap is an
    `inspect` on an eleven-second budget. Both used to happen between the
    click and the response, with the stepper disabled the whole time.
    Nothing is lost by not waiting: the cap in the database is what the
    system obeys, the sizing pass re-reads it every minute, and the table
    polls, so the live columns catch up on their own. If the web process dies
    before the background task runs, that sweep is the backstop — which is
    the same guarantee the awaited version had, since a push could fail
    there too.
    Refusals still happen inline, because they are decided from the value and
    the machine's ceiling alone and never touch the broker:
    * **400** — the value is not allowed (negative, or above what this
      container can hold). Nothing was stored. The body carries `detail`,
      which is the sentence the UI shows; a refused control with no reason
      reads as a bug.
    """
    lane = LANES_BY_NAME.get(name)
    if lane is None:
        return _bad("unknown_lane", detail=name, known=sorted(LANES_BY_NAME))
    body = await request.get_json()
    if not isinstance(body, dict):
        return _bad("invalid_body", detail="body must be a JSON object")
    if "slots_cap" not in body:
        return _bad("invalid_body", detail="give slots_cap")
    value = body["slots_cap"]
    # Rejected rather than coerced: `True` is an int in Python, and silently
    # reading it as a cap of 1 would be a control that appears to work and
    # sets something nobody asked for.
    if not isinstance(value, int) or isinstance(value, bool):
        return _bad("invalid_body", detail="slots_cap must be an integer")
    # Store, close the session, THEN hand off. The session must not be held
    # across broker work — that is what made this page block the whole site
    # (see `worker_control.LaneSettings`) — and now the request does not wait
    # for that work either.
    async with get_session() as session:
        try:
            was_cap = await store_lane_cap(session, lane, value)
        except LaneUpdateRefused as exc:
            return _bad("refused", detail=str(exc))
    _schedule_push(lane, value, was_cap)
    return jsonify({
        "name": lane.name,
        "slots_cap": value,
        "ceiling": derived_ceiling(lane),
        "enabled": value > 0,
        # The value is stored; the live lane is being told separately. The UI
        # patches its row from this and lets the next poll bring the live
        # columns, rather than refetching and paying for an inspect it just
        # avoided.
        "queued": True,
        # Raising the cap off zero is what downloads the model (step 6), and
        # the background task does it. Reported here so the UI can say a
        # download has started rather than leaving the operator to wonder why
        # a lane they just turned on is busy.
        "fetching_models": value > 0 and was_cap == 0 and bool(lane.models),
    })
 def _schedule_push(lane: Lane, slots_cap: int, was_cap: int) -> None:
    """Run the live push after the response has gone out.
    A seam, not an abstraction: it is one call, and it exists so the tests can
    hold the push still — a background task that outlived a test's patches
    would reach the real broker during teardown.
    Quart tracks the task on the app and awaits it at shutdown, so an
    in-flight push survives a graceful restart. `partial` rather than passing
    `was_cap=` through `add_background_task`, so nothing depends on how that
    forwards keyword arguments.
    """
    current_app.add_background_task(
        partial(push_lane_cap, lane, slots_cap, was_cap=was_cap)
    )
@@ -0,0 +1,77 @@
 """Celery beat that remembers when each job last ran — from task_run, not a file.
 Celery's default PersistentScheduler keeps its memory in a shelve file in the
 working directory. Nothing mounts that directory, so every container recreate
 forgets it, and a scheduler that remembers nothing seeds every entry with
 `last_run_at = now`: each job waits a FULL interval after startup. A daily job
 therefore needs 24 hours without a redeploy to fire. Since the one-container
 image (172e33d) every redeploy restarts beat, and on 2026-09-24 no daily or
 weekly job had run since the 21st (#4408).
 task_run already records every task that starts (celery_signals), indexed on
 (task_name, started_at DESC), and prune_task_runs keeps the newest row of each
 task however old it is. So on startup each entry takes its last_run_at from
 there:
  - a job that is overdue runs at once;
  - a job that is not due waits only the remainder of its interval;
  - a job that has never run is due now.
 Beat keeps last_run_at in memory from then on, as the default scheduler does;
 only the startup seed changes. If the database cannot be read, the entries keep
 Celery's own default rather than beat failing to start.
 """
 from __future__ import annotations
 import logging
 from datetime import UTC, datetime
 from celery.beat import Scheduler
 from sqlalchemy import func, select
 log = logging.getLogger(__name__)
 # Seed for a job with no recorded run: far enough back that any interval or
 # crontab reads as due.
 NEVER = datetime(2000, 1, 1, tzinfo=UTC)
 def last_runs(session, task_names: list[str]) -> dict[str, datetime]:
    """The latest recorded start of each task, by task name."""
    from .models import TaskRun
    if not task_names:
        return {}
    rows = session.execute(
        select(TaskRun.task_name, func.max(TaskRun.started_at))
        .where(TaskRun.task_name.in_(sorted(set(task_names))))
        .group_by(TaskRun.task_name)
    ).all()
    return dict(rows)
 def seed(entries, last: dict[str, datetime]) -> None:
    """Set each entry's last_run_at from `last`; a task with none is due now."""
    for entry in entries:
        entry.last_run_at = last.get(entry.task, NEVER)
 class TaskRunScheduler(Scheduler):
    """An in-memory beat seeded from task_run history at startup."""
    def setup_schedule(self):
        super().setup_schedule()
        try:
            from .tasks._sync_engine import sync_session_factory
            with sync_session_factory()() as session:
                last = last_runs(session, [e.task for e in self.schedule.values()])
        except Exception:
            log.exception("beat: could not read task_run; every job waits a full interval")
            return
        seed(self.schedule.values(), last)
        due = sum(1 for e in self.schedule.values() if e.is_due()[0])
        log.info(
            "beat: seeded %d job(s) from task_run, %d due now",
            len(self.schedule), due,
        )
@@ -14,6 +14,7 @@ Queues:
 from celery import Celery
 from .config import get_config
 from .services.worker_lanes import SWEEP_PERIOD_SECONDS
 def make_celery() -> Celery:
@@ -61,6 +62,13 @@ def make_celery() -> Celery:
            # can never starve the quick self-healing sweeps (operator-flagged
            # 2026-06-07: a 2h audit blocked vacuum/backup/normalize for hours).
            "backend.app.tasks.maintenance.*": {"queue": "maintenance"},
            # The one long job in maintenance.py: a whole-library phash
            # recompute (35 min hard limit; the library was cleared for
            # re-hashing by migration 0098). On the quick lane it held a
            # scheduler process for its whole run, and the minute ticks queued
            # up behind it (2026-09-24: 7 waiting, "all workers busy for 18
            # minutes"). An exact name wins over the glob above.
            "backend.app.tasks.maintenance.backfill_phash": {"queue": "maintenance_long"},
            "backend.app.tasks.backup.*": {"queue": "maintenance_long"},
            "backend.app.tasks.admin.*": {"queue": "maintenance_long"},
            "backend.app.tasks.library_audit.*": {"queue": "maintenance_long"},
@@ -111,6 +119,34 @@ def make_celery() -> Celery:
                "task": "backend.app.tasks.maintenance.recover_interrupted_tasks",
                "schedule": 300.0,  # every 5 minutes
            },
            "size-worker-lanes": {
                "task": "backend.app.tasks.maintenance.size_worker_lanes",
                "schedule": SWEEP_PERIOD_SECONDS,
                #
                # The number lives in `services/worker_lanes` because three
                # places must agree on it: this schedule, the freshness of the
                # sample the System tab reads, and the roster staleness
                # thresholds in `api/system_health` — which now depend on this
                # sweep rather than on a browser being open, and assert their
                # headroom over it at import.
                #
                # ONE entry, replacing `autoscale-worker-lanes` (60s) and
                # `reconcile-worker-lanes` (300s) on 2026-09-23. They were two
                # sweeps over one number and most of the autoscaler's design
                # existed to stop the reconcile undoing its work; with the
                # stored `slots` gone there is nothing to disagree about.
                #
                # Fast enough to react to a BACKLOG — a five-minute reaction to
                # a queue filling up is no reaction. It also carries what the
                # reconcile was for: a worker restarted at its ENV concurrency
                # is corrected on the next tick rather than after five.
                #
                # Cheap when settled: one inspect plus one LLEN sweep, and no
                # control messages at all once every lane matches. It is also
                # now the ONLY thing that inspects — nothing on a request path
                # does — so this is the whole broker cost of the System tab,
                # whether nobody or ten tabs are watching.
            },
            "cleanup-old-tasks": {
                "task": "backend.app.tasks.maintenance.cleanup_old_tasks",
                "schedule": 86400.0,  # daily
@@ -120,6 +156,14 @@ def make_celery() -> Celery:
                "schedule": 86400.0,  # daily — sweep .part/.partial left by a
                # download/import killed mid-write (graceful-shutdown fallout)
            },
            "backfill-phash-daily": {
                "task": "backend.app.tasks.maintenance.backfill_phash",
                "schedule": 86400.0,  # daily — NULL-only, so a no-op once the
                # library is hashed. This is what makes migration 0098's
                # re-hash happen on its own: 0098 NULLs every phash, and
                # without a scheduled refill the library would sit
                # dedup-disabled until someone ran a deep scan (#4223).
            },
            "train-heads-nightly": {
                "task": "backend.app.tasks.ml.scheduled_train_heads",
                "schedule": 86400.0,  # passive cadence; manual retrain stays available
@@ -316,6 +360,9 @@ def make_celery() -> Celery:
            },
        },
        timezone="UTC",
        # Beat's memory of when each job last ran comes from task_run, not a
        # shelve file nothing persists — see beat_scheduler (#4408).
        beat_scheduler="backend.app.beat_scheduler:TaskRunScheduler",
    )
    # FC-3i: register task_run signal handlers (side-effect import).
    from . import celery_signals  # noqa: F401
@@ -8,6 +8,8 @@ from .backup_run import BackupRun
 from .base import Base
 from .character_prototype import CcipPrototypeState, CharacterPrototype
 from .credential import Credential
 from .discord_failed_media import DiscordFailedMedia
 from .discord_seen_media import DiscordSeenMedia
 from .download_event import DownloadEvent
 from .external_link import ExternalLink
 from .gpu_job import GpuJob
@@ -26,8 +28,6 @@ from .membership_sync import MembershipSync
 from .ml_settings import MLSettings
 from .patreon_failed_media import PatreonFailedMedia
 from .patreon_seen_media import PatreonSeenMedia
 from .pixiv_failed_media import PixivFailedMedia
 from .pixiv_seen_media import PixivSeenMedia
 from .platform_membership import PlatformMembership
 from .post import Post
 from .post_association import PostAssociation
@@ -46,6 +46,8 @@ from .tag_head import TagHead
 from .tag_positive_confirmation import TagPositiveConfirmation
 from .tag_suggestion_rejection import TagSuggestionRejection
 from .task_run import TaskRun
 from .worker_lane import WorkerLane
 from .worker_lane_sample import WorkerLaneSample
 __all__ = [
    "Base",
@@ -56,10 +58,10 @@ __all__ = [
    "BackupRun",
    "Source",
    "Credential",
    "DiscordFailedMedia",
    "DiscordSeenMedia",
    "PatreonFailedMedia",
    "PatreonSeenMedia",
    "PixivFailedMedia",
    "PixivSeenMedia",
    "SubscribeStarFailedMedia",
    "SubscribeStarSeenMedia",
    "Post",
@@ -98,4 +100,6 @@ __all__ = [
    "TagPositiveConfirmation",
    "TagSuggestionRejection",
    "TaskRun",
    "WorkerLane",
    "WorkerLaneSample",
 ]
@@ -1,16 +1,9 @@
-"""PixivFailedMedia — per-source dead-letter ledger of Pixiv media that keeps
+"""DiscordFailedMedia — per-source dead-letter ledger of Discord files that
-failing to download/validate.
+keep failing to download or validate.
-Mirror of PatreonFailedMedia/SubscribeStarFailedMedia. Media that fails every
+Mirror of SubscribeStarFailedMedia. After `attempts` reaches the dead-letter
-walk (404'd pximg URL, deleted work, persistently-corrupt bytes) would
+threshold a routine walk skips the file (recovery still retries it); a later
-otherwise re-error forever and re-burn backfill chunks. After ``attempts``
+clean download clears the row. `filehash` is the seen-ledger's key.
 reaches the dead-letter threshold the ingester skips it on routine
 tick/backfill walks (recovery still re-attempts). A later clean download
 clears the row.
 `filehash` is the same synthesized ``<illust_id>:p<num>`` /
 ``<illust_id>:ugoira`` key the seen-ledger uses. UNIQUE (source_id, filehash)
 is the upsert key.
 """
 from datetime import datetime
@@ -22,12 +15,10 @@ from sqlalchemy.types import DateTime
 from .base import Base
-class PixivFailedMedia(Base):
+class DiscordFailedMedia(Base):
-    __tablename__ = "pixiv_failed_media"
+    __tablename__ = "discord_failed_media"
    __table_args__ = (
-        UniqueConstraint(
+        UniqueConstraint("source_id", "filehash", name="uq_discord_failed_media_source_id"),
            "source_id", "filehash", name="uq_pixiv_failed_media_source_id"
        ),
    )
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
@@ -0,0 +1,34 @@
 """DiscordSeenMedia — per-source ledger of Discord files already downloaded.
 Mirror of SubscribeStarSeenMedia for the native Discord ingester (milestone
 428). `filehash` holds the ingester's per-file key, `<message_id>:<media_id>`:
 the attachment id, or for an embed a hash of its URL path. Not the file's
 position in the message — an edit that removes a file renumbers the rest
 (see `discord_client.MediaItem`). The message record's own gate is the
 synthetic `message:<id>` key in the same column.
 """
 from datetime import datetime
 from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
 from sqlalchemy.orm import Mapped, mapped_column
 from sqlalchemy.types import DateTime
 from .base import Base
 class DiscordSeenMedia(Base):
    __tablename__ = "discord_seen_media"
    __table_args__ = (
        UniqueConstraint("source_id", "filehash", name="uq_discord_seen_media_source_id"),
    )
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    source_id: Mapped[int] = mapped_column(
        ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
    )
    filehash: Mapped[str] = mapped_column(String(128), nullable=False)
    post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
    seen_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True), nullable=False, server_default=func.now()
    )
@@ -64,7 +64,11 @@ class ImageRecord(Base):
    # that 0001 also built was an exact duplicate of it — dropped in 0089
    # (#3301). Lookups by sha256 use the constraint's index.
    sha256: Mapped[str] = mapped_column(String(64), nullable=False)
-    phash: Mapped[str | None] = mapped_column(String(32), nullable=True, index=True)
+    # 64 hex chars = the 256-bit hash utils.phash emits at hash_size=16. Was
    # String(32) (64-bit) until migration 0098; the narrow column was the
    # reason for the undersized hash, and the undersized hash was collapsing
    # variant artwork into one record (#4223).
    phash: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
    size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
    mime: Mapped[str] = mapped_column(String(64), nullable=False)
    width: Mapped[int | None] = mapped_column(Integer, nullable=True)
@@ -42,7 +42,12 @@ class ImportSettings(Base):
    single_color_threshold: Mapped[float] = mapped_column(Float, nullable=False, default=0.95, server_default="0.95")
    single_color_tolerance: Mapped[int] = mapped_column(Integer, nullable=False, default=30, server_default="30")
-    phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=10, server_default="10")
+    # Hamming distance over a 256-bit pHash (utils.phash, hash_size=16). The
    # unit CHANGED in migration 0098 — it used to be bits out of 64 — so the
    # old default of 10 is not this scale's 10, and 0098 resets every row.
    # This is now the cheap PRE-FILTER: the aspect + pixel gates decide, which
    # is what lets it be generous enough to catch a re-encoded rescale.
    phash_threshold: Mapped[int] = mapped_column(Integer, nullable=False, default=24, server_default="24")
    # FC-3c downloader knobs
    download_rate_limit_seconds: Mapped[float] = mapped_column(
@@ -63,6 +68,16 @@ class ImportSettings(Base):
        Integer, nullable=False, default=90,
        server_default="90",
    )
    # How far back a routine tick keeps looking after it has run out of new
    # posts, so a creator who EDITS an older post to attach a hotfix build is
    # still reached (ingest_core.DEFAULT_REVISIT_DAYS carries the reasoning).
    # A knob rather than a constant because how long a creator keeps editing is
    # a property of the creator, not of FabledCurator: 0 turns the revisit off
    # and restores the pure count early-out.
    download_revisit_days: Mapped[int] = mapped_column(
        Integer, nullable=False, default=30,
        server_default="30",
    )
    download_failure_warning_threshold: Mapped[int] = mapped_column(
        Integer, nullable=False, default=5,
        server_default="5",
@@ -124,6 +139,50 @@ class ImportSettings(Base):
        server_default="24",
    )
    # Whether FC links a CONCLUSIVE pair without asking.
    #
    # Operator, 2026-09-24: *"I don't want this to be manual that defeats the
    # convenience that I'm going for."* Confirm-only was the right default
    # while the only signals were circumstantial — proximity and a body that
    # mentions Discord can never be more than suggestive, and asking was the
    # honest response to that. A shared working name is different in kind: when
    # the name appears in exactly these two posts and nowhere else in the
    # artist's library, there is nothing for the operator to adjudicate.
    #
    # Only the conclusive band is affected (post_association_service.
    # AUTO_LINK_FLOOR). Everything weaker still queues, and an accepted link is
    # a row the operator can dismiss, so this is reversible in the UI rather
    # than only in the database.
    discord_link_auto: Mapped[bool] = mapped_column(
        Boolean, nullable=False, default=True,
        server_default="true",
    )
    # The unified card (#4402). A linked Discord drop is NOT absorbed into its
    # teaser — it keeps its own post, date and provenance, and the teaser's card
    # shows it by REFERENCE. Operator, 2026-09-24: *"discord 'posts' land as
    # normal and only hidden from the post view they're posted the same day."*
    #
    # So the drop's own card leaves the feed only when it sits within this many
    # hours of the teaser that references it — the adjacency that reads as the
    # same thing twice. Hours rather than a calendar day: a teaser at 23:00 and
    # its drop at 01:00 are one release, and "the same day" has no timezone
    # the server can know.
    discord_link_fold_hours: Mapped[float] = mapped_column(
        Float, nullable=False, default=24.0,
        server_default="24",
    )
    # How far from the teaser the card reaches for the rest of a piece's
    # variants — the wips, alts and censor passes a creator trickles out under
    # one working name (#4401). Measured on artist 8: named families spread a
    # median 5 days and up to 44, while every name collision found spreads
    # over 500. A reference, not a regrouping, so a generous value costs one
    # extra thumbnail at worst — never a post moved or hidden.
    discord_family_window_days: Mapped[float] = mapped_column(
        Float, nullable=False, default=60.0,
        server_default="60",
    )
    # #830 off-platform file-host downloads — per-host enable lever (default on,
    # rule #26). Column names are extdl_<host>_enabled so the worker reads them
    # via getattr(settings, f"extdl_{host}_enabled", True).
@@ -1,42 +0,0 @@
 """PixivSeenMedia — per-source ledger of Pixiv media already
 downloaded+processed.
 Mirror of PatreonSeenMedia/SubscribeStarSeenMedia for the Pixiv native
 ingester (replacing gallery-dl). One queryable row per (source, media) so
 routine walks skip media we've already ingested; recovery mode bypasses the
 ledger to re-walk.
 Pixiv original URLs carry no content hash, so `filehash` is always the
 synthesized ``<illust_id>:p<num>`` (page) / ``<illust_id>:ugoira`` (frame
 zip) key — stable across any URL-shape drift. String(128) matches the sibling
 ledgers.
 """
 from datetime import datetime
 from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
 from sqlalchemy.orm import Mapped, mapped_column
 from sqlalchemy.types import DateTime
 from .base import Base
 class PixivSeenMedia(Base):
    __tablename__ = "pixiv_seen_media"
    __table_args__ = (
        # Dedup key the downloader upserts against: one ledger row per
        # (source, media). A second sighting of the same media is a no-op.
        UniqueConstraint(
            "source_id", "filehash", name="uq_pixiv_seen_media_source_id"
        ),
    )
    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    source_id: Mapped[int] = mapped_column(
        ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
    )
    filehash: Mapped[str] = mapped_column(String(128), nullable=False)
    post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
    seen_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True), nullable=False, server_default=func.now()
    )
@@ -91,6 +91,13 @@ class PostAssociation(Base):
    status: Mapped[str] = mapped_column(
        String(16), nullable=False, server_default="pending", index=True
    )
    # WHO linked it: "fc" when the matcher linked a conclusive pair by itself
    # (discord_link_auto), "operator" when a person accepted it. The card needs
    # this to be honest — a link FC asserted on its own says so and offers an
    # undo, which the operator chose over a silent merge (#4402). NULL on a row
    # that is not linked, and on rows linked before the column existed, all of
    # which an operator accepted: auto-linking shipped in the same release.
    linked_by: Mapped[str | None] = mapped_column(String(16), nullable=True)
    created_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True), nullable=False, server_default=func.now()
    )
@@ -0,0 +1,85 @@
 """worker_lane — the most workers the operator will let each lane use.
 Milestone 422 step 1, reshaped 2026-09-23. One row per lane in
 `services/worker_lanes.LANES`, and ONE COLUMN an operator sets.
 ## Why there is only one number now
 There were three — `slots`, `slots_cap` and `autoscale` — because the manual
 dial was built first and the autoscaler arrived last, beside a control that
 already existed rather than in place of it.
 Operator: *"auto should be always on, not a setting, so that idle instances
 quiet down when not running. the number that is visible and something the
 user can tweak and manage should be the cap itself the number of running
 workers is handled by the autoscaling function which is always on."*
 So `slots` is gone. How many workers a lane is running right now is a
 MEASUREMENT — read live from the worker, moved by the autoscaler, never
 stored. Storing it made it look like a preference, which meant the operator
 had to keep two numbers in agreement and the autoscaler had to be told it was
 allowed to touch one of them.
 `autoscale` is gone for the same reason: it gated the mechanism behind a
 choice, and a lane nobody opted in simply never gave its slots back.
 `enabled` is gone too, and is now DERIVED: a cap of zero means no consumers.
 "Off" and "may use no workers" were two spellings of one fact, free to
 disagree.
 ## What remains
    1  <=  live pool  <=  slots_cap  <=  derived_ceiling
           (autoscaler)   (this row)     (computed)
 The floor is one PROCESS, not zero: billiard will not run an empty pool, and
 the parked process is what `add_consumer` lands on when the cap goes back up.
 The DERIVED CEILING is deliberately absent from this table. It is computed
 from the container's cgroup limits on every read, so a row written on a 32GB
 host and later run in a 4GB container is bounded by the 4GB — a stored
 ceiling would quietly authorise what the box can no longer hold.
 """
 from datetime import datetime
 from sqlalchemy import (
    CheckConstraint,
    DateTime,
    Integer,
    String,
    func,
 )
 from sqlalchemy.orm import Mapped, mapped_column
 from .base import Base
 class WorkerLane(Base):
    __tablename__ = "worker_lane"
    __table_args__ = (
        # Bare name — Base.metadata's naming convention prepends
        # ck_worker_lane_. Pre-prefixing here doubles it, which is what
        # alembic 0088 had to rename four constraints for (#3275).
        CheckConstraint("slots_cap >= 0", name="cap_non_negative"),
    )
    # The lane name from services/worker_lanes.LANES — never a container
    # hostname. See models/service_seen.py for why: celery's worker names here
    # are `celery@<container id>`, minted fresh on every deploy.
    name: Mapped[str] = mapped_column(String(32), primary_key=True)
    # The most workers this lane may use. Zero means off — no consumers, so
    # the lane takes no work and (for ML) downloads no model.
    #
    # There is no upper CHECK here, because the bound it would need is the
    # derived ceiling, and no column holds that: it depends on the cgroup the
    # container is running in right now. Enforced at write instead.
    slots_cap: Mapped[int] = mapped_column(Integer, nullable=False)
    updated_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True),
        nullable=False,
        server_default=func.now(),
        onupdate=func.now(),
    )
@@ -0,0 +1,83 @@
 """worker_lane_sample — the last thing the sizing sweep measured about a lane.
 Milestone 422, 2026-09-23. A MEASUREMENT table, deliberately separate from
 `worker_lane`, which holds the one number an operator sets.
 ## Why this exists
 Operator, 2026-09-23, looking at the System tab: *"there is a repull every
 time this page loads — is there a reason this info isn't being tracked in the
 background and stored in some way?"*
 There was not a good one. `/api/system/workers` ran a full celery inspect —
 four broadcast round trips on an eleven-second budget — on every call, and
 the page polls it every fifteen seconds. Meanwhile `size_worker_lanes` was
 already inspecting on a timer to decide pool sizes, computing exactly these
 numbers, using them, and throwing them away. The browser then asked the
 broker for them again.
 So the sweep writes what it saw here, and the endpoint reads this table. The
 request path makes no broker call at all any more.
 ## Why NOT columns on `worker_lane`
 Because that is the mistake this milestone already made once and undid. That
 table used to carry `slots` — how many workers were running — beside
 `slots_cap`, and a measurement sitting next to a preference reads as a second
 preference: the operator had to keep two numbers in agreement, and the
 autoscaler had to be granted permission to move one of them.
 The distinction is the whole design, so it is a table boundary. Nothing an
 operator sets lives here; nothing here is ever an input to a decision about
 what they wanted.
 ## Freshness is a value, not an assumption
 `measured_at` is returned to the UI, which says how old the reading is rather
 than implying it is live. A sample is a fact about a moment, and a page that
 presents a one-minute-old number as current is how an operator ends up
 mistrusting the whole surface.
 """
 from datetime import datetime
 from sqlalchemy import Boolean, DateTime, Integer, String, func
 from sqlalchemy.orm import Mapped, mapped_column
 from .base import Base
 class WorkerLaneSample(Base):
    __tablename__ = "worker_lane_sample"
    # The lane name from services/worker_lanes.LANES. One row per lane,
    # overwritten in place: this is the LATEST reading, not a history. A time
    # series would be a different table with a different retention problem,
    # and nothing has asked for one.
    lane: Mapped[str] = mapped_column(String(32), primary_key=True)
    # Whether anything answered for this lane. NOT the same as "zero workers"
    # — an unswept absence is not a verdict (snippet #3969). False here means
    # the inspect came back without this lane, so every count below is
    # meaningless and the UI must say "not answering" rather than "0".
    present: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
    replicas: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
    # Pool size of ONE process, nullable because a worker that answered
    # without reporting its pool is unknown rather than empty.
    pool: Mapped[int | None] = mapped_column(Integer, nullable=True)
    active: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
    reserved: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
    # Redis LLEN across the lane's queues. Nullable for the same reason as
    # `pool`: a queue the broker did not answer for is unknown, and summing it
    # as zero would report a buried lane as idle.
    queue_depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
    measured_at: Mapped[datetime] = mapped_column(
        DateTime(timezone=True),
        nullable=False,
        server_default=func.now(),
    )
@@ -0,0 +1,227 @@
 """Emit a supervisord config for the single-container layout.
 Milestone 422 step 5. Writes to stdout; `entrypoint.sh all` redirects it to a
 file and execs supervisord against it.
 ## Why this is generated and not a checked-in .conf
 A static config would spell out each lane's `-Q` list, and that would be a
 FIFTH hand-kept copy of the queue names — after `celery_app.task_routes`, and
 the three collapsed in steps 1, 2 and 4 (`service_roster.ROLE_NAMES`,
 `system_activity._QUEUE_NAMES`, and the Activity filter). Every one of those
 had already drifted by the time it was found.
 Generating from `worker_lanes.LANES` makes a stronger guarantee than "they
 match today": the processes this container runs and the lanes the application
 believes in are the same list, so a lane added to `LANES` gets a process
 without anyone remembering to add one, and a queue can never end up with no
 consumer because a config file was missed.
 ## Why supervisord
 It is one pip dependency on an image that is already Python, and it does the
 four things this needs without being clever: restart a program that exits,
 give each one its OWN stop timeout, signal the process GROUP rather than the
 leader, and put every program's output on one stdout.
 The process-group part is not a detail. Celery's prefork pool forks children,
 and a TERM delivered only to the parent leaves them running — which is how a
 "graceful" shutdown turns into orphaned workers holding tasks. `stopasgroup`
 and `killasgroup` are both set for every program.
 s6-overlay is the other standard answer and would work; it needs a build-time
 download and a second mental model, and its advantage (correct PID-1 signal
 and zombie handling) is available here from `init: true` in compose, which
 puts tini in front of supervisord. Neither choice reaches the application —
 nothing in FC talks to the supervisor — so this is reversible without touching
 a line of product code.
 ## Every lane, including ml
 Step 6 merged the images, so this one carries torch and the ML requirements
 and the `ml` lane gets a program like any other. It starts at one slot with
 its consumers CANCELLED — `enabled=false` in the seeded settings — so it
 holds a process and no model. That matters: `add_consumer` needs a running
 worker to reach, and without one the UI switch would have nothing to switch.
 Nothing is downloaded by starting it. The model fetch is enqueued when the
 lane is enabled, which is what lets rule 164 permit a runtime fetch at all.
 """
 from __future__ import annotations
 import argparse
 import shlex
 import sys
 from ..services.worker_lanes import LANES, MIN_POOL_SLOTS, Lane
 # One number for the whole container, and it must cover the SLOWEST lane —
 # docker gives the container a single stop timeout, where compose today gives
 # each service its own (90/60/180/120s). `maintenance_long` is the 180s one:
 # DB backups, library audits and translation backfill. Anything less turns a
 # routine restart into a SIGKILL mid-backup.
 #
 # Per-program values below are the old per-service ones, preserved: supervisord
 # waits `stopwaitsecs` for each, and they stop in parallel, so the container's
 # own timeout needs to cover the max rather than the sum.
 STOP_WAIT_SECONDS: dict[str, int] = {
    "worker": 90,
    "scheduler": 60,
    "maintenance_long": 180,
    "ml": 120,
 }
 DEFAULT_STOP_WAIT = 60
 def _program(lane: Lane, *, slots: int) -> str:
    """One [program:x] block.
    `stdout_logfile=/dev/fd/1` with maxbytes 0 puts the lane's output straight
    on the container's stdout unbuffered, so `docker logs` shows every lane
    interleaved rather than supervisord swallowing them into rotated files.
    The output is prefixed through `sed` so a line can be attributed to a lane
    — four celery workers and hypercorn on one stream are otherwise
    indistinguishable. The shell that the pipe requires is exactly why
    `stopasgroup` matters: the signal has to reach the celery process, not the
    `sh` holding the pipeline.
    """
    inner = f"./entrypoint.sh {lane.entrypoint_role}"
    prefixed = f"{inner} 2>&1 | sed -u 's/^/[{lane.name}] /'"
    stop_wait = STOP_WAIT_SECONDS.get(lane.name, DEFAULT_STOP_WAIT)
    return "\n".join([
        f"[program:{lane.name}]",
        f"command=sh -c {shlex.quote(prefixed)}",
        # QUOTED, and that is load-bearing. supervisord parses `environment`
        # as a COMMA-separated KEY=VALUE list, so an unquoted queue list reads
        # as CELERY_QUEUES=default followed by three malformed entries — and
        # the lane would consume only its first queue. Silent: the worker
        # starts, reports healthy, and simply never picks up `import`.
        f'environment=CELERY_QUEUES="{",".join(lane.queues)}",'
        f"CELERY_CONCURRENCY={slots},"
        # A UNIQUE celery node name per lane, and the reason is not cosmetic.
        # These processes share one hostname, so celery's default
        # `celery@<hostname>` made all four the SAME node: inspect collapsed
        # their replies, three lanes read as absent, and which three varied
        # per call (run 7319). The healthcheck could never pass, and
        # pool_grow's `destination` would have addressed an arbitrary lane.
        f"CELERY_NODENAME={lane.name}",
        "autostart=true",
        "autorestart=true",
        # A lane that dies instantly and repeatedly is a broken image, not a
        # transient fault. Backing off stops it burning a core in a restart
        # loop while still recovering from a one-off crash.
        "startretries=3",
        "startsecs=5",
        f"stopwaitsecs={stop_wait}",
        "stopasgroup=true",
        "killasgroup=true",
        "stdout_logfile=/dev/fd/1",
        "stdout_logfile_maxbytes=0",
        "redirect_stderr=true",
        "",
    ])
 # supervisord's control socket. /tmp for the same reason the generated config
 # lives there — writable by every role, and per-container by nature.
 SOCKET_PATH = "/tmp/supervisor.sock"
 def _web_program() -> str:
    """hypercorn. Started FIRST (priority) because its role runs
    `alembic upgrade head`, and a worker that boots against an un-migrated
    schema fails in a way that looks like application breakage."""
    prefixed = "./entrypoint.sh web 2>&1 | sed -u 's/^/[web] /'"
    return "\n".join([
        "[program:web]",
        f"command=sh -c {shlex.quote(prefixed)}",
        "priority=1",
        "autostart=true",
        "autorestart=true",
        "startretries=3",
        "startsecs=5",
        # Short: HTTP requests and the occasional file download. Matches the
        # 30s the operator's production stack gives the web service.
        "stopwaitsecs=30",
        "stopasgroup=true",
        "killasgroup=true",
        "stdout_logfile=/dev/fd/1",
        "stdout_logfile_maxbytes=0",
        "redirect_stderr=true",
        "",
    ])
 def render() -> str:
    parts = [
        "\n".join([
            "[supervisord]",
            # PID 1 in the container, so it must not daemonise.
            "nodaemon=true",
            # supervisord's OWN log. /dev/fd/1 keeps it on the container's
            # stdout beside the programs rather than in a file nobody reads.
            "logfile=/dev/fd/1",
            "logfile_maxbytes=0",
            "loglevel=info",
            "",
        ]),
        # THE CONTROL SOCKET, and it is not optional furniture.
        #
        # Without these three sections supervisord runs perfectly and
        # `supervisorctl` cannot talk to it at all:
        #
        #     Error: .ini file does not include supervisorctl section
        #
        # Which is the first thing anyone reaches for when a lane misbehaves
        # in the consolidated container — `docker exec <c> supervisorctl
        # status` to see which processes are up, or `restart ml` to bounce one
        # without taking the whole application down with it. Consolidation
        # took away `docker ps` as the way to see the lanes; this is what
        # replaces it, and shipping without it would have left an operator
        # with one container, five processes inside it, and no way to ask
        # about any of them.
        #
        # Found by the smoke's own diagnostic line on run 7322, which printed
        # this error instead of a process list. It was behind `|| true`, so it
        # cost nothing and said so anyway — the argument for printing evidence
        # even where nothing depends on it.
        #
        # /tmp, like the generated config itself: writable by every role
        # without assuming a volume, and per-container state that must not
        # outlive the container.
        "\n".join([
            "[unix_http_server]",
            f"file={SOCKET_PATH}",
            "chmod=0700",
            "",
            "[rpcinterface:supervisor]",
            "supervisor.rpcinterface_factory = "
            "supervisor.rpcinterface:make_main_rpcinterface",
            "",
            "[supervisorctl]",
            f"serverurl=unix://{SOCKET_PATH}",
            "",
        ]),
        _web_program(),
    ]
    # Lanes after web, in LANES order, so the log reads in a stable sequence.
    for lane in LANES:
        # A lane configured at zero slots still gets a PROCESS, at one slot
        # with its consumers cancelled by the reconcile. Without a running
        # worker there is nothing for `add_consumer` to reach, so enabling the
        # lane from the UI could not work at all — the process has to exist for
        # the switch to have something to switch.
        parts.append(_program(lane, slots=MIN_POOL_SLOTS))
    return "\n".join(parts)
 def main(argv: list[str] | None = None) -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.parse_args(argv)
    sys.stdout.write(render())
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,181 @@
 """The container's healthcheck. Picks the right check from the role it runs.
 Exit 0 healthy, non-zero unhealthy.
 ## Why this is in the IMAGE and not in every compose file
 Because the image is the only thing that knows what it is running. A
 deployment had to declare a healthcheck per service, which meant every
 compose file, stack file and README repeated the same knowledge:
    web         -> curl /api/health
    worker      -> celery inspect ping -d celery@$HOSTNAME
    all         -> both, for every lane
 Three checks, written out by hand, once per service, in every file anyone
 ever wrote — and none of them wrong until a role changed. Operator, 2026-09-23:
 *"why isn't the healthcheck built into the image or base on what command runs
 if one is passed in."* There was no reason. The role is already a fact the
 container holds; asking the deployment to restate it is the same duplication
 the lane table exists to remove one level down.
 So `entrypoint.sh` records the role it started, the Dockerfile declares ONE
 `HEALTHCHECK` that runs this, and a stack file says nothing at all. Declaring
 one anyway still works — docker lets a service override the image's — which
 is the escape hatch for a deployment that genuinely wants something else.
 ## What each role is asked
 * **web** — hypercorn answers `/api/health`. No database: the endpoint is a
  no-DB 200 that proves the app booted and is serving after `alembic upgrade
  head`, which is what a rolling deploy needs to know.
 * **worker / scheduler / ml-worker** — THIS container's celery node answers a
  ping over the broker. Not "some worker answered": the node name is pinned
  to this container, or a healthy sibling would keep a dead one looking alive.
 * **all** — both halves, for every lane in the table. The failure mode
  consolidation creates is that docker can no longer see the lanes as
  separate services, so a web-only check reports a healthy container with
  every worker dead.
 * **shell / alembic / anything else** — nothing to check. These are one-shot
  or interactive; a liveness probe on them has no meaning, so it passes
  rather than inventing a verdict.
 ## An unrecorded role passes rather than failing
 If the role file is missing, the entrypoint did not run — someone used
 `--entrypoint` or ran a bare command. That is a debugging shape, and a
 healthcheck that cannot tell what it is looking at must not assert that the
 thing is broken (snippet #3969: an unswept read is not a verdict). It says so
 on stdout and exits 0.
 """
 from __future__ import annotations
 import os
 import socket
 import sys
 import urllib.error
 import urllib.request
 # Written by entrypoint.sh at boot. /tmp because it is the one path writable
 # by every role without assuming a volume, and the value is per-container
 # state that must NOT survive into a new container.
 ROLE_FILE = os.environ.get("FC_ROLE_FILE", "/tmp/fc-role")
 WEB_URL = "http://localhost:8080/api/health"
 WEB_TIMEOUT = 5.0
 # A broker round trip, so it gets a deadline (rule 156). Generous relative to
 # `inspect`'s 2s elsewhere: this runs every 30s with retries, and a transient
 # blip flagging a worker unhealthy would roll back a deployment that is fine.
 PING_TIMEOUT = 10.0
 CELERY_ROLES = {"worker", "scheduler", "ml-worker"}
 # Roles with nothing to probe. Listed rather than treated as the default, so
 # an unknown role takes the "I cannot tell" path and says so.
 NO_CHECK_ROLES = {"shell", "bash", "alembic"}
 def current_role() -> str | None:
    """The role this container was started with, or None if nothing recorded."""
    env = os.environ.get("FC_ROLE")
    if env:
        return env.strip()
    try:
        with open(ROLE_FILE) as fh:
            return fh.read().strip() or None
    except OSError:
        return None
 def _web_ok() -> tuple[bool, str]:
    try:
        with urllib.request.urlopen(WEB_URL, timeout=WEB_TIMEOUT) as resp:
            if resp.status == 200:
                return True, ""
            return False, f"web returned {resp.status}"
    except (urllib.error.URLError, OSError) as exc:
        return False, f"web unreachable: {exc}"
 def _this_node_ok() -> tuple[bool, str]:
    """Ping THIS container's celery node, by name.
    Pinned to this node deliberately. A bare `ping()` is answered by any
    worker on the broker, so in a stack with several replicas a dead one
    would go on reporting healthy for as long as a sibling was alive — the
    healthcheck would be measuring the cluster, not the container it is in.
    """
    from ..celery_app import celery as celery_app
    node = f"{os.environ.get('CELERY_NODENAME', 'celery')}@{socket.gethostname()}"
    try:
        replies = celery_app.control.ping(destination=[node], timeout=PING_TIMEOUT)
    except Exception as exc:  # noqa: BLE001 — a probe reports, never raises
        return False, f"could not reach the broker: {exc}"
    if not replies:
        return False, f"{node} did not answer a ping"
    return True, ""
 def _lanes_ok() -> tuple[bool, str]:
    """Every lane in the table is answering.
    Deliberately ignores whether a lane is ON: a lane at cap 0 still runs its
    process with its consumers cancelled, so it answers `inspect` and is
    healthy. Health is "is the process alive"; whether it should be consuming
    is a settings question the sizing pass owns, and conflating them would
    make turning a lane off mark the container unhealthy.
    That was not merely a risk — it was happening. Until 2026-09-23 a worker
    was attributed to its lane by the queues it was CONSUMING, and a lane with
    its consumers cancelled reports none, so it read as absent and this check
    failed. ML ships at cap 0, so a fresh install was permanently unhealthy
    and Swarm restarts an unhealthy task forever. The docstring above said the
    right thing while the code did the opposite; `worker_lanes.lane_for_node`
    is what makes it true.
    """
    from ..services.worker_control import inspect_lanes_sync
    from ..services.worker_lanes import LANES
    live = inspect_lanes_sync()
    missing = sorted(lane.name for lane in LANES if not live[lane.name].present)
    if missing:
        return False, "lanes not answering: " + ", ".join(missing)
    return True, ""
 def main(argv: list[str] | None = None) -> int:
    role = current_role()
    if role is None:
        # Not a failure. See the module docstring: the entrypoint did not run,
        # so there is no role to check against and no basis for a verdict.
        print("no role recorded; nothing to check")
        return 0
    if role in NO_CHECK_ROLES:
        print(f"{role}: nothing to check")
        return 0
    checks = []
    if role == "all":
        checks = [_web_ok, _lanes_ok]
    elif role == "web":
        checks = [_web_ok]
    elif role in CELERY_ROLES:
        checks = [_this_node_ok]
    else:
        print(f"unknown role {role!r}; nothing to check")
        return 0
    for check in checks:
        ok, detail = check()
        if not ok:
            print(detail, file=sys.stderr)
            return 1
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,146 @@
 """Block until Postgres and Redis accept connections. Exit 0 ready, 1 timed out.
 ## Why the container has to do this itself
 Compose has `depends_on: {condition: service_healthy}`, and **Swarm ignores
 it**. `docker stack deploy` has no ordering primitive at all: every service in
 the stack starts at once, so FabledCurator races Postgres on every cold
 deploy and always has.
 The multi-service stack hid how sharp that is. `web` ran `alembic upgrade
 head`, failed against a Postgres that was still doing `initdb`, and the task
 died — but Swarm restarts a failed task forever, so the service came up a few
 seconds later and nobody saw a problem worth naming.
 Consolidation removes that safety net. supervisord gives each program
 `startretries=3`, so a web program that fails three times in the first
 seconds goes FATAL and **stays** FATAL: supervisord keeps running, the
 container keeps running, and the application never starts. The healthcheck
 catches it — but as a container that is permanently unhealthy for a reason
 that has nothing to do with the image, on a stack whose database simply took
 twenty seconds to initialise.
 Operator, 2026-09-23: *"it's a single container that need to connect
 successfully to redis and postgres before starting work shouldn't that simply
 be a check (with retries) at the start of the container."* Yes.
 ## A TCP connect, not a query
 The same probe `build.yml`'s integration lane and the build smoke already use.
 It answers the question that is actually being asked — is something listening
 — and it cannot fail for a reason that retrying will never fix.
 A real query would be a stronger readiness signal and a worse gate: a wrong
 password or a missing database is not a transient condition, and a loop that
 waits for one to heal turns a five-second misconfiguration into a two-minute
 timeout with a misleading message. Those belong to alembic, which runs
 seconds later and says exactly what is wrong.
 The Postgres image is well behaved here: during `initdb` it serves on a unix
 socket only and opens TCP when it is ready for clients, so the connect is a
 good proxy for "ready" rather than merely "process exists".
 """
 from __future__ import annotations
 import argparse
 import os
 import socket
 import sys
 import time
 from urllib.parse import urlparse
 # Long enough for a first-ever `initdb` on a slow disk, which is the worst
 # case this exists for and is measured in tens of seconds, not minutes. A
 # deploy that is genuinely misconfigured should fail while someone is still
 # watching it rather than hold the container open for a quarter of an hour.
 DEFAULT_TIMEOUT = 120.0
 CONNECT_TIMEOUT = 2.0
 RETRY_DELAY = 1.0
 # Progress every N attempts. `docker logs` on a container that is waiting must
 # say what it is waiting for — silence is indistinguishable from a hang.
 REPORT_EVERY = 5
 def _target(url: str | None, default_port: int) -> tuple[str, int] | None:
    """(host, port) from a connection URL, or None if there is nothing to wait for."""
    if not url:
        return None
    parsed = urlparse(url)
    if not parsed.hostname:
        return None
    return parsed.hostname, parsed.port or default_port
 def targets() -> list[tuple[str, tuple[str, int]]]:
    """What this container must reach, read from the same env the app reads.
    Derived rather than passed in, so the wait cannot drift from what the
    application will actually connect to — a gate that checks a different
    host than the app uses is worse than no gate.
    """
    out: list[tuple[str, tuple[str, int]]] = []
    host = os.environ.get("DB_HOST")
    if host:
        out.append(("postgres", (host, int(os.environ.get("DB_PORT") or 5432))))
    broker = _target(os.environ.get("CELERY_BROKER_URL"), 6379)
    if broker:
        out.append(("redis", broker))
    return out
 def _accepts(host: str, port: int) -> bool:
    try:
        with socket.create_connection((host, port), timeout=CONNECT_TIMEOUT):
            return True
    except OSError:
        return False
 def wait(
    name: str, host: str, port: int, deadline: float, now=time.monotonic,
 ) -> bool:
    attempt = 0
    while True:
        if _accepts(host, port):
            print(f"[wait] {name} at {host}:{port} is accepting connections")
            return True
        attempt += 1
        if now() >= deadline:
            print(
                f"[wait] TIMEOUT: {name} at {host}:{port} never accepted a "
                f"connection ({attempt} attempts)",
                file=sys.stderr,
            )
            return False
        if attempt % REPORT_EVERY == 0:
            left = int(deadline - now())
            print(f"[wait] {name} at {host}:{port} not ready yet, {left}s left")
        time.sleep(RETRY_DELAY)
 def main(argv: list[str] | None = None) -> int:
    ap = argparse.ArgumentParser(description="Wait for Postgres and Redis.")
    ap.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT)
    args = ap.parse_args(argv)
    wanted = targets()
    if not wanted:
        # Nothing configured to wait for. Not an error: `shell` and one-off
        # runs are legitimate, and refusing to start would make this gate the
        # reason a debugging container will not boot.
        print("[wait] no database or broker configured; nothing to wait for")
        return 0
    deadline = time.monotonic() + args.timeout
    for name, (host, port) in wanted:
        if not wait(name, host, port, deadline):
            return 1
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -300,6 +300,18 @@ class ArtistService:
        await self.session.commit()
        return artist
    async def all_names(self) -> list[tuple[int, str, str]]:
        """Every artist as (id, name, slug), alphabetical.
        For pickers that should show a full list before anything is typed (the
        Latest feed's artist filter). Three columns and no joins, so it stays
        cheap on a library of thousands of artists.
        """
        rows = (await self.session.execute(
            select(Artist.id, Artist.name, Artist.slug).order_by(func.lower(Artist.name))
        )).all()
        return [(r.id, r.name, r.slug) for r in rows]
    async def autocomplete(self, prefix: str, limit: int = 20) -> list[Artist]:
        cleaned = (prefix or "").strip()
        if not cleaned:
@@ -24,6 +24,25 @@ from pathlib import Path
 _BACKUPS_DIRNAME = "_backups"
 # Excluded from the images tarball, and each for its own reason (#4233, #4234):
 #
 #   _backups    — the archive would otherwise contain every previous archive.
 #                 This is not hypothetical: the 2026-05-23/24 runs, taken before
 #                 this exclude existed, grew 43G -> 107G -> ... -> 2123G as each
 #                 swallowed its predecessors, and cost 4.3T of the images
 #                 filesystem until they were reclaimed on 2026-09-21.
 #   _quarantine — holds files deliberately pulled OUT of the library.
 #   secrets     — `credential_key.b64`, the key that decrypts the stored
 #                 Patreon/SubscribeStar session credentials.
 #   cookies     — those session cookies themselves.
 #
 # The last two are the ones worth stating plainly: an images tarball is a media
 # archive, and a media archive that carries the key to the operator's accounts
 # is a credential leak wearing a backup's name. Encryption at rest buys nothing
 # when the key rides along in the same file. A restore therefore does NOT
 # re-establish credentials — you sign in again, which is the correct outcome.
 _IMAGES_EXCLUDED_DIRNAMES = ("_backups", "_quarantine", "secrets", "cookies")
 # Subprocess-level guardrails BEYOND the Celery soft_time_limit. The Celery
 # soft limit signals the Python process; subprocess.Popen in a blocking syscall
 # ignores that signal, so these bound the worst case directly. Each sits just
@@ -173,8 +192,10 @@ def backup_images(
        [
            "tar", "--zstd", "-cf", str(tar_path),
            "-C", str(images_root.parent), images_root.name,
-            f"--exclude={images_root.name}/_backups",
+            *(
-            f"--exclude={images_root.name}/_quarantine",
+                f"--exclude={images_root.name}/{name}"
                for name in _IMAGES_EXCLUDED_DIRNAMES
            ),
        ],
        _IMAGES_SUBPROCESS_TIMEOUT_S,
    )
@@ -24,6 +24,7 @@ rows undecryptable (recovery = delete the rows and re-upload).
 import logging
 import os
 import tempfile
 from pathlib import Path
 from cryptography.fernet import Fernet, InvalidToken
@@ -80,10 +81,59 @@ class CredentialCrypto:
        parent = self._key_path.parent
        parent.mkdir(parents=True, exist_ok=True)
        os.chmod(parent, 0o700)
        # Written to a temp file and LINKED into place, not written directly.
        #
        # hypercorn starts several worker processes and each one builds the
        # app, so on a first boot they all reach this at once. A plain
        # `write_bytes` creates the file at size zero and fills it a moment
        # later, which gives a second process an `exists()` of True and a
        # `read_bytes()` of b"" — and the app dies with
        #
        #     ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
        #
        # Seen on run 7368's smoke, and it is a race rather than a certainty:
        # the same image had booted cleanly on the three runs before it. A
        # first boot that fails one time in five is worse than one that fails
        # every time, because it looks like the deployment rather than the code.
        #
        # `os.link` is the atomic part: it either creates the name or raises
        # FileExistsError, and it cannot expose a half-written file. NOT
        # `os.replace`, which would succeed — so two processes that both
        # generated a key would each think they had won, and the loser's key
        # would overwrite the one the winner had already handed to Fernet.
        # `mkstemp`, not a pid-derived name. The first cut spelled the temp
        # file `.credential_key.b64.<pid>.tmp`, which assumes one bootstrap per
        # process — and the test that exercises this with eight THREADS shares
        # one pid, so all eight raced the same filename and six died with
        # FileNotFoundError when another had already unlinked it. The
        # assumption held for hypercorn's workers and would have held in
        # production; it was still an assumption the code did not need to make.
        key = Fernet.generate_key()
-        self._key_path.write_bytes(key)
+        fd, tmp_name = tempfile.mkstemp(
-        os.chmod(self._key_path, 0o600)
+            dir=parent, prefix=f".{self._key_path.name}.", suffix=".tmp",
-        return key
+        )
        tmp = Path(tmp_name)
        try:
            with os.fdopen(fd, "wb") as fh:
                fh.write(key)
            os.chmod(tmp, 0o600)
            try:
                os.link(tmp, self._key_path)
            except FileExistsError:
                # Another process created it between our `exists()` check and
                # here. Theirs is as good as ours, and using it is what keeps
                # every worker on ONE key.
                log.info(
                    "another process created %s first; using that key",
                    self._key_path,
                )
        finally:
            tmp.unlink(missing_ok=True)
        # Read back rather than returning `key`: on the losing branch the file
        # holds somebody else's, and returning ours would leave this worker
        # encrypting with a key no other worker can read.
        return self._key_path.read_bytes()
    def encrypt(self, plaintext: str) -> bytes:
        return self._fernet.encrypt(plaintext.encode("utf-8"))
@@ -16,7 +16,7 @@ from __future__ import annotations
 from collections.abc import Awaitable, Callable
-from sqlalchemy import Select
+from sqlalchemy import Select, and_
 from sqlalchemy.exc import IntegrityError
 from sqlalchemy.ext.asyncio import AsyncSession
@@ -67,12 +67,26 @@ async def get_or_create[T](
 def failing_sources_clause():
-    """A source is FAILING when its runs are actually erroring.
+    """A source is FAILING when it is ENABLED and its runs are erroring.
    Deliberately not `last_error IS NOT NULL` — a tier-limited source clears
    last_error and keeps a chip, and must never be counted as broken.
    The `enabled` half was folded in 2026-09-21 (#4279). A disabled source is
    one FC deliberately stopped — most often because the membership sweep saw
    `former_patron` — and "stopped because you no longer subscribe" is not
    "failing". Worse, it is a failure nobody can clear: a disabled source is
    never scheduled, so no successful run ever resets the counter, and the
    card's Retry button routes to `/check`, which refuses a disabled source
    outright. Ebi77 sat in the banner for six days with no action available.
    This also settles a disagreement the two callers already had. The
    scheduler's status count paired this clause with `enabled.is_(True)`;
    `SourceService.list(failing=True)` did not. One counted Ebi77, the other
    did not — the exact drift the note above this function warns about, which
    is why the `enabled` test belongs IN the predicate rather than beside it.
    """
-    return Source.consecutive_failures > 0
+    return and_(Source.enabled.is_(True), Source.consecutive_failures > 0)
 def no_access_sources_clause():
@@ -0,0 +1,562 @@
 """Native Discord read client — the Discord counterpart to subscribestar_client.
 Mirrors gallery-dl 1.32.13's `extractor/discord.py` (rule 130: gallery-dl is the
 known-working base), adapted to the native core's client contract
 (`ingest_core` module docstring): `iter_posts` / `extract_media`, plus the
 post-first `post_record_key` and the `post_meta` date the revisit window reads.
 What is mirrored exactly, because drift in any of it changes what we fetch or
 where it lands on disk:
  - API v10, `Authorization: <user token>` (a USER token, not a bot token).
  - gallery-dl's request profile: its date-derived Firefox User-Agent,
    `Accept: */*`, `Accept-Language`, `Referer: https://discord.com/`.
  - `GET /channels/{id}/messages?limit=100&before=<last id>`, newest first,
    stopping on a short page. Message types {0, 19, 21} only.
  - The walk: a text/news channel's own messages then its threads, a forum's
    threads, a category's children, a server's text/news/forum channels.
  - Files: attachments, then embeds of type image/gifv/video (FC configures
    `embeds: all`, which for files is the same three plus rich/link embeds that
    carry an image), then forwarded `message_snapshots`, numbered from 1 across
    the lot — the `num` in `{date}_{message_id}_{num}_{filename}`.
  - Text: `content`, rich-embed author/title/description/fields/footer, poll.
 Two deliberate departures, both about the walk order, neither about content:
  - Threads are walked newest-CREATED first (by id), not by last-message time.
    A backfill resumes from a checkpointed channel; last-message order shifts
    between chunks whenever someone posts, which can move an unwalked thread
    above the resume point and skip it. Creation order only ever grows at the
    front, where the next tick finds it.
  - A 403 on a thread or a nested channel skips that feed instead of failing
    the walk. gallery-dl skips only nested channels; one private thread the
    token cannot read would otherwise stop every channel after it.
 FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
 """
 from __future__ import annotations
 import hashlib
 import logging
 import re
 import time
 from collections.abc import Iterator
 from dataclasses import dataclass
 from datetime import date
 from urllib.parse import unquote
 import requests
 from .native_ingest_common import (
    NativeAuthError,
    NativeDriftError,
    NativeIngestError,
    retry_after_seconds,
 )
 log = logging.getLogger(__name__)
 API_ROOT = "https://discord.com/api/v10"
 _ROOT = "https://discord.com"
 _TIMEOUT_SECONDS = 60.0
 _MESSAGES_BATCH = 100
 _THREADS_BATCH = 25
 # gallery-dl retries a 429 up to its default 4 retries, waiting
 # `request_interval_429` (60s) between them. Discord's Retry-After is exact, so
 # it is honoured when present; 60s is the fallback and the cap.
 _MAX_429_RETRIES = 4
 _429_WAIT_SECONDS = 60.0
 # https://discord.com/developers/docs/resources/message#message-object-message-types
 # DEFAULT, REPLY, CHAT_INPUT_COMMAND — the ones that carry user content.
 MESSAGE_TYPES = frozenset({0, 19, 21})
 # https://discord.com/developers/docs/resources/channel#channel-object-channel-types
 _TEXT = frozenset({0, 5})               # text, announcement: messages + threads
 _DIRECT = frozenset({1, 3, 10, 11, 12})  # DMs and threads: messages only
 _FORUM = frozenset({15, 16})            # forum, media: threads only
 _CATEGORY = 4
 _SERVER_WALK = _TEXT | _FORUM
 _EMBED_TYPES = frozenset({"image", "gifv", "video"})
 _URL_RE = re.compile(
    r"^(?:https?://)?(?:www\.|ptb\.|canary\.)?discord(?:app)?\.com/channels/"
    r"(?P<server>@me|\d+)(?:/(?:\d+/threads/)?(?P<channel>\d+))?(?P<rest>/.*)?/?$"
 )
 class DiscordAPIError(NativeIngestError):
    """Base for native Discord client failures."""
 class DiscordAuthError(DiscordAPIError, NativeAuthError):
    """401 (the token is invalid or expired) or a 403 on the channel the
    source names. The fix is a new token, not a new client."""
 class DiscordDriftError(DiscordAPIError, NativeDriftError):
    """A response did not have the shape the walk depends on."""
 def firefox_user_agent(today: date | None = None) -> str:
    """gallery-dl's default User-Agent: a Firefox whose version advances every
    four weeks (`util._ff_ver`, "147 on 2026-01-13"). Computed the same way so
    the profile keeps matching the gallery-dl this replaced."""
    ver = ((today or date.today()).toordinal() - 735_513) // 28
    return (
        f"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:{ver}.0) "
        f"Gecko/20100101 Firefox/{ver}.0"
    )
 def nameext_from_url(url: str) -> tuple[str, str]:
    """gallery-dl's `text.nameext_from_url`: the URL's last path segment,
    unquoted, split at the last dot when the extension is at most 16 chars
    (lowercased); otherwise the whole name and no extension."""
    filename = unquote(url.partition("?")[0].rpartition("/")[2])
    name, _, ext = filename.rpartition(".")
    if name and len(ext) <= 16:
        return name, ext.lower()
    return filename, ""
 def parse_source_url(url: str) -> tuple[str | None, str | None]:
    """`(server_id, channel_id)` from a Discord channel/server URL. `server_id`
    is None for a DM (`@me`); `channel_id` is None for a whole server. Raises
    DiscordAPIError for anything else, including a link to a single message —
    a message is not something a source can subscribe to."""
    m = _URL_RE.match((url or "").strip())
    if not m or (m.group("rest") or "").strip("/"):
        raise DiscordAPIError(
            f"Not a Discord channel or server link: {url!r} "
            "(expected https://discord.com/channels/<server>[/<channel>])"
        )
    server = m.group("server")
    channel = m.group("channel")
    if server == "@me":
        if not channel:
            raise DiscordAPIError(f"A DM link needs a channel id: {url!r}")
        return None, channel
    return server, channel
 def message_text(message: dict) -> str:
    """gallery-dl's `extract_message_text`: the body plus the text of rich
    embeds and polls, newline-joined, empties dropped."""
    parts = [message.get("content") or ""]
    for embed in message.get("embeds") or []:
        if embed.get("type") != "rich":
            continue
        parts.append((embed.get("author") or {}).get("name") or "")
        parts.append(embed.get("title") or "")
        parts.append(embed.get("description") or "")
        for fld in embed.get("fields") or []:
            parts.append(fld.get("name") or "")
            parts.append(fld.get("value") or "")
        parts.append((embed.get("footer") or {}).get("text") or "")
    poll = message.get("poll")
    if poll:
        parts.append(((poll.get("question") or {}).get("text")) or "")
        for answer in poll.get("answers") or []:
            parts.append(((answer.get("poll_media") or {}).get("text")) or "")
    return "\n".join(p for p in parts if p)
@dataclass
 class MediaItem:
    """One file of a Discord message. `filename`/`extension` are gallery-dl's
    split of the URL; `num` is its 1-based position across the message's files,
    which is what names it on disk.
    `media_id` is what the seen-ledger keys on, and it is deliberately NOT
    `num`: an edit that removes a file renumbers the ones after it, and a
    positional key would then call a different file seen. It is the
    attachment's id, or for an embed (which has none) a hash of its URL path —
    the query string is a signature that changes on every fetch. `filehash` is
    always None; nothing in a signed CDN URL is a content hash."""
    url: str
    filename: str
    extension: str
    kind: str
    post_id: str
    num: int
    media_id: str
    filehash: str | None = None
 class DiscordClient:
    """Synchronous Discord API v10 read client for one user token."""
    def __init__(
        self,
        token: str | None,
        *,
        request_sleep: float = 0.0,
        max_retries: int = _MAX_429_RETRIES,
        session: requests.Session | None = None,
    ):
        self._session = session or requests.Session()
        self._session.headers.update({
            "User-Agent": firefox_user_agent(),
            "Accept": "*/*",
            "Accept-Language": "en-US,en;q=0.5",
            "Referer": _ROOT + "/",
        })
        if token:
            self._session.headers["Authorization"] = token
        self._token = token
        self._request_sleep = request_sleep or 0.0
        self._max_retries = max_retries
        self._server: dict = {}
        self._channels: dict[str, dict] = {}
        self._skip_feed = False
    # -- request -----------------------------------------------------------
    def _get(self, endpoint: str, params: dict | None = None):
        if not self._token:
            raise DiscordAuthError("No Discord token is configured for this source")
        if self._request_sleep > 0:
            time.sleep(self._request_sleep)
        url = API_ROOT + endpoint
        attempt = 0
        while True:
            try:
                resp = self._session.get(url, params=params, timeout=_TIMEOUT_SECONDS)
            except requests.RequestException as exc:
                raise DiscordAPIError(f"Discord request failed ({endpoint}): {exc}") from exc
            if resp.status_code == 429 and attempt < self._max_retries:
                attempt += 1
                delay = retry_after_seconds(
                    resp, attempt, base=_429_WAIT_SECONDS, cap=_429_WAIT_SECONDS,
                )
                log.warning(
                    "Discord 429 (%s) — waiting %.1fs (retry %d/%d)",
                    endpoint, delay, attempt, self._max_retries,
                )
                time.sleep(delay)
                continue
            break
        if resp.status_code == 401:
            raise DiscordAuthError(
                "Discord rejected the token (HTTP 401) — it is invalid or has "
                "expired; copy a fresh one from the browser",
                status_code=401,
            )
        if resp.status_code != 200:
            raise DiscordAPIError(
                f"Discord returned HTTP {resp.status_code} ({endpoint})",
                status_code=resp.status_code,
                retry_after=_retry_after(resp),
            )
        try:
            return resp.json()
        except ValueError as exc:
            raise DiscordDriftError(
                f"Discord returned non-JSON for {endpoint} ({len(resp.content)} bytes)"
            ) from exc
    # -- metadata (gallery-dl parse_server / parse_channel) -----------------
    def _load_server(self, server_id: str) -> None:
        server = self._get(f"/guilds/{server_id}")
        if not isinstance(server, dict) or "id" not in server:
            raise DiscordDriftError(f"Discord server {server_id} came back without an id")
        self._server = {
            "server": server.get("name") or "",
            "server_id": str(server["id"]),
            "owner_id": server.get("owner_id"),
        }
        channels = self._get(f"/guilds/{server_id}/channels")
        if not isinstance(channels, list):
            raise DiscordDriftError(f"Discord server {server_id} channel list is not a list")
        # Categories first, so every child can name its parent.
        for channel in sorted(channels, key=lambda ch: ch.get("type") != _CATEGORY):
            self._parse_channel(channel)
    def _parse_channel(self, channel: dict) -> dict:
        parent_id = channel.get("parent_id")
        meta = {
            "channel": channel.get("name") or "",
            "channel_id": str(channel.get("id")),
            "channel_type": channel.get("type"),
            "channel_topic": channel.get("topic") or "",
            "parent_id": parent_id,
            "is_thread": "thread_metadata" in channel,
        }
        parent = self._channels.get(parent_id) if parent_id else None
        if parent:
            meta["parent"] = parent["channel"]
            meta["parent_type"] = parent["channel_type"]
        if meta["channel_type"] in {1, 3}:
            recipients = channel.get("recipients") or []
            meta["channel"] = "DMs"
            meta["recipients"] = [u.get("username") for u in recipients]
            meta["recipients_id"] = [u.get("id") for u in recipients]
        self._channels[meta["channel_id"]] = meta
        return meta
    def _channel_meta(self, channel_id: str) -> dict:
        if channel_id not in self._channels:
            self._parse_channel(self._get(f"/channels/{channel_id}"))
        return self._channels[channel_id]
    def _threads(self, channel_id: str) -> list[dict]:
        """Every thread of a channel or forum, newest-created first (see the
        module docstring for why not last-message order)."""
        threads: list[dict] = []
        offset = 0
        while True:
            data = self._get(f"/channels/{channel_id}/threads/search", {
                "sort_by": "last_message_time",
                "sort_order": "desc",
                "limit": _THREADS_BATCH,
                "offset": offset,
            })
            batch = (data.get("threads") or []) if isinstance(data, dict) else []
            threads.extend(batch)
            if len(batch) < _THREADS_BATCH:
                break
            offset += len(batch)
        threads.sort(key=lambda t: int(t.get("id") or 0), reverse=True)
        return threads
    # -- the walk ------------------------------------------------------------
    def _feeds(self, channel_id: str, *, safe: bool) -> Iterator[tuple[str, bool]]:
        """`(channel_id, safe)` for every message feed under `channel_id`, in
        gallery-dl's order. `safe` feeds are skipped on a 403."""
        try:
            ctype = self._channel_meta(channel_id)["channel_type"]
        except DiscordAPIError as exc:
            if exc.status_code != 403:
                raise
            if not safe:
                raise DiscordAuthError(
                    f"The Discord token cannot see channel {channel_id} (HTTP 403)",
                    status_code=403,
                ) from exc
            log.info("Discord: no access to channel %s — skipped", channel_id)
            return
        if ctype in _TEXT or ctype in _DIRECT:
            yield channel_id, safe
        if ctype in _TEXT or ctype in _FORUM:
            try:
                threads = self._threads(channel_id)
            except DiscordAPIError as exc:
                if exc.status_code != 403:
                    raise
                log.info("Discord: cannot list threads of %s — skipped", channel_id)
                threads = []
            for thread in threads:
                yield self._parse_channel(thread)["channel_id"], True
        elif ctype == _CATEGORY:
            for child in list(self._channels.values()):
                if child.get("parent_id") == channel_id:
                    yield from self._feeds(child["channel_id"], safe=True)
        elif ctype not in _DIRECT and not safe:
            raise DiscordAPIError(
                f"Discord channel {channel_id} is of type {ctype}, which has no messages"
            )
    def _source_feeds(self, url: str) -> Iterator[tuple[str, bool]]:
        server_id, channel_id = parse_source_url(url)
        self._server, self._channels = {}, {}
        if server_id is not None:
            self._load_server(server_id)
        if channel_id is not None:
            yield from self._feeds(channel_id, safe=False)
            return
        for meta in list(self._channels.values()):
            if meta["channel_type"] in _SERVER_WALK:
                yield from self._feeds(meta["channel_id"], safe=True)
    def skip_feed(self) -> None:
        """Optional core seam (#4413): end the current channel and go on to the
        next one. A tick's early-out means THIS channel has nothing new, not
        that the server has nothing new."""
        self._skip_feed = True
    def iter_posts(
        self, campaign_id: str, cursor: str | None = None
    ) -> Iterator[tuple[dict, dict, str | None]]:
        """Yield `(message, channel_meta, page_cursor)` for every content
        message the source reaches, channel by channel, each newest first.
        `campaign_id` is the source URL. The cursor is `<channel_id>:<before>`
        — the channel and the `before` id that fetched the page (empty for a
        channel's first page) — so a backfill resumes inside the right channel
        and re-fetches the page it was cut in. A cursor naming a channel the
        walk no longer reaches (a deleted thread) restarts from the top rather
        than walking nothing.
        """
        resume_channel, _, resume_before = (cursor or "").partition(":")
        resuming = bool(resume_channel)
        feeds = list(self._source_feeds(campaign_id)) if resuming else None
        if feeds is not None and resume_channel not in {cid for cid, _ in feeds}:
            log.warning(
                "Discord: resume channel %s is no longer in %s — restarting",
                resume_channel, campaign_id,
            )
            resuming = False
        for channel_id, safe in feeds if feeds is not None else self._source_feeds(campaign_id):
            before = None
            if resuming:
                if channel_id != resume_channel:
                    continue
                resuming = False
                before = resume_before or None
            yield from self._iter_channel(channel_id, before, safe=safe)
    def _iter_channel(
        self, channel_id: str, before: str | None, *, safe: bool
    ) -> Iterator[tuple[dict, dict, str | None]]:
        self._skip_feed = False
        meta = {**self._server, **self._channels.get(channel_id, {})}
        while True:
            page_cursor = f"{channel_id}:{before or ''}"
            try:
                messages = self._get(
                    f"/channels/{channel_id}/messages",
                    {"limit": _MESSAGES_BATCH, "before": before},
                )
            except DiscordAPIError as exc:
                if exc.status_code != 403:
                    raise
                if not safe:
                    raise DiscordAuthError(
                        f"The Discord token cannot read channel {channel_id} (HTTP 403)",
                        status_code=403,
                    ) from exc
                log.info("Discord: no access to messages of %s — skipped", channel_id)
                return
            if not isinstance(messages, list):
                raise DiscordDriftError(
                    f"Discord messages of {channel_id} came back as "
                    f"{type(messages).__name__}, not a list"
                )
            for message in messages:
                if message.get("type") not in MESSAGE_TYPES:
                    continue
                message["_meta"] = meta
                yield message, meta, page_cursor
                if self._skip_feed:
                    return
            if len(messages) < _MESSAGES_BATCH:
                return
            before = str(messages[-1]["id"])
    # -- per-message -------------------------------------------------------
    @staticmethod
    def extract_media(post: dict, included: dict | None = None) -> list[MediaItem]:
        """gallery-dl's file list for one message: attachments, then the first
        of video/image/thumbnail `proxy_url` of each file-bearing embed, then
        the same for every forwarded snapshot; numbered from 1 across them."""
        mid = str(post.get("id") or "")
        snapshots = [post] + [
            (s or {}).get("message") or {}
            for s in post.get("message_snapshots") or []
            if ((s or {}).get("message") or {}).get("type", 0) in MESSAGE_TYPES
        ]
        found: list[tuple[str, str, str | None]] = []
        for snap in snapshots:
            for att in snap.get("attachments") or []:
                if att.get("url"):
                    aid = att.get("id")
                    found.append((att["url"], "attachment", str(aid) if aid else None))
            for embed in snap.get("embeds") or []:
                if embed.get("type") not in _EMBED_TYPES:
                    continue
                for fld in ("video", "image", "thumbnail"):
                    url = (embed.get(fld) or {}).get("proxy_url")
                    if url:
                        found.append((url, "embed", None))
                        break
        items = []
        for num, (url, kind, fid) in enumerate(found, start=1):
            name, ext = nameext_from_url(url)
            if fid is None:
                path = url.partition("?")[0].encode()
                fid = "u" + hashlib.sha1(path, usedforsecurity=False).hexdigest()[:32]
            items.append(MediaItem(
                url=url, filename=name, extension=ext, kind=kind, post_id=mid,
                num=num, media_id=fid,
            ))
        return items
    @staticmethod
    def post_meta(post: dict) -> dict:
        """No title (Discord has none); `date` is the message timestamp, ISO
        with an offset — what the core's revisit window reads."""
        return {"title": None, "date": post.get("timestamp")}
    @classmethod
    def post_record_key(cls, post: dict) -> tuple[str, str] | None:
        """`(message:<id>, <id>)` — gates the message record through the seen
        ledger, like `post:<id>` on the other platforms.
        None for a message with no files. gallery-dl wrote a sidecar only
        beside a file, so a text-only chat line never became a post, and the
        drop grouping (discord_grouping) is built on that: a channel's chatter
        recorded as posts would bury the drops it exists to surface."""
        mid = post.get("id")
        mid = str(mid) if mid is not None else ""
        if not mid or not cls.extract_media(post):
            return None
        return (f"message:{mid}", mid)
    # -- verify ------------------------------------------------------------
    def describe(self, server_id: str | None, channel_id: str | None) -> dict:
        """The display names behind a server/channel pair, for the browser
        extension's Add panel. Best-effort per name: one that can't be read
        comes back None, and the other is still returned."""
        out: dict = {"server": None, "channel": None, "parent": None}
        if server_id:
            try:
                out["server"] = (self._get(f"/guilds/{server_id}") or {}).get("name") or None
            except DiscordAPIError:
                pass
        if channel_id:
            try:
                meta = self._parse_channel(self._get(f"/channels/{channel_id}"))
                out["channel"] = meta.get("channel") or None
            except (DiscordAPIError, AttributeError):
                pass
        return out
    def verify_auth(self, url: str) -> tuple[bool | None, str]:
        """Is the token valid, and can it see what the source names?"""
        try:
            server_id, channel_id = parse_source_url(url)
        except DiscordAPIError as exc:
            return None, str(exc)
        try:
            me = self._get("/users/@me")
            if channel_id is not None:
                self._get(f"/channels/{channel_id}")
            elif server_id is not None:
                self._get(f"/guilds/{server_id}")
        except DiscordAuthError as exc:
            return False, f"Discord rejected the token — {exc}"
        except DiscordAPIError as exc:
            if exc.status_code in (403, 404):
                return False, (
                    "The token is valid, but its account cannot see "
                    f"{'this channel' if channel_id else 'this server'} "
                    f"(HTTP {exc.status_code})"
                )
            return None, f"Couldn't verify (network/HTTP issue): {exc}"
        who = (me or {}).get("username") if isinstance(me, dict) else None
        return True, f"Token valid{f' ({who})' if who else ''} — the source is readable."
 def _retry_after(resp: requests.Response) -> float | None:
    hdr = resp.headers.get("Retry-After")
    try:
        return float(hdr) if hdr else None
    except (TypeError, ValueError):
        return None
@@ -0,0 +1,212 @@
 """Native Discord media downloader — the Discord counterpart to
 subscribestar_downloader.
 Writes files exactly where gallery-dl wrote them, so a cutover finds every
 existing file on disk (`skipped_disk`) instead of fetching it again:
    <images>/<artist>/discord/<channel>/<YYYYMMDD>_<message_id>_<NN>_<name>.<ext>
 That is what FC's gallery-dl config produced (directory `{channel}`, filename
 `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}.{extension}`, under the
 per-source base directory `<images>/<artist>/<platform>`), retired from that
 config once Discord moved here; tests/test_discord_naming.py pins the match
 against a real gallery-dl sidecar. The name is cleaned the way gallery-dl cleans it
 on Linux — `/` becomes `_` and control characters are removed, nothing else
 (`path-restrict: auto`, `path-remove` defaults). It is NOT `sanitize_segment`,
 whose Windows set would turn a `:` in a channel or file name into `_` and miss
 the file gallery-dl wrote.
 Post-first (rule 120): each file gets a minimal sidecar named like it minus the
 extension (what `find_sidecar` pairs first), and the message itself gets one
 record, `<YYYYMMDD>_<message_id>_post.json`, carrying gallery-dl's metadata keys
 — `message_id` for the post id, `server_id`/`channel_id` for the permalink,
 `message` for the body, `date` — so `parse_sidecar` reads it exactly as it read
 the gallery-dl sidecars. Neither file carries an `id` or `post_id` key: both
 outrank `message_id` in the post-id chain (`platforms.base`).
 PURE: no DB; the seen-skip is an injected predicate.
 """
 from __future__ import annotations
 import json
 import logging
 import re
 import time
 from collections.abc import Callable
 from datetime import UTC, datetime
 from pathlib import Path
 import requests
 from .discord_client import firefox_user_agent, message_text
 from .native_ingest_common import (
    BaseNativeDownloader,
    MediaOutcome,
    PostRecordOutcome,
    make_session,
 )
 log = logging.getLogger(__name__)
 PLATFORM = "discord"
 _CONTROL = re.compile("[\x00-\x1f\x7f]")
 # gallery-dl falls back to the response's type for a URL with no extension;
 # we never see the response before naming, and such URLs do not occur for
 # Discord attachments or embed proxies in practice.
 _NO_EXTENSION = "bin"
 def gdl_clean(segment: str) -> str:
    """One path segment as gallery-dl writes it on Linux."""
    return _CONTROL.sub("", segment.replace("/", "_"))
 def message_date(post: dict) -> datetime | None:
    raw = post.get("timestamp")
    if not isinstance(raw, str) or not raw:
        return None
    try:
        dt = datetime.fromisoformat(raw.replace("Z", "+00:00"))
    except ValueError:
        return None
    return (dt if dt.tzinfo else dt.replace(tzinfo=UTC)).astimezone(UTC)
 def channel_dir(images_root: Path, artist_slug: str, post: dict) -> Path:
    """gallery-dl's `{channel}` directory; an empty name adds no segment."""
    base = Path(images_root) / artist_slug / PLATFORM
    channel = gdl_clean(((post.get("_meta") or {}).get("channel") or "").strip())
    return base / channel if channel else base
 def media_stem(post: dict, media) -> str:
    """`<YYYYMMDD>_<message_id>_<NN>_<name>` — the file's name minus `.<ext>`."""
    when = message_date(post)
    day = f"{when:%Y%m%d}" if when else "None"
    return gdl_clean(f"{day}_{post.get('id')}_{media.num:>02}_{media.filename}")
 class DiscordDownloader(BaseNativeDownloader):
    """Download a message's files to gallery-dl's layout. The CDN gets
    gallery-dl's browser profile and no token — gallery-dl sends the token only
    to the API, and the CDN URLs are pre-signed."""
    def __init__(
        self,
        images_root: Path,
        cookies_path: str | None = None,
        *,
        validate: bool = True,
        rate_limit: float = 0.0,
        session: requests.Session | None = None,
    ):
        super().__init__(
            images_root, None, platform=PLATFORM,
            validate=validate, rate_limit=rate_limit,
            session=session if session is not None else make_session(None, extra_headers={
                "User-Agent": firefox_user_agent(),
                "Accept-Language": "en-US,en;q=0.5",
                "Referer": "https://discord.com/",
            }),
        )
    def download_post(
        self,
        post: dict,
        media_items: list,
        artist_slug: str,
        *,
        is_seen: Callable[[object], bool] = lambda m: False,
        should_stop: Callable[[], bool] = lambda: False,
        recapture: bool = False,
    ) -> list[MediaOutcome]:
        """Every file of one message; per-file outcomes, one failure isolated."""
        folder = channel_dir(self.images_root, artist_slug, post)
        outcomes: list[MediaOutcome] = []
        for media in media_items:
            if should_stop():
                break
            try:
                outcomes.append(self._download_one(
                    post, media, folder, artist_slug, is_seen, recapture=recapture,
                ))
            except Exception as exc:  # resilient: isolate one item's failure
                log.warning(
                    "Discord media failed (message %s, file %d): %s",
                    post.get("id"), media.num, exc,
                )
                outcomes.append(
                    MediaOutcome(media=media, status="error", path=None, error=str(exc))
                )
        return outcomes
    def _download_one(
        self,
        post: dict,
        media,
        folder: Path,
        artist_slug: str,
        is_seen: Callable[[object], bool],
        *,
        recapture: bool = False,
    ) -> MediaOutcome:
        seen = is_seen(media)
        if seen and not recapture:
            return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
        stem = media_stem(post, media)
        path = folder / f"{stem}.{media.extension or _NO_EXTENSION}"
        if path.exists():  # tier-2: gallery-dl (or an earlier walk) wrote it
            return MediaOutcome(media=media, status="skipped_disk", path=path, error=None)
        if seen:  # recapture never re-fetches a seen file that is gone
            return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
        folder.mkdir(parents=True, exist_ok=True)
        if self._rate_limit > 0:
            time.sleep(self._rate_limit)
        out = self._fetch_get(media.url, path)
        reason, quarantined = self._validate_path(out, artist_slug, media.url)
        if reason is not None:
            return MediaOutcome(media=media, status="quarantined", path=quarantined, error=reason)
        sidecar = {"category": PLATFORM, "message_id": str(post.get("id") or "")}
        sidecar["source_url"] = media.url
        (folder / f"{stem}.json").write_text(json.dumps(sidecar, indent=2))
        return MediaOutcome(media=media, status="downloaded", path=out, error=None)
    def write_post_record(
        self, post: dict, artist_slug: str, *, revisit: bool = False,
    ) -> PostRecordOutcome:
        """The message record — the one writer of a Discord post's body, date
        and permalink ids. `revisit` re-reads a message already captured (an
        edit); an empty re-read writes nothing, so it never blanks a body."""
        mid = str(post.get("id") or "")
        body = message_text(post)
        if not mid or (revisit and not body.strip()):
            return PostRecordOutcome(path=None, post_type=None, title=None, body_chars=0)
        meta = post.get("_meta") or {}
        author = post.get("author") or {}
        record = {
            "category": PLATFORM,
            "message_id": mid,
            "server": meta.get("server"),
            "server_id": meta.get("server_id"),
            "channel": meta.get("channel"),
            "channel_id": meta.get("channel_id") or post.get("channel_id"),
            "parent": meta.get("parent"),
            "is_thread": meta.get("is_thread"),
            "author": author.get("username"),
            "author_id": author.get("id"),
            "message": body,
            "date": post.get("timestamp"),
        }
        folder = channel_dir(self.images_root, artist_slug, post)
        folder.mkdir(parents=True, exist_ok=True)
        when = message_date(post)
        day = f"{when:%Y%m%d}" if when else "None"
        path = folder / f"{day}_{mid}_post.json"
        path.write_text(json.dumps(
            {k: v for k, v in record.items() if v is not None}, indent=2,
        ))
        return PostRecordOutcome(
            path=path, post_type=None, title=None, body_chars=len(body),
        )
@@ -59,14 +59,23 @@ from __future__ import annotations
 import logging
 import math
 from collections import Counter
 from dataclasses import dataclass, field
 from datetime import UTC, datetime, timedelta
-from sqlalchemy import Select, func, select, update
+from sqlalchemy import Select, delete, func, select, union, update
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
-from ..models import ImageProvenance, ImageRecord, MLSettings, Post, Source
+from ..models import (
    ImageProvenance,
    ImageRecord,
    MLSettings,
    Post,
    PostAssociation,
    Source,
 )
 from .post_naming import FAMILY_MAX_POSTS, leading_name, rarity, token_frequencies
 log = logging.getLogger(__name__)
@@ -117,6 +126,31 @@ def cosine_distance(a, b) -> float:
    return 1.0 - (dot / (na * nb))
 def _message_images(posts):
    """(post_id, image_id) for every image a message carries — owned AND re-posted.
    A message owns an image through `primary_post_id`, but only the FIRST
    message imported with a given file does. The same file posted again is a
    provenance link, and the backfill runs newest-first, so it is usually the
    ORIGINAL message that ends up owning nothing. Reading ownership alone left
    101 of Yellowroom's messages (2018–2020 mostly) ungroupable: every image
    they carried also sat in another message.
    `posts` is a list of ids or a select of them; filtering both branches by it
    keeps the union to the messages in hand rather than the whole library.
    Callers pass MESSAGE posts only — a drop's own provenance rows would read
    as images it carries.
    """
    owned = select(
        ImageRecord.primary_post_id.label("post_id"), ImageRecord.id.label("image_id"),
    ).where(ImageRecord.primary_post_id.in_(posts))
    reposted = select(
        ImageProvenance.post_id.label("post_id"),
        ImageProvenance.image_record_id.label("image_id"),
    ).where(ImageProvenance.post_id.in_(posts))
    return union(owned, reposted).subquery()
 def _candidate_stmt(source_id: int, *, not_after: datetime) -> Select:
    """Ungrouped Discord message-posts, one representative image each, OLDEST
    FIRST — which is the order `build_groups` requires.
@@ -134,13 +168,15 @@ def _candidate_stmt(source_id: int, *, not_after: datetime) -> Select:
    take the OLDEST candidates instead of the lowest-numbered ones.
    """
    sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
    carried = _message_images(select(Post.id).where(Post.source_id == source_id))
    inner = (
        select(
            Post.id.label("post_id"),
            sort_key.label("occurred_at"),
            ImageRecord.siglip_embedding.label("embedding"),
        )
-        .join(ImageRecord, ImageRecord.primary_post_id == Post.id)
+        .join(carried, carried.c.post_id == Post.id)
        .join(ImageRecord, ImageRecord.id == carried.c.image_id)
        .where(
            Post.source_id == source_id,
            # Never absorb a post FC wrote, and never re-absorb one already
@@ -286,9 +322,10 @@ async def _link_member_images(
    """
    if not member_ids:
        return 0
-    image_rows = (await session.execute(
+    carried = _message_images(member_ids)
-        select(ImageRecord.id).where(ImageRecord.primary_post_id.in_(member_ids))
+    image_rows = sorted(set((await session.execute(
-    )).scalars().all()
+        select(carried.c.image_id)
    )).scalars().all()))
    if not image_rows:
        return 0
    await session.execute(
@@ -412,9 +449,11 @@ async def _group_seed(session: AsyncSession, post_id: int) -> list[float] | None
    free to disagree; this way there is one.
    """
    sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
    carried = _message_images(select(Post.id).where(Post.absorbed_by_post_id == post_id))
    return (await session.execute(
        select(ImageRecord.siglip_embedding)
-        .join(Post, ImageRecord.primary_post_id == Post.id)
+        .join(carried, carried.c.image_id == ImageRecord.id)
        .join(Post, Post.id == carried.c.post_id)
        .where(
            Post.absorbed_by_post_id == post_id,
            ImageRecord.siglip_embedding.is_not(None),
@@ -600,6 +639,352 @@ async def join_open_groups(
    return joined
 # ---------------------------------------------------------------------------
 # #4390: a trickle is one drop — later drops of the same piece merge in.
 # ---------------------------------------------------------------------------
 #
 # Operator, 2026-09-24, on a feed of "Grouped from 1 Discord message" cards:
 # *"the groups are still single image even when they can clearly be seen as
 # group"*. 665 of Yellowroom's 714 drops were one message.
 #
 # Both paths above join on cosine distance to the group's SEED, and the stages
 # of one piece fail it: `svtt_wip4` did not join `svtt_wip3` from the day
 # before. A creator trickles a piece out as sketch -> wip -> wip -> release, and
 # each stage is nearest to the one before it, not to the first.
 #
 # Measured on artist 8 before any of this was written (#4390 log):
 #
 # * phash cannot see it. Stages of one piece sit 68-134 bits apart; unrelated
 #   pieces by the same artist sit at a median of 126, p5 110. Lesson #4400.
 # * The embedding's NEAREST neighbour can. Every stage of three real trickles
 #   had a sibling stage as its single nearest image in the artist's whole
 #   library, while siblings further along ranked 40-100 — which is exactly why
 #   seed distance fails. Negative control over all 137 recent Discord images:
 #   where the nearest neighbour was another message within 7 days, the two
 #   carried the same working name 53 times out of 53. Disagreements start past
 #   7 days.
 # * The working name sees it directly, when there is one.
 #
 # So a later drop merges into an earlier one when the two are within
 # `discord_group_close_after_hours` of each other (168h — the measured 7 days)
 # AND either they share a gated LEADING working name, or one's image has the
 # other's image as its nearest neighbour. A drop reaching SEVERAL earlier drops
 # pulls them all together — unless two of them are named as different pieces,
 # in which case nothing moves (see `_compatible`): leaving a drop alone is
 # recoverable, a wrong merge asserts that unrelated art belongs together.
 #
 # Chaining is permitted here and was forbidden above, deliberately. The seed
 # rule exists because tiny steps can drift from one piece to another; the
 # measured precision of nearest-neighbour inside 7 days is what bounds drift
 # for this route, and each link is between neighbours in time, never across a
 # quiet week.
 # How many unchecked drops one sweep examines per source. A first run over an
 # established library drains over successive sweeps, oldest first, rather than
 # issuing one nearest-neighbour query per image of the whole history at once.
 TRICKLE_BATCH = 300
@dataclass
 class _Drop:
    post: Post
    members: set[int]
    first_at: datetime
    last_at: datetime
    names: set[str]
    images: list[int]
    nearest: set[int] | None
 async def _nearest_message(
    session: AsyncSession, *, artist_id: int, image_id: int, exclude: set[int],
 ) -> int | None:
    """The post that owns the nearest image in the artist's whole library.
    The whole LIBRARY, not this source, because that is what was measured: a
    neighbour that turns out to be a Patreon re-post simply yields no Discord
    drop to merge into, which errs toward leaving things alone. `exclude` is
    the drop's own messages — an image is always nearest to its own siblings
    in the same drop, which says nothing.
    """
    embedding = (await session.execute(
        select(ImageRecord.siglip_embedding).where(ImageRecord.id == image_id)
    )).scalar_one_or_none()
    if embedding is None:
        return None
    stmt = (
        select(ImageRecord.primary_post_id)
        .where(
            ImageRecord.artist_id == artist_id,
            ImageRecord.id != image_id,
            ImageRecord.siglip_embedding.is_not(None),
            ImageRecord.primary_post_id.is_not(None),
        )
        .order_by(ImageRecord.siglip_embedding.cosine_distance(embedding))
        .limit(1)
    )
    if exclude:
        stmt = stmt.where(ImageRecord.primary_post_id.not_in(exclude))
    return (await session.execute(stmt)).scalar_one_or_none()
 async def _load_drops(session: AsyncSession, source: Source) -> list[_Drop]:
    """Every live drop of this source, with what the merge rule reads, oldest first."""
    posts = (await session.execute(
        select(Post).where(
            Post.source_id == source.id,
            Post.synthesized_by == DROP_GROUPER,
            Post.absorbed_by_post_id.is_(None),
        )
    )).scalars().all()
    if not posts:
        return []
    by_id = {p.id: p for p in posts}
    msg_at = func.coalesce(Post.post_date, Post.downloaded_at)
    members: dict[int, set[int]] = {pid: set() for pid in by_id}
    times: dict[int, list[datetime]] = {pid: [] for pid in by_id}
    for mid, owner, at in (await session.execute(
        select(Post.id, Post.absorbed_by_post_id, msg_at)
        .where(Post.absorbed_by_post_id.in_(list(by_id)))
    )).all():
        members[owner].add(mid)
        times[owner].append(at)
    owner_of = {m: d for d, ms in members.items() for m in ms}
    names: dict[int, set[str]] = {pid: set() for pid in by_id}
    images: dict[int, list[int]] = {pid: [] for pid in by_id}
    if owner_of:
        carried = _message_images(list(owner_of))
        for iid, message, path in (await session.execute(
            select(ImageRecord.id, carried.c.post_id, ImageRecord.path)
            .join(carried, carried.c.image_id == ImageRecord.id)
            .order_by(ImageRecord.id)
        )).all():
            drop = owner_of[message]
            if iid in images[drop]:
                continue  # one file carried by two of the drop's messages
            images[drop].append(iid)
            if (name := leading_name(path)) is not None:
                names[drop].add(name)
    out = []
    for pid, post in by_id.items():
        if not times[pid]:
            continue
        stored = (post.synthesis_details or {}).get("nearest_message_ids")
        out.append(_Drop(
            post=post, members=members[pid],
            first_at=min(times[pid]), last_at=max(times[pid]),
            names=names[pid], images=images[pid],
            nearest=set(stored) if stored is not None else None,
        ))
    return sorted(out, key=lambda d: (d.first_at, d.post.id))
 async def _name_posts(session: AsyncSession, artist_id: int) -> Counter[str]:
    """Post-span counts of the artist's working names — the same corpus the
    teaser card and the announcement matcher count against."""
    by_post: dict[int, list[str]] = {}
    for pid, path in (await session.execute(
        select(ImageRecord.primary_post_id, ImageRecord.path).where(
            ImageRecord.artist_id == artist_id,
            ImageRecord.primary_post_id.is_not(None),
        )
    )).all():
        by_post.setdefault(pid, []).append(path)
    return token_frequencies(by_post.values())
 async def _repoint_associations(
    session: AsyncSession, *, from_id: int, to_id: int,
 ) -> None:
    """Move announcement links from a drop about to merge onto the one it joins.
    Without this the merge would silently undo a teaser link: the association's
    payload FK cascades on delete. Where the teaser already points at the
    surviving drop, the stronger claim is kept — a link over a proposal over a
    dismissal — and the duplicate goes.
    """
    rank = {"linked": 2, "pending": 1, "dismissed": 0}
    moving = (await session.execute(
        select(PostAssociation).where(PostAssociation.payload_post_id == from_id)
    )).scalars().all()
    for a in moving:
        existing = (await session.execute(
            select(PostAssociation).where(
                PostAssociation.announcement_post_id == a.announcement_post_id,
                PostAssociation.payload_post_id == to_id,
            )
        )).scalar_one_or_none()
        if existing is None:
            a.payload_post_id = to_id
            continue
        if rank.get(a.status, 0) > rank.get(existing.status, 0):
            existing.status = a.status
            existing.linked_by = a.linked_by
        await session.delete(a)
    await session.flush()
 async def merge_trickles(
    session: AsyncSession,
    source: Source,
    *,
    gap: timedelta,
    min_images: int,
    cooldown: timedelta,
    batch: int = TRICKLE_BATCH,
 ) -> int:
    """Fold later drops of the same piece into the earlier one. Returns merges."""
    drops = await _load_drops(session, source)
    if len(drops) < 2:
        return 0
    name_posts = await _name_posts(session, source.artist_id)
    def gated(names: set[str]) -> set[str]:
        return {n for n in names if rarity(name_posts.get(n, 0), FAMILY_MAX_POSTS) > 0}
    alive: list[_Drop] = []
    merged = 0
    checked = 0
    for drop in drops:
        details = drop.post.synthesis_details or {}
        if details.get("trickle_checked"):
            alive.append(drop)
            continue
        if checked >= batch:
            # Unchecked and out of budget: still a candidate for LATER drops'
            # reverse edges, just not examined itself this run.
            alive.append(drop)
            continue
        checked += 1
        if drop.nearest is None:
            found: set[int] = set()
            for iid in drop.images:
                pid = await _nearest_message(
                    session, artist_id=source.artist_id, image_id=iid,
                    exclude=drop.members,
                )
                if pid is not None:
                    found.add(pid)
            drop.nearest = found
        mine = gated(drop.names)
        targets: dict[int, tuple[_Drop, str]] = {}
        for earlier in alive:
            if drop.first_at - earlier.last_at > gap:
                continue
            shared = mine & gated(earlier.names)
            if set(drop.images) & set(earlier.images):
                # The creator posted the very same file again — the strongest
                # evidence there is, and one nearest-neighbour cannot see: it
                # skips the image itself, which is the one they share.
                targets[earlier.post.id] = (earlier, "same_image")
            elif shared:
                targets[earlier.post.id] = (earlier, f"name:{min(shared)}")
            elif drop.nearest & earlier.members or (earlier.nearest or set()) & drop.members:
                targets[earlier.post.id] = (earlier, "nearest")
        record = dict(details)
        record["nearest_message_ids"] = sorted(drop.nearest)
        record["trickle_checked"] = True
        drop.post.synthesis_details = record
        if not targets or not _compatible(
            [gated(t.names) for t, _route in targets.values()] + [mine]
        ):
            alive.append(drop)
            continue
        # Every target is the same piece as this drop, so they are the same
        # piece as each other: fold them all into the earliest, then this drop.
        ordered = sorted(targets.values(), key=lambda tr: (tr[0].first_at, tr[0].post.id))
        into = ordered[0][0]
        for other, route in ordered[1:]:
            await _merge_drop(
                session, into=into, drop=other, route=route,
                min_images=min_images, cooldown=cooldown,
            )
            alive.remove(other)
            merged += 1
        await _merge_drop(
            session, into=into, drop=drop, route=ordered[0][1],
            min_images=min_images, cooldown=cooldown,
        )
        merged += 1
    return merged
 def _compatible(name_sets: list[set[str]]) -> bool:
    """May drops carrying these working names become one post?
    Refused only when two of them are NAMED AS DIFFERENT PIECES — both carry a
    gated name, and they share none. An unnamed drop (a canvas screenshot)
    fits anywhere, which is the whole of the Marin case: two early stages both
    nearest to the same later one are one trickle, not an ambiguity.
    What it does NOT refuse is a drop the creator made two pieces in
    themselves. Measured on artist 8: one November message carries both
    `AdL01_wip4` and `Year_20k_wip_z4`, so its drop holds both names, and a
    later drop of either piece joins it on its own name. That is the creator's
    co-posting carried forward — Discord shows those two together too — not a
    bridge FC built.
    """
    named = [n for n in name_sets if n]
    return all(a & b for i, a in enumerate(named) for b in named[i + 1:])
 async def _merge_drop(
    session: AsyncSession,
    *,
    into: _Drop,
    drop: _Drop,
    route: str,
    min_images: int,
    cooldown: timedelta,
 ) -> None:
    """Absorb `drop`'s messages into `into`, carry its links over, delete it.
    Growth is stamped at the merged messages' OWN time, not the wall clock.
    Merging history must not drag a two-year-old drop to the top of the feed,
    and the time the group actually grew is when those messages arrived.
    """
    await _repoint_associations(session, from_id=drop.post.id, to_id=into.post.id)
    grew_before = into.post.last_grew_at
    await _absorb_into(
        session, group=into.post, member_ids=sorted(drop.members),
        source_id=into.post.source_id, now=drop.last_at,
        min_images=min_images, cooldown=cooldown,
    )
    # Never backwards: a group that already grew later than these messages
    # keeps that later date.
    if grew_before is not None and grew_before > drop.last_at:
        into.post.last_grew_at = grew_before
    details = dict(into.post.synthesis_details or {})
    if grew_before is not None and grew_before > drop.last_at:
        details["last_grew_at"] = grew_before.isoformat()
    # The honesty rule, extended: a grouping FC invented says what it was
    # built from, and a merge says WHY — "name:svtt" or "nearest".
    details["merged"] = [
        *details.get("merged", []),
        {"post_id": drop.post.id, "route": route, "message_ids": sorted(drop.members)},
    ]
    details["nearest_message_ids"] = sorted((into.nearest or set()) | (drop.nearest or set()))
    into.post.synthesis_details = details
    into.members |= drop.members
    into.names |= drop.names
    into.images += drop.images
    into.nearest = (into.nearest or set()) | (drop.nearest or set())
    into.last_at = max(into.last_at, drop.last_at)
    await session.execute(delete(ImageProvenance).where(ImageProvenance.post_id == drop.post.id))
    await session.delete(drop.post)
    await session.flush()
 async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
    """Group every enabled Discord source. No-op when the switch is off.
@@ -612,6 +997,7 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
    if not settings.discord_grouping_enabled:
        return {
            "enabled": False, "sources": 0, "posts_created": 0, "images_joined": 0,
            "drops_merged": 0,
        }
    sources = (await session.execute(
@@ -626,6 +1012,7 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
    created = 0
    joined = 0
    merged = 0
    for source in sources:
        joined += await join_open_groups(
            session, source,
@@ -644,12 +1031,20 @@ async def sweep(session: AsyncSession, *, now: datetime | None = None) -> dict:
            window_minutes=window_minutes,
            now=now,
        )
        # Last, so the drops the two passes above just wrote are merged in
        # the same sweep rather than showing as singletons for an hour.
        merged += await merge_trickles(
            session, source,
            gap=timedelta(hours=float(settings.discord_group_close_after_hours)),
            min_images=int(settings.discord_group_resurface_min_images),
            cooldown=timedelta(hours=float(settings.discord_group_resurface_cooldown_hours)),
        )
    log.info(
        "discord drop grouping: %d source(s), %d synthetic post(s) created, "
-        "%d image(s) joined to open groups",
+        "%d image(s) joined to open groups, %d trickle drop(s) merged",
-        len(sources), created, joined,
+        len(sources), created, joined, merged,
    )
    return {
        "enabled": True, "sources": len(sources),
-        "posts_created": created, "images_joined": joined,
+        "posts_created": created, "images_joined": joined, "drops_merged": merged,
    }
@@ -0,0 +1,85 @@
 """Native Discord ingester — the Discord ADAPTER over `ingest_core.Ingester`.
 Thin counterpart to subscribestar_ingester (milestone 428). The walk's modes,
 both ledgers, cursor checkpointing and the post-first capture live in the core;
 this wires in the Discord client, downloader, ledger models and key.
 Two things differ from the cookie platforms:
  - Discord authenticates with a user TOKEN, so `auth_token` is the credential
    here rather than an argument accepted and ignored.
  - The body canary is off. It fails a walk whose first 30+ captured posts all
    came back without text, on the theory that a creator nearly always writes
    something; a Discord drop is routinely files and nothing else, so on
    Discord that is an ordinary backfill, not a broken parser.
 `campaign_id` is the source URL (a server, channel, thread or category link).
 FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
 """
 from __future__ import annotations
 import asyncio
 from collections.abc import Callable
 from pathlib import Path
 from ..models import DiscordFailedMedia, DiscordSeenMedia
 from .discord_client import DiscordAPIError, DiscordClient, MediaItem
 from .discord_downloader import DiscordDownloader
 from .ingest_core import Ingester
 _LEDGER_KEY_MAX = 128
 def _ledger_key(media: MediaItem) -> str:
    """`<message_id>:<media_id>` — stable across edits (see MediaItem)."""
    return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX]
 class DiscordIngester(Ingester):
    """Walk a Discord source's channels, download unseen files, return a
    `DownloadResult`. `client` / `downloader` are injectable for tests."""
    def __init__(
        self,
        images_root: Path,
        cookies_path: str | None,
        session_factory: Callable[[], object],
        *,
        validate: bool = True,
        rate_limit: float = 0.0,
        request_sleep: float = 0.0,
        auth_token: str | None = None,
        client: DiscordClient | None = None,
        downloader: DiscordDownloader | None = None,
    ):
        del cookies_path  # Discord authenticates by token (uniform signature)
        self.images_root = Path(images_root)
        super().__init__(
            client=client if client is not None else DiscordClient(
                auth_token, request_sleep=request_sleep,
            ),
            downloader=downloader if downloader is not None else DiscordDownloader(
                self.images_root, validate=validate, rate_limit=rate_limit,
            ),
            session_factory=session_factory,
            seen_model=DiscordSeenMedia,
            failed_model=DiscordFailedMedia,
            seen_constraint="uq_discord_seen_media_source_id",
            failed_constraint="uq_discord_failed_media_source_id",
            ledger_key=_ledger_key,
            platform="discord",
            error_base=DiscordAPIError,
            drift_label="Discord API",
            body_canary=False,
        )
 async def verify_discord_credential(url: str, auth_token: str | None) -> tuple[bool | None, str]:
    """The uniform `(ok, message)` probe: is the token valid, and can its
    account see the channel or server the source names?"""
    if not auth_token:
        return False, "No Discord token is saved — add one under Credentials."
    client = DiscordClient(auth_token)
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(None, client.verify_auth, url)
@@ -0,0 +1,183 @@
 """Repair the Discord downloads made before the naming fix (issue #3999).
 Until dc840fe, gallery-dl's Discord patterns asked for keys the extractor never
 emits, so every Discord download landed as
 `<artist>/discord/None/<date>_None_<original name>`, next to a sidecar named
 after the attachment's ORIGINAL name. That broke two things:
 * **No Post, no date.** `find_sidecar` can never pair those names, so these
  files were imported as loose images with no Post, and a card shows the
  download time.
 * **No trustworthy metadata to relink from.** Every `image.png` in a channel
  wrote the same `image.json`, so the surviving sidecar describes whichever
  message was written last. The message id is gone from the filename too.
 The operator chose a clean re-download (2026-09-13) over relinking in place:
 delete the broken files and their records, make gallery-dl forget it fetched
 them, and backfill every Discord source again under the fixed naming.
 ## Why the archive is cleared for ALL of Discord
 gallery-dl records a download as `discord{message_id}_{num}` (upstream
 `DiscordExtractor.archive_fmt`, prefixed with the category). The broken files
 lost their message ids, so there is no way to forget one source's entries and
 not another's. Every Discord download made before the fix is broken, so
 forgetting all of them is exactly right. Anything downloaded AFTER the fix still
 exists on disk under its correct name, and gallery-dl's `skip` sees the file and
 does not fetch it again.
 ## What it does not touch
 Discord posts FC grouped itself (#388 E2) are built from Posts, and these files
 never had one, so there is nothing grouped to unwind. Images outside a
 `discord/None/` directory are never selected: the path filter requires both the
 `None` directory and the `_None_` filename, the pair only the bug produced.
 Operator-triggered only (Settings, preview first). Never on a beat.
 """
 from __future__ import annotations
 import logging
 import sqlite3
 from pathlib import Path
 from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 from ..models import ImageRecord, Source
 from .cleanup_service import delete_images
 from .gallery_dl import archive_path
 from .source_service import arm_backfill
 log = logging.getLogger(__name__)
 # `%` and `_` are LIKE wildcards, so the literal underscores around None are
 # escaped. `________` is the eight-digit date prefix the old pattern wrote.
 _BROKEN_PATH_LIKE = r"%/discord/None/________\_None\_%"
 # Upstream keys asset downloads as `asset_{server_id}_{id}`. FC never fetches
 # server assets, but excluding them keeps this to exactly the message
 # attachments the bug mangled.
 _ARCHIVE_SQL_MATCH = r"entry LIKE 'discord%' AND entry NOT LIKE 'discordasset\_%' ESCAPE '\'"
 _COUNT_SQL = "SELECT COUNT(*) FROM archive WHERE " + _ARCHIVE_SQL_MATCH
 _DELETE_SQL = "DELETE FROM archive WHERE " + _ARCHIVE_SQL_MATCH
 def broken_directories(images_root: Path) -> list[Path]:
    """Every `<artist>/discord/None` directory. Artist folders that differ only by
    case (`Conto` and `conto`) are separate directories and both are found."""
    return sorted(d for d in Path(images_root).glob("*/discord/None") if d.is_dir())
 def count_archive_entries(archive: Path) -> int:
    return _archive(archive, delete=False)
 def forget_archive_entries(archive: Path) -> int:
    return _archive(archive, delete=True)
 def _archive(archive: Path, *, delete: bool) -> int:
    if not archive.is_file():
        return 0
    # A download running at the same moment holds this file briefly. Waiting
    # 30s for its lock beats failing the repair over a transient contention.
    conn = sqlite3.connect(str(archive), timeout=30)
    try:
        has_table = conn.execute(
            "SELECT 1 FROM sqlite_master WHERE type='table' AND name='archive'"
        ).fetchone()
        if not has_table:
            return 0
        if not delete:
            return conn.execute(_COUNT_SQL).fetchone()[0]
        cur = conn.execute(_DELETE_SQL)
        conn.commit()
        return cur.rowcount
    finally:
        conn.close()
 def _sweep_directory(directory: Path) -> tuple[int, bool]:
    """Remove what the record deletes left behind: the collided sidecars, plus any
    file that never became a record (a quarantined or rejected download). Then
    the directory itself, if it is empty. Returns (files removed, dir removed)."""
    removed = 0
    for f in directory.iterdir():
        if f.is_file():
            try:
                f.unlink()
                removed += 1
            except OSError as exc:
                log.warning("discord repair: could not remove %s: %s", f, exc)
    try:
        directory.rmdir()
        return removed, True
    except OSError:
        return removed, False
 def repair_discord_downloads(
    session: Session, *, images_root: Path, dry_run: bool,
 ) -> dict:
    images_root = Path(images_root)
    archive = archive_path(images_root)
    broken = select(ImageRecord.id, ImageRecord.size_bytes).where(
        ImageRecord.path.like(_BROKEN_PATH_LIKE, escape="\\")
    )
    rows = session.execute(broken).all()
    image_ids = [r.id for r in rows]
    directories = broken_directories(images_root)
    sources = session.execute(
        select(Source).where(Source.platform == "discord")
    ).scalars().all()
    summary = {
        "images": len(image_ids),
        "bytes": sum(r.size_bytes or 0 for r in rows),
        "directories": len(directories),
        "sources": len(sources),
        "enabled_sources": sum(1 for s in sources if s.enabled),
    }
    if dry_run:
        summary["archive_entries"] = count_archive_entries(archive)
        return summary
    deleted = delete_images(session, image_ids=image_ids, images_root=images_root)
    swept = 0
    directories_removed = 0
    for d in directories:
        n, gone = _sweep_directory(d)
        swept += n
        directories_removed += int(gone)
    # Only after the files are gone. Forgetting first and failing half way would
    # leave gallery-dl free to re-fetch into a directory still full of the old
    # copies.
    forgotten = forget_archive_entries(archive)
    for source in sources:
        arm_backfill(source)
    session.commit()
    remaining = session.execute(
        select(func.count(ImageRecord.id)).where(
            ImageRecord.path.like(_BROKEN_PATH_LIKE, escape="\\")
        )
    ).scalar_one()
    summary.update(
        images_deleted=deleted["images_deleted"],
        files_failed=deleted["files_failed"],
        leftover_files_removed=swept,
        directories_removed=directories_removed,
        archive_entries=forgotten,
        backfills_started=len(sources),
        remaining=remaining,
    )
    log.info("discord repair applied: %s", summary)
    return summary
@@ -23,18 +23,22 @@ from __future__ import annotations
 import asyncio
 from pathlib import Path
 from .discord_ingester import DiscordIngester
 from .gallery_dl import DownloadResult, ErrorType
 from .ingest_core import DEFAULT_REVISIT_DAYS
 from .patreon_ingester import PatreonIngester
 from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
 from .pixiv_client import user_id_from_url
 from .pixiv_ingester import PixivIngester
 from .platforms import known_platform_keys
 from .subscribestar_ingester import SubscribeStarIngester
 # Platforms whose download + verify go through the native ingester rather than
-# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord) until
+# gallery-dl. gallery-dl still serves the rest (hentaifoundry) until it
-# they migrate too. pixiv left this set when it was retired (milestone #406).
+# migrates too. Discord joined in milestone 428.
-NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar"})
+NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar", "discord"})
 # Native platforms whose feed id IS the source URL, so there is nothing to
 # resolve: SubscribeStar's creator page, Discord's server/channel link.
 _URL_IS_FEED = frozenset({"subscribestar", "discord"})
 def _unsupported_platform_message(platform: str) -> str | None:
@@ -67,8 +71,8 @@ def _native_ingester_cls(platform: str):
    dispatch pick up the replacement."""
    return {
        "patreon": PatreonIngester,
        "pixiv": PixivIngester,
        "subscribestar": SubscribeStarIngester,
        "discord": DiscordIngester,
    }[platform]
@@ -86,6 +90,7 @@ async def run_download(
    mode: str | None,
    gdl,
    sync_session_factory,
    revisit_days: int = DEFAULT_REVISIT_DAYS,
 ) -> tuple[DownloadResult, str | None]:
    """Uniform download across backends — the download counterpart to
    `verify_source_credential`, so this module is the ONE place that knows how
@@ -109,7 +114,7 @@ async def run_download(
        ), None
    if uses_native_ingester(platform):
        return await _run_native_ingester(
-            ctx, source_config, mode, gdl, sync_session_factory
+            ctx, source_config, mode, gdl, sync_session_factory, revisit_days
        )
    result = await gdl.download(
        url=ctx["url"],
@@ -127,25 +132,17 @@ async def _resolve_native_campaign_id(
    platform: str, url: str, cookies_path: str | None, overrides: dict,
 ) -> tuple[str | None, str | None]:
    """`(campaign_id, resolved_campaign_id)` for a native source. SubscribeStar's
-    feed id IS the creator URL; Pixiv's is the numeric user id parsed straight
+    and Discord's feed id IS the source URL (no lookup → resolved None). Patreon
-    from it (no lookup → resolved None either way). Patreon resolves the
+    resolves the campaign id from the vanity URL (resolved non-None when a lookup
-    campaign id from the vanity URL (resolved non-None when a lookup actually ran,
+    actually ran, so phase 3 caches it)."""
-    so phase 3 caches it)."""
+    if platform in _URL_IS_FEED:
    if platform == "subscribestar":
        return url, None
    if platform == "pixiv":
        return user_id_from_url(url), None
    return await resolve_campaign_id_for_source(url, cookies_path, overrides)
 def _campaign_resolution_error(platform: str, url: str) -> str:
    """Operator-facing message for a native source whose campaign id could not
    be resolved — names the platform's own lookup mechanism."""
    if platform == "pixiv":
        return (
            f"Could not extract a pixiv user id. source_url={url!r} — expected "
            "a URL like https://www.pixiv.net/users/<id>."
        )
    vanity = extract_vanity(url)
    return (
        f"Could not resolve Patreon campaign id. source_url={url!r}; "
@@ -157,6 +154,7 @@ def _campaign_resolution_error(platform: str, url: str) -> str:
 async def _run_native_ingester(
    ctx: dict, source_config, mode: str | None, gdl, sync_session_factory,
    revisit_days: int = DEFAULT_REVISIT_DAYS,
 ) -> tuple[DownloadResult, str | None]:
    """Run the native ingester for a native platform in a worker thread (sync
    requests/subprocess). Patreon resolves a campaign id from the vanity URL;
@@ -172,8 +170,8 @@ async def _run_native_ingester(
        platform, ctx["url"], ctx["cookies_path"], overrides
    )
    if not campaign_id:
-        # Patreon: vanity lookup failed. Pixiv: no numeric user id in the URL.
+        # Patreon: vanity lookup failed. (SubscribeStar's campaign id is the
-        # (SubscribeStar's campaign id is the URL itself — never lands here.)
+        # URL itself — never lands here.)
        url = ctx["url"]
        return (
            DownloadResult(
@@ -205,7 +203,7 @@ async def _run_native_ingester(
        validate=gdl._validate_files,
        rate_limit=rate_limit,
        request_sleep=request_sleep,
-        # Uniform across adapters: token platforms (pixiv) authenticate with
+        # Uniform across adapters: a token platform would authenticate with
        # it, cookie platforms accept-and-ignore — so this construction stays
        # platform-agnostic.
        auth_token=ctx["auth_token"],
@@ -221,6 +219,9 @@ async def _run_native_ingester(
            mode=mode,
            resume_cursor=source_config.resume_cursor,
            time_budget_seconds=source_config.timeout,
            # How far back a tick keeps looking for EDITED posts. The ingester
            # applies it to ticks only; a backfill ignores it.
            revisit_days=revisit_days,
            posts_base=int(overrides.get("_backfill_posts", 0)),
            # plan #709: live progress writes to this running event mid-walk.
            event_id=ctx.get("event_id"),
@@ -252,16 +253,15 @@ async def verify_source_credential(
    if uses_native_ingester(platform):
        # Native ingester platforms verify via their own lightweight auth probe.
        # SubscribeStar's probe takes the creator URL directly; Patreon's
-        # resolves the campaign id first; Pixiv's is one OAuth refresh (the
+        # resolves the campaign id first.
        # exact call that fails when the token is bad — no feed walk).
        if platform == "subscribestar":
            from .subscribestar_ingester import verify_subscribestar_credential
            return await verify_subscribestar_credential(url, cookies_path, config_overrides)
-        if platform == "pixiv":
+        if platform == "discord":
-            from .pixiv_ingester import verify_pixiv_credential
+            from .discord_ingester import verify_discord_credential
-            return await verify_pixiv_credential(auth_token)
+            return await verify_discord_credential(url, auth_token)
        from .patreon_ingester import verify_patreon_credential
        return await verify_patreon_credential(url, cookies_path, config_overrides)
@@ -39,6 +39,7 @@ from .gallery_dl import (
    walk_completed,
 )
 from .importer import Importer
 from .ingest_core import DEFAULT_REVISIT_DAYS
 from .platforms import auth_type_for
 from .scheduler_service import set_platform_cooldown
@@ -60,6 +61,7 @@ class DownloadService:
        importer: Importer,
        cred_service: CredentialService,
        sync_session_factory=None,
        revisit_days: int = DEFAULT_REVISIT_DAYS,
    ):
        self.async_session = async_session
        self.sync_session = sync_session
@@ -71,6 +73,12 @@ class DownloadService:
        # the multi-minute walk — see PatreonIngester). Only the patreon branch
        # of phase 2 uses it; gallery-dl sources leave it None.
        self.sync_session_factory = sync_session_factory
        # ImportSettings.download_revisit_days — how far back a tick keeps
        # looking for EDITED posts (ingest_core.DEFAULT_REVISIT_DAYS). Passed in
        # rather than read here because the task already loads the settings row
        # for rate_limit/validate_files, and a second load on every download
        # would be the same row twice for one number.
        self.revisit_days = revisit_days
    async def download_source(self, source_id: int) -> int:
        """Returns DownloadEvent.id. Idempotent: in-flight events are returned as-is."""
@@ -178,6 +186,7 @@ class DownloadService:
        return await run_download(
            ctx=ctx, source_config=source_config, skip_value=skip_value, mode=mode,
            gdl=self.gdl, sync_session_factory=self.sync_session_factory,
            revisit_days=self.revisit_days,
        )
    async def _phase1_setup(self, source_id: int) -> dict[str, Any]:
@@ -414,6 +423,13 @@ class DownloadService:
            await loop.run_in_executor(None, _upsert)
        # Only now is it safe to call this walk's media seen: every file above
        # has been through the importer. Had the run died before here they stay
        # unmarked, and the next walk imports them from disk (ingest_core).
        mark_seen = getattr(dl_result, "mark_seen_after_import", None)
        if mark_seen is not None:
            await loop.run_in_executor(None, mark_seen)
        # #830 recapture: backfill source_filehash on EXISTING on-disk images so
        # their post-body inline <img src=CDN> remaps to the local copy. A
        # SEPARATE non-deleting channel (NOT the import list — that would unlink
@@ -21,6 +21,9 @@ from .source_service import BACKFILL_MAX_CHUNKS
 log = logging.getLogger(__name__)
 # The probe runs while the chip is drawing; names that take longer are skipped.
 _NAME_LOOKUP_SECONDS = 6.0
 class UnknownPlatformError(Exception):
    """URL didn't match any platform pattern."""
@@ -30,6 +33,10 @@ class InvalidUrlError(Exception):
    """URL was empty or missing a scheme."""
 class UnknownArtistError(Exception):
    """quick-add named an `artist_id` that does not exist."""
 # Mirrored byte-for-byte from extension/lib/platforms.js
 # PLATFORM_ARTIST_PATTERNS. Keep these two copies in sync by hand —
 # reviewers catch drift.
@@ -55,40 +62,104 @@ _PLATFORM_PATTERNS: list[tuple[str, re.Pattern[str]]] = [
        r"^https?://(?:www\.)?hentai-foundry\.com/user/(?P<slug>[^/?#]+)",
        re.IGNORECASE,
    )),
    # A Discord URL names a server or a channel, never a creator, so the slug is
    # `<server>` or `<server>/<channel>` and the artist is chosen, not derived.
    # A trailing message id (a jump link) still names its channel. DMs (`@me`)
    # are not sources; thread links (`/threads/`) are left to the manual form.
    ("discord", re.compile(
        r"^https?://(?:www\.|ptb\.|canary\.)?discord\.com/channels/"
        r"(?P<slug>\d+(?:/\d+)?)(?:/\d+)?/?(?:[?#].*)?$",
        re.IGNORECASE,
    )),
 ]
 DISCORD = "discord"
 def canonical_source_url(platform: str, url: str, slug: str) -> str:
    """The URL a new source is stored under. Discord's is rebuilt from the ids
    — the form the manual Add form and the ingester use — so a jump link, a
    ptb/canary host or a trailing slash never makes a second source for the
    same channel. Every other platform keeps the URL as given."""
    if platform == DISCORD:
        return f"https://discord.com/channels/{slug}"
    return url
 def _discord_ids(url: str) -> tuple[str | None, str | None] | None:
    """`(server_id, channel_id)` of a stored Discord source URL, None if it
    does not parse (a DM or thread link, or an old malformed row)."""
    from .discord_client import DiscordAPIError, parse_source_url
    try:
        return parse_source_url(url)
    except DiscordAPIError:
        return None
 class ExtensionService:
    def __init__(self, session: AsyncSession, crypto=None) -> None:
        self.session = session
-        # Optional decryptor for resolving a token-auth platform's display name
+        # Optional decryptor for resolving a platform's display name at
-        # (pixiv) at add-time. None → skip resolution, fall back to the handle.
+        # add-time. None → skip resolution, fall back to the handle.
        self._crypto = crypto
-    async def quick_add_source(self, url: str) -> dict:
+    async def quick_add_source(
        self,
        url: str,
        *,
        artist_id: int | None = None,
        artist_name: str | None = None,
    ) -> dict:
        """Add `url` as a source. `artist_id` connects it to an existing
        artist, `artist_name` to that artist (created if new); with neither,
        the artist is resolved from the platform as before."""
        platform, raw_slug = self._derive(url)
        url = canonical_source_url(platform, url, raw_slug)
        # Identity by SOURCE handle (#130): an existing (platform, url) source
        # keeps its artist on re-add — even if that artist was since renamed (its
-        # frozen slug no longer matches the current name). Only a genuinely new
+        # frozen slug no longer matches the current name), and even when the
-        # source resolves/creates an artist.
+        # add named a different artist. Only a genuinely new source
-        existing = (await self.session.execute(
+        # resolves/creates an artist.
-            select(Source).where(Source.platform == platform, Source.url == url)
+        existing = await self._existing_source(platform, url)
        )).scalar_one_or_none()
        if existing is not None:
            artist = (await self.session.execute(
                select(Artist).where(Artist.id == existing.artist_id)
            )).scalar_one()
            return self._shape(existing, artist, created_source=False, created_artist=False)
-        # New source → name the artist properly by resolving the real display
+        if artist_id is not None:
-        # name from the platform (falls back to the URL handle).
+            artist = (await self.session.execute(
-        name = await self._resolve_artist_name(platform, raw_slug, url)
+                select(Artist).where(Artist.id == artist_id)
-        artist, created_artist = await self._find_or_create_artist(name)
+            )).scalar_one_or_none()
            if artist is None:
                raise UnknownArtistError(f"no artist with id {artist_id}")
            created_artist = False
        else:
            name = (artist_name or "").strip()
            if not name:
                # Name the artist properly by resolving the real display name
                # from the platform (falls back to the URL handle).
                name = await self._resolve_artist_name(platform, raw_slug, url)
            artist, created_artist = await self._find_or_create_artist(name)
        source, created_source = await self._find_or_create_source(
            artist_id=artist.id, platform=platform, url=url,
        )
        return self._shape(source, artist, created_source, created_artist)
    async def _existing_source(self, platform: str, url: str) -> Source | None:
        """The source this URL already is, whichever artist owns it. Discord
        compares ids, not strings, so a row stored before canonicalisation (a
        ptb host, a trailing slash) is still found."""
        if platform != DISCORD:
            return (await self.session.execute(
                select(Source).where(Source.platform == platform, Source.url == url)
            )).scalars().first()
        want = _discord_ids(url)
        rows = (await self.session.execute(
            select(Source).where(Source.platform == DISCORD).order_by(Source.id)
        )).scalars().all()
        return next((s for s in rows if _discord_ids(s.url) == want), None)
    @staticmethod
    def _shape(source, artist, created_source: bool, created_artist: bool) -> dict:
        return {
@@ -112,12 +183,17 @@ class ExtensionService:
        self, platform: str, raw_slug: str, url: str
    ) -> str:
        """The real display name for a new artist, resolved from the platform at
-        add-time (#130). Our native platforms each have a name source — pixiv the
+        add-time (#130). Our native platforms each have a name source — patreon
-        app API (token), patreon the campaigns API, subscribestar the profile
+        the campaigns API, subscribestar the profile page (both cookies). Other
-        page (both cookies). Other platforms (and any failure — no credential,
+        platforms (and any failure — no credential, network error) fall back to
-        network error) fall back to the URL handle, which is already readable.
+        the URL handle, which is already readable.
        The resolvers are sync, so they run in an executor."""
-        if self._crypto is None or platform not in ("pixiv", "patreon", "subscribestar"):
+        if platform == DISCORD:
            # The server's name: what the operator knows the community as.
            server_id = raw_slug.split("/", 1)[0]
            names = await self._discord_names(server_id, None)
            return names.get("server") or f"Discord {server_id}"
        if self._crypto is None or platform not in ("patreon", "subscribestar"):
            return raw_slug
        import asyncio
@@ -125,15 +201,7 @@ class ExtensionService:
        cred = CredentialService(self.session, self._crypto)
        loop = asyncio.get_running_loop()
        try:
-            if platform == "pixiv":
+            if platform == "patreon":
                token = await cred.get_token("pixiv")
                if not token:
                    return raw_slug
                from .pixiv_client import PixivClient
                name = await loop.run_in_executor(
                    None, PixivClient(token).resolve_display_name, raw_slug
                )
            elif platform == "patreon":
                cookies = await cred.get_cookies_path("patreon")
                from .patreon_resolver import resolve_display_name
                name = await loop.run_in_executor(
@@ -174,6 +242,8 @@ class ExtensionService:
            platform, raw_slug = self._derive(url)
        except (UnknownPlatformError, InvalidUrlError):
            return {"state": "unknown_platform"}
        if platform == DISCORD:
            return await self._probe_discord(raw_slug)
        slug = slugify(raw_slug)
        artist = (await self.session.execute(
@@ -213,6 +283,106 @@ class ExtensionService:
            },
        }
    async def _probe_discord(self, raw_slug: str) -> dict:
        """probe for a Discord server or channel. The states mean what they
        mean elsewhere, but the artist is never read off the URL:
          - source_match: this channel is a source — or the whole server is
            (`covered_by_server`), which already walks every channel;
          - artist_match: another source on this server belongs to an artist,
            the one this channel most likely belongs to too (a suggestion the
            Add panel preselects, not a decision);
          - new: nothing on this server yet.
        `discord` carries the ids, both canonical URLs and the display names,
        read with the stored token; a name that can't be read is None."""
        server_id, _, channel_id = raw_slug.partition("/")
        channel_id = channel_id or None
        rows = (await self.session.execute(
            select(Source, Artist)
            .join(Artist, Artist.id == Source.artist_id)
            .where(Source.platform == DISCORD)
            .order_by(Source.id)
        )).all()
        exact = server_whole = on_server = None
        for source, artist in rows:
            ids = _discord_ids(source.url)
            if ids is None or ids[0] != server_id:
                continue
            if ids[1] == channel_id and exact is None:
                exact = (source, artist)
            elif ids[1] is None and server_whole is None:
                server_whole = (source, artist)
            if on_server is None:
                on_server = (source, artist)
        names = await self._discord_names(server_id, channel_id)
        base = f"https://discord.com/channels/{server_id}"
        result: dict = {
            "platform": DISCORD,
            "slug": raw_slug,
            "discord": {
                "server_id": server_id,
                "channel_id": channel_id,
                "server_name": names.get("server"),
                "channel_name": names.get("channel"),
                "server_url": base,
                "channel_url": f"{base}/{channel_id}" if channel_id else None,
            },
        }
        hit = exact or server_whole
        if hit is not None:
            source, artist = hit
            result.update(
                state="source_match",
                artist=self._artist_payload(artist),
                source=self._source_payload(source),
                covered_by_server=exact is None,
            )
        elif on_server is not None:
            result.update(state="artist_match", artist=self._artist_payload(on_server[1]))
        else:
            result["state"] = "new"
        return result
    async def _discord_names(self, server_id: str | None, channel_id: str | None) -> dict:
        """Server/channel display names via the stored Discord token. Never
        raises and never waits out a rate limit: it runs while the operator
        looks at a page, so a slow or missing answer just means no names."""
        if self._crypto is None:
            return {}
        import asyncio
        from .credential_service import CredentialService
        from .discord_client import DiscordClient
        try:
            token = await CredentialService(self.session, self._crypto).get_token(DISCORD)
            if not token:
                return {}
            client = DiscordClient(token, max_retries=0)
            loop = asyncio.get_running_loop()
            return await asyncio.wait_for(
                loop.run_in_executor(None, client.describe, server_id, channel_id),
                timeout=_NAME_LOOKUP_SECONDS,
            )
        except Exception as exc:      # names are decoration — never fail the call
            log.info("Discord name lookup failed: %s", exc)
            return {}
    @staticmethod
    def _artist_payload(artist) -> dict:
        return {"id": artist.id, "name": artist.name, "slug": artist.slug}
    @staticmethod
    def _source_payload(source) -> dict:
        return {
            "id": source.id,
            "artist_id": source.artist_id,
            "platform": source.platform,
            "url": source.url,
            "enabled": source.enabled,
        }
    def _derive(self, url: str) -> tuple[str, str]:
        if not isinstance(url, str) or not url.strip():
            raise InvalidUrlError("url is empty")
@@ -17,6 +17,7 @@ import subprocess
 import sys
 import tempfile
 import time
 from collections.abc import Callable
 from dataclasses import dataclass, field
 from datetime import UTC, datetime
 from enum import StrEnum
@@ -94,6 +95,16 @@ BACKFILL_CHUNK_SECONDS = 600
 _DEFAULT_GDL_TIMEOUT_SECONDS = 870
 def archive_path(images_root: Path) -> Path:
    """gallery-dl's download archive: the record of what it has already fetched.
    One definition, because the Discord repair (services/discord_repair.py) has
    to find the same file the downloader writes, without constructing a service
    whose __init__ creates directories.
    """
    return Path(images_root) / ".gallery-dl" / "archive.sqlite3"
@dataclass
 class SourceConfig:
    """Per-source overrides loaded from Source.config_overrides JSON.
@@ -178,6 +189,13 @@ class DownloadResult:
    # the platform cooldown matches the hint instead of a flat default. None when
    # unknown (no header, or not a rate-limit failure).
    retry_after_seconds: float | None = None
    # Native ingester only: marks this walk's fetched media seen in its ledger.
    # Phase 3 calls it AFTER the import loop, never before — a file marked seen
    # but not yet imported is invisible to every later walk, so a run killed in
    # between orphaned it for good (TamadaHeijun's 12PCG post lost 7 of 13
    # images to a stranded run, 2026-09-24). Unmarked, the next walk finds the
    # file on disk with no ImageRecord and imports it. None on gallery-dl.
    mark_seen_after_import: Callable[[], None] | None = None
 def extract_errors_warnings(stderr: str) -> str:
@@ -365,24 +383,16 @@ class GalleryDLService:
    # (services/patreon_ingester.py), not gallery-dl.
    PLATFORM_DEFAULTS = {
        # subscribestar removed — native-ingester platform now (#71); pixiv
-        # removed likewise (#129); deviantart removed at #3069 as a dropped
+        # removed likewise (#129); discord likewise (milestone 428, whose
-        # platform, not a migrated one. The remaining entries are the
+        # downloader keeps this config's on-disk naming); deviantart removed at
-        # gallery-dl platforms not yet migrated.
+        # #3069 as a dropped platform, not a migrated one. HentaiFoundry is the
        # one platform left here, by the operator's choice not to migrate it.
        "hentaifoundry": {
            "content_types": ["all"],
            "directory": [],
            "filename": "{category}_{index:>03}_{title[:50]}.{extension}",
            "include": "all",
        },
        "discord": {
            "content_types": ["all"],
            "directory": ["{channel[name]}"],
            "filename": "{date:%Y%m%d}_{id}_{filename}.{extension}",
            "embeds": "all",
            "stickers": True,
            "reactions": False,
            "threads": True,
        },
    }
    def __init__(
@@ -402,7 +412,7 @@ class GalleryDLService:
        config = {
            "extractor": {
                "base-directory": str(self.images_root),
-                "archive": str(self._config_dir / "archive.sqlite3"),
+                "archive": str(archive_path(self.images_root)),
                "skip": True,
                "sleep": self._rate_limit,
                "sleep-request": max(0.5, self._rate_limit / 4),
@@ -744,9 +754,6 @@ class GalleryDLService:
        if cookies_path:
            config["extractor"]["cookies"] = cookies_path
        if auth_token and platform == "discord":
            config["extractor"].setdefault("discord", {})
            config["extractor"]["discord"]["token"] = auth_token
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
@@ -930,8 +937,6 @@ class GalleryDLService:
        config = self._build_config_for_source(platform, source_config, artist_slug)
        if cookies_path:
            config["extractor"]["cookies"] = cookies_path
        if auth_token and platform == "discord":
            config["extractor"].setdefault("discord", {})["token"] = auth_token
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".json", delete=False, dir=str(self._config_dir),
@@ -45,7 +45,7 @@ from .tag_query import (
 # provenance (filesystem imports). Returned by facets() as a null-valued
 # bucket; the frontend maps that null back to this sentinel in the URL so the
 # bucket is selectable. Underscore-wrapped so it can't collide with a real
-# gallery-dl platform name (patreon/pixiv/...).
+# gallery-dl platform name (patreon/hentaifoundry/...).
 UNSOURCED_PLATFORM = "__unsourced__"
@@ -322,7 +322,7 @@ def _gallery_images(rows, artists: dict[int, dict]) -> list[GalleryImage]:
    ]
-def _diversify_similar(src, rows, limit, *, dup_threshold=8, lam=0.40):
+def _diversify_similar(src, rows, limit, *, dup_threshold=32, lam=0.40):
    """Trim a nearest-cosine candidate pool down to `limit` diverse picks.
    1. pHash collapse: drop any candidate whose perceptual hash is within
@@ -338,6 +338,11 @@ def _diversify_similar(src, rows, limit, *, dup_threshold=8, lam=0.40):
    2026-07-01 — dropped 0.55→0.40, dup 6→8, paired with a wider pool in
    `similar()`).
    `dup_threshold` counts Hamming bits, so it moved 8→32 when the pHash went
    from 64 to 256 bits (#4223, migration 0098) — the same fraction of the
    hash, i.e. the tuning the operator chose, unchanged. This collapse is
    DISPLAY-only: it hides a near-dup from one rail, it never drops a record.
    Falls back to nearest-order (`rows[:limit]`) on any failure or a small pool.
    """
    if len(rows) <= 1:
@@ -33,13 +33,14 @@ from ..models import (
 )
 from ..utils import safe_probe
 from ..utils.paths import (
    canonical_subdir,
    derive_subdir,
    derive_top_level_artist,
    filehash_from_url,
    hash_suffixed_name,
    safe_ext,
 )
-from ..utils.phash import compute_phash, find_similar
+from ..utils.phash import compute_phash, find_similar, fingerprint_path, fingerprints_match
 from ..utils.sidecar import find_sidecar, parse_sidecar
 from ..utils.slug import slugify
 from .archive_extractor import extract_archive, is_archive
@@ -234,6 +235,38 @@ class Importer:
                (phash, width or 0, height or 0, image_id)
            )
    def _pixel_confirmer(self, source: Path):
        """Build `find_similar`'s gate-3 callback for an incoming file.
        pHash proposes; this accepts. A candidate is a duplicate only if its
        file really is the same picture as `source` at a different size —
        which is the only merge the operator asked for (#4223). Everything
        else (a missing file, an unreadable one, a deleted row) returns
        False: the destructive outcomes here are dropping a download and
        overwriting a kept file, so an unanswerable question must not read
        as "yes".
        Both sides' fingerprints are computed lazily and cached, so an
        import that matches nothing costs no I/O at all and an archive
        member that keeps hitting the same candidate pays for it once.
        """
        new_fp: list = []
        cand_fps: dict[int, object] = {}
        def confirm(candidate_id: int) -> bool:
            if not new_fp:
                new_fp.append(fingerprint_path(source))
            if new_fp[0] is None:
                return False
            if candidate_id not in cand_fps:
                rec = self.session.get(ImageRecord, candidate_id)
                cand_fps[candidate_id] = (
                    fingerprint_path(Path(rec.path)) if rec and rec.path else None
                )
            return fingerprints_match(new_fp[0], cand_fps[candidate_id])
        return confirm
    def _get_or_create(self, stmt, factory):
        """Race-safe find-or-create. Run `stmt` (scalar_one_or_none); if a
        row exists, return it. Otherwise open a savepoint and INSERT
@@ -862,6 +895,7 @@ class Importer:
                rel, match_id = find_similar(
                    phash, width or 0, height or 0,
                    candidates, self.settings.phash_threshold,
                    confirm=self._pixel_confirmer(source),
                )
                if rel == "larger_exists":
                    # Enrich-on-duplicate (parity with attach_in_place).
@@ -911,7 +945,7 @@ class Importer:
                )
                return ImportResult(status="superseded", image_id=match_id)
-        dest = self._copy_to_library(source, sha, attribution_path)
+        dest = self._copy_to_library(source, sha, attribution_path, path_artist)
        record = ImageRecord(
            path=str(dest),
@@ -1241,6 +1275,7 @@ class Importer:
                rel, match_id = find_similar(
                    phash, width or 0, height or 0,
                    candidates, self.settings.phash_threshold,
                    confirm=self._pixel_confirmer(path),
                )
                if rel == "larger_exists":
                    # Enrich-on-duplicate: link the near-dup's post to the
@@ -1566,7 +1601,8 @@ class Importer:
        )
    def _copy_to_library(
-        self, source: Path, sha: str, attribution_path: Path
+        self, source: Path, sha: str, attribution_path: Path,
        artist: Artist | None = None,
    ) -> Path:
        """Copy `source` to its final library path. Returns the destination.
@@ -1574,8 +1610,18 @@ class Importer:
        _import_media (filesystem scan) and _supersede (when new_path is
        not passed). FC-3c's attach_in_place skips this helper entirely
        — the file is already at its final home.
        `artist`, when resolved, decides the top-level directory: the
        library is keyed on the Artist row's slug, NOT on however the
        import folder happened to be capitalised. Without that, an import
        from `/import/Conto/` and a download for the same artist write to
        `Conto/` and `conto/` respectively and the library grows a second
        home for one artist (milestone #421).
        """
-        subdir = derive_subdir(attribution_path, self.import_root)
+        subdir = canonical_subdir(
            derive_subdir(attribution_path, self.import_root),
            artist.slug if artist else None,
        )
        dest_dir = self.images_root / subdir if subdir else self.images_root
        dest_dir.mkdir(parents=True, exist_ok=True)
        dest_name = hash_suffixed_name(source.stem, sha, source.suffix)
@@ -1610,7 +1656,16 @@ class Importer:
        that path (FC-3c attach_in_place case) — skip the copy step.
        Otherwise the file is copied via _copy_to_library."""
        if new_path is None:
-            dest = self._copy_to_library(source, sha, source)
+            # The KEPT row's artist decides the destination, not the incoming
            # file's folder — a supersede rewrites `existing.path`, so writing
            # it anywhere but that artist's canonical directory would move a
            # row OUT of the tree milestone #421 is consolidating. ImageRecord
            # carries `artist_id` with no relationship attribute, so this is a
            # lookup rather than `existing.artist`.
            kept_artist = artist
            if kept_artist is None and existing.artist_id is not None:
                kept_artist = self.session.get(Artist, existing.artist_id)
            dest = self._copy_to_library(source, sha, source, kept_artist)
        else:
            dest = new_path
@@ -31,10 +31,12 @@ import json
 import logging
 import time
 from collections.abc import Callable
 from datetime import UTC, datetime, timedelta
 from sqlalchemy import delete, func, select, text
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from ..models import ImageRecord
 from .gallery_dl import (
    DownloadResult,
    ErrorType,
@@ -51,6 +53,33 @@ log = logging.getLogger(__name__)
 # per-file HEADs. Headroom against paywalled/undownloadable items interleaving.
 _TICK_SEEN_THRESHOLD = 20
 # How far back a tick keeps looking even once everything is already-have-it —
 # the REVISIT WINDOW. Operator, 2026-09-23, holding up a Floppystack post:
 # *"this post has been updated as he implements hot fixes — any chance we have a
 # way to scan for or see updated posts so we can update ours to match and pull
 # the new attachments and pictures etc."*
 #
 # A creator who edits a three-day-old post to append a hotfix build was
 # structurally unreachable: that post sits twenty-odd already-seen items down
 # the feed, so the count early-out above fired before the walk ever got to it.
 # Not a bug in the early-out — a COUNT cannot express "recent".
 #
 # So the early-out now needs BOTH conditions: the run of already-seen items AND
 # a post published before the horizon. Strictly a widening. Two properties this
 # shape has and a plain "walk the last N days" would not:
 #
 #   * window 0 is exactly the old behaviour, so the feature has an off switch
 #     that costs nothing to reason about;
 #   * no window can make a tick stop EARLIER than it used to. A source paused
 #     for months has an unseen backlog stretching well past any horizon, and
 #     the walk still runs to the end of it — the horizon is a FLOOR on how far
 #     to look, never a ceiling.
 #
 # The live value is `ImportSettings.download_revisit_days` (rule 25 — an
 # operator tuning how far back their creators edit should not need a redeploy).
 # This is the fallback for a caller that passes none.
 DEFAULT_REVISIT_DAYS = 30
 # plan #705 #7: after this many failed download/validate attempts a media is
 # "dead-lettered" and skipped on routine tick/backfill walks (recovery still
 # re-attempts it). Stops a permanently-broken media re-erroring forever.
@@ -77,6 +106,44 @@ _LIVE_PROGRESS_INTERVAL = 5.0
 # recapture (the operator's schema-test flow) reaches the sample.
 _CANARY_MIN_SAMPLE = 30
 # The walk's time budget covers only the walk, but phase 3 runs in the SAME
 # Celery task, under the same soft limit (tasks/download.py: 1350s). A walk that
 # finds a lot of work for phase 3 must stop early and leave it to the next chunk,
 # or phase 3 is killed mid-import: TamadaHeijun's recapture, 2026-09-24, walked
 # for ~2 min and then spent 20 min importing 431 orphans and relinking ~3000
 # on-disk files, and died at the soft limit.
 #
 # So the walk also stops when its elapsed time PLUS phase 3's estimated cost
 # would pass CHUNK_TOTAL_SECONDS. Costs measured on the live instance: 431
 # imports took 976s (~2.3s each: hash, pHash, sidecar, provenance); a relink is
 # a sha256 over NFS, 0.15s for the 8.8 MB average file, plus a lookup.
 # test_download_source_task pins CHUNK_TOTAL_SECONDS under the soft limit.
 CHUNK_TOTAL_SECONDS = 1200.0
 PHASE3_IMPORT_SECONDS = 2.5
 PHASE3_RELINK_SECONDS = 0.25
 def _parse_published(raw: object) -> datetime | None:
    """An ISO-8601 post date from either native client, as aware UTC.
    Patreon's `published_at` is tz-aware with a `Z` or `+00:00` offset;
    SubscribeStar's is NAIVE (`_parse_ss_datetime` renders a parsed local
    timestamp with no zone). A naive value is read as UTC — the alternative is
    discarding it, and a post whose date we refuse to read is a post the revisit
    window can never reach.
    Anything unparseable returns None, which reads downstream as "not provably
    recent" and leaves the walk on its pre-revisit behaviour. Never raises: a
    date we cannot read must not fail a walk that is otherwise working.
    """
    if not isinstance(raw, str) or not raw.strip():
        return None
    try:
        parsed = datetime.fromisoformat(raw.strip().replace("Z", "+00:00"))
    except ValueError:
        return None
    return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
 class Ingester:
    """Generic native-ingest orchestration. Subclass with a platform adapter
@@ -113,9 +180,9 @@ class Ingester:
        # (e.g. "Patreon API", "SubscribeStar markup").
        self._drift_label = drift_label or platform
        # #862 canary opt-out: platforms whose posts legitimately have empty
-        # bodies across large samples (pixiv — caption-less artists are common)
+        # bodies across large samples would false-positive the
-        # would false-positive the zero-bodies-means-drift alarm; their clients
+        # zero-bodies-means-drift alarm; their clients catch drift structurally
-        # catch drift structurally (response-shape checks) instead. The
+        # (response-shape checks) instead. The
        # "bodies X/N" summary line still surfaces the ratio either way.
        self._body_canary = body_canary
@@ -132,11 +199,17 @@ class Ingester:
        resume_cursor: str | None = None,
        time_budget_seconds: float = 870.0,
        seen_threshold: int = _TICK_SEEN_THRESHOLD,
        revisit_days: int = DEFAULT_REVISIT_DAYS,
        posts_base: int = 0,
        event_id: int | None = None,
    ) -> DownloadResult:
        """Walk + download for one source, returning a gallery-dl-shaped result.
        `revisit_days` is the tick's revisit window (see DEFAULT_REVISIT_DAYS):
        inside it a tick neither early-outs nor trusts the post-record gate, so
        a post edited after we first captured it is re-read and its new
        attachments downloaded. 0 turns the window off.
        `mode` is "tick" | "backfill" | "recovery" | "recapture". Recovery
        bypasses the tier-1 seen-ledger AND the dead-letter ledger (tier-2 disk
        still skips kept files). Recapture (#830) is the cheap "re-grab post
@@ -179,6 +252,29 @@ class Ingester:
        # no media download, no post-record stub. Absent on stub/not-yet-migrated
        # clients → nothing is ever treated as gated.
        post_is_gated = getattr(self.client, "post_is_gated", None)
        # The revisit window (see DEFAULT_REVISIT_DAYS). `post_meta` is an
        # existing client seam — both native clients already implement it, for
        # a preview sample whose caller has since gone, so this needed no new
        # contract, only a live consumer for one. Absent seam, an unreadable
        # date or a window of 0 → `horizon` never matches and the walk behaves
        # exactly as it did before 2026-09-23.
        #
        # The window applies to TICKS only. A backfill is gated on purpose
        # (capture each post once) and `recapture` mode already exists for the
        # operator-driven "re-read every body" pass; a horizon there would be a
        # third overlapping answer to a question that has two.
        post_meta = getattr(self.client, "post_meta", None)
        # #4413: optional client seam for a source that is several feeds walked
        # one after another (a Discord server: every channel and thread). The
        # tick early-out means "this feed has nothing new", and without the
        # seam it ends the WHOLE walk — so the first quiet channel would hide
        # every channel after it. With it, the early-out asks the client to
        # move on and the walk continues. Absent → the early-out ends the walk,
        # exactly as before (Patreon and SubscribeStar are one feed each).
        skip_feed = getattr(self.client, "skip_feed", None)
        horizon: datetime | None = None
        if mode == "tick" and revisit_days > 0 and post_meta is not None:
            horizon = datetime.now(UTC) - timedelta(days=revisit_days)
        start = time.monotonic()
        last_live = start  # plan #709: last live-progress write timestamp
        log_lines: list[str] = []
@@ -190,6 +286,9 @@ class Ingester:
        # source_filehash and (b) link the on-disk image to its Post (#1288) —
        # WITHOUT re-downloading or unlinking the file. Empty outside recapture.
        relink: list[tuple[str, str, str]] = []
        # Media handed to phase 3 for import. Marked seen by phase 3 once the
        # import has run (`mark_seen_after_import`), not here — see there.
        fetched: list[tuple[str, str]] = []
        downloaded = 0
        errors = 0
        quarantined = 0
@@ -210,11 +309,18 @@ class Ingester:
        # absolute across chunks instead of an inflating sum. posts_processed
        # stays the gross per-chunk count used for the run summary.
        chunk_new_posts = 0
        # Posts inside the revisit window that we had already captured, and the
        # media those revisits turned up. Reported in the run summary — the
        # operator's ask was to SEE the updated posts, not only to end up with
        # their files ("so we can update ours to match").
        revisited = 0
        revisit_downloads = 0
        consecutive_seen = 0
        emitted_cursor: str | None = None
        reached_bottom = False
        budget_hit = False
        early_out = False
        feeds_caught_up = 0  # #4413: feeds a tick left early via skip_feed
        stopped = False  # plan #708 B4: operator hit Stop mid-walk
        cancel_armed = False  # latched once we observe a live "running" state
@@ -236,6 +342,7 @@ class Ingester:
                written_paths=written,
                post_record_paths=list(post_records),
                relink_source_paths=list(relink),
                mark_seen_after_import=lambda: self._mark_seen(source_id, fetched),
                stdout="\n".join(log_lines),
                stderr="",
                return_code=return_code,
@@ -311,7 +418,17 @@ class Ingester:
                # Time-box check at the post boundary (coarse, like a gallery-dl
                # chunk). Backfill/recovery resume from emitted_cursor next chunk.
-                if time.monotonic() - start >= time_budget_seconds:
+                # The second half is phase 3's share of the task — see
                # CHUNK_TOTAL_SECONDS. A mid-page stop resumes the same page.
                elapsed = time.monotonic() - start
                phase3 = (
                    len(written) * PHASE3_IMPORT_SECONDS
                    + len(relink) * PHASE3_RELINK_SECONDS
                )
                if (
                    elapsed >= time_budget_seconds
                    or elapsed + phase3 >= CHUNK_TOTAL_SECONDS
                ):
                    budget_hit = True
                    break
@@ -322,6 +439,20 @@ class Ingester:
                # resume_cursor None, so everything counts.
                if not (resume_cursor and page_cursor == resume_cursor):
                    chunk_new_posts += 1
                # Inside the revisit window? Computed per post rather than
                # "stop once one post is old" because the feed is only MOSTLY
                # date-ordered — a pinned or re-pinned post can sit above older
                # ones, and one such post must not end the walk.
                in_window = False
                if horizon is not None:
                    published = _parse_published((post_meta(post) or {}).get("date"))
                    in_window = published is not None and published >= horizon
                # Set by the post-record block below when this post was already
                # captured on an earlier walk. Stays False when the platform has
                # no post-record seam, so the revisit accounting simply reports
                # nothing rather than guessing.
                post_already_recorded = False
                downloaded_before = downloaded
                # Tier-gated post (#874): the account can't fully view it, so
                # Patreon serves only blurred locked-preview media. Skip it
                # ENTIRELY — no media download AND no post-record stub (operator
@@ -353,11 +484,31 @@ class Ingester:
                            set() if recapture_records
                            else self._seen_keys(source_id, [pkey])
                        )
-                        if pkey not in already:
+                        post_already_recorded = pkey in already
-                            rec = write_post_record(post, artist_slug)
+                        # A post inside the revisit window is re-read even
-                            posts_recorded += 1
+                        # though the gate has it: that gate's whole job is to
-                            if rec.body_chars:
+                        # stop us paying for a post twice, and an EDITED post is
-                                posts_with_body += 1
+                        # not the same post. `revisit=True` keeps the cost at
                        # zero requests — the downloader re-reads the body from
                        # the feed response already in hand and declines to
                        # write at all if that body came back empty, so a
                        # detail-fetched body is never overwritten by a blank.
                        if not post_already_recorded or in_window:
                            rec = write_post_record(
                                post, artist_slug, revisit=post_already_recorded,
                            )
                            if not post_already_recorded:
                                # FIRST captures only feed the #862 body canary.
                                # A revisit legitimately comes back empty — a
                                # post whose body only ever arrived from the
                                # detail endpoint has none in the feed, and the
                                # downloader declines to write it. Counting
                                # those into the sample would walk the canary
                                # toward firing on healthy ticks, which is the
                                # one thing a drift alarm must never do.
                                posts_recorded += 1
                                if rec.body_chars:
                                    posts_with_body += 1
                            if rec.path is not None:
                                post_records.append(str(rec.path))
                                self._mark_seen(source_id, [(pkey, ppid)])
@@ -367,7 +518,8 @@ class Ingester:
                            # a 0-char body is the "why is this one empty" answer.
                            log_lines.append(
                                f"  post {ppid} [{rec.post_type or '?'}] "
-                                f"body: {rec.body_chars} chars"
+                                + ("re-read, " if post_already_recorded else "")
                                + f"body: {rec.body_chars} chars"
                                + ("" if rec.body_chars else " — EMPTY")
                                + (f" — {rec.title}" if rec.title else "")
                            )
@@ -400,6 +552,13 @@ class Ingester:
                    recapture=recapture,
                )
                # An on-disk file is only "done" if something imported it. One
                # with no ImageRecord at its path was written by a run that died
                # before phase 3 — it goes to import, not to the ledger.
                imported_paths = self._recorded_paths([
                    str(o.path) for o in outcomes
                    if o.status == "skipped_disk" and o.path is not None
                ])
                to_mark: list[tuple[str, str]] = []
                to_clear: list[str] = []  # recovered → drop any dead-letter row
                to_fail: list[tuple[str, str, str]] = []  # (key, post_id, error)
@@ -411,11 +570,29 @@ class Ingester:
                        downloaded += 1
                        if outcome.path is not None:
                            written.append(str(outcome.path))
-                        to_mark.append((key, media_item.post_id))
+                        fetched.append((key, media_item.post_id))
                        to_clear.append(key)
                        consecutive_seen = 0
                    elif (
                        outcome.status == "skipped_disk"
                        and outcome.path is not None
                        and str(outcome.path) not in imported_paths
                    ):
                        # On disk, never imported: a prior run wrote it and died
                        # before phase 3. Import it now. Safe to feed to
                        # attach_in_place because no record owns this path —
                        # the unlink below is about a file that IS the record.
                        written.append(str(outcome.path))
                        fetched.append((key, media_item.post_id))
                        to_clear.append(key)
                        skipped_count += 1
                        consecutive_seen += 1
                        log_lines.append(
                            f"  post {media_item.post_id} — on disk but never "
                            f"imported: {outcome.path.name}"
                        )
                    elif outcome.status == "skipped_disk":
-                        # Already on disk (a prior run). Reconcile the ledger so a
+                        # Already on disk and imported. Reconcile the ledger so a
                        # later tick skips it at tier-1 without a disk stat, but
                        # do NOT re-feed it to phase 3 — attach_in_place would see
                        # the duplicate sha256 and unlink the on-disk copy.
@@ -451,7 +628,15 @@ class Ingester:
                        to_fail.append((key, media_item.post_id, outcome.error or "error"))
                        # An error neither advances nor resets the run-of-seen.
-                    if mode == "tick" and consecutive_seen >= seen_threshold:
+                    # `not in_window` is the revisit window's half of the
                    # early-out: a run of already-seen items is only permission
                    # to stop once the walk is BELOW the horizon. Both halves,
                    # never either alone — see DEFAULT_REVISIT_DAYS.
                    if (
                        mode == "tick"
                        and not in_window
                        and consecutive_seen >= seen_threshold
                    ):
                        early_out = True
                        break
@@ -465,6 +650,21 @@ class Ingester:
                if to_fail:
                    self._record_failures(source_id, to_fail)
                # An already-captured post that yielded NEW media is an edited
                # post — the operator's Floppystack case, and the one thing in
                # this walk worth naming individually in the run log. The media
                # half needed no new detection: `extract_media` reads the media
                # list off the live feed response, so a hotfix build appended
                # last night is simply a ledger key we have never seen.
                new_here = downloaded - downloaded_before
                if post_already_recorded and new_here:
                    revisited += 1
                    revisit_downloads += new_here
                    log_lines.append(
                        f"  post {post.get('id')} — updated: "
                        f"{new_here} new file(s)"
                    )
                # plan #709: time-throttled live progress to the running event so
                # the Downloads view ticks ~every 5s, independent of page size.
                now = time.monotonic()
@@ -483,9 +683,15 @@ class Ingester:
                    })
                if early_out:
-                    break
+                    if skip_feed is None:
                        break
                    skip_feed()
                    feeds_caught_up += 1
                    early_out = False
                    consecutive_seen = 0
            else:
-                reached_bottom = True
+                # A walk that left feeds early did not read to their ends.
                reached_bottom = not feeds_caught_up
        except self._error_base as exc:
            # The platform's client-error base — _failure_result (adapter)
            # maps it to a typed error.
@@ -527,6 +733,13 @@ class Ingester:
            # visible in the Raw stdout (e.g. "bodies 3/180" reads as off).
            + (f", bodies {posts_with_body}/{posts_recorded}" if posts_recorded else "")
            + (f", {gated_skipped} gated-skipped" if gated_skipped else "")
            # Only when it happened: on a quiet tick this is 0 and saying so
            # every run would bury the times it is not.
            + (
                f", {revisited} post(s) updated ({revisit_downloads} new file(s))"
                if revisited else ""
            )
            + (f", {feeds_caught_up} feed(s) caught up" if feeds_caught_up else "")
            + (", reached end" if reached_bottom else "")
            + (", time-boxed" if budget_hit else "")
        )
@@ -540,7 +753,13 @@ class Ingester:
            # next chunk resumes from the emitted cursor. No progress → TIMEOUT,
            # which feeds download_service's backfill stall-guard. rc<0 mirrors
            # subprocess TimeoutExpired so completion detection stays false.
-            made_progress = downloaded > 0 or emitted_cursor != resume_cursor
+            # Work handed to phase 3 is progress too: a recapture chunk that
            # stopped for its imports downloaded nothing, and may not have left
            # its first page.
            made_progress = (
                downloaded > 0 or bool(written) or bool(relink)
                or emitted_cursor != resume_cursor
            )
            if made_progress:
                return _result(
                    success=False, return_code=-1,
@@ -746,6 +965,16 @@ class Ingester:
            )
            session.commit()
    def _recorded_paths(self, paths: list[str]) -> set[str]:
        """Which of `paths` an ImageRecord already points at."""
        if not paths:
            return set()
        with self.session_factory() as session:
            rows = session.execute(
                select(ImageRecord.path).where(ImageRecord.path.in_(paths))
            ).scalars().all()
        return set(rows)
    def _mark_seen(self, source_id: int, items: list[tuple[str, str]]) -> None:
        """Idempotent upsert of (filehash, post_id) seen-ledger rows for a page.
@@ -8,9 +8,10 @@ trustworthy enough to act on.
 1. `subscribed_not_tracked` — you pay for this and FC does not follow it. The
   adoption win, and the only bucket carrying an action.
 2. `tracked_not_subscribed` — FC follows this and the roster does not show you
-   paying for it. REPORT ONLY, by the operator's decision (2026-09-11): it says
+   paying for it. No longer shown on the card: the operator reversed the
-   what it sees and links to the existing Subscriptions row, and offers no
+   2026-09-11 "report only" call on 2026-09-13. The lapsed half of it now ACTS,
-   one-click disable.
+   in `apply_membership_lapses` below (#3995). The absent half still only
   reports, because absence proves nothing.
 3. `matched` — the healthy set. Counted, not listed loudly.
 4. `unidentified` — sources this join cannot speak to at all. Reported as
   exactly that, because the alternative is filing them under a verdict.
@@ -39,7 +40,7 @@ rendered as lapsed. That is the whole reason it returns a tri-state.
 from __future__ import annotations
-from datetime import datetime
+from datetime import UTC, datetime
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
@@ -47,12 +48,12 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from ..models import Artist, MembershipSync, PlatformMembership, Source
 from .membership_roster import (
    get_sync_state,
    has_paid_access,
    identity_keys_for_source,
    pair_sources_with_memberships,
    roster_is_fresh,
    url_tail,
 )
 from .native_ingest_common import has_paid_access
 # Why a source appears in `tracked_not_subscribed`. Ordered strongest first —
 # the UI renders a different sentence per basis, because collapsing them into
@@ -233,3 +234,126 @@ async def reconcile_all(session: AsyncSession, now: datetime | None = None) -> d
            await reconcile(session, platform=p, now=now) for p in sorted(platforms)
        ]
    }
 # ---------------------------------------------------------------------------
 # Stop pulling what the account no longer pays for (#3995)
 # ---------------------------------------------------------------------------
 #
 # Operator decision, 2026-09-13, reversing the 2026-09-11 "report only" call
 # for this direction: "if I kill a subscription on patreon I would like the
 # pulling to stop on curator as well", with automatic resume on resubscribing.
 #
 # This is a SOURCE-level action taken by the daily sweep, visible on the source
 # row and reversible there. It is not a fetch-path decision. The line C5 draws,
 # that the roster never decides a POST is inaccessible, still holds: nothing
 # here reads per-post access, and no download path reads the roster
 # (`test_no_fetch_path_can_read_the_roster`). The scheduler keeps selecting on
 # `enabled` alone.
 #
 # Acts ONLY on positive evidence. A source whose matched membership says access
 # has ended is stopped. A source with NO matched membership is left alone,
 # because absence has innocent causes: a creator rename, a source never walked
 # so no id is cached, a membership the platform stopped listing. Stopping on
 # absence would switch off things the operator still pays for.
 #
 # Two app-managed config_overrides keys carry the state. The `_` prefix is
 # already the "FC writes this, an operator edit preserves it" family.
 #   _membership_stopped  set when the sweep stops a source; the sweep resumes
 #                        ONLY sources carrying it, so a source the operator
 #                        switched off by hand is never switched back on
 #   _membership_kept     set by SourceService.update when the operator turns a
 #                        stopped source back ON: a deliberate choice to keep
 #                        pulling a lapsed creator, which the next sweep must
 #                        not undo. Cleared when the membership is paid again.
 STOPPED_KEY = "_membership_stopped"
 KEPT_KEY = "_membership_kept"
 def _access_expires_at(m: PlatformMembership) -> datetime | None:
    """When paid access actually ends, if the platform says.
    Patreon keeps a cancelled membership's access until the end of the billing
    period and reports that date (`member.access_expires_at`, note #3992).
    SubscribeStar's page gives no such date, so a cancelled SubscribeStar
    membership stops at once. Returns None when there is no usable date.
    """
    details = m.details or {}
    raw = details.get("access_expires_at") or (details.get("member") or {}).get("access_expires_at")
    if not isinstance(raw, str) or not raw:
        return None
    try:
        parsed = datetime.fromisoformat(raw.replace("Z", "+00:00"))
    except ValueError:
        return None
    return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
 async def apply_membership_lapses(
    session: AsyncSession, *, platform: str, now: datetime | None = None,
 ) -> dict:
    """Stop sources whose paid access has ended; resume the ones this stopped.
    Refuses to act on a roster that isn't fresh, for the same reason C4 refuses
    to draw conclusions from one.
    """
    now = now or datetime.now(UTC)
    state = await get_sync_state(session, platform)
    if not roster_is_fresh(state, now=now):
        return {"platform": platform, "skipped": "roster not fresh", "stopped": 0, "resumed": 0}
    memberships = (await session.execute(
        select(PlatformMembership).where(PlatformMembership.platform == platform)
    )).scalars().all()
    sources = (await session.execute(
        select(Source).where(Source.platform == platform)
    )).scalars().all()
    pairs = pair_sources_with_memberships(list(sources), list(memberships))
    stopped: list[int] = []
    resumed: list[int] = []
    for source in sources:
        pair = pairs.get(source.id)
        if pair is None:
            continue  # absence is never acted on, see above
        m, _kind = pair
        paid = has_paid_access(
            m.platform, m.status,
            is_free_member=bool((m.details or {}).get("is_free_member")),
        )
        co = dict(source.config_overrides or {})
        if paid is True:
            changed = co.pop(KEPT_KEY, None) is not None
            if STOPPED_KEY in co:
                co.pop(STOPPED_KEY)
                source.enabled = True
                resumed.append(source.id)
                changed = True
            if changed:
                source.config_overrides = co
            continue
        # Unknown status: never a reason to stop something (has_paid_access's
        # tri-state exists for exactly this).
        if paid is None:
            continue
        if not source.enabled or co.get(KEPT_KEY):
            continue
        expires = _access_expires_at(m)
        if expires is not None and expires > now:
            continue  # still inside the paid-through period
        co[STOPPED_KEY] = {"at": now.isoformat(), "status": m.status}
        source.config_overrides = co
        source.enabled = False
        # The same clean slate a manual disable gives (SourceService.update,
        # #1285), so a stopped source doesn't linger as failing or gated.
        source.last_error = None
        source.error_type = None
        source.consecutive_failures = 0
        stopped.append(source.id)
    await session.commit()
    return {"platform": platform, "stopped": len(stopped), "resumed": len(resumed)}
@@ -20,8 +20,10 @@ own word — `active_patron`, not some normalised FC value. The mapping from
 those words to FC's meaning is a read-site concern and belongs in code that can
 be corrected without a migration, because the vocabulary comes from whatever
 each platform says and will be discovered per platform rather than designed up
-front. `MEMBERSHIP_STATUS` below is a place for that knowledge to accumulate as
+front. `native_ingest_common.MEMBERSHIP_STATUS` is where that knowledge
-platforms are characterised; it is deliberately empty of guesses today.
+accumulates as platforms are characterised, and it holds no guesses. It lives
 there rather than here because platform clients need it, and a client may not
 import this module (test_gated_reason.py).
 """
 from __future__ import annotations
@@ -35,78 +37,10 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 from ..models import MembershipSync, PlatformMembership, Source
 from .native_ingest_common import has_paid_access
 log = logging.getLogger(__name__)
 # Platform word -> whether the account currently has paid access.
 #
 # Every entry here must come from a CHARACTERISED response, never from API docs
 # or a plausible guess — project rule 130, and inventing a status before seeing
 # it in a real payload is exactly the failure it names.
 #
 # patreon: from a live capture of the operator's own session, 2026-09-10
 # (Scribe note #3886). Only two values were OBSERVED in `patron_status` and
 # only those two are here.
 #
 # `declined_patron` is deliberately ABSENT even though it looks obviously
 # right. It appears in the request's `filter[membership_type]`, and the capture
 # proved that filter is NOT the same vocabulary as the attribute — a row
 # selected by the filter as `free_member` came back with
 # `patron_status: former_patron`, a word the filter does not contain. Reading
 # the filter as an enum is the specific mistake the capture caught; adding
 # `declined_patron` on the strength of it would be repeating that mistake one
 # step later.
 #
 # Unknown words are NOT an error: an unrecognised status means the roster
 # records evidence it cannot yet interpret, which is a better state than
 # dropping the row or asserting a meaning for it.
 #
 # subscribestar: from a live capture of the account's /subscriptions page,
 # 2026-09-13 (Scribe note #3989). SubscribeStar gives NO per-row status word —
 # a membership's state is which of two tables it sits in — so the "word" stored
 # is the table card's own `data-identifier`, verbatim. Those two identifiers are
 # the whole vocabulary; there is nothing further to characterise later.
 MEMBERSHIP_STATUS: dict[str, dict[str, bool]] = {
    "patreon": {
        "active_patron": True,
        "former_patron": False,
    },
    "subscribestar": {
        "active_subscriptions": True,
        "cancelled_subscriptions": False,
    },
 }
 def has_paid_access(
    platform: str, status: str | None, *, is_free_member: bool = False,
 ) -> bool | None:
    """Does this membership mean the account currently PAYS for access?
    Returns None for a status this code has not been taught, which callers must
    treat as "unknown" rather than as False. The difference matters: False says
    the operator has lost access, and asserting that from an unrecognised word
    would tell them to cancel a source they are still paying for.
    `is_free_member` is a second axis, not a status, and that is Patreon's
    design rather than ours: the capture shows a free follow expressed as a
    boolean alongside `patron_status`, so a "current" membership can still be
    one nobody is paying for. Taking status alone would report a free follower
    as a paying patron, and C4 would then never offer to clean it up.
    (Honest limit: the capture contains no ACTIVE free member, so it cannot
    demonstrate the two axes coming apart. The separation is what the payload's
    shape says; the sample only shows it is possible, not that it happens.)
    """
    if status is None:
        return None
    known = MEMBERSHIP_STATUS.get(platform, {}).get(status)
    if known is None:
        return None
    if not known:
        return False
    return not is_free_member
 async def touch_membership(
    session: AsyncSession,
@@ -35,6 +35,58 @@ DEFAULT_SIM_THRESHOLD = 0.85
 _FIGURE_KINDS = ("face", "figure")
 # How many cosine scores to hold in memory at once, per matmul block.
 # 4M float32 is 16 MB — small enough to stay in cache-friendly territory on the
 # shared ml lane, large enough that the per-call overhead stops mattering.
 _MAX_SCORE_ELEMS = 4_000_000
 def char_maxima(q_by_image, allref, seg, np, *, max_elems=_MAX_SCORE_ELEMS):
    """(n_images, n_chars) — each image's best cosine to each character.
    `q_by_image` is one L2-normalised `(n_figures, dim)` array per image, in
    the order the answer comes back in. `allref` is every character's
    prototypes stacked, and `seg` their per-character start offsets into it.
    ## Why this is batched, and why that is safe
    `scheduled_ccip_auto_apply` did this one image at a time — a `(nq, dim) @
    (dim, total)` product per image, over every image in the library on every
    run. At ~119k images that is 119k separate matmuls, each too small to pay
    for its own BLAS setup, and on 2026-09-23 the daily sweep hit its 1800s
    soft limit on the operator's instance.
    Batching changes no arithmetic. The score a character gets for an image is
    a max over that image's figures AND over that character's prototypes, and
    max does not care in what order or grouping it is taken — so reducing the
    prototype axis first (per row, inside a block) and the figure axis after
    (per image, across blocks) gives exactly what the per-image loop gave.
    That equivalence is what `test_char_maxima_matches_the_per_image_loop`
    pins, against the naive form written out longhand.
    Blocked by ROWS rather than done in one product, because the full score
    matrix is (all figures in the chunk x every prototype) and that grows with
    the library on both axes. The block bound is on elements, so the memory
    this uses stays flat as either axis grows.
    """
    counts = [len(q) for q in q_by_image]
    rows = np.vstack(q_by_image)
    total = max(int(allref.shape[0]), 1)
    block = max(1, max_elems // total)
    per_row = np.empty((rows.shape[0], len(seg)), dtype=np.float32)
    for a in range(0, rows.shape[0], block):
        scores = rows[a:a + block] @ allref.T
        per_row[a:a + block] = np.maximum.reduceat(scores, seg, axis=1)
    # Start offset of each image's rows. Every image has at least one figure —
    # it is in `q_by_image` because a region produced it — so these strictly
    # increase, which is what `reduceat` needs to reduce rather than pass a row
    # through untouched.
    starts = np.cumsum([0] + counts[:-1])
    return np.maximum.reduceat(per_row, starts, axis=0)
 async def _settings_threshold(session: AsyncSession) -> float:
    val = (
        await session.execute(
@@ -11,12 +11,21 @@ from pathlib import Path
 import numpy as np
 from PIL import Image, ImageFile
 from ..worker_lanes import LANES_BY_NAME
 ImageFile.LOAD_TRUNCATED_IMAGES = True
 # Cap torch's intra-op threads so each ml-worker replica is a bounded core
-# consumer on a shared node (torch otherwise uses all cores). Keep
+# consumer on a shared node (torch otherwise uses all cores).
-# N_replicas × this within the cores allotted to ML to avoid oversubscription.
+#
-_INTRA_OP_THREADS = 4
+# Read from the lane rather than restated here. This was a literal 4 beside a
 # comment reading "keep N_replicas x this within the cores allotted to ML" —
 # a constraint written where nothing could act on it, and nothing did: the ML
 # ceiling came from memory alone, offered the operator ~49 slots on a
 # large-memory host, and the lane spent 2026-09-23 with ~200 torch threads on
 # it. `derived_ceiling` now divides the cores by this number, which only means
 # anything while the two are the same number.
 _INTRA_OP_THREADS = LANES_BY_NAME["ml"].threads_per_slot
 DEFAULT_MODEL_NAME = os.environ.get(
    "SIGLIP_MODEL_NAME", "google/siglip-so400m-patch14-384"
@@ -231,7 +231,7 @@ class Membership:
    `status` carries the PLATFORM's own word, verbatim and unmapped
    (`active_patron`, `former_patron`, ...). Deciding what it means is the read
-    site's job — `membership_roster.has_paid_access` — precisely so an
+    site's job — `has_paid_access`, below — precisely so an
    unrecognised word records as evidence rather than as a decision.
    `is_free_member` is SEPARATE from status and must stay that way. Patreon
@@ -396,3 +396,81 @@ class BaseNativeDownloader:
        sidecar_path = media_path.with_suffix(".json")
        sidecar_path.write_text(json.dumps(data, indent=2))
        return sidecar_path
 # --- membership status vocabulary (#387) ------------------------------------
 #
 # Lives here, beside `Membership`, rather than in `membership_roster`. It is
 # pure platform knowledge with no database behind it, and the platform clients
 # need it too. Patreon's must tell a lapsed membership to a deleted creator
 # (skippable) from a paid one it cannot attribute (drift), and a client may not
 # import `membership_roster`: test_gated_reason.py forbids any fetch path from
 # reaching the roster, so the roster can explain a skip but never cause one.
 #
 # Platform word -> whether the account currently has paid access.
 #
 # Every entry here must come from a CHARACTERISED response, never from API docs
 # or a plausible guess — project rule 130, and inventing a status before seeing
 # it in a real payload is exactly the failure it names.
 #
 # patreon: from a live capture of the operator's own session, 2026-09-10
 # (Scribe note #3886). Only two values were OBSERVED in `patron_status` and
 # only those two are here.
 #
 # `declined_patron` is deliberately ABSENT even though it looks obviously
 # right. It appears in the request's `filter[membership_type]`, and the capture
 # proved that filter is NOT the same vocabulary as the attribute — a row
 # selected by the filter as `free_member` came back with
 # `patron_status: former_patron`, a word the filter does not contain. Reading
 # the filter as an enum is the specific mistake the capture caught; adding
 # `declined_patron` on the strength of it would be repeating that mistake one
 # step later.
 #
 # Unknown words are NOT an error: an unrecognised status means the roster
 # records evidence it cannot yet interpret, which is a better state than
 # dropping the row or asserting a meaning for it.
 #
 # subscribestar: from a live capture of the account's /subscriptions page,
 # 2026-09-13 (Scribe note #3989). SubscribeStar gives NO per-row status word —
 # a membership's state is which of two tables it sits in — so the "word" stored
 # is the table card's own `data-identifier`, verbatim. Those two identifiers are
 # the whole vocabulary; there is nothing further to characterise later.
 MEMBERSHIP_STATUS: dict[str, dict[str, bool]] = {
    "patreon": {
        "active_patron": True,
        "former_patron": False,
    },
    "subscribestar": {
        "active_subscriptions": True,
        "cancelled_subscriptions": False,
    },
 }
 def has_paid_access(
    platform: str, status: str | None, *, is_free_member: bool = False,
 ) -> bool | None:
    """Does this membership mean the account currently PAYS for access?
    Returns None for a status this code has not been taught, which callers must
    treat as "unknown" rather than as False. The difference matters: False says
    the operator has lost access, and asserting that from an unrecognised word
    would tell them to cancel a source they are still paying for.
    `is_free_member` is a second axis, not a status, and that is Patreon's
    design rather than ours: the capture shows a free follow expressed as a
    boolean alongside `patron_status`, so a "current" membership can still be
    one nobody is paying for. Taking status alone would report a free follower
    as a paying patron, and C4 would then never offer to clean it up.
    (Honest limit: the capture contains no ACTIVE free member, so it cannot
    demonstrate the two axes coming apart. The separation is what the payload's
    shape says; the sample only shows it is possible, not that it happens.)
    """
    if status is None:
        return None
    known = MEMBERSHIP_STATUS.get(platform, {}).get(status)
    if known is None:
        return None
    if not known:
        return False
    return not is_free_member
@@ -58,6 +58,7 @@ from .native_ingest_common import (
    NativeDriftError,
    NativeIngestError,
    basename_from_url,
    has_paid_access,
    make_session,
    retry_after_seconds,
 )
@@ -482,8 +483,14 @@ class PatreonClient:
    @staticmethod
    def post_meta(post: dict) -> dict:
-        """Title + published date for a post — for the preview sample (plan #708
+        """Title + published date for a post. Part of the client contract.
-        B4). Part of the client contract `ingest_core.Ingester.preview` calls."""
+
        Written for a preview sample (plan #708 B4) whose caller has since gone;
        as of 2026-09-23 its consumer is the core's REVISIT WINDOW, which needs
        a post's date to know whether a tick is still inside it. Both native
        clients answer in the same shape — an ISO-8601 string under `date`, or
        None — so the core reads a date without knowing the platform.
        """
        attrs = post.get("attributes") or {}
        title = attrs.get("title")
        published = attrs.get("published_at")
@@ -631,7 +638,25 @@ class PatreonClient:
                "cannot tell a complete roster from a truncated one"
            )
-    def _membership(self, member: dict, index: dict) -> Membership:
+    def _membership(self, member: dict, index: dict) -> Membership | None:
        """One member row as a Membership, or None for a row the roster can skip.
        The one skippable row is a LAPSED membership whose creator no longer
        exists. The live roster (note #3886, CORRECTION 3) returned 104 rows,
        because FC sends no membership-type filter and so gets lapses going back
        years. One of them, a membership that ended in 2017, carried no
        `campaign` relationship at all: the key is absent, not null, and its
        reward names no campaign either. The creator's page is gone.
        Raising on that row made the whole roster unusable over one membership
        nobody can act on. Skipping it changes no conclusion. A lapsed
        membership already means "not paying", absence means the same, and no
        Source can be matched to a campaign that no longer has an id.
        The refusal stays for every other row. An active or unrecognised
        membership without a creator is something FC cannot vouch for, and
        dropping it would read downstream as a cancellation.
        """
        attrs = member.get("attributes") or {}
        if "patron_status" not in attrs:
            raise PatreonDriftError(
@@ -640,6 +665,17 @@ class PatreonClient:
        campaign_ids = self._related_ids(member, "campaign")
        if not campaign_ids:
            paid = has_paid_access(
                "patreon", attrs.get("patron_status"),
                is_free_member=bool(attrs.get("is_free_member")),
            )
            if paid is False:
                log.info(
                    "Patreon roster: skipping a lapsed membership with no campaign "
                    "(creator deleted); status=%s access_expires_at=%s",
                    attrs.get("patron_status"), attrs.get("access_expires_at"),
                )
                return None
            raise PatreonDriftError(
                "Patreon member resource has no campaign relationship — a "
                "membership we cannot attribute to a creator is not usable"
@@ -700,7 +736,9 @@ class PatreonClient:
            index = self._transform(response)
            rows = [m for m in (response.get("data") or []) if isinstance(m, dict)]
            for member in rows:
-                yield self._membership(member, index)
+                membership = self._membership(member, index)
                if membership is not None:
                    yield membership
            seen += len(rows)
            total = int(response["meta"]["pagination"]["total"] or 0)
@@ -340,7 +340,7 @@ class PatreonDownloader(BaseNativeDownloader):
    def _write_sidecar_data(
        self, post: dict, sidecar_path: Path, *, source_url: str | None = None,
-        minimal: bool = False,
+        minimal: bool = False, detail_fetch: bool = True,
    ) -> Path:
        """Serialize the post's metadata to `sidecar_path`. The post-only record
        (`write_post_record`) writes the FULL post (body/title/date/url); the
@@ -364,7 +364,13 @@ class PatreonDownloader(BaseNativeDownloader):
        # dict — so a multi-image post fetches detail at most once, the post-record
        # body-length read reuses it, and a fully-seen post (no fresh download → no
        # sidecar write) never pays the extra GET.
-        if (not content or not content.strip()) and self._content_fetcher:
+        # `detail_fetch=False` on a REVISIT (a post inside the tick's revisit
        # window that we already captured): re-read the body from the feed
        # response we are holding and pay nothing. Without this a 30-day window
        # would buy one detail GET per body-less post per tick, forever — a
        # per-creator cost that grows with how prolific they are, to re-fetch a
        # body we already stored.
        if (not content or not content.strip()) and self._content_fetcher and detail_fetch:
            fetched = self._content_fetcher(str(post.get("id") or ""))
            if fetched:
                content = fetched
@@ -386,7 +392,9 @@ class PatreonDownloader(BaseNativeDownloader):
        sidecar_path.write_text(json.dumps(data, indent=2))
        return sidecar_path
-    def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
+    def write_post_record(
        self, post: dict, artist_slug: str, *, revisit: bool = False,
    ) -> PostRecordOutcome:
        """Write a post-ONLY sidecar (no media file) for a media-less post, so
        the importer can still upsert the Post + its body — text posts often hold
        the only copy of an external <a href> link. Named `_post.json`: the
@@ -397,6 +405,18 @@ class PatreonDownloader(BaseNativeDownloader):
        Returns a PostRecordOutcome (path None when the post has no id) carrying
        the captured body's shape — post_type + final char count — so the engine
        can log per-post handling without re-reading the post itself.
        `revisit=True` is the tick re-reading a post it already captured
        (ingest_core's revisit window, #...). Two differences, both about not
        making an update cost more than it is worth:
        * no detail-fetch — the body comes from the feed response already in
          hand, so a revisit costs zero requests;
        * a body that comes back empty writes NOTHING and returns `path=None`.
          On a first capture an empty body is the truth about the post; on a
          revisit it usually just means this post's body only ever came from
          the detail endpoint we just declined to call, and writing it would
          blank a stored body to say something we never learned.
        """
        attrs = post.get("attributes") or {}
        title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
@@ -406,9 +426,17 @@ class PatreonDownloader(BaseNativeDownloader):
            return PostRecordOutcome(
                path=None, post_type=post_type, title=title, body_chars=0,
            )
        if revisit:
            feed_body = post_body_html(attrs)
            if not (isinstance(feed_body, str) and feed_body.strip()):
                return PostRecordOutcome(
                    path=None, post_type=post_type, title=title, body_chars=0,
                )
        post_dir = self.images_root / artist_slug / "patreon" / post_dir_name(post)
        post_dir.mkdir(parents=True, exist_ok=True)
-        path = self._write_sidecar_data(post, post_dir / "_post.json")
+        path = self._write_sidecar_data(
            post, post_dir / "_post.json", detail_fetch=not revisit,
        )
        # _write_sidecar_data has by now memoized any detail-fetched body onto
        # post["attributes"]["content"], so re-read it for the FINAL char count.
        body = (post.get("attributes") or {}).get("content")
@@ -1,579 +0,0 @@
 """Native Pixiv client — the Pixiv adapter's read path.
 Pixiv has a real (if unofficial) API: the mobile app API gallery-dl drives
 (`PixivAppAPI`). Per the downloader ground rule — gallery-dl is the
 known-working base — this client mirrors gallery-dl 1.32.5's request profile
 EXACTLY: the same iOS app headers on every request, the same OAuth
 refresh-token dance against oauth.secure.pixiv.net (X-Client-Time +
 X-Client-Hash), and the same `/v1/user/illusts` walk paginated by `next_url`.
 Deviating from that profile is how the SubscribeStar/Patreon spikes broke, so
 any change here should be diffed against gallery-dl's extractor first.
 Feed shape (characterized from gallery-dl 1.32.5, extractor/pixiv.py):
  - `GET /v1/user/illusts?user_id=<id>` returns `{"illusts": [work...],
    "next_url": "https://app-api...?user_id=..&offset=30" | null}`.
  - Pagination: re-issue the SAME endpoint with `next_url`'s query params. The
    query string doubles as our resumable page cursor (re-fetching it re-serves
    the same page — the ingest-core resume contract).
  - A work carries id/title/type(illust|manga|ugoira)/caption(HTML)/
    create_date(ISO+09:00)/tags[{name,translated_name}]/user/page_count/
    x_restrict/series/total_view/total_bookmarks/meta_single_page/meta_pages.
  - Files: multi-page → meta_pages[].image_urls.original; single page →
    meta_single_page.original_image_url; ugoira → `/v1/ugoira/metadata` zip
    (600x600 → 1920x1080 URL swap, gallery-dl's default non-original mode).
 `campaign_id` for Pixiv is the numeric user id (extracted from the source URL
 by `user_id_from_url` — no network resolver needed).
 Gated works: pixiv serves a `https://s.pximg.net/common/images/limit_*.png`
 placeholder as the "original" when a work is blocked for this account
 (sanity-level filter, my-pixiv lock, deleted). gallery-dl's fallback for those
 is a web-AJAX scrape that needs PHPSESSID browser cookies — FC stores only the
 OAuth refresh token, so (exactly like our previous gallery-dl configuration,
 which warned "No PHPSESSID cookie set") those works are skipped, via the
 post_is_gated seam. Auth failures are loud (rotate the refresh token); a
 response missing the fields we depend on is DRIFT (update this client).
 FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
 """
 from __future__ import annotations
 import hashlib
 import logging
 import time
 from collections.abc import Iterator
 from dataclasses import dataclass
 from datetime import UTC, datetime
 from urllib.parse import parse_qsl, urlsplit
 import requests
 from ..utils.paths import safe_ext
 from .native_ingest_common import (
    _MAX_429_RETRIES,
    NativeAuthError,
    NativeDriftError,
    NativeIngestError,
    make_session,
    retry_after_seconds,
 )
 log = logging.getLogger(__name__)
 _TIMEOUT_SECONDS = 30.0
 _API_ROOT = "https://app-api.pixiv.net"
 _OAUTH_URL = "https://oauth.secure.pixiv.net/auth/token"
 # gallery-dl's public Pixiv-app credentials (PixivAppAPI, also pixivpy's) —
 # these identify the official iOS app to the API, NOT the operator; the
 # operator's identity is the OAuth refresh token.
 _CLIENT_ID = "MOBrBDS8blbauoSck0ZfDbtuzpyT"
 _CLIENT_SECRET = "lsACyCD94FhDUtGTXi3QzcFE2uU1hqtDaKeqrdwj"
 _HASH_SECRET = (
    "28c1fdd170a5204386cb1313c7077b34"
    "f83e4aaf4aa829ce78c231e05b0bae2c"
 )
 # The exact header set gallery-dl 1.32.5 installs on its session — the proven
 # app-API request profile. The Referer also unlocks i.pximg.net media GETs
 # (403 without it), so the downloader reuses this constant.
 PIXIV_APP_HEADERS = {
    "App-OS": "ios",
    "App-OS-Version": "16.7.2",
    "App-Version": "7.19.1",
    "User-Agent": "PixivIOSApp/7.19.1 (iOS 16.7.2; iPhone12,8)",
    "Referer": "https://app-api.pixiv.net/",
 }
 # Placeholder image prefix pixiv serves instead of a blocked work's original
 # (limit_sanity_level / limit_mypixiv / limit_unknown variants).
 _LIMIT_URL = "https://s.pximg.net/common/images/limit_"
 # The app API reports rate-limiting as an error MESSAGE (often on HTTP 403),
 # not only as HTTP 429. gallery-dl sleeps 300s in-walk; sleeping that long
 # inside our time-boxed chunk would eat the whole budget, so we surface it as
 # a typed 429 and let download_service's cooldown machinery honor the wait.
 _RATE_LIMIT_RETRY_AFTER = 300.0
 _TITLE_MAX = 50  # gallery-dl pixiv filename template: {title[:50]}
 _RATINGS = {0: "General", 1: "R-18", 2: "R-18G"}
 class PixivAPIError(NativeIngestError):
    """Base for native Pixiv client failures. status_code / retry_after are
    inherited from NativeIngestError."""
 class PixivAuthError(PixivAPIError, NativeAuthError):
    """Auth failure — missing/expired/revoked OAuth refresh token. Fix =
    rotate the credential (Settings → Credentials → Pixiv), not update the
    client. Maps to error_type 'auth_error'."""
 class PixivDriftError(PixivAPIError, NativeDriftError):
    """A response did not match the shape this client depends on (missing
    `illusts`, un-parseable JSON where JSON was promised). Fail loud so the
    run flags 'the Pixiv app API changed' instead of silently importing
    nothing. Maps to API_DRIFT."""
@dataclass
 class MediaItem:
    """One resolved downloadable file belonging to a Pixiv work.
    Fields mirror the other native clients' MediaItem so the downloader and
    ledger are structurally the same. Pixiv original URLs carry no content
    hash, so `filehash` is always None and the ledger keys on
    `<post_id>:<media_id>` where media_id is `p<num>` (page) or `ugoira`
    (the frame zip) — stable across URL-shape drift.
    """
    url: str
    filename: str
    kind: str
    filehash: str | None
    post_id: str
    media_id: str
 def user_id_from_url(url: str) -> str | None:
    """The numeric pixiv user id from a source URL, or None.
    Handles the modern forms FC accepts as sources
    (https://www.pixiv.net/users/<id>, /en/users/<id>) plus the legacy
    member.php?id=<id>. This IS the campaign id — no network resolver.
    """
    parts = urlsplit(url or "")
    if "pixiv.net" not in parts.netloc:
        return None
    segs = [s for s in parts.path.split("/") if s]
    if segs and segs[0] == "en":
        segs = segs[1:]
    if len(segs) >= 2 and segs[0] == "users" and segs[1].isdigit():
        return segs[1]
    if segs and segs[0] == "member.php":
        qid = dict(parse_qsl(parts.query)).get("id", "")
        if qid.isdigit():
            return qid
    return None
 def _work_filename(work: dict, num: int, url: str) -> str:
    """gallery-dl layout parity: `{id}_{title[:50]}_{num:>02}.{extension}`
    (the downloader sanitizes the final segment)."""
    title = work.get("title")
    title50 = (title if isinstance(title, str) else "")[:_TITLE_MAX]
    ext = safe_ext(urlsplit(url).path.rsplit("/", 1)[-1])
    return f"{work.get('id')}_{title50}_{num:02d}{ext}"
 class PixivClient:
    """Synchronous Pixiv app-API read client. Construct with the operator's
    OAuth refresh token (the same token-type Credential the gallery-dl path
    consumed as `extractor.pixiv.refresh-token`)."""
    def __init__(
        self,
        refresh_token: str | None,
        *,
        request_sleep: float = 0.0,
        max_retries: int = _MAX_429_RETRIES,
        session: requests.Session | None = None,
    ):
        self.refresh_token = refresh_token
        self._request_sleep = request_sleep or 0.0
        self._max_retries = max_retries
        # No cookies — the app API authenticates via the Bearer token _login
        # installs. make_session still supplies the retry/UA plumbing; the
        # extra_headers overwrite its browser UA with the app profile.
        self._session = (
            session if session is not None
            else make_session(None, extra_headers=PIXIV_APP_HEADERS)
        )
        self._authed_user: dict = {}
        # Monotonic deadline after which the access token must be refreshed;
        # 0 forces a refresh on first use.
        self._token_deadline = 0.0
    # -- auth ----------------------------------------------------------------
    def _login(self) -> None:
        """Exchange the refresh token for a Bearer access token (gallery-dl's
        `_login_impl`, including the X-Client-Time/X-Client-Hash pair the
        endpoint validates). No-op while the current token is still fresh."""
        if time.monotonic() < self._token_deadline:
            return
        if not self.refresh_token:
            raise PixivAuthError(
                "No Pixiv refresh token configured — add the OAuth refresh "
                "token as the Pixiv credential (token type)."
            )
        # gallery-dl stamps naive-UTC with a literal +00:00 suffix.
        now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S+00:00")
        headers = {
            "X-Client-Time": now,
            "X-Client-Hash": hashlib.md5(
                (now + _HASH_SECRET).encode()
            ).hexdigest(),
        }
        data = {
            "client_id": _CLIENT_ID,
            "client_secret": _CLIENT_SECRET,
            "grant_type": "refresh_token",
            "refresh_token": self.refresh_token,
            "get_secure_url": "1",
        }
        try:
            resp = self._session.post(
                _OAUTH_URL, data=data, headers=headers,
                timeout=_TIMEOUT_SECONDS,
            )
        except requests.RequestException as exc:
            raise PixivAPIError(f"Pixiv OAuth request failed: {exc}") from exc
        if resp.status_code >= 400:
            raise PixivAuthError(
                "Pixiv rejected the refresh token (HTTP "
                f"{resp.status_code}) — rotate the Pixiv credential.",
                status_code=resp.status_code,
            )
        try:
            payload = resp.json()["response"]
            access = payload["access_token"]
        except (ValueError, KeyError, TypeError) as exc:
            raise PixivDriftError(
                f"Pixiv OAuth response shape changed: {exc}"
            ) from exc
        self._authed_user = payload.get("user") or {}
        self._session.headers["Authorization"] = f"Bearer {access}"
        # expires_in is 3600 today; refresh 60s early so a long walk never
        # rides an expiring token into a spurious 400.
        expires_in = payload.get("expires_in")
        lifetime = float(expires_in) if isinstance(expires_in, (int, float)) else 3600.0
        self._token_deadline = time.monotonic() + max(60.0, lifetime - 60.0)
    # -- request -------------------------------------------------------------
    def _call(self, endpoint: str, params: dict) -> dict:
        """Authenticated app-API GET → parsed JSON body, with the shared 429
        backoff and the loud auth/drift/rate-limit mapping."""
        self._login()
        if self._request_sleep > 0:
            time.sleep(self._request_sleep)
        url = _API_ROOT + endpoint
        attempt = 0
        while True:
            try:
                resp = self._session.get(
                    url, params=params, timeout=_TIMEOUT_SECONDS
                )
            except requests.RequestException as exc:
                raise PixivAPIError(
                    f"Pixiv request failed ({endpoint}): {exc}"
                ) from exc
            if resp.status_code == 429 and attempt < self._max_retries:
                attempt += 1
                delay = retry_after_seconds(resp, attempt)
                log.warning(
                    "Pixiv 429 (%s) — backing off %.1fs (retry %d/%d)",
                    endpoint, delay, attempt, self._max_retries,
                )
                time.sleep(delay)
                continue
            break
        try:
            body = resp.json()
        except ValueError as exc:
            raise PixivDriftError(
                f"Pixiv returned non-JSON for {endpoint} "
                f"(HTTP {resp.status_code})"
            ) from exc
        error = body.get("error") if isinstance(body, dict) else None
        message = ""
        if isinstance(error, dict):
            message = str(
                error.get("user_message") or error.get("message") or ""
            )
        # Rate limiting first: the app API reports it as an error MESSAGE
        # (often on HTTP 403), which must not be mistaken for an auth failure.
        if resp.status_code == 429 or "rate limit" in message.lower():
            raise PixivAPIError(
                f"Pixiv rate limit hit ({endpoint}): {message or 'HTTP 429'}",
                status_code=429,
                retry_after=_RATE_LIMIT_RETRY_AFTER,
            )
        if resp.status_code in (400, 401, 403):
            # Invalid/expired access token surfaces as 400 invalid_grant-style
            # errors on the app API; 401/403 are straight auth rejections.
            raise PixivAuthError(
                f"Pixiv rejected the request ({endpoint}, HTTP "
                f"{resp.status_code}): {message or 'auth rejected'} — "
                "rotate the Pixiv refresh token.",
                status_code=resp.status_code,
            )
        if resp.status_code >= 400:
            raise PixivAPIError(
                f"Pixiv API error ({endpoint}, HTTP {resp.status_code}): "
                f"{message or 'unknown error'}",
                status_code=resp.status_code,
            )
        if error:
            # HTTP 200 carrying an error object — unexpected, but never
            # silently treat it as data.
            raise PixivAPIError(
                f"Pixiv API error ({endpoint}): {message or error}",
                status_code=resp.status_code,
            )
        return body
    # -- normalization -------------------------------------------------------
    @staticmethod
    def _normalize(work: dict) -> dict:
        """Wrap an app-API work in the `{"id", "attributes", ...}` post shape
        the platform-agnostic core and shared helpers read. The raw work rides
        along under `_work` for extract_media / the post record."""
        title = work.get("title")
        caption = work.get("caption")
        wtype = work.get("type")
        return {
            "id": work.get("id"),
            "attributes": {
                "title": title if isinstance(title, str) else "",
                "content": caption if isinstance(caption, str) else "",
                "published_at": work.get("create_date"),
                "post_type": wtype if isinstance(wtype, str) else "illust",
            },
            "_work": work,
        }
    # -- post-first seams ----------------------------------------------------
    @staticmethod
    def post_record_key(post: dict) -> tuple[str, str] | None:
        """`(ledger_key, post_id)` gating post-record capture through the seen
        ledger (`post:<id>` synthetic key), or None when the work has no id."""
        pid = post.get("id")
        pid = str(pid) if pid is not None else ""
        if not pid:
            return None
        return (f"post:{pid}", pid)
    @staticmethod
    def post_meta(post: dict) -> dict:
        attrs = post.get("attributes") or {}
        return {"title": attrs.get("title") or None, "date": attrs.get("published_at")}
    @staticmethod
    def post_is_gated(post: dict) -> bool:
        """True when this account cannot fetch the work's real files: pixiv
        substitutes a `limit_*` placeholder for the original (sanity-level
        filter / my-pixiv lock / deleted), or zeroes the author (deleted
        account). Mirrors #874 semantics: gated content leaves NO trace — a
        placeholder thumbnail and an empty stub would only pollute the
        archive. (gallery-dl's PHPSESSID web-scrape fallback for these is out
        of scope: FC holds no pixiv browser cookies — module docstring.)"""
        work = post.get("_work") or {}
        user = work.get("user") or {}
        if not user.get("id"):
            return True
        if work.get("meta_pages"):
            return False
        single = work.get("meta_single_page") or {}
        original = single.get("original_image_url")
        return isinstance(original, str) and original.startswith(_LIMIT_URL)
    # -- media ---------------------------------------------------------------
    def extract_media(self, post: dict, included_index: dict) -> list[MediaItem]:
        """Resolve a work's downloadable files (gallery-dl's `_extract_files`):
        multi-page originals, the single-page original, or the ugoira frame
        zip. `included_index` is unused (pixiv works are self-contained)."""
        work = post.get("_work") or {}
        pid = str(post.get("id") or "")
        if not pid or self.post_is_gated(post):
            return []
        if work.get("type") == "ugoira":
            return self._ugoira_media(work, pid)
        meta_pages = work.get("meta_pages") or []
        if meta_pages:
            items = []
            for num, page in enumerate(meta_pages):
                urls = page.get("image_urls") or {}
                url = urls.get("original")
                if not isinstance(url, str) or not url:
                    continue
                items.append(
                    MediaItem(
                        url=url,
                        filename=_work_filename(work, num, url),
                        kind="image",
                        filehash=None,
                        post_id=pid,
                        media_id=f"p{num}",
                    )
                )
            return items
        single = work.get("meta_single_page") or {}
        url = single.get("original_image_url")
        if not isinstance(url, str) or not url or url.startswith(_LIMIT_URL):
            return []
        return [
            MediaItem(
                url=url,
                filename=_work_filename(work, 0, url),
                kind="image",
                filehash=None,
                post_id=pid,
                media_id="p0",
            )
        ]
    def _ugoira_meta(self, work: dict, pid: str) -> dict | None:
        """Fetch + memoize the ugoira metadata (frames + zip urls) for a work.
        Idempotent and cached on the work dict, so the post record and the
        media extraction share ONE `/v1/ugoira/metadata` call regardless of
        which runs first (the core writes the post record BEFORE it extracts
        media). Returns None — and caches the miss — on a non-auth failure
        (matching gallery-dl's downgrade); auth failures stay loud."""
        if "_ugoira_meta" in work:
            return work["_ugoira_meta"]
        try:
            body = self._call("/v1/ugoira/metadata", {"illust_id": pid})
            meta = body["ugoira_metadata"]
        except PixivAuthError:
            raise
        except (PixivAPIError, KeyError, TypeError) as exc:
            log.warning("Pixiv ugoira metadata failed for %s: %s", pid, exc)
            work["_ugoira_meta"] = None
            return None
        work["_ugoira_meta"] = meta
        # Frame delays: a future ugoira→video conversion needs the timings (the
        # zip alone has none), so the post record captures them.
        work["_ugoira_frames"] = meta.get("frames") or []
        return meta
    def fetch_ugoira_frames(self, post: dict) -> None:
        """Populate `post['_work']['_ugoira_frames']` for an ugoira post (no-op
        otherwise). The core writes the post record BEFORE extract_media, so
        without this the frame timings would never reach the record; this
        fetches (and memoizes, so extract_media reuses it) the metadata. Injected
        into the downloader by the ingester, mirroring Patreon's content_fetcher.
        Auth errors propagate; other failures leave frames unset."""
        work = post.get("_work") or {}
        if work.get("type") != "ugoira":
            return
        pid = str(post.get("id") or "")
        if pid:
            self._ugoira_meta(work, pid)
    def _ugoira_media(self, work: dict, pid: str) -> list[MediaItem]:
        """The ugoira frame zip (gallery-dl's default non-original mode):
        `/v1/ugoira/metadata` → zip_urls.medium with the 600x600→1920x1080
        swap. A metadata failure downgrades to 'no media' with a warning
        (matching gallery-dl) instead of failing the walk — except auth
        failures, which stay loud."""
        meta = self._ugoira_meta(work, pid)
        if meta is None:
            return []
        try:
            zip_url = meta["zip_urls"]["medium"]
        except (KeyError, TypeError) as exc:
            log.warning("Pixiv ugoira zip url missing for %s: %s", pid, exc)
            return []
        url = zip_url.replace("_ugoira600x600", "_ugoira1920x1080", 1)
        return [
            MediaItem(
                url=url,
                filename=_work_filename(work, 0, url),
                kind="ugoira",
                filehash=None,
                post_id=pid,
                media_id="ugoira",
            )
        ]
    # -- iteration -----------------------------------------------------------
    def iter_posts(
        self, campaign_id: str, cursor: str | None = None
    ) -> Iterator[tuple[dict, dict, str | None]]:
        """Yield (post, {}, page_cursor) for every work in the user's feed.
        `campaign_id` is the numeric pixiv user id. `cursor` is the query
        string of the app API's `next_url` (offset pagination); None fetches
        page 1. The yielded `page_cursor` is the cursor that FETCHED this
        work's page, so the core checkpoints a value that re-serves the same
        page on resume (the shared cursor contract)."""
        if not str(campaign_id or "").isdigit():
            raise PixivDriftError(
                f"Pixiv campaign id must be a numeric user id, got "
                f"{campaign_id!r}"
            )
        current = cursor
        while True:
            page_cursor = current
            if current is None:
                params: dict = {"user_id": campaign_id}
            else:
                params = dict(parse_qsl(current))
            data = self._call("/v1/user/illusts", params)
            works = data.get("illusts")
            if not isinstance(works, list):
                raise PixivDriftError(
                    "Pixiv user-illusts response had no 'illusts' list "
                    f"(keys: {sorted(data)[:8]})"
                )
            for work in works:
                if not isinstance(work, dict):
                    continue
                yield self._normalize(work), {}, page_cursor
            next_url = data.get("next_url")
            if not next_url:
                return
            current = str(next_url).rpartition("?")[2]
    # -- user detail ---------------------------------------------------------
    def resolve_display_name(self, user_id: str) -> str | None:
        """The pixiv user's display name via `/v1/user/detail` (gallery-dl's
        user_detail) — used to name the Artist when a source is added by numeric
        id. None on any failure (the caller falls back to the id)."""
        try:
            body = self._call("/v1/user/detail", {"user_id": str(user_id)})
        except PixivAPIError:
            return None
        name = (body.get("user") or {}).get("name") if isinstance(body, dict) else None
        return name if isinstance(name, str) and name.strip() else None
    # -- verify --------------------------------------------------------------
    def verify_auth(self) -> tuple[bool | None, str]:
        """Cheap credential probe: run the OAuth refresh (the thing that fails
        when the token is bad) without walking any feed."""
        try:
            self._token_deadline = 0.0  # force a real refresh
            self._login()
        except PixivAuthError as exc:
            return False, f"Pixiv rejected the credential — {exc}"
        except PixivAPIError as exc:
            return None, f"Couldn't verify (network/HTTP issue): {exc}"
        account = self._authed_user.get("account") or self._authed_user.get("name")
        suffix = f" as {account}" if account else ""
        return True, f"Credentials valid — Pixiv OAuth refresh succeeded{suffix}."
 def rating_label(x_restrict) -> str | None:
    """Human rating from pixiv's x_restrict (0/1/2) — written into the post
    record so the archive keeps the R-18 flag without the reader needing to
    know pixiv's numeric scheme."""
    if isinstance(x_restrict, bool) or not isinstance(x_restrict, int):
        return None
    return _RATINGS.get(x_restrict)
@@ -1,276 +0,0 @@
 """Native Pixiv media downloader — the Pixiv counterpart to
 patreon_downloader / subscribestar_downloader.
 Given a normalized Pixiv work and its resolved `MediaItem`s
 (pixiv_client.extract_media), download the originals to gallery-dl's on-disk
 layout (so pre-cutover gallery-dl downloads are recognized on disk and not
 re-fetched), write the post-first sidecars the importer consumes, and report
 per-media outcomes.
 On-disk layout (matches FC's gallery-dl pixiv config, PLATFORM_DEFAULTS:
 base-directory `<images_root>/<artist_slug>/pixiv` + `directory:
 ["{category}"]` + filename `{id}_{title[:50]}_{num:>02}.{extension}`):
    <images_root>/<artist_slug>/pixiv/pixiv/<id>_<title50>_<NN>.<ext>
 — note the intentional DOUBLE `pixiv` segment: gallery-dl appended
 `{category}` under a base-directory that already ended in the platform name,
 and tier-2 disk-skip parity requires reproducing that exactly. The layout is
 FLAT (no per-post directory), so the post-first record is `_post_<id>.json`
 in the same directory (the id suffix prevents the collisions a bare
 `_post.json` would have here; phase 3 receives explicit post_record_paths, so
 the name is a convention, not a discovery key).
 Simpler than Patreon (no Mux/yt-dlp video branch) — the one special file is
 the ugoira frame zip, downloaded as-is; FC's archive-containment import
 extracts the frames, and the frame DELAYS ride the post record (the zip
 carries none — a future ugoira→video conversion needs them).
 PURE: no DB; the seen-skip is an injected predicate. FC runs on a plain-HTTP
 homelab; nothing here uses a secure-context Web API.
 """
 from __future__ import annotations
 import json
 import logging
 import re
 import time
 from collections.abc import Callable
 from pathlib import Path
 import requests
 from .native_ingest_common import (
    BaseNativeDownloader,
    MediaOutcome,
    PostRecordOutcome,
 )
 from .pixiv_client import PIXIV_APP_HEADERS, rating_label
 log = logging.getLogger(__name__)
 # Control chars (0x00–0x1f + 0x7f DEL) — gallery-dl's default `path-remove`.
 _GDL_PATH_REMOVE_RE = re.compile(r"[\x00-\x1f\x7f]")
 def gdl_clean_filename(name: str) -> str:
    """Reproduce gallery-dl's on-disk filename EXACTLY as it wrote it on this
    Linux host, so the tier-2 disk-skip recognizes pre-cutover files instead of
    re-downloading them.
    gallery-dl's PathFormat.build_filename is `clean_path(clean_segment(name))`.
    On Linux (verified against gallery-dl 1.32.5 path.py) the defaults resolve to:
      - path-restrict "auto" → "/"          → clean_segment replaces ONLY "/" → "_"
      - path-remove   "\\x00-\\x1f\\x7f"       → clean_path DELETES control chars
      - path-strip    "auto" → ""           → NO trailing dot/space stripping
    Crucially it does NOT touch the Windows-forbidden set (<>:"|?*) — those stay
    raw in titles on disk. A stricter sanitizer here would rename any such title,
    miss the on-disk match, and re-pull the whole work. Order mirrors gallery-dl
    (segment inner, path outer); for these disjoint char sets it's commutative.
    """
    return _GDL_PATH_REMOVE_RE.sub("", name.replace("/", "_"))
 # Enrichment keys copied verbatim from the app-API work dict into the post
 # record (they're already JSON scalars/objects). Everything lands in
 # Post.raw_metadata via the importer, so the archive keeps pixiv's stats and
 # structure without a schema change.
 _WORK_PASSTHROUGH_KEYS = (
    "type",
    "page_count",
    "width",
    "height",
    "total_view",
    "total_bookmarks",
    "total_comments",
    "is_bookmarked",
    "illust_ai_type",
    "series",
 )
 class PixivDownloader(BaseNativeDownloader):
    """Download resolved Pixiv media to gallery-dl's on-disk layout.
    Subclasses BaseNativeDownloader for the shared streaming GET
    (transient-retry + Range-resume) and validation/quarantine. PURE: no DB."""
    def __init__(
        self,
        images_root: Path,
        cookies_path: str | None = None,
        *,
        validate: bool = True,
        rate_limit: float = 0.0,
        session: requests.Session | None = None,
        ugoira_frames_fetcher: Callable[[dict], None] | None = None,
    ):
        super().__init__(
            images_root, cookies_path, platform="pixiv",
            validate=validate, rate_limit=rate_limit, session=session,
        )
        # Injected by the ingester (client.fetch_ugoira_frames) so write_post_record
        # can populate frame timings — which extract_media memoizes, but the core
        # writes the post record FIRST. Mirrors Patreon's content_fetcher.
        self._ugoira_frames_fetcher = ugoira_frames_fetcher
        if session is None:
            # i.pximg.net 403s any GET without the app Referer; mirror the
            # client's full app-header profile (gallery-dl serves media off
            # the same session it drives the API with). An injected session
            # (tests) owns its own headers.
            self.session.headers.update(PIXIV_APP_HEADERS)
    # -- public ------------------------------------------------------------
    def download_post(
        self,
        post: dict,
        media_items: list,
        artist_slug: str,
        *,
        is_seen: Callable[[object], bool] = lambda m: False,
        should_stop: Callable[[], bool] = lambda: False,
        recapture: bool = False,
    ) -> list[MediaOutcome]:
        """Download every media item of one work; return per-item outcomes.
        Mirrors SubscribeStarDownloader.download_post (two-tier skip, mid-post
        time-box, recapture surfacing)."""
        flat_dir = self._flat_dir(artist_slug)
        outcomes: list[MediaOutcome] = []
        for media in media_items:
            if should_stop():
                break
            try:
                outcomes.append(
                    self._download_one(
                        post, media, flat_dir, artist_slug, is_seen,
                        recapture=recapture,
                    )
                )
            except Exception as exc:  # resilient: isolate one item's failure
                log.warning(
                    "Pixiv media failed (work %s, %s): %s",
                    post.get("id"), getattr(media, "media_id", "?"), exc,
                )
                outcomes.append(
                    MediaOutcome(media=media, status="error", path=None, error=str(exc))
                )
        return outcomes
    def _flat_dir(self, artist_slug: str) -> Path:
        # Double platform segment — gallery-dl layout parity (module docstring).
        return self.images_root / artist_slug / "pixiv" / "pixiv"
    # -- per-item ----------------------------------------------------------
    def _download_one(
        self,
        post: dict,
        media,
        flat_dir: Path,
        artist_slug: str,
        is_seen: Callable[[object], bool],
        *,
        recapture: bool = False,
    ) -> MediaOutcome:
        seen = is_seen(media)
        if seen and not recapture:
            return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
        # The client's filename already carries the {id}_{title50}_{NN} shape
        # (raw title, gallery-dl-template order); clean it to the byte-exact
        # name gallery-dl wrote on disk so tier-2 disk-skip matches (else a
        # re-download of the whole work). See gdl_clean_filename.
        media_path = flat_dir / gdl_clean_filename(media.filename)
        if media_path.exists():  # tier-2: already on disk
            return MediaOutcome(
                media=media, status="skipped_disk", path=media_path, error=None
            )
        # recapture: a seen item not on disk is NOT re-downloaded (recovery's job).
        if seen:
            return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
        flat_dir.mkdir(parents=True, exist_ok=True)
        if self._rate_limit > 0:
            time.sleep(self._rate_limit)
        out_path = self._fetch_get(media.url, media_path)
        reason, quarantine_dest = self._validate_path(out_path, artist_slug, media.url)
        if reason is not None:
            return MediaOutcome(
                media=media, status="quarantined", path=quarantine_dest, error=reason,
            )
        self._write_minimal_sidecar(post, out_path, source_url=media.url)
        return MediaOutcome(media=media, status="downloaded", path=out_path, error=None)
    # -- post record ---------------------------------------------------------
    def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
        """Write the post-first `_post_<id>.json` — the sole writer of the post
        body/metadata on the native path. Beyond the standard body fields, the
        record carries pixiv's own structure (tags + EN translations, rating,
        series, view/bookmark counts, AI flag, dimensions, author, ugoira frame
        delays) so the archive keeps what the platform knows about the work."""
        attrs = post.get("attributes") or {}
        work = post.get("_work") or {}
        title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
        post_type = attrs.get("post_type") if isinstance(attrs.get("post_type"), str) else None
        pid = str(post.get("id") or "")
        if not pid:
            return PostRecordOutcome(
                path=None, post_type=post_type, title=title, body_chars=0,
            )
        content = attrs.get("content")
        content = content if isinstance(content, str) else ""
        data: dict = {
            "category": "pixiv",
            "id": pid,
            "title": title or "",
            "content": content,
            "published_at": attrs.get("published_at"),
            # The post permalink is synthesized by platforms/pixiv.py
            # derive_post_url from `id` at parse time — no url key here.
            "rating": rating_label(work.get("x_restrict")),
        }
        for key in _WORK_PASSTHROUGH_KEYS:
            if key in work:
                data[key] = work[key]
        tags = work.get("tags")
        if isinstance(tags, list):
            data["tags"] = [
                {
                    "name": t.get("name"),
                    "translated_name": t.get("translated_name"),
                }
                for t in tags
                if isinstance(t, dict)
            ]
        user = work.get("user")
        if isinstance(user, dict):
            data["user"] = {
                "id": user.get("id"),
                "account": user.get("account"),
                "name": user.get("name"),
            }
        # Ugoira frame timings. extract_media memoizes these, but the core writes
        # the post record BEFORE extracting media, so fetch them here (shared +
        # idempotent via the client's memoization) so the record actually keeps
        # them — the zip carries no timings.
        if (
            work.get("type") == "ugoira"
            and not work.get("_ugoira_frames")
            and self._ugoira_frames_fetcher is not None
        ):
            self._ugoira_frames_fetcher(post)
        frames = work.get("_ugoira_frames")
        if frames:
            data["ugoira_frames"] = frames
        flat_dir = self._flat_dir(artist_slug)
        flat_dir.mkdir(parents=True, exist_ok=True)
        path = flat_dir / f"_post_{pid}.json"
        path.write_text(json.dumps(data, indent=2, ensure_ascii=False))
        return PostRecordOutcome(
            path=path, post_type=post_type, title=title, body_chars=len(content),
        )
@@ -1,121 +0,0 @@
 """Native Pixiv ingester — the Pixiv ADAPTER over the platform-agnostic core
 (`ingest_core.Ingester`).
 Thin counterpart to patreon_ingester / subscribestar_ingester: wires the Pixiv
 client/downloader/ledger models/constraints/key into the core and supplies the
 Pixiv failure mapping. The modes (tick / backfill / recovery / recapture), the
 seen + dead-letter ledgers, cursor checkpointing, and the post-first capture
 all live in the core. `download_service.download_source` drives
 `PixivIngester.run` exactly as it drives the other two.
 `campaign_id` is the numeric pixiv user id (download_backends extracts it from
 the source URL — no network resolver). Auth is the operator's OAuth refresh
 token (the token-type Credential), passed as `auth_token` — pixiv is the first
 native platform authenticating by token rather than cookies, so the uniform
 constructor accepts both and ignores what it doesn't need.
 FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
 """
 from __future__ import annotations
 import asyncio
 import logging
 from collections.abc import Callable
 from pathlib import Path
 from ..models import PixivFailedMedia, PixivSeenMedia
 from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
 from .pixiv_client import MediaItem, PixivAPIError, PixivClient
 from .pixiv_downloader import PixivDownloader
 __all__ = [
    "DEAD_LETTER_THRESHOLD",
    "PixivIngester",
    "_ledger_key",
    "verify_pixiv_credential",
 ]
 log = logging.getLogger(__name__)
 _LEDGER_KEY_MAX = 128
 def _ledger_key(media: MediaItem) -> str:
    """Stable per-media identity for the cross-run seen-ledger. Pixiv original
    URLs carry no content hash, so the key is the page/zip identity scoped to
    its work: `<illust_id>:p<num>` / `<illust_id>:ugoira`. Bounded to the
    column width."""
    if media.filehash:
        return media.filehash
    return f"{media.post_id}:{media.media_id}"[:_LEDGER_KEY_MAX]
 class PixivIngester(Ingester):
    """Walk a pixiv user's works, download unseen originals, return a
    `DownloadResult`. A thin adapter over `ingest_core.Ingester`; `client` /
    `downloader` are injectable seams so unit tests run without network."""
    def __init__(
        self,
        images_root: Path,
        cookies_path: str | None,
        session_factory: Callable[[], object],
        *,
        validate: bool = True,
        rate_limit: float = 0.0,
        request_sleep: float = 0.0,
        auth_token: str | None = None,
        client: PixivClient | None = None,
        downloader: PixivDownloader | None = None,
    ):
        self.images_root = Path(images_root)
        self.cookies_path = str(cookies_path) if cookies_path else None
        resolved_client = (
            client
            if client is not None
            else PixivClient(auth_token, request_sleep=request_sleep)
        )
        resolved_downloader = (
            downloader
            if downloader is not None
            else PixivDownloader(
                self.images_root, cookies_path, validate=validate, rate_limit=rate_limit,
                # write_post_record runs before extract_media in the core, so it
                # fetches ugoira frame timings via the SAME client (shared,
                # memoized) — else the record's ugoira_frames stays empty.
                ugoira_frames_fetcher=resolved_client.fetch_ugoira_frames,
            )
        )
        super().__init__(
            client=resolved_client,
            downloader=resolved_downloader,
            session_factory=session_factory,
            seen_model=PixivSeenMedia,
            failed_model=PixivFailedMedia,
            seen_constraint="uq_pixiv_seen_media_source_id",
            failed_constraint="uq_pixiv_failed_media_source_id",
            ledger_key=_ledger_key,
            platform="pixiv",
            error_base=PixivAPIError,
            # API_DRIFT message phrasing; the base Ingester._failure_result owns
            # the auth/drift/HTTP→error_type mapping (shared across platforms).
            drift_label="Pixiv app API",
            # Captions are legitimately empty for many pixiv artists, so the
            # zero-bodies #862 canary would false-positive here; the client's
            # response-shape checks (missing `illusts` → drift) cover the same
            # failure class structurally.
            body_canary=False,
        )
 async def verify_pixiv_credential(
    auth_token: str | None,
 ) -> tuple[bool | None, str]:
    """Native Pixiv credential probe — one OAuth refresh via
    PixivClient.verify_auth (the exact call that fails when the token is
    bad; no feed walk). Returns the uniform `(ok, message)` contract so
    download_backends.verify_source_credential treats it like the others."""
    client = PixivClient(auth_token)
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(None, client.verify_auth)
@@ -23,8 +23,10 @@ log = logging.getLogger(__name__)
 # Platforms walked one-at-a-time. gallery-dl platforms are intentionally NOT
 # here: each runs as a self-pacing subprocess and they're lower-volume. The
 # native-ingester platforms are serialized (one paced scrape/API walk at a time).
-# Add a platform here to cap it to a single concurrent walk.
+# Add a platform here to cap it to a single concurrent walk. Discord most of
-SERIALIZED_PLATFORMS = frozenset({"patreon", "subscribestar"})
+# all: every source walks on the operator's ONE user token, and parallel walks
 # on a user account are both how its rate limit trips and what gets it flagged.
 SERIALIZED_PLATFORMS = frozenset({"patreon", "subscribestar", "discord"})
 _LOCK_PREFIX = "fc:download_lock:"
@@ -13,9 +13,8 @@ URL patterns match GS exactly so the existing browser extension
 hits FC unmodified. deviantart was dropped at #3069 (2026-08-27) —
 FC downloaders are art-dedicated services only. pixiv was retired at
 milestone #406 (2026-09-13, rule #171): unregistered here first, which
-switches it off everywhere this registry is consulted; `pixiv.py` and the
+switched it off everywhere this registry is consulted, then removed from
-pixiv client/downloader/ingester stay in the tree, uncalled, until the
+the tree entirely in the milestone's phase 2 (2026-09-21).
 milestone's phase 2 deletes them.
 """
 from .base import (
@@ -24,8 +24,8 @@ from typing import Literal
 # external_post_id chain: `post_id` MUST come before `id` because
 # SubscribeStar gallery-dl puts the per-attachment id in `id` and the
 # actual post id in `post_id`; picking `id` first fragments
-# multi-image SubscribeStar posts into N Post rows. Patreon/Pixiv have
+# multi-image SubscribeStar posts into N Post rows. Patreon has
-# no `post_id` so `id` still wins for them; HF uses `index`, Discord
+# no `post_id` so `id` still wins for it; HF uses `index`, Discord
 # uses `message_id` — all reached via the remaining chain entries.
 # (Banked 2026-05-27 during the sidecar audit.)
 DEFAULT_EXTERNAL_POST_ID_KEYS: tuple[str, ...] = (
@@ -62,7 +62,7 @@ class PlatformInfo:
    # --- Behavioral hooks ---
    # Synthesize a post permalink from sidecar data. Required when
    # gallery-dl's `url` field is the file/CDN URL rather than the post
-    # permalink (subscribestar/pixiv/hf/discord). None = trust the bare
+    # permalink (subscribestar/hf/discord). None = trust the bare
    # `url` field (patreon).
    derive_post_url: Callable[[dict], str | None] | None = None
@@ -1,38 +0,0 @@
 """Pixiv — one quirk.
 post_url: the sidecar's `url` (legacy gallery-dl era) is the image URL
 on `i.pximg.net`, and the native post record (#129) writes no url key
 at all — the permalink is synthesized from `id` here either way:
 /artworks/<id>. external_post_id (= `id`) was already correct, so no
 override there.
 Downloads run through the native ingester (pixiv_ingester.py), not
 gallery-dl; this registry entry still owns URL validation, sidecar
 parsing, and the credential surface (the OAuth refresh token).
 """
 from .base import GD_DEFAULTS, PlatformInfo, str_id_value
 def derive_post_url(data: dict) -> str | None:
    pid = str_id_value(data.get("id"))
    if pid:
        return f"https://www.pixiv.net/artworks/{pid}"
    return None
 INFO = PlatformInfo(
    key="pixiv",
    name="Pixiv",
    description="Download artwork from Pixiv artists",
    auth_type="token",
    requires_auth=True,
    url_pattern=r"^https?://(www\.)?pixiv\.net/",
    url_examples=[
        "https://www.pixiv.net/users/12345678",
        "https://www.pixiv.net/en/users/12345678",
    ],
    default_config={**GD_DEFAULTS, "content_types": ["all"]},
    notes="Requires OAuth refresh token. Run `gallery-dl oauth:pixiv` to obtain one.",
    derive_post_url=derive_post_url,
 )
@@ -15,24 +15,56 @@ where they already were, a wrong one actively misinforms and then propagates
 into whatever reads the association. So the matcher's job is to make a SHORT
 list worth reading, not a long list worth trusting.
-## Signals, and the one deliberately NOT built
+## Two routes, because the evidence is of two different kinds
 CIRCUMSTANTIAL evidence says two things happened near each other. It is
 additive, weighted, and no single one of its signals may reach the threshold:
 1. **Time proximity.** The Patreon post exists in order to announce the drop,
   so the two are minutes-to-hours apart. Nearly free, and strong.
 2. **The post says so.** These announcements routinely name Discord or carry
   an invite link, which is close to a declaration.
 3. **A shared marker.** The creator's own tie-back — `🍈🍈` in the Patreon
   title and `@everyone 🍈 🍈` in the Discord message — gated on how rare that
   marker is in THIS artist's posts, because a habitual emoji is punctuation.
-3. **Crop-to-source matching is HELD, on the plan's own instruction** — it is
+IDENTITY evidence says two things are the same thing, and it gets its own
-   real work with real false-positive risk, and it is only worth building once
+route (see `IDENTITY_FLOOR`). Two signals, and the stronger one stands rather
-   1 and 2 are shown to be insufficient against the operator's actual artists.
+than them being summed — saying "the same piece" twice is not more true:
   Nothing here should be read as evidence it is unnecessary; it is deferred,
   and the thing that would justify it is an empty review queue on a pair the
   operator can see with their own eyes.
-   Note also that a naive whole-image SigLIP similarity is NOT that signal. A
+4. **A shared working name.** The creator exports the teaser and the release
-   cropped teaser and its full version are exactly the pair a whole-image
+   from one file, and the internal name survives into both platforms
-   comparison handles worst, so adding one as a "bonus" would mostly add noise
+   untouched. Measured on the operator's artist: `ConnFront` ↔ `ConnFront`.
-   while looking like progress.
+   This is the only signal that reaches a pair 23.8 hours apart, which
   proximity scores at 0.005.
 5. **The drop contains the teaser's image.** Rare, and near-certain when it
   happens. It is the one signal needing no cooperation from the creator: it
   works on a teaser called `Screenshot 2026-08-13`, and on a creator whose
   two platforms share no naming convention.
 ## The one deliberately NOT built
 **Crop-to-source matching stays held, and now for a measured reason rather
 than a cautious one.**
 It was deferred until the cheap signals could be shown insufficient. They can:
 of artist 8's 27 teasers with a drop inside a day, 11 still go unlinked, and
 five of those are screenshot teasers carrying no working name at all.
 So it was tried, on those exact pairs. Every teaser image was correlated
 against every window of every nearby drop image at five scales, with the pairs
 the working name independently confirms as ground truth and unrelated
 same-artist posts a month away as a control. **It does not separate.** True
 pairs score as low as 0.401 while the control reaches 0.605 — the two
 distributions overlap, and no threshold divides them.
 The reason is the reason the naive version was rejected in the first place,
 and it turns out to apply just as hard to the sophisticated one: one artist's
 work is stylistically homogeneous, so any whole-image comparison between two
 of their pieces is high whether or not it is the same piece. Signal 5 above is
 what survived that experiment — it asks a narrower question ("is this the same
 image") that the measurement shows is answerable, instead of a broader one
 ("is this a crop of that") that it shows is not.
 ## Creator identity comes free, so E4 is not actually a prerequisite
@@ -56,14 +88,27 @@ from __future__ import annotations
 import logging
 import re
 from collections import Counter
 from dataclasses import dataclass
 from datetime import UTC, datetime, timedelta
-from sqlalchemy import func, or_, select
+from sqlalchemy import and_, func, or_, select
 from sqlalchemy.ext.asyncio import AsyncSession
-from ..models import ImportSettings, Post, PostAssociation
+from ..models import ImageRecord, ImportSettings, Post, PostAssociation
 from ..utils.phash import hamming, hash_bits
 from ..utils.text import html_to_plain
 from .discord_grouping import DROP_GROUPER
 from .post_naming import (
    IDENTITY_FLOOR,
    MAX_TOKEN_POSTS,
    marker_frequencies,
    marker_overlap,
    rarity,
    shared_identity,
    token_frequencies,
    working_name_tokens,
 )
 log = logging.getLogger(__name__)
@@ -76,18 +121,83 @@ log = logging.getLogger(__name__)
 # enough" arithmetic rather than aspirational: on a busy day an artist posts
 # several times, and a matcher that could pair on proximity alone would turn
 # every busy day into false pairs. A guard test pins this.
-WEIGHTS = {"proximity": 0.55, "declared": 0.45}
+WEIGHTS = {"proximity": 0.45, "declared": 0.35, "marker": 0.20}
 # A Discord INVITE in the body is close to a declaration; the bare word is
 # weaker but still meaningful, because these posts are short and on-topic.
 #
 # "the server" and its possessives are here because the word `discord` is NOT
 # how these creators actually write. Measured across 20,558 Patreon bodies:
 # `discord` appears in 486 and `the server` in 37 — but the distribution is the
 # point, not the totals. For the artist this step was built for, 21 of 42 posts
 # say `discord` and 7 say `the server`, and it is the RECENT ones that say the
 # latter: the phrasing drifted once the audience already knew where the server
 # was. A vocabulary list written from old posts silently stops matching.
 _INVITE = re.compile(r"discord\.(?:gg|com/invite)/", re.I)
-_MENTION = re.compile(r"\bdiscord\b", re.I)
+_MENTION = re.compile(r"\b(?:discord|(?:the|our|my)\s+server)\b", re.I)
 DECLARED_INVITE = 1.0
 DECLARED_MENTION = 0.6
 MAX_CANDIDATES = 25
 # How many just-grouped drops one sweep will look around. A ceiling on the
 # `or_` the sweep builds, not a policy — a backfill that authors thousands of
 # drops at once should not turn one sweep into a full-library rescan, which is
 # the manual button's job.
 MAX_RECENT_DROPS = 200
 # What a shared name must reach before FC links a pair WITHOUT asking.
 #
 # 1.0, which under post_naming's post-span counting means the name appears in
 # exactly these two posts and nowhere else in the artist's library. That is not
 # "strong evidence" — within the library it is conclusive, and the remaining
 # ways to be wrong are a mis-parse or the creator reusing a name for a genuinely
 # different piece on the same day.
 #
 # Deliberately above IDENTITY_FLOOR, which is what a name needs to PROPOSE.
 # The gap between them is the review queue: real evidence, not certain enough
 # for FC to act on by itself. Measured on artist 8, 15 name-sharing pairs: 11
 # are conclusive, 2 more propose, 2 fall short of both.
 AUTO_LINK_FLOOR = 1.0
 # When the drop simply CONTAINS the teaser's image — a pHash within this many
 # of 256 bits.
 #
 # 32, the same number and unit `gallery_service._diversify_similar` already
 # calls a near-duplicate. Measured on artist 8, comparing every teaser against
 # every drop within a day: pairs the working name independently confirms score
 # 0, 0 and 20, and the nearest unrelated same-artist pair in a 29-sample
 # control scores **108**. A 76-bit gap, so the threshold is not finely tuned
 # and does not need to be.
 #
 # `utils/phash` warns that the hash alone must not decide a MERGE, because
 # variants of one piece collide at this distance. That warning does not invert
 # here, it is the point: merging destroys a file, so a variant colliding with
 # its original is a loss, while this is asking whether two POSTS are about the
 # same piece — and a variant of the drop's image is exactly that. Nothing is
 # deleted either way, so no pixel confirm is needed to accept.
 DUPLICATE_MAX_DISTANCE = 32
@dataclass(frozen=True)
 class _Corpus:
    """One artist's rare-token evidence, gathered once rather than per pair.
    Both rare-token signals are scoped to a single artist — a working name and
    a marker belong to the person who chose them — so the counts are useless
    across artists and expensive to rebuild per candidate. A sweep touches an
    artist's posts many times over; this is loaded on the first touch and kept
    for the life of the service.
    """
    tokens_by_post: dict[int, set[str]]
    token_posts: Counter[str]
    text_by_post: dict[int, str]
    marker_posts: Counter[str]
    hashes_by_post: dict[int, list[int]]
    hash_posts: Counter[int]
 def proximity_signal(gap: timedelta, window: timedelta) -> float:
    """1.0 when the two posts are simultaneous, decaying linearly to 0 at the
@@ -123,6 +233,40 @@ def declared_signal(description: str | None) -> float:
    return 0.0
 def shared_image(
    left: list[int],
    right: list[int],
    hash_posts: Counter[int],
    *,
    max_distance: int = DUPLICATE_MAX_DISTANCE,
    max_frequency: int = MAX_TOKEN_POSTS,
 ) -> float:
    """Strength in [0, 1] that the drop contains the teaser's own image.
    IDENTITY evidence, and the only one of the three that needs no cooperation
    from the creator — it works on a teaser named `Screenshot 2026-08-13`, and
    on a creator whose two platforms share no naming convention at all. Where
    it fires it is close to certain; it is simply quiet most of the time,
    because a teaser is usually a crop rather than a copy.
    Rarity-gated on POSTS like the other two: an image the creator puts on many
    posts is a banner, not a piece.
    """
    if not left or not right:
        return 0.0
    best = None
    for a in left:
        for b in right:
            d = hamming(a, b)
            if d is None or d > max_distance:
                continue
            span = max(hash_posts.get(a, 1), hash_posts.get(b, 1), 1)
            strength = rarity(span, max_frequency)
            if best is None or strength > best:
                best = strength
    return round(best, 4) if best is not None else 0.0
 def weighted_score(signals: dict) -> float:
    return round(sum(WEIGHTS[k] * signals.get(k, 0.0) for k in WEIGHTS), 4)
@@ -134,81 +278,298 @@ def _post_time(post: Post) -> datetime:
 class PostAssociationService:
    def __init__(self, session: AsyncSession):
        self.session = session
        self._corpora: dict[int, _Corpus] = {}
-    async def _decided(self, announcement_id: int) -> set[int]:
+    async def _corpus(self, artist_id: int) -> _Corpus:
-        """Payload posts already proposed for this announcement, in ANY status.
+        if artist_id in self._corpora:
            return self._corpora[artist_id]
-        Dismissed pairs are included deliberately: re-proposing a pair the
+        paths_by_post: dict[int, list[str]] = {}
-        operator has already rejected on every subsequent scan is the single
+        hashes_by_post: dict[int, list[int]] = {}
-        behaviour that makes a review queue get ignored.
+        # An image is counted under the post a reader SEES it on: a Discord
        # message absorbed into a drop contributes to the drop, not to itself.
        #
        # Keyed on `primary_post_id` alone until 2026-09-24, which on the live
        # instance meant a drop never had a name or a hash at all — its images
        # are owned by its member messages, and the drop only claims them
        # through provenance. The identity route therefore never fired there;
        # the tests missed it because they attached images to the drop itself,
        # which discord_grouping never does.
        #
        # Counting by the drop also makes the span honest: five wips of one
        # piece posted as five messages and grouped into one drop are ONE post
        # as far as "how many posts carry this name" is concerned.
        owner = Post.__table__.alias("owner")
        rows = await self.session.execute(
            select(
                func.coalesce(owner.c.absorbed_by_post_id, ImageRecord.primary_post_id),
                ImageRecord.path, ImageRecord.phash,
            )
            .select_from(ImageRecord)
            .join(owner, owner.c.id == ImageRecord.primary_post_id)
            .where(
                ImageRecord.artist_id == artist_id,
                ImageRecord.primary_post_id.is_not(None),
            )
        )
        for post_id, path, phash in rows:
            paths_by_post.setdefault(post_id, []).append(path)
            bits = hash_bits(phash)
            if bits is not None:
                hashes_by_post.setdefault(post_id, []).append(bits)
        text_by_post: dict[int, str] = {}
        rows = await self.session.execute(
            select(Post.id, Post.post_title, Post.description).where(
                Post.artist_id == artist_id
            )
        )
        for post_id, title, description in rows:
            text_by_post[post_id] = "\n".join(
                part for part in (title, html_to_plain(description) or "") if part
            )
        corpus = _Corpus(
            tokens_by_post={
                pid: {t for path in paths for t in working_name_tokens(path)}
                for pid, paths in paths_by_post.items()
            },
            # Both counts take POSTS, which is why they are built from these
            # groupings rather than from flat lists — see post_naming.
            token_posts=token_frequencies(paths_by_post.values()),
            text_by_post=text_by_post,
            marker_posts=marker_frequencies(text_by_post.values()),
            hashes_by_post=hashes_by_post,
            # An image the creator puts on many posts — a banner, a watermark
            # plate, a recurring title card — is a habit exactly as a character
            # name is, and gets gated the same way. Counted on the EXACT hash,
            # which is what a reused file produces.
            hash_posts=Counter(
                h for hs in hashes_by_post.values() for h in set(hs)
            ),
        )
        self._corpora[artist_id] = corpus
        return corpus
    async def _decided(self, announcement_id: int) -> dict[int, PostAssociation]:
        """Pairs already recorded for this announcement, keyed by payload.
        Linked and dismissed pairs are never touched again: re-proposing a pair
        the operator has already rejected on every subsequent scan is the
        single behaviour that makes a review queue get ignored.
        A PENDING pair is different — nobody has decided it — so the caller
        re-scores it. Otherwise a pair queued by an older, weaker matcher sits
        in the queue forever even once the evidence is conclusive, which is the
        chore the operator asked FC not to hand them.
        """
        rows = (await self.session.execute(
-            select(PostAssociation.payload_post_id)
+            select(PostAssociation)
            .where(PostAssociation.announcement_post_id == announcement_id)
        )).scalars().all()
-        return set(rows)
+        return {a.payload_post_id: a for a in rows}
    async def _candidate_groups(
        self, announcement: Post, *, window: timedelta,
-    ) -> list[Post]:
+    ) -> list[tuple[Post, datetime]]:
-        """Synthetic Discord groupings by the SAME artist, inside the window.
+        """Synthetic Discord groupings by the SAME artist with a message inside
        the window — each with the time of its message CLOSEST to the teaser.
        Same-artist is the identity signal and it is free (see the module
        docstring on E4). It is also a hard filter rather than a scored one:
        two different creators posting minutes apart is a coincidence, not
        evidence, and letting it score at all would mean a busy hour across the
        library could out-vote everything else.
        Measured on the MESSAGES, not the drop's own date. A drop is dated by
        its first message, and since #4390 merges a creator's trickle into one
        drop, that can be days before the release the teaser announces —
        "Very early Marin" on Sep 7, `MarinaraSauce_base` on Sep 11. Matching
        on the drop's date would put every merged trickle outside the window.
        """
        at = _post_time(announcement)
-        sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
+        member = Post.__table__.alias("member")
-        return (await self.session.execute(
+        member_at = func.coalesce(member.c.post_date, member.c.downloaded_at)
-            select(Post)
+        rows = list((await self.session.execute(
            select(member.c.absorbed_by_post_id, member_at)
            .where(
                member.c.artist_id == announcement.artist_id,
                member.c.absorbed_by_post_id.is_not(None),
                member_at >= at - window,
                member_at <= at + window,
            )
        )).all())
        # The drop's own date counts too — the first message's, so it adds
        # nothing for a real drop, but it keeps a drop with no member rows
        # (hand-built, or one whose messages were removed) matchable.
        own_at = func.coalesce(Post.post_date, Post.downloaded_at)
        rows += (await self.session.execute(
            select(Post.id, own_at).where(
                Post.artist_id == announcement.artist_id,
                Post.synthesized_by == DROP_GROUPER,
                own_at >= at - window,
                own_at <= at + window,
            )
        )).all()
        closest: dict[int, datetime] = {}
        for group_id, when in rows:
            if group_id not in closest or abs(when - at) < abs(closest[group_id] - at):
                closest[group_id] = when
        if not closest:
            return []
        groups = (await self.session.execute(
            select(Post).where(
                Post.id.in_(list(closest)),
                Post.synthesized_by == DROP_GROUPER,
                Post.id != announcement.id,
                sort_key >= at - window,
                sort_key <= at + window,
            )
            .order_by(sort_key)
            .limit(MAX_CANDIDATES)
        )).scalars().all()
        ranked = sorted(groups, key=lambda g: (abs(closest[g.id] - at), g.id))
        return [(g, closest[g.id]) for g in ranked[:MAX_CANDIDATES]]
    async def _claimed(self, announcement_id: int, payload_id: int) -> bool:
        """Is either end of this pair already spoken for by an accepted link?
        An auto-link is FC asserting something the operator never saw, so it
        only happens where there is nothing to contradict. A drop already
        linked to a different announcement is exactly such a contradiction, and
        resolving it is a judgement about which one is right — which is the
        operator's, not FC's.
        """
        return (await self.session.execute(
            select(PostAssociation.id).where(
                PostAssociation.status == "linked",
                or_(
                    PostAssociation.payload_post_id == payload_id,
                    PostAssociation.announcement_post_id == announcement_id,
                ),
            ).limit(1)
        )).scalar() is not None
    async def match_post(
        self, announcement_id: int, *, threshold: float, window_hours: float,
-    ) -> int:
+        auto_link: bool = False,
-        """Score one announcement against nearby groupings. Returns proposals made."""
+    ) -> tuple[int, int]:
        """Score one announcement against nearby groupings.
        Returns `(proposed, linked)` — how many pairs were written, and how
        many of those were linked outright rather than queued.
        """
        announcement = await self.session.get(Post, announcement_id)
        if announcement is None or announcement.synthesized_by is not None:
            # A synthetic post cannot announce anything — FC wrote it.
-            return 0
+            return 0, 0
        window = timedelta(hours=window_hours)
        declared = declared_signal(announcement.description)
        already = await self._decided(announcement_id)
        corpus = await self._corpus(announcement.artist_id)
        here = corpus.tokens_by_post.get(announcement.id, set())
        here_text = corpus.text_by_post.get(announcement.id, "")
        here_hashes = corpus.hashes_by_post.get(announcement.id, [])
        made = 0
-        for group in await self._candidate_groups(announcement, window=window):
+        scored: list[tuple[Post, float, dict, float]] = []
-            if group.id in already:
+        for group, group_at in await self._candidate_groups(announcement, window=window):
            prior = already.get(group.id)
            if prior is not None and prior.status != "pending":
                continue
-            signals = {
+            named, token = shared_identity(
                here,
                corpus.tokens_by_post.get(group.id, set()),
                corpus.token_posts,
            )
            # The two identity signals answer the same question by different
            # means, so the stronger one stands rather than them being summed:
            # a name and a shared image both say "the same piece", and saying
            # it twice is not more true.
            copied = shared_image(
                here_hashes,
                corpus.hashes_by_post.get(group.id, []),
                corpus.hash_posts,
            )
            identity = max(named, copied)
            circumstantial = {
                "proximity": proximity_signal(
-                    _post_time(group) - _post_time(announcement), window,
+                    group_at - _post_time(announcement), window,
                ),
                "declared": declared,
                "marker": marker_overlap(
                    here_text,
                    corpus.text_by_post.get(group.id, ""),
                    corpus.marker_posts,
                ),
            }
-            score = weighted_score(signals)
+            score = weighted_score(circumstantial)
            # THE TWO ROUTES, and why identity is not simply a fourth weight.
            #
            # Circumstance and identity answer different questions. Proximity
            # and a declaration say two things happened near each other and
            # that one of them mentioned Discord; a working name the creator
            # uses on these two posts and nowhere else says they are the same
            # piece. Averaging those makes the threshold uninterpretable, and
            # it costs both: adding identity as a weight dilutes the others
            # enough that measured teaser/drop pairs an hour apart stop
            # proposing, while capping identity's contribution at its weight
            # means the strongest evidence available can never carry a pair on
            # its own.
            #
            # So identity may override, never dilute. Below the floor it is
            # recorded for the operator to read and moves nothing — which is
            # the conservative direction, since a wrong link asserts that two
            # different pieces are one.
            if identity >= IDENTITY_FLOOR:
                score = max(score, identity)
            if score < threshold:
                continue
            signals = {**circumstantial, "identity": identity}
            if copied:
                signals["identity_image"] = copied
            if token and named >= copied:
                # Carried so the queue can say WHY. A review queue that cannot
                # explain itself is one the operator learns to click through.
                signals["identity_token"] = token
            scored.append((group, score, signals, identity))
        # Who, if anyone, FC links without asking.
        #
        # EXACTLY ONE candidate may be conclusive. Two drops sharing a name
        # with one teaser at full strength is not a tie to be broken by score —
        # it means the name identifies something other than what FC thinks it
        # does, and the right response is to queue both and say nothing.
        auto_id = None
        if auto_link:
            conclusive = [c for c in scored if c[3] >= AUTO_LINK_FLOOR]
            if len(conclusive) == 1 and not await self._claimed(
                announcement.id, conclusive[0][0].id
            ):
                auto_id = conclusive[0][0].id
        linked = 0
        for group, score, signals, _identity in scored:
            status = "linked" if group.id == auto_id else "pending"
            if status == "linked":
                linked += 1
            prior = already.get(group.id)
            if prior is not None:
                # Re-scored in place: a pending pair keeps its row (and id),
                # and only an upgrade to linked counts as news.
                prior.score = score
                prior.signals = signals
                if status == "linked":
                    prior.status = "linked"
                    prior.linked_by = "fc"
                continue
            self.session.add(PostAssociation(
                announcement_post_id=announcement.id,
                payload_post_id=group.id,
                score=score,
                signals=signals,
-                status="pending",
+                status=status,
                linked_by="fc" if status == "linked" else None,
            ))
            made += 1
-        return made
+        return made, linked
    async def list_pending(self) -> list[dict]:
        rows = (await self.session.execute(
@@ -232,14 +593,19 @@ class PostAssociationService:
        if a is None:
            return None
        a.status = "linked"
        a.linked_by = "operator"
        return {"id": a.id, "status": a.status}
    async def dismiss(self, association_id: int) -> dict | None:
        a = await self.session.get(PostAssociation, association_id)
        if a is None:
            return None
-        # Kept, not deleted — the row is what remembers the rejection.
+        # Kept, not deleted — the row is what remembers the rejection. It is
        # also the undo for a link FC made itself (#4402): the unified card
        # dismisses the pair, and the dismissed row stops the next sweep from
        # linking it straight back.
        a.status = "dismissed"
        a.linked_by = None
        return {"id": a.id, "status": a.status}
    async def linked_for(self, post_ids: list[int]) -> dict[int, list[dict]]:
@@ -273,36 +639,124 @@ class PostAssociationService:
        return out
-async def rescan(session: AsyncSession, *, now: datetime | None = None) -> dict:
+async def rescan(
-    """Score every recent non-synthetic post against nearby groupings."""
+    session: AsyncSession, *, now: datetime | None = None, full: bool = False,
 ) -> dict:
    """Score recent non-synthetic posts against nearby groupings.
    `full=True` scores EVERY post by an artist who has Discord drops at all —
    the manual button's job, for history that predates the feature or that a
    trickle merge (#4390) has just rearranged. It used to share the sweep's
    48-hour horizon, so the button described as "a first run over a library
    that predates the feature" could not reach that library.
    """
    settings = await ImportSettings.load(session)
    if not settings.discord_link_enabled:
-        return {"enabled": False, "scanned": 0, "proposed": 0}
+        return {"enabled": False, "scanned": 0, "proposed": 0, "linked": 0}
    now = now or datetime.now(UTC)
    window_hours = float(settings.discord_link_window_hours)
    window = timedelta(hours=window_hours)
    # Only look at announcements that could still have a partner in range —
    # a full-library rescan is the manual button's job, not the sweep's.
    horizon = now - timedelta(hours=window_hours * 2)
    sort_key = func.coalesce(Post.post_date, Post.downloaded_at)
-    ids = (await session.execute(
+    if full:
        with_drops = select(Post.artist_id).where(
            Post.synthesized_by == DROP_GROUPER
        ).distinct()
        ids = set((await session.execute(
            select(Post.id).where(
                Post.synthesized_by.is_(None),
                Post.absorbed_by_post_id.is_(None),
                Post.artist_id.in_(with_drops),
            )
        )).scalars().all())
        return await _score(session, settings, ids, window_hours)
    ids = set((await session.execute(
        select(Post.id).where(
            Post.synthesized_by.is_(None),
            Post.absorbed_by_post_id.is_(None),
            sort_key >= horizon,
        )
-    )).scalars().all()
+    )).scalars().all())
    # ...and announcements sitting next to a drop FC has only JUST authored.
    #
    # A drop's `post_date` is backdated to its first message, but FC cannot
    # write the drop until the message has an embedding and the hourly grouper
    # has run — so a drop created this minute can land weeks back in the feed.
    # Its neighbours were last swept before it existed, and a sweep keyed only
    # on how recent the ANNOUNCEMENT is will never look at them again.
    #
    # That is #4392's third cause, and it is the one that left a measured 0.800
    # pair with an empty review queue on the live instance. The other two were
    # about scoring; this one meant nothing was scored at all.
    #
    # The times are the drops' MESSAGES, not the drops' own dates: a drop that
    # grew today by a trickle merge (#4390) is dated by its first stage, days
    # earlier, while the teaser sits beside the message that just joined.
    recent = (
        select(Post.id).where(
            Post.synthesized_by == DROP_GROUPER,
            func.coalesce(Post.last_grew_at, Post.downloaded_at) >= horizon,
        )
        .order_by(func.coalesce(Post.last_grew_at, Post.downloaded_at).desc())
        .limit(MAX_RECENT_DROPS)
    )
    member = Post.__table__.alias("member")
    drop_times = set((await session.execute(
        select(func.coalesce(member.c.post_date, member.c.downloaded_at))
        .where(member.c.absorbed_by_post_id.in_(recent))
    )).scalars().all())
    # Plus each drop's own date — its first message's, so a duplicate for a
    # real drop, but what a drop with no member rows is placed by.
    drop_times |= set((await session.execute(
        select(sort_key).where(Post.id.in_(recent))
    )).scalars().all())
    # The interval arithmetic is done in Python rather than SQL: a handful of
    # literal ranges is portable, and `now - INTERVAL` is not.
    ranges = [
        and_(sort_key >= at - window, sort_key <= at + window)
        for at in drop_times
        if at is not None
    ]
    if ranges:
        ids |= set((await session.execute(
            select(Post.id).where(
                Post.synthesized_by.is_(None),
                Post.absorbed_by_post_id.is_(None),
                or_(*ranges),
            )
        )).scalars().all())
    return await _score(session, settings, ids, window_hours)
 async def _score(
    session: AsyncSession, settings: ImportSettings, ids: set[int], window_hours: float,
 ) -> dict:
    svc = PostAssociationService(session)
    proposed = 0
-    for pid in ids:
+    linked = 0
-        proposed += await svc.match_post(
+    # Sorted because `ids` is now a union of two queries: set iteration order
    # is arbitrary, and a sweep that visits posts in a different order each
    # run is one whose failures cannot be reproduced.
    for pid in sorted(ids):
        made, auto = await svc.match_post(
            pid,
            threshold=float(settings.discord_link_threshold),
            window_hours=window_hours,
            auto_link=bool(settings.discord_link_auto),
        )
        proposed += made
        linked += auto
    log.info(
-        "discord announcement matcher: scanned %d post(s), proposed %d pair(s)",
+        "discord announcement matcher: scanned %d post(s), proposed %d pair(s), "
-        len(ids), proposed,
+        "linked %d outright",
        len(ids), proposed, linked,
    )
-    return {"enabled": True, "scanned": len(ids), "proposed": proposed}
+    return {
        "enabled": True, "scanned": len(ids), "proposed": proposed,
        "linked": linked,
    }
@@ -19,6 +19,7 @@ from ..models import (
    ExternalLink,
    ImageProvenance,
    ImageRecord,
    ImportSettings,
    Post,
    PostAttachment,
    Source,
@@ -118,6 +119,15 @@ class PostFeedService:
        # from. `around` and `get_post` deliberately do NOT apply this: reaching
        # a member by id is how you inspect a grouping.
        stmt = stmt.where(Post.absorbed_by_post_id.is_(None))
        # A linked Discord drop is shown ON its teaser's card by reference
        # (#4402), so its own card sitting beside that teaser is the same
        # release twice. Only then is it left out — an older drop keeps its
        # place, because a reference does not take anything out of history.
        fold_hours = await self._fold_hours()
        if fold_hours > 0:
            from .post_unification import fold_clause
            stmt = stmt.where(fold_clause(fold_hours))
        if artist_id is not None:
            stmt = stmt.where(Post.artist_id == artist_id)
        if platform is not None:
@@ -169,9 +179,12 @@ class PostFeedService:
        thumbs_map = await self._thumbnails_for(post_ids)
        atts_map = await self._attachments_for(post_ids)
        links_map = await self._links_for(post_ids)
        unified_map = await self._unified_for([p for p, _, _ in rows])
        items = [
-            self._to_dict(post, artist, source, thumbs_map, atts_map, links_map)
+            self._to_dict(
                post, artist, source, thumbs_map, atts_map, links_map, unified_map,
            )
            for post, artist, source in rows
        ]
        return {"items": items, "next_cursor": next_cursor}
@@ -213,6 +226,7 @@ class PostFeedService:
        anchor_item = self._to_dict(
            anchor_post, anchor_artist, anchor_source, thumbs_map, atts_map,
            await self._links_for([anchor_post.id]),
            await self._unified_for([anchor_post]),
        )
        return {
            "items": newer["items"] + [anchor_item] + older["items"],
@@ -239,6 +253,7 @@ class PostFeedService:
        item = self._to_dict(
            post, artist, source, thumbs_map, atts_map,
            await self._links_for([post.id]),
            await self._unified_for([post]),
        )
        item["description_full"] = html_to_plain(post.description)
        # Full (uncapped) translated description for the detail view (#143).
@@ -385,9 +400,27 @@ class PostFeedService:
        return await PostAssociationService(self.session).linked_for(post_ids)
    async def _unified_for(self, posts: list[Post]) -> dict[int, dict]:
        """The reference set each teaser's card shows (#4402). Local import for
        the same reason as `_links_for`: it reaches the association service."""
        from .post_unification import PostUnificationService
        return await PostUnificationService(self.session).unified_for(posts)
    async def _fold_hours(self) -> float:
        """`discord_link_fold_hours`, read without assuming the row exists.
        The feed is the one surface that must not fail on a settings row the
        caller never needed, so a missing row folds nothing rather than
        raising — which is also exactly how the feed behaved before this.
        """
        settings = await self.session.get(ImportSettings, 1)
        return float(settings.discord_link_fold_hours) if settings is not None else 0.0
    def _to_dict(
        self, post: Post, artist: Artist, source: Source | None,
        thumbs_map: dict, atts_map: dict, links_map: dict | None = None,
        unified_map: dict | None = None,
    ) -> dict:
        plain_full = html_to_plain(post.description) if post.description else None
        if plain_full is None:
@@ -436,6 +469,11 @@ class PostFeedService:
            # post is the drop). Always a list so the UI never branches on
            # absence.
            "associations": (links_map or {}).get(post.id, []),
            # #4402. On a teaser with a linked drop: what the card shows BY
            # REFERENCE — the drop's images, the piece's older variants, and
            # the text of each — plus who made each link, so one FC made by
            # itself can say so and offer the undo. None on every other post.
            "unified": (unified_map or {}).get(post.id),
            # Non-null on a chat message a synthetic post absorbed. The feed
            # filters these out, but `around`/`get_post` still reach them, and
            # the UI uses this to explain why a post it linked to is not in the
@@ -0,0 +1,426 @@
 """The creator's own working name for a piece, recovered from a filename.
 Milestone 388. Pure functions, no DB and no ML — the whole point is that this
 signal is free.
 ## What this is for
 Two of the operator's artists post a censored or cropped teaser on Patreon and
 the real release in their Discord. Matching those by IMAGE is the pair a
 whole-image comparison handles worst: the teaser is a crop with a censor bar,
 which is exactly the local edit that moves a perceptual hash and blurs a
 semantic embedding.
 But the creator names both exports after the same internal working title, and
 that name survives into both platforms untouched. Measured on the live instance
 2026-09-24, artist 8:
    01_((0-k           <->  0-k_base          (1.3h apart)
    01_680LC           <->  680LC_Border      (21.0h apart)
    01_cnni18x         <->  cnni18x           (21.5h apart)
 Three pairs, no false positives, and **two of them are 21 hours apart** — far
 enough that time proximity scores them ~0.10 and could never propose them. The
 naming signal is orthogonal to the timing one: each finds pairs the other
 cannot, which is why both are kept rather than one being tuned to cover both.
 ## Why a filename and not a perceptual hash
 A shared working-name token is IDENTITY evidence — `680lc` appearing on both
 platforms is not a coincidence. Proximity is CIRCUMSTANTIAL: it says two things
 happened near each other, never that they are the same thing. The distinction
 drives the weighting in `post_association_service`, and it is why a rare enough
 token is allowed to carry a proposal on its own while no amount of circumstance
 is.
 ## The one false-positive class found, and why the fix is shaped this way
 A first pass matched `01_Screenshot 2026-08-13 000004` to
 `Screenshot_2026-08-13_032144` on the token `2026-08-13`, twice.
 A screenshot filename is a camera artifact. It carries no working name, and the
 date inside it collides across platforms on the same day BY CONSTRUCTION — the
 teaser and the release are posted the same day, so their screenshot names
 always share a date token. That is a signal that fires precisely when it is
 least informative.
 So a filename with no working name contributes NOTHING, rather than the
 plausible-looking date match it could be squeezed for. Re-run with that rule:
 the same three true pairs, zero false. Half of this creator's recent teasers
 are screenshots, and those pairs are simply out of this signal's reach — which
 is where crop-to-source matching earns its cost, and nowhere else.
 """
 from __future__ import annotations
 import re
 from collections import Counter
 from collections.abc import Iterable
 from pathlib import PurePosixPath
 # A screenshot name, on either platform. Patreon's importer writes
 # `01_Screenshot 2026-08-13 000004`; gallery-dl's Discord naming writes
 # `Screenshot_2026-09-22_003651`. Matched after the index/message prefixes are
 # stripped, so both shapes reach this as a bare `Screenshot ...`.
 # NOT `\b` after "shot": `\b` needs a word/non-word transition and `_` is a
 # WORD character, so `Screenshot_2026-08-13_032144` — gallery-dl's Discord
 # spelling — sailed straight past the guard while the space-separated Patreon
 # spelling was caught. Found by running this against the live library rather
 # than by reading it. Assert the next character is not a letter instead.
 _SCREENSHOT = re.compile(r"^(?:screen[ _-]?shot(?![a-z])|\u30b9\u30af\u30ea\u30fc\u30f3\u30b7\u30e7\u30c3\u30c8|\u30b9\u30af\u30b7\u30e7)", re.I)
 # The importer's per-post media index: `01_`, `02_`. Not part of any name.
 _MEDIA_INDEX = re.compile(r"^\d{1,3}_")
 # gallery-dl's Discord filename pattern (#3999):
 # `{date:%Y%m%d}_{message_id}_{num:>02}_{filename}`.
 _DISCORD_PREFIX = re.compile(r"^\d{8}_\d{6,}_\d{1,3}_")
 # The legacy era (#4002): images sit FLAT at the artist root as
 # `<post id>_media_<media id>_<name>`. Stripping it is not cosmetic — the
 # SCREENSHOT guard below matches from the start of the stem, so while this
 # prefix was left on, a legacy screenshot never looked like one. Measured on
 # tamadaheijun: `109078417_media_334848471_Screenshot 2025-07-27 182450ab`
 # sailed through and contributed `2025-07-27`, which is precisely the
 # same-day date collision this module was built to refuse.
 _LEGACY_PREFIX = re.compile(r"^\d+_media_\d+_")
 # The importer's content-hash suffix, `__<10 hex>`, sometimes doubled on files
 # that went through an older import era.
 _HASH_SUFFIX = re.compile(r"(?:__[0-9a-f]{10})+$")
 # Generic export decorations. Stripped as SUFFIXES so the stem survives:
 # `cnni18x_wip3` and `cnni18x` must yield the same token, or a work-in-progress
 # would never match the piece it became.
 _DECORATION = re.compile(
    r"(?:[_-]?(?:wip|base|final|alt|alts|edit|edits|border|clean|raw|hd|full|"
    r"censored|uncensored|nsfw|sfw|ver|v)\d*)+$",
    re.I,
 )
 # Tokens that carry no identity even when they survive the rules above.
 _STOPWORDS = frozenset({
    "img", "image", "untitled", "new", "test", "page", "final", "copy",
    "post", "media", "file", "avatar", "cover", "banner", "icon", "splash",
    # Each of these was MEASURED carrying a false match in the 3..6 frequency
    # band, where the rarity gate still admits a token: `capture` across six
    # unrelated knuxy posts, `the` and `patreon` out of legacy title-derived
    # names, `main` out of `Anya Main CST`, `timeline` and `gif` out of
    # tamadaheijun's exports.
    "the", "gif", "main", "patreon", "capture", "timeline", "screenshot",
    # Literally the string "None": issue #3999's Discord naming rendered
    # `{user[name]}` as it for ~1,600 files, so it is the single most common
    # "name" in the library and identifies nothing.
    "none",
    # gallery-dl's fallback when a Discord attachment has no filename of its
    # own. Measured on artist 8: four unrelated images across 1,974 days, and
    # the one false family the leading-name rule admitted inside 60 days.
    "image0",
 })
 # A bare year: still needed for the TEXT signal, where words and numbers are
 # tokenised separately.
 _YEAR = re.compile(r"^(?:19|20)\d{2}$")
 # An identity token must contain a LETTER. This replaces separate "all digits"
 # and "bare year" rules with the property behind both, and it is the rule that
 # holds once hyphens are kept inside tokens for `0-k`'s sake: without it
 # `2025-07-27` and `3-0002` survive as single tokens, and both were measured
 # linking unrelated posts — the second across three of them, out of
 # tamadaheijun's `timeline 3-0002` exports.
 #
 # `0-k`, `680lc`, `cnni18x` and `p59` all keep a letter and are unaffected.
 _HAS_LETTER = re.compile(r"[A-Za-z]")
 MIN_TOKEN_LEN = 3
 # A token appearing in more than this many of ONE ARTIST's POSTS is a habit,
 # not an identity — a character name, a series tag, a recurring export preset.
 #
 # POSTS, not files, and the difference is not bookkeeping. Counting files
 # punishes a piece for having many exports, which is the one thing a working
 # name is GUARANTEED to do. Measured across the operator's four dual-platform
 # artists: knuxy's comic pages carry `p217` on four files spread over exactly
 # two posts — the Patreon post and the Discord drop — and file-counting scored
 # every one of ~200 such tokens at half strength for it. Counting posts scores
 # them 1.00 while still catching the real habits, which span many posts:
 # tamadaheijun's `comic2` spans 8, conto's `seth2` 5, `maid` 4.
 #
 # Six rather than two, although two posts IS the shape of a teaser and its
 # drop, because a creator legitimately revisits one working name: a wip post,
 # then an alt, then the release. Measured on artist 8, `cnni18x` and `680lc`
 # each span four posts and both are genuine. IDENTITY_FLOOR below is what
 # decides how much span a link may carry on its own.
 MAX_TOKEN_POSTS = 6
 # A shared name at or above this strength is enough to propose a link with NO
 # corroboration — it is identity evidence, and the whole reason this module
 # exists is that identity survives where circumstance does not.
 #
 # 0.75 is a token spanning three posts or fewer. Measured on artist 8: of the
 # 15 same-artist pairs that share a name, 13 clear this bar, including the
 # operator's own example (`0-k`, three posts, 1.3h apart). The two that do not
 # — `680lc` and `cnni18x`, four posts each, 21h apart — are real pairs this
 # signal will not carry alone; they are the measured cost of not admitting the
 # four-post band, where conto's `illustration9` and `maid` also sit.
 IDENTITY_FLOOR = 0.75
 # A LEADING name spanning this many of ONE ARTIST's posts is a habit — a
 # character the creator returns to — not one piece's trickle. Used wherever a
 # name gathers a FAMILY: the teaser card's variants (#4401) and the Discord
 # grouper's trickle merge (#4390), so the two cannot disagree about what a
 # family is.
 #
 # Its own value rather than MAX_TOKEN_POSTS (6), which is calibrated for
 # PAIRING two posts and measured too tight for a family. On artist 8,
 # `tentacooler` spans 6 posts over 7 days and `0-k1` 6 posts over 10: both real
 # families, both gated out at 6. At 8, `anya` (7) passes the cap — and has no
 # pair inside any family window, which is what the window is for.
 FAMILY_MAX_POSTS = 8
 def _strip_prefixes(stem: str) -> str:
    """Remove the framing each platform's importer adds around the real name."""
    stem = _DISCORD_PREFIX.sub("", stem)
    stem = _LEGACY_PREFIX.sub("", stem)
    stem = _MEDIA_INDEX.sub("", stem)
    return _HASH_SUFFIX.sub("", stem)
 def _ordered_tokens(path: str) -> list[str]:
    """The identity-bearing tokens of one filename, in the order written.
    The one tokenizer both public readings share, so the set of names and the
    leading name cannot disagree about what counts as a name.
    """
    stem = _strip_prefixes(PurePosixPath(path).stem)
    if _SCREENSHOT.match(stem.strip()):
        return []
    out: list[str] = []
    # Hyphens are kept INSIDE tokens — `0-k` is a real working name on the live
    # instance, and splitting on hyphen would reduce it to a single character
    # and then discard it for being too short.
    for raw in re.split(r"[^0-9A-Za-z-]+", stem.lower()):
        tok = _DECORATION.sub("", raw).strip("-")
        if len(tok) < MIN_TOKEN_LEN:
            continue
        if tok in _STOPWORDS or not _HAS_LETTER.search(tok):
            continue
        if tok not in out:
            out.append(tok)
    return out
 def working_name_tokens(path: str) -> set[str]:
    """The identity-bearing tokens in one image's filename.
    Returns an EMPTY set for a name that carries no working title — a
    screenshot, a bare number, a stopword. Empty means "no evidence", which the
    caller must treat as silence rather than as a weak match; see the module
    docstring for the false positive that rule exists for.
    """
    return set(_ordered_tokens(path))
 def leading_name(path: str) -> str | None:
    """The FIRST identity token of a filename — the piece, not its decoration.
    Creators lead with what the piece is and trail with what this export of it
    is: `Year_20k_wip1`, `not_sombra_21-cumpeen`, `Tentacooler_c_ins`. Content
    words sit at the tail, and they span too FEW posts for any frequency cap
    to catch — `nude`, `cum` and `top` are on three of artist 8's posts each.
    Position is what separates them from a name.
    Measured on artist 8, Discord messages 2-60 days apart sharing a gated
    token: 121 pairs. The 106 sharing the leading name all read as one piece's
    trickle; of the 15 sharing only a trailing word, 14 are sibling pieces
    (`Bea_Machamp_Shiny_*` / `Bea_Machoke_Shiny_*`) and one is a plain
    collision (`Undyne_insert_bottom_only-C` / `Lichgalclc_Lingerie_Bottom_21`).
    None when the name carries no identity at all — see `working_name_tokens`.
    """
    tokens = _ordered_tokens(path)
    return tokens[0] if tokens else None
 def token_frequencies(posts: Iterable[Iterable[str]]) -> Counter[str]:
    """How many of ONE ARTIST's POSTS each working-name token appears in.
    Takes posts — each an iterable of that post's image paths — rather than a
    flat list of paths, because the unit of the count is the post. See
    MAX_TOKEN_POSTS for what that buys; the short version is that a piece with
    six exports in one post has used its name once.
    Scoped to the ARTIST, not the library: a working name belongs to the person
    who chose it, and the same string can be one creator's piece and another's
    boilerplate. Built once per artist per sweep, not per candidate pair.
    """
    counts: Counter[str] = Counter()
    for paths in posts:
        counts.update({t for path in paths for t in working_name_tokens(path)})
    return counts
 def rarity(freq: int, max_frequency: int) -> float:
    """Rarity of one token within an artist's own corpus, in [0, 1].
    Shared by EVERY rarity-gated signal deliberately. They carried one each
    until 2026-09-24, and the copies drifted: the filename signal grew a
    frequency gate and the marker signal never did, so a creator's habitual
    emoji scored the same 1.00 as a marker they had used twice. One
    definition cannot drift from itself.
    Full strength at 2 rather than 1: a genuine match means the token is on
    at least two things, so demanding uniqueness would reject every real
    pair. Decays to zero AT the cap rather than falling off it, so nothing
    sits on a cliff edge.
    """
    if freq <= 2:
        return 1.0
    if freq >= max_frequency:
        return 0.0
    return (max_frequency - freq) / (max_frequency - 2)
 def shared_identity(
    left: Iterable[str],
    right: Iterable[str],
    frequencies: Counter[str],
    *,
    max_frequency: int = MAX_TOKEN_POSTS,
 ) -> tuple[float, str | None]:
    """Strength in [0, 1] that two sets of filenames name the SAME piece.
    Returns `(strength, token)` — the token is carried back so the proposal can
    say WHY it was made. A review queue that cannot explain itself is one the
    operator learns to click through without reading.
    Strength is a function of the winning token's rarity within the artist's
    own library, not of how many tokens matched. One decisive token beats three
    vague ones, and a token appearing across forty of this artist's posts is a
    habit rather than an identity however exactly it matches.
    `frequencies` must be the POST counts from `token_frequencies`.
    """
    shared = {t for t in set(left) & set(right) if frequencies.get(t, 0) <= max_frequency}
    if not shared:
        return 0.0, None
    # The rarest shared token decides — one decisive token beats three vague
    # ones.
    token = min(shared, key=lambda t: (frequencies.get(t, 0), -len(t), t))
    strength = round(rarity(max(frequencies.get(token, 1), 1), max_frequency), 4)
    # A token sitting exactly ON the cap decays to zero, and naming it anyway
    # would hand the review queue a reason that carries no weight — "matched on
    # loislanetb2", with nothing behind it. Measured: that token is on 6 of this
    # artist's images. Report a token only when it is doing work.
    return (strength, token) if strength > 0 else (0.0, None)
 # --- the body/title signal ---------------------------------------------------
 #
 # The same idea applied to TEXT. The operator's example pair carries `🍈🍈` in
 # the Patreon title and `@everyone 🍈 🍈` in the Discord message — a marker the
 # creator uses to tie the two together, which no vocabulary list would predict.
 #
 # Rarity-gated, exactly as the filename signal is, and the gate is here because
 # the first pass did NOT have one. Measured on artist 8, 300 posts:
 #
 #     💦 11 posts (4%)   🫴 6   🌰 5   🍗 5   🫣 4
 #
 # 💦 is punctuation for this creator — about one post in twenty-five. Ungated it
 # scored a full 1.00 and was the DECIDING term in a proposal that proximity
 # alone (0.441) could not carry. A habitual marker riding along with proximity
 # is just proximity wearing a hat, which is the exact failure the matcher's
 # threshold sits above 0.55 to prevent. The operator's 🍈🍈 is the opposite
 # case: two posts, and they are the pair itself.
 _WORD = re.compile(r"[0-9A-Za-z]{3,}")
 # Anything outside the Basic Multilingual Plane's text ranges: emoji, symbols,
 # kaomoji parts. These are the tokens creators actually use as markers, and
 # they are rare enough in prose to be evidence on their own.
 # U+1F000-1FAFF is the emoji planes; U+2190-2BFF covers arrows, dingbats and
 # the miscellaneous-symbol blocks, which already contains U+2600-27BF.
 _SYMBOL = re.compile(r"[\U0001F000-\U0001FAFF\u2190-\u2BFF]")
 _COMMON_TEXT = frozenset({
    "the", "and", "for", "you", "new", "out", "now", "this", "that", "with",
    "everyone", "here", "post", "all", "art", "one", "get", "has", "are",
 })
 # A marker in more than this many of ONE ARTIST's posts is a signature, not a
 # tie-back. A marker tying an announcement to its drop lands on two posts —
 # the two.
 #
 # Tighter than MAX_TOKEN_POSTS. Both count posts, so the numbers are directly
 # comparable and the gap between them is the claim being made: a working name
 # is the creator's private label for one piece and may honestly recur as they
 # revisit it, while a marker is public decoration and stops being evidence the
 # moment it is reused. Measured on artist 8: 💦 spans 13 posts, the word
 # "like" 43, and 🌗 — a real tie-back — exactly 2.
 MAX_MARKER_POSTS = 4
 def text_markers(text: str | None) -> set[str]:
    """Distinctive tokens in a post body or title: symbols, and rare-ish words.
    Symbols count individually rather than as a run, so `🍈🍈` and `🍈 🍈` —
    which is how the same marker appears on the two platforms — reduce to the
    same token. Spacing is a platform's rendering, not the creator's intent.
    """
    if not text:
        return set()
    out = {m.group(0) for m in _SYMBOL.finditer(text)}
    out |= {
        w.lower() for w in _WORD.findall(text)
        if w.lower() not in _COMMON_TEXT and not _YEAR.match(w)
    }
    return out
 def marker_frequencies(texts: Iterable[str | None]) -> Counter[str]:
    """How many of ONE ARTIST's posts each marker appears in.
    Per POST, not per occurrence: a creator who repeats an emoji six times in
    one body has used it once as far as identity goes. Scoped to the artist for
    the same reason `token_frequencies` is — a marker is a personal habit, and
    one creator's signature is another's whole vocabulary.
    """
    counts: Counter[str] = Counter()
    for t in texts:
        counts.update(text_markers(t))
    return counts
 def marker_overlap(
    left: str | None,
    right: str | None,
    frequencies: Counter[str],
    *,
    max_frequency: int = MAX_MARKER_POSTS,
 ) -> float:
    """Strength in [0, 1] that two texts share a DELIBERATE marker.
    `frequencies` is required rather than defaulted to "no gate". An ungated
    call is the bug this signature exists to make impossible to write by
    accident, and a default would have kept it one keyword away.
    Symbols weigh full and words a quarter, because prose shares words by
    accident: a creator who writes "commission" in both posts on a Tuesday has
    told us nothing that the timestamps did not already say.
    There is no divisor. An earlier pass halved the total so that a long body
    could not out-vote a short one, which the rarity gate now does properly —
    and halving meant the operator's own 🍈🍈 pair, a marker on exactly two
    posts, could reach only 0.5. One marker the creator uses nowhere else is
    the whole signal, not half of it.
    """
    shared = text_markers(left) & text_markers(right)
    if not shared:
        return 0.0
    score = sum(
        (1.0 if _SYMBOL.match(t) else 0.25) * rarity(frequencies.get(t, 1), max_frequency)
        for t in shared
    )
    return round(min(1.0, score), 4)
@@ -0,0 +1,417 @@
 """The unified post card — a teaser shows what it points at (#4402, #4401).
 Milestone 388. A Patreon teaser is a POINTER: a cropped, censored fragment
 whose job is to say "the full set is in Discord". Until this module the card
 rendered the fragment and a text link, and the reader had to make the join FC
 had already made.
 Operator, 2026-09-24: *"the teaser from the patreon post doesn't show the items
 that it's supposed to reference so I'm trying to unify the teaser post with the
 content it's meant to draw attention to."*
 ## A reference, never an absorption
 `discord_grouping` folds chat messages into a synthetic post by transferring
 ownership (`absorbed_by_post_id`). That is the wrong primitive here, and the
 operator said so directly: *"the nested items on the unified post are a
 duplicate or reference of existing content. that's why they can show similar
 items and not erase or invalidate the way the discord items landed."*
 So nothing here writes. The Discord posts keep their own rows, dates and
 places in the feed; the teaser's card DISPLAYS them. That is also what makes
 reaching back for older variants safe at all: a wrong reference shows one
 extra thumbnail in one place, where a wrong regrouping would move content.
 ## What a teaser references
 1. The Discord drops a `linked` PostAssociation joins it to (#4392) — accepted
   by the operator, or linked by FC on a conclusive name match.
 2. The rest of that piece's VARIANT FAMILY (#4401): the wips, alts and censor
   passes a creator trickles out under one working name, days or weeks apart.
 Families are found by the creator's LEADING working name, not by any shared
 token and not by image similarity — see `post_naming.leading_name` for the
 measurement, and lesson #4400 for why a whole-image comparison between two
 works by one artist cannot separate "same piece" from "same artist".
 """
 from __future__ import annotations
 from collections import Counter
 from collections.abc import Iterable
 from dataclasses import dataclass
 from datetime import datetime, timedelta
 from sqlalchemy import and_, exists, extract, func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy.orm import aliased
 from ..models import (
    ImageProvenance,
    ImageRecord,
    ImportSettings,
    Post,
    PostAssociation,
    Source,
 )
 from ..utils.phash import hamming, hash_bits
 from ..utils.text import html_to_plain, truncate_at_word
 from .discord_grouping import PLATFORM as DISCORD
 from .gallery_service import thumbnail_url
 from .post_association_service import DUPLICATE_MAX_DISTANCE
 from .post_naming import FAMILY_MAX_POSTS, leading_name, rarity, token_frequencies
 # The text each referenced post contributes to the card, per post. The card
 # clamps it again; this keeps a long Discord thread from making the feed
 # payload the size of the thread.
 TEXT_LIMIT = 280
@dataclass(frozen=True)
 class Candidate:
    """One image as the family search sees it."""
    image_id: int
    post_id: int
    path: str
    phash: int | None
    at: datetime
 def family(
    seed: Iterable[Candidate],
    pool: Iterable[Candidate],
    name_posts: Counter[str],
    hash_posts: Counter[int],
    *,
    anchor: datetime,
    window: timedelta,
    max_posts: int = FAMILY_MAX_POSTS,
 ) -> list[Candidate]:
    """The images in `pool` that belong to the same piece as `seed`.
    A member shares a seed image's LEADING working name, or is a perceptual
    near-duplicate of one (the same file re-posted), and lies within `window`
    of `anchor` — the teaser's date. Oldest first, so the card reads as the
    trickle it was.
    ONE hop from the seed, never transitive. Every measured family is one hop
    from any of its members, because the members share the name; chaining is
    what lets a family drift from `Year_20k` to whatever `Year_20k_Base`'s
    other tokens happen to touch.
    Both identity routes are rarity-gated against `max_posts`, exactly as the
    matcher gates them. A leading name the creator uses across many posts is a
    character, and a hash on many posts is a banner.
    """
    seed = list(seed)
    names = {
        name for c in seed
        if (name := leading_name(c.path)) is not None
        and rarity(name_posts.get(name, 0), max_posts) > 0
    }
    hashes = [
        c.phash for c in seed
        if c.phash is not None and rarity(hash_posts.get(c.phash, 0), max_posts) > 0
    ]
    taken = {c.image_id for c in seed}
    out: list[Candidate] = []
    for c in pool:
        if c.image_id in taken or abs(c.at - anchor) > window:
            continue
        named = leading_name(c.path) in names
        copied = c.phash is not None and any(
            (d := hamming(c.phash, h)) is not None and d <= DUPLICATE_MAX_DISTANCE
            for h in hashes
        )
        if named or copied:
            out.append(c)
            taken.add(c.image_id)
    return sorted(out, key=lambda c: (c.at, c.image_id))
 def _when(post: Post) -> datetime:
    return post.post_date or post.downloaded_at
 def _text(post: Post) -> str | None:
    plain = html_to_plain(post.description) if post.description else None
    if not plain or not plain.strip():
        return None
    return truncate_at_word(plain.strip(), TEXT_LIMIT)[0]
@dataclass
 class _Artist:
    """Everything the family search needs about one artist, loaded once."""
    rows: dict[int, tuple]          # image_id -> (post_id, path, phash, sha, mime, thumb)
    posts: dict[int, Post]
    platform: dict[int, str | None]  # post_id -> platform
    name_posts: Counter[str]
    hash_posts: Counter[int]
 class PostUnificationService:
    def __init__(self, session: AsyncSession):
        self.session = session
        self._artists: dict[int, _Artist] = {}
    async def _artist(self, artist_id: int) -> _Artist:
        if artist_id in self._artists:
            return self._artists[artist_id]
        posts: dict[int, Post] = {}
        platform: dict[int, str | None] = {}
        for post, plat in (await self.session.execute(
            select(Post, Source.platform)
            .outerjoin(Source, Post.source_id == Source.id)
            .where(Post.artist_id == artist_id)
        )).all():
            posts[post.id] = post
            platform[post.id] = plat
        rows: dict[int, tuple] = {}
        paths_by_post: dict[int, list[str]] = {}
        hashes_by_post: dict[int, set[int]] = {}
        for img_id, post_id, path, phash, sha, mime, thumb in (await self.session.execute(
            select(
                ImageRecord.id, ImageRecord.primary_post_id, ImageRecord.path,
                ImageRecord.phash, ImageRecord.sha256, ImageRecord.mime,
                ImageRecord.thumbnail_path,
            ).where(
                ImageRecord.artist_id == artist_id,
                ImageRecord.primary_post_id.is_not(None),
            )
        )).all():
            bits = hash_bits(phash)
            rows[img_id] = (post_id, path, bits, sha, mime, thumb)
            paths_by_post.setdefault(post_id, []).append(path)
            if bits is not None:
                hashes_by_post.setdefault(post_id, set()).add(bits)
        # Counted over EVERY post the artist has, exactly as the matcher's
        # corpus counts them — a family is judged against the whole library,
        # not against the slice inside the window, or a character name would
        # look rare in any quiet month.
        found = _Artist(
            rows=rows,
            posts=posts,
            platform=platform,
            name_posts=token_frequencies(paths_by_post.values()),
            hash_posts=Counter(h for hs in hashes_by_post.values() for h in hs),
        )
        self._artists[artist_id] = found
        return found
    async def _drop_images(self, drop_ids: list[int]) -> dict[int, list[int]]:
        """drop post id -> its image ids, through provenance as the feed reads them.
        A synthetic drop owns no image outright: its images belong to the
        member messages, and `discord_grouping` gives the drop a provenance row
        for each. The primary_post_id arm keeps any image that has one and no
        row, the same union `PostFeedService._thumbnails_for` takes.
        """
        out: dict[int, list[int]] = {pid: [] for pid in drop_ids}
        if not drop_ids:
            return out
        links = (
            select(
                ImageProvenance.image_record_id.label("image_id"),
                ImageProvenance.post_id.label("post_id"),
            )
            .where(ImageProvenance.post_id.in_(drop_ids))
            .union(
                select(
                    ImageRecord.id.label("image_id"),
                    ImageRecord.primary_post_id.label("post_id"),
                ).where(ImageRecord.primary_post_id.in_(drop_ids))
            )
            .subquery()
        )
        for img_id, pid in (await self.session.execute(
            select(links.c.image_id, links.c.post_id).order_by(links.c.image_id)
        )).all():
            out[pid].append(img_id)
        return out
    async def unified_for(self, posts: Iterable[Post]) -> dict[int, dict]:
        """post id -> the card's reference set, for each post that HAS one.
        Only teasers get one: a post with at least one `linked` association on
        the announcing side. Every other post is absent from the result, and
        the card renders exactly as it did before this module existed.
        """
        teasers = {p.id: p for p in posts if p.synthesized_by is None}
        if not teasers:
            return {}
        links = (await self.session.execute(
            select(PostAssociation)
            .where(
                PostAssociation.status == "linked",
                PostAssociation.announcement_post_id.in_(list(teasers)),
            )
            .order_by(PostAssociation.id)
        )).scalars().all()
        if not links:
            return {}
        settings = await self.session.get(ImportSettings, 1)
        window = timedelta(days=float(
            settings.discord_family_window_days if settings is not None else 60.0
        ))
        drop_images = await self._drop_images(
            sorted({a.payload_post_id for a in links})
        )
        by_teaser: dict[int, list[PostAssociation]] = {}
        for a in links:
            by_teaser.setdefault(a.announcement_post_id, []).append(a)
        out: dict[int, dict] = {}
        for teaser_id, assocs in by_teaser.items():
            teaser = teasers[teaser_id]
            artist = await self._artist(teaser.artist_id)
            out[teaser_id] = self._compose(teaser, assocs, drop_images, artist, window)
        return out
    def _compose(
        self,
        teaser: Post,
        assocs: list[PostAssociation],
        drop_images: dict[int, list[int]],
        artist: _Artist,
        window: timedelta,
    ) -> dict:
        def candidate(img_id: int) -> Candidate | None:
            row = artist.rows.get(img_id)
            if row is None:
                return None
            post_id, path, bits, *_ = row
            post = artist.posts.get(post_id)
            if post is None:
                return None
            return Candidate(img_id, post_id, path, bits, _when(post))
        drop_ids = [a.payload_post_id for a in assocs]
        shown = [i for d in drop_ids for i in drop_images.get(d, [])]
        own = [i for i, row in artist.rows.items() if row[0] == teaser.id]
        seed = [c for i in own + shown if (c := candidate(i)) is not None]
        # Variants come from Discord only. That is where a creator trickles
        # them out, it is the corpus the family rule was measured on, and it
        # keeps one teaser from pulling a DIFFERENT teaser's crop onto its card.
        pool = [
            c for i, row in artist.rows.items()
            if artist.platform.get(row[0]) == DISCORD
            and (c := candidate(i)) is not None
        ]
        variants = family(
            seed, pool, artist.name_posts, artist.hash_posts,
            anchor=_when(teaser), window=window,
        )
        def thumb(img_id: int, post_id: int, role: str) -> dict | None:
            row = artist.rows.get(img_id)
            if row is None:
                return None
            _pid, _path, _bits, sha, mime, tp = row
            return {
                "image_id": img_id,
                "thumbnail_url": thumbnail_url(tp, sha, mime),
                "mime": mime,
                "post_id": post_id,
                "role": role,
            }
        own_ids = set(own)
        thumbnails: list[dict] = []
        seen: set[int] = set(own_ids)
        for drop_id in drop_ids:
            for img_id in drop_images.get(drop_id, []):
                if img_id in seen:
                    continue
                if (t := thumb(img_id, drop_id, "drop")) is not None:
                    thumbnails.append(t)
                    seen.add(img_id)
        for c in variants:
            if c.image_id in seen:
                continue
            if (t := thumb(c.image_id, c.post_id, "variant")) is not None:
                thumbnails.append(t)
                seen.add(c.image_id)
        # The text of every item the card unifies — the operator's *"the
        # unified card should also contain the text for any of the items
        # unified on it"*. A drop's own description already joins its member
        # messages, so a variant's text is its MESSAGE, read off the member
        # post that owns the image. A line said twice (`@everyone 🍈🍈` on
        # every message of a drop) is shown once.
        texts: list[dict] = []
        said: set[str] = set()
        def add_text(post: Post | None, role: str) -> None:
            if post is None:
                return
            text = _text(post)
            if text is None or text in said:
                return
            said.add(text)
            texts.append({
                "post_id": post.id,
                "role": role,
                "date": _when(post).isoformat(),
                "text": text,
            })
        for drop_id in drop_ids:
            add_text(artist.posts.get(drop_id), "drop")
        for post_id in dict.fromkeys(c.post_id for c in variants):
            add_text(artist.posts.get(post_id), "variant")
        return {
            "links": [
                {
                    "association_id": a.id,
                    "post_id": a.payload_post_id,
                    # "fc" | "operator" | None (linked before the column
                    # existed — an operator accept, every one of them).
                    "linked_by": a.linked_by,
                    "token": (a.signals or {}).get("identity_token"),
                }
                for a in assocs
            ],
            "thumbnails": thumbnails,
            "variant_count": sum(1 for t in thumbnails if t["role"] == "variant"),
            "texts": texts,
        }
 def fold_clause(fold_hours: float):
    """WHERE clause: this post is NOT a linked drop sitting beside its teaser.
    Operator: *"discord 'posts' land as normal and only hidden from the post
    view they're posted the same day."* Everything else stays — an older
    variant the teaser also references is history, and a reference does not
    remove it from history.
    Built on `post_date`/`downloaded_at`, not the feed's `resurfaced_at`-led
    sort key: whether two posts are the same release is a question about when
    they were published, not about where the feed has since moved one.
    """
    teaser = aliased(Post)
    # The SQL-standard EXTRACT(epoch FROM …), which every Postgres accepts.
    gap = func.abs(extract(
        "epoch",
        func.coalesce(Post.post_date, Post.downloaded_at)
        - func.coalesce(teaser.post_date, teaser.downloaded_at),
    ))
    return ~exists(
        select(PostAssociation.id)
        .join(teaser, teaser.id == PostAssociation.announcement_post_id)
        .where(and_(
            PostAssociation.payload_post_id == Post.id,
            PostAssociation.status == "linked",
            gap <= fold_hours * 3600,
        ))
    )
@@ -115,6 +115,32 @@ async def active_platform_cooldowns(session: AsyncSession) -> dict[str, datetime
    return active
 def backfill_ready(source: Source) -> bool:
    """A deep walk the operator started, with budget left and no failure
    backing it off — due NOW rather than at its next scheduled check.
    A backfill runs one time-boxed chunk per download (plan #693), and nothing
    queued the next chunk: each waited for the source's regular interval. At
    the 8-hour default a freshly armed backfill sat untouched until the next
    check (the operator armed one on 2026-09-25 and saw nothing happen) and a
    five-chunk walk took most of two days. The tick's in-flight guard keeps
    one chunk at a time per source and the platform lock one walk per
    platform, so "due every tick" means "next chunk as soon as the last one
    ends".
    The failure gate is what keeps a broken source from retrying every
    minute: any failed chunk raises `consecutive_failures`, which drops the
    source back onto its backed-off interval. A chunk that fails to progress
    twice marks the walk stalled (download_service), which ends it here too.
    """
    co = source.config_overrides or {}
    return (
        co.get("_backfill_state") == "running"
        and (source.backfill_runs_remaining or 0) > 0
        and not (source.consecutive_failures or 0)
    )
 async def select_due_sources(session: AsyncSession) -> list[Source]:
    """Sources where (enabled, artist.auto_check) and now >= last_checked_at + effective_interval.
@@ -123,6 +149,9 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
    cooldown is the preventive half of the burst-prevention pair (per-source
    consecutive_failures backoff handles the offending source itself).
    A running backfill (`backfill_ready`) is due on every tick, and whether
    or not its artist is on auto-check — the operator started it by hand.
    Ordering: last_checked_at ASC NULLS FIRST, then id. Never-checked
    sources go first, then the longest-since-checked, so the most overdue
    sources hit Celery's FIFO download queue first. Anti-starvation: if
@@ -135,7 +164,6 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
        .options(selectinload(Source.artist))
        .join(Artist, Source.artist_id == Artist.id)
        .where(Source.enabled.is_(True))
        .where(Artist.auto_check.is_(True))
        .order_by(Source.last_checked_at.asc().nulls_first(), Source.id)
    )).scalars().all()
@@ -147,6 +175,11 @@ async def select_due_sources(session: AsyncSession) -> list[Source]:
    for s in rows:
        if s.platform in cooldowns:
            continue
        if backfill_ready(s):
            due.append(s)
            continue
        if not s.artist.auto_check:
            continue
        interval = compute_effective_interval(s, s.artist, settings)
        if s.last_checked_at is None:
            due.append(s)
@@ -163,6 +196,8 @@ def compute_next_check_at(
    """Return the projected datetime of the next check, or None if never checked."""
    if source.last_checked_at is None:
        return None
    if backfill_ready(source):
        return datetime.now(UTC)
    interval = compute_effective_interval(source, artist, settings)
    return source.last_checked_at + timedelta(seconds=interval)
@@ -229,7 +264,7 @@ async def scheduler_status(session: AsyncSession) -> dict:
    # links to cannot disagree about what they are counting.
    failing_sources = (await session.execute(
        select(func.count()).select_from(Source)
-        .where(Source.enabled.is_(True), failing_sources_clause())
+        .where(failing_sources_clause())
    )).scalar_one()
    no_access_sources = (await session.execute(
        select(func.count()).select_from(Source)
@@ -31,18 +31,20 @@ from __future__ import annotations
 import asyncio
 import logging
-from sqlalchemy import func, select
+from sqlalchemy import func
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 from ..models import ServiceSeen
 from .worker_lanes import LANES, lane_for_node
 log = logging.getLogger(__name__)
-# How stale the roster may be before a health request refreshes it. Comfortably
+# There is no refresh TTL any more. It existed because the HEALTH REQUEST
-# under the staleness thresholds that decide a service is missing, so the
+# refreshed the roster, rate-limited to 20s so that a page open in two tabs
-# verdict is never limited by how often anyone looked.
+# did not inspect twice as often. `size_worker_lanes` owns the refresh now, on
-REFRESH_TTL_SECONDS = 20.0
+# `SWEEP_PERIOD_SECONDS`, so the cadence is a schedule rather than a side
 # effect of someone looking.
 # celery inspect is a broker round trip and this sits on a request path, so it
 # gets a deadline (rule 156). A broker that has stopped answering must make the
@@ -50,16 +52,44 @@ REFRESH_TTL_SECONDS = 20.0
 # page that exists to explain it.
 INSPECT_TIMEOUT_SECONDS = 2.0
 # How many broadcast round trips `_inspect_celery_sync` makes. Named, because
 # the wrapper's budget is derived from it and the two must not drift.
 #
 # `active_queues()` and `active()` are separate broadcasts, and a broadcast
 # with no `destination` cannot know how many replies to expect — so each one
 # waits out its full timeout rather than returning on the last reply. The sync
 # call therefore costs ~2 x INSPECT_TIMEOUT_SECONDS in the ordinary case, not
 # once.
 INSPECT_ROUND_TRIPS = 2
 # Slack for the thread handoff. `asyncio.to_thread` hands work to the default
 # executor, and on a loaded web process — the operator's showcase page pulling
 # ninety thumbnails a second — the thread may not even be scheduled inside the
 # budget, let alone finish.
 #
 # This exists because the wrapper used to allow `INSPECT_TIMEOUT_SECONDS * 2`,
 # which LOOKS like a safety factor and is exactly the worst case with nothing
 # left over. Observed on the operator's first consolidated deploy, 2026-09-23:
 # a TimeoutError traceback per refresh while the two inspect calls were
 # working perfectly. A budget equal to the work is a budget that fails under
 # any load at all.
 INSPECT_SLACK_SECONDS = 3.0
 # Queue set -> the name an operator recognises. Sorted-tuple keys, because the
 # order celery reports them in is not guaranteed.
 #
-# A deployment that slices CELERY_QUEUES differently falls through to the raw
+# DERIVED from `worker_lanes.LANES` (milestone 422 step 1) rather than written
-# queue list rather than being given a name this table invented for it: a
+# out here. It was a hand-kept second copy of the same fact, and it had already
 # drifted: `maintenance_long` is a live lane with four task routes pointing at
 # it and a dedicated worker in the operator's stack, and this map did not know
 # it — so the System tab labelled it `Worker (maintenance_long)`. One list of
 # lanes now names them everywhere.
 #
 # A deployment that slices CELERY_QUEUES differently still falls through to the
 # raw queue list rather than being given a name this code invented for it: a
 # wrong-but-confident label on a status page is worse than an ugly true one.
 ROLE_NAMES: dict[tuple[str, ...], str] = {
-    ("default", "download", "import", "thumbnail"): "Worker",
+    lane.queue_key: lane.display_name for lane in LANES
    ("maintenance", "scan"): "Scheduler",
    ("ml",): "ML worker",
 }
@@ -80,12 +110,29 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
    from ..celery_app import celery as celery_app
    insp = celery_app.control.inspect(timeout=INSPECT_TIMEOUT_SECONDS)
    # TWO broadcasts, each waiting out its own timeout — see
    # INSPECT_ROUND_TRIPS, which the caller's budget is derived from. Adding a
    # third call here without updating that constant puts the wrapper back
    # under the work it is waiting for.
    active_queues = insp.active_queues() or {}
    active_tasks = insp.active() or {}
    grouped: dict[tuple[str, ...], dict] = {}
    for hostname, queues in active_queues.items():
-        key = tuple(sorted({q["name"] for q in queues}))
+        # Keyed on the LANE's queue set when the node name identifies one, so
        # a lane keeps the same roster row whether or not it is consuming.
        #
        # Grouping on the ACTIVE queues alone meant a lane at cap 0 — which
        # cancels its consumers — reported an empty set, landed under the key
        # `celery:`, and rendered as a phantom row named `Worker ()` while its
        # real row went stale beside it. Both symptoms on the operator's
        # screen, 2026-09-23, from this one line.
        #
        # Deriving the key from `lane.queue_key` rather than inventing a new
        # one keeps every existing row: it is the same string the lane already
        # had while it was running.
        lane = lane_for_node(hostname)
        key = lane.queue_key if lane else tuple(sorted({q["name"] for q in queues}))
        entry = grouped.setdefault(key, {"hostnames": [], "active": 0})
        entry["hostnames"].append(hostname)
        entry["active"] += len(active_tasks.get(hostname, []))
@@ -94,10 +141,13 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
    return grouped
-async def touch_service(
+def touch_service_stmt(*, key: str, kind: str, display_name: str, details: dict):
-    session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict
+    """The upsert that records a check-in, as a statement.
-) -> None:
+
-    """Record that a part checked in just now.
+    Built here rather than inline so the async caller (an agent lease, over
    the API) and the sync one (the sizing sweep, in a celery task) run the
    SAME write. Two spellings of one upsert is the kind of duplication that
    stays correct right up until one of them gains a column.
    Upsert rather than read-modify-write: several web processes and several
    agents can be doing this at once, and the last writer is simply the most
@@ -108,7 +158,7 @@ async def touch_service(
    stmt = pg_insert(ServiceSeen).values(
        key=key, kind=kind, display_name=display_name, details=details,
    )
-    stmt = stmt.on_conflict_do_update(
+    return stmt.on_conflict_do_update(
        index_elements=[ServiceSeen.key],
        set_={
            "kind": stmt.excluded.kind,
@@ -117,7 +167,75 @@ async def touch_service(
            "last_seen_at": func.now(),
        },
    )
-    await session.execute(stmt)
+
 async def touch_service(
    session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict
 ) -> None:
    """Record that a part checked in just now."""
    await session.execute(touch_service_stmt(
        key=key, kind=kind, display_name=display_name, details=details,
    ))
 def _roster_rows(grouped: dict[tuple[str, ...], dict]) -> list[dict]:
    """The `touch_service` arguments for everything that answered.
    Split from the write so the async and sync refreshes below share the
    mapping as well as the statement — what a roster row IS should not depend
    on which kind of session is writing it.
    """
    return [
        {
            "key": "celery:" + ",".join(queues),
            "kind": "celery",
            "display_name": role_display_name(queues),
            "details": {
                "queues": list(queues),
                "hostnames": entry["hostnames"],
                "replicas": len(entry["hostnames"]),
                "active": entry["active"],
            },
        }
        for queues, entry in grouped.items()
    ]
 def refresh_celery_roster_sync(session) -> None:
    """The roster refresh, from the sizing sweep's sync session.
    ## Why the sweep owns this now
    It used to run on the request path, rate-limited to once every 20s by the
    newest celery row. So the roster only advanced while somebody had a
    browser open — the liveness of the workers was a function of whether
    anyone was looking at them, which is the observer-effect version of the
    bug this roster exists to prevent.
    Operator, 2026-09-23: *"there is a repull every time this page loads — is
    there a reason this info isn't being tracked in the background and stored
    in some way?"*
    Now a timer writes it and the page only reads. The cadence is
    `SWEEP_PERIOD_SECONDS`, and `api/system_health` asserts it leaves headroom
    under the staleness thresholds — because a sweep period and a stale
    threshold chosen in different files and never compared is exactly how the
    idle GPU agent came to read as stopped (lesson #4355).
    Never raises. A failure means the roster does not advance, and the rows
    going stale is then a TRUE report about a broker nobody can reach.
    """
    try:
        grouped = _inspect_celery_sync()
    except Exception:
        log.warning(
            "service roster: celery inspect failed; roster not refreshed",
            exc_info=True,
        )
        return
    for row in _roster_rows(grouped):
        session.execute(touch_service_stmt(**row))
    session.commit()
 async def refresh_celery_roster(session: AsyncSession) -> None:
@@ -131,45 +249,14 @@ async def refresh_celery_roster(session: AsyncSession) -> None:
    try:
        grouped = await asyncio.wait_for(
            asyncio.to_thread(_inspect_celery_sync),
-            timeout=INSPECT_TIMEOUT_SECONDS * 2,
+            timeout=(
                INSPECT_TIMEOUT_SECONDS * INSPECT_ROUND_TRIPS
                + INSPECT_SLACK_SECONDS
            ),
        )
    except Exception:
        log.warning("service roster: celery inspect failed; roster not refreshed", exc_info=True)
        return
-    for queues, entry in grouped.items():
+    for row in _roster_rows(grouped):
-        await touch_service(
+        await touch_service(session, **row)
            session,
            key="celery:" + ",".join(queues),
            kind="celery",
            display_name=role_display_name(queues),
            details={
                "queues": list(queues),
                "hostnames": entry["hostnames"],
                "replicas": len(entry["hostnames"]),
                "active": entry["active"],
            },
        )
 async def refresh_if_stale(session: AsyncSession) -> None:
    """Refresh the celery roster if nobody has for REFRESH_TTL_SECONDS.
    Rate-limited by the data rather than by a lock: the gate is the newest
    last_seen_at across the celery rows, which every web process can see. Two
    processes racing through the gate costs one redundant inspect and writes
    the same values twice, so the benign outcome needs no coordination to
    prevent.
    """
    newest = (
        await session.execute(
            select(func.max(ServiceSeen.last_seen_at)).where(ServiceSeen.kind == "celery")
        )
    ).scalar_one_or_none()
    if newest is not None:
        age = (await session.execute(select(func.now()))).scalar_one() - newest
        if age.total_seconds() < REFRESH_TTL_SECONDS:
            return
    await refresh_celery_roster(session)
@@ -18,7 +18,9 @@ from ..models import (
    Source,
 )
 from .db_helpers import failing_sources_clause
 from .download_backends import uses_native_ingester
 from .gallery_dl import ErrorType
 from .membership_reconcile import KEPT_KEY, STOPPED_KEY
 from .membership_roster import gated_reasons_for_sources
 from .platforms import known_platform_keys
 from .scheduler_service import compute_next_check_at
@@ -124,6 +126,11 @@ class SourceRecord:
            "backfill_posts": self.backfill_posts,
            "tier_gated_count": self.tier_gated_count,
            "gated_reason": self.gated_reason,
            # Recover / recapture exist only on the native ingester. Sent so the
            # UI asks the backend's own predicate instead of keeping a copy of
            # the platform list — the copy said "patreon, subscribestar" for a
            # day after Discord went native (milestone 428).
            "native_ingester": uses_native_ingester(self.platform),
        }
@@ -158,6 +165,38 @@ def _is_app_managed(key: str) -> bool:
 BACKFILL_MAX_CHUNKS = 200
 def arm_backfill(source: Source) -> None:
    """Arm a fresh run-until-done backfill on `source` (plan #693). Mutation
    only — the caller commits. Shared by `SourceService.start_backfill` and the
    sync Discord repair, so both clear exactly the same resume state."""
    co = dict(source.config_overrides or {})
    co["_backfill_state"] = "running"
    for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
              "_backfill_posts"):
        co.pop(k, None)
    source.config_overrides = co
    source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
 def _record_manual_enable_choice(source: Source, *, enabled: bool) -> None:
    """Keep the membership sweep (#3995) from overriding the operator.
    Turning a source the sweep STOPPED back on is a deliberate choice to keep
    pulling a lapsed creator, so it is marked kept and the next sweep leaves it
    alone. Turning a source off by hand drops any sweep marker, so the sweep
    never switches back on something the operator switched off themselves.
    """
    co = dict(source.config_overrides or {})
    if enabled and STOPPED_KEY in co:
        co.pop(STOPPED_KEY)
        co[KEPT_KEY] = True
    elif not enabled:
        co.pop(STOPPED_KEY, None)
    else:
        return
    source.config_overrides = co
 class SourceService:
    def __init__(self, session: AsyncSession):
        self.session = session
@@ -415,6 +454,9 @@ class SourceService:
        for key, value in fields.items():
            setattr(source, key, value)
        if "enabled" in fields:
            _record_manual_enable_choice(source, enabled=bool(fields["enabled"]))
        if url_changed:
            # Repointing a source at a different creator makes a cached campaign
            # id WRONG, not merely stale, and `patreon_resolver` consults that
@@ -473,13 +515,7 @@ class SourceService:
        )).scalar_one_or_none()
        if source is None:
            raise LookupError(f"source id={source_id} not found")
-        co = dict(source.config_overrides or {})
+        arm_backfill(source)
        co["_backfill_state"] = "running"
        for k in ("_backfill_cursor", "_backfill_cursor_stalls", "_backfill_chunks",
                  "_backfill_posts"):
            co.pop(k, None)
        source.config_overrides = co
        source.backfill_runs_remaining = BACKFILL_MAX_CHUNKS
        await self.session.commit()
        return await self._row_to_record(source)
@@ -521,8 +557,8 @@ class SourceService:
        whole source); the two flags are mutually exclusive, so arming recapture
        clears bypass_seen. Clears prior cursor/chunk/stall state so it walks
        fresh from the top. The flag is cleared on completion (download_service)
-        and on stop. Recapture is Patreon-only (the native ingester's post-record
+        and on stop. Recapture needs the native ingester's post-record capture,
-        capture); inert elsewhere. The UI gates the action to Patreon sources."""
+        so the UI offers it on native sources only (`native_ingester`)."""
        source = (await self.session.execute(
            select(Source).where(Source.id == source_id)
        )).scalar_one_or_none()
@@ -316,7 +316,7 @@ _ROSTER_URL = f"{_ROSTER_BASE}/subscriptions"
 # `data-identifier`, the one vocabulary that names a state: the table class
 # inside the cancelled card says `for-unsubscribed_users`, a different word for
 # the same list (note #3989, CORRECTION 1). The identifier is stored verbatim as
-# Membership.status and mapped in membership_roster.MEMBERSHIP_STATUS.
+# Membership.status and mapped in native_ingest_common.MEMBERSHIP_STATUS.
 _ROSTER_ACTIVE = "active_subscriptions"
 _ROSTER_CANCELLED = "cancelled_subscriptions"
@@ -742,8 +742,15 @@ class SubscribeStarClient:
    @staticmethod
    def post_meta(post: dict) -> dict:
-        """Title + date for the preview sample. Title is synthesized from the body
+        """Title + date. Title is None — SubscribeStar has no title field, and
-        (SubscribeStar has no title field)."""
+        the importer synthesizes one from the body.
        `date` is the contract the core's REVISIT WINDOW reads (2026-09-23):
        ISO-8601 or None. NAIVE here, because `_parse_ss_datetime` renders a
        parsed local timestamp with no zone; the core reads a naive date as UTC
        rather than discarding it, since a date it refuses to read is a post
        the window can never reach.
        """
        attrs = post.get("attributes") or {}
        return {"title": None, "date": attrs.get("published_at")}
@@ -184,10 +184,20 @@ class SubscribeStarDownloader(BaseNativeDownloader):
        sidecar_path.write_text(json.dumps(data, indent=2))
        return sidecar_path
-    def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
+    def write_post_record(
        self, post: dict, artist_slug: str, *, revisit: bool = False,
    ) -> PostRecordOutcome:
        """Write the post-first `_post.json` (body/links/metadata) — the sole
        writer of the post record on the native path. SubscribeStar's body is
-        already in the feed HTML, so no detail-fetch is needed."""
+        already in the feed HTML, so no detail-fetch is needed.
        `revisit=True` is the tick re-reading a post it already captured
        (ingest_core's revisit window). The no-detail-fetch half of that
        contract is free here — there is no detail endpoint — but the
        don't-blank-a-stored-body half still applies: a chunk that parsed with
        no content must not overwrite a body we already have. Same guarantee as
        the Patreon downloader, for the same reason, so a walk behaves the same
        on both platforms."""
        attrs = post.get("attributes") or {}
        title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
        post_type = attrs.get("post_type") if isinstance(attrs.get("post_type"), str) else None
@@ -196,6 +206,12 @@ class SubscribeStarDownloader(BaseNativeDownloader):
            return PostRecordOutcome(
                path=None, post_type=post_type, title=title, body_chars=0,
            )
        if revisit:
            body = attrs.get("content")
            if not (isinstance(body, str) and body.strip()):
                return PostRecordOutcome(
                    path=None, post_type=post_type, title=title, body_chars=0,
                )
        post_dir = self.images_root / artist_slug / "subscribestar" / post_dir_name(post)
        post_dir.mkdir(parents=True, exist_ok=True)
        path = self._write_sidecar_data(post, post_dir / "_post.json")
@@ -0,0 +1,996 @@
 """Read and change a lane's live pool, over the broker.
 Milestone 422 step 2. The half of the milestone that does something.
 ## No docker socket is involved, and that is the point
 Milestone 365 put "acting on the state" out of scope because restarting a
 dead worker needs a docker socket the web container deliberately does not
 have. That is true of RESTARTING a container. It is not true of changing how
 much work a RUNNING worker does: celery's remote control sends a message over
 the broker and the worker resizes its own pool. Same Redis the app already
 uses, no new privilege, no new surface.
    pool_grow / pool_shrink   how many slots a lane runs
    add_consumer / cancel_consumer   whether it consumes its queues at all
 The operator ruled the socket out independently (2026-09-22: *"this feature
 is a very invasive idea in my mind and I'd like to avoid it"*), and nothing
 here raises the question.
 ## The setting is PER PROCESS, not per lane total
 `pool_grow(n, destination=[...])` adds n slots to EACH destination it names.
 While the stack still runs several containers per lane — the operator's
 production `worker` is `replicas: 2` — a single delta applied to a lane's
 total would be wrong for every replica.
 So `slots` means what `CELERY_CONCURRENCY` means: the pool size of one
 process. The reconcile below drives EACH replica to that number
 independently, computing its own delta from that replica's current pool, so
 replicas that have drifted apart (one restarted, one was grown) converge
 rather than being moved in lockstep from a shared baseline.
 After step 5 there is one process per lane and the distinction disappears.
 It matters now, and getting it wrong now would be invisible — the totals
 would simply be double what the UI claimed.
 ## Why reserved() is read alongside the queue depth
 Celery PREFETCHES: a worker pulls more messages than it can run and holds
 them in memory. Those have already left the Redis list, so `LLEN` — which is
 what `/api/system/activity/queues` reports — can read 0 while thirty tasks
 are waiting inside a worker. Any judgement about backlog that uses only LLEN
 under-reports, which matters for the UI and is disqualifying for step 7's
 autoscaler.
 """
 from __future__ import annotations
 import asyncio
 import logging
 from dataclasses import dataclass, field
 from datetime import UTC, datetime
 from sqlalchemy import func, select
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 from ..models import TaskRun, WorkerLane, WorkerLaneSample
 from .worker_lanes import (
    LANES,
    LANES_BY_QUEUE_KEY,
    MIN_POOL_SLOTS,
    Lane,
    derived_ceiling,
    lane_for_node,
 )
 log = logging.getLogger(__name__)
 # celery control is a broker round trip on a request path, so it gets a
 # deadline (rule 156) — the same reasoning and the same budget as
 # service_roster's inspect. A broker that stopped answering must make this
 # report "not present", which is true, rather than hang the page.
 CONTROL_TIMEOUT_SECONDS = 2.0
 # The WORST case of `inspect_lanes_sync`, for callers that need a deadline.
 #
 # One broadcast plus three targeted reads. The targeted three normally return
 # as soon as the named nodes answer; each can still cost a full timeout if a
 # node disappears mid-read, so the bound stays four.
 CONTROL_ROUND_TRIPS = 4
 # Slack for the `asyncio.to_thread` handoff. A budget equal to the work is a
 # budget that fails under load — the roster carried exactly that bug into the
 # operator's first consolidated deploy and logged a TimeoutError per refresh
 # while the inspect calls underneath were working fine.
 CONTROL_SLACK_SECONDS = 3.0
 INSPECT_BUDGET_SECONDS = (
    CONTROL_TIMEOUT_SECONDS * CONTROL_ROUND_TRIPS + CONTROL_SLACK_SECONDS
 )
@dataclass
 class LaneLiveState:
    """What `celery inspect` says about one lane right now.
    `present=False` is NOT "zero slots" — it is "nothing answered". A lane
    whose worker is restarting, or whose broker is unreachable, must read as
    unknown rather than as stopped: an unswept absence is not a verdict
    (snippet #3969). The reconcile in step 3 skips an absent lane rather than
    correcting it, which is only safe because this distinction is kept.
    """
    present: bool = False
    replicas: int = 0
    active: int = 0
    reserved: int = 0
    hostnames: list[str] = field(default_factory=list)
    # The queues this lane is actually consuming right now, across replicas.
    # Distinct from the lane's CONFIGURED queues: `cancel_consumer` stops a
    # worker consuming one without changing what it was started with, which
    # is how `enabled=false` is implemented. The reconcile needs this to tell
    # "already disabled" from "needs disabling" — without it, it would re-send
    # add_consumer for every queue on every tick forever (lesson #4183).
    consuming: set[str] = field(default_factory=set)
    # Pool size PER HOSTNAME, not aggregated. The resize below computes each
    # replica's own delta from its own current pool, so replicas that have
    # drifted apart converge instead of being moved in lockstep from a shared
    # baseline — which is what an aggregate here would silently reintroduce.
    pools: dict[str, int] = field(default_factory=dict)
    @property
    def pool(self) -> int | None:
        """One number for the UI. `max` rather than a sum: `slots` means the
        pool size of ONE process (see the module docstring), so the largest
        replica is the honest answer to "what is this lane set to". None when
        no replica reported — unknown, never zero."""
        return max(self.pools.values()) if self.pools else None
    @property
    def capacity(self) -> int:
        """Total slots across replicas — how many tasks this lane can run at
        once. Distinct from `pool`, and the two must not be confused: `pool`
        is the DIAL (one process's size, what grow/shrink move), `capacity` is
        the CAPABILITY. Asking "is this lane saturated" compares `active`,
        which is summed across replicas, against this — against `pool` it
        would call two half-busy replicas of 4 saturated at 4 active."""
        return sum(self.pools.values())
 def _lane_for_queues(queues: tuple[str, ...]) -> Lane | None:
    return LANES_BY_QUEUE_KEY.get(tuple(sorted(queues)))
 def inspect_lanes_sync() -> dict[str, LaneLiveState]:
    """Live state per lane name. Sync — callers wrap in asyncio.to_thread.
    Never raises. Every lane is present in the result; ones nothing answered
    for carry `present=False`, so a caller cannot accidentally read a missing
    lane as an empty one by iterating only what came back.
    """
    out = {lane.name: LaneLiveState() for lane in LANES}
    try:
        from ..celery_app import celery as celery_app
        # ONE broadcast, then three TARGETED reads.
        #
        # A broadcast with no `destination` cannot know how many replies to
        # expect, so it waits out its whole timeout rather than returning on
        # the last one. Four of those is four full timeouts — about eight
        # seconds — and `lane_view` sits on the Settings card, so that was the
        # load time of the Worker lanes page every time it was opened.
        #
        # Naming the destinations lets celery stop as soon as those nodes have
        # answered, which for workers in this same container is milliseconds.
        # The worst case is unchanged: a node that vanishes between the
        # broadcast and the targeted reads costs a full timeout waiting for a
        # reply that is not coming.
        insp = celery_app.control.inspect(timeout=CONTROL_TIMEOUT_SECONDS)
        active_queues = insp.active_queues() or {}
        # Nothing answered — and the three reads below exist only to describe
        # what did. Returning here also makes the broker-down case FAST
        # (one timeout, not four), which is exactly when the healthcheck and
        # the card need an answer rather than a long wait.
        if not active_queues:
            return out
        targeted = celery_app.control.inspect(
            destination=sorted(active_queues),
            timeout=CONTROL_TIMEOUT_SECONDS,
        )
        stats = targeted.stats() or {}
        active = targeted.active() or {}
        reserved = targeted.reserved() or {}
    except Exception:
        log.warning("worker_control: celery inspect failed", exc_info=True)
        return out
    for hostname, queues in active_queues.items():
        # The NODE NAME first — see `lane_for_node`. A lane at cap 0 has its
        # consumers cancelled and answers with an empty queue list, which
        # matches no lane, so attributing by queues alone dropped every lane
        # the operator had turned off and reported it as "not answering".
        lane = lane_for_node(hostname) or _lane_for_queues(
            tuple(q["name"] for q in queues)
        )
        if lane is None:
            # A deployment slicing CELERY_QUEUES differently. Reported by the
            # roster under its raw queue list; it simply has no lane row to
            # control, which is honest rather than an error.
            continue
        state = out[lane.name]
        state.present = True
        state.replicas += 1
        state.hostnames.append(hostname)
        state.active += len(active.get(hostname, []))
        state.reserved += len(reserved.get(hostname, []))
        state.consuming.update(q["name"] for q in queues)
        # Absent on a worker whose stats did not answer, which leaves
        # pool=None — unknown, not zero.
        pool = pool_size((stats.get(hostname) or {}).get("pool") or {})
        if pool is not None:
            state.pools[hostname] = pool
    for state in out.values():
        state.hostnames.sort()
    return out
 def pool_size(pool_stats: dict) -> int | None:
    """How many processes a prefork pool is running NOW, from `inspect stats`.
    The length of `processes`, not `max-concurrency`. Celery reports
    `max-concurrency` as the pool's `limit`, set once at boot; `pool_grow` and
    `pool_shrink` hand straight to billiard and never touch it (celery 5.6
    `concurrency/prefork.py`: `self.grow = P.grow`). So it read 1 forever on a
    lane booted at 1, and the sizing sweep, computing `target - 1` on every
    tick, grew a scheduler capped at 2 to six processes while the System tab
    showed "6 / 1" (2026-09-24) — and could never shrink one, since 1 - 1 is 0.
    `max-concurrency` is the fallback only for a pool that lists no processes
    (a non-prefork pool), where it is the best number there is.
    """
    procs = pool_stats.get("processes")
    if isinstance(procs, list):
        return len(procs)
    limit = pool_stats.get("max-concurrency")
    return limit if isinstance(limit, int) else None
 def effective_slots(target: int) -> int:
    """What a pool can actually be set to. Never below one process.
    Used wherever a target is COMPARED as well as wherever one is sent: a
    reconcile that compares against the unclamped number sees a difference
    that no control message can ever close, and re-sends it every tick.
    """
    return max(MIN_POOL_SLOTS, target)
 def set_lane_slots_sync(
    lane: Lane, target: int, live: LaneLiveState | None = None,
 ) -> tuple[bool, str | None]:
    """Drive every replica of `lane` to `target` slots. Returns (applied, err).
    Per-replica deltas rather than one shared delta: see the module docstring.
    A replica already at the target is issued nothing at all, which is what
    makes step 3's periodic reconcile converge instead of re-sending a grow of
    zero forever (lesson #4183 — an enforcer without a reachable fixed point
    re-does its own work every tick).
    `applied=False` is not a failure of the SETTING. The caller has already
    stored the value; this says only that the live push did not land, and the
    reconcile will carry it when the lane answers again.
    """
    target = effective_slots(target)
    try:
        from ..celery_app import celery as celery_app
        if live is None:
            live = inspect_lanes_sync()[lane.name]
        if not live.present:
            return False, "lane is not running"
        if not live.pools:
            return False, "worker did not report its pool size"
        control = celery_app.control
        unreported = [h for h in live.hostnames if h not in live.pools]
        for hostname, current in live.pools.items():
            delta = target - current
            if delta > 0:
                control.pool_grow(delta, destination=[hostname])
            elif delta < 0:
                control.pool_shrink(-delta, destination=[hostname])
        if unreported:
            # Resized what could be resized, and said which could not. Silence
            # here would leave a replica running at a size the UI claims it is
            # not, with nothing anywhere recording the gap.
            return False, f"no pool size reported by {', '.join(sorted(unreported))}"
        return True, None
    except Exception as exc:  # noqa: BLE001 — reported, never raised at a caller
        log.warning("worker_control: could not resize %s", lane.name, exc_info=True)
        return False, str(exc)
 def set_lane_enabled_sync(
    lane: Lane, enabled: bool, live: LaneLiveState | None = None,
 ) -> tuple[bool, str | None]:
    """Start or stop `lane` consuming its queues, without killing the process.
    `cancel_consumer` rather than a shutdown: a stopped consumer keeps its
    worker alive and answering `inspect`, so a disabled lane stays visible and
    can be turned back on. A killed worker would read as absent, which is the
    same signal as a crash — and the whole point of the roster (#365) is that
    those two must not look alike.
    """
    try:
        from ..celery_app import celery as celery_app
        if live is None:
            live = inspect_lanes_sync()[lane.name]
        if not live.present:
            return False, "lane is not running"
        control = celery_app.control
        for queue in lane.queues:
            if enabled:
                control.add_consumer(queue, destination=live.hostnames)
            else:
                control.cancel_consumer(queue, destination=live.hostnames)
        return True, None
    except Exception as exc:  # noqa: BLE001
        log.warning(
            "worker_control: could not %s %s",
            "enable" if enabled else "disable", lane.name, exc_info=True,
        )
        return False, str(exc)
 # --- the settings half, which is async ----------------------------------------
 #
 # Sync celery control above, async DB below, in one module. Same split
 # `service_roster` already runs (`_inspect_celery_sync` beside `touch_service`)
 # — the boundary is the transport, not the concern, and "control the workers"
 # is one concern.
 async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]:
    """Every lane's row, creating any that are missing from its LANES defaults.
    Self-heals rather than depending on a migration having run for a lane
    added later: alembic 0103 seeded the four that existed on 2026-09-22, and
    a fifth added to LANES afterwards gets its row the first time anything
    asks. Without this, a new lane would read as absent and the UI would
    simply not show it.
    """
    rows = {
        row.name: row
        for row in (await session.execute(select(WorkerLane))).scalars()
    }
    missing = [lane for lane in LANES if lane.name not in rows]
    for lane in missing:
        row = WorkerLane(name=lane.name, slots_cap=lane.default_slots_cap)
        session.add(row)
        rows[lane.name] = row
    if missing:
        await session.commit()
    return rows
@dataclass(frozen=True)
 class LaneSample:
    """What the sizing sweep last measured about one lane.
    The same fields `LaneLiveState` carries, plus the queue depth and WHEN —
    because this one is read from a table rather than from the broker, and a
    reading with no timestamp invites being presented as current.
    `measured_at=None` means no sweep has written this lane yet: a fresh
    install inside its first period, or a stack whose beat is not running.
    Distinct from `present=False` (something asked, nothing answered), and the
    UI says different things about the two.
    """
    present: bool = False
    replicas: int = 0
    pool: int | None = None
    active: int = 0
    reserved: int = 0
    queue_depth: int | None = None
    measured_at: datetime | None = None
 def _lane_depth(lane: Lane, depths: dict[str, int | None]) -> int | None:
    """A lane's backlog across its queues — None when NOTHING answered.
    A queue the broker did not answer for must not be summed as zero: an
    unknown depth is not an empty one, and reporting a buried lane as idle is
    the direction that matters.
    """
    known = [depths.get(q) for q in lane.queues]
    if not any(d is not None for d in known):
        return None
    return sum(d for d in known if d is not None)
 def store_lane_samples_sync(session, live: dict[str, LaneLiveState], depths) -> None:
    """Write what the sweep just measured. SYNC — the celery task owns a sync
    session, and this is the only place these rows are written.
    Upsert per lane, last writer wins, same shape as `service_roster`'s
    `touch_service`: two processes sweeping at once is a benign race that
    needs no coordination, because both are recording what they actually saw.
    A lane that did not answer is STILL written, with `present=False`. Skipping
    it would leave the previous reading in place and let the page go on showing
    a pool that is no longer there — the stale row would read as a current one
    (lesson #4202: the row is the thing that has to change).
    """
    now = datetime.now(UTC)
    for lane in LANES:
        state = live.get(lane.name) or LaneLiveState()
        values = {
            "lane": lane.name,
            "present": state.present,
            "replicas": state.replicas,
            "pool": state.pool,
            "active": state.active,
            "reserved": state.reserved,
            "queue_depth": _lane_depth(lane, depths),
            "measured_at": now,
        }
        stmt = pg_insert(WorkerLaneSample).values(**values)
        session.execute(stmt.on_conflict_do_update(
            index_elements=[WorkerLaneSample.lane],
            set_={k: v for k, v in values.items() if k != "lane"},
        ))
    session.commit()
@dataclass
 class LaneSettings:
    """What the DATABASE knows about the lanes — read and finished with before
    anything touches the broker.
    This exists because holding a Postgres connection across a celery round
    trip is what made the System tab block the whole site (operator,
    2026-09-23: *"something about changing the cap number is blocking to the
    website"*).
    `lane_view` used to take the session and keep it open through an inspect
    whose budget is eleven seconds — and that page polls every fifteen. With a
    lane not answering, every inspect ran to nearly its full budget, so each
    poll pinned a connection for ten seconds. SQLAlchemy's default pool is
    five connections plus ten overflow; a couple of browser tabs, the health
    endpoint doing the same thing, and a cap change adding two more inspects
    exhausts that, and every OTHER request then waits on a connection.
    Nothing was slow in itself. The slowness was a scarce resource held across
    it, which is why it surfaced as the whole site stalling rather than as one
    slow page.
    """
    caps: dict[str, int]
    oldest_by_queue: dict[str, datetime]
    # The sizing sweep's last reading per lane. Since 2026-09-23 this is where
    # the live numbers come from: the endpoint no longer inspects at all.
    samples: dict[str, LaneSample] = field(default_factory=dict)
 async def lane_settings(session: AsyncSession) -> LaneSettings:
    """Every DB read the lane view needs, in one short-lived session.
    Which is now ALL of them. `lane_view` below takes what this returns and
    talks to nothing.
    """
    rows = await _rows_by_name(session)
    samples = {
        row.lane: LaneSample(
            present=row.present,
            replicas=row.replicas,
            pool=row.pool,
            active=row.active,
            reserved=row.reserved,
            queue_depth=row.queue_depth,
            measured_at=row.measured_at,
        )
        for row in (
            await session.execute(select(WorkerLaneSample))
        ).scalars()
    }
    return LaneSettings(
        caps={name: row.slots_cap for name, row in rows.items()},
        oldest_by_queue=await _oldest_running_by_queue(session),
        samples=samples,
    )
 def lane_view(settings: LaneSettings) -> list[dict]:
    """Every lane: what is configured, what was last measured, what it may
    grow to. NO broker call, and no database — `settings` is the whole input.
    ## It used to inspect, on every request
    Four broadcast round trips on an eleven-second budget, on a page that
    polls every fifteen seconds. Operator, 2026-09-23: *"there is a repull
    every time this page loads — is there a reason this info isn't being
    tracked in the background and stored in some way?"*
    There was one, and it had expired. The docstring here used to say the
    endpoint was deliberately uncached because *"this is the surface an
    operator watches while dragging a stepper, and a cached reply would show
    them the value from before their own change"*. True while a cap change
    refetched the table — and that refetch is exactly what was removed in
    `1353d34`, so the UI now patches its own row from the write's reply and
    nothing depends on this being live.
    Meanwhile `size_worker_lanes` was already inspecting on a timer to decide
    pool sizes: the same numbers, computed, used, and discarded, while the
    browser asked the broker for them again four times a minute.
    So the sweep writes `worker_lane_sample` and this reads it. The reading is
    up to `SWEEP_PERIOD_SECONDS` old, and `measured_at` travels with it so the
    UI can say so rather than implying it is current.
    `pending` is still the honest backlog — depth PLUS reserved — because
    celery prefetches and LLEN alone reads 0 while a worker holds tasks in
    memory.
    """
    oldest = settings.oldest_by_queue
    now = datetime.now(UTC)
    out = []
    for lane in LANES:
        cap = settings.caps[lane.name]
        # A lane with no row yet is not-measured, which is distinct from
        # measured-as-absent. The default carries `measured_at=None`, and the
        # UI says "not measured yet" rather than "not answering".
        sample = settings.samples.get(lane.name) or LaneSample()
        depth = sample.queue_depth
        out.append({
            "name": lane.name,
            "display_name": lane.display_name,
            "queues": list(lane.queues),
            "slots_cap": cap,
            "ceiling": derived_ceiling(lane),
            # DERIVED, never stored. A cap of zero means no consumers, so
            # "off" and "may use no workers" cannot disagree.
            "enabled": cap > 0,
            "memory_bound": lane.memory_bound,
            "optional": lane.optional,
            # What raising this lane's cap will download, so the UI can say
            # WHICH model and how big BEFORE the first slot is asked for
            # rather than after a multi-GB fetch has started. `measured`
            # travels with the numbers: the UI must not present an estimate
            # as a fact.
            "models": [
                {
                    "repo": m.repo,
                    "download_bytes": m.approx_download_bytes,
                    "resident_bytes": m.approx_resident_bytes,
                    "measured": m.measured,
                }
                for m in lane.models
            ],
            "live": {
                "present": sample.present,
                "replicas": sample.replicas,
                "pool": sample.pool,
                "active": sample.active,
                "reserved": sample.reserved,
            },
            "queue_depth": depth,
            "pending": None if depth is None else depth + sample.reserved,
            # When the numbers above were read. Per lane rather than one for
            # the response, because a lane whose row has never been written
            # has no reading at all and must not borrow another lane's.
            "measured_at": (
                sample.measured_at.isoformat() if sample.measured_at else None
            ),
            # How long the oldest still-running task on this lane has been
            # going, in minutes. Read from `task_run`, not from the sweep, so
            # this one IS current. The operator asked for a trigger here —
            # grow a lane whose tasks run past some duration — and it stayed a
            # REPORT: a long task does not finish sooner because the lane
            # gained a slot, so scaling on it would spend memory to change
            # nothing. Shown so they can see a lane wedged on one slow job,
            # which is the genuinely useful half of the idea.
            "oldest_running_minutes": _minutes_since(
                min(
                    (oldest[q] for q in lane.queues if q in oldest),
                    default=None,
                ),
                now,
            ),
        })
    return out
 async def _oldest_running_by_queue(session: AsyncSession) -> dict[str, datetime]:
    """When the longest-running unfinished task on each queue started.
    Read from `task_run`, which is OUR OWN table on OUR OWN wall clock, and
    deliberately not from celery's `inspect active()`. Those entries carry a
    `time_start` taken from the WORKER's `time.monotonic()` — a clock with an
    arbitrary origin per process. Subtracting it from this process's wall
    clock produces a number that looks like a duration and is meaningless, and
    it would be meaningless in the direction that matters: plausible.
    `task_run` also already carries the per-queue staleness thresholds the
    recovery sweep uses, so a row still `running` here is one the system
    itself considers legitimately in flight rather than abandoned.
    """
    result = await session.execute(
        select(TaskRun.queue, func.min(TaskRun.started_at))
        .where(TaskRun.status == "running", TaskRun.finished_at.is_(None))
        .group_by(TaskRun.queue)
    )
    return {queue: started for queue, started in result if started is not None}
 def _minutes_since(started: datetime | None, now: datetime) -> int | None:
    """Whole minutes, or None when nothing is running. Never negative: a row
    written by a container whose clock is a few seconds ahead must read as 0
    rather than as a task that starts in the future."""
    if started is None:
        return None
    return max(0, int((now - started).total_seconds() // 60))
 def _queue_depths_sync() -> dict[str, int | None]:
    """Redis LLEN per queue. None for one that did not answer — see lane_view.
    Sync; the caller threads it. A per-queue try/except so one bad queue does
    not cost the whole report, matching `api/system_activity._read_queues_sync`.
    """
    import redis
    from ..config import get_config
    out: dict[str, int | None] = {}
    try:
        client = redis.Redis.from_url(get_config().celery_broker_url)
    except Exception:
        log.warning("worker_control: no broker for queue depths", exc_info=True)
        return {q: None for lane in LANES for q in lane.queues}
    for lane in LANES:
        for queue in lane.queues:
            try:
                out[queue] = int(client.llen(queue))
            except Exception:  # noqa: BLE001 — a hiccup must not break the UI
                out[queue] = None
    return out
 class LaneUpdateRefused(ValueError):
    """A requested value is outside what the lane may hold. Carries the reason
    the UI shows — a greyed control with no explanation reads as a bug."""
 async def store_lane_cap(
    session: AsyncSession, lane: Lane, slots_cap: int,
 ) -> int:
    """Validate and store the cap. Returns the PREVIOUS cap. DB only.
    Split from the live push for the reason `LaneSettings` gives at length: a
    Postgres connection must not be held across a celery round trip. Everything
    here is fast and finished with before `push_lane_cap` starts.
    """
    rows = await _rows_by_name(session)
    row = rows[lane.name]
    ceiling = derived_ceiling(lane)
    if slots_cap < 0:
        raise LaneUpdateRefused("a cap cannot be negative")
    if slots_cap > ceiling:
        raise LaneUpdateRefused(
            f"a cap of {slots_cap} is above what this container can hold "
            f"({ceiling} for {lane.display_name})"
        )
    was_cap = row.slots_cap
    row.slots_cap = slots_cap
    await session.commit()
    # The previous value, because the push needs the DIRECTION: lowering a cap
    # has to reach the running lane now, and raising one has nothing to say.
    return was_cap
 async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
    """Make the running lane obey a cap that is already stored. NO database.
    Runs OFF the request path since 2026-09-23 — the endpoint stores the cap,
    answers, and hands this to a background task (operator: *"the change
    should be queued so that it isn't blocking of the webui"*). Nothing here
    changed as a result except who waits for it: the return value is now read
    by the log rather than by a browser, and every branch below already
    treated failure as "the sizing pass will carry it".
    ## What is pushed, and what is not
    Consumers follow the cap immediately in BOTH directions: zero means off,
    and off must take effect when it is asked for rather than up to a minute
    later.
    The pool is only ever pushed DOWNWARD. Raising a cap is permission, not a
    request — growing on permission would put workers on a lane with nothing
    to do — so the sizing pass spends it on its next tick if there is work.
    That also makes the common case (raising a cap) free: no broker round trip
    AT ALL, which is the difference between a control that answers instantly
    and one that takes ten seconds. Keyed on the previous cap rather than on
    "is it on" — the first cut only knew on/off, so it inspected on every
    raise to find out whether the pool needed lowering, and the control it was
    meant to make instant still waited out an inspect.
    A failed push is not a failed setting. The value is already stored and the
    sizing pass carries it within a minute; `applied: false` with a reason
    lets the UI say "saved, not yet live" rather than "that didn't work"
    (lesson #4202 — a live change that does not survive, with nothing saying
    so).
    """
    was_on, now_on = was_cap > 0, slots_cap > 0
    applied, error = True, None
    if now_on != was_on:
        applied, error = await asyncio.to_thread(set_lane_enabled_sync, lane, now_on)
    if applied and not now_on:
        # Down to the floor at once. The pool cannot be emptied, so "off" is
        # one parked process with its consumers cancelled.
        applied, error = await asyncio.to_thread(
            set_lane_slots_sync, lane, MIN_POOL_SLOTS,
        )
    elif applied and now_on and slots_cap < was_cap:
        # LOWERED on a running lane. Only this direction needs a message, and
        # only when the pool is actually above the new cap — so it reads the
        # live pool rather than resizing blind. A raise never reaches here.
        #
        # Bounded (rule 156): `to_thread` on its own is an await with no
        # deadline, and this runs in a background task where a hang would be
        # silent rather than visible as a slow page. On a timeout the lane is
        # simply not resized here and the sizing sweep carries it.
        try:
            live = await asyncio.wait_for(
                asyncio.to_thread(inspect_lanes_sync),
                timeout=INSPECT_BUDGET_SECONDS,
            )
        except TimeoutError:
            log.warning(
                "worker_control: inspect exceeded %ss lowering %s; leaving the "
                "pool to the sizing pass", INSPECT_BUDGET_SECONDS, lane.name,
            )
            return _cap_result(lane, slots_cap, now_on, applied, error, False)
        current = live[lane.name].pool
        if current is not None and current > slots_cap:
            applied, error = await asyncio.to_thread(
                set_lane_slots_sync, lane, slots_cap, live=live[lane.name],
            )
    # Raising the cap off zero is what triggers the model download (milestone
    # 422 step 6). Never at boot: that made every start of the ML role reach
    # HuggingFace for ~3.5GB, and rule 164 permits a runtime fetch only for a
    # feature that is optional and clearly OFF.
    #
    # On the TRANSITION, so re-saving a cap on a lane already running does not
    # re-enqueue.
    #
    # NOT gated on the consumer change having landed, which it was until
    # 2026-09-23. The reasoning then was that enqueueing onto a queue nothing
    # consumes leaves the task pending — true, and it is the right place for
    # it to wait. Gated, a cap raised while the lane was restarting stored the
    # cap, let the sizing pass start the consumers a minute later, and left
    # the lane running with no model, because nothing else ever asks for one.
    # A task parked on the `ml` queue is picked up the moment that happens.
    fetching = False
    if now_on and not was_on and lane.models:
        fetching = _enqueue_model_fetch()
    # Nobody is waiting on this any more, so the log is where a push that did
    # not land has to be visible. Not an error: the value is stored and the
    # sizing pass carries it within a minute.
    if not applied:
        log.info(
            "worker_control: %s cap %s stored, not pushed (%s); "
            "the sizing pass will carry it",
            lane.name, slots_cap, error,
        )
    return _cap_result(lane, slots_cap, now_on, applied, error, fetching)
 def _cap_result(
    lane: Lane, slots_cap: int, now_on: bool, applied: bool,
    error: str | None, fetching: bool,
 ) -> dict:
    """The push's outcome. One builder, because `push_lane_cap` has two exits
    and a second literal would be free to disagree with the first."""
    return {
        "name": lane.name,
        "slots_cap": slots_cap,
        "ceiling": derived_ceiling(lane),
        "enabled": now_on,
        "applied": applied,
        "apply_error": error,
        # Tells the UI to say a download has started rather than leaving the
        # operator to wonder why a lane they just turned on is busy.
        "fetching_models": fetching,
    }
 def _enqueue_model_fetch() -> bool:
    """Queue the model download. Returns whether it was accepted.
    Import inside the function: `backend.app.tasks.ml` pulls in torch, and web
    must not pay that import cost on a module that every settings request
    touches.
    Never raises. A broker that will not take the task is worth reporting, but
    the SETTING has already been stored and the lane is already enabled — so
    failing the whole request here would roll back nothing and tell the
    operator their change did not happen when it did.
    """
    try:
        from ..tasks.ml import ensure_models
        ensure_models.delay()
        return True
    except Exception:  # noqa: BLE001 — reported, never raised at a caller
        log.warning("worker_control: could not enqueue the model fetch", exc_info=True)
        return False
 # --- the sizing pass: one sweep, always on ------------------------------------
 #
 # This replaced BOTH `reconcile_lanes_sync` (step 3) and `autoscale_lanes_sync`
 # (step 7) on 2026-09-23. They were two enforcers over one number, and the
 # whole of step 7's hardest reasoning — a stored value that is a FLOOR, a
 # target of `max(stored, current)` so the reconcile does not undo what the
 # autoscaler added — existed only to stop them fighting. Delete one of them and
 # the problem is not solved, it is absent.
 #
 # Operator: *"auto should be always on, not a setting, so that idle instances
 # quiet down when not running. the number that is visible and something the
 # user can tweak and manage should be the cap itself the number of running
 # workers is handled by the autoscaling function which is always on."*
 #
 # So there is one pass, it runs every minute, it reads the live pool rather
 # than any stored number, and the only thing it obeys is the cap.
 #
 # It also subsumes what the reconcile existed for. `pool_grow` is not durable:
 # a worker restarted by its supervisor comes back at its ENV concurrency,
 # silently below what the lane should be running. This pass reads the live
 # pool every minute and sizes from the backlog, so that worker is corrected on
 # the next tick — sooner than the five-minute reconcile managed, and without a
 # second sweep that could disagree with this one.
 # How much work justifies a slot. `pending` is depth + reserved, so it already
 # counts what celery has prefetched into worker memory — one task, one slot.
 #
 # Growth is IMMEDIATE and shrink is one slot per tick, deliberately asymmetric.
 # A backlog of four thousand should not take an hour to reach the cap, and a
 # lane that idles for one minute should not drop every process it has: the
 # cost of being one slot too large for a minute is a sleeping process, and the
 # cost of being too small is work not happening. For ML the asymmetry matters
 # most — every new slot reloads a multi-GB model, so the slow shrink is what
 # stops a quiet patch from paying that cost again a minute later.
 SHRINK_STEP = 1
@dataclass
 class LaneSizing:
    """What the pass did to one lane, and why — in the operator's terms.
    A reason on every outcome including "held", because a sizing pass that
    only speaks when it acts is one nobody can debug when it does not.
    """
    lane: str
    action: str  # "grew" | "shrank" | "held" | "skipped"
    slots: int
    reason: str
 def wanted_slots(cap: int, active: int, pending: int | None) -> int:
    """How many workers this lane has work for right now, within its cap.
    One slot per task in flight or waiting, floored at one process and
    ceilinged by the cap. `pending` of None means the broker did not answer
    for this lane's queues — an unknown backlog is not an empty one (snippet
    #3969), so it contributes nothing rather than being read as zero.
    A cap of zero still returns one: billiard cannot run an empty pool, and
    the parked process is what `add_consumer` lands on when the cap goes back
    up. "Off" is expressed by cancelling consumers, not by emptying the pool.
    """
    if cap <= 0:
        return MIN_POOL_SLOTS
    return max(MIN_POOL_SLOTS, min(cap, active + (pending or 0)))
 def size_lanes_sync(
    caps: dict[str, int],
    *,
    live: dict[str, LaneLiveState] | None = None,
    depths: dict[str, int | None] | None = None,
 ) -> list[LaneSizing]:
    """Size every lane to its backlog, within the cap. The whole control loop.
    `caps` is lane name -> slots_cap, read from the database by the caller.
    This function touches no database: the celery task that schedules it owns
    the session, and keeping the DB out of here is what lets it be called from
    anywhere that already knows the caps.
    `live` and `depths` are the measurements. Passing them in is not an
    optimisation — it is how the caller gets to KEEP them. The sweep now
    stores what it measured (`worker_lane_sample`) so the System tab reads a
    table instead of inspecting on every page load, and that is only possible
    if the same reading serves both purposes. Measured here when not given, so
    every existing caller and test is unaffected.
    ## It must converge and then go quiet
    One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing to a
    replica already at its target. A settled system therefore performs one
    broker round trip plus one LLEN sweep per tick and sends no control
    messages at all — the reachable fixed point lesson #4183 is about. An
    enforcer that re-sent a grow of zero every tick would churn forever and
    bury a real correction in its own noise.
    ## An absent lane is SKIPPED, not corrected
    `present=False` means nothing answered — a worker restarting, or an
    unreachable broker. It does NOT mean zero slots. Deciding from that would
    be a verdict drawn from an unswept read, and here it is worse than
    useless: there is nothing to send the message to.
    """
    if live is None:
        live = inspect_lanes_sync()
    if depths is None:
        depths = _queue_depths_sync()
    out: list[LaneSizing] = []
    for lane in LANES:
        cap = caps.get(lane.name)
        if cap is None:
            continue
        state = live[lane.name]
        if not state.present:
            out.append(LaneSizing(lane.name, "skipped", 0, "lane is not answering"))
            continue
        # Consumers first, and only when they DISAGREE. Sending add_consumer
        # for every queue on every tick of a settled system is the exact churn
        # above, and invisible: add_consumer on a queue already consumed is
        # harmless and reports success.
        should_consume = cap > 0
        if should_consume != state.consuming.issuperset(lane.queues):
            ok, err = set_lane_enabled_sync(lane, should_consume, live=state)
            if not ok:
                out.append(LaneSizing(
                    lane.name, "held", state.pool or 0,
                    f"could not {'start' if should_consume else 'stop'} "
                    f"consuming: {err}",
                ))
                continue
        current = state.pool
        if current is None:
            out.append(LaneSizing(
                lane.name, "held", 0, "worker did not report its pool size",
            ))
            continue
        depth = _lane_depth(lane, depths)
        pending = None if depth is None else depth + state.reserved
        want = wanted_slots(cap, state.active, pending)
        if want > current:
            new = want
            verb = "grew"
        elif want < current:
            # One at a time on the way down. See SHRINK_STEP.
            new = max(want, current - SHRINK_STEP)
            verb = "shrank"
        else:
            out.append(LaneSizing(
                lane.name, "held", current,
                f"{pending if pending is not None else '?'} waiting, "
                f"{state.active} busy, cap {cap}",
            ))
            continue
        ok, err = set_lane_slots_sync(lane, new, live=state)
        if not ok:
            out.append(LaneSizing(
                lane.name, "held", current, f"could not resize: {err}",
            ))
            continue
        out.append(LaneSizing(
            lane.name, verb, new,
            f"{pending if pending is not None else '?'} waiting, "
            f"{state.active} busy, cap {cap}",
        ))
    return out
@@ -0,0 +1,464 @@
 """The worker lanes: what they are, and how many slots each may be given.
 Milestone 422 step 1. This module is the ONE place that knows the lane set;
 `models/worker_lane.py` holds only what the operator can change about them.
 ## Why the queues are here and not in the table
 A lane's queue set is not a preference — it is decided by `celery_app.py`'s
 `task_routes`, which is what puts a backup on `maintenance_long` and a
 thumbnail on `thumbnail`. An operator cannot move a task to another lane, so
 storing the queues as settings would create a row that can disagree with the
 routing table, and nothing would notice until a queue had no consumer.
 So: queues and display names are code, slots and caps are data. The table
 stores three numbers and a flag, and nothing that could contradict celery.
 This also collapses a duplicate rather than adding one.
 `service_roster.ROLE_NAMES` was a second copy of "queue set -> the name an
 operator recognises", and it had already drifted: `maintenance_long` is a
 live lane with four task routes pointing at it, and the roster did not know
 its name, so the System tab rendered it as `Worker (maintenance_long)`. That
 map is now derived from `LANES` below, so a lane added here is named
 everywhere at once.
 ## Why the ceiling is derived rather than configured
 Consolidating the stack into one container (step 5) widens the OOM blast
 radius: today an ml-worker that exhausts memory is killed by Docker on its
 own, and web keeps serving. In one container the kernel picks a victim from
 the whole cgroup, and it may pick hypercorn — so a tagging task can take the
 UI down with it, on exactly the modest hardware least able to spare the
 memory.
 Operator, 2026-09-22: *"ram isn't an issue for me but some users might run
 this on weaker hardware and I don't want it to kill their servers."*
 So the maximum is computed from what the container actually has, and the
 operator's own `slots_cap` must fit under it. Three numbers, not two, and the
 ordering is the point:
    slots  <=  slots_cap  <=  derived_ceiling
    (live)     (operator)     (this module)
 The operator can always lower their cap. They cannot raise it past what the
 box can hold. The derived ceiling is never stored — a row that outlived a
 change in container limits must not carry a stale one.
 """
 from __future__ import annotations
 import logging
 import os
 from dataclasses import dataclass
 from pathlib import Path
 log = logging.getLogger(__name__)
 # --- the lanes ---------------------------------------------------------------
 GIB = 1024 ** 3
@dataclass(frozen=True)
 class ModelRequirement:
    """A model a lane must download before it can do anything.
    Surfaced to the UI so the operator is told WHICH model, how big, and what
    it costs to hold — before they turn the lane on, not after a multi-GB
    download has already started. The lane is optional and its cost is not
    obvious from its name, which is the whole reason this is structured data
    rather than a sentence in a component.
    `measured=False` means the numbers are ESTIMATES and the UI must say so.
    They come from the checkpoint's parameter count and dtype, not from a
    build — and a number presented as fact decides whether someone's server
    survives, so it is labelled rather than rounded confidently.
    """
    # The Hugging Face repo id, which is the honest answer to "which model".
    repo: str
    # Roughly what the download costs, for the operator's bandwidth and disk.
    approx_download_bytes: int
    # Roughly what ONE slot holds while running. Prefork forks a child per
    # slot and each loads its own copy, so this multiplies.
    approx_resident_bytes: int
    measured: bool = False
 # SigLIP so400m — the only model FabledCurator itself downloads.
 #
 # What it is for, which is NOT obvious from the lane's name: it produces the
 # image embeddings that back similarity search, duplicate grouping and the
 # tag heads. WD14 tagging is the GPU AGENT's job, not this lane's — the
 # comment in celery_app.py naming both is stale since B3 (#1238), when the
 # agent took over and this lane was left as the CPU embed fallback for stacks
 # running no agent at all (see MLSettings.cpu_embed_enabled).
 #
 # Both numbers are ESTIMATES, derived from the checkpoint rather than from a
 # build: ~877M parameters at fp32 is ~3.5GB of weights, and holding them plus
 # activations and the torch runtime is what the resident figure covers. They
 # err high. Replace them with measurements — download the repo and read its
 # size; run one embed and read the worker child's VmHWM — and set
 # `measured=True` when you do.
 SIGLIP_MODEL = ModelRequirement(
    repo="google/siglip-so400m-patch14-384",
    approx_download_bytes=3_500_000_000,
    approx_resident_bytes=4 * GIB,
    measured=False,
 )
@dataclass(frozen=True)
 class Lane:
    """A worker lane. `name` is the stable key the settings row is keyed on.
    Keyed on a lane NAME rather than a container hostname for the reason
    `models/service_seen.py` gives at length: celery's worker names here are
    `celery@<container id>` and are minted fresh on every deploy, so anything
    keyed on them records a death and a birth every time the stack updates.
    """
    name: str
    display_name: str
    queues: tuple[str, ...]
    # Which `entrypoint.sh` role starts this lane. NOT always the lane name:
    # `maintenance_long` is the plain `worker` role pointed at a different
    # queue, exactly as docker-compose starts it today (`command: ["worker"]`
    # with CELERY_QUEUES=maintenance_long). Recorded here so the generated
    # supervisord config and the compose file cannot disagree about it.
    entrypoint_role: str
    # THE cap a lane starts with — and, since 2026-09-23, the only number an
    # operator sets for it. How many workers actually run is the autoscaler's
    # job; this is the most it may use. Zero means the lane is off.
    #
    # One, and zero for ML. Deliberately far below the operator's own
    # production numbers, which are tuned for their hardware and are not a
    # sane first boot for a stranger — and low enough that a busy instance
    # tells them to raise it rather than quietly consuming the machine.
    default_slots_cap: int
    # True when a slot costs a copy of the ML model rather than just a process.
    # Such a lane is bounded by memory AS WELL AS by cores, never instead of.
    memory_bound: bool = False
    # CPU threads ONE slot uses. More than one for a lane whose work is an
    # inference library with its own thread pool: `services/ml/embedder.py`
    # calls `torch.set_num_threads` with this number, so a slot is four cores'
    # worth of demand rather than one process's.
    #
    # It lives here because the CEILING has to know it. It was a private
    # constant in the embedder with a comment saying "keep N_replicas x this
    # within the cores allotted to ML" — a rule stated where nothing could
    # enforce it. Nothing did: the ML ceiling was computed from memory alone,
    # so a large-memory host offered ~49 slots, the operator took them, and
    # 2026-09-23's log shows ~200 torch threads fighting over the box —
    # embeds at 107-246s each, and the daily CCIP sweep sharing that pool
    # timing out at 1800s.
    threads_per_slot: int = 1
    # Models this lane downloads the first time it is enabled. Empty for every
    # lane that needs none, which is how the UI knows whether to warn at all.
    models: tuple[ModelRequirement, ...] = ()
    # An optional lane is one the product works without. Shown as such, so
    # nobody turns on a multi-GB download believing it is required.
    optional: bool = False
    @property
    def queue_key(self) -> tuple[str, ...]:
        """The sorted queue set, which is how `service_seen` identifies a
        running worker. The join between what is configured here and what
        `celery inspect` reports."""
        return tuple(sorted(self.queues))
 # ONE CAP PER LANE, and that is the whole of what an operator sets.
 #
 # Operator, 2026-09-23: *"auto should be always on, not a setting, so that
 # idle instances quiet down when not running. the number that is visible and
 # something the user can tweak and manage should be the cap itself the number
 # of running workers is handled by the autoscaling function which is always
 # on."*
 #
 # Until then a lane had THREE operator values — `slots`, `slots_cap` and
 # `autoscale` — because the manual dial was built first (steps 2-4) and the
 # autoscaler arrived last (step 7) as an opt-in beside a control that already
 # existed. Nothing ever asked whether the dial should still exist once
 # something could move it automatically. It should not: "how many are running
 # right now" is a measurement, not a preference.
 #
 # One of each, and ML at zero. ML at zero is also rule 164's carve-out: a cap
 # of zero means no consumers, so a fresh install never loads a model or
 # reaches HuggingFace, and raising the cap is what triggers the fetch.
 #
 # These are far below the operator's own production numbers, and deliberately
 # so — they are what a stranger's first boot should do, not what a tuned
 # machine can. The UI is what closes that gap: a lane sitting at its cap with
 # a backlog says so, and says raising the cap is the fix. Without that a
 # conservative default is just a slow instance nobody knows how to speed up.
 LANES: tuple[Lane, ...] = (
    Lane(
        name="worker",
        display_name="Worker",
        queues=("default", "import", "thumbnail", "download"),
        entrypoint_role="worker",
        default_slots_cap=1,
    ),
    Lane(
        name="scheduler",
        display_name="Scheduler",
        queues=("maintenance", "scan"),
        entrypoint_role="scheduler",
        default_slots_cap=1,
    ),
    Lane(
        name="maintenance_long",
        display_name="Long maintenance",
        queues=("maintenance_long",),
        entrypoint_role="worker",
        default_slots_cap=1,
    ),
    Lane(
        name="ml",
        display_name="ML tagging",
        queues=("ml",),
        entrypoint_role="ml-worker",
        default_slots_cap=0,
        memory_bound=True,
        threads_per_slot=4,
        models=(SIGLIP_MODEL,),
        optional=True,
    ),
 )
 LANES_BY_NAME: dict[str, Lane] = {lane.name: lane for lane in LANES}
 LANES_BY_QUEUE_KEY: dict[tuple[str, ...], Lane] = {
    lane.queue_key: lane for lane in LANES
 }
 # --- what the container actually has -----------------------------------------
 # cgroup v2 first, then v1. A container started without an explicit memory
 # limit reports "max" on v2 and a sentinel near 2**63 on v1; both mean "no
 # limit", and the answer then is the host's RAM.
 _CGROUP_V2_MEMORY = Path("/sys/fs/cgroup/memory.max")
 _CGROUP_V1_MEMORY = Path("/sys/fs/cgroup/memory/memory.limit_in_bytes")
 _CGROUP_V2_CPU = Path("/sys/fs/cgroup/cpu.max")
 _CGROUP_V1_CPU_QUOTA = Path("/sys/fs/cgroup/cpu/cpu.cfs_quota_us")
 _CGROUP_V1_CPU_PERIOD = Path("/sys/fs/cgroup/cpu/cpu.cfs_period_us")
 # A v1 "unlimited" is PAGE_SIZE-aligned LONG_MAX, not a round number, so it is
 # recognised by magnitude rather than by equality. Anything claiming more than
 # a petabyte is a sentinel, not a machine.
 _UNLIMITED_ABOVE = 1 << 50
 # DERIVED from the model requirement above, never restated. The ceiling and
 # the number shown to the operator before they enable the lane have to be the
 # same figure, or the UI promises something the cap will then refuse.
 ML_BYTES_PER_SLOT = SIGLIP_MODEL.approx_resident_bytes
 # Held back for hypercorn and the non-ML lanes before any ML slot is offered.
 # In the consolidated container these share one cgroup with ML, and they are
 # the processes an OOM kill must not take (see the module docstring).
 RESERVED_BYTES = 2 * GIB
 # The smallest pool a lane can actually run: ONE process, never zero.
 #
 # billiard refuses to remove the last worker in a pool, so a lane asked to
 # shrink to nothing gets `ValueError("Can't shrink pool. All processes
 # busy!")` and the sizing pass re-sends the doomed message forever. Found on
 # the operator's live deploy, 2026-09-23.
 #
 # It is also what makes "off" expressible: a lane at cap 0 keeps this one
 # parked process with its consumers cancelled, so it still answers `inspect`
 # (and so reads as present rather than crashed), and `add_consumer` has
 # something to reach when the cap goes back up.
 #
 # Lives HERE rather than in `worker_control` because `gen_supervisord` needs
 # it at container boot and must not import the models package to get it.
 MIN_POOL_SLOTS = 1
 # The floor a cores-derived ceiling never goes below. A single-core box still
 # needs to be able to run its lanes; the ceiling exists to stop absurd values,
 # not to make a small machine unusable.
 MIN_CEILING = 1
 # How often `size_worker_lanes` runs — the beat schedule, and the freshness of
 # everything the System tab shows.
 #
 # It is here, in the import-light module, because three places have to agree
 # about it and they are in different packages: the beat entry in `celery_app`,
 # the sample the sweep writes (`worker_lane_sample`), and the roster's
 # staleness thresholds in `api/system_health`, which now depend on this sweep
 # rather than on a browser being open.
 #
 # 30s, down from 60s, because the sweep became the ONLY writer of the celery
 # roster on 2026-09-23. A part is called stale after 90s of silence, so a
 # 60-second sweep left one missed tick between "normal" and "everything is
 # yellow". That is the shape of lesson #4355 — a reader's threshold and an
 # emitter's cadence chosen in different files and never compared — and the
 # fix is headroom plus a test that asserts it, not a number that happens to
 # work today.
 #
 # The cost is one inspect every 30s instead of every 60s; the saving is every
 # inspect that used to run on a request path, which with a single tab open
 # was roughly four a minute against this two. Consequence worth knowing: the
 # pass also SHRINKS an idle lane by one slot per tick, so an idle lane now
 # gives its workers back twice as fast. That is the direction the operator
 # asked for — *"idle instances quiet down when not running"*.
 SWEEP_PERIOD_SECONDS = 30.0
 # What an unreadable limit yields. Low rather than unlimited, on purpose: not
 # knowing how much memory there is must never read as "plenty". An unswept
 # absence is not a verdict.
 UNKNOWN_CEILING = 1
 def _read_int(path: Path) -> int | None:
    try:
        raw = path.read_text().strip()
    except OSError:
        return None
    if raw == "max":
        return None
    try:
        return int(raw)
    except ValueError:
        return None
 def container_memory_bytes() -> int | None:
    """The memory this container may use, or None when it cannot be read.
    None means UNKNOWN, never UNLIMITED. Every caller must treat it as the
    conservative case — the whole point of the ceiling is to protect a machine
    whose size we are unsure of.
    """
    for path in (_CGROUP_V2_MEMORY, _CGROUP_V1_MEMORY):
        value = _read_int(path)
        if value is not None and value < _UNLIMITED_ABOVE:
            return value
        if value is not None:
            # A sentinel: the cgroup exists but sets no limit, so the real
            # bound is the host's.
            break
    try:
        return os.sysconf("SC_PHYS_PAGES") * os.sysconf("SC_PAGE_SIZE")
    except (ValueError, OSError, AttributeError):
        return None
 def container_cpu_count() -> int | None:
    """Effective cores, honouring a cgroup CPU quota.
    `os.cpu_count()` reports the HOST's cores from inside a container, so a
    quota of 2.0 on a 32-core host would otherwise offer 32 slots. The
    operator's own stack sets `cpus: '4.0'` on ml-worker, so this is a real
    configuration here and not a hypothetical.
    """
    quota: float | None = None
    try:
        raw = _CGROUP_V2_CPU.read_text().strip().split()
        if raw and raw[0] != "max":
            quota = int(raw[0]) / int(raw[1])
    except (OSError, ValueError, IndexError, ZeroDivisionError):
        pass
    if quota is None:
        q = _read_int(_CGROUP_V1_CPU_QUOTA)
        p = _read_int(_CGROUP_V1_CPU_PERIOD)
        if q is not None and p and q > 0:
            quota = q / p
    if quota is not None and quota > 0:
        return max(1, int(quota))
    return os.cpu_count()
 def lane_for_node(hostname: str) -> Lane | None:
    """`ml@7f3c9a1b` -> the ml lane. None for a node this build did not name.
    ## Why the node name, and not the queues it is consuming
    Because a lane that is OFF is consuming nothing, and "nothing" identifies
    no lane at all.
    Both the roster and `inspect_lanes_sync` used to map a worker to its lane
    through `active_queues()`. That is exact while the lane is running and
    useless the moment it is not: a lane at cap 0 has its consumers cancelled,
    so it answers the broadcast with an EMPTY queue list, matches no lane, and
    is dropped. Three things followed, and the operator saw all three at once
    on 2026-09-23:
    1. The lanes table showed the lane as **not answering** — which is the
       signal for a crashed worker, not for one the operator turned off.
    2. The roster grew a phantom row called **`Worker ()`**, the empty queue
       set rendered as a display name, "running" beside the real lane's row
       going stale.
    3. **The container went unhealthy.** `healthcheck._lanes_ok` requires
       every lane in the table to be present, and its docstring asserted the
       opposite of what the code did — *"a disabled lane still runs its
       process with its consumers cancelled, so it answers inspect and is
       healthy"*. It answers; it is not attributed. ML ships at cap 0, so a
       fresh install would have been permanently unhealthy, and Swarm
       restarts an unhealthy task forever.
    The node name survives all of that: `gen_supervisord` sets
    `CELERY_NODENAME={lane.name}` per program and the entrypoint passes it to
    `celery -n`, so the identity travels with the PROCESS rather than with
    what it happens to be doing. Falls back to the queue set for a deployment
    that sets no node name — the multi-service compose stack, where every node
    is `celery@<host>`.
    """
    return LANES_BY_NAME.get(hostname.split("@", 1)[0])
 def _cpu_bound_slots(lane: Lane) -> int:
    """How many slots this container's cores can feed, at `threads_per_slot`.
    Never zero: a machine with fewer cores than one slot wants still runs the
    lane, just slowly. That is a real trade an operator may want, and refusing
    to offer the lane at all on a small box would make ML unreachable there —
    unlike the memory bound, where the honest answer IS zero, because the
    first task would OOM the container rather than merely be slow.
    """
    cores = container_cpu_count()
    if cores is None:
        return UNKNOWN_CEILING
    return max(MIN_CEILING, cores // lane.threads_per_slot)
 def derived_ceiling(lane: Lane) -> int:
    """The most slots `lane` may be given on this container.
    Never stored. Recomputed on every read so a container whose limits changed
    is bounded by what it has NOW rather than by what it had when its row was
    written.
    """
    by_cpu = _cpu_bound_slots(lane)
    if not lane.memory_bound:
        return by_cpu
    total = container_memory_bytes()
    if total is None:
        log.warning(
            "worker_lanes: cannot read a memory limit; capping %s at %d",
            lane.name, UNKNOWN_CEILING,
        )
        return UNKNOWN_CEILING
    usable = total - RESERVED_BYTES
    if usable < ML_BYTES_PER_SLOT:
        # Honestly zero. A box that cannot hold one model alongside the web
        # process must be told it cannot run tagging, not sold a slot that
        # will OOM the container the first time it is used.
        return 0
    # BOTH bounds, whichever binds first. Memory alone was the whole answer
    # until 2026-09-23, and on a large-memory host that is the wrong one: RAM
    # said ~49 slots, and each of those slots wants `threads_per_slot` cores.
    # The operator raised the cap to what the dial offered and the lane
    # starved itself — a control is not allowed to offer a number the machine
    # cannot feed.
    return min(int(usable // ML_BYTES_PER_SLOT), by_cpu)
 def ceilings() -> dict[str, int]:
    """Every lane's ceiling, for the settings API and the UI."""
    return {lane.name: derived_ceiling(lane) for lane in LANES}
@@ -122,6 +122,28 @@ def prune_missing_file_records_task(self) -> dict:
    return {"checked": checked, "missing": len(missing_ids), "deleted": deleted}
@celery.task(
    name="backend.app.tasks.admin.repair_discord_downloads_task",
    bind=True,
    autoretry_for=(OperationalError, DBAPIError),
    retry_backoff=15, retry_backoff_max=180, max_retries=1,
    soft_time_limit=1800, time_limit=2400,  # 30 min / 40 min
 )
 def repair_discord_downloads_task(self, dry_run: bool = True) -> dict:
    """Clean re-download of the Discord files broken by the `None` naming
    (#3999). dry_run (the default) returns the projection; apply deletes the
    broken images and their files, makes gallery-dl forget every Discord
    download, and restarts every Discord source's backfill. Defaults to the SAFE
    preview because the apply deletes files. Operator-triggered only."""
    from ..services.discord_repair import repair_discord_downloads
    SessionLocal = _sync_session_factory()
    with SessionLocal() as session:
        return repair_discord_downloads(
            session, images_root=IMAGES_ROOT, dry_run=dry_run,
        )
@celery.task(
    name="backend.app.tasks.admin.dedup_videos_task",
    bind=True,
@@ -177,6 +177,10 @@ def download_source(self, source_id: int, _serialize_waits: int = 0) -> int:
                settings = ImportSettings.load_sync(sync_session)
                rate_limit = settings.download_rate_limit_seconds
                validate_files = settings.download_validate_files
                # How far back a tick keeps looking for EDITED posts. Read here
                # with the other downloader knobs, off the row this block is
                # already holding open.
                revisit_days = settings.download_revisit_days
            gdl = GalleryDLService(
                images_root=IMAGES_ROOT,
@@ -207,6 +211,7 @@ def download_source(self, source_id: int, _serialize_waits: int = 0) -> int:
                        # the walk). Same factory the importer's sync session
                        # comes from — a different DB connection per checkout.
                        sync_session_factory=SyncFactory,
                        revisit_days=revisit_days,
                    )
                    return await svc.download_source(source_id)
        finally:
@@ -473,6 +473,10 @@ def prune_task_runs() -> dict:
      (recover_stalled_task_runs) is the mechanism that flips them to
      terminal state; prune doesn't touch in-flight state.
    - 'retry' rows: treated as failures (>7d).
    - The NEWEST row of each task is never deleted, whatever its age: it is
      what the beat scheduler reads to know when a job last ran (#4408).
      Without it a weekly job's last run would be pruned after a day, and beat
      would think it had never run and fire it on every restart.
    Returns dict of how many rows were deleted in each bucket.
    """
@@ -480,16 +484,19 @@ def prune_task_runs() -> dict:
    now = datetime.now(UTC)
    ok_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_OK_SECONDS)
    fail_cutoff = now - timedelta(seconds=TASK_RUN_KEEP_FAILURE_SECONDS)
    newest = select(func.max(TaskRun.id)).group_by(TaskRun.task_name)
    with SessionLocal() as session:
        ok_deleted = session.execute(
            delete(TaskRun)
            .where(TaskRun.status == "ok")
            .where(TaskRun.finished_at < ok_cutoff)
            .where(TaskRun.id.not_in(newest))
        ).rowcount or 0
        fail_deleted = session.execute(
            delete(TaskRun)
            .where(TaskRun.status.in_(["error", "timeout", "retry"]))
            .where(TaskRun.finished_at < fail_cutoff)
            .where(TaskRun.id.not_in(newest))
        ).rowcount or 0
        session.commit()
        return {"ok_deleted": ok_deleted, "failures_deleted": fail_deleted}
@@ -502,10 +509,16 @@ def prune_task_runs() -> dict:
    soft_time_limit=1800, time_limit=2100,
 )
 def backfill_phash() -> int:
-    """Recompute phash for stored images that have none (imported before
+    """Recompute phash for stored images that have none. Keyset-paginated by
-    FC-2d-i+ii). Keyset-paginated by id (restart-safe), NULL-only fill,
+    id (restart-safe), NULL-only fill, idempotent. Videos legitimately keep
-    idempotent. Videos legitimately keep phash NULL. A missing/unreadable
+    phash NULL. A missing/unreadable file is logged and left NULL — never
-    file is logged and left NULL — never fails the task."""
+    fails the task.
    Two sources of NULLs: images imported before FC-2d-i+ii, and migration
    0098, which cleared every phash so the library could be re-hashed at
    hash_size=16 (#4223). The daily beat entry exists for the second — until
    a row is refilled it takes no part in dedup, which is why this runs on a
    schedule rather than waiting for a deep scan."""
    SessionLocal = _sync_session_factory()
    updated = 0
    last_id = 0
@@ -1169,7 +1182,7 @@ def group_discord_drops() -> str:
        return "disabled"
    return (
        f"sources={res['sources']} created={res['posts_created']} "
-        f"joined={res['images_joined']}"
+        f"joined={res['images_joined']} merged={res['drops_merged']}"
    )
@@ -1178,12 +1191,16 @@ def group_discord_drops() -> str:
    soft_time_limit=900, time_limit=1200,
 )
 def match_post_associations() -> str:
-    """Milestone 388 E5: propose which Patreon post announced which Discord drop.
+    """Milestone 388 E5: which Patreon post announced which Discord drop.
-    Proposes only — every pair lands in a review queue and nothing is linked
+    A CONCLUSIVE pair — one the creator's own working name identifies, where
-    until the operator accepts. Maintenance lane for the same reason as the
+    that name appears in these two posts and nowhere else in their library — is
-    grouper: no inference, no ML library, and it must not depend on the
+    linked outright when `discord_link_auto` is on, because there is nothing
-    optional ml-worker being present.
+    there for the operator to adjudicate. Everything weaker lands in the review
    queue and stays unlinked until they accept it.
    Maintenance lane for the same reason as the grouper: no inference, no ML
    library, and it must not depend on the optional ml-worker being present.
    """
    import asyncio
@@ -1247,6 +1264,7 @@ def sync_memberships() -> str:
    from ..services.artist_membership_service import rescan as membership_rescan
    from ..services.credential_crypto import CredentialCrypto
    from ..services.credential_service import CredentialService
    from ..services.membership_reconcile import apply_membership_lapses
    from ..services.membership_roster import roster_user_id, sync_platform
    from ..services.patreon_client import PatreonClient
    from ..services.subscribestar_client import SubscribeStarClient
@@ -1298,9 +1316,18 @@ def sync_memberships() -> str:
                    )
                async with async_factory() as session:
-                    results.append(
+                    result = await sync_platform(session, platform=platform, fetch=fetch)
-                        await sync_platform(session, platform=platform, fetch=fetch)
+                results.append(result)
-                    )
+
                # #3995: stop pulling sources whose paid access has ended, and
                # resume the ones this stopped once they are paid again. Only
                # right after a successful sync, so it always acts on the roster
                # just written, never on a stale one.
                if result.get("ok"):
                    async with async_factory() as session:
                        result["lapses"] = await apply_membership_lapses(
                            session, platform=platform,
                        )
            # #388 E4: offer the freshly-synced roster to the artists FC already
            # tracks. Chained here rather than given its own beat entry because
@@ -1321,9 +1348,100 @@ def sync_memberships() -> str:
        if "skipped" in r:
            parts.append(f"{r['platform']}=skipped({r['skipped']})")
        elif r.get("ok"):
-            parts.append(f"{r['platform']}={r['count']}")
+            lapses = r.get("lapses") or {}
            detail = (
                f"(stopped={lapses['stopped']},resumed={lapses['resumed']})"
                if lapses.get("stopped") or lapses.get("resumed") else ""
            )
            parts.append(f"{r['platform']}={r['count']}{detail}")
        else:
            parts.append(f"{r['platform']}=FAILED({r['error']})")
    if res.get("suggested") is not None:
        parts.append(f"suggested={res['suggested']}")
    return " ".join(parts) or "no platforms"
@celery.task(name="backend.app.tasks.maintenance.size_worker_lanes")
 def size_worker_lanes() -> dict:
    """Size every lane to its backlog, within the cap the operator set.
    ONE sweep, replacing `reconcile_worker_lanes` and `autoscale_worker_lanes`
    (2026-09-23). They were two enforcers over one number: the reconcile drove
    the pool to a stored `slots`, the autoscaler moved it away from that same
    value, and most of the autoscaler's design existed to keep the reconcile
    from undoing its work. Deleting the stored number deletes the conflict.
    It still does what the reconcile existed for. `pool_grow` is not durable —
    a worker restarted by its supervisor comes back at its ENV concurrency,
    silently below what the lane should run — and this reads the LIVE pool
    every minute, so that worker is corrected on the next tick rather than
    after five.
    Returns every lane's outcome INCLUDING the ones it held, each with a
    reason. A pass that only speaks when it acts cannot be debugged on the day
    it does not.
    ## It is also the only thing that MEASURES, since 2026-09-23
    It always inspected the broker to decide pool sizes, and then threw the
    reading away — while `/api/system/workers` ran the same inspect on every
    page load and the System tab polls it four times a minute. Operator:
    *"there is a repull every time this page loads — is there a reason this
    info isn't being tracked in the background and stored in some way?"*
    So one inspect now feeds three things: the sizing decision, the stored
    sample the System tab reads, and the celery roster. No request path
    touches the broker any more.
    Order matters. The sample is stored BEFORE the roster refresh, because
    that refresh does its own broadcast and a broker that has just started
    failing must not cost us the reading we already have.
    """
    from ..models import WorkerLane
    from ..services.service_roster import refresh_celery_roster_sync
    from ..services.worker_control import (
        _queue_depths_sync,
        inspect_lanes_sync,
        size_lanes_sync,
        store_lane_samples_sync,
    )
    # Read INSIDE the session. Reading a column off a detached instance
    # happens to work while the attribute is still loaded and stops working
    # the moment anything expires it — a failure that would appear long after
    # this line, in a sweep nobody is watching.
    with _sync_session_factory()() as session:
        caps = {
            row.name: row.slots_cap
            for row in session.execute(select(WorkerLane)).scalars()
        }
    if not caps:
        # Migration 0103/0105 seed these, so an empty table means they have
        # not run yet. Nothing to assert — and inventing defaults here would
        # let this task disagree with the seed it is meant to be enforcing.
        return {"sized": []}
    # Measured ONCE, here, and then used three times. Passing them down is
    # what makes the reading keepable rather than an implementation detail of
    # a function that returns decisions.
    live = inspect_lanes_sync()
    depths = _queue_depths_sync()
    sized = size_lanes_sync(caps, live=live, depths=depths)
    with _sync_session_factory()() as session:
        store_lane_samples_sync(session, live, depths)
        refresh_celery_roster_sync(session)
    for d in sized:
        if d.action not in ("held", "skipped"):
            log.info(
                "worker lanes: %s %s to %s slots — %s",
                d.lane, d.action, d.slots, d.reason,
            )
    return {
        "sized": [
            {"lane": d.lane, "action": d.action, "slots": d.slots,
             "reason": d.reason}
            for d in sized
        ],
    }
@@ -488,7 +488,7 @@ def scheduled_ccip_auto_apply() -> str:
    from ..models import ImageRegion, MLSettings, Tag, TagKind
    from ..models.tag import image_tag
-    from ..services.ml.ccip import _FIGURE_KINDS
+    from ..services.ml.ccip import _FIGURE_KINDS, char_maxima
    from ..services.ml.training_data import _applied_or_rejected, _l2norm
    SessionLocal = _sync_session_factory()
@@ -553,11 +553,22 @@ def scheduled_ccip_auto_apply() -> str:
            by_img: dict[int, list] = {}
            for iid, vec in rows:
                by_img.setdefault(iid, []).append(vec)
-            for iid, vecs in by_img.items():
+            if not by_img:
-                q = _l2norm(np.asarray(vecs, dtype=np.float32), np)  # (nq, 768)
+                continue
-                colmax = (q @ allref.T).max(axis=0)              # (total,)
+
-                charmax = np.maximum.reduceat(colmax, seg)       # (n_chars,)
+            # One matmul per BLOCK of figures, not one per image. This loop ran
-                for ci in np.where(charmax >= thr)[0]:
+            # over every image in the library on every daily run and did a
            # matmul too small to pay for itself each time; it hit the 1800s
            # soft limit on the operator's instance on 2026-09-23. Same
            # arithmetic — see `char_maxima`.
            iids = list(by_img)
            charmax = char_maxima(
                [_l2norm(np.asarray(by_img[i], dtype=np.float32), np) for i in iids],
                allref, seg, np,
            )                                                    # (n_img, n_chars)
            for row, iid in enumerate(iids):
                for ci in np.where(charmax[row] >= thr)[0]:
                    t = ref_tags[int(ci)]
                    if iid in skip[t]:
                        continue
@@ -668,3 +679,31 @@ def scheduled_retract_auto_tags() -> str:
    with SessionLocal() as session:
        n_ccip = retract_auto_applied_ccip(session)
    return f"head={n_head} ccip={n_ccip}"
@celery.task(name="backend.app.tasks.ml.ensure_models", bind=True)
 def ensure_models(self) -> dict:
    """Fetch the models this lane needs, if they are not already present.
    Milestone 422 step 6. This used to run in `entrypoint.sh` before celery
    started, which made every boot of the ML role reach HuggingFace for
    ~3.5GB — a startup dependency on a third party, for a feature the operator
    may never use. Rule 164 permits a runtime fetch only for something
    "optional and clearly off", so it moved here: enqueued the moment the lane
    is ENABLED, never at boot.
    Being a task rather than a startup step is what makes it visible: it gets
    a TaskRun row like any other, so the download shows in Activity with a
    duration and a status, and a failure is something the operator can see and
    retry rather than a container that quietly never became useful.
    Idempotent — `download_models` fetches only what is missing — so enabling
    an already-provisioned lane costs one no-op task rather than a re-download.
    That matters because the reconcile may enqueue it again.
    """
    from ..scripts.download_models import main as download
    rc = download()
    if rc != 0:
        raise RuntimeError(f"model download failed with exit code {rc}")
    return {"ok": True}
@@ -60,6 +60,37 @@ def derive_subdir(source_path: Path, import_root: Path) -> str:
    return str(rel) if str(rel) != "." else ""
 def canonical_subdir(subdir: str, artist_slug: str | None) -> str:
    """`subdir` with its TOP-LEVEL segment replaced by the artist's slug.
        canonical_subdir("Conto/patreon", "conto")  -> "conto/patreon"
        canonical_subdir("Conto", "conto")          -> "conto"
        canonical_subdir("Conto/patreon", None)     -> "Conto/patreon"
        canonical_subdir("", "conto")               -> ""
    `derive_subdir` mirrors the IMPORT tree's folder names verbatim, so a
    filesystem import out of `/import/Conto/...` used to write
    `<images_root>/Conto/...` while the download path wrote `<root>/conto/...`
    for the very same Artist row. One artist, two directories, forever — 57
    such families had accumulated by 2026-09-21, and the database never had
    duplicate artists at all (milestone #421).
    The slug is the canonical name because it is the Artist row's own
    identifier: it is what `/api/artists` reports, what the ingesters already
    write, and the one spelling that cannot vary with how a folder happened to
    be capitalised on the way in.
    Two deliberate pass-throughs. NO artist resolved means there is nothing
    authoritative to canonicalise against, and an EMPTY subdir is a file
    landing at the images root — those have no artist folder to correct, and
    what becomes of them is its own decision (task #4247), not a side effect
    of this helper.
    """
    if not artist_slug or not subdir:
        return subdir
    return str(Path(artist_slug, *Path(subdir).parts[1:]))
 def hash_suffixed_name(stem: str, sha256_hex: str, ext: str) -> str:
    """Builds 'stem__<first10ofhash><ext>'.
--- a/Show More
+++ b/Show More