feat(agent): survive + auto-recover when curator is unreachable

For redeploying curator while away with nobody to restart the agent:

- _process now distinguishes a TRANSPORT error (curator down/redeploying, 5xx,
  401/403/408/409/429, or our lease reclaimed mid-flight) from a genuine job
  fault. On a transport error it hands the job back (best effort) and signals
  the loop to back off — instead of calling fail(), which would burn the job's
  server-side attempt budget (MAX_ATTEMPTS=3) and permanently error good jobs
  across a redeploy. Job-specific 4xx (404 image gone) still fail so they don't
  re-lease forever.
- lease loop retries with capped exponential backoff (poll_idle → 60s) and
  resets on the first successful lease, so a long outage is gentle and recovery
  is automatic within ≤60s of curator returning. Sleeps are interruptible so
  Stop / pool-shrink stays responsive.
- AUTO_START env (default on in compose) resumes the worker on container start,
  so a host reboot / crash-restart (restart: unless-stopped) self-heals with
  nobody at the desktop.
- control UI shows a "waited out" counter + an "curator unreachable, holding
  work" banner so the recovering state reads as recovery, not failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

.forgejo/workflows/build.yml

@@ -329,3 +329,41 @@ jobs:
           file: Dockerfile.ml
           push: true
           tags: ${{ steps.tag.outputs.tags }}
+
+  # The desktop GPU agent (#114) — published so the operator pulls + runs it on
+  # the GPU machine instead of building locally. Independent of web/ml (its own
+  # CUDA + onnxruntime-gpu image, context = agent/). Same tag cadence.
+  build-agent:
+    runs-on: python-ci
+    container:
+      image: git.fabledsword.com/bvandeusen/ci-python:3.14
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Determine tag
+        id: tag
+        run: |
+          SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
+          if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
+            TAG_NAME="${GITHUB_REF#refs/tags/}"
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:${TAG_NAME}" >> "$GITHUB_OUTPUT"
+          elif [ "${GITHUB_REF##*/}" = "main" ]; then
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:main,git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
+          else
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Login to Forgejo registry
+        uses: docker/login-action@v3
+        with:
+          registry: git.fabledsword.com
+          username: ${{ github.actor }}
+          password: ${{ secrets.RELEASE_TOKEN }}
+
+      - name: Build and push agent image
+        uses: docker/build-push-action@v5
+        with:
+          context: agent
+          file: agent/Dockerfile
+          push: true
+          tags: ${{ steps.tag.outputs.tags }}

Dockerfile

@@ -18,13 +18,16 @@ ENV PYTHONUNBUFFERED=1 \
 
 # System deps: ffmpeg (transcode + thumbnails, FC-2), unar (archives, FC-2),
 # libpq for psycopg, postgresql-client + zstd for FC-5 backup/restore
-# (pg_dump + tar --zstd), image libs.
+# (pg_dump + tar --zstd), image libs, megatools (mega.nz public-link downloads
+# for off-platform file-host links, #830 — `megatools dl`; Debian-native, no
+# external MEGA apt repo needed).
 RUN apt-get update && apt-get install -y --no-install-recommends \
         ffmpeg \
         unar \
         libpq5 \
         postgresql-client \
         zstd \
+        megatools \
         libjpeg62-turbo \
         libwebp7 \
         libpng16-16 \

agent/Dockerfile

@@ -0,0 +1,27 @@
+# FabledCurator GPU agent — runs on the desktop with the GPU.
+# CUDA + cuDNN runtime so onnxruntime-gpu can use the card (it needs cuDNN 9 —
+# the plain -runtime image lacks it: "libcudnn.so.9: cannot open shared object
+# file"); ffmpeg for video frames.
+FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
+
+ENV DEBIAN_FRONTEND=noninteractive PYTHONUNBUFFERED=1
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends python3 python3-pip ffmpeg \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+# torch from the CUDA-12.4 wheel index (matches the base image); its wheels
+# bundle their own CUDA + cuDNN and coexist with onnxruntime-gpu. Installed
+# first + separately so the GPU build of torch is deterministic and layer-cached.
+RUN pip3 install --no-cache-dir torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
+COPY requirements.txt .
+RUN pip3 install --no-cache-dir -r requirements.txt
+COPY fc_agent ./fc_agent
+
+# imgutils ONNX models + the transformers SigLIP weights both cache here; mount
+# a volume to persist them across restarts (the SigLIP download is ~3.5 GB once).
+ENV HF_HOME=/models
+EXPOSE 8770
+
+# The control UI; the worker is started from it (or POST /start).
+CMD ["uvicorn", "fc_agent.app:app", "--host", "0.0.0.0", "--port", "8770"]

agent/README.md

@@ -0,0 +1,71 @@
+# FabledCurator GPU agent
+
+A desktop-GPU worker that embeds characters (CCIP) + figure crops for
+FabledCurator. It talks to FC **only over HTTP** — it leases jobs, fetches image
+pixels, runs the models on your GPU, and posts results back. Your FC database and
+Redis stay private; the agent never touches them.
+
+You run it when you want a burst and stop it to reclaim the card.
+
+## 0. Host prerequisite — NVIDIA Container Toolkit
+Docker needs the toolkit to hand the GPU to a container (else: *"could not select
+device driver nvidia with capabilities [[gpu]]"*). On Arch/CachyOS:
+```sh
+sudo pacman -S nvidia-container-toolkit
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+# verify:
+docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
+```
+
+## 1. Get a token
+In FC: **Settings → Tagging → GPU agent → Generate token** (or Rotate). Copy it.
+
+## 2. Pull (CI publishes it alongside the web/ml images)
+```sh
+docker pull git.fabledsword.com/bvandeusen/fabledcurator-agent:latest
+```
+> Local build for development instead: `docker build -t fc-gpu-agent agent/`
+
+## 3. Run (on the machine with the GPU)
+```sh
+docker run --rm --gpus all -p 8770:8770 \
+  -e FC_URL=http://curator.traefik.internal \
+  -e FC_TOKEN=<paste-the-token> \
+  -v fc-agent-models:/models \
+  git.fabledsword.com/bvandeusen/fabledcurator-agent:latest
+```
+Then open <http://localhost:8770> — the control page. Click **Start** to begin
+draining the queue; **Pause**/**Stop** to yield the GPU. The `-v fc-agent-models`
+volume caches the downloaded ONNX models so restarts are fast.
+
+Kick off a backfill from FC (**GPU agent card → Queue character embedding**), then
+watch the queue counts on the control page (or FC's card) drain.
+
+## Config (env)
+| var | default | meaning |
+|---|---|---|
+| `FC_URL` | `http://localhost:8000` | FC base URL |
+| `FC_TOKEN` | — | the bearer token (required) |
+| `AGENT_ID` | `desktop-agent` | identifies this agent's leases |
+| `BATCH_SIZE` | `4` | jobs leased per round (still processed one at a time) |
+| `CCIP_MODEL` | imgutils default | CCIP model name |
+| `DETECTOR_LEVEL` | `m` | person-detector size: `n` < `s` < `m` < `x` |
+| `POLL_IDLE_SECONDS` | `10` | wait between empty leases |
+
+## ⚠️ Verify on first run
+This part can't be CI-tested (no GPU/models in CI), so confirm against your
+installed `dghs-imgutils` (`pip show dghs-imgutils`) — see `fc_agent/models.py`:
+- `imgutils.detect.detect_person(image, level=...)` returns
+  `[((x0,y0,x1,y1), label, score), ...]`.
+- `imgutils.metrics.ccip_extract_feature(image, model=...)` returns a vector
+  (768-d for caformer). If you want the F1-0.94 variant, set
+  `CCIP_MODEL=ccip-caformer_b36-24` (verify the exact string in imgutils).
+
+If FC's matcher under/over-fires, tune the cosine threshold in
+`backend/app/services/ml/ccip.py` (`DEFAULT_SIM_THRESHOLD`) and use
+`GET /api/ccip/overview` + `/api/ccip/images/<id>` to spot-check.
+
+## CPU fallback
+Swap `onnxruntime-gpu` → `onnxruntime` in `requirements.txt` and drop `--gpus all`
+to grind it slowly on the server instead. Same agent, no card.

agent/docker-compose.yml

@@ -0,0 +1,53 @@
+# FabledCurator GPU agent — desktop run via docker compose.
+#
+# Usage:
+#   1. Generate a token: FC → Settings → Tagging → GPU agent → Generate token.
+#   2. Create a .env next to this file:
+#        FC_URL=http://curator.traefik.internal
+#        FC_TOKEN=<paste-the-token>
+#        # optional: CCIP_MODEL=ccip-caformer_b36-24   (the F1-0.94 variant)
+#   3. docker compose up -d     (pulls the published image)
+#   4. Open http://localhost:8770 → Start. Pause/Stop hands the GPU back.
+#      docker compose down       to stop the container entirely.
+#
+# Surviving a curator redeploy (you're away, can't touch the agent):
+#   - A running agent rides out curator being unreachable on its own — it retries
+#     leasing with capped backoff and resumes when the server is back. In-flight
+#     work is handed back (not failed), so a redeploy never poisons good jobs.
+#   - AUTO_START=1 (below) also resumes the worker if the AGENT container itself
+#     restarts (host reboot / crash via `restart: unless-stopped`) — no click.
+#
+# Needs the NVIDIA Container Toolkit installed on the host for --gpus.
+
+services:
+  fc-gpu-agent:
+    image: git.fabledsword.com/bvandeusen/fabledcurator-agent:latest
+    pull_policy: always
+    ports:
+      - "8770:8770"
+    environment:
+      FC_URL: ${FC_URL:-http://curator.traefik.internal}
+      FC_TOKEN: ${FC_TOKEN:?set FC_TOKEN in .env (FC → GPU agent → Generate token)}
+      CCIP_MODEL: ${CCIP_MODEL:-}
+      DETECTOR_LEVEL: ${DETECTOR_LEVEL:-m}
+      BATCH_SIZE: ${BATCH_SIZE:-4}
+      # Resume the worker automatically on container start (survive a reboot /
+      # crash-restart while you're away). Set to 0 to require a manual Start.
+      AUTO_START: ${AUTO_START:-1}
+      # Crop embedder (SigLIP concept bag): float16 keeps VRAM low on a shared
+      # desktop GPU; the model itself is announced by the server.
+      SIGLIP_DTYPE: ${SIGLIP_DTYPE:-float16}
+    volumes:
+      # Persist the downloaded ONNX models so restarts are fast.
+      - fc-agent-models:/models
+    restart: unless-stopped
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+volumes:
+  fc-agent-models:

agent/fc_agent/app.py

@@ -0,0 +1,134 @@
+"""FastAPI control surface for the agent (served on localhost).
+
+Start / stop the worker pool, tune the worker count live (trades desktop
+responsiveness for throughput), and watch GPU load + progress + the server-side
+queue. Config is env-seeded; the worker count is adjustable here on the fly.
+"""
+from fastapi import FastAPI, Request
+from fastapi.responses import HTMLResponse, JSONResponse
+
+from .config import Config
+from .gpu import read_gpu
+from .worker import Worker
+
+cfg = Config.from_env()
+worker = Worker(cfg)
+app = FastAPI(title="FabledCurator GPU agent")
+
+
+@app.on_event("startup")
+def _maybe_autostart() -> None:
+    # With AUTO_START set, a container restart (host reboot, or `restart:
+    # unless-stopped` after a crash) resumes the worker on its own — the slots
+    # then ride out a still-down curator via lease backoff. Lets the agent
+    # survive a redeploy with nobody at the desktop to click Start.
+    if cfg.auto_start and cfg.token:
+        worker.start()
+
+
+@app.get("/", response_class=HTMLResponse)
+def index() -> str:
+    return _PAGE
+
+
+@app.post("/start")
+def start():
+    worker.start()
+    return JSONResponse(worker.status())
+
+
+@app.post("/stop")
+def stop():
+    worker.stop()
+    return JSONResponse(worker.status())
+
+
+@app.post("/concurrency")
+async def concurrency(request: Request):
+    body = await request.json()
+    worker.set_concurrency(int(body.get("value", 1)))
+    return JSONResponse(worker.status())
+
+
+@app.get("/status")
+def status():
+    s = worker.status()
+    s["fc_url"] = cfg.fc_url
+    s["configured"] = bool(cfg.token)
+    s["gpu"] = read_gpu()
+    try:
+        s["queue"] = worker.client.queue_status()
+    except Exception:
+        s["queue"] = None
+    return JSONResponse(s)
+
+
+_PAGE = """<!doctype html><html><head><meta charset=utf-8>
+<title>FabledCurator GPU agent</title>
+<style>
+ body{font:14px system-ui;margin:2rem;max-width:680px;background:#14171a;color:#e8e8e8}
+ h1{font-size:18px} button{font:14px system-ui;padding:.5rem 1rem;border:0;border-radius:6px;
+ margin-right:.5rem;cursor:pointer;color:#fff} .start{background:#2e7d32}.stop{background:#b3261e}
+ .step{background:#33373b;padding:.4rem .7rem;font-weight:700}
+ .stat{display:inline-block;margin-right:1.5rem;vertical-align:top}
+ .n{font-size:22px;font-weight:700} code{background:#222;padding:2px 6px;border-radius:4px}
+ .q,.gpu{margin-top:1rem;color:#9aa} .bar{height:8px;border-radius:4px;background:#222;overflow:hidden;
+ max-width:320px;margin-top:4px} .bar>i{display:block;height:100%;background:#3f7d3f}
+ .row{margin:.8rem 0}
+</style></head><body>
+<h1>FabledCurator GPU agent</h1>
+<p>FC: <code id=fc>—</code> · token <code id=cfg>—</code></p>
+<div class=row>
+ <button class=start onclick=act('start')>Start</button>
+ <button class=stop onclick=act('stop')>Stop</button>
+</div>
+<div class=row>
+ workers
+ <button class=step onclick=setc(-1)>−</button>
+ <input id=conc type=number min=1 value=1
+  style="width:3.5rem;font:700 16px system-ui;text-align:center;background:#222;color:#e8e8e8;border:1px solid #444;border-radius:6px;padding:.3rem"
+  onchange="setv(this.value)">
+ <button class=step onclick=setc(1)>+</button>
+ <span class=cap style=color:#9aa>(more = overlap I/O, fill the GPU) max <b id=capn>8</b></span>
+</div>
+<div class=row>
+ <span class=stat><span class=n id=state>stopped</span><br>state</span>
+ <span class=stat><span class=n id=active>0</span><br>active now</span>
+ <span class=stat><span class=n id=done>0</span><br>processed</span>
+ <span class=stat><span class=n id=err>0</span><br>errors</span>
+ <span class=stat><span class=n id=wait>0</span><br>waited out</span>
+</div>
+<div id=banner style="display:none;margin:.6rem 0;padding:.5rem .8rem;border-radius:6px;background:#5a4a17;color:#ffe28a">
+ curator unreachable — holding work + retrying, will resume on its own (no restart needed)
+</div>
+<div class=gpu id=gpu>GPU — …</div>
+<div class=bar><i id=gpubar style=width:0%></i></div>
+<div class=q id=queue></div>
+<script>
+ let CAP=8
+ async function act(p){await fetch('/'+p,{method:'POST'});refresh()}
+ function setc(d){ setv((parseInt(conc.value||'1'))+d) }
+ async function setv(v){
+  v=Math.max(1,Math.min(CAP,parseInt(v)||1)); conc.value=v
+  await fetch('/concurrency',{method:'POST',headers:{'Content-Type':'application/json'},
+   body:JSON.stringify({value:v})});refresh()
+ }
+ async function refresh(){
+  const s=await (await fetch('/status')).json()
+  CAP=s.max_concurrency||8; capn.textContent=CAP
+  state.textContent=s.state; active.textContent=s.active; done.textContent=s.processed
+  err.textContent=s.errors; fc.textContent=s.fc_url; wait.textContent=s.transient||0
+  // Running but the queue read failed → curator is unreachable; show we're
+  // riding it out rather than erroring.
+  banner.style.display=(s.state==='running' && !s.queue)?'block':'none'
+  if(document.activeElement!==conc) conc.value=s.concurrency
+  conc.max=CAP
+  cfg.textContent=s.configured?'set':'MISSING'
+  if(s.gpu){
+   gpu.textContent=`GPU — ${s.gpu.util_pct}% util · VRAM ${s.gpu.mem_used_mb}/${s.gpu.mem_total_mb} MB · ${s.gpu.temp_c}°C`
+   gpubar.style.width=Math.round(100*s.gpu.mem_used_mb/s.gpu.mem_total_mb)+'%'
+  } else { gpu.textContent='GPU — n/a (CPU fallback?)'; gpubar.style.width='0%' }
+  queue.textContent=s.queue?`queue — pending ${s.queue.pending} · in flight ${s.queue.leased} · done ${s.queue.done} · errored ${s.queue.error}`:'queue — unreachable'
+ }
+ refresh(); setInterval(refresh,3000)
+</script></body></html>"""

agent/fc_agent/client.py

@@ -0,0 +1,85 @@
+"""HTTP client for the FabledCurator GPU-job API.
+
+The agent's ONLY contact with FC — lease/submit/heartbeat/fail + fetch image
+bytes, all over HTTP with the bearer token. No DB/Redis.
+"""
+import requests
+from requests.adapters import HTTPAdapter
+
+
+class FcClient:
+    def __init__(self, base_url: str, token: str, agent_id: str):
+        self.base = base_url.rstrip("/")
+        self.agent_id = agent_id
+        self.s = requests.Session()
+        self.s.headers["Authorization"] = f"Bearer {token}"
+        # Many worker threads share this Session; the default pool (10) would
+        # throttle them + spam "connection pool is full". Size it for the cap.
+        adapter = HTTPAdapter(pool_connections=64, pool_maxsize=64)
+        self.s.mount("http://", adapter)
+        self.s.mount("https://", adapter)
+
+    def lease(self, batch_size: int) -> list[dict]:
+        r = self.s.post(
+            f"{self.base}/api/gpu/jobs/lease",
+            json={"agent_id": self.agent_id, "batch_size": batch_size},
+            timeout=30,
+        )
+        r.raise_for_status()
+        return r.json().get("jobs", [])
+
+    def submit(self, job_id: int, regions: list[dict], replace_kinds: list[str]) -> dict:
+        r = self.s.post(
+            f"{self.base}/api/gpu/jobs/submit",
+            json={
+                "agent_id": self.agent_id, "job_id": job_id,
+                "regions": regions, "replace_kinds": replace_kinds,
+            },
+            timeout=120,
+        )
+        r.raise_for_status()
+        return r.json()
+
+    def heartbeat(self, job_ids: list[int]) -> None:
+        try:
+            self.s.post(
+                f"{self.base}/api/gpu/jobs/heartbeat",
+                json={"agent_id": self.agent_id, "job_ids": job_ids},
+                timeout=30,
+            )
+        except requests.RequestException:
+            pass
+
+    def fail(self, job_id: int, error: str) -> None:
+        try:
+            self.s.post(
+                f"{self.base}/api/gpu/jobs/fail",
+                json={"agent_id": self.agent_id, "job_id": job_id, "error": error},
+                timeout=30,
+            )
+        except requests.RequestException:
+            pass
+
+    def release(self, job_ids: list[int]) -> None:
+        # Graceful hand-back on stop so orphaned work is re-leased at once.
+        if not job_ids:
+            return
+        try:
+            self.s.post(
+                f"{self.base}/api/gpu/jobs/release",
+                json={"agent_id": self.agent_id, "job_ids": job_ids},
+                timeout=30,
+            )
+        except requests.RequestException:
+            pass
+
+    def fetch_image(self, image_url: str) -> bytes:
+        # image_url is a server-relative path ("/images/...").
+        r = self.s.get(f"{self.base}{image_url}", timeout=180)
+        r.raise_for_status()
+        return r.content
+
+    def queue_status(self) -> dict:
+        r = self.s.get(f"{self.base}/api/gpu/status", timeout=15)
+        r.raise_for_status()
+        return r.json()

agent/fc_agent/config.py

@@ -0,0 +1,36 @@
+"""Agent config, all from env (the control container is configured at run)."""
+import os
+from dataclasses import dataclass
+
+
+@dataclass
+class Config:
+    fc_url: str          # base URL of the FabledCurator web service
+    token: str           # the bearer token from Settings → Tagging → GPU agent
+    agent_id: str        # identifies this agent's leases
+    batch_size: int      # jobs a worker leases per round
+    concurrency: int     # INITIAL parallel workers (tunable live from the UI)
+    ccip_model: str      # imgutils CCIP model name ("" → imgutils default)
+    detector_level: str  # imgutils person-detector level: n|s|m|x
+    poll_idle_seconds: float   # wait between empty leases
+    embed_dtype: str     # torch dtype for the crop embedder: float16|float32
+    embed_model_override: str  # force a SigLIP-family model ("" → use the one
+    #                            the server announces in the lease)
+    auto_start: bool     # start the worker pool on boot (so a container restart
+    #                      resumes processing without anyone clicking Start)
+
+    @classmethod
+    def from_env(cls) -> "Config":
+        return cls(
+            fc_url=os.environ.get("FC_URL", "http://localhost:8000").rstrip("/"),
+            token=os.environ.get("FC_TOKEN", ""),
+            agent_id=os.environ.get("AGENT_ID", "desktop-agent"),
+            batch_size=int(os.environ.get("BATCH_SIZE", "4")),
+            concurrency=int(os.environ.get("CONCURRENCY", "1")),
+            ccip_model=os.environ.get("CCIP_MODEL", ""),
+            detector_level=os.environ.get("DETECTOR_LEVEL", "m"),
+            poll_idle_seconds=float(os.environ.get("POLL_IDLE_SECONDS", "10")),
+            embed_dtype=os.environ.get("SIGLIP_DTYPE", "float16"),
+            embed_model_override=os.environ.get("EMBED_MODEL_NAME", ""),
+            auto_start=os.environ.get("AUTO_START", "").lower() in ("1", "true", "yes"),
+        )

agent/fc_agent/crops.py

@@ -0,0 +1,36 @@
+"""Crop primitive — vendored from backend/app/services/ml/crops.py so the agent
+is self-contained. Keep in sync if the floor logic changes."""
+from PIL import Image
+
+MIN_CROP_FRACTION = 0.10
+MIN_CROP_PX = 64
+
+
+def crop_region(
+    img: Image.Image,
+    bbox: tuple[float, float, float, float],
+    *,
+    pad: float = 0.0,
+    min_fraction: float = MIN_CROP_FRACTION,
+    min_px: int = MIN_CROP_PX,
+) -> Image.Image | None:
+    """Crop a NORMALIZED bbox (x, y, w, h in [0,1]); None if below the size
+    floor (max of a fraction-of-short-side and an absolute pixel floor)."""
+    iw, ih = img.size
+    x, y, w, h = bbox
+    px, py, pw, ph = x * iw, y * ih, w * iw, h * ih
+    if pad:
+        px -= pw * pad / 2.0
+        py -= ph * pad / 2.0
+        pw *= (1.0 + pad)
+        ph *= (1.0 + pad)
+    left = max(0, int(round(px)))
+    top = max(0, int(round(py)))
+    right = min(iw, int(round(px + pw)))
+    bottom = min(ih, int(round(py + ph)))
+    if right <= left or bottom <= top:
+        return None
+    floor = max(min_px, int(min_fraction * min(iw, ih)))
+    if min(right - left, bottom - top) < floor:
+        return None
+    return img.crop((left, top, right, bottom)).convert("RGB")

agent/fc_agent/embedder.py

@@ -0,0 +1,69 @@
+"""Crop EMBEDDER for the concept bag — model-agnostic (CLIP/SigLIP-family).
+
+The server trains its per-concept heads in the embedding space of whatever model
+its `embedder_model_version` names; a crop must be embedded with the SAME model
+or its vector lands in a different coordinate system and every head misfires. So
+the model identity (HF name + version) is ANNOUNCED BY THE SERVER in the lease —
+nothing here is hardcoded to SigLIP. Whatever name the server sends is loaded via
+transformers `get_image_features` (the CLIP/SigLIP-family image-tower call); a
+non-CLIP backbone (e.g. a DINO encoder) would need its own pooling adapter.
+
+torch on CUDA, fp16 by default to keep VRAM low on a shared desktop GPU — the
+tiny fp16-vs-fp32 difference is negligible for the linear heads (cosine ~0.999).
+A single inference lock serializes the forward pass: the pipeline is I/O-bound,
+so the GPU isn't the bottleneck, and one model shared across worker threads is
+safest behind a lock.
+"""
+import threading
+
+import numpy as np
+from PIL import Image
+
+
+class CropEmbedder:
+    def __init__(self, model_name: str, dtype: str = "float16"):
+        self._name = model_name
+        self._dtype_name = dtype
+        self._model = None
+        self._processor = None
+        self._torch = None
+        self._device = None
+        self._dt = None
+        self._load_lock = threading.Lock()
+        self._infer_lock = threading.Lock()
+
+    @property
+    def model_name(self) -> str:
+        return self._name
+
+    def load(self) -> None:
+        if self._model is not None:
+            return
+        with self._load_lock:
+            if self._model is not None:
+                return
+            import torch
+            from transformers import AutoImageProcessor, AutoModel
+
+            self._torch = torch
+            self._device = "cuda" if torch.cuda.is_available() else "cpu"
+            dt = getattr(torch, self._dtype_name, torch.float16)
+            if self._device == "cpu":
+                dt = torch.float32   # fp16 matmul is unsupported/slow on CPU
+            self._dt = dt
+            self._processor = AutoImageProcessor.from_pretrained(self._name)
+            model = AutoModel.from_pretrained(self._name, torch_dtype=dt)
+            model.eval().to(self._device)
+            self._model = model
+
+    def embed(self, image: Image.Image) -> list[float]:
+        """A crop → its embedding as a plain float list, ready to POST."""
+        self.load()
+        torch = self._torch
+        enc = self._processor(images=image, return_tensors="pt")
+        pixel_values = enc["pixel_values"].to(self._device, self._dt)
+        with self._infer_lock, torch.no_grad():
+            out = self._model.get_image_features(pixel_values=pixel_values)
+        pooled = out.pooler_output if hasattr(out, "pooler_output") else out
+        vec = pooled[0].float().cpu().numpy().astype(np.float32).reshape(-1)
+        return vec.tolist()

agent/fc_agent/gpu.py

@@ -0,0 +1,30 @@
+"""GPU load readout via nvidia-smi (present in the container thanks to the
+NVIDIA Container Toolkit's `utility` capability). Returns None if unavailable —
+the UI just shows n/a (e.g. CPU-fallback run)."""
+import subprocess
+
+
+def read_gpu() -> dict | None:
+    try:
+        out = subprocess.run(
+            [
+                "nvidia-smi",
+                "--query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu",
+                "--format=csv,noheader,nounits",
+            ],
+            capture_output=True, text=True, timeout=5, check=True,
+        ).stdout.strip().splitlines()
+    except (OSError, subprocess.SubprocessError):
+        return None
+    if not out:
+        return None
+    parts = [p.strip() for p in out[0].split(",")]
+    try:
+        return {
+            "util_pct": int(float(parts[0])),
+            "mem_used_mb": int(float(parts[1])),
+            "mem_total_mb": int(float(parts[2])),
+            "temp_c": int(float(parts[3])),
+        }
+    except (ValueError, IndexError):
+        return None

agent/fc_agent/media.py

@@ -0,0 +1,63 @@
+"""Image + video handling. Stills load directly; videos are sampled into frames
+(ffmpeg) at the cadence FC sends — so a video becomes a bag of per-frame
+instances, each with a timestamp."""
+import io
+import os
+import subprocess
+import tempfile
+
+from PIL import Image
+
+
+def is_video(mime: str) -> bool:
+    return bool(mime) and (mime.startswith("video/") or mime in {"image/gif"})
+
+
+def to_rgb(img: Image.Image) -> Image.Image:
+    """RGB, flattening any transparency onto white first. A naive convert('RGB')
+    on a palette-with-transparency image (common for character PNGs on a clear
+    background) lets PIL guess the transparent pixels — usually black artifacts
+    that bleed into the crop + the embedding (and the "should be converted to
+    RGBA" warning). Compositing over white gives a clean, consistent background."""
+    if img.mode in ("RGBA", "LA", "PA") or (
+        img.mode == "P" and "transparency" in img.info
+    ):
+        img = img.convert("RGBA")
+        bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
+        return Image.alpha_composite(bg, img).convert("RGB")
+    return img.convert("RGB")
+
+
+def load_image(data: bytes) -> Image.Image:
+    return to_rgb(Image.open(io.BytesIO(data)))
+
+
+def sample_frames(
+    data: bytes, interval_seconds: float, max_frames: int
+) -> list[tuple[float, Image.Image]]:
+    """Extract up to max_frames frames at one-every-interval_seconds via ffmpeg.
+    Returns [(timestamp_seconds, frame)]. Empty on failure (caller falls back)."""
+    interval = max(0.5, float(interval_seconds or 4.0))
+    cap = max(1, int(max_frames or 64))
+    with tempfile.TemporaryDirectory() as tmp:
+        src = os.path.join(tmp, "in")
+        with open(src, "wb") as fh:
+            fh.write(data)
+        pattern = os.path.join(tmp, "f_%05d.jpg")
+        try:
+            subprocess.run(
+                [
+                    "ffmpeg", "-nostdin", "-loglevel", "error", "-i", src,
+                    "-vf", f"fps=1/{interval}", "-frames:v", str(cap),
+                    "-q:v", "3", pattern,
+                ],
+                check=True, timeout=600,
+            )
+        except (subprocess.SubprocessError, FileNotFoundError):
+            return []
+        out: list[tuple[float, Image.Image]] = []
+        names = sorted(n for n in os.listdir(tmp) if n.startswith("f_"))
+        for i, name in enumerate(names[:cap]):
+            with Image.open(os.path.join(tmp, name)) as im:
+                out.append((round(i * interval, 2), to_rgb(im)))
+        return out

agent/fc_agent/models.py

@@ -0,0 +1,39 @@
+"""imgutils model wrappers — the figure DETECTOR + the CCIP EMBEDDER.
+
+⚠️ VERIFY ON FIRST RUN: the exact imgutils function names/signatures + the CCIP
+model string can drift between dghs-imgutils releases. These are the two seams to
+check against your installed version (`pip show dghs-imgutils`):
+  - detect_person(image, level=...) -> [((x0,y0,x1,y1), label, score), ...]
+  - ccip_extract_feature(image, model=...) -> a vector (768-d for caformer)
+imgutils auto-downloads the ONNX models from HuggingFace on first use; GPU is
+used when onnxruntime-gpu is installed.
+"""
+import numpy as np
+from PIL import Image
+
+
+def detect_figures(image: Image.Image, level: str = "m") -> list[tuple[tuple, float | None]]:
+    """Person/figure bounding boxes, NORMALIZED (x, y, w, h in [0,1]) + score.
+    Returns [] if detection finds nothing (caller falls back to whole-image)."""
+    from imgutils.detect import detect_person
+
+    iw, ih = image.size
+    out = []
+    for (x0, y0, x1, y1), _label, score in detect_person(image, level=level):
+        out.append((
+            (x0 / iw, y0 / ih, (x1 - x0) / iw, (y1 - y0) / ih),
+            float(score),
+        ))
+    return out
+
+
+def ccip_vector(image: Image.Image, model: str | None = None) -> list[float]:
+    """The CCIP identity embedding of a (cropped) character image, as a plain
+    float list ready to POST."""
+    from imgutils.metrics import ccip_extract_feature
+
+    feat = (
+        ccip_extract_feature(image, model=model)
+        if model else ccip_extract_feature(image)
+    )
+    return np.asarray(feat, dtype=np.float32).reshape(-1).tolist()

agent/fc_agent/worker.py

@@ -0,0 +1,274 @@
+"""The lease → fetch → detect+embed → submit loop, run by a pool of worker
+slots whose count is tunable live from the UI.
+
+Each slot is an independent loop (its own leases; the server's SKIP-LOCKED lease
+keeps them from colliding). More slots = more GPU load + throughput; the model is
+loaded once and shared, so slots add concurrent inference, not N× model VRAM.
+That's the dial the operator turns to trade desktop responsiveness for speed.
+
+Stop (or shrinking the pool) RELEASES a slot's still-leased jobs immediately so
+orphaned work is re-picked at once rather than waiting out the lease.
+"""
+import threading
+
+import requests
+
+from . import media, models
+from .client import FcClient
+from .config import Config
+from .crops import crop_region
+
+# Cap on the lease-retry backoff: when curator is unreachable (e.g. you redeploy
+# it while away), each slot retries leasing with exponential backoff up to this
+# many seconds, then resumes within this window once the server is back — no
+# restart needed.
+MAX_BACKOFF_SECONDS = 60.0
+
+
+def _is_transient(exc: "requests.RequestException") -> bool:
+    """A server/transport problem (wait it out) vs a job-specific fault (fail it).
+    No response → connection refused/timeout → curator is down → transient. With
+    a response: 5xx, auth (401/403, e.g. a token blip on redeploy), 408/409/429
+    (timeout / our lease reclaimed / rate-limited) are all 'not this job's fault'.
+    A specific 4xx like 404 (image gone) / 400 IS the job's fault → fail it."""
+    resp = getattr(exc, "response", None)
+    if resp is None:
+        return True
+    return resp.status_code >= 500 or resp.status_code in (401, 403, 408, 409, 429)
+
+# Generous cap: the pipeline is usually I/O-bound (downloading + decoding images
+# over HTTP), so the GPU stays underused until many workers overlap that I/O.
+# Push it up while watching the GPU util + VRAM in the UI.
+MAX_CONCURRENCY = 32
+
+# Fallbacks only — the server ANNOUNCES the embedding model (name + version) in
+# the lease so the agent stays model-agnostic and in lock-step with the space
+# the heads were trained in. These cover an older server that doesn't send them.
+DEFAULT_EMBED_MODEL = "google/siglip-so400m-patch14-384"
+DEFAULT_EMBED_VERSION = "siglip-so400m-patch14-384"
+
+
+class _Slot:
+    """One worker loop. `inflight` = jobs leased but not yet processed, so a
+    graceful stop can hand them back."""
+    __slots__ = ("stop", "inflight")
+
+    def __init__(self):
+        self.stop = threading.Event()
+        self.inflight: list[int] = []
+
+
+class Worker:
+    def __init__(self, cfg: Config):
+        self.cfg = cfg
+        self.client = FcClient(cfg.fc_url, cfg.token, cfg.agent_id)
+        self._lock = threading.Lock()
+        self._running = False
+        self._target = max(1, min(MAX_CONCURRENCY, cfg.concurrency))
+        self._slots: list[_Slot] = []
+        self.processed = 0
+        self.errors = 0
+        self.transient = 0        # jobs handed back due to a server outage (NOT
+        #                           failed) — the "waiting out curator" counter
+        self._active = 0          # slots currently mid-image
+        # The crop embedder (SigLIP-family) is built lazily on the first job that
+        # needs it, from the model the server announces — one shared instance.
+        self._embedder = None
+        self._embedder_lock = threading.Lock()
+
+    # --- control -----------------------------------------------------------
+    def start(self):
+        with self._lock:
+            self._running = True
+            self._reconcile_locked()
+
+    def stop(self):
+        with self._lock:
+            self._running = False
+            slots, self._slots = self._slots, []
+        for s in slots:
+            s.stop.set()          # each slot releases its inflight on exit
+
+    def set_concurrency(self, n: int):
+        with self._lock:
+            self._target = max(1, min(MAX_CONCURRENCY, int(n)))
+            if self._running:
+                self._reconcile_locked()
+
+    def _reconcile_locked(self):
+        while len(self._slots) < self._target:
+            slot = _Slot()
+            self._slots.append(slot)
+            threading.Thread(target=self._loop, args=(slot,), daemon=True).start()
+        while len(self._slots) > self._target:
+            self._slots.pop().stop.set()
+
+    def status(self) -> dict:
+        with self._lock:
+            return {
+                "state": "running" if self._running else "stopped",
+                "concurrency": self._target,
+                "max_concurrency": MAX_CONCURRENCY,
+                "workers": len(self._slots),
+                "active": self._active,
+                "processed": self.processed,
+                "errors": self.errors,
+                "transient": self.transient,
+            }
+
+    def _bump(self, *, processed=0, errors=0, active=0, transient=0):
+        with self._lock:
+            self.processed += processed
+            self.errors += errors
+            self.transient += transient
+            self._active += active
+
+    # --- per-slot loop -----------------------------------------------------
+    def _loop(self, slot: _Slot):
+        backoff = self.cfg.poll_idle_seconds
+        while not slot.stop.is_set() and self._running:
+            try:
+                jobs = self.client.lease(self.cfg.batch_size)
+                backoff = self.cfg.poll_idle_seconds      # server answered → reset
+            except Exception:
+                # curator unreachable (redeploy, network drop): wait it out with
+                # exponential backoff, capped — resume on our own when it returns.
+                self._interruptible_sleep(slot, backoff)
+                backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
+                continue
+            if not jobs:
+                self._interruptible_sleep(slot, self.cfg.poll_idle_seconds)
+                continue
+            slot.inflight = [j["job_id"] for j in jobs]
+            for job in jobs:
+                if slot.stop.is_set() or not self._running:
+                    break
+                ok = self._process(job)
+                slot.inflight = [i for i in slot.inflight if i != job["job_id"]]
+                if not ok:
+                    # Server went away mid-batch: hand the rest back (best effort)
+                    # and back off instead of hammering a recovering server or
+                    # burning the jobs' attempt budgets on fail().
+                    if slot.inflight:
+                        self.client.release(slot.inflight)
+                        slot.inflight = []
+                    self._interruptible_sleep(slot, backoff)
+                    backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
+                    break
+                if slot.inflight:
+                    self.client.heartbeat(slot.inflight)
+        # Graceful hand-back of anything leased but not processed.
+        if slot.inflight:
+            self.client.release(slot.inflight)
+            slot.inflight = []
+
+    def _interruptible_sleep(self, slot: _Slot, seconds: float):
+        """Sleep, but wake immediately if the slot is told to stop — so a Stop or
+        a pool-shrink doesn't hang for a full backoff window."""
+        slot.stop.wait(timeout=seconds)
+
+    def _ensure_embedder(self, model_name: str):
+        if self._embedder is not None:
+            return self._embedder
+        with self._embedder_lock:
+            if self._embedder is None:
+                from .embedder import CropEmbedder
+                self._embedder = CropEmbedder(model_name, self.cfg.embed_dtype)
+        return self._embedder
+
+    def _process(self, job: dict) -> bool:
+        """Process one job. Returns True when handled (completed, or hard-failed
+        because the job itself is bad) and False on a TRANSPORT error (curator
+        unreachable / 5xx / our lease was reclaimed mid-flight) — which is not
+        the job's fault, so the caller backs off and the job is left to be
+        re-leased rather than fail()ed into its attempt budget."""
+        self._bump(active=1)
+        try:
+            data = self.client.fetch_image(job["image_url"])
+            if media.is_video(job.get("mime", "")):
+                frames = media.sample_frames(
+                    data, job.get("frame_interval_seconds", 4.0),
+                    job.get("max_frames", 64),
+                ) or [(None, media.load_image(data))]
+            else:
+                frames = [(None, media.load_image(data))]
+
+            # task picks what to produce per crop:
+            #   'siglip' (backfill existing images) → concept (SigLIP) regions
+            #     ONLY, so it never churns their figure/CCIP regions or the
+            #     character-reference cache.
+            #   'ccip' / 'both' (a new image's first pass) → figure (CCIP) AND
+            #     concept (SigLIP) in one go, off the same crop.
+            task = job.get("task") or "ccip"
+            want_ccip = task in ("ccip", "both")
+            want_siglip = task in ("ccip", "siglip", "both")
+            replace_kinds = (
+                ["concept"] if task == "siglip" else ["figure", "face", "concept"]
+            )
+
+            embed_version = job.get("embed_version") or DEFAULT_EMBED_VERSION
+            embedder = None
+            if want_siglip:
+                model_name = (
+                    self.cfg.embed_model_override
+                    or job.get("embed_model_name")
+                    or DEFAULT_EMBED_MODEL
+                )
+                embedder = self._ensure_embedder(model_name)
+
+            regions = []
+            ccip_ev = self.cfg.ccip_model or "ccip-default"
+            dv = f"person-{self.cfg.detector_level}"
+            for t, frame in frames:
+                figs = models.detect_figures(frame, self.cfg.detector_level)
+                if not figs:
+                    figs = [((0.0, 0.0, 1.0, 1.0), None)]   # whole-frame fallback
+                for bbox, score in figs:
+                    crop = crop_region(frame, bbox)
+                    if crop is None:
+                        continue
+                    if want_ccip:
+                        regions.append({
+                            "kind": "figure",
+                            "bbox": list(bbox),
+                            "frame_time": t,
+                            "score": score,
+                            "ccip_embedding": models.ccip_vector(
+                                crop, self.cfg.ccip_model or None
+                            ),
+                            "embedding_version": ccip_ev,
+                            "detector_version": dv,
+                        })
+                    if want_siglip:
+                        regions.append({
+                            "kind": "concept",
+                            "bbox": list(bbox),
+                            "frame_time": t,
+                            "score": score,
+                            "siglip_embedding": embedder.embed(crop),
+                            "embedding_version": embed_version,
+                            "detector_version": dv,
+                        })
+            self.client.submit(job["job_id"], regions, replace_kinds)
+            self._bump(processed=1)
+            return True
+        except requests.RequestException as exc:
+            if _is_transient(exc):
+                # curator down/redeploying, a 5xx, or our lease was reclaimed
+                # while we worked. NOT the job's fault — hand it back (best
+                # effort; no-ops if the server is still down, then the server's
+                # orphan-recovery reclaims it) and signal the loop to wait.
+                self._bump(transient=1)
+                self.client.release([job["job_id"]])
+                return False
+            # A job-specific HTTP fault (404 image gone, 400) → fail it so it
+            # doesn't re-lease forever.
+            self._bump(errors=1)
+            self.client.fail(job["job_id"], str(exc)[:500])
+            return True
+        except Exception as exc:   # noqa: BLE001 — a genuine job fault: report it
+            self._bump(errors=1)
+            self.client.fail(job["job_id"], str(exc)[:500])
+            return True
+        finally:
+            self._bump(active=-1)

agent/requirements.txt

@@ -0,0 +1,15 @@
+# CCIP + figure detection (ONNX models, auto-downloaded from HuggingFace).
+dghs-imgutils>=0.4
+# GPU inference for the ONNX models. Swap to onnxruntime (CPU) for a slow
+# server-side fallback run.
+onnxruntime-gpu
+# The crop EMBEDDER (concept bag). torch is installed separately in the
+# Dockerfile from the CUDA-12.4 wheel index so the GPU build is deterministic;
+# transformers loads whatever SigLIP-family model the server announces.
+transformers>=4.45
+# Control surface + HTTP.
+fastapi
+uvicorn[standard]
+requests
+pillow
+numpy

alembic/versions/0046_drop_tagger_predictions.py

@@ -0,0 +1,43 @@
+"""drop image_record.tagger_predictions (predictions normalized to image_prediction)
+
+Final step of #768. The per-tag predictions now live in the image_prediction
+table (backfilled from the JSON, read by suggestions + allowlist, written by
+tag_and_embed). The old JSON column is dead weight — and it's the ~100 GB of
+sub-0.70 score tail that bloated image_record's TOAST and broke DB backups
+(#739). Dropping it is a fast catalog change; it does NOT reclaim the disk on
+its own — run `VACUUM FULL image_record` (or pg_repack) afterward, off-hours,
+to return the space to the OS so backups go small.
+
+DROP COLUMN needs a brief ACCESS EXCLUSIVE lock on image_record; env.py's
+lock_timeout guards it, so quiesce the ml-worker if a tagging run is in flight
+(see the migration-lock reference). tagger_model_version is kept — it's the
+"has this been tagged / is it current?" signal the backfill sweep reads.
+
+Revision ID: 0046
+Revises: 0045
+Create Date: 2026-06-11
+
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0046"
+down_revision: Union[str, None] = "0045"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.drop_column("image_record", "tagger_predictions")
+
+
+def downgrade() -> None:
+    # Re-add the column empty. The JSON data is not restored (it lived only in
+    # this column); a downgrade would re-tag or backfill from image_prediction
+    # separately if ever needed.
+    op.add_column(
+        "image_record",
+        sa.Column("tagger_predictions", sa.JSON(), nullable=True),
+    )

alembic/versions/0047_series_chapter_dividers.py

@@ -0,0 +1,175 @@
+"""series chapters become cosmetic dividers; pages become one series-global run
+
+FC-6.x reframe (#789). A series is now ONE flat, series-global ordered run of
+pages; chapters stop owning pages and become labeled dividers anchored to the
+page that begins them.
+
+Migration (order matters — series_page.chapter_id cascades, so it must be
+dropped BEFORE any chapter row is deleted, or pages would cascade away):
+  a. Renumber series_page.page_number to a series-global 1..N (ordered by the
+     OLD (chapter_number, page_number)).
+  b. Add series_chapter.anchor_page_id and populate it with each chapter's first
+     page (lowest new page_number).
+  c. Drop series_page.chapter_id (severs the cascade link).
+  d. Prune chapters that shouldn't become dividers: empty/placeholder ones (no
+     anchor) and the redundant unlabeled chapter that would sit at page 1.
+  e. Reshape series_chapter into the divider: drop chapter_number,
+     is_placeholder, stated_page_start/end; make anchor_page_id NOT NULL +
+     UNIQUE + FK→series_page ON DELETE CASCADE.
+
+Revision ID: 0047
+Revises: 0046
+Create Date: 2026-06-11
+
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0047"
+down_revision: Union[str, None] = "0046"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # a. series-global page numbering, preserving the old reading order.
+    op.execute(
+        """
+        WITH ordered AS (
+            SELECT sp.id,
+                   ROW_NUMBER() OVER (
+                       PARTITION BY sp.series_tag_id
+                       ORDER BY sc.chapter_number, sp.page_number, sp.id
+                   ) AS rn
+            FROM series_page sp
+            JOIN series_chapter sc ON sc.id = sp.chapter_id
+        )
+        UPDATE series_page sp
+        SET page_number = ordered.rn
+        FROM ordered
+        WHERE sp.id = ordered.id
+        """
+    )
+
+    # b. anchor each existing chapter at its first page (lowest new page_number).
+    op.add_column(
+        "series_chapter",
+        sa.Column("anchor_page_id", sa.Integer(), nullable=True),
+    )
+    op.execute(
+        """
+        WITH firsts AS (
+            SELECT DISTINCT ON (sp.chapter_id)
+                   sp.chapter_id, sp.id AS page_id
+            FROM series_page sp
+            ORDER BY sp.chapter_id, sp.page_number, sp.id
+        )
+        UPDATE series_chapter sc
+        SET anchor_page_id = firsts.page_id
+        FROM firsts
+        WHERE firsts.chapter_id = sc.id
+        """
+    )
+
+    # c. sever the ownership link (drops the FK + index with the column) BEFORE
+    #    pruning chapters, so deleting a chapter can't cascade-delete its pages.
+    op.drop_column("series_page", "chapter_id")
+
+    # d. prune chapters that don't become dividers: placeholders / empty ones
+    #    (no anchor), and the unlabeled chapter that would land redundantly at
+    #    page 1 (the series just starts — no divider needed there).
+    op.execute(
+        """
+        DELETE FROM series_chapter sc
+        USING (
+            SELECT sc2.id
+            FROM series_chapter sc2
+            LEFT JOIN series_page sp ON sp.id = sc2.anchor_page_id
+            WHERE sc2.anchor_page_id IS NULL
+               OR (sp.page_number = 1
+                   AND sc2.title IS NULL
+                   AND sc2.stated_part IS NULL)
+        ) gone
+        WHERE sc.id = gone.id
+        """
+    )
+
+    # e. reshape into the divider model.
+    op.drop_column("series_chapter", "chapter_number")
+    op.drop_column("series_chapter", "is_placeholder")
+    op.drop_column("series_chapter", "stated_page_start")
+    op.drop_column("series_chapter", "stated_page_end")
+    op.alter_column("series_chapter", "anchor_page_id", nullable=False)
+    op.create_unique_constraint(
+        "uq_series_chapter_anchor_page", "series_chapter", ["anchor_page_id"]
+    )
+    op.create_foreign_key(
+        "fk_series_chapter_anchor_page",
+        "series_chapter",
+        "series_page",
+        ["anchor_page_id"],
+        ["id"],
+        ondelete="CASCADE",
+    )
+
+
+def downgrade() -> None:
+    # Lossy: dividers can't be reconstructed as owning chapters. Collapse back to
+    # exactly one chapter per series that owns all its pages in order.
+    op.add_column(
+        "series_page", sa.Column("chapter_id", sa.Integer(), nullable=True)
+    )
+    op.drop_constraint(
+        "fk_series_chapter_anchor_page", "series_chapter", type_="foreignkey"
+    )
+    op.drop_constraint(
+        "uq_series_chapter_anchor_page", "series_chapter", type_="unique"
+    )
+    op.drop_column("series_chapter", "anchor_page_id")
+    op.add_column(
+        "series_chapter",
+        sa.Column(
+            "chapter_number", sa.Integer(), nullable=False, server_default="1"
+        ),
+    )
+    op.add_column(
+        "series_chapter",
+        sa.Column(
+            "is_placeholder", sa.Boolean(), nullable=False,
+            server_default="false",
+        ),
+    )
+    op.add_column(
+        "series_chapter",
+        sa.Column("stated_page_start", sa.Integer(), nullable=True),
+    )
+    op.add_column(
+        "series_chapter",
+        sa.Column("stated_page_end", sa.Integer(), nullable=True),
+    )
+    op.execute("DELETE FROM series_chapter")
+    op.execute(
+        """
+        INSERT INTO series_chapter (series_tag_id, chapter_number)
+        SELECT DISTINCT series_tag_id, 1 FROM series_page
+        """
+    )
+    op.execute(
+        """
+        UPDATE series_page sp
+        SET chapter_id = sc.id
+        FROM series_chapter sc
+        WHERE sc.series_tag_id = sp.series_tag_id
+        """
+    )
+    op.alter_column("series_page", "chapter_id", nullable=False)
+    op.create_foreign_key(
+        "fk_series_page_chapter",
+        "series_page",
+        "series_chapter",
+        ["chapter_id"],
+        ["id"],
+        ondelete="CASCADE",
+    )

alembic/versions/0048_series_page_pending_status.py

@@ -0,0 +1,45 @@
+"""series_page pending staging: status + nullable page_number (#789 Phase 2)
+
+Pages added from a post no longer append straight into the run — they land
+'pending' with a NULL page_number, staged grouped by their source post so the
+operator can drop junk (text-free alts, bumpers) and place the keepers into the
+sequence. A page only gets a series-global page_number once it's 'placed'.
+
+Revision ID: 0048
+Revises: 0047
+Create Date: 2026-06-11
+
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0048"
+down_revision: Union[str, None] = "0047"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "series_page",
+        sa.Column(
+            "status", sa.String(length=16), nullable=False,
+            server_default="placed",
+        ),
+    )
+    op.alter_column(
+        "series_page", "page_number",
+        existing_type=sa.Integer(), nullable=True,
+    )
+
+
+def downgrade() -> None:
+    # Lossy: pending pages are unsorted staging rows with no order — drop them.
+    op.execute("DELETE FROM series_page WHERE status = 'pending'")
+    op.alter_column(
+        "series_page", "page_number",
+        existing_type=sa.Integer(), nullable=False,
+    )
+    op.drop_column("series_page", "status")

alembic/versions/0049_external_link_table.py

@@ -0,0 +1,90 @@
+"""external_link table — off-platform file-host links found in post bodies
+
+Creators host the real files on mega.nz / Google Drive / MediaFire / Dropbox /
+Pixeldrain and link them in the post text. This table records each such link
+(so nothing is silently dropped), and doubles as the dedup + dead-letter ledger
+the download worker (a later slice) walks. `url` keeps the FULL link including
+the `#fragment` — mega.nz's decryption key lives there; truncating it makes the
+file undownloadable.
+
+CHECK whitelists for host + status include the full enum up front (incl. the
+download-worker statuses) so the worker slice needs no constraint migration.
+
+Revision ID: 0049
+Revises: 0048
+Create Date: 2026-06-14
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0049"
+down_revision: Union[str, None] = "0048"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "external_link",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "post_id", sa.Integer(),
+            sa.ForeignKey("post.id", ondelete="CASCADE"), nullable=False,
+        ),
+        sa.Column(
+            "artist_id", sa.Integer(),
+            sa.ForeignKey("artist.id", ondelete="SET NULL"), nullable=True,
+        ),
+        sa.Column("host", sa.String(length=16), nullable=False),
+        sa.Column("url", sa.Text(), nullable=False),
+        sa.Column("label", sa.Text(), nullable=True),
+        sa.Column(
+            "status", sa.String(length=16), nullable=False,
+            server_default="pending",
+        ),
+        sa.Column("attempts", sa.Integer(), nullable=False, server_default="0"),
+        sa.Column("last_error", sa.Text(), nullable=True),
+        sa.Column(
+            "attachment_id", sa.Integer(),
+            sa.ForeignKey("post_attachment.id", ondelete="SET NULL"),
+            nullable=True,
+        ),
+        sa.Column(
+            "created_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+        sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("duration_seconds", sa.Float(), nullable=True),
+        sa.CheckConstraint(
+            "host IN ('mega','gdrive','mediafire','dropbox','pixeldrain')",
+            name="ck_external_link_host",
+        ),
+        sa.CheckConstraint(
+            "status IN ('pending','downloading','downloaded','failed',"
+            "'skipped','dead')",
+            name="ck_external_link_status",
+        ),
+    )
+    op.create_index(
+        "ix_external_link_post_id", "external_link", ["post_id"],
+    )
+    op.create_index(
+        "ix_external_link_artist_id", "external_link", ["artist_id"],
+    )
+    op.create_index(
+        "ix_external_link_status", "external_link", ["status"],
+    )
+    op.create_index(
+        "uq_external_link_post_url", "external_link", ["post_id", "url"],
+        unique=True,
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("uq_external_link_post_url", table_name="external_link")
+    op.drop_index("ix_external_link_status", table_name="external_link")
+    op.drop_index("ix_external_link_artist_id", table_name="external_link")
+    op.drop_index("ix_external_link_post_id", table_name="external_link")
+    op.drop_table("external_link")

alembic/versions/0050_external_link_host_toggles.py

@@ -0,0 +1,38 @@
+"""import_settings: per-host enable toggles for external file-host downloads
+
+Operator levers (#830): disable a single host (e.g. mega.nz when it's
+rate-limiting/banning) without touching the others. The worker reads these via
+getattr and defaults to enabled, so the toggles default TRUE (works out of the
+box, rule #26).
+
+Revision ID: 0050
+Revises: 0049
+Create Date: 2026-06-14
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0050"
+down_revision: Union[str, None] = "0049"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+_HOSTS = ("mega", "gdrive", "mediafire", "dropbox", "pixeldrain")
+
+
+def upgrade() -> None:
+    for host in _HOSTS:
+        op.add_column(
+            "import_settings",
+            sa.Column(
+                f"extdl_{host}_enabled", sa.Boolean(), nullable=False,
+                server_default=sa.true(),
+            ),
+        )
+
+
+def downgrade() -> None:
+    for host in _HOSTS:
+        op.drop_column("import_settings", f"extdl_{host}_enabled")

alembic/versions/0051_image_source_provenance.py

@@ -0,0 +1,38 @@
+"""image_record: source_url + source_filehash (inline-image localization)
+
+#830 Phase 2. To render a post body faithfully we serve LOCAL copies of inline
+images instead of hotlinking the public CDN. The join key between a body
+`<img src=CDN>` and the local file is the CDN's 32-hex filehash (the same
+identity extract_media dedups by). Persist it (indexed) plus the full source
+URL for provenance/debugging. Both NULL for filesystem-imported / pre-existing
+rows — those fall back to hotlinking until re-downloaded.
+
+Revision ID: 0051
+Revises: 0050
+Create Date: 2026-06-14
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0051"
+down_revision: Union[str, None] = "0050"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column("image_record", sa.Column("source_url", sa.Text(), nullable=True))
+    op.add_column(
+        "image_record", sa.Column("source_filehash", sa.String(length=32), nullable=True)
+    )
+    op.create_index(
+        "ix_image_record_source_filehash", "image_record", ["source_filehash"]
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_image_record_source_filehash", table_name="image_record")
+    op.drop_column("image_record", "source_filehash")
+    op.drop_column("image_record", "source_url")

alembic/versions/0052_image_duration_seconds.py

@@ -0,0 +1,32 @@
+"""image_record: duration_seconds (Tier-1 video near-dup key)
+
+#871. Videos previously deduped on sha256 only (pHash is images-only), so a
+different encode/remux of the same video imported as a distinct record. Persist
+the container duration so the importer can treat same-artist videos with matching
+duration (+ aspect ratio) as the same content and dedup/supersede like images.
+NULL for images and for video rows imported before this column existed (a
+backfill re-probes those so they participate in dedup).
+
+Revision ID: 0052
+Revises: 0051
+Create Date: 2026-06-16
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0052"
+down_revision: Union[str, None] = "0051"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "image_record", sa.Column("duration_seconds", sa.Float(), nullable=True)
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("image_record", "duration_seconds")

alembic/versions/0053_ml_settings_video_tagging.py

@@ -0,0 +1,49 @@
+"""ml_settings: video tagging knobs (cadence sampling + noise floor)
+
+#747. Video tag quality/perf: sample frames at a fixed cadence (interval) so a
+tag's frame-presence reflects real screen time, cap total frames so long videos
+stay bounded, and keep a tag only if it appears in >= min_tag_frames sampled
+frames. Operator-tunable via Settings → ML (replaces the VIDEO_ML_FRAMES env var).
+
+Revision ID: 0053
+Revises: 0052
+Create Date: 2026-06-16
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0053"
+down_revision: Union[str, None] = "0052"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "video_frame_interval_seconds", sa.Float(), nullable=False,
+            server_default="4.0",
+        ),
+    )
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "video_max_frames", sa.Integer(), nullable=False, server_default="64",
+        ),
+    )
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "video_min_tag_frames", sa.Integer(), nullable=False,
+            server_default="3",
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("ml_settings", "video_min_tag_frames")
+    op.drop_column("ml_settings", "video_max_frames")
+    op.drop_column("ml_settings", "video_frame_interval_seconds")

alembic/versions/0054_subscribestar_ledgers.py

@@ -0,0 +1,82 @@
+"""subscribestar_seen_media + subscribestar_failed_media: per-source ledgers
+
+Revision ID: 0054
+Revises: 0053
+Create Date: 2026-06-17
+
+SubscribeStar native ingester (phase 1 of the gallery-dl → native-core
+migration). Mirrors the Patreon ledger tables (0037/0038): a seen-ledger so
+routine walks skip already-ingested media (recovery bypasses it) and a
+dead-letter ledger so persistently-failing media stops re-burning backfill
+chunks. `filehash` is a CDN content hash when present, else a synthesized
+``<post_id>:<filename>`` key — hence String(128). UNIQUE (source_id, filehash)
+is the upsert key on each.
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0054"
+down_revision: Union[str, None] = "0053"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "subscribestar_seen_media",
+        sa.Column("id", sa.Integer, primary_key=True),
+        sa.Column(
+            "source_id",
+            sa.Integer,
+            sa.ForeignKey("source.id", ondelete="CASCADE"),
+            nullable=False,
+            index=True,
+        ),
+        sa.Column("filehash", sa.String(128), nullable=False),
+        sa.Column("post_id", sa.String(64), nullable=True),
+        sa.Column(
+            "seen_at",
+            sa.DateTime(timezone=True),
+            nullable=False,
+            server_default=sa.text("NOW()"),
+        ),
+        sa.UniqueConstraint(
+            "source_id", "filehash", name="uq_subscribestar_seen_media_source_id"
+        ),
+    )
+    op.create_table(
+        "subscribestar_failed_media",
+        sa.Column("id", sa.Integer, primary_key=True),
+        sa.Column(
+            "source_id",
+            sa.Integer,
+            sa.ForeignKey("source.id", ondelete="CASCADE"),
+            nullable=False,
+            index=True,
+        ),
+        sa.Column("filehash", sa.String(128), nullable=False),
+        sa.Column("attempts", sa.Integer, nullable=False, server_default="1"),
+        sa.Column("last_error", sa.Text, nullable=True),
+        sa.Column(
+            "first_failed_at",
+            sa.DateTime(timezone=True),
+            nullable=False,
+            server_default=sa.text("NOW()"),
+        ),
+        sa.Column(
+            "last_failed_at",
+            sa.DateTime(timezone=True),
+            nullable=False,
+            server_default=sa.text("NOW()"),
+        ),
+        sa.UniqueConstraint(
+            "source_id", "filehash", name="uq_subscribestar_failed_media_source_id"
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("subscribestar_failed_media")
+    op.drop_table("subscribestar_seen_media")

alembic/versions/0055_image_provenance_from_attachment.py

@@ -0,0 +1,55 @@
+"""image_provenance: from_attachment_id (which archive an image was extracted from)
+
+Milestone #87. When an image is pulled out of a .zip/.rar, record WHICH archive
+PostAttachment it came from, so the provenance UI can show the single archive a
+file lives inside instead of every attachment on the post. Nullable FK with
+ON DELETE SET NULL — a loose (non-archive) download leaves it NULL, and deleting
+the archive attachment forgets the linkage without destroying the (image, post)
+provenance edge. Existing rows are NULL until the reextract backfill stamps them.
+
+Revision ID: 0055
+Revises: 0054
+Create Date: 2026-06-22
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0055"
+down_revision: Union[str, None] = "0054"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "image_provenance",
+        sa.Column("from_attachment_id", sa.Integer(), nullable=True),
+    )
+    op.create_index(
+        "ix_image_provenance_from_attachment_id",
+        "image_provenance",
+        ["from_attachment_id"],
+    )
+    op.create_foreign_key(
+        "fk_image_provenance_from_attachment",
+        "image_provenance",
+        "post_attachment",
+        ["from_attachment_id"],
+        ["id"],
+        ondelete="SET NULL",
+    )
+
+
+def downgrade() -> None:
+    op.drop_constraint(
+        "fk_image_provenance_from_attachment",
+        "image_provenance",
+        type_="foreignkey",
+    )
+    op.drop_index(
+        "ix_image_provenance_from_attachment_id",
+        table_name="image_provenance",
+    )
+    op.drop_column("image_provenance", "from_attachment_id")

alembic/versions/0056_tag_eval_run.py

@@ -0,0 +1,43 @@
+"""tag_eval_run: persisted head-vs-centroid tagging eval runs (#1130)
+
+Milestone #114 slice 1. A long ml-queue eval whose full report must SURVIVE
+navigation, so the run + report live in a row the admin card rehydrates from
+(mirrors library_audit_run). running -> ready / error.
+
+Revision ID: 0056
+Revises: 0055
+Create Date: 2026-06-28
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects.postgresql import JSONB
+
+revision: str = "0056"
+down_revision: Union[str, None] = "0055"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "tag_eval_run",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column("params", JSONB(), nullable=False),
+        sa.Column("status", sa.String(length=16), nullable=False, server_default="running"),
+        sa.Column(
+            "started_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+        sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("report", JSONB(), nullable=True),
+        sa.Column("error", sa.Text(), nullable=True),
+        sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
+    )
+    op.create_index("ix_tag_eval_run_status", "tag_eval_run", ["status"])
+
+
+def downgrade() -> None:
+    op.drop_index("ix_tag_eval_run_status", table_name="tag_eval_run")
+    op.drop_table("tag_eval_run")

alembic/versions/0057_tag_positive_confirmation.py

@@ -0,0 +1,40 @@
+"""tag_positive_confirmation: operator-affirmed correct positives (#1130)
+
+Mirror of tag_suggestion_rejection. "Keep" on a doubted positive records here so
+the eval's doubts list stops resurfacing confirmed-correct images every run.
+
+Revision ID: 0057
+Revises: 0056
+Create Date: 2026-06-28
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0057"
+down_revision: Union[str, None] = "0056"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "tag_positive_confirmation",
+        sa.Column(
+            "image_record_id", sa.Integer(),
+            sa.ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True,
+        ),
+        sa.Column(
+            "tag_id", sa.Integer(),
+            sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True,
+        ),
+        sa.Column(
+            "confirmed_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("tag_positive_confirmation")

alembic/versions/0058_tag_head.py

@@ -0,0 +1,95 @@
+"""tag_head + head_training_run: production heads that learn from tags (#114)
+
+The eval (#1130) proved the frozen-embedding + trained-head spine; this lands its
+production form. tag_head stores one logistic-regression head per concept (the
+new suggestion source, replacing Camie + centroid); head_training_run tracks the
+batch that (re)trains them. Adds two head-training tunables to ml_settings.
+
+Revision ID: 0058
+Revises: 0057
+Create Date: 2026-06-28
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+from pgvector.sqlalchemy import Vector
+from sqlalchemy.dialects.postgresql import JSONB
+
+revision: str = "0058"
+down_revision: Union[str, None] = "0057"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+_HEAD_DIM = 1152
+
+
+def upgrade() -> None:
+    op.create_table(
+        "tag_head",
+        sa.Column(
+            "tag_id", sa.Integer(),
+            sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
+        ),
+        sa.Column("embedding_version", sa.String(length=128), nullable=False),
+        sa.Column("weights", Vector(_HEAD_DIM), nullable=False),
+        sa.Column("bias", sa.Float(), nullable=False),
+        sa.Column("suggest_threshold", sa.Float(), nullable=False),
+        sa.Column("auto_apply_threshold", sa.Float(), nullable=True),
+        sa.Column("n_pos", sa.Integer(), nullable=False),
+        sa.Column("n_neg", sa.Integer(), nullable=False),
+        sa.Column("ap", sa.Float(), nullable=False),
+        sa.Column("precision_cv", sa.Float(), nullable=False),
+        sa.Column("recall", sa.Float(), nullable=False),
+        sa.Column(
+            "trained_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+        sa.Column("metrics", JSONB(), nullable=True),
+    )
+
+    op.create_table(
+        "head_training_run",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column("params", JSONB(), nullable=False),
+        sa.Column(
+            "status", sa.String(length=16), nullable=False,
+            server_default="running",
+        ),
+        sa.Column(
+            "started_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+        sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("n_trained", sa.Integer(), nullable=True),
+        sa.Column("n_skipped", sa.Integer(), nullable=True),
+        sa.Column("error", sa.Text(), nullable=True),
+        sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
+    )
+    op.create_index(
+        "ix_head_training_run_status", "head_training_run", ["status"],
+    )
+
+    # Head-training tunables on the ml_settings singleton.
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "head_min_positives", sa.Integer(), nullable=False,
+            server_default="8",
+        ),
+    )
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "head_auto_apply_precision", sa.Float(), nullable=False,
+            server_default="0.97",
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("ml_settings", "head_auto_apply_precision")
+    op.drop_column("ml_settings", "head_min_positives")
+    op.drop_index("ix_head_training_run_status", table_name="head_training_run")
+    op.drop_table("head_training_run")
+    op.drop_table("tag_head")

alembic/versions/0059_head_auto_apply.py

@@ -0,0 +1,70 @@
+"""head_auto_apply_run + earned-auto-apply settings (#114)
+
+A graduated head can apply its tag without a human, gated by a master switch +
+a support floor. head_auto_apply_run tracks each sweep / dry-run preview.
+
+Revision ID: 0059
+Revises: 0058
+Create Date: 2026-06-29
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects.postgresql import JSONB
+
+revision: str = "0059"
+down_revision: Union[str, None] = "0058"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "head_auto_apply_run",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "dry_run", sa.Boolean(), nullable=False, server_default=sa.false()
+        ),
+        sa.Column("params", JSONB(), nullable=False),
+        sa.Column(
+            "status", sa.String(length=16), nullable=False,
+            server_default="running",
+        ),
+        sa.Column(
+            "started_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+        sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("n_applied", sa.Integer(), nullable=True),
+        sa.Column("report", JSONB(), nullable=True),
+        sa.Column("error", sa.Text(), nullable=True),
+        sa.Column("last_progress_at", sa.DateTime(timezone=True), nullable=True),
+    )
+    op.create_index(
+        "ix_head_auto_apply_run_status", "head_auto_apply_run", ["status"],
+    )
+
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "head_auto_apply_enabled", sa.Boolean(), nullable=False,
+            server_default=sa.true(),  # opt-out: on by default (operator-asked)
+        ),
+    )
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "head_auto_apply_min_positives", sa.Integer(), nullable=False,
+            server_default="30",
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("ml_settings", "head_auto_apply_min_positives")
+    op.drop_column("ml_settings", "head_auto_apply_enabled")
+    op.drop_index(
+        "ix_head_auto_apply_run_status", table_name="head_auto_apply_run"
+    )
+    op.drop_table("head_auto_apply_run")

alembic/versions/0060_head_metrics.py

@@ -0,0 +1,74 @@
+"""head_metric + head_metrics_snapshot: auto-apply observability (#114)
+
+Running misfire/under-fire counters per concept (captured at correction time,
+since image_tag.source is lost on delete) + a daily per-concept time-series so
+the operator can tune the precision target + support floor from real data.
+
+Revision ID: 0060
+Revises: 0059
+Create Date: 2026-06-29
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0060"
+down_revision: Union[str, None] = "0059"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "head_metric",
+        sa.Column(
+            "tag_id", sa.Integer(),
+            sa.ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True,
+        ),
+        sa.Column("n_misfires", sa.Integer(), nullable=False, server_default="0"),
+        sa.Column("n_underfires", sa.Integer(), nullable=False, server_default="0"),
+        sa.Column(
+            "updated_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+    )
+
+    op.create_table(
+        "head_metrics_snapshot",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "tag_id", sa.Integer(),
+            sa.ForeignKey("tag.id", ondelete="CASCADE"),
+        ),
+        sa.Column("name", sa.String(length=255), nullable=False),
+        sa.Column(
+            "snapshot_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+        sa.Column("n_auto_applied", sa.Integer(), nullable=False, server_default="0"),
+        sa.Column("n_misfires", sa.Integer(), nullable=False, server_default="0"),
+        sa.Column("n_underfires", sa.Integer(), nullable=False, server_default="0"),
+        sa.Column("ap", sa.Float(), nullable=True),
+        sa.Column("precision_cv", sa.Float(), nullable=True),
+        sa.Column("recall", sa.Float(), nullable=True),
+        sa.Column("n_pos", sa.Integer(), nullable=True),
+    )
+    op.create_index(
+        "ix_head_metrics_snapshot_tag_id", "head_metrics_snapshot", ["tag_id"],
+    )
+    op.create_index(
+        "ix_head_metrics_snapshot_snapshot_at", "head_metrics_snapshot",
+        ["snapshot_at"],
+    )
+
+
+def downgrade() -> None:
+    op.drop_index(
+        "ix_head_metrics_snapshot_snapshot_at", table_name="head_metrics_snapshot"
+    )
+    op.drop_index(
+        "ix_head_metrics_snapshot_tag_id", table_name="head_metrics_snapshot"
+    )
+    op.drop_table("head_metrics_snapshot")
+    op.drop_table("head_metric")

alembic/versions/0061_image_region.py

@@ -0,0 +1,59 @@
+"""image_region: detected/proposed regions + their crop embeddings (#114)
+
+Storage backbone of the crop pipeline. A region = normalized bbox + the crop's
+embedding (CCIP for face/figure → character id; SigLIP for concept regions →
+head bag-of-embeddings). Also serves as grounded-tag bbox provenance.
+
+Revision ID: 0061
+Revises: 0060
+Create Date: 2026-06-29
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+from pgvector.sqlalchemy import Vector
+
+revision: str = "0061"
+down_revision: Union[str, None] = "0060"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+_CCIP_DIM = 768
+_SIGLIP_DIM = 1152
+
+
+def upgrade() -> None:
+    op.create_table(
+        "image_region",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "image_record_id", sa.Integer(),
+            sa.ForeignKey("image_record.id", ondelete="CASCADE"), nullable=False,
+        ),
+        sa.Column("kind", sa.String(length=16), nullable=False),
+        # Video/animated: source frame timestamp (seconds); NULL for stills.
+        sa.Column("frame_time", sa.Float(), nullable=True),
+        sa.Column("rx", sa.Float(), nullable=False),
+        sa.Column("ry", sa.Float(), nullable=False),
+        sa.Column("rw", sa.Float(), nullable=False),
+        sa.Column("rh", sa.Float(), nullable=False),
+        sa.Column("score", sa.Float(), nullable=True),
+        sa.Column("detector_version", sa.String(length=64), nullable=True),
+        sa.Column("crop_version", sa.String(length=64), nullable=True),
+        sa.Column("embedding_version", sa.String(length=128), nullable=True),
+        sa.Column("ccip_embedding", Vector(_CCIP_DIM), nullable=True),
+        sa.Column("siglip_embedding", Vector(_SIGLIP_DIM), nullable=True),
+        sa.Column(
+            "created_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+    )
+    op.create_index(
+        "ix_image_region_image_record_id", "image_region", ["image_record_id"],
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_image_region_image_record_id", table_name="image_region")
+    op.drop_table("image_region")

alembic/versions/0062_gpu_job.py

@@ -0,0 +1,55 @@
+"""gpu_job: the HTTP-leased GPU work queue for the desktop agent (#114)
+
+The agent stays HTTP-only — the server enqueues per-(image, task) jobs here and
+the agent leases/submits over the web API; Redis/Postgres stay private.
+
+Revision ID: 0062
+Revises: 0061
+Create Date: 2026-06-29
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0062"
+down_revision: Union[str, None] = "0061"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "gpu_job",
+        sa.Column("id", sa.Integer(), primary_key=True),
+        sa.Column(
+            "image_record_id", sa.Integer(),
+            sa.ForeignKey("image_record.id", ondelete="CASCADE"), nullable=False,
+        ),
+        sa.Column("task", sa.String(length=32), nullable=False),
+        sa.Column(
+            "status", sa.String(length=16), nullable=False,
+            server_default="pending",
+        ),
+        sa.Column("lease_token", sa.String(length=64), nullable=True),
+        sa.Column("leased_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("lease_expires_at", sa.DateTime(timezone=True), nullable=True),
+        sa.Column("attempts", sa.Integer(), nullable=False, server_default="0"),
+        sa.Column("error", sa.Text(), nullable=True),
+        sa.Column(
+            "created_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+        sa.Column(
+            "updated_at", sa.DateTime(timezone=True), nullable=False,
+            server_default=sa.func.now(),
+        ),
+    )
+    op.create_index("ix_gpu_job_image_record_id", "gpu_job", ["image_record_id"])
+    op.create_index("ix_gpu_job_status", "gpu_job", ["status"])
+
+
+def downgrade() -> None:
+    op.drop_index("ix_gpu_job_status", table_name="gpu_job")
+    op.drop_index("ix_gpu_job_image_record_id", table_name="gpu_job")
+    op.drop_table("gpu_job")

alembic/versions/0063_ccip_match_threshold.py

@@ -0,0 +1,33 @@
+"""ml_settings.ccip_match_threshold — tunable CCIP character-match cut (#114)
+
+The v1 matcher used a flat 0.75 cosine; live data showed that over-fires (a
+high-reference character matched a scatter of images). 0.85 keeps the confident
+single-character matches and drops the noise. Tunable from the GPU agent card.
+
+Revision ID: 0063
+Revises: 0062
+Create Date: 2026-06-29
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0063"
+down_revision: Union[str, None] = "0062"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "ccip_match_threshold", sa.Float(), nullable=False,
+            server_default="0.85",
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("ml_settings", "ccip_match_threshold")

alembic/versions/0064_ccip_auto_apply.py

@@ -0,0 +1,42 @@
+"""ml_settings: CCIP auto-apply switch + threshold (#114)
+
+Confident CCIP character matches auto-tag (source='ccip_auto') on a daily sweep,
+so identity tags keep flowing without pressing a button. ON by default (opt-out,
+like head auto-apply); the high threshold (0.92, above the 0.85 suggest cut) +
+single-character references keep it safe, and every auto-tag is reversible.
+
+Revision ID: 0064
+Revises: 0063
+Create Date: 2026-06-30
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0064"
+down_revision: Union[str, None] = "0063"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "ccip_auto_apply_enabled", sa.Boolean(), nullable=False,
+            server_default=sa.true(),
+        ),
+    )
+    op.add_column(
+        "ml_settings",
+        sa.Column(
+            "ccip_auto_apply_threshold", sa.Float(), nullable=False,
+            server_default="0.92",
+        ),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("ml_settings", "ccip_auto_apply_threshold")
+    op.drop_column("ml_settings", "ccip_auto_apply_enabled")

backend/app/api/__init__.py

@@ -20,11 +20,14 @@ def all_blueprints() -> list[Blueprint]:
     from .artist import artist_bp
     from .artists import artists_bp
     from .attachments import attachments_bp
+    from .ccip import ccip_bp
     from .cleanup import cleanup_bp
     from .credentials import credentials_bp
     from .downloads import downloads_bp
     from .extension import extension_bp
     from .gallery import gallery_bp
+    from .gpu import gpu_bp
+    from .heads import heads_bp
     from .import_admin import import_admin_bp
     from .ml_admin import ml_admin_bp
     from .platforms import platforms_bp
@@ -36,6 +39,7 @@ def all_blueprints() -> list[Blueprint]:
     from .suggestions import suggestions_bp
     from .system_activity import system_activity_bp
     from .system_backup import system_backup_bp
+    from .tag_eval import tag_eval_bp
     from .tags import tags_bp
     from .thumbnails import thumbnails_bp
     return [
@@ -56,6 +60,10 @@ def all_blueprints() -> list[Blueprint]:
         suggestions_bp,
         allowlist_bp,
         aliases_bp,
+        tag_eval_bp,
+        heads_bp,
+        gpu_bp,
+        ccip_bp,
         ml_admin_bp,
         thumbnails_bp,
         sources_bp,

backend/app/api/admin.py

@@ -39,6 +39,31 @@ def _bulk_image_confirm_token(image_ids: list[int]) -> str:
     return digest[:8]
 
 
+async def _run_dry_run_op(service_fn, **service_kwargs):
+    """Shared body for the Tier-A dry-run/apply endpoints: read the `dry_run`
+    flag, run the cleanup_service predicate under `run_sync`, and return its
+    result dict. The SAME `service_fn` drives both preview and apply (the flag
+    just toggles), so a handler physically can't let its preview diverge from
+    its delete (rule 93). Default False preserves the existing contract — the UI
+    always passes `dry_run` explicitly (true to preview, false to apply). Extra
+    service kwargs (e.g. `source_id`) pass straight through."""
+    body = await request.get_json(silent=True) or {}
+    dry_run = bool(body.get("dry_run", False))
+    async with get_session() as session:
+        result = await session.run_sync(
+            lambda sync_sess: service_fn(sync_sess, dry_run=dry_run, **service_kwargs)
+        )
+    return jsonify(result)
+
+
+def _queued(async_result):
+    """Standard 202 for an operator-triggered maintenance task: hand the UI the
+    Celery task id so it can tail /maintenance/task-result (or the activity
+    dashboard) for the summary. (trigger_vacuum stays bespoke — the UI doesn't
+    poll it, so it returns no task id.)"""
+    return jsonify({"task_id": async_result.id, "status": "queued"}), 202
+
+
 @admin_bp.route("/artists/<slug>/cascade-delete", methods=["POST"])
 async def artist_cascade_delete(slug: str):
     body = await request.get_json(silent=True) or {}
@@ -146,6 +171,30 @@ async def tag_merge(dest_id: int):
     if not isinstance(source_id, int) or source_id == dest_id:
         return _bad("invalid_source_id", detail="source_id must be int and differ from dest")
 
+    # dry_run: non-mutating preview (counts + sample) so the operator can
+    # confirm the target before the irreversible merge (#8, rule 93 parity).
+    if body.get("dry_run"):
+        async with get_session() as session:
+            try:
+                p = await TagService(session).merge_preview(
+                    source_id=source_id, target_id=dest_id,
+                )
+            except TagValidationError as exc:
+                return _bad("tag_not_found", status=404, detail=str(exc))
+        return jsonify({
+            "preview": {
+                "source_id": p.source_id, "source_name": p.source_name,
+                "target_id": p.target_id, "target_name": p.target_name,
+                "compatible": p.compatible,
+                "images_moving": p.images_moving,
+                "images_already_on_target": p.images_already_on_target,
+                "source_total": p.source_total,
+                "series_pages": p.series_pages,
+                "will_alias": p.will_alias,
+                "sample_thumbnails": p.sample_thumbnails,
+            },
+        })
+
     async with get_session() as session:
         try:
             result = await TagService(session).merge(
@@ -193,16 +242,7 @@ async def tags_prune_unused():
     re-call with dry_run=false."""
     from ..services.cleanup_service import prune_unused_tags
 
-    body = await request.get_json(silent=True) or {}
-    dry_run = bool(body.get("dry_run", False))
-
-    async with get_session() as session:
-        result = await session.run_sync(
-            lambda sync_sess: prune_unused_tags(
-                sync_sess, dry_run=dry_run,
-            )
-        )
-    return jsonify(result)
+    return await _run_dry_run_op(prune_unused_tags)
 
 
 @admin_bp.route("/posts/prune-bare", methods=["POST"])
@@ -214,16 +254,27 @@ async def posts_prune_bare():
     prune itself, so the preview can't diverge from the delete."""
     from ..services.cleanup_service import prune_bare_posts
 
-    body = await request.get_json(silent=True) or {}
-    dry_run = bool(body.get("dry_run", False))
+    return await _run_dry_run_op(prune_bare_posts)
 
-    async with get_session() as session:
-        result = await session.run_sync(
-            lambda sync_sess: prune_bare_posts(
-                sync_sess, dry_run=dry_run,
-            )
-        )
-    return jsonify(result)
+
+@admin_bp.route("/posts/reconcile-duplicates", methods=["POST"])
+async def posts_reconcile_duplicates():
+    """Tier-A: unify duplicate post rows for the same real post — the gallery-dl
+    (attachment-id) + native (post-id) duplicates — onto ONE post-id-keyed keeper,
+    moving image/provenance/attachment/link rows over. Images are untouched.
+    dry_run=true returns {groups, posts_to_merge, sample}; dry_run=false applies
+    and returns {groups, merged, sample}. Optional source_id scopes to one source.
+    Same find_duplicate_post_groups predicate drives preview + apply (rule 93)."""
+    from ..services.cleanup_service import reconcile_duplicate_posts
+
+    body = await request.get_json(silent=True) or {}
+    raw_source = body.get("source_id")
+    try:
+        source_id = int(raw_source) if raw_source is not None else None
+    except (TypeError, ValueError):
+        return _bad("invalid_source_id", detail="source_id must be an integer")
+
+    return await _run_dry_run_op(reconcile_duplicate_posts, source_id=source_id)
 
 
 @admin_bp.route("/tags/purge-legacy", methods=["POST"])
@@ -236,14 +287,7 @@ async def tags_purge_legacy():
     operator confirms with dry_run=false."""
     from ..services.cleanup_service import purge_legacy_tags
 
-    body = await request.get_json(silent=True) or {}
-    dry_run = bool(body.get("dry_run", False))
-
-    async with get_session() as session:
-        result = await session.run_sync(
-            lambda sync_sess: purge_legacy_tags(sync_sess, dry_run=dry_run)
-        )
-    return jsonify(result)
+    return await _run_dry_run_op(purge_legacy_tags)
 
 
 @admin_bp.route("/tags/reset-content", methods=["POST"])
@@ -251,20 +295,13 @@ async def tags_reset_content():
     """Tier-A: delete ALL general + character tags (the Camie-suggestable
     content vocabulary) so the operator can re-tag from scratch via
     auto-suggest. fandom + series tags + series_page ordering are preserved,
-    and image tagger_predictions are untouched so suggestions repopulate.
+    and image_prediction rows are untouched so suggestions repopulate.
     dry-run preview returns per-kind counts + applications + a sample so the
     UI shows exactly what'll go before the operator confirms (dry_run=false).
     Irreversible except via DB backup restore."""
     from ..services.cleanup_service import reset_content_tagging
 
-    body = await request.get_json(silent=True) or {}
-    dry_run = bool(body.get("dry_run", False))
-
-    async with get_session() as session:
-        result = await session.run_sync(
-            lambda sync_sess: reset_content_tagging(sync_sess, dry_run=dry_run)
-        )
-    return jsonify(result)
+    return await _run_dry_run_op(reset_content_tagging)
 
 
 @admin_bp.route("/tags/normalize", methods=["POST"])
@@ -290,7 +327,7 @@ async def tags_normalize():
     from ..tasks.admin import normalize_tags_task
 
     async_result = normalize_tags_task.delay()
-    return jsonify({"task_id": async_result.id, "status": "queued"}), 202
+    return _queued(async_result)
 
 
 @admin_bp.route("/maintenance/db-stats", methods=["GET"])
@@ -347,29 +384,64 @@ async def trigger_reextract_archives():
     from ..tasks.admin import reextract_archive_attachments_task
 
     async_result = reextract_archive_attachments_task.delay()
-    return jsonify({"task_id": async_result.id, "status": "queued"}), 202
+    return _queued(async_result)
 
 
-@admin_bp.route("/maintenance/prune-predictions", methods=["POST"])
-async def trigger_prune_predictions():
-    """Operator-triggered #764 backfill: drop stored tagger predictions below
-    the current ml_settings.tagger_store_floor and clamp allowlist thresholds
-    up to it. Shrinks image_record's TOAST (~100 GB of sub-0.70 scores).
-    Idempotent + self-resuming; runs on the maintenance_long lane."""
-    from ..tasks.admin import prune_low_confidence_predictions_task
+@admin_bp.route("/maintenance/prune-missing-files", methods=["POST"])
+async def trigger_prune_missing_files():
+    """Operator-triggered orphan repair (#859): delete ImageRecords whose backing
+    file is gone from disk (e.g. left by the external-attach unlink bug), so they
+    stop 404-ing on playback. The task aborts WITHOUT deleting if a large fraction
+    of files look missing (a filesystem/NFS stall). Maintenance queue;
+    operator-triggered only — never an unattended sweep."""
+    from ..tasks.admin import prune_missing_file_records_task
 
-    async_result = prune_low_confidence_predictions_task.delay()
-    return jsonify({"task_id": async_result.id, "status": "queued"}), 202
+    async_result = prune_missing_file_records_task.delay()
+    return _queued(async_result)
 
 
-@admin_bp.route("/maintenance/backfill-predictions", methods=["POST"])
-async def trigger_backfill_predictions():
-    """Operator-triggered #768 backfill: copy stored tagger predictions from the
-    image_record.tagger_predictions JSON into the normalized image_prediction
-    table. Batched + resumable + idempotent; runs on the maintenance_long lane.
-    Run this once after deploying migration 0045 (which creates the empty table)
-    to populate predictions for the existing library."""
-    from ..tasks.admin import backfill_image_predictions_task
+@admin_bp.route("/maintenance/dedup-videos", methods=["POST"])
+async def trigger_dedup_videos():
+    """Tier-1 video dedup (#871). Body {"dry_run": bool}: dry_run=true previews
+    what would be removed (groups / redundant count / reclaimable bytes) WITHOUT
+    deleting; dry_run=false applies it (re-link posts to the keeper, then delete
+    the redundant copies). Either way it first re-probes NULL-duration videos so
+    the existing library participates. Returns the Celery task id — poll
+    /maintenance/task-result/<id> for the summary."""
+    from ..tasks.admin import dedup_videos_task
 
-    async_result = backfill_image_predictions_task.delay()
-    return jsonify({"task_id": async_result.id, "status": "queued"}), 202
+    body = await request.get_json(silent=True) or {}
+    dry_run = bool(body.get("dry_run", True))  # default to the SAFE preview
+    async_result = dedup_videos_task.delay(dry_run=dry_run)
+    return _queued(async_result)
+
+
+@admin_bp.route("/maintenance/purge-gated-previews", methods=["POST"])
+async def trigger_purge_gated_previews():
+    """Cleanup (#874 follow-up). Body {"dry_run": bool}: dry_run=true previews how
+    many blurred locked-preview images (grabbed from tier-gated Patreon posts
+    before the fix) would be removed WITHOUT deleting; dry_run=false applies it.
+    Re-walks every enabled Patreon source read-only and matches by content hash, so
+    real content downloaded when access existed is provably spared. Returns the
+    Celery task id — poll /maintenance/task-result/<id> for the summary."""
+    from ..tasks.admin import purge_gated_previews_task
+
+    body = await request.get_json(silent=True) or {}
+    dry_run = bool(body.get("dry_run", True))  # default to the SAFE preview
+    async_result = purge_gated_previews_task.delay(dry_run=dry_run)
+    return _queued(async_result)
+
+
+@admin_bp.route("/maintenance/task-result/<task_id>", methods=["GET"])
+async def maintenance_task_result(task_id: str):
+    """Poll a maintenance Celery task's result (the summary dict it returns).
+    Used by the video-dedup card to show the dry-run projection before apply."""
+    from ..celery_app import celery
+
+    res = celery.AsyncResult(task_id)
+    ready = res.ready()
+    return jsonify({
+        "ready": ready,
+        "successful": res.successful() if ready else None,
+        "result": res.result if (ready and res.successful()) else None,
+    })

backend/app/api/allowlist.py

@@ -20,12 +20,37 @@ async def list_allowlist():
                 "tag_name": r.tag_name,
                 "tag_kind": r.tag_kind,
                 "min_confidence": r.min_confidence,
+                "applied_count": r.applied_count,
+                "coverage_count": r.coverage_count,
             }
             for r in rows
         ]
     )
 
 
+@allowlist_bp.route("/tags/<int:tag_id>/allowlist/coverage", methods=["GET"])
+async def coverage(tag_id: int):
+    """Live "at threshold T, a sweep would cover ~N images" projection for the
+    allowlist tuning dashboard. Defaults to the tag's stored threshold."""
+    raw = request.args.get("threshold")
+    async with get_session() as session:
+        svc = AllowlistService(session)
+        if raw is not None:
+            try:
+                threshold = float(raw)
+            except ValueError:
+                return jsonify({"error": "threshold must be a float"}), 400
+            if not (0 < threshold <= 1):
+                return jsonify({"error": "threshold must be in (0, 1]"}), 400
+        else:
+            row = await session.get(TagAllowlist, tag_id)
+            if row is None:
+                return jsonify({"error": "not on allowlist"}), 404
+            threshold = row.min_confidence
+        count = await svc.coverage(tag_id, threshold)
+    return jsonify({"count": count, "threshold": threshold})
+
+
 @allowlist_bp.route("/tags/<int:tag_id>/allowlist", methods=["GET"])
 async def get_one(tag_id: int):
     async with get_session() as session:

backend/app/api/ccip.py

@@ -0,0 +1,124 @@
+"""CCIP / region observability API (#114) — read-only, analysis-shaped.
+
+So the work can be checked through an API as the agent fills in vectors: overall
+coverage (regions by kind, how many images have figure CCIP vectors, which
+characters have enough reference examples to match on) + a per-image drill-down
+(its regions + the CCIP character matches it would get). Mirrors the heads
+metrics endpoint; no GPU, just reads what's stored.
+"""
+
+from quart import Blueprint, jsonify
+from sqlalchemy import distinct, func, select
+
+from ..extensions import get_session
+from ..models import ImageRegion, Tag, TagKind
+from ..models.tag import image_tag
+from ..services.ml.ccip import match_image
+
+ccip_bp = Blueprint("ccip", __name__, url_prefix="/api/ccip")
+
+_FIGURE_KINDS = ("face", "figure")
+
+
+@ccip_bp.route("/overview", methods=["GET"])
+async def overview():
+    async with get_session() as session:
+        by_kind = dict(
+            (
+                await session.execute(
+                    select(ImageRegion.kind, func.count()).group_by(ImageRegion.kind)
+                )
+            ).all()
+        )
+        images_with_figure_ccip = (
+            await session.execute(
+                select(func.count(distinct(ImageRegion.image_record_id)))
+                .where(ImageRegion.kind.in_(_FIGURE_KINDS))
+                .where(ImageRegion.ccip_embedding.is_not(None))
+            )
+        ).scalar_one()
+        # Concept-crop (SigLIP bag) coverage — how far the back-catalogue embed
+        # has progressed, so the max-over-bag scorer's reach is checkable.
+        images_with_concept_siglip = (
+            await session.execute(
+                select(func.count(distinct(ImageRegion.image_record_id)))
+                .where(ImageRegion.kind == "concept")
+                .where(ImageRegion.siglip_embedding.is_not(None))
+            )
+        ).scalar_one()
+        # Per-character reference counts (no vectors loaded) — which characters
+        # have enough examples to match on.
+        ref_rows = (
+            await session.execute(
+                select(image_tag.c.tag_id, Tag.name, func.count())
+                .select_from(ImageRegion)
+                .join(
+                    image_tag,
+                    image_tag.c.image_record_id == ImageRegion.image_record_id,
+                )
+                .join(Tag, Tag.id == image_tag.c.tag_id)
+                .where(Tag.kind == TagKind.character)
+                .where(ImageRegion.kind.in_(_FIGURE_KINDS))
+                .where(ImageRegion.ccip_embedding.is_not(None))
+                .group_by(image_tag.c.tag_id, Tag.name)
+                .order_by(func.count().desc())
+            )
+        ).all()
+        versions = [
+            v for (v,) in (
+                await session.execute(
+                    select(distinct(ImageRegion.embedding_version))
+                )
+            ).all() if v
+        ]
+        auto_applied = (
+            await session.execute(
+                select(func.count()).select_from(image_tag).where(
+                    image_tag.c.source == "ccip_auto"
+                )
+            )
+        ).scalar_one()
+    return jsonify({
+        "regions_by_kind": by_kind,
+        "images_with_figure_ccip": images_with_figure_ccip,
+        "images_with_concept_siglip": images_with_concept_siglip,
+        "characters_with_references": len(ref_rows),
+        "character_references": [
+            {"tag_id": t, "name": n, "n_refs": c} for (t, n, c) in ref_rows
+        ],
+        "embedding_versions": versions,
+        "auto_applied": auto_applied,
+    })
+
+
+@ccip_bp.route("/images/<int:image_id>", methods=["GET"])
+async def image_detail(image_id: int):
+    """An image's stored regions + the CCIP character matches it would get —
+    for spot-checking the agent's output + the matcher."""
+    async with get_session() as session:
+        regions = (
+            await session.execute(
+                select(ImageRegion)
+                .where(ImageRegion.image_record_id == image_id)
+                .order_by(ImageRegion.id)
+            )
+        ).scalars().all()
+        matches = await match_image(session, image_id)
+    return jsonify({
+        "image_id": image_id,
+        "regions": [
+            {
+                "id": r.id,
+                "kind": r.kind,
+                "bbox": [r.rx, r.ry, r.rw, r.rh],
+                "frame_time": r.frame_time,
+                "score": r.score,
+                "detector_version": r.detector_version,
+                "embedding_version": r.embedding_version,
+                "has_ccip": r.ccip_embedding is not None,
+                "has_siglip": r.siglip_embedding is not None,
+            }
+            for r in regions
+        ],
+        "ccip_matches": matches,
+    })

backend/app/api/gallery.py

@@ -37,16 +37,30 @@ def _parse_filters():
     """Parse the composable gallery filters from query args, returning
     ``(filters_dict, sort)``. Raises ValueError (→ 400) on malformed ids/dates.
 
-    `tag_id` accepts a single id or a comma-separated list (AND); `media` is
-    image|video; `sort` is newest|oldest; `platform` selects one platform
-    (or the UNSOURCED_PLATFORM sentinel); `untagged`/`no_artist` are boolean
-    flags; `date_from`/`date_to` are inclusive calendar-day bounds (date_to is
-    widened by a day so the whole day is covered by the service's half-open
-    `< date_to`)."""
+    The structured tag filter (#6) is AND-of-OR plus exclusions:
+    - `tag_id` accepts a single id or a comma-separated list — all ANDed
+      (the include common case; back-compat).
+    - `tag_or` is REPEATABLE; each instance is a comma-separated OR-group, and
+      the image must match at least one tag from EACH group (groups ANDed).
+    - `tag_not` is a comma-separated exclude list (image must carry none).
+
+    `media` is image|video; `sort` is newest|oldest; `platform` selects one
+    platform (or the UNSOURCED_PLATFORM sentinel); `untagged`/`no_artist` are
+    boolean flags; `date_from`/`date_to` are inclusive calendar-day bounds
+    (date_to is widened by a day so the whole day is covered by the service's
+    half-open `< date_to`)."""
     tag_raw = request.args.get("tag_id")
     tag_ids = (
         [int(x) for x in tag_raw.split(",") if x.strip()] if tag_raw else None
     ) or None
+    tag_or_groups = [
+        grp for raw in request.args.getlist("tag_or")
+        if (grp := [int(x) for x in raw.split(",") if x.strip()])
+    ] or None
+    not_raw = request.args.get("tag_not")
+    tag_exclude = (
+        [int(x) for x in not_raw.split(",") if x.strip()] if not_raw else None
+    ) or None
     post_id_raw = request.args.get("post_id")
     post_id = int(post_id_raw) if post_id_raw else None
     artist_id_raw = request.args.get("artist_id")
@@ -64,7 +78,9 @@ def _parse_filters():
         date_to += timedelta(days=1)  # inclusive of the date_to calendar day
     filters = {
         "tag_ids": tag_ids, "post_id": post_id, "artist_id": artist_id,
-        "media_type": media_type, "platform": platform,
+        "media_type": media_type,
+        "tag_or_groups": tag_or_groups, "tag_exclude": tag_exclude,
+        "platform": platform,
         "untagged": untagged, "no_artist": no_artist,
         "date_from": date_from, "date_to": date_to,
     }

backend/app/api/gpu.py

@@ -0,0 +1,220 @@
+"""GPU-job API (#114): the HTTP surface the desktop agent pulls work from.
+
+The agent stays HTTP-only — it leases jobs, fetches image pixels via the normal
+FC image URLs, and submits embeddings/regions back, all over this API. Redis and
+Postgres are never exposed. The agent endpoints are gated by a bearer token
+(Authorization: Bearer <token>) stored in AppSetting; the admin endpoints
+(token / backfill / status) ride the browser session like the rest of FC's
+homelab admin.
+"""
+
+import secrets
+
+from quart import Blueprint, jsonify, request
+from sqlalchemy import func, select
+from sqlalchemy.dialects.postgresql import insert as pg_insert
+
+from ..extensions import get_session
+from ..models import AppSetting, GpuJob, ImageRecord, MLSettings
+from ..services.gallery_service import image_url
+from ..services.ml.embedder import MODEL_NAME as EMBED_MODEL_NAME
+from ..services.ml.gpu_jobs import GpuJobService
+from ..services.ml.regions import RegionService
+
+gpu_bp = Blueprint("gpu", __name__, url_prefix="/api/gpu")
+
+_TOKEN_KEY = "gpu_agent_token"
+
+
+def _bearer() -> str | None:
+    h = request.headers.get("Authorization", "")
+    return h[7:].strip() if h.startswith("Bearer ") else None
+
+
+async def _agent_authed(session) -> bool:
+    supplied = _bearer()
+    if not supplied:
+        return False
+    stored = (
+        await session.execute(
+            select(AppSetting.value).where(AppSetting.key == _TOKEN_KEY)
+        )
+    ).scalar_one_or_none()
+    return stored is not None and secrets.compare_digest(supplied, stored)
+
+
+# --- Admin (browser): token + backfill + status -------------------------
+
+@gpu_bp.route("/token", methods=["GET"])
+async def get_token():
+    async with get_session() as session:
+        tok = (
+            await session.execute(
+                select(AppSetting.value).where(AppSetting.key == _TOKEN_KEY)
+            )
+        ).scalar_one_or_none()
+    return jsonify({"token": tok, "configured": tok is not None})
+
+
+@gpu_bp.route("/token/rotate", methods=["POST"])
+async def rotate_token():
+    token = secrets.token_urlsafe(32)
+    async with get_session() as session:
+        await session.execute(
+            pg_insert(AppSetting)
+            .values(key=_TOKEN_KEY, value=token)
+            .on_conflict_do_update(index_elements=["key"], set_={"value": token})
+        )
+        await session.commit()
+    return jsonify({"token": token})
+
+
+@gpu_bp.route("/status", methods=["GET"])
+async def status():
+    async with get_session() as session:
+        rows = (
+            await session.execute(
+                select(GpuJob.status, func.count()).group_by(GpuJob.status)
+            )
+        ).all()
+    counts = dict(rows)
+    return jsonify({
+        "pending": counts.get("pending", 0),
+        "leased": counts.get("leased", 0),
+        "done": counts.get("done", 0),
+        "error": counts.get("error", 0),
+    })
+
+
+@gpu_bp.route("/backfill", methods=["POST"])
+async def backfill():
+    """Enqueue a job for every image that doesn't already have one for `task`."""
+    body = await request.get_json(silent=True) or {}
+    task = str(body.get("task") or "ccip")
+    from ..tasks.ml import enqueue_gpu_backfill
+
+    r = enqueue_gpu_backfill.delay(task)
+    return jsonify({"celery_task_id": r.id, "task": task}), 202
+
+
+# --- Agent (bearer token): lease / submit / heartbeat / fail ------------
+
+@gpu_bp.route("/jobs/lease", methods=["POST"])
+async def lease():
+    body = await request.get_json(silent=True) or {}
+    agent_id = str(body.get("agent_id") or "agent")
+    try:
+        batch = min(max(int(body.get("batch_size", 8)), 1), 64)
+    except (TypeError, ValueError):
+        batch = 8
+    async with get_session() as session:
+        if not await _agent_authed(session):
+            return jsonify({"error": "unauthorized"}), 401
+        jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
+        ml = (
+            await session.execute(select(MLSettings).where(MLSettings.id == 1))
+        ).scalar_one()
+        # image rows for url/mime in one shot
+        ids = [j.image_record_id for j in jobs]
+        imgs = {
+            i.id: i for i in (
+                await session.execute(
+                    select(ImageRecord).where(ImageRecord.id.in_(ids))
+                )
+            ).scalars()
+        } if ids else {}
+        await session.commit()
+        out = []
+        for j in jobs:
+            img = imgs.get(j.image_record_id)
+            if img is None:
+                continue
+            out.append({
+                "job_id": j.id,
+                "image_id": j.image_record_id,
+                "task": j.task,
+                "mime": img.mime,
+                "image_url": image_url(img.path),
+                # For video/animated: the agent samples at this cadence.
+                "frame_interval_seconds": ml.video_frame_interval_seconds,
+                "max_frames": ml.video_max_frames,
+                # The embedding model the agent must use for concept crops, so
+                # its region vectors land in the SAME space the heads trained in.
+                # Server-announced → the agent stays model-agnostic; a swap is a
+                # server setting + a re-embed migration, never an agent change.
+                "embed_model_name": EMBED_MODEL_NAME,
+                "embed_version": ml.embedder_model_version,
+            })
+    return jsonify({"jobs": out})
+
+
+@gpu_bp.route("/jobs/heartbeat", methods=["POST"])
+async def heartbeat():
+    body = await request.get_json(silent=True) or {}
+    agent_id = str(body.get("agent_id") or "agent")
+    job_ids = [int(x) for x in (body.get("job_ids") or [])]
+    async with get_session() as session:
+        if not await _agent_authed(session):
+            return jsonify({"error": "unauthorized"}), 401
+        n = await GpuJobService(session).heartbeat(agent_id, job_ids)
+        await session.commit()
+    return jsonify({"extended": n})
+
+
+@gpu_bp.route("/jobs/submit", methods=["POST"])
+async def submit():
+    """Store a job's regions + close it. regions: [{kind, bbox:[x,y,w,h],
+    frame_time?, score?, *_version?, ccip_embedding?, siglip_embedding?}].
+    replace_kinds defaults to the kinds present in the submitted regions."""
+    body = await request.get_json(silent=True) or {}
+    agent_id = str(body.get("agent_id") or "agent")
+    job_id = body.get("job_id")
+    regions = body.get("regions") or []
+    if job_id is None:
+        return jsonify({"error": "job_id required"}), 400
+    kinds = body.get("replace_kinds") or sorted({r["kind"] for r in regions})
+    async with get_session() as session:
+        if not await _agent_authed(session):
+            return jsonify({"error": "unauthorized"}), 401
+        job = await session.get(GpuJob, int(job_id))
+        if job is None or job.status != "leased" or job.lease_token != agent_id:
+            return jsonify({"error": "lease_invalid"}), 409
+        if kinds:
+            await RegionService(session).replace_regions(
+                job.image_record_id, kinds, regions
+            )
+        await GpuJobService(session).complete(agent_id, int(job_id))
+        await session.commit()
+    return jsonify({"ok": True, "stored": len(regions)})
+
+
+@gpu_bp.route("/jobs/fail", methods=["POST"])
+async def fail():
+    body = await request.get_json(silent=True) or {}
+    agent_id = str(body.get("agent_id") or "agent")
+    job_id = body.get("job_id")
+    if job_id is None:
+        return jsonify({"error": "job_id required"}), 400
+    async with get_session() as session:
+        if not await _agent_authed(session):
+            return jsonify({"error": "unauthorized"}), 401
+        ok = await GpuJobService(session).fail(
+            agent_id, int(job_id), str(body.get("error") or "")
+        )
+        await session.commit()
+    return jsonify({"ok": ok})
+
+
+@gpu_bp.route("/jobs/release", methods=["POST"])
+async def release():
+    """Graceful stop: the agent hands its still-leased jobs back to pending so
+    they're picked up immediately instead of waiting out the lease."""
+    body = await request.get_json(silent=True) or {}
+    agent_id = str(body.get("agent_id") or "agent")
+    job_ids = [int(x) for x in (body.get("job_ids") or [])]
+    async with get_session() as session:
+        if not await _agent_authed(session):
+            return jsonify({"error": "unauthorized"}), 401
+        n = await GpuJobService(session).release(agent_id, job_ids)
+        await session.commit()
+    return jsonify({"released": n})

backend/app/api/heads.py

@@ -0,0 +1,285 @@
+"""Heads API (#114): train + inspect the per-concept heads that power
+suggestions (replacing Camie + centroid).
+
+POST /api/heads/train   — (re)train all eligible heads (one run at a time).
+GET  /api/heads         — status: head count, last-trained, running run, the
+                          per-concept head table (strength + auto-apply ready),
+                          and recent training runs. The card rehydrates from
+                          here so status survives navigation.
+"""
+
+from quart import Blueprint, jsonify, request
+from sqlalchemy import desc, func, select
+
+from ..extensions import get_session
+from ..models import (
+    HeadAutoApplyRun,
+    HeadMetric,
+    HeadMetricsSnapshot,
+    HeadTrainingRun,
+    Tag,
+    TagHead,
+)
+from ..models.tag import image_tag
+from ..services.ml.heads import (
+    HeadAutoApplyAlreadyRunning,
+    HeadAutoApplyDisabled,
+    HeadTrainingAlreadyRunning,
+    start_head_auto_apply_run,
+    start_head_training_run,
+)
+
+heads_bp = Blueprint("heads", __name__, url_prefix="/api/heads")
+
+
+def _serialize_run(run: HeadTrainingRun) -> dict:
+    return {
+        "id": run.id,
+        "params": run.params,
+        "status": run.status,
+        "started_at": run.started_at.isoformat() if run.started_at else None,
+        "finished_at": run.finished_at.isoformat() if run.finished_at else None,
+        "n_trained": run.n_trained,
+        "n_skipped": run.n_skipped,
+        "error": run.error,
+    }
+
+
+@heads_bp.route("/train", methods=["POST"])
+async def train():
+    body = await request.get_json(silent=True) or {}
+    params = body.get("params") or body or {}
+    async with get_session() as session:
+        try:
+            run_id = await session.run_sync(
+                lambda s: start_head_training_run(s, params)
+            )
+        except HeadTrainingAlreadyRunning as running:
+            return jsonify({
+                "error": "training_already_running",
+                "running_id": int(running.args[0]),
+            }), 409
+        await session.commit()
+    return jsonify({"run_id": run_id, "status": "running"}), 202
+
+
+@heads_bp.route("", methods=["GET"])
+async def status():
+    async with get_session() as session:
+        count, last_trained = (
+            await session.execute(
+                select(func.count(), func.max(TagHead.trained_at))
+            )
+        ).one()
+        graduated = (
+            await session.execute(
+                select(func.count()).where(
+                    TagHead.auto_apply_threshold.is_not(None)
+                )
+            )
+        ).scalar_one()
+        running = (
+            await session.execute(
+                select(HeadTrainingRun.id)
+                .where(HeadTrainingRun.status == "running")
+                .order_by(HeadTrainingRun.id.desc())
+                .limit(1)
+            )
+        ).scalar_one_or_none()
+        runs = (
+            await session.execute(
+                select(HeadTrainingRun)
+                .order_by(HeadTrainingRun.id.desc())
+                .limit(10)
+            )
+        ).scalars().all()
+        # The per-concept table: strongest first, capped for the admin card.
+        head_rows = (
+            await session.execute(
+                select(
+                    TagHead.tag_id, Tag.name, Tag.kind,
+                    TagHead.n_pos, TagHead.n_neg, TagHead.ap,
+                    TagHead.precision_cv, TagHead.recall,
+                    TagHead.auto_apply_threshold, TagHead.trained_at,
+                )
+                .join(Tag, Tag.id == TagHead.tag_id)
+                .order_by(desc(TagHead.ap))
+                .limit(500)
+            )
+        ).all()
+        heads = [
+            {
+                "tag_id": r.tag_id,
+                "name": r.name,
+                "category": r.kind.value if hasattr(r.kind, "value") else str(r.kind),
+                "n_pos": r.n_pos,
+                "n_neg": r.n_neg,
+                "ap": r.ap,
+                "precision": r.precision_cv,
+                "recall": r.recall,
+                "auto_apply": r.auto_apply_threshold is not None,
+                "trained_at": r.trained_at.isoformat() if r.trained_at else None,
+            }
+            for r in head_rows
+        ]
+    return jsonify({
+        "head_count": count,
+        "graduated_count": graduated,
+        "last_trained_at": last_trained.isoformat() if last_trained else None,
+        "running_id": running,
+        "runs": [_serialize_run(r) for r in runs],
+        "heads": heads,
+    })
+
+
+def _serialize_apply_run(run: HeadAutoApplyRun) -> dict:
+    return {
+        "id": run.id,
+        "dry_run": run.dry_run,
+        "status": run.status,
+        "started_at": run.started_at.isoformat() if run.started_at else None,
+        "finished_at": run.finished_at.isoformat() if run.finished_at else None,
+        "n_applied": run.n_applied,
+        "report": run.report,
+        "error": run.error,
+    }
+
+
+@heads_bp.route("/auto-apply", methods=["POST"])
+async def auto_apply():
+    """Trigger an earned-auto-apply sweep. {dry_run:true} previews (writes
+    nothing); a real sweep needs head_auto_apply_enabled on."""
+    body = await request.get_json(silent=True) or {}
+    params = {"dry_run": bool(body.get("dry_run", False))}
+    async with get_session() as session:
+        try:
+            run_id = await session.run_sync(
+                lambda s: start_head_auto_apply_run(s, params)
+            )
+        except HeadAutoApplyAlreadyRunning as running:
+            return jsonify({
+                "error": "auto_apply_already_running",
+                "running_id": int(running.args[0]),
+            }), 409
+        except HeadAutoApplyDisabled:
+            return jsonify({"error": "auto_apply_disabled"}), 400
+        await session.commit()
+    return jsonify({"run_id": run_id, "status": "running"}), 202
+
+
+@heads_bp.route("/auto-apply", methods=["GET"])
+async def auto_apply_status():
+    async with get_session() as session:
+        running = (
+            await session.execute(
+                select(HeadAutoApplyRun.id)
+                .where(HeadAutoApplyRun.status == "running")
+                .order_by(HeadAutoApplyRun.id.desc())
+                .limit(1)
+            )
+        ).scalar_one_or_none()
+        runs = (
+            await session.execute(
+                select(HeadAutoApplyRun)
+                .order_by(HeadAutoApplyRun.id.desc())
+                .limit(10)
+            )
+        ).scalars().all()
+    return jsonify({
+        "running_id": running,
+        "runs": [_serialize_apply_run(r) for r in runs],
+    })
+
+
+@heads_bp.route("/metrics", methods=["GET"])
+async def metrics():
+    """Auto-apply observability: per-concept current counts (volume, misfires,
+    under-fires, realized misfire rate, head quality) + the daily time-series so
+    the operator can tune the precision target + support floor from real data."""
+    async with get_session() as session:
+        head_rows = (
+            await session.execute(
+                select(
+                    TagHead.tag_id, Tag.name, TagHead.ap, TagHead.precision_cv,
+                    TagHead.recall, TagHead.auto_apply_threshold, TagHead.n_pos,
+                ).join(Tag, Tag.id == TagHead.tag_id)
+            )
+        ).all()
+        heads = {r.tag_id: r for r in head_rows}
+        metric_rows = (
+            await session.execute(
+                select(
+                    HeadMetric.tag_id, HeadMetric.n_misfires, HeadMetric.n_underfires
+                )
+            )
+        ).all()
+        mets = {r.tag_id: r for r in metric_rows}
+        applied = dict(
+            (
+                await session.execute(
+                    select(image_tag.c.tag_id, func.count())
+                    .where(image_tag.c.source == "head_auto")
+                    .group_by(image_tag.c.tag_id)
+                )
+            ).all()
+        )
+        names = {r.tag_id: r.name for r in head_rows}
+        # Names for metric-only tags (head pruned but corrections recorded).
+        missing = [t for t in mets if t not in names]
+        if missing:
+            for tid, nm in (
+                await session.execute(
+                    select(Tag.id, Tag.name).where(Tag.id.in_(missing))
+                )
+            ).all():
+                names[tid] = nm
+
+        concepts = []
+        for tid in set(heads) | set(mets):
+            h = heads.get(tid)
+            m = mets.get(tid)
+            n_applied = applied.get(tid, 0)
+            n_mis = m.n_misfires if m else 0
+            denom = n_applied + n_mis
+            concepts.append({
+                "tag_id": tid,
+                "name": names.get(tid, str(tid)),
+                "n_auto_applied": n_applied,
+                "n_misfires": n_mis,
+                "n_underfires": m.n_underfires if m else 0,
+                # Of everything this head ever auto-applied, the fraction you
+                # removed — the misfire rate (null until something fired).
+                "misfire_rate": round(n_mis / denom, 4) if denom else None,
+                "ap": h.ap if h else None,
+                "precision_cv": h.precision_cv if h else None,
+                "recall": h.recall if h else None,
+                "auto_apply": bool(h and h.auto_apply_threshold is not None),
+                "n_pos": h.n_pos if h else None,
+            })
+        concepts.sort(key=lambda c: (c["n_misfires"], c["n_auto_applied"]), reverse=True)
+
+        snaps = (
+            await session.execute(
+                select(HeadMetricsSnapshot)
+                .order_by(HeadMetricsSnapshot.snapshot_at.desc())
+                .limit(1000)
+            )
+        ).scalars().all()
+    return jsonify({
+        "concepts": concepts,
+        "snapshots": [
+            {
+                "tag_id": s.tag_id,
+                "name": s.name,
+                "snapshot_at": s.snapshot_at.isoformat() if s.snapshot_at else None,
+                "n_auto_applied": s.n_auto_applied,
+                "n_misfires": s.n_misfires,
+                "n_underfires": s.n_underfires,
+                "ap": s.ap,
+                "precision_cv": s.precision_cv,
+                "recall": s.recall,
+                "n_pos": s.n_pos,
+            }
+            for s in snaps
+        ],
+    })

backend/app/api/ml_admin.py

@@ -14,6 +14,16 @@ _EDITABLE = (
     "centroid_similarity_threshold",
     "min_reference_images",
     "tagger_store_floor",
+    "video_frame_interval_seconds",
+    "video_max_frames",
+    "video_min_tag_frames",
+    "head_min_positives",
+    "head_auto_apply_precision",
+    "head_auto_apply_enabled",
+    "head_auto_apply_min_positives",
+    "ccip_match_threshold",
+    "ccip_auto_apply_enabled",
+    "ccip_auto_apply_threshold",
 )
 
 
@@ -32,8 +42,18 @@ async def get_settings():
             "centroid_similarity_threshold": s.centroid_similarity_threshold,
             "min_reference_images": s.min_reference_images,
             "tagger_store_floor": s.tagger_store_floor,
+            "video_frame_interval_seconds": s.video_frame_interval_seconds,
+            "video_max_frames": s.video_max_frames,
+            "video_min_tag_frames": s.video_min_tag_frames,
             "tagger_model_version": s.tagger_model_version,
             "embedder_model_version": s.embedder_model_version,
+            "head_min_positives": s.head_min_positives,
+            "head_auto_apply_precision": s.head_auto_apply_precision,
+            "head_auto_apply_enabled": s.head_auto_apply_enabled,
+            "head_auto_apply_min_positives": s.head_auto_apply_min_positives,
+            "ccip_match_threshold": s.ccip_match_threshold,
+            "ccip_auto_apply_enabled": s.ccip_auto_apply_enabled,
+            "ccip_auto_apply_threshold": s.ccip_auto_apply_threshold,
         }
     )
 
@@ -85,6 +105,26 @@ def _validate(p: dict) -> str | None:
                 f"suggestion_threshold_{cat} cannot be below tagger_store_floor "
                 f"({floor}) — predictions below the floor are not stored"
             )
+    # Video tagging (#747).
+    if p["video_frame_interval_seconds"] <= 0:
+        return "video_frame_interval_seconds must be > 0"
+    if p["video_max_frames"] < 1:
+        return "video_max_frames must be >= 1"
+    if p["video_min_tag_frames"] < 1:
+        return "video_min_tag_frames must be >= 1"
+    if p["video_min_tag_frames"] > p["video_max_frames"]:
+        return "video_min_tag_frames cannot exceed video_max_frames"
+    # Head training (#114).
+    if int(p["head_min_positives"]) < 1:
+        return "head_min_positives must be >= 1"
+    if not (0.5 <= float(p["head_auto_apply_precision"]) <= 0.999):
+        return "head_auto_apply_precision must be between 0.5 and 0.999"
+    if int(p["head_auto_apply_min_positives"]) < 1:
+        return "head_auto_apply_min_positives must be >= 1"
+    if not (0.5 <= float(p["ccip_match_threshold"]) <= 0.999):
+        return "ccip_match_threshold must be between 0.5 and 0.999"
+    if not (0.5 <= float(p["ccip_auto_apply_threshold"]) <= 0.999):
+        return "ccip_auto_apply_threshold must be between 0.5 and 0.999"
     return None
 
 

backend/app/api/posts.py

@@ -17,6 +17,7 @@ async def list_posts():
     cursor = args.get("cursor") or None
     artist_id_raw = args.get("artist_id")
     platform = args.get("platform") or None
+    q = (args.get("q") or "").strip() or None
     limit_raw = args.get("limit", "24")
     direction = args.get("direction", "older")
     around_raw = args.get("around")
@@ -56,7 +57,7 @@ async def list_posts():
         if around_id is not None:
             result = await svc.around(
                 post_id=around_id, artist_id=artist_id,
-                platform=platform, limit=limit,
+                platform=platform, q=q, limit=limit,
             )
             if result is None:
                 return _bad("not_found", status=404, detail=f"post id={around_id}")
@@ -64,7 +65,7 @@ async def list_posts():
         try:
             page = await svc.scroll(
                 cursor=cursor, artist_id=artist_id,
-                platform=platform, limit=limit, direction=direction,
+                platform=platform, q=q, limit=limit, direction=direction,
             )
         except ValueError as exc:
             # Service raises ValueError for malformed cursors only;

backend/app/api/settings.py

@@ -27,6 +27,20 @@ _EDITABLE_FIELDS = (
     "download_failure_warning_threshold",
     "series_suggest_enabled",
     "series_suggest_threshold",
+    "extdl_mega_enabled",
+    "extdl_gdrive_enabled",
+    "extdl_mediafire_enabled",
+    "extdl_dropbox_enabled",
+    "extdl_pixeldrain_enabled",
+)
+
+# Per-host external-download toggles — all plain booleans, validated uniformly.
+_EXTDL_TOGGLE_FIELDS = (
+    "extdl_mega_enabled",
+    "extdl_gdrive_enabled",
+    "extdl_mediafire_enabled",
+    "extdl_dropbox_enabled",
+    "extdl_pixeldrain_enabled",
 )
 
 
@@ -50,6 +64,11 @@ async def get_import_settings():
         "download_failure_warning_threshold": row.download_failure_warning_threshold,
         "series_suggest_enabled": row.series_suggest_enabled,
         "series_suggest_threshold": row.series_suggest_threshold,
+        "extdl_mega_enabled": row.extdl_mega_enabled,
+        "extdl_gdrive_enabled": row.extdl_gdrive_enabled,
+        "extdl_mediafire_enabled": row.extdl_mediafire_enabled,
+        "extdl_dropbox_enabled": row.extdl_dropbox_enabled,
+        "extdl_pixeldrain_enabled": row.extdl_pixeldrain_enabled,
     })
 
 
@@ -106,6 +125,9 @@ async def update_import_settings():
         return jsonify(
             {"error": "series_suggest_enabled must be a boolean"}
         ), 400
+    for tog in _EXTDL_TOGGLE_FIELDS:
+        if tog in body and not isinstance(body[tog], bool):
+            return jsonify({"error": f"{tog} must be a boolean"}), 400
     if "series_suggest_threshold" in body:
         v = body["series_suggest_threshold"]
         if not isinstance(v, (int, float)) or isinstance(v, bool) or v < 0 or v > 1:

backend/app/api/sources.py

@@ -138,14 +138,16 @@ async def delete_source(source_id: int):
 
 @sources_bp.route("/<int:source_id>/backfill", methods=["POST"])
 async def set_backfill(source_id: int):
-    """Plan #693/#697: start/stop a run-until-done backfill, or start a recovery.
-    Body: `{"action": "start" | "stop" | "recover"}` (default "start"). 'start'
-    walks the full post history in time-boxed chunks until it reaches the bottom
-    (then the source shows 'complete'); 'recover' is the same walk but bypasses
-    the Patreon seen-ledger to re-fetch dropped-and-deleted near-dups under the
-    current pHash threshold; 'stop' cancels either back to tick mode. Returns the
-    updated source dict (incl. backfill_state / backfill_chunks /
-    backfill_bypass_seen)."""
+    """Plan #693/#697 + #830: start/stop a backfill, or start a recovery /
+    recapture. Body: `{"action": "start" | "stop" | "recover" | "recapture"}`
+    (default "start"). 'start' walks the full post history in time-boxed chunks
+    until it reaches the bottom (then the source shows 'complete'); 'recover' is
+    the same walk but bypasses the Patreon seen-ledger to re-fetch
+    dropped-and-deleted near-dups under the current pHash threshold; 'recapture'
+    re-grabs EVERY post's body + external links and localizes on-disk inline
+    images WITHOUT re-downloading media; 'stop' cancels any back to tick mode.
+    Returns the updated source dict (incl. backfill_state / backfill_chunks /
+    backfill_bypass_seen / backfill_recapture)."""
     from pathlib import Path
 
     from ..services.credential_service import CredentialService
@@ -157,10 +159,10 @@ async def set_backfill(source_id: int):
 
     payload = await request.get_json(silent=True) or {}
     action = payload.get("action", "start")
-    if action not in ("start", "stop", "recover"):
+    if action not in ("start", "stop", "recover", "recapture"):
         return _bad(
             "invalid_action",
-            detail="action must be 'start', 'stop', or 'recover'",
+            detail="action must be 'start', 'stop', 'recover', or 'recapture'",
         )
 
     # Pre-flight (plan #703 #2): before arming a deep walk on a native-ingester
@@ -170,7 +172,7 @@ async def set_backfill(source_id: int):
     # platforms: gallery-dl verify is a slow --simulate subprocess, too heavy for
     # an arm action. The credential read happens in a session that's CLOSED
     # before the verify network call (don't hold a DB conn across the request).
-    if action in ("start", "recover"):
+    if action in ("start", "recover", "recapture"):
         async with get_session() as session:
             rec = await SourceService(session).get(source_id)
             if rec is None:
@@ -200,6 +202,8 @@ async def set_backfill(source_id: int):
                 record = await svc.start_backfill(source_id)
             elif action == "recover":
                 record = await svc.start_recovery(source_id)
+            elif action == "recapture":
+                record = await svc.start_recapture(source_id)
             else:
                 record = await svc.stop_backfill(source_id)
         except LookupError:

backend/app/api/suggestions.py

@@ -3,12 +3,31 @@
 from quart import Blueprint, jsonify, request
 
 from ..extensions import get_session
+from ..models import Tag, TagAllowlist
 from ..services.ml.allowlist import AllowlistService
 from ..services.ml.suggestions import SuggestionService
 
 suggestions_bp = Blueprint("suggestions", __name__, url_prefix="/api")
 
 
+async def _accept_payload(session, svc, newly_added: bool, tag_id: int) -> dict:
+    """Shape the accept/alias response. When accepting newly allowlists a tag,
+    include the coverage PROJECTION (at the tag's threshold) so the UI can show
+    a non-blocking "auto-applying to ~N images" toast — the actual apply runs
+    async via apply_allowlist_tags, so this is an estimate, not a post-hoc
+    count (#7)."""
+    payload = {"allowlisted": newly_added}
+    if newly_added:
+        tag = await session.get(Tag, tag_id)
+        row = await session.get(TagAllowlist, tag_id)
+        payload["tag_id"] = tag_id
+        payload["tag_name"] = tag.name if tag is not None else None
+        payload["projected_count"] = await svc.coverage(
+            tag_id, row.min_confidence if row is not None else 0.90,
+        )
+    return payload
+
+
 @suggestions_bp.route("/images/<int:image_id>/suggestions", methods=["GET"])
 async def get_suggestions(image_id: int):
     # ?min=<float> overrides the configured per-category thresholds so the typed
@@ -37,6 +56,15 @@ async def get_suggestions(image_id: int):
                         "score": round(s.score, 4),
                         "source": s.source,
                         "creates_new_tag": s.creates_new_tag,
+                        # raw model key (alias is stored under this) + whether an
+                        # operator alias produced this suggestion — drive the
+                        # modal's "Treat as alias"/"Remove alias" affordances.
+                        "raw_name": s.raw_name,
+                        "via_alias": s.via_alias,
+                        # operator dismissed this tag for this image — surfaced
+                        # (not dropped) so the rail can show it rejected + offer
+                        # one-click un-reject.
+                        "rejected": s.rejected,
                     }
                     for s in items
                 ]
@@ -55,13 +83,15 @@ async def accept_suggestion(image_id: int):
         return jsonify({"error": "tag_id required"}), 400
     tag_id = body["tag_id"]
     async with get_session() as session:
-        newly_added = await AllowlistService(session).accept(image_id, tag_id)
+        svc = AllowlistService(session)
+        newly_added = await svc.accept(image_id, tag_id)
+        payload = await _accept_payload(session, svc, newly_added, tag_id)
         await session.commit()
     if newly_added:
         from ..tasks.ml import apply_allowlist_tags
 
         apply_allowlist_tags.delay(tag_id=tag_id)
-    return "", 204
+    return jsonify(payload)
 
 
 @suggestions_bp.route(
@@ -72,19 +102,24 @@ async def alias_suggestion(image_id: int):
     required = {"alias_string", "alias_category", "canonical_tag_id"}
     if not body or not required.issubset(body):
         return jsonify({"error": f"required: {sorted(required)}"}), 400
+    canonical_tag_id = body["canonical_tag_id"]
     async with get_session() as session:
-        newly_added = await AllowlistService(session).add_alias_and_accept(
+        svc = AllowlistService(session)
+        newly_added = await svc.add_alias_and_accept(
             image_id,
             body["alias_string"],
             body["alias_category"],
-            body["canonical_tag_id"],
+            canonical_tag_id,
+        )
+        payload = await _accept_payload(
+            session, svc, newly_added, canonical_tag_id,
         )
         await session.commit()
     if newly_added:
         from ..tasks.ml import apply_allowlist_tags
 
-        apply_allowlist_tags.delay(tag_id=body["canonical_tag_id"])
-    return "", 204
+        apply_allowlist_tags.delay(tag_id=canonical_tag_id)
+    return jsonify(payload)
 
 
 @suggestions_bp.route(
@@ -100,6 +135,21 @@ async def dismiss_suggestion(image_id: int):
     return "", 204
 
 
+@suggestions_bp.route(
+    "/images/<int:image_id>/suggestions/undismiss", methods=["POST"]
+)
+async def undismiss_suggestion(image_id: int):
+    """Reverse a per-image dismissal (reject-recovery). Idempotent — undoing a
+    tag that isn't rejected is a no-op delete."""
+    body = await request.get_json()
+    if not body or "tag_id" not in body:
+        return jsonify({"error": "tag_id required"}), 400
+    async with get_session() as session:
+        await AllowlistService(session).undismiss(image_id, body["tag_id"])
+        await session.commit()
+    return "", 204
+
+
 @suggestions_bp.route("/suggestions/bulk", methods=["POST"])
 async def bulk_suggestions():
     body = await request.get_json()

backend/app/api/tag_eval.py

@@ -0,0 +1,70 @@
+"""Tag-eval API (#1130): trigger + revisit the head-vs-centroid eval.
+
+The run + full report live in the tag_eval_run row, so the admin card rehydrates
+from GET (history / detail) on mount — the report survives navigation rather than
+living in transient frontend state.
+"""
+
+from quart import Blueprint, jsonify, request
+from sqlalchemy import select
+
+from ..extensions import get_session
+from ..models import TagEvalRun
+from ..services.ml.tag_eval import EvalAlreadyRunning, start_tag_eval_run
+
+tag_eval_bp = Blueprint("tag_eval", __name__, url_prefix="/api/tag-eval")
+
+
+def _serialize(run: TagEvalRun, *, include_report: bool) -> dict:
+    out = {
+        "id": run.id,
+        "params": run.params,
+        "status": run.status,
+        "started_at": run.started_at.isoformat() if run.started_at else None,
+        "finished_at": run.finished_at.isoformat() if run.finished_at else None,
+        "error": run.error,
+    }
+    if include_report:
+        out["report"] = run.report
+    return out
+
+
+@tag_eval_bp.route("", methods=["POST"])
+async def create():
+    body = await request.get_json(silent=True) or {}
+    params = body.get("params") or body or {}
+    async with get_session() as session:
+        try:
+            run_id = await session.run_sync(
+                lambda s: start_tag_eval_run(s, params)
+            )
+        except EvalAlreadyRunning as running:
+            return jsonify({
+                "error": "eval_already_running",
+                "running_id": int(running.args[0]),
+            }), 409
+        await session.commit()
+    return jsonify({"run_id": run_id, "status": "running"}), 202
+
+
+@tag_eval_bp.route("", methods=["GET"])
+async def history():
+    try:
+        limit = min(int(request.args.get("limit", "20")), 100)
+    except ValueError:
+        return jsonify({"error": "invalid_limit"}), 400
+    async with get_session() as session:
+        rows = (await session.execute(
+            select(TagEvalRun).order_by(TagEvalRun.id.desc()).limit(limit)
+        )).scalars().all()
+        # List is light — no full report (the detail endpoint carries it).
+        return jsonify({"runs": [_serialize(r, include_report=False) for r in rows]})
+
+
+@tag_eval_bp.route("/<int:run_id>", methods=["GET"])
+async def detail(run_id: int):
+    async with get_session() as session:
+        run = await session.get(TagEvalRun, run_id)
+        if run is None:
+            return jsonify({"error": "not_found"}), 404
+        return jsonify(_serialize(run, include_report=True))

backend/app/api/tags.py

@@ -2,12 +2,14 @@
 
 from quart import Blueprint, jsonify, request
 from sqlalchemy import exists, select
+from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.exc import IntegrityError
 
 from ..extensions import get_session
-from ..models import Tag, TagKind
+from ..models import Tag, TagKind, TagPositiveConfirmation
 from ..models.tag_allowlist import TagAllowlist
 from ..services.bulk_tag_service import BulkTagService
+from ..services.ml.aliases import AliasService
 from ..services.series_match_service import SeriesMatchService
 from ..services.series_service import SeriesError, SeriesService
 from ..services.tag_directory_service import TagDirectoryService
@@ -182,6 +184,20 @@ async def remove_tag_from_image(image_id: int, tag_id: int):
     return "", 204
 
 
+@tags_bp.route("/images/<int:image_id>/tags/<int:tag_id>/confirm", methods=["POST"])
+async def confirm_tag_on_image(image_id: int, tag_id: int):
+    """Operator affirmed an applied tag is correct ("keep" on a doubted positive).
+    Idempotent; recorded so the eval's doubts list stops resurfacing it (#1130)."""
+    async with get_session() as session:
+        await session.execute(
+            pg_insert(TagPositiveConfirmation)
+            .values(image_record_id=image_id, tag_id=tag_id)
+            .on_conflict_do_nothing(index_elements=["image_record_id", "tag_id"])
+        )
+        await session.commit()
+    return "", 204
+
+
 @tags_bp.route("/tags/<int:tag_id>", methods=["GET"])
 async def get_tag(tag_id: int):
     """Resolve a single tag (used by the gallery to label its active
@@ -200,6 +216,25 @@ async def get_tag(tag_id: int):
         )
 
 
+@tags_bp.route("/tags/<int:tag_id>/aliases", methods=["GET"])
+async def list_tag_aliases(tag_id: int):
+    """Model keys that fold into this tag (tag-side alias view). Remove via the
+    shared DELETE /api/aliases/<string>/<category>."""
+    async with get_session() as session:
+        if await session.get(Tag, tag_id) is None:
+            return jsonify({"error": "tag not found"}), 404
+        rows = await AliasService(session).list_for_tag(tag_id)
+    return jsonify(
+        [
+            {
+                "alias_string": r.alias_string,
+                "alias_category": r.alias_category,
+            }
+            for r in rows
+        ]
+    )
+
+
 @tags_bp.route("/tags/<int:tag_id>", methods=["PATCH"])
 async def update_tag(tag_id: int):
     """Rename and/or re-fandom a tag. Body may carry `name` and/or
@@ -391,14 +426,9 @@ async def series_add(tag_id: int):
     ids, err = _parse_bulk_ids(body, max_ids=500)
     if err:
         return err
-    chapter_id, cerr = _opt_int(body, "chapter_id")
-    if cerr:
-        return cerr
     async with get_session() as session:
         try:
-            n = await SeriesService(session).add_images(
-                tag_id, ids, chapter_id=chapter_id
-            )
+            n = await SeriesService(session).add_images(tag_id, ids)
         except SeriesError as exc:
             return _series_err(exc)
         await session.commit()
@@ -420,15 +450,26 @@ async def series_remove(tag_id: int):
     return jsonify({"removed_count": n})
 
 
-@tags_bp.route("/series/<int:tag_id>/reorder", methods=["POST"])
-async def series_reorder(tag_id: int):
-    body = await request.get_json()
-    ids, err = _parse_bulk_ids(body, max_ids=500)
-    if err:
-        return err
+@tags_bp.route("/series/<int:tag_id>/pages/number", methods=["POST"])
+async def series_set_page_number(tag_id: int):
+    """Set one placed page's number — the operator's value (sparse, gaps
+    allowed); pass page_number: null to leave it unnumbered."""
+    body = await request.get_json() or {}
+    image_id, ierr = _opt_int(body, "image_id")
+    if ierr:
+        return ierr
+    if image_id is None:
+        return jsonify({"error": "image_id required"}), 400
+    if "page_number" not in body:
+        return jsonify({"error": "page_number required (may be null)"}), 400
+    page_number, perr = _opt_int(body, "page_number")
+    if perr:
+        return perr
     async with get_session() as session:
         try:
-            await SeriesService(session).reorder(tag_id, ids)
+            await SeriesService(session).set_page_number(
+                tag_id, image_id, page_number
+            )
         except SeriesError as exc:
             return _series_err(exc)
         await session.commit()
@@ -453,30 +494,31 @@ async def series_cover(tag_id: int):
     return jsonify({"ok": True})
 
 
-# ---- chapters (FC-6.1) ----------------------------------------------------
+# ---- chapter dividers (FC-6.x) -------------------------------------------
+# A chapter is a cosmetic divider anchored to the page that begins it; it owns
+# no pages. Page ordering follows each page's operator-set number (the
+# /pages/number endpoint), so there is no per-chapter reorder/merge — those are
+# gone.
 
 
 @tags_bp.route("/series/<int:tag_id>/chapters", methods=["POST"])
 async def series_chapter_create(tag_id: int):
     body = await request.get_json() or {}
+    anchor, aerr = _opt_int(body, "anchor_image_id")
+    if aerr:
+        return aerr
+    if anchor is None:
+        return jsonify({"error": "anchor_image_id required"}), 400
     title = body.get("title")
     if title is not None and not isinstance(title, str):
         return jsonify({"error": "title must be a string"}), 400
-    is_placeholder = bool(body.get("is_placeholder", False))
-    start, serr = _opt_int(body, "stated_page_start")
-    if serr:
-        return serr
-    end, eerr = _opt_int(body, "stated_page_end")
-    if eerr:
-        return eerr
+    part, perr = _opt_int(body, "stated_part")
+    if perr:
+        return perr
     async with get_session() as session:
         try:
-            ch = await SeriesService(session).create_chapter(
-                tag_id,
-                title=title,
-                is_placeholder=is_placeholder,
-                stated_page_start=start,
-                stated_page_end=end,
+            ch = await SeriesService(session).create_divider(
+                tag_id, anchor, title=title, stated_part=part,
             )
         except SeriesError as exc:
             return _series_err(exc)
@@ -484,21 +526,6 @@ async def series_chapter_create(tag_id: int):
     return jsonify(ch)
 
 
-@tags_bp.route("/series/<int:tag_id>/chapters/reorder", methods=["POST"])
-async def series_chapter_reorder(tag_id: int):
-    body = await request.get_json()
-    ids, err = _parse_int_list(body, "chapter_ids")
-    if err:
-        return err
-    async with get_session() as session:
-        try:
-            await SeriesService(session).reorder_chapters(tag_id, ids)
-        except SeriesError as exc:
-            return _series_err(exc)
-        await session.commit()
-    return jsonify({"ok": True})
-
-
 @tags_bp.route(
     "/series/<int:tag_id>/chapters/<int:chapter_id>", methods=["PATCH"]
 )
@@ -514,19 +541,18 @@ async def series_chapter_update(tag_id: int, chapter_id: int):
         if perr:
             return perr
         kwargs.update(set_part=True, stated_part=part)
-    if "stated_page_start" in body:
-        start, serr = _opt_int(body, "stated_page_start")
-        if serr:
-            return serr
-        kwargs.update(set_start=True, stated_page_start=start)
-    if "stated_page_end" in body:
-        end, eerr = _opt_int(body, "stated_page_end")
-        if eerr:
-            return eerr
-        kwargs.update(set_end=True, stated_page_end=end)
+    if "anchor_image_id" in body:
+        anchor, aerr = _opt_int(body, "anchor_image_id")
+        if aerr:
+            return aerr
+        if anchor is None:
+            return jsonify(
+                {"error": "anchor_image_id must be an integer"}
+            ), 400
+        kwargs.update(set_anchor=True, anchor_image_id=anchor)
     async with get_session() as session:
         try:
-            await SeriesService(session).update_chapter(
+            await SeriesService(session).update_divider(
                 tag_id, chapter_id, **kwargs
             )
         except SeriesError as exc:
@@ -541,45 +567,7 @@ async def series_chapter_update(tag_id: int, chapter_id: int):
 async def series_chapter_delete(tag_id: int, chapter_id: int):
     async with get_session() as session:
         try:
-            await SeriesService(session).delete_chapter(tag_id, chapter_id)
-        except SeriesError as exc:
-            return _series_err(exc)
-        await session.commit()
-    return jsonify({"ok": True})
-
-
-@tags_bp.route(
-    "/series/<int:tag_id>/chapters/<int:chapter_id>/merge", methods=["POST"]
-)
-async def series_chapter_merge(tag_id: int, chapter_id: int):
-    body = await request.get_json()
-    target, terr = _opt_int(body, "target_chapter_id")
-    if terr:
-        return terr
-    if target is None:
-        return jsonify({"error": "target_chapter_id required"}), 400
-    async with get_session() as session:
-        try:
-            moved = await SeriesService(session).merge_chapter(
-                tag_id, chapter_id, target
-            )
-        except SeriesError as exc:
-            return _series_err(exc)
-        await session.commit()
-    return jsonify({"moved_count": moved})
-
-
-@tags_bp.route(
-    "/series/<int:tag_id>/chapters/<int:chapter_id>/reorder", methods=["POST"]
-)
-async def series_chapter_reorder_pages(tag_id: int, chapter_id: int):
-    body = await request.get_json()
-    ids, err = _parse_bulk_ids(body, max_ids=500)
-    if err:
-        return err
-    async with get_session() as session:
-        try:
-            await SeriesService(session).reorder_pages(tag_id, chapter_id, ids)
+            await SeriesService(session).delete_divider(tag_id, chapter_id)
         except SeriesError as exc:
             return _series_err(exc)
         await session.commit()
@@ -635,13 +623,35 @@ async def series_add_post(tag_id: int):
         return jsonify({"error": "post_id required"}), 400
     async with get_session() as session:
         try:
-            out = await SeriesService(session).add_post_as_chapter(tag_id, post_id)
+            out = await SeriesService(session).add_post(tag_id, post_id)
         except SeriesError as exc:
             return _series_err(exc)
         await session.commit()
     return jsonify(out)
 
 
+@tags_bp.route("/series/<int:tag_id>/pending/place", methods=["POST"])
+async def series_place_pending(tag_id: int):
+    """Place staged (pending) pages into the run, numbered sequentially from
+    `start_page` in the given order (#789). start_page null → unnumbered."""
+    body = await request.get_json()
+    ids, err = _parse_bulk_ids(body, max_ids=500)
+    if err:
+        return err
+    start, serr = _opt_int(body, "start_page")
+    if serr:
+        return serr
+    async with get_session() as session:
+        try:
+            n = await SeriesService(session).place_pending(
+                tag_id, ids, start_page=start
+            )
+        except SeriesError as exc:
+            return _series_err(exc)
+        await session.commit()
+    return jsonify({"placed_count": n})
+
+
 # ---- suggestion queue (FC-6.3) --------------------------------------------
 
 

backend/app/celery_app.py

@@ -30,6 +30,7 @@ def make_celery() -> Celery:
             "backend.app.tasks.maintenance",
             "backend.app.tasks.ml",
             "backend.app.tasks.download",
+            "backend.app.tasks.external",
             "backend.app.tasks.backup",
             "backend.app.tasks.admin",
             "backend.app.tasks.library_audit",
@@ -42,6 +43,9 @@ def make_celery() -> Celery:
             "backend.app.tasks.ml.*": {"queue": "ml"},
             "backend.app.tasks.thumbnail.*": {"queue": "thumbnail"},
             "backend.app.tasks.download.*": {"queue": "download"},
+            # External file-host fetches are downloads — same lane (they can run
+            # long, but the download worker already tolerates long backfills).
+            "backend.app.tasks.external.*": {"queue": "download"},
             "backend.app.tasks.scan.*": {"queue": "scan"},
             # `maintenance` is the QUICK lane — recovery sweeps, vacuum, cleanup
             # (concurrency-1 on the scheduler). The long one-shots (DB backups,
@@ -57,7 +61,33 @@ def make_celery() -> Celery:
         # Heavy ML tasks need fair dispatch — see ImageRepo's precedent.
         task_acks_late=True,
         worker_prefetch_multiplier=1,
+        # Broker resilience (2026-06-24): a swarm overlay-network blip after a
+        # redeploy left Redis healthy but transiently unreachable, and a worker
+        # starting in that window crash-looped on the initial broker connect
+        # (kombu OperationalError) instead of waiting it out — needing a manual
+        # Redis reset to recover. Retry the broker FOREVER (None) on startup and
+        # at runtime so a transient outage self-heals when routing returns,
+        # rather than the worker exiting.
         broker_connection_retry_on_startup=True,
+        broker_connection_retry=True,
+        broker_connection_max_retries=None,
+        # Redis-transport socket options (apply to the BROKER connection): a
+        # short connect timeout + TCP keepalive so a dead/blocked socket is
+        # noticed and retried, and a periodic health check that proactively
+        # reconnects a live worker through a network hiccup.
+        broker_transport_options={
+            "socket_connect_timeout": 5,
+            "socket_timeout": 30,
+            "socket_keepalive": True,
+            "retry_on_timeout": True,
+            "health_check_interval": 30,
+        },
+        # Same hardening for the Redis RESULT backend (separate connection pool).
+        redis_socket_connect_timeout=5,
+        redis_socket_timeout=30,
+        redis_socket_keepalive=True,
+        redis_retry_on_timeout=True,
+        redis_backend_health_check_interval=30,
         beat_schedule={
             "recover-interrupted-tasks": {
                 "task": "backend.app.tasks.maintenance.recover_interrupted_tasks",
@@ -79,6 +109,36 @@ def make_celery() -> Celery:
                 "task": "backend.app.tasks.ml.apply_allowlist_tags",
                 "schedule": 86400.0,
             },
+            "train-heads-nightly": {
+                "task": "backend.app.tasks.ml.scheduled_train_heads",
+                "schedule": 86400.0,  # passive cadence; manual retrain stays available
+            },
+            "apply-head-tags-daily": {
+                "task": "backend.app.tasks.ml.scheduled_apply_head_tags",
+                "schedule": 86400.0,  # no-op unless head_auto_apply_enabled
+            },
+            "recover-orphaned-gpu-jobs": {
+                "task": "backend.app.tasks.ml.recover_orphaned_gpu_jobs",
+                "schedule": 60.0,  # quick pickup of work a dead agent orphaned
+            },
+            "enqueue-ccip-backfill-hourly": {
+                "task": "backend.app.tasks.ml.enqueue_gpu_backfill",
+                "schedule": 3600.0,  # auto-feed new images (+ retry errored) so
+                "args": ("ccip",),   # the queue keeps moving without the button
+            },
+            "enqueue-siglip-backfill-daily": {
+                "task": "backend.app.tasks.ml.enqueue_gpu_backfill",
+                "schedule": 86400.0,  # drain the concept-crop back-catalogue +
+                "args": ("siglip",),  # retry failed embeds, no button needed
+            },
+            "ccip-auto-apply-daily": {
+                "task": "backend.app.tasks.ml.scheduled_ccip_auto_apply",
+                "schedule": 86400.0,  # no-op unless ccip_auto_apply_enabled
+            },
+            "snapshot-head-metrics-daily": {
+                "task": "backend.app.tasks.maintenance.snapshot_head_metrics",
+                "schedule": 86400.0,
+            },
             "integrity-verify-weekly": {
                 "task": "backend.app.tasks.maintenance.verify_integrity",
                 "schedule": 604800.0,  # weekly
@@ -126,6 +186,18 @@ def make_celery() -> Celery:
                 "task": "backend.app.tasks.maintenance.recover_stalled_library_audit_runs",
                 "schedule": 300.0,
             },
+            "recover-stalled-tag-eval-runs": {
+                "task": "backend.app.tasks.maintenance.recover_stalled_tag_eval_runs",
+                "schedule": 300.0,
+            },
+            "recover-stalled-head-training-runs": {
+                "task": "backend.app.tasks.maintenance.recover_stalled_head_training_runs",
+                "schedule": 300.0,
+            },
+            "recover-stalled-head-auto-apply-runs": {
+                "task": "backend.app.tasks.maintenance.recover_stalled_head_auto_apply_runs",
+                "schedule": 300.0,
+            },
             "recover-stalled-import-batches": {
                 "task": "backend.app.tasks.maintenance.recover_stalled_import_batches",
                 "schedule": 300.0,
@@ -150,6 +222,21 @@ def make_celery() -> Celery:
                 "task": "backend.app.tasks.thumbnail.backfill_thumbnails",
                 "schedule": 86400.0,
             },
+            # External file-host downloads (#830): a steady sweep catches links
+            # the post-download hook missed (worker down, etc.); recovery re-tries
+            # dead links daily; retention prunes long-dead rows.
+            "extdl-sweep": {
+                "task": "backend.app.tasks.external.sweep_external_links",
+                "schedule": 600.0,  # every 10 min
+            },
+            "extdl-recover-daily": {
+                "task": "backend.app.tasks.external.recover_external_links",
+                "schedule": 86400.0,
+            },
+            "extdl-prune-daily": {
+                "task": "backend.app.tasks.external.prune_external_links",
+                "schedule": 86400.0,
+            },
         },
         timezone="UTC",
     )

backend/app/celery_signals.py

@@ -69,7 +69,15 @@ def _queue_for(task) -> str:
         return "ml"
     if name.startswith("backend.app.tasks.thumbnail."):
         return "thumbnail"
-    if name.startswith("backend.app.tasks.download."):
+    if name.startswith((
+        "backend.app.tasks.download.",
+        # External file-host fetches share the download lane (celery_app
+        # routes external.* → download). Mirror it here or TaskRun.queue
+        # lies 'default' for them, so per-queue dashboard filters and the
+        # per-queue threshold override miss them — the same gap the
+        # 2026-06-02 audit fixed for backup/admin/library_audit.
+        "backend.app.tasks.external.",
+    )):
         return "download"
     if name.startswith("backend.app.tasks.scan."):
         return "scan"

backend/app/models/__init__.py

@@ -7,9 +7,16 @@ from .backup_run import BackupRun
 from .base import Base
 from .credential import Credential
 from .download_event import DownloadEvent
+from .external_link import ExternalLink
+from .gpu_job import GpuJob
+from .head_auto_apply_run import HeadAutoApplyRun
+from .head_metric import HeadMetric
+from .head_metrics_snapshot import HeadMetricsSnapshot
+from .head_training_run import HeadTrainingRun
 from .image_prediction import ImagePrediction
 from .image_provenance import ImageProvenance
 from .image_record import ImageRecord
+from .image_region import ImageRegion
 from .import_batch import ImportBatch
 from .import_settings import ImportSettings
 from .import_task import ImportTask
@@ -23,9 +30,14 @@ from .series_chapter import SeriesChapter
 from .series_page import SeriesPage
 from .series_suggestion import SeriesSuggestion
 from .source import Source
+from .subscribestar_failed_media import SubscribeStarFailedMedia
+from .subscribestar_seen_media import SubscribeStarSeenMedia
 from .tag import Tag, TagKind, image_tag
 from .tag_alias import TagAlias
 from .tag_allowlist import TagAllowlist
+from .tag_eval_run import TagEvalRun
+from .tag_head import TagHead
+from .tag_positive_confirmation import TagPositiveConfirmation
 from .tag_reference_embedding import TagReferenceEmbedding
 from .tag_suggestion_rejection import TagSuggestionRejection
 from .task_run import TaskRun
@@ -40,6 +52,8 @@ __all__ = [
     "Credential",
     "PatreonFailedMedia",
     "PatreonSeenMedia",
+    "SubscribeStarFailedMedia",
+    "SubscribeStarSeenMedia",
     "Post",
     "PostAttachment",
     "SeriesChapter",
@@ -48,17 +62,27 @@ __all__ = [
     "ImageRecord",
     "ImagePrediction",
     "ImageProvenance",
+    "ImageRegion",
     "Tag",
     "TagKind",
     "image_tag",
     "DownloadEvent",
+    "ExternalLink",
+    "GpuJob",
     "ImportBatch",
     "ImportTask",
     "ImportSettings",
     "LibraryAuditRun",
     "MLSettings",
+    "HeadAutoApplyRun",
+    "HeadMetric",
+    "HeadMetricsSnapshot",
+    "HeadTrainingRun",
     "TagAlias",
     "TagAllowlist",
+    "TagEvalRun",
+    "TagHead",
+    "TagPositiveConfirmation",
     "TagReferenceEmbedding",
     "TagSuggestionRejection",
     "TaskRun",

backend/app/models/external_link.py

@@ -0,0 +1,73 @@
+"""ExternalLink — an off-platform file-host link found in a post body.
+
+Creators host the actual files (films, packs) on mega.nz / Google Drive /
+MediaFire / Dropbox / Pixeldrain and drop the link in the post text. This row
+is the record that the link existed (so nothing is silently dropped), the
+dedup + dead-letter ledger for fetching it, and the driver the download worker
+walks. `url` keeps the FULL link including the `#fragment` (mega's decryption
+key) — truncating it makes the file undownloadable.
+
+status lifecycle: pending → downloading → downloaded | failed | dead
+(too many attempts) | skipped (host disabled). `attachment_id` links the
+captured file once a download lands (SET NULL so deleting the attachment
+doesn't delete the link record).
+"""
+
+from datetime import datetime
+
+from sqlalchemy import (
+    DateTime,
+    Float,
+    ForeignKey,
+    Index,
+    Integer,
+    String,
+    Text,
+    func,
+    text,
+)
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+# Kept in sync with link_extract.SUPPORTED_HOSTS and the CHECK in migration 0049.
+HOSTS = ("mega", "gdrive", "mediafire", "dropbox", "pixeldrain")
+STATUSES = ("pending", "downloading", "downloaded", "failed", "skipped", "dead")
+
+
+class ExternalLink(Base):
+    __tablename__ = "external_link"
+    __table_args__ = (
+        # One row per (post, url). The full url (incl. #fragment) is the identity
+        # — the same file linked twice in a post collapses to one row.
+        Index("uq_external_link_post_url", "post_id", "url", unique=True),
+        Index("ix_external_link_status", "status"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    post_id: Mapped[int] = mapped_column(
+        ForeignKey("post.id", ondelete="CASCADE"), nullable=False, index=True
+    )
+    artist_id: Mapped[int | None] = mapped_column(
+        ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True
+    )
+    host: Mapped[str] = mapped_column(String(16), nullable=False)
+    url: Mapped[str] = mapped_column(Text, nullable=False)
+    label: Mapped[str | None] = mapped_column(Text, nullable=True)
+    status: Mapped[str] = mapped_column(
+        String(16), nullable=False, server_default="pending"
+    )
+    attempts: Mapped[int] = mapped_column(
+        Integer, nullable=False, server_default=text("0")
+    )
+    last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
+    attachment_id: Mapped[int | None] = mapped_column(
+        ForeignKey("post_attachment.id", ondelete="SET NULL"), nullable=True
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    completed_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    duration_seconds: Mapped[float | None] = mapped_column(Float, nullable=True)

backend/app/models/gpu_job.py

@@ -0,0 +1,50 @@
+"""GpuJob — a unit of GPU work the desktop agent pulls over HTTP (#114).
+
+The durable work list that lets the agent stay HTTP-only: the server enqueues a
+job per (image, task) — e.g. detect figures + CCIP-embed — and the agent LEASES a
+batch, computes on its GPU, then SUBMITS results, all over the already-exposed web
+API. Redis/Postgres stay private. A lease has an expiry; the lease query itself
+re-claims expired leases (agent died / stopped mid-batch), so the queue is
+self-healing without a separate sweep. One job is per ITEM; the agent fans a
+VIDEO out into per-frame instances internally (see image_region.frame_time).
+
+State: pending → leased → done | error (a failure under the attempt cap returns to
+pending for another agent).
+"""
+
+from datetime import datetime
+
+from sqlalchemy import DateTime, ForeignKey, Integer, String, Text, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class GpuJob(Base):
+    __tablename__ = "gpu_job"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    image_record_id: Mapped[int] = mapped_column(
+        ForeignKey("image_record.id", ondelete="CASCADE"), index=True
+    )
+    # What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'.
+    task: Mapped[str] = mapped_column(String(32), nullable=False)
+    status: Mapped[str] = mapped_column(
+        String(16), nullable=False, default="pending", index=True
+    )
+    # pending | leased | done | error
+    lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True)
+    leased_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    lease_expires_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    error: Mapped[str | None] = mapped_column(Text, nullable=True)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )

backend/app/models/head_auto_apply_run.py

@@ -0,0 +1,46 @@
+"""HeadAutoApplyRun — persisted lifecycle of an earned-auto-apply sweep (#114).
+
+A graduated head can apply its tag to images it scores above the head's
+auto-apply threshold, without a human. This row tracks one such sweep (or a
+dry-run PREVIEW of it) so the result survives navigation and the admin card can
+show what fired / what would fire. Mirrors HeadTrainingRun. State machine:
+running → ready / error. The `report` JSONB holds per-concept counts
+(applied / projected / scanned).
+"""
+
+from datetime import datetime
+from typing import Any
+
+from sqlalchemy import Boolean, DateTime, Integer, String, Text, func
+from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class HeadAutoApplyRun(Base):
+    __tablename__ = "head_auto_apply_run"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    # dry_run=True is a PREVIEW: scores + counts what WOULD apply, writes nothing
+    # (preview/apply parity, rule 93).
+    dry_run: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
+    params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
+    status: Mapped[str] = mapped_column(
+        String(16), nullable=False, default="running", index=True
+    )
+    # running | ready | error
+    started_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    finished_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    # Total tags applied across all heads this sweep (0 for a clean dry-run).
+    n_applied: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    # Per-concept breakdown: [{tag_id, name, applied, scanned, threshold}, ...].
+    report: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True)
+    error: Mapped[str | None] = mapped_column(Text, nullable=True)
+    last_progress_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )

backend/app/models/head_metric.py

@@ -0,0 +1,32 @@
+"""HeadMetric — running correction counters per concept (#114 observability).
+
+Earned auto-apply fires graduated heads; to TUNE it we need to know how often a
+head's auto-applied tag was wrong (the operator removed it = a MISFIRE) and how
+often the operator had to add a tag a head exists for by hand (an UNDER-FIRE,
+the head missed it). image_tag.source is lost when a row is deleted, so these
+are captured as durable cumulative counters at correction time — they survive
+head retrain/prune (keyed by tag, not by the head row). The daily snapshot reads
+them into the time-series.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import DateTime, ForeignKey, Integer, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class HeadMetric(Base):
+    __tablename__ = "head_metric"
+
+    tag_id: Mapped[int] = mapped_column(
+        ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
+    )
+    # An auto-applied (source='head_auto') tag the operator later REMOVED.
+    n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    # A tag with a head that the operator added by HAND (the head missed it).
+    n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )

backend/app/models/head_metrics_snapshot.py

@@ -0,0 +1,38 @@
+"""HeadMetricsSnapshot — a daily per-concept time-series point (#114).
+
+The "amount of change over time" reporting the operator asked for: once a day,
+record each concept's auto-applied VOLUME (current head_auto tags), cumulative
+misfires/under-fires, and the head's measured quality. Plotting these rows over
+time shows whether auto-apply is landing better/worse and whether tagging more is
+sharpening a concept — the signal for tuning the precision target + support floor.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import DateTime, Float, ForeignKey, Integer, String, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class HeadMetricsSnapshot(Base):
+    __tablename__ = "head_metrics_snapshot"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    tag_id: Mapped[int] = mapped_column(
+        ForeignKey("tag.id", ondelete="CASCADE"), index=True
+    )
+    # Denormalized so a snapshot stays readable even if the tag is later renamed.
+    name: Mapped[str] = mapped_column(String(255), nullable=False)
+    snapshot_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now(), index=True
+    )
+    # Current count of source='head_auto' applications still standing.
+    n_auto_applied: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    n_misfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    n_underfires: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    # The head's measured quality at snapshot time (null if no head exists).
+    ap: Mapped[float | None] = mapped_column(Float, nullable=True)
+    precision_cv: Mapped[float | None] = mapped_column(Float, nullable=True)
+    recall: Mapped[float | None] = mapped_column(Float, nullable=True)
+    n_pos: Mapped[int | None] = mapped_column(Integer, nullable=True)

backend/app/models/head_training_run.py

@@ -0,0 +1,44 @@
+"""HeadTrainingRun — persisted lifecycle of a head-training batch (#114).
+
+Mirrors TagEvalRun so the run SURVIVES navigation and the admin card can show
+live + historical status instead of holding it in transient frontend state.
+Training is idempotent (it upserts tag_head rows), so a SIGKILL'd run is harmless
+— a maintenance recovery sweep flips a stalled `running` row to `error`, and the
+next run re-trains. State machine: running → ready / error.
+"""
+
+from datetime import datetime
+from typing import Any
+
+from sqlalchemy import DateTime, Integer, String, Text, func
+from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class HeadTrainingRun(Base):
+    __tablename__ = "head_training_run"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    # Training parameters: {min_positives, neg_ratio, precision_target, ...}.
+    params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
+    status: Mapped[str] = mapped_column(
+        String(16), nullable=False, default="running", index=True
+    )
+    # running | ready | error
+    started_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    finished_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
+    # How many concepts got a (re)trained head vs were skipped (too few labels).
+    n_trained: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    n_skipped: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    error: Mapped[str | None] = mapped_column(Text, nullable=True)
+    # Last time the task made progress — the recovery sweep tells a live run from
+    # a SIGKILL'd one by this (mirrors TagEvalRun).
+    last_progress_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )

backend/app/models/image_provenance.py

@@ -41,6 +41,16 @@ class ImageProvenance(Base):
     source_id: Mapped[int | None] = mapped_column(
         ForeignKey("source.id", ondelete="SET NULL"), nullable=True, index=True
     )
+    # The archive PostAttachment this image was extracted FROM, when it came
+    # out of a .zip/.rar rather than as a loose file (milestone #87). Lets the
+    # provenance UI show the exact archive a file lives inside instead of every
+    # attachment on the post. NULL for loose downloads and pre-backfill rows.
+    # SET NULL so deleting the archive attachment never destroys the (image,
+    # post) edge — it just forgets which archive it came from.
+    from_attachment_id: Mapped[int | None] = mapped_column(
+        ForeignKey("post_attachment.id", ondelete="SET NULL"),
+        nullable=True, index=True,
+    )
     captured_metadata: Mapped[dict | None] = mapped_column(JSON, nullable=True)
     captured_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), nullable=False, server_default=func.now()

backend/app/models/image_record.py

@@ -13,6 +13,7 @@ from sqlalchemy import (
     BigInteger,
     DateTime,
     Enum,
+    Float,
     ForeignKey,
     Integer,
     String,
@@ -39,6 +40,10 @@ class ImageRecord(Base):
     mime: Mapped[str] = mapped_column(String(64), nullable=False)
     width: Mapped[int | None] = mapped_column(Integer, nullable=True)
     height: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    # Video container duration (seconds); NULL for images. The Tier-1 video
+    # near-dup key (#871): two videos of the same artist with matching duration
+    # (+ aspect) are the same content across re-encodes — dedup like image pHash.
+    duration_seconds: Mapped[float | None] = mapped_column(Float, nullable=True)
 
     # Integrity verification status. FC-2e populates this; FC-2a leaves rows at 'unknown'.
     # Values: 'unknown' (default), 'ok', 'corrupt', 'failed_verification'.
@@ -49,6 +54,18 @@ class ImageRecord(Base):
     # Thumbnail (populated by FC-2)
     thumbnail_path: Mapped[str | None] = mapped_column(Text, nullable=True)
 
+    # Source provenance for downloaded media (#830 Phase 2). `source_url` is the
+    # CDN/origin URL the file was fetched from (debugging + future re-fetch).
+    # `source_filehash` is the URL's 32-hex CDN identity segment
+    # (utils.paths.filehash_from_url) — the JOIN KEY that maps a post body's
+    # inline `<img src=CDN>` back to this local copy so the rendered body serves
+    # our stored image instead of hotlinking the public source. Indexed for the
+    # render-time lookup. NULL for filesystem-imported / pre-Phase-2 rows.
+    source_url: Mapped[str | None] = mapped_column(Text, nullable=True)
+    source_filehash: Mapped[str | None] = mapped_column(
+        String(32), nullable=True, index=True
+    )
+
     # Origin / provenance pointers
     origin: Mapped[str] = mapped_column(Enum(*ORIGIN_CHOICES, name="origin_enum"), nullable=False)
     primary_post_id: Mapped[int | None] = mapped_column(
@@ -60,8 +77,10 @@ class ImageRecord(Base):
         ForeignKey("artist.id", ondelete="SET NULL"), nullable=True, index=True
     )
 
-    # ML fields (populated by FC-2's ml-worker)
-    tagger_predictions: Mapped[dict | None] = mapped_column(JSON, nullable=True)
+    # ML fields (populated by FC-2's ml-worker). Per-tag predictions live in the
+    # normalized image_prediction table (#768) — the tagger_predictions JSON
+    # column was dropped in migration 0046. tagger_model_version stays as the
+    # "has this been tagged / is it current?" signal the backfill sweep reads.
     tagger_model_version: Mapped[str | None] = mapped_column(String(128), nullable=True)
     # 1152 = SigLIP-so400m embedding dim. Swapping models in FC-2 may require
     # a column-width migration.

backend/app/models/image_region.py

@@ -0,0 +1,62 @@
+"""ImageRegion — a detected/proposed sub-region of an image + its crop embedding.
+
+The storage backbone of the crop pipeline (#114). A region is a normalized bbox
+plus the embedding of its crop:
+- kind='face' / 'figure' → embedded by CCIP for cross-artist character identity.
+- kind='concept' → embedded by SigLIP, a localized instance for a concept head's
+  bag-of-embeddings (a concept is "present if ANY instance matches").
+One row carries the embedding appropriate to its kind (the other is null). The
+bbox doubles as grounded-tag provenance (hover a tag → highlight its region; a
+wrong box is a precise negative). The GPU agent writes these via the job API;
+the few-shot character matcher + bag scorer read them — both server-side, no GPU.
+"""
+
+from datetime import datetime
+
+from pgvector.sqlalchemy import Vector
+from sqlalchemy import DateTime, Float, ForeignKey, Integer, String, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+CCIP_DIM = 768       # deepghs/imgutils CCIP character embedding
+SIGLIP_DIM = 1152    # matches image_record.siglip_embedding
+
+
+class ImageRegion(Base):
+    __tablename__ = "image_region"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    image_record_id: Mapped[int] = mapped_column(
+        ForeignKey("image_record.id", ondelete="CASCADE"), index=True
+    )
+    # 'frame' (a whole video frame → SigLIP bag) | 'face' | 'figure' (→ CCIP
+    # character id) | 'concept' (→ SigLIP head bag).
+    kind: Mapped[str] = mapped_column(String(16), nullable=False)
+    # For video/animated media: the source frame's timestamp in SECONDS. NULL for
+    # static images. Lets a video be a BAG of per-frame instances (fixes the
+    # mean-embedding muddle) + grounds a tag to "appears at 0:42".
+    frame_time: Mapped[float | None] = mapped_column(Float, nullable=True)
+    # Normalized bbox in [0,1]: top-left (rx, ry) + size (rw, rh). Named rx/ry/…
+    # rather than x/y/by to dodge SQL keyword ambiguity ('by').
+    rx: Mapped[float] = mapped_column(Float, nullable=False)
+    ry: Mapped[float] = mapped_column(Float, nullable=False)
+    rw: Mapped[float] = mapped_column(Float, nullable=False)
+    rh: Mapped[float] = mapped_column(Float, nullable=False)
+    # Proposer/detector confidence (null for deterministic proposers).
+    score: Mapped[float | None] = mapped_column(Float, nullable=True)
+    # Version stamps so a re-detect / re-crop / re-embed can be gated (compute
+    # once; only redo when the producing model version changes).
+    detector_version: Mapped[str | None] = mapped_column(String(64), nullable=True)
+    crop_version: Mapped[str | None] = mapped_column(String(64), nullable=True)
+    embedding_version: Mapped[str | None] = mapped_column(String(128), nullable=True)
+    # Exactly one is set, per kind.
+    ccip_embedding: Mapped[list[float] | None] = mapped_column(
+        Vector(CCIP_DIM), nullable=True
+    )
+    siglip_embedding: Mapped[list[float] | None] = mapped_column(
+        Vector(SIGLIP_DIM), nullable=True
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )

backend/app/models/import_settings.py

@@ -73,6 +73,25 @@ class ImportSettings(Base):
         Float, nullable=False, default=0.5,
     )
 
+    # #830 off-platform file-host downloads — per-host enable lever (default on,
+    # rule #26). Column names are extdl_<host>_enabled so the worker reads them
+    # via getattr(settings, f"extdl_{host}_enabled", True).
+    extdl_mega_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, default=True, server_default="true",
+    )
+    extdl_gdrive_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, default=True, server_default="true",
+    )
+    extdl_mediafire_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, default=True, server_default="true",
+    )
+    extdl_dropbox_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, default=True, server_default="true",
+    )
+    extdl_pixeldrain_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, default=True, server_default="true",
+    )
+
     @classmethod
     async def load(cls, session) -> ImportSettings:
         """The singleton settings row (id=1), via an async session."""

backend/app/models/ml_settings.py

@@ -2,7 +2,15 @@
 
 from datetime import datetime
 
-from sqlalchemy import CheckConstraint, DateTime, Float, Integer, String, func
+from sqlalchemy import (
+    Boolean,
+    CheckConstraint,
+    DateTime,
+    Float,
+    Integer,
+    String,
+    func,
+)
 from sqlalchemy.orm import Mapped, mapped_column
 
 from .base import Base
@@ -40,6 +48,59 @@ class MLSettings(Base):
     min_reference_images: Mapped[int] = mapped_column(
         Integer, nullable=False, default=5
     )
+    # Video tagging (#747). Sample one frame every N seconds (fixed CADENCE, not a
+    # fixed count) so a tag's frame-presence reflects real screen time regardless
+    # of video length; cap the total so a long video can't explode into hundreds
+    # of inferences (the cadence stretches past the cap). A tag is kept only if it
+    # appears in >= video_min_tag_frames sampled frames (≈ that many × interval
+    # seconds on screen) — duration-independent noise rejection. Operator-tunable.
+    video_frame_interval_seconds: Mapped[float] = mapped_column(
+        Float, nullable=False, default=4.0
+    )
+    video_max_frames: Mapped[int] = mapped_column(
+        Integer, nullable=False, default=64
+    )
+    video_min_tag_frames: Mapped[int] = mapped_column(
+        Integer, nullable=False, default=3
+    )
+    # Tagging-v2 head training (#114). The head is the suggestion source that
+    # LEARNS from the operator's tags (replacing Camie + centroid). A concept
+    # needs >= head_min_positives labelled images before a head is trained;
+    # head_auto_apply_precision is the precision bar a head must clear (at some
+    # operating point) to "graduate" into earned auto-apply. Operator-tunable.
+    head_min_positives: Mapped[int] = mapped_column(
+        Integer, nullable=False, default=8
+    )
+    head_auto_apply_precision: Mapped[float] = mapped_column(
+        Float, nullable=False, default=0.97
+    )
+    # Earned auto-apply (#114). A graduated head fires (tags images without a
+    # human) when this master switch is on AND the head has at least
+    # head_auto_apply_min_positives clean labels — so a precise-looking but
+    # under-supported low-N head can't spray tags across the library. ON by
+    # default (operator-asked 2026-06-29: opt-OUT, not opt-in); the support +
+    # measured-precision gates keep it safe, and every auto-tag is reversible.
+    head_auto_apply_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, default=True
+    )
+    head_auto_apply_min_positives: Mapped[int] = mapped_column(
+        Integer, nullable=False, default=30
+    )
+    # CCIP character-match cosine cut (#114). 0.85 default — the v1 flat 0.75
+    # over-fired (high-reference characters matched a scatter of images); 0.85
+    # keeps the confident single-character matches. Tunable from the agent card.
+    ccip_match_threshold: Mapped[float] = mapped_column(
+        Float, nullable=False, default=0.85
+    )
+    # CCIP auto-apply (#114). Confident matches (>= ccip_auto_apply_threshold,
+    # above the suggest cut) auto-tag on a daily sweep. ON by default (opt-out);
+    # single-character references + the high bar keep it safe, every tag reversible.
+    ccip_auto_apply_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, default=True
+    )
+    ccip_auto_apply_threshold: Mapped[float] = mapped_column(
+        Float, nullable=False, default=0.92
+    )
     tagger_model_version: Mapped[str] = mapped_column(
         String(128), nullable=False, default="camie-tagger-v2"
     )

backend/app/models/series_chapter.py

@@ -1,28 +1,22 @@
-"""SeriesChapter — an ordered chapter/part within a series.
+"""SeriesChapter — a cosmetic chapter DIVIDER within a series (FC-6.x reframe).
 
-A series IS a Tag(kind='series'); a chapter groups ordered SeriesPages under it.
-Reading order is (chapter.chapter_number, series_page.page_number): chapter_number
-sets the order of chapters, page_number orders pages within a chapter.
+A series is ONE flat, series-global ordered run of SeriesPages. A chapter is NOT
+a container — it owns no pages. It is a labeled divider anchored to the page that
+BEGINS the chapter (anchor_page_id → series_page): "a new chapter starts here."
+A page's chapter is derived at read time as the nearest preceding divider.
 
-chapter_number is an ordering key only (not unique) — reorder rewrites 1..N
-wholesale, mirroring series_page.page_number, so a reorder can't transiently
-collide on a unique index.
+Dividers never affect page ordering or the series-global page numbers; they stay
+pinned to their anchor page across reorders. anchor_page_id is UNIQUE — at most
+one chapter begins at a given page — and FK-cascades, so removing the anchor page
+from the series drops the divider (the chapter merges into the preceding run).
 
-A chapter may be a placeholder (is_placeholder=True) — a reserved empty slot for
-a section the operator doesn't have yet; it holds no pages and shows as a gap in
-the reader. stated_page_start/end carry the page range parsed from the source
-post (FC-6.2), used to flag missing-page gaps; both are nullable when unknown.
-
-stated_part is the operator-facing "Part N" label (FC-6.4), separate from the
-positional chapter_number: chapter_number is auto-managed ordering (rewritten
-1..N on reorder/delete), while stated_part is the real installment number the
-operator types — e.g. a series authored from a post that is Part 2 of a story.
-Nullable when unset (the UI then falls back to showing chapter_number).
+title is the optional chapter name; stated_part is the optional operator-facing
+"Part N" label (shown instead of a derived ordinal when set).
 """
 
 from datetime import datetime
 
-from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, Text, func
+from sqlalchemy import DateTime, ForeignKey, Integer, Text, func
 from sqlalchemy.orm import Mapped, mapped_column
 
 from .base import Base
@@ -35,14 +29,13 @@ class SeriesChapter(Base):
     series_tag_id: Mapped[int] = mapped_column(
         ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
     )
-    chapter_number: Mapped[int] = mapped_column(Integer, nullable=False)
-    stated_part: Mapped[int | None] = mapped_column(Integer, nullable=True)
-    title: Mapped[str | None] = mapped_column(Text, nullable=True)
-    is_placeholder: Mapped[bool] = mapped_column(
-        Boolean, nullable=False, server_default="false"
+    anchor_page_id: Mapped[int] = mapped_column(
+        ForeignKey("series_page.id", ondelete="CASCADE"),
+        nullable=False,
+        unique=True,
     )
-    stated_page_start: Mapped[int | None] = mapped_column(Integer, nullable=True)
-    stated_page_end: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    title: Mapped[str | None] = mapped_column(Text, nullable=True)
+    stated_part: Mapped[int | None] = mapped_column(Integer, nullable=True)
     created_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), nullable=False, server_default=func.now()
     )

backend/app/models/series_page.py

@@ -1,17 +1,20 @@
 """SeriesPage — ordered image membership for a series-kind Tag.
 
-A series IS a Tag with kind='series'; series_page gives it ordered pages,
-grouped into chapters (FC-6). An image belongs to at most one series
-(UNIQUE image_id) ⇒ at most one chapter. Reading order is
-(chapter.chapter_number, series_page.page_number): page_number orders pages
-WITHIN a chapter and is an ordering key only (not unique) — reorder rewrites
-1..N wholesale. stated_page carries the page number parsed from the source
-post (FC-6.2), nullable when unknown.
+A series IS a Tag with kind='series'; series_page gives it a SINGLE flat,
+series-global ordered run of pages (FC-6.x divider reframe). An image belongs to
+at most one series (UNIQUE image_id). Reading order is `page_number` alone — a
+series-wide ordering key (not unique), rewritten 1..N wholesale on reorder so a
+reorder can't transiently collide on an index.
+
+Chapters are cosmetic DIVIDERS anchored to a page (see SeriesChapter); they do
+NOT own pages, so there is no chapter_id here — a page's chapter is derived at
+read time as the nearest preceding divider. stated_page carries the printed page
+number parsed from the source post, nullable when unknown.
 """
 
 from datetime import datetime
 
-from sqlalchemy import DateTime, ForeignKey, Integer, func
+from sqlalchemy import DateTime, ForeignKey, Integer, String, func
 from sqlalchemy.orm import Mapped, mapped_column
 
 from .base import Base
@@ -24,17 +27,17 @@ class SeriesPage(Base):
     series_tag_id: Mapped[int] = mapped_column(
         ForeignKey("tag.id", ondelete="CASCADE"), nullable=False, index=True
     )
-    chapter_id: Mapped[int] = mapped_column(
-        ForeignKey("series_chapter.id", ondelete="CASCADE"),
-        nullable=False,
-        index=True,
-    )
     image_id: Mapped[int] = mapped_column(
         ForeignKey("image_record.id", ondelete="CASCADE"),
         nullable=False,
         unique=True,
     )
-    page_number: Mapped[int] = mapped_column(Integer, nullable=False)
+    # 'placed' = in the series-global run (page_number set); 'pending' = staged
+    # from a post awaiting the operator's sort (page_number NULL). (#789 P2)
+    status: Mapped[str] = mapped_column(
+        String(16), nullable=False, server_default="placed"
+    )
+    page_number: Mapped[int | None] = mapped_column(Integer, nullable=True)
     stated_page: Mapped[int | None] = mapped_column(Integer, nullable=True)
     created_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), nullable=False, server_default=func.now()

backend/app/models/subscribestar_failed_media.py

@@ -0,0 +1,44 @@
+"""SubscribeStarFailedMedia — per-source dead-letter ledger of SubscribeStar
+media that keeps failing to download/validate.
+
+Mirror of PatreonFailedMedia. Media that fails every walk (404'd CDN URL,
+deleted post, persistently-corrupt bytes) would otherwise re-error forever and
+re-burn backfill chunks. After ``attempts`` reaches the dead-letter threshold
+the ingester skips it on routine tick/backfill walks (recovery still
+re-attempts). A later clean download clears the row.
+
+`filehash` is the same per-media key the seen-ledger uses (CDN content hash or a
+synthesized ``<post_id>:<filename>`` key) — hence String(128). UNIQUE
+(source_id, filehash) is the upsert key.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import ForeignKey, Integer, String, Text, UniqueConstraint, func
+from sqlalchemy.orm import Mapped, mapped_column
+from sqlalchemy.types import DateTime
+
+from .base import Base
+
+
+class SubscribeStarFailedMedia(Base):
+    __tablename__ = "subscribestar_failed_media"
+    __table_args__ = (
+        UniqueConstraint(
+            "source_id", "filehash", name="uq_subscribestar_failed_media_source_id"
+        ),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    source_id: Mapped[int] = mapped_column(
+        ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
+    )
+    filehash: Mapped[str] = mapped_column(String(128), nullable=False)
+    attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=1)
+    last_error: Mapped[str | None] = mapped_column(Text, nullable=True)
+    first_failed_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    last_failed_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )

backend/app/models/subscribestar_seen_media.py

@@ -0,0 +1,40 @@
+"""SubscribeStarSeenMedia — per-source ledger of SubscribeStar media already
+downloaded+processed.
+
+Mirror of PatreonSeenMedia for the SubscribeStar native ingester (replacing
+gallery-dl). One queryable row per (source, media) so routine walks skip media
+we've already ingested; recovery mode bypasses the ledger to re-walk.
+
+`filehash` is a CDN content hash when the media URL carries one, else a
+synthesized ``<post_id>:<filename>`` key (SubscribeStar URLs aren't always
+content-addressed) — hence String(128) rather than 32.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import ForeignKey, Integer, String, UniqueConstraint, func
+from sqlalchemy.orm import Mapped, mapped_column
+from sqlalchemy.types import DateTime
+
+from .base import Base
+
+
+class SubscribeStarSeenMedia(Base):
+    __tablename__ = "subscribestar_seen_media"
+    __table_args__ = (
+        # Dedup key the downloader upserts against: one ledger row per
+        # (source, media). A second sighting of the same media is a no-op.
+        UniqueConstraint(
+            "source_id", "filehash", name="uq_subscribestar_seen_media_source_id"
+        ),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    source_id: Mapped[int] = mapped_column(
+        ForeignKey("source.id", ondelete="CASCADE"), nullable=False, index=True
+    )
+    filehash: Mapped[str] = mapped_column(String(128), nullable=False)
+    post_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
+    seen_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )

backend/app/models/tag_alias.py

@@ -1,6 +1,6 @@
 """TagAlias — maps a model's (name, category) prediction to the operator's
-canonical tag. Resolved at suggestion-read time so raw predictions stay
-unmolested in image_record.tagger_predictions.
+canonical tag. Resolved at suggestion-read time so the raw predictions stored
+in image_prediction stay unmolested.
 """
 
 from datetime import datetime

backend/app/models/tag_eval_run.py

@@ -0,0 +1,45 @@
+"""TagEvalRun — persisted lifecycle of a head-vs-centroid tagging eval (#1130).
+
+Mirrors LibraryAuditRun so the result SURVIVES navigation: the run + its full
+report live in this row, and the admin card rehydrates from it on mount instead
+of holding the report in transient frontend state. State machine:
+running → ready / error. The async ml-queue task writes `report` (JSONB) when
+done; a maintenance recovery sweep flips a stalled `running` row to `error`.
+"""
+
+from datetime import datetime
+from typing import Any
+
+from sqlalchemy import DateTime, Integer, String, Text, func
+from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class TagEvalRun(Base):
+    __tablename__ = "tag_eval_run"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    # The eval parameters: {concepts: [...], curve_points: [...], neg_ratio,
+    # cv_folds, ...} — echoed back so the report is self-describing.
+    params: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False)
+    status: Mapped[str] = mapped_column(
+        String(16), nullable=False, default="running", index=True,
+    )
+    # running | ready | error
+    started_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now(),
+    )
+    finished_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True,
+    )
+    # The full result: per-concept metrics (head vs centroid), learning-curve
+    # points, and example image ids. Null until the task finishes.
+    report: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True)
+    error: Mapped[str | None] = mapped_column(Text, nullable=True)
+    # Last time the task made progress — the recovery sweep tells a live run
+    # from a SIGKILL'd one by this (mirrors LibraryAuditRun).
+    last_progress_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True,
+    )

backend/app/models/tag_head.py

@@ -0,0 +1,77 @@
+"""TagHead — a small per-concept classifier trained on the operator's tags.
+
+Milestone #114, tagging-v2: the production form of the head the eval (#1130)
+proved. One row per concept (general or character) that has enough labelled
+positives. The head is a logistic-regression boundary over the FROZEN SigLIP
+embedding (L2-normalized), trained on the operator's positives + negatives
+(rejections + sampled unlabeled). It REPLACES the Camie prediction + per-tag
+centroid as the suggestion source — and unlike them it LEARNS: every accept /
+reject re-trains it sharper.
+
+Scoring (suggestion path, API worker, NO numpy): p = sigmoid(weights · x̂ + bias)
+where x̂ is the L2-normalized image embedding. Surface as a suggestion when
+p >= suggest_threshold; auto-apply only once auto_apply_threshold is set (the
+head "graduated" — a precision-targeted operating point was achievable). The
+thresholds come from CROSS-VALIDATED out-of-fold scores so they're honest, not
+in-sample-optimistic; the deployable weights are fit on all data.
+"""
+
+from datetime import datetime
+from typing import Any
+
+from pgvector.sqlalchemy import Vector
+from sqlalchemy import (
+    DateTime,
+    Float,
+    ForeignKey,
+    Integer,
+    String,
+    func,
+)
+from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+# Matches image_record.siglip_embedding's dimensionality — the head operates in
+# the same space. A model-version change re-embeds AND retrains (embedding_version
+# guards staleness).
+HEAD_DIM = 1152
+
+
+class TagHead(Base):
+    __tablename__ = "tag_head"
+
+    # One head per concept tag; cascade so deleting a tag retires its head.
+    tag_id: Mapped[int] = mapped_column(
+        ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True
+    )
+    # The embedding the head was trained against (image_record's
+    # embedder_model_version). A mismatch with the current embedder means the
+    # head is stale and must be retrained, not scored.
+    embedding_version: Mapped[str] = mapped_column(String(128), nullable=False)
+    # Logistic-regression coefficients over the L2-normalized embedding, stored
+    # as a pgvector for compactness + a future in-DB dot-product path. NOT a
+    # similarity target, just a serialized weight vector.
+    weights: Mapped[list[float]] = mapped_column(Vector(HEAD_DIM), nullable=False)
+    bias: Mapped[float] = mapped_column(Float, nullable=False)
+    # Probability cutoff for SURFACING as a suggestion (F1-best on CV scores).
+    suggest_threshold: Mapped[float] = mapped_column(Float, nullable=False)
+    # Probability cutoff for EARNED auto-apply: the operating point that holds
+    # precision >= the configured target while maximizing recall. NULL = the head
+    # hasn't graduated (can't auto-apply without a human yet).
+    auto_apply_threshold: Mapped[float | None] = mapped_column(Float, nullable=True)
+    # Training-set sizes + cross-validated quality, surfaced in the admin card so
+    # the operator can see which concepts are strong / need more tags.
+    n_pos: Mapped[int] = mapped_column(Integer, nullable=False)
+    n_neg: Mapped[int] = mapped_column(Integer, nullable=False)
+    ap: Mapped[float] = mapped_column(Float, nullable=False)
+    # 'precision' is a SQL reserved word → store as precision_cv (the
+    # cross-validated precision at the suggest operating point).
+    precision_cv: Mapped[float] = mapped_column(Float, nullable=False)
+    recall: Mapped[float] = mapped_column(Float, nullable=False)
+    trained_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    # Extra detail (auto-apply operating point, F1, etc.) — non-load-bearing.
+    metrics: Mapped[dict[str, Any] | None] = mapped_column(JSONB, nullable=True)

backend/app/models/tag_positive_confirmation.py

@@ -0,0 +1,28 @@
+"""TagPositiveConfirmation — operator affirmed an applied tag is correct.
+
+The mirror of TagSuggestionRejection (#1130). When the operator "keeps" a
+positive the head doubts (low-scoring), record it so the eval's doubts list
+stops resurfacing the same confirmed-correct images every run. Does not change
+training (it's already a positive) — purely a "I've reviewed this" marker.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import DateTime, ForeignKey, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class TagPositiveConfirmation(Base):
+    __tablename__ = "tag_positive_confirmation"
+
+    image_record_id: Mapped[int] = mapped_column(
+        ForeignKey("image_record.id", ondelete="CASCADE"), primary_key=True
+    )
+    tag_id: Mapped[int] = mapped_column(
+        ForeignKey("tag.id", ondelete="CASCADE"), primary_key=True, index=True
+    )
+    confirmed_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )

backend/app/services/cleanup_service.py

@@ -17,14 +17,17 @@ from datetime import UTC, datetime, timedelta
 from pathlib import Path
 from typing import Any
 
-from sqlalchemy import func, or_, select, update
-from sqlalchemy.orm import Session
+from sqlalchemy import delete, func, or_, select, update
+from sqlalchemy.orm import Session, aliased
 
 from ..models import (
     Artist,
+    ExternalLink,
     ImageProvenance,
     ImageRecord,
     LibraryAuditRun,
+    PatreonFailedMedia,
+    PatreonSeenMedia,
     Post,
     PostAttachment,
     Tag,
@@ -32,9 +35,18 @@ from ..models import (
 from ..models.series_chapter import SeriesChapter
 from ..models.series_page import SeriesPage
 from ..models.tag import image_tag
+from ..utils import safe_probe
+from .importer import _VIDEO_DUP_ASPECT_TOL, _VIDEO_DUP_DURATION_TOL_SECONDS
+from .platforms import PLATFORMS
 
 log = logging.getLogger(__name__)
 
+# Sentinel written to duration_seconds when a video was probed but ffprobe
+# reported no usable duration (missing/corrupt file) — distinct from NULL (never
+# probed) so the backfill doesn't re-probe it forever, and < 0 so it can never
+# match a real duration in the dedup grouping (#871).
+_VIDEO_DURATION_UNKNOWN = -1.0
+
 
 def project_artist_cascade(session: Session, *, slug: str) -> dict:
     """Read-only projection of what delete_artist_cascade would touch.
@@ -139,11 +151,22 @@ def project_bulk_image_delete(
 
 
 def count_tag_associations(session: Session, *, tag_id: int) -> int:
-    """COUNT(*) FROM image_tag WHERE tag_id=?. For Tier-B prompt."""
+    """Images affected by deleting this tag — the Tier-B blast-radius prompt.
+    Mirrors the gallery/directory membership predicate: images carrying the tag
+    DIRECTLY, plus — when it's a fandom — images carrying one of its characters
+    (member.fandom_id == tag_id). DISTINCT so each image counts once. Without
+    the character leg a fandom would report 0 here yet its delete still strips
+    the fandom off every character, badly understating the prompt."""
+    member = aliased(Tag)
     return session.execute(
-        select(func.count())
-        .select_from(image_tag)
-        .where(image_tag.c.tag_id == tag_id)
+        select(func.count(image_tag.c.image_record_id.distinct())).where(
+            or_(
+                image_tag.c.tag_id == tag_id,
+                image_tag.c.tag_id.in_(
+                    select(member.id).where(member.fandom_id == tag_id)
+                ),
+            )
+        )
     ).scalar_one()
 
 
@@ -497,6 +520,205 @@ def prune_bare_posts(session: Session, *, dry_run: bool = False) -> dict:
     return {"deleted": result.rowcount or 0, "sample_names": sample}
 
 
+# -- duplicate-post reconciliation (gallery-dl → native migration) ----------
+# An artist first downloaded by gallery-dl gets Post rows keyed by the per-
+# ATTACHMENT id (gallery-dl's `id`); a later native walk keys the SAME real post
+# by the post id. The two never dedup (uq_post_source_external_id is on
+# external_post_id) → duplicate post rows. The real post id is recoverable in-DB
+# from raw_metadata["post_id"] (both eras store the sidecar there). We unify each
+# group onto ONE post row keyed the way the CURRENT native downloader keys it
+# (post id), so future native walks match and the dup can't recur. Images are
+# untouched (content-addressed/deduped already); only post rows + their link
+# rows move. Milestone #73 / note #917.
+
+
+def _canonical_post_id(post: Post) -> str | None:
+    """The real platform post id used to group duplicate rows: raw_metadata
+    ['post_id'] when present (the true id, stored by both gallery-dl and native
+    imports), else external_post_id. None when neither is usable."""
+    rm = post.raw_metadata or {}
+    pid = rm.get("post_id")
+    if pid is not None and str(pid).strip():
+        return str(pid)
+    return post.external_post_id or None
+
+
+def find_duplicate_post_groups(
+    session: Session, *, source_id: int | None = None,
+) -> list[list[Post]]:
+    """Groups of >1 Post that are the SAME real post (same source_id + canonical
+    post id) — the gallery-dl(attachment-id) / native(post-id) duplicates. Shared
+    by the dry-run preview and the live reconcile (preview/apply parity)."""
+    stmt = select(Post)
+    if source_id is not None:
+        stmt = stmt.where(Post.source_id == source_id)
+    groups: dict[tuple, list[Post]] = {}
+    for post in session.execute(stmt.order_by(Post.id)).scalars().all():
+        cpid = _canonical_post_id(post)
+        if not cpid:
+            continue
+        groups.setdefault((post.source_id, cpid), []).append(post)
+    return [posts for posts in groups.values() if len(posts) > 1]
+
+
+def _choose_keeper(posts: list[Post], cpid: str) -> Post:
+    """The surviving row for a dup group: prefer one already keyed by the
+    canonical post id (the native format we keep), then the most complete
+    (has description, has date), then the lowest id for stability."""
+    native = [p for p in posts if p.external_post_id == cpid]
+    pool = native or posts
+    return min(pool, key=lambda p: (not p.description, not p.post_date, p.id))
+
+
+def _repoint_post_links(session: Session, loser_id: int, keeper_id: int) -> None:
+    """Move every link row from a loser post to the keeper, conflict-safe against
+    each table's uniqueness (drop the loser's row when the keeper already has the
+    equivalent, else re-point). Images themselves are never touched."""
+    # ImageRecord.primary_post_id — no uniqueness; straight re-point.
+    session.execute(
+        update(ImageRecord)
+        .where(ImageRecord.primary_post_id == loser_id)
+        .values(primary_post_id=keeper_id)
+    )
+    # ImageProvenance — unique (image_record_id, post_id).
+    dup_imgs = select(ImageProvenance.image_record_id).where(
+        ImageProvenance.post_id == keeper_id
+    )
+    # Before dropping the colliding loser rows, carry their from_attachment_id
+    # (which archive the file came out of, milestone #87) onto the keeper's
+    # surviving row when the keeper didn't record one. For the gallery-dl→native
+    # case this very milestone targets, the keeper is the native stub (no
+    # archive) and the loser is the gallery-dl row that extracted the member, so
+    # a blind delete would silently lose the containing-archive linkage.
+    for img_id, att_id in session.execute(
+        select(ImageProvenance.image_record_id, ImageProvenance.from_attachment_id)
+        .where(
+            ImageProvenance.post_id == loser_id,
+            ImageProvenance.image_record_id.in_(dup_imgs),
+            ImageProvenance.from_attachment_id.is_not(None),
+        )
+    ).all():
+        session.execute(
+            update(ImageProvenance)
+            .where(
+                ImageProvenance.post_id == keeper_id,
+                ImageProvenance.image_record_id == img_id,
+                ImageProvenance.from_attachment_id.is_(None),
+            )
+            .values(from_attachment_id=att_id)
+        )
+    session.execute(
+        delete(ImageProvenance).where(
+            ImageProvenance.post_id == loser_id,
+            ImageProvenance.image_record_id.in_(dup_imgs),
+        )
+    )
+    session.execute(
+        update(ImageProvenance)
+        .where(ImageProvenance.post_id == loser_id)
+        .values(post_id=keeper_id)
+    )
+    # PostAttachment — partial unique (post_id, sha256) where post_id NOT NULL.
+    dup_shas = select(PostAttachment.sha256).where(PostAttachment.post_id == keeper_id)
+    session.execute(
+        delete(PostAttachment).where(
+            PostAttachment.post_id == loser_id,
+            PostAttachment.sha256.in_(dup_shas),
+        )
+    )
+    session.execute(
+        update(PostAttachment)
+        .where(PostAttachment.post_id == loser_id)
+        .values(post_id=keeper_id)
+    )
+    # ExternalLink — unique (post_id, url).
+    dup_urls = select(ExternalLink.url).where(ExternalLink.post_id == keeper_id)
+    session.execute(
+        delete(ExternalLink).where(
+            ExternalLink.post_id == loser_id,
+            ExternalLink.url.in_(dup_urls),
+        )
+    )
+    session.execute(
+        update(ExternalLink)
+        .where(ExternalLink.post_id == loser_id)
+        .values(post_id=keeper_id)
+    )
+
+
+def _fill_missing_post_fields(keeper: Post, loser: Post) -> None:
+    """Backfill the keeper's empty metadata from a loser that has it (the native
+    stub is often bare; the gallery-dl row carries date/title/body/raw_metadata)."""
+    if not keeper.post_date and loser.post_date:
+        keeper.post_date = loser.post_date
+    if not keeper.post_title and loser.post_title:
+        keeper.post_title = loser.post_title
+    if not keeper.description and loser.description:
+        keeper.description = loser.description
+    if not keeper.raw_metadata and loser.raw_metadata:
+        keeper.raw_metadata = loser.raw_metadata
+    if not keeper.attachment_count and loser.attachment_count:
+        keeper.attachment_count = loser.attachment_count
+
+
+def _canonical_post_url(post: Post, cpid: str) -> str | None:
+    """Permalink for the unified post, via the platform's derive_post_url hook
+    (subscribestar/patreon synthesize `…/posts/<id>`). Falls back to the keeper's
+    existing url when no hook applies."""
+    platform = (post.raw_metadata or {}).get("category")
+    info = PLATFORMS.get(platform) if platform else None
+    if info is not None and info.derive_post_url is not None:
+        derived = info.derive_post_url({"post_id": cpid})
+        if derived:
+            return derived
+    return post.post_url
+
+
+def reconcile_duplicate_posts(
+    session: Session, *, source_id: int | None = None, dry_run: bool = False,
+) -> dict:
+    """Unify duplicate post rows (gallery-dl attachment-id + native post-id) onto
+    one keeper per real post, re-keyed to the post id. Images untouched.
+
+    Returns:
+      dry_run=True:  {"groups": G, "posts_to_merge": L, "sample": [...]}
+      dry_run=False: {"groups": G, "merged": L, "sample": [...]}
+    where L = rows that would be (were) deleted after merging into keepers. The
+    SAME find_duplicate_post_groups predicate drives preview and apply (rule 93).
+    """
+    groups = find_duplicate_post_groups(session, source_id=source_id)
+    sample: list[dict] = []
+    losers_total = 0
+    for posts in groups:
+        cpid = _canonical_post_id(posts[0])
+        keeper = _choose_keeper(posts, cpid)
+        losers = [p for p in posts if p.id != keeper.id]
+        losers_total += len(losers)
+        if len(sample) < 50:
+            sample.append({
+                "post_id": cpid,
+                "rows": len(posts),
+                "keeper_id": keeper.id,
+                "title": keeper.post_title or f"Post {cpid}",
+            })
+        if dry_run:
+            continue
+        for loser in losers:
+            _repoint_post_links(session, loser.id, keeper.id)
+            _fill_missing_post_fields(keeper, loser)
+        keeper.external_post_id = cpid
+        new_url = _canonical_post_url(keeper, cpid)
+        if new_url:
+            keeper.post_url = new_url
+        session.flush()
+        for loser in losers:
+            session.delete(loser)
+    if dry_run:
+        return {"groups": len(groups), "posts_to_merge": losers_total, "sample": sample}
+    session.commit()
+    return {"groups": len(groups), "merged": losers_total, "sample": sample}
+
+
 # Legacy tags FC no longer uses, in two shapes:
 #   (1) kinds the tag input never produces — archive/post/artist.
 #       provenance (post grouping) + archive membership are their own
@@ -574,7 +796,7 @@ def reset_content_tagging(session: Session, *, dry_run: bool = False) -> dict:
     can re-tag from scratch via the Camie auto-suggest.
 
     PRESERVED: fandom + series tags and their series_page ordering, plus every
-    image's image_record.tagger_predictions (untouched) so suggestions
+    image's image_prediction rows (untouched) so suggestions
     repopulate immediately. CASCADE on image_tag / tag_alias / tag_allowlist /
     tag_reference_embedding / tag_suggestion_rejection clears each deleted
     tag's applications + metadata. Tag.fandom_id is SET NULL, so deleting
@@ -940,3 +1162,389 @@ def reextract_archive_attachments(
         except Exception as exc:
             log.warning("re-extract enqueue failed for image %s: %s", img_id, exc)
     return summary
+
+
+# ---- Tier-1 video dedup (#871) ------------------------------------------
+
+
+def _aspect_matches(w, h, cw, ch) -> bool:
+    """Same aspect ratio within tolerance; missing dims don't block (duration is
+    the primary signal). Mirrors Importer._video_aspect_matches."""
+    if not (w and h and cw and ch):
+        return True
+    return abs((w / h) - (cw / ch)) <= _VIDEO_DUP_ASPECT_TOL
+
+
+def backfill_video_durations(session: Session) -> int:
+    """Populate image_record.duration_seconds for video rows imported before #871
+    (NULL). Idempotent — only NULL rows are touched, so a re-run after a timeout
+    naturally resumes. A probe that yields no duration writes the
+    _VIDEO_DURATION_UNKNOWN sentinel so the file isn't re-probed forever (and can
+    never match a real duration). Returns the count of rows given a real duration.
+    """
+    populated = 0
+    while True:
+        rows = session.execute(
+            select(ImageRecord.id, ImageRecord.path)
+            .where(
+                ImageRecord.mime.like("video/%"),
+                ImageRecord.duration_seconds.is_(None),
+            )
+            .order_by(ImageRecord.id)
+            .limit(500)
+        ).all()
+        if not rows:
+            break
+        for rid, path in rows:
+            probe = safe_probe.probe_video(Path(path))
+            dur = probe.duration if probe.ok and probe.duration else None
+            session.execute(
+                update(ImageRecord)
+                .where(ImageRecord.id == rid)
+                .values(
+                    duration_seconds=dur if dur is not None else _VIDEO_DURATION_UNKNOWN
+                )
+            )
+            if dur is not None:
+                populated += 1
+        session.commit()
+    return populated
+
+
+def _video_dup_group(members: list) -> dict:
+    """Pick the keeper (highest pixel area, then largest bytes, then lowest id for
+    stability) and describe the group."""
+    keeper = max(
+        members,
+        key=lambda m: ((m.width or 0) * (m.height or 0), m.size_bytes or 0, -m.id),
+    )
+    losers = [m for m in members if m.id != keeper.id]
+    return {
+        "artist_id": keeper.artist_id,
+        "keeper_id": keeper.id,
+        "loser_ids": [m.id for m in losers],
+        "duration": keeper.duration_seconds,
+        "count": len(members),
+        "reclaim_bytes": sum((m.size_bytes or 0) for m in losers),
+    }
+
+
+def find_video_dup_groups(session: Session) -> list[dict]:
+    """Cluster videos that are the same content (#871): same artist, duration
+    within tolerance, matching aspect ratio. Returns groups of >1 member. Greedy
+    sweep over duration-sorted rows, anchored to each cluster's first member so the
+    cluster's duration span never exceeds the tolerance (no chain drift)."""
+    rows = session.execute(
+        select(
+            ImageRecord.id, ImageRecord.artist_id, ImageRecord.duration_seconds,
+            ImageRecord.width, ImageRecord.height, ImageRecord.size_bytes,
+        )
+        .where(
+            ImageRecord.mime.like("video/%"),
+            ImageRecord.duration_seconds.is_not(None),
+            ImageRecord.duration_seconds > 0,
+            ImageRecord.artist_id.is_not(None),
+        )
+        .order_by(
+            ImageRecord.artist_id, ImageRecord.duration_seconds, ImageRecord.id
+        )
+    ).all()
+    groups: list[dict] = []
+    cluster: list = []
+    anchor = None
+    for r in rows:
+        if (
+            anchor is not None
+            and r.artist_id == anchor.artist_id
+            and (r.duration_seconds - anchor.duration_seconds)
+            <= _VIDEO_DUP_DURATION_TOL_SECONDS
+            and _aspect_matches(r.width, r.height, anchor.width, anchor.height)
+        ):
+            cluster.append(r)
+        else:
+            if len(cluster) > 1:
+                groups.append(_video_dup_group(cluster))
+            cluster = [r]
+            anchor = r
+    if len(cluster) > 1:
+        groups.append(_video_dup_group(cluster))
+    return groups
+
+
+def _relink_provenance_to_keeper(
+    session: Session, *, loser_id: int, keeper_id: int
+) -> int:
+    """Ensure the keeper has an ImageProvenance row for every post the loser was
+    linked to, so deleting the loser never drops the video off a post. Returns the
+    number of new keeper↔post links added."""
+    rows = session.execute(
+        select(ImageProvenance.post_id, ImageProvenance.source_id)
+        .where(ImageProvenance.image_record_id == loser_id)
+    ).all()
+    added = 0
+    for post_id, source_id in rows:
+        exists = session.execute(
+            select(ImageProvenance.id).where(
+                ImageProvenance.image_record_id == keeper_id,
+                ImageProvenance.post_id == post_id,
+            )
+        ).scalar_one_or_none()
+        if exists is None:
+            session.add(ImageProvenance(
+                image_record_id=keeper_id, post_id=post_id, source_id=source_id,
+            ))
+            session.flush()
+            added += 1
+    return added
+
+
+def dedup_videos(
+    session: Session, *, images_root: Path, dry_run: bool = False
+) -> dict:
+    """Find and (unless dry_run) collapse Tier-1 video duplicates (#871).
+
+    Re-probes NULL-duration videos first so the existing library participates,
+    then clusters by artist + duration + aspect and keeps the highest-res copy per
+    cluster. On apply, each loser's post links are re-pointed to the keeper BEFORE
+    the loser record + file are deleted, so no post loses the video. dry_run shares
+    the same discovery predicate and returns the projection without deleting
+    (rule 93).
+
+    NOTE: tags/curation on a loser are NOT merged onto the keeper — videos rarely
+    carry hand-curation and merging would add FK-juggling risk. Flagged as a
+    follow-up if it ever matters.
+    """
+    backfill_video_durations(session)
+    groups = find_video_dup_groups(session)
+    redundant = sum(len(g["loser_ids"]) for g in groups)
+    reclaim = sum(g["reclaim_bytes"] for g in groups)
+    sample = [
+        {
+            "keeper_id": g["keeper_id"],
+            "redundant": len(g["loser_ids"]),
+            "duration": round(g["duration"], 1),
+        }
+        for g in groups[:50]
+    ]
+    if dry_run:
+        return {
+            "groups": len(groups), "redundant": redundant,
+            "reclaim_bytes": reclaim, "sample": sample,
+        }
+
+    relinked = 0
+    for g in groups:
+        for loser_id in g["loser_ids"]:
+            relinked += _relink_provenance_to_keeper(
+                session, loser_id=loser_id, keeper_id=g["keeper_id"]
+            )
+        session.commit()
+
+    loser_ids = [lid for g in groups for lid in g["loser_ids"]]
+    deleted = delete_images(session, image_ids=loser_ids, images_root=images_root)
+    log.info(
+        "video dedup: %d group(s), %d redundant removed, %d post link(s) re-pointed",
+        len(groups), deleted["images_deleted"], relinked,
+    )
+    return {
+        "groups": len(groups), "redundant": redundant,
+        "reclaim_bytes": reclaim, "deleted": deleted["images_deleted"],
+        "files_deleted": deleted["files_deleted"],
+        "relinked_posts": relinked, "sample": sample,
+    }
+
+
+# ---- Gated-post blurred-preview cleanup (#874 follow-up) -----------------
+#
+# Before the #874 ingester fix, tier-gated Patreon posts (current_user_can_view
+# == False) had their BLURRED locked-preview media downloaded as if real. That
+# flag was never persisted (the sidecar/Post.raw_metadata store only
+# category/id/title/content/url), so we can't tell from the DB which stored
+# images are blurred previews. This cleanup RE-WALKS each feed to re-derive what
+# Patreon serves as gated NOW, then matches by CONTENT HASH: an ImageRecord whose
+# stored `source_filehash` equals a currently-served blurred file's hash IS that
+# blurred preview. Because the hash is content-addressed, a real file the operator
+# downloaded when they HAD access has a different hash and can never match — so
+# regained-then-lost-access content is provably spared (operator's hard
+# requirement, 2026-06-16). NULL source_filehash → can't verify → kept + reported.
+
+
+def collect_gated_previews(client, campaign_id: str) -> dict[str, list[str]]:
+    """Read-only walk of one Patreon feed → `{external_post_id: [filehashes]}` for
+    every GATED post, where the list is the CDN filehashes of the blurred preview
+    media Patreon is serving for it right now. No downloads.
+
+    `client` is a PatreonClient (or test stub) exposing the same seams the
+    ingester uses: `post_is_gated`, `iter_posts`, `extract_media`. extract_media
+    on a gated post yields exactly the blurred preview items (the original bug), so
+    their filehashes are the match keys the purge needs.
+    """
+    gated: dict[str, list[str]] = {}
+    is_gated = getattr(client, "post_is_gated", None)
+    if is_gated is None:
+        return gated
+    for post, included, _cursor in client.iter_posts(campaign_id):
+        if not is_gated(post):
+            continue
+        pid = str(post.get("id") or "")
+        if not pid:
+            continue
+        hashes = sorted(
+            {m.filehash for m in client.extract_media(post, included) if m.filehash}
+        )
+        gated[pid] = hashes
+    return gated
+
+
+def _gated_source_hashes(gated_map: dict) -> dict[int, set[str]]:
+    """{source_id: set(blurred filehashes)} from the discovery map
+    {source_id: {external_post_id: [filehashes]}}."""
+    out: dict[int, set[str]] = {}
+    for sid, posts in gated_map.items():
+        bucket: set[str] = set()
+        for hashes in posts.values():
+            bucket.update(hashes)
+        out[int(sid)] = bucket
+    return out
+
+
+def _match_gated_preview_images(
+    session: Session, all_hashes: set[str],
+) -> dict[int, int]:
+    """{image_id: size_bytes} for ImageRecords whose source_filehash is one of the
+    currently-served blurred filehashes — the confirmed blurred previews. Chunked
+    under the psycopg parameter ceiling."""
+    matched: dict[int, int] = {}
+    hash_list = list(all_hashes)
+    for i in range(0, len(hash_list), 500):
+        chunk = hash_list[i:i + 500]
+        for rid, size in session.execute(
+            select(ImageRecord.id, ImageRecord.size_bytes)
+            .where(ImageRecord.source_filehash.in_(chunk))
+        ).all():
+            matched[rid] = size or 0
+    return matched
+
+
+def _count_unverifiable_gated_images(session: Session, gated_map: dict) -> int:
+    """Images linked (via provenance) to a gated post but with a NULL
+    source_filehash — we can't prove they're blurred previews, so they're KEPT and
+    only reported. Counts distinct images so one shared across posts counts once."""
+    image_ids: set[int] = set()
+    for sid, posts in gated_map.items():
+        ext_ids = list(posts.keys())
+        if not ext_ids:
+            continue
+        post_ids = session.execute(
+            select(Post.id).where(
+                Post.source_id == int(sid), Post.external_post_id.in_(ext_ids)
+            )
+        ).scalars().all()
+        if not post_ids:
+            continue
+        rows = session.execute(
+            select(ImageProvenance.image_record_id)
+            .join(ImageRecord, ImageRecord.id == ImageProvenance.image_record_id)
+            .where(
+                ImageProvenance.post_id.in_(list(post_ids)),
+                ImageRecord.source_filehash.is_(None),
+            )
+        ).scalars().all()
+        image_ids.update(rows)
+    return len(image_ids)
+
+
+def purge_gated_previews(
+    session: Session, *, gated_map: dict, images_root: Path, dry_run: bool = False,
+) -> dict:
+    """Find and (unless dry_run) delete the blurred locked-preview images grabbed
+    from gated posts before the #874 fix (see module section above).
+
+    `gated_map` is the discovery output `{source_id: {external_post_id:
+    [filehashes]}}` (assembled by collect_gated_previews per source). Matching is
+    by content hash, so real content is provably spared. dry_run shares the exact
+    same match predicate and returns the projection without deleting (rule 93).
+
+    On apply: delete the matched ImageRecords + files (provenance cascades), clear
+    the seen/dead-letter ledger rows for those blurred hashes so the REAL media
+    re-ingests if access is later regained, and delete the gated post records that
+    are left bare.
+    """
+    per_source = _gated_source_hashes(gated_map)
+    all_hashes: set[str] = set()
+    for bucket in per_source.values():
+        all_hashes |= bucket
+    gated_posts = sum(len(posts) for posts in gated_map.values())
+
+    matched = _match_gated_preview_images(session, all_hashes)
+    reclaim = sum(matched.values())
+    unverifiable = _count_unverifiable_gated_images(session, gated_map)
+    sample = [{"image_id": rid} for rid in sorted(matched)[:50]]
+
+    projection = {
+        "gated_posts": gated_posts,
+        "matched": len(matched),
+        "reclaim_bytes": reclaim,
+        "unverifiable": unverifiable,
+        "sample": sample,
+    }
+    if dry_run:
+        return projection
+
+    deleted = delete_images(
+        session, image_ids=list(matched), images_root=images_root,
+    )
+
+    # Clear the seen + dead-letter ledger for the blurred hashes so a later
+    # recovery/backfill re-ingests the REAL media if access is regained.
+    ledger_cleared = 0
+    for sid, hashes in per_source.items():
+        hl = list(hashes)
+        for i in range(0, len(hl), 500):
+            chunk = hl[i:i + 500]
+            res = session.execute(
+                delete(PatreonSeenMedia).where(
+                    PatreonSeenMedia.source_id == sid,
+                    PatreonSeenMedia.filehash.in_(chunk),
+                )
+            )
+            ledger_cleared += res.rowcount or 0
+            session.execute(
+                delete(PatreonFailedMedia).where(
+                    PatreonFailedMedia.source_id == sid,
+                    PatreonFailedMedia.filehash.in_(chunk),
+                )
+            )
+    session.commit()
+
+    # Delete gated post records left bare by the deletion (shares the bare-post
+    # predicate so a gated post that still has real content is never removed).
+    bare = _bare_post_conditions()
+    posts_deleted = 0
+    for sid, posts in gated_map.items():
+        ext_ids = list(posts.keys())
+        for i in range(0, len(ext_ids), 500):
+            chunk = ext_ids[i:i + 500]
+            res = session.execute(
+                Post.__table__.delete().where(
+                    Post.source_id == int(sid),
+                    Post.external_post_id.in_(chunk),
+                    *bare,
+                )
+            )
+            posts_deleted += res.rowcount or 0
+    session.commit()
+
+    log.info(
+        "gated-preview purge: %d gated post(s), %d blurred image(s) deleted, "
+        "%d ledger row(s) cleared, %d bare post(s) removed, %d unverifiable kept",
+        gated_posts, deleted["images_deleted"], ledger_cleared, posts_deleted,
+        unverifiable,
+    )
+    return {
+        **projection,
+        "deleted": deleted["images_deleted"],
+        "files_deleted": deleted["files_deleted"],
+        "ledger_cleared": ledger_cleared,
+        "posts_deleted": posts_deleted,
+    }

backend/app/services/download_backends.py

@@ -24,19 +24,32 @@ import asyncio
 from pathlib import Path
 
 from .gallery_dl import DownloadResult, ErrorType
+from .native_ingest_common import NativeIngestError
 from .patreon_ingester import PatreonIngester
 from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
+from .subscribestar_ingester import SubscribeStarIngester
 
 # Platforms whose download + verify go through the native ingester rather than
-# gallery-dl. gallery-dl still serves every other platform (subscribestar,
-# hentaifoundry, discord, pixiv, deviantart) unchanged.
-NATIVE_INGESTER_PLATFORMS = frozenset({"patreon"})
+# gallery-dl. gallery-dl still serves the rest (hentaifoundry, discord, pixiv,
+# deviantart) until they migrate too.
+NATIVE_INGESTER_PLATFORMS = frozenset({"patreon", "subscribestar"})
 
 # Mirrors patreon_resolver._CAMPAIGNS_URL — surfaced in resolution-failure
 # messages so the operator sees the exact lookup endpoint that was hit.
 _CAMPAIGNS_API = "https://www.patreon.com/api/campaigns"
 
 
+def _native_ingester_cls(platform: str):
+    """The native ingester class for `platform` (uniform constructor signature).
+    A call-time lookup (not a module-level dict captured at import) so tests can
+    monkeypatch db_mod.PatreonIngester / SubscribeStarIngester and have the
+    dispatch pick up the replacement."""
+    return {
+        "patreon": PatreonIngester,
+        "subscribestar": SubscribeStarIngester,
+    }[platform]
+
+
 def uses_native_ingester(platform: str) -> bool:
     """True when `platform` is served by the native ingester (not gallery-dl).
     The single predicate the download path and verify both route on."""
@@ -81,22 +94,36 @@ async def run_download(
     return result, None
 
 
+async def _resolve_native_campaign_id(
+    platform: str, url: str, cookies_path: str | None, overrides: dict,
+) -> tuple[str | None, str | None]:
+    """`(campaign_id, resolved_campaign_id)` for a native source. SubscribeStar's
+    feed id IS the creator URL (no lookup → resolved None). Patreon resolves the
+    campaign id from the vanity URL (resolved non-None when a lookup actually ran,
+    so phase 3 caches it)."""
+    if platform == "subscribestar":
+        return url, None
+    return await resolve_campaign_id_for_source(url, cookies_path, overrides)
+
+
 async def _run_native_ingester(
     ctx: dict, source_config, mode: str | None, gdl, sync_session_factory,
 ) -> tuple[DownloadResult, str | None]:
-    """Patreon (today the only native platform): resolve the campaign id, then run
-    the native ingester in a worker thread (it is sync requests/subprocess).
+    """Run the native ingester for a native platform in a worker thread (sync
+    requests/subprocess). Patreon resolves a campaign id from the vanity URL;
+    SubscribeStar's feed id is the creator URL itself. A campaign id we cannot
+    resolve is a loud NOT_FOUND — never a silent empty success.
 
-    `resolved_campaign_id` is non-None only when we had to look it up from the
-    vanity URL this run, so phase 3 caches it the way the old gallery-dl retry
-    did. A campaign id we cannot resolve is a loud NOT_FOUND — never a silent
-    empty success.
+    `resolved_campaign_id` is non-None only when a lookup ran this call, so phase
+    3 caches it the way the old gallery-dl retry did.
     """
+    platform = ctx["platform"]
     overrides = ctx["config_overrides"] or {}
-    campaign_id, resolved_campaign_id = await resolve_campaign_id_for_source(
-        ctx["url"], ctx["cookies_path"], overrides
+    campaign_id, resolved_campaign_id = await _resolve_native_campaign_id(
+        platform, ctx["url"], ctx["cookies_path"], overrides
     )
     if not campaign_id:
+        # Only reachable for Patreon (SubscribeStar's campaign id is the URL).
         url = ctx["url"]
         vanity = extract_vanity(url)
         return (
@@ -104,7 +131,7 @@ async def _run_native_ingester(
                 success=False,
                 url=url,
                 artist_slug=ctx["artist_slug"],
-                platform="patreon",
+                platform=platform,
                 error_type=ErrorType.NOT_FOUND,
                 error_message=(
                     f"Could not resolve Patreon campaign id. source_url={url!r}; "
@@ -127,7 +154,7 @@ async def _run_native_ingester(
         if source_config.sleep_request is not None
         else max(0.5, rate_limit / 4)
     )
-    ingester = PatreonIngester(
+    ingester = _native_ingester_cls(platform)(
         images_root=gdl.images_root,
         cookies_path=ctx["cookies_path"],
         session_factory=sync_session_factory,
@@ -174,10 +201,8 @@ async def preview_source(
     """
     import asyncio
 
-    from .patreon_client import PatreonAPIError
-
-    campaign_id, _ = await resolve_campaign_id_for_source(
-        url, cookies_path, config_overrides or {}
+    campaign_id, _ = await _resolve_native_campaign_id(
+        platform, url, cookies_path, config_overrides or {}
     )
     if not campaign_id:
         vanity = extract_vanity(url)
@@ -188,7 +213,7 @@ async def preview_source(
                 "(cookies expired, or the creator moved/renamed?)."
             )
         }
-    ingester = PatreonIngester(
+    ingester = _native_ingester_cls(platform)(
         images_root=images_root,
         cookies_path=cookies_path,
         session_factory=sync_session_factory,
@@ -199,7 +224,7 @@ async def preview_source(
             None,
             lambda: ingester.preview(source_id, campaign_id, page_limit=page_limit),
         )
-    except PatreonAPIError as exc:
+    except NativeIngestError as exc:
         return {"error": f"Couldn't preview: {exc}"}
     return result
 
@@ -221,7 +246,12 @@ async def verify_source_credential(
     """
     if uses_native_ingester(platform):
         # Native ingester platforms verify via their own lightweight auth probe
-        # (resolve campaign id + one authenticated API page). Patreon today.
+        # (one authenticated feed fetch). SubscribeStar's probe takes the creator
+        # URL directly; Patreon's resolves the campaign id first.
+        if platform == "subscribestar":
+            from .subscribestar_ingester import verify_subscribestar_credential
+
+            return await verify_subscribestar_credential(url, cookies_path, config_overrides)
         from .patreon_ingester import verify_patreon_credential
 
         return await verify_patreon_credential(url, cookies_path, config_overrides)

backend/app/services/download_service.py

@@ -111,6 +111,10 @@ class DownloadService:
         # `_backfill_bypass_seen` flag rides alongside the running backfill state;
         # download mode is "recovery" when both are set.
         bypass_seen = bool(overrides.get("_backfill_bypass_seen"))
+        # #830 recapture: re-grab post bodies/links + localize on-disk inline
+        # images, WITHOUT re-downloading media. Rides alongside the backfill
+        # state like _backfill_bypass_seen; mutually exclusive with it.
+        recapture = bool(overrides.get("_backfill_recapture"))
         if in_backfill:
             skip_value: bool | str = BACKFILL_SKIP_VALUE
             source_config.timeout = BACKFILL_CHUNK_SECONDS
@@ -130,6 +134,8 @@ class DownloadService:
         if uses_native_ingester(ctx["platform"]):
             if in_backfill and bypass_seen:
                 mode = "recovery"
+            elif in_backfill and recapture:
+                mode = "recapture"
             elif in_backfill:
                 mode = "backfill"
             else:
@@ -268,6 +274,14 @@ class DownloadService:
             artist = self.sync_session.get(Artist, ctx["artist_id"])
         source_row = self.sync_session.get(Source, ctx["source_id"])
 
+        # Post-first (#856): on the native ingester the post-record is the sole
+        # writer of the post body/links, so the per-media import must NOT apply post
+        # fields. gallery-dl platforms keep writing them via the sidecar. The
+        # Importer is per-task, so this per-instance flag is safe. uses_native_ingester
+        # is the future-proof seam — a platform migrating onto the native core
+        # flips to post-first automatically (milestone #67, step 3).
+        self.importer.post_first = uses_native_ingester(ctx["platform"])
+
         import_summary = {"attached": 0, "skipped": 0, "errors": 0}
         # Archives detected but captured WITHOUT extracting any image (probe
         # rejected / corrupt / missing extractor backend). Surfaced on the event
@@ -380,6 +394,54 @@ class DownloadService:
             else:
                 import_summary["errors"] += 1
 
+        # Post-only records: media-less posts (pure-text) the native ingester
+        # captured so the artist archive is complete. Upsert each (keyed on
+        # external_post_id → updates the same Post a media import would create,
+        # never doubles). No file to clean up; the sidecar stays on disk.
+        for rec_str in getattr(dl_result, "post_record_paths", None) or []:
+            rec_path = Path(rec_str)
+            if not rec_path.exists():  # noqa: ASYNC240
+                continue
+
+            def _upsert(p=rec_path):
+                return self.importer.upsert_post_record(
+                    p, artist=artist, source=source_row,
+                )
+
+            await loop.run_in_executor(None, _upsert)
+
+        # #830 recapture: backfill source_filehash on EXISTING on-disk images so
+        # their post-body inline <img src=CDN> remaps to the local copy. A
+        # SEPARATE non-deleting channel (NOT the import list — that would unlink
+        # the duplicate file). Empty outside recapture mode.
+        relink_pairs = getattr(dl_result, "relink_source_paths", None) or []
+        relinked = 0
+        for rel_str, rel_url in relink_pairs:
+            rel_path = Path(rel_str)
+            if not rel_path.exists():  # noqa: ASYNC240
+                continue
+
+            def _relink(p=rel_path, u=rel_url):
+                return self.importer.relink_source_filehash(p, u, artist=artist)
+
+            if await loop.run_in_executor(None, _relink):
+                relinked += 1
+        if relink_pairs:
+            # recapture diagnostic: how many on-disk images got their
+            # source_filehash backfilled (inline-image localization). < total is
+            # normal — files already carrying a filehash are skipped (NULL-only).
+            log.info(
+                "recap: relinked source_filehash on %d/%d on-disk image(s)",
+                relinked, len(relink_pairs),
+            )
+
+        # Kick the off-platform file-host downloader for any links this run
+        # recorded (mega/gdrive/…). Global + idempotent (only claims pending/
+        # retryable rows); the beat sweep is the backstop. Lazy import dodges a
+        # task-module import cycle.
+        from ..tasks.external import sweep_external_links
+        sweep_external_links.delay()
+
         ev = (await self.async_session.execute(
             select(DownloadEvent).where(DownloadEvent.id == event_id)
         )).scalar_one()
@@ -485,6 +547,8 @@ class DownloadService:
             # plan #697: a recovery walk shares this lifecycle; clear its bypass
             # flag on completion so the next routine tick honors the seen-ledger.
             new_overrides.pop("_backfill_bypass_seen", None)
+            # #830: same for the recapture flag.
+            new_overrides.pop("_backfill_recapture", None)
             src.config_overrides = new_overrides
             src.backfill_runs_remaining = 0
             return

backend/app/services/external_fetch.py

@@ -0,0 +1,292 @@
+"""Fetchers for off-platform file hosts (mega / gdrive / mediafire / dropbox /
+pixeldrain).
+
+A shared, reusable subsystem: given an external_link URL, fetch the file(s) into
+a destination directory and report the outcome. The download worker (separate
+slice) drives these off the external_link ledger; any in-house downloader can
+call `fetch_external()` directly.
+
+No single tool covers all five hosts, so a small registry maps host → fetch
+function behind one signature:
+
+    fetch_external(host, url, dest_dir, *, read_timeout, total_timeout, should_stop)
+        -> FetchResult
+
+Backends:
+  - dropbox    : force the direct-download variant (dl=1) + stream GET.
+  - pixeldrain : GET the /api/file/{id} endpoint.
+  - mediafire  : scrape the download page for the direct link + stream GET.
+  - gdrive     : gdown (handles the confirm-token + virus-scan interstitial).
+  - mega       : `megatools dl` subprocess (public link incl. #key); needs the
+                 `megatools` binary in the runtime image (Debian apt package).
+
+Public links work credential-free (rule 26); per-host creds are a later Settings
+concern. Plain-HTTP homelab — no secure-context API. The HTTP / gdown /
+subprocess calls go through module-level seams so unit tests run without
+network, gdown, or MEGAcmd.
+"""
+from __future__ import annotations
+
+import logging
+import os
+import re
+import subprocess
+import time
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from pathlib import Path
+from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
+
+import requests
+
+log = logging.getLogger(__name__)
+
+_CHUNK = 1 << 16
+# Two distinct limits, because conflating them (the old single 3000s value) meant
+# a stalled HTTP connection tied up a download-worker slot + the per-host lock for
+# ~50 min before failing (operator-flagged 2026-06-17):
+#   * READ timeout — max idle gap between bytes on an HTTP socket. A stalled host
+#     (socket open, nothing flowing) is the common failure mode; a short read
+#     timeout fails it fast. This is what requests' `timeout` actually enforces —
+#     per-read, never a total.
+#   * TOTAL budget — generous wall-clock cap for a file that IS actively
+#     transferring (big films/packs). Enforced as a deadline across chunks, since
+#     no HTTP client timeout bounds the total. Also the subprocess total for mega.
+_CONNECT_TIMEOUT = 30.0
+_READ_TIMEOUT = 60.0
+_TOTAL_TIMEOUT = 1800.0  # 30 min per fetch
+_USER_AGENT = (
+    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
+)
+# MediaFire's download page embeds the real file URL in a download-button href.
+_MEDIAFIRE_RE = re.compile(
+    r'href="(https://download[^"]+?\.mediafire\.com/[^"]+)"', re.IGNORECASE
+)
+_CD_FILENAME_RE = re.compile(r'filename\*?=(?:UTF-8\'\')?"?([^";]+)"?', re.IGNORECASE)
+
+
+@dataclass
+class FetchResult:
+    files: list[Path] = field(default_factory=list)
+    bytes: int = 0
+    error: str | None = None
+
+    @property
+    def ok(self) -> bool:
+        return self.error is None and bool(self.files)
+
+
+class ExternalFetchError(Exception):
+    """A fetch failed in a way worth recording on the link's last_error."""
+
+
+# -- seams (monkeypatched in tests) ----------------------------------------
+
+def _http_get(url: str, *, timeout: float, headers: dict | None = None,
+              stream: bool = True) -> requests.Response:
+    hdrs = {"User-Agent": _USER_AGENT}
+    if headers:
+        hdrs.update(headers)
+    return requests.get(url, timeout=timeout, headers=hdrs, stream=stream)
+
+
+def _gdown_download(url: str, out_dir: str) -> str | None:
+    """Download a Google Drive url into out_dir via gdown; return the written
+    path. Imported lazily so the dep is optional at module-import time."""
+    # Lazy import: keeps gdown an optional dep that isn't needed to import this
+    # module (e.g. the no-DB unit lane that never exercises a real fetch).
+    import gdown
+    # A trailing sep tells gdown to keep the server-side filename inside out_dir.
+    return gdown.download(url, output=out_dir + os.sep, quiet=True, fuzzy=True)
+
+
+def _run_mega_get(url: str, out_dir: str, *, timeout: float) -> None:
+    """Download a mega.nz public link (key in the #fragment) into out_dir via
+    `megatools dl` (the Debian `megatools` package). Raises ExternalFetchError
+    on non-zero exit."""
+    # Fixed argv (not shell): only `url` is external input, passed positionally,
+    # so there's no shell-injection surface.
+    proc = subprocess.run(
+        ["megatools", "dl", "--path", out_dir, url],
+        capture_output=True, text=True, timeout=timeout, check=False,
+    )
+    if proc.returncode != 0:
+        raise ExternalFetchError(
+            f"megatools dl exit {proc.returncode}: {(proc.stderr or '').strip()[:300]}"
+        )
+
+
+# -- helpers ---------------------------------------------------------------
+
+def _safe_name(name: str, fallback: str) -> str:
+    name = os.path.basename((name or "").strip().strip('"'))
+    # Strip path separators / control chars; never empty.
+    name = re.sub(r'[<>:"/\\|?*\x00-\x1f]', "_", name).strip(". ")
+    return name or fallback
+
+
+def _filename_from(resp: requests.Response, url: str, fallback: str) -> str:
+    cd = resp.headers.get("Content-Disposition", "")
+    m = _CD_FILENAME_RE.search(cd)
+    if m:
+        return _safe_name(m.group(1), fallback)
+    path_name = os.path.basename(urlsplit(url).path)
+    return _safe_name(path_name, fallback)
+
+
+def _stream_to_file(resp: requests.Response, dest: Path,
+                    should_stop: Callable[[], bool],
+                    *, deadline: float | None = None) -> int:
+    """Stream a response body to `dest` (atomic via .part). Returns byte count.
+    Honors should_stop and the total-budget `deadline` (a time.monotonic() value)
+    between chunks; the partial file is removed on either abort."""
+    part = dest.with_name(dest.name + ".part")
+    total = 0
+    try:
+        with part.open("wb") as fh:
+            for chunk in resp.iter_content(chunk_size=_CHUNK):
+                if should_stop():
+                    raise ExternalFetchError("stopped")
+                if deadline is not None and time.monotonic() > deadline:
+                    raise ExternalFetchError("exceeded total fetch budget")
+                if chunk:
+                    fh.write(chunk)
+                    total += len(chunk)
+    except BaseException:
+        part.unlink(missing_ok=True)
+        raise
+    os.replace(part, dest)
+    return total
+
+
+def _get_to_dir(url: str, dest_dir: Path, *, read_timeout: float,
+                total_timeout: float, should_stop: Callable[[], bool],
+                fallback: str, headers: dict | None = None) -> FetchResult:
+    # (connect, read): a short read timeout fails a stalled socket fast; the total
+    # budget is enforced separately as a deadline across chunks (requests has no
+    # total-download timeout).
+    resp = _http_get(
+        url, timeout=(_CONNECT_TIMEOUT, read_timeout), headers=headers, stream=True
+    )
+    if resp.status_code != 200:
+        return FetchResult(error=f"HTTP {resp.status_code} for {url}")
+    name = _filename_from(resp, url, fallback)
+    dest = dest_dir / name
+    written = _stream_to_file(
+        resp, dest, should_stop, deadline=time.monotonic() + total_timeout
+    )
+    return FetchResult(files=[dest], bytes=written)
+
+
+# -- per-host fetchers -----------------------------------------------------
+
+def _fetch_dropbox(url: str, dest_dir: Path, *, read_timeout: float,
+                   total_timeout: float,
+                   should_stop: Callable[[], bool]) -> FetchResult:
+    # Force the direct-download variant: dl=1 (Dropbox serves an HTML preview
+    # for dl=0). Rewrite/insert the param rather than string-replace so ?dl=0,
+    # &dl=0, and a missing param all resolve.
+    parts = urlsplit(url)
+    q = dict(parse_qsl(parts.query))
+    q["dl"] = "1"
+    direct = urlunsplit(parts._replace(query=urlencode(q)))
+    return _get_to_dir(direct, dest_dir, read_timeout=read_timeout,
+                       total_timeout=total_timeout, should_stop=should_stop,
+                       fallback="dropbox-file")
+
+
+def _fetch_pixeldrain(url: str, dest_dir: Path, *, read_timeout: float,
+                      total_timeout: float,
+                      should_stop: Callable[[], bool]) -> FetchResult:
+    # /u/{id} (and /l/{id}) → the API file endpoint.
+    file_id = urlsplit(url).path.rstrip("/").split("/")[-1]
+    if not file_id:
+        return FetchResult(error=f"no pixeldrain id in {url}")
+    api = f"https://pixeldrain.com/api/file/{file_id}"
+    return _get_to_dir(api, dest_dir, read_timeout=read_timeout,
+                       total_timeout=total_timeout, should_stop=should_stop,
+                       fallback=f"{file_id}.bin")
+
+
+def _fetch_mediafire(url: str, dest_dir: Path, *, read_timeout: float,
+                     total_timeout: float,
+                     should_stop: Callable[[], bool]) -> FetchResult:
+    page = _http_get(url, timeout=(_CONNECT_TIMEOUT, read_timeout), stream=False)
+    if page.status_code != 200:
+        return FetchResult(error=f"HTTP {page.status_code} for mediafire page")
+    m = _MEDIAFIRE_RE.search(page.text or "")
+    if not m:
+        return FetchResult(error="mediafire direct link not found on page")
+    return _get_to_dir(m.group(1), dest_dir, read_timeout=read_timeout,
+                       total_timeout=total_timeout, should_stop=should_stop,
+                       fallback="mediafire-file")
+
+
+def _fetch_gdrive(url: str, dest_dir: Path, *, read_timeout: float,
+                  total_timeout: float,
+                  should_stop: Callable[[], bool]) -> FetchResult:
+    # gdown manages its own HTTP session/timeouts; the task's celery hard limit is
+    # the outer backstop. read_timeout/total_timeout are accepted for a uniform
+    # registry signature but not separately enforceable here.
+    out = _gdown_download(url, str(dest_dir))
+    if not out:
+        return FetchResult(error="gdown returned no file (quota / private?)")
+    p = Path(out)
+    if not p.exists():
+        return FetchResult(error=f"gdown reported {out} but it is missing")
+    return FetchResult(files=[p], bytes=p.stat().st_size)
+
+
+def _fetch_mega(url: str, dest_dir: Path, *, read_timeout: float,
+                total_timeout: float,
+                should_stop: Callable[[], bool]) -> FetchResult:
+    before = set(dest_dir.iterdir()) if dest_dir.exists() else set()
+    # megatools is a subprocess: its timeout IS a total wall-clock cap (the read
+    # timeout has no analogue here), so the total budget applies directly.
+    _run_mega_get(url, str(dest_dir), timeout=total_timeout)
+    new = [p for p in dest_dir.iterdir() if p not in before and p.is_file()]
+    if not new:
+        return FetchResult(error="mega-get wrote no new file")
+    return FetchResult(files=new, bytes=sum(p.stat().st_size for p in new))
+
+
+_REGISTRY: dict[str, Callable[..., FetchResult]] = {
+    "dropbox": _fetch_dropbox,
+    "pixeldrain": _fetch_pixeldrain,
+    "mediafire": _fetch_mediafire,
+    "gdrive": _fetch_gdrive,
+    "mega": _fetch_mega,
+}
+
+SUPPORTED_HOSTS = tuple(_REGISTRY)
+
+
+def fetch_external(host: str, url: str, dest_dir: Path, *,
+                   read_timeout: float = _READ_TIMEOUT,
+                   total_timeout: float = _TOTAL_TIMEOUT,
+                   should_stop: Callable[[], bool] = lambda: False) -> FetchResult:
+    """Fetch `url` (a `host` link) into `dest_dir`. Returns a FetchResult; never
+    raises — any backend error (transport, read/total timeout, non-200, scrape
+    miss, subprocess failure, stop) is captured on `.error` so the worker can
+    record it and move on.
+
+    `read_timeout` fails a stalled HTTP socket fast (idle gap between bytes);
+    `total_timeout` is the generous wall-clock cap for a large file that is
+    actively transferring (and the subprocess total for mega)."""
+    fetcher = _REGISTRY.get(host)
+    if fetcher is None:
+        return FetchResult(error=f"unsupported host {host!r}")
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    try:
+        return fetcher(url, dest_dir, read_timeout=read_timeout,
+                       total_timeout=total_timeout, should_stop=should_stop)
+    except requests.RequestException as exc:
+        return FetchResult(error=f"transport error: {exc}")
+    except subprocess.TimeoutExpired:
+        return FetchResult(error="timed out")
+    except ExternalFetchError as exc:
+        return FetchResult(error=str(exc))
+    except Exception as exc:  # never let a backend quirk kill the worker
+        log.warning("external fetch (%s) failed for %s: %s", host, url, exc)
+        return FetchResult(error=f"{type(exc).__name__}: {exc}")

backend/app/services/gallery_dl.py

@@ -147,6 +147,17 @@ class DownloadResult:
     files_quarantined: int = 0
     quarantined_paths: list[str] = field(default_factory=list)
     written_paths: list[str] = field(default_factory=list)
+    # Native ingester only: post-only sidecar paths for posts that have NO
+    # downloadable media (pure-text posts), so the importer can still upsert the
+    # Post + its body. Empty on the gallery-dl path. Phase 3 imports these via
+    # Importer.upsert_post_record (keyed on external_post_id → updates, never
+    # doubles, the same Post a media import would create).
+    post_record_paths: list[str] = field(default_factory=list)
+    # Native ingester, recapture mode only (#830): (on-disk path, CDN source_url)
+    # pairs for already-present media whose ImageRecord.source_filehash should be
+    # backfilled (inline-image localization) WITHOUT re-download or unlink. Empty
+    # on the gallery-dl path and outside recapture.
+    relink_source_paths: list[tuple[str, str]] = field(default_factory=list)
     stdout: str = ""
     stderr: str = ""
     return_code: int = 0
@@ -287,11 +298,8 @@ class GalleryDLService:
     # removed at the plan-#697 cutover — it now uses the native ingester
     # (services/patreon_ingester.py), not gallery-dl.
     PLATFORM_DEFAULTS = {
-        "subscribestar": {
-            "content_types": ["all"],
-            "directory": ["{date:%Y-%m-%d}_{id}_{title[:40]}"],
-            "filename": "{num:>02}_{filename}.{extension}",
-        },
+        # subscribestar removed — it's a native-ingester platform now (#71); the
+        # remaining entries are the gallery-dl platforms not yet migrated.
         "hentaifoundry": {
             "content_types": ["all"],
             "directory": [],

backend/app/services/gallery_service.py

@@ -16,6 +16,7 @@ translates that to HTTP 400.
 
 from dataclasses import dataclass
 from datetime import datetime
+from urllib.parse import quote
 
 from sqlalchemy import Select, and_, distinct, exists, func, or_, select
 from sqlalchemy.ext.asyncio import AsyncSession
@@ -24,7 +25,13 @@ from sqlalchemy.orm import aliased
 from ..models import Artist, ImageProvenance, ImageRecord, Post, Source, Tag
 from ..models.tag import image_tag
 from .pagination import decode_cursor, encode_cursor
-from .tag_query import fandom_join_alias, serialize_tag, tag_columns
+from .tag_query import (
+    fandom_join_alias,
+    image_in_any_tag_scope,
+    image_in_tag_scope,
+    serialize_tag,
+    tag_columns,
+)
 
 # Reserved `platform` filter value selecting images with NO platformed
 # provenance (filesystem imports). Returned by facets() as a null-valued
@@ -120,11 +127,30 @@ def thumbnail_url(thumbnail_path: str | None, sha256_hex: str, mime: str) -> str
     return f"/images/thumbs/{bucket}/{sha256_hex}{ext}"
 
 
-def _require_single_filter(tag_ids, post_id, artist_id) -> None:
+def image_url(path: str) -> str:
+    """Return the URL to fetch the full-size original from /images.
+
+    The on-disk `path` mirrors the source tree (artist/post folders), so it can
+    contain characters that are special in a URL — most importantly '#' (post
+    titles like 'BLUE#59'), but also spaces, '?', '%'. The serve_image route
+    URL-decodes its <path:subpath> fine, but only if the browser sends the whole
+    path; an unencoded '#' is parsed as a fragment, so '#59/01_timelapse.jpg'
+    never reaches the server and the original 404s while the (hash-named)
+    thumbnail still loads. Percent-encode the path, keeping '/' as the segment
+    separator. Operator-flagged 2026-06-12."""
+    rel = path.split("/images/", 1)[-1]
+    return f"/images/{quote(rel, safe='/')}"
+
+
+def _require_single_filter(
+    tag_ids, post_id, artist_id, tag_or_groups=None, tag_exclude=None,
+) -> None:
     """post_id is the post-detail view — it can't combine with the
-    composable filters. tag_ids + artist_id (+ media_type) compose freely
-    (AND)."""
-    if post_id is not None and (tag_ids or artist_id is not None):
+    composable filters. tag_ids / tag_or_groups / tag_exclude + artist_id
+    (+ media_type) compose freely (AND)."""
+    if post_id is not None and (
+        tag_ids or artist_id is not None or tag_or_groups or tag_exclude
+    ):
         raise ValueError(
             "post_id cannot be combined with tag or artist filters"
         )
@@ -132,6 +158,7 @@ def _require_single_filter(tag_ids, post_id, artist_id) -> None:
 
 def _apply_scope(
     stmt, *, tag_ids, post_id, artist_id, media_type,
+    tag_or_groups=None, tag_exclude=None,
     platform=None, untagged=False, no_artist=False,
     date_from=None, date_to=None,
 ):
@@ -142,7 +169,14 @@ def _apply_scope(
     be present on `stmt` (the artist/platform paths alias Post/Source inside
     their own EXISTS).
 
-    - tag_ids: image must carry ALL of them — one correlated EXISTS per tag.
+    Tag filtering is one structured model (#6): AND-of-OR plus exclusions.
+    - tag_ids: image must carry ALL of them — one correlated EXISTS per tag
+      (the AND-of-singletons "include" common case; light editor + back-compat).
+    - tag_or_groups: list of OR-groups; the image must carry AT LEAST ONE tag
+      from EACH group — one EXISTS(tag_id IN group) per group, AND'd across
+      groups. (advanced editor)
+    - tag_exclude: image must carry NONE of these — a single NOT EXISTS(tag_id
+      IN exclude). (light "exclude" chips + advanced NOT)
     - post_id / artist_id: provenance EXISTS (post_id is exclusive, guarded
       by _require_single_filter).
     - media_type: 'image' | 'video' narrows by mime prefix.
@@ -152,13 +186,17 @@ def _apply_scope(
     - no_artist: ImageRecord.artist_id IS NULL.
     - date_from / date_to: half-open [from, to) bounds on effective_date.
     """
+    # Every tag clause goes through image_in_tag_scope/_any: a fandom tag also
+    # matches images carrying any of its characters (Tag.fandom_id). Include,
+    # OR-group, and exclude are all symmetric on that membership.
     for tid in tag_ids or []:
-        stmt = stmt.where(
-            exists().where(
-                image_tag.c.image_record_id == ImageRecord.id,
-                image_tag.c.tag_id == tid,
-            )
-        )
+        stmt = stmt.where(image_in_tag_scope(tid))
+    for group in tag_or_groups or []:
+        if not group:
+            continue  # an empty OR-group would match nothing; treat as absent
+        stmt = stmt.where(image_in_any_tag_scope(group))
+    if tag_exclude:
+        stmt = stmt.where(~image_in_any_tag_scope(tag_exclude))
     prov = _provenance_clause(post_id, artist_id)
     if prov is not None:
         stmt = stmt.where(prov)
@@ -280,6 +318,8 @@ class GalleryService:
         artist_id: int | None = None,
         media_type: str | None = None,
         sort: str = "newest",
+        tag_or_groups: list[list[int]] | None = None,
+        tag_exclude: list[int] | None = None,
         platform: str | None = None,
         untagged: bool = False,
         no_artist: bool = False,
@@ -288,7 +328,9 @@ class GalleryService:
     ) -> GalleryPage:
         if limit < 1 or limit > 200:
             raise ValueError("limit must be between 1 and 200")
-        _require_single_filter(tag_ids, post_id, artist_id)
+        _require_single_filter(
+            tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
+        )
 
         eff = _effective_date_col()
         stmt = select(ImageRecord, Post.post_date, eff.label("eff"))
@@ -296,6 +338,7 @@ class GalleryService:
         stmt = _apply_scope(
             stmt, tag_ids=tag_ids, post_id=post_id,
             artist_id=artist_id, media_type=media_type,
+            tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
             platform=platform, untagged=untagged, no_artist=no_artist,
             date_from=date_from, date_to=date_to,
         )
@@ -343,6 +386,8 @@ class GalleryService:
         post_id: int | None = None,
         artist_id: int | None = None,
         media_type: str | None = None,
+        tag_or_groups: list[list[int]] | None = None,
+        tag_exclude: list[int] | None = None,
         platform: str | None = None,
         untagged: bool = False,
         no_artist: bool = False,
@@ -356,10 +401,13 @@ class GalleryService:
             year_col, month_col, func.count(ImageRecord.id).label("cnt")
         )
         stmt = _outer_join_primary_post(stmt)
-        _require_single_filter(tag_ids, post_id, artist_id)
+        _require_single_filter(
+            tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
+        )
         stmt = _apply_scope(
             stmt, tag_ids=tag_ids, post_id=post_id,
             artist_id=artist_id, media_type=media_type,
+            tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
             platform=platform, untagged=untagged, no_artist=no_artist,
             date_from=date_from, date_to=date_to,
         )
@@ -371,6 +419,8 @@ class GalleryService:
         self, year: int, month: int, tag_ids: list[int] | None = None,
         post_id: int | None = None, artist_id: int | None = None,
         media_type: str | None = None, sort: str = "newest",
+        tag_or_groups: list[list[int]] | None = None,
+        tag_exclude: list[int] | None = None,
         platform: str | None = None, untagged: bool = False,
         no_artist: bool = False, date_from: datetime | None = None,
         date_to: datetime | None = None,
@@ -387,10 +437,13 @@ class GalleryService:
             extract("month", eff) == month,
         )
         stmt = _outer_join_primary_post(stmt)
-        _require_single_filter(tag_ids, post_id, artist_id)
+        _require_single_filter(
+            tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
+        )
         stmt = _apply_scope(
             stmt, tag_ids=tag_ids, post_id=post_id,
             artist_id=artist_id, media_type=media_type,
+            tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
             platform=platform, untagged=untagged, no_artist=no_artist,
             date_from=date_from, date_to=date_to,
         )
@@ -411,7 +464,10 @@ class GalleryService:
     async def facets(
         self, *, tag_ids: list[int] | None = None,
         post_id: int | None = None, artist_id: int | None = None,
-        media_type: str | None = None, platform: str | None = None,
+        media_type: str | None = None,
+        tag_or_groups: list[list[int]] | None = None,
+        tag_exclude: list[int] | None = None,
+        platform: str | None = None,
         untagged: bool = False, no_artist: bool = False,
         date_from: datetime | None = None, date_to: datetime | None = None,
     ) -> GalleryFacets:
@@ -421,10 +477,13 @@ class GalleryService:
         No outer join is needed — every clause is a correlated EXISTS or a
         column predicate on ImageRecord.
         """
-        _require_single_filter(tag_ids, post_id, artist_id)
+        _require_single_filter(
+            tag_ids, post_id, artist_id, tag_or_groups, tag_exclude,
+        )
         common = {
             "tag_ids": tag_ids, "post_id": post_id,
             "artist_id": artist_id, "media_type": media_type,
+            "tag_or_groups": tag_or_groups, "tag_exclude": tag_exclude,
         }
 
         # total — the full active filter (the headline result count).
@@ -499,7 +558,10 @@ class GalleryService:
     async def similar(
         self, image_id: int, limit: int = 100, *,
         tag_ids: list[int] | None = None, artist_id: int | None = None,
-        media_type: str | None = None, platform: str | None = None,
+        media_type: str | None = None,
+        tag_or_groups: list[list[int]] | None = None,
+        tag_exclude: list[int] | None = None,
+        platform: str | None = None,
         untagged: bool = False, no_artist: bool = False,
         date_from: datetime | None = None, date_to: datetime | None = None,
     ) -> list[GalleryImage] | None:
@@ -531,6 +593,7 @@ class GalleryService:
         stmt = _apply_scope(
             stmt, tag_ids=tag_ids, post_id=None,
             artist_id=artist_id, media_type=media_type,
+            tag_or_groups=tag_or_groups, tag_exclude=tag_exclude,
             platform=platform, untagged=untagged, no_artist=no_artist,
             date_from=date_from, date_to=date_to,
         )
@@ -589,7 +652,7 @@ class GalleryService:
             "created_at": record.created_at.isoformat(),
             "posted_at": posted_at.isoformat() if posted_at else None,
             "thumbnail_url": thumbnail_url(record.thumbnail_path, record.sha256, record.mime),
-            "image_url": f"/images/{record.path.split('/images/', 1)[-1]}",
+            "image_url": image_url(record.path),
             "artist": (
                 {"id": artist.id, "name": artist.name, "slug": artist.slug}
                 if artist is not None else None

backend/app/services/importer.py

@@ -17,12 +17,13 @@ from enum import StrEnum
 from pathlib import Path
 
 from PIL import Image
-from sqlalchemy import select
+from sqlalchemy import select, update
 from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import Session
 
 from ..models import (
     Artist,
+    ExternalLink,
     ImageProvenance,
     ImageRecord,
     ImportSettings,
@@ -34,6 +35,7 @@ from ..utils import safe_probe
 from ..utils.paths import (
     derive_subdir,
     derive_top_level_artist,
+    filehash_from_url,
     hash_suffixed_name,
     safe_ext,
 )
@@ -42,6 +44,7 @@ from ..utils.sidecar import find_sidecar, parse_sidecar
 from ..utils.slug import slugify
 from .archive_extractor import extract_archive, is_archive
 from .attachment_store import AttachmentStore
+from .link_extract import extract_external_links
 from .thumbnailer import Thumbnailer
 
 log = logging.getLogger(__name__)
@@ -76,6 +79,21 @@ IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".tif", ".tiff"}
 VIDEO_EXTS = {".mp4", ".mov", ".avi", ".mkv", ".webm", ".m4v", ".wmv", ".flv"}
 ALL_EXTS = IMAGE_EXTS | VIDEO_EXTS
 
+# A "high-resolution files" pack often wraps per-chapter .rar/.zip inside one
+# outer archive (incase #718) — recurse into nested archives so their images
+# land instead of being silently dropped. Cap the depth so a maliciously
+# deep-nested archive can't recurse forever; the bomb-probe re-runs per level.
+_ARCHIVE_MAX_DEPTH = 3
+
+# Tier-1 video near-dup (#871). pHash is images-only, so videos deduped on sha256
+# alone — a different encode/remux of the same clip imported as a distinct record.
+# Identity = container duration within a tight tolerance + matching aspect ratio,
+# scoped to the same artist (codec/bitrate are NOT part of identity — the point is
+# to match ACROSS re-encodes). Quality axis for supersede = pixel dimensions, same
+# as image pHash. Tight tolerances because a wrong video merge is destructive.
+_VIDEO_DUP_DURATION_TOL_SECONDS = 1.0
+_VIDEO_DUP_ASPECT_TOL = 0.02
+
 
 def is_supported(path: Path) -> bool:
     return path.suffix.lower() in ALL_EXTS
@@ -146,6 +164,14 @@ class Importer:
         self.thumbnailer = thumbnailer
         self.settings = settings
         self.deep = deep
+        # Post-first ingest (#856, milestone #67): on the NATIVE core ingester the
+        # post-record (`upsert_post_record`) is the SOLE writer of a post's
+        # body/links/metadata; the per-media import only links image provenance +
+        # localization, NOT the post body. download_service sets this True for
+        # native platforms before the phase-3 media loop. Default False keeps the
+        # gallery-dl path writing post fields via `_apply_sidecar` (unchanged).
+        # Safe as a per-instance flag: Importer is constructed per Celery task.
+        self.post_first = False
         self.attachments = AttachmentStore(images_root)
         # phash near-dup candidate cache. Archive imports call _import_media
         # per-member; without this cache the per-member SELECT *FROM
@@ -305,6 +331,32 @@ class Importer:
             ),
         )
 
+    def _sync_external_links(self, post: Post) -> None:
+        """Record off-platform file-host links (mega/gdrive/mediafire/dropbox/
+        pixeldrain) found in the post body, so they're never silently dropped
+        and the download worker can fetch them later. Shared by every platform's
+        import (runs off Post.description). INSERT-MISSING only — an existing row
+        keeps its status/attempts (a re-import must not reset a link already
+        downloaded or dead-lettered). Identity is (post_id, url); the full url
+        incl. #fragment is preserved by the extractor."""
+        links = extract_external_links(post.description)
+        if not links:
+            return
+        self.session.flush()  # ensure post.id is assigned before we reference it
+        existing = set(self.session.execute(
+            select(ExternalLink.url).where(ExternalLink.post_id == post.id)
+        ).scalars().all())
+        for link in links:
+            if link.url in existing:
+                continue
+            self.session.add(ExternalLink(
+                post_id=post.id,
+                artist_id=post.artist_id,
+                host=link.host,
+                url=link.url,
+                label=link.label,
+            ))
+
     def import_one(self, source: Path) -> ImportResult:
         """Dispatch by kind. Media → normal pipeline. Archive → extract
         media members (one Post via the archive-adjacent sidecar) and
@@ -454,40 +506,212 @@ class Importer:
         artist_use = artist if artist is not None else self._resolve_artist(source)
         post = self._post_for_sidecar(source, artist_use)
         member_ids: list[int] = []
-        member_total = 0
-        with extract_archive(source) as members:
-            for _name, member_path in members:
-                member_total += 1
-                if not is_supported(member_path):
-                    continue  # non-media preserved via the stored archive
-                res = self._import_media(
-                    member_path, source, explicit_source=source_row,
-                )
-                if res.status in ("imported", "superseded") and res.image_id:
-                    member_ids.append(res.image_id)
+        # Every member image touched (new + superseded + deduped), so the
+        # from_attachment_id stamp below covers files that already existed in the
+        # library and were merely re-linked to this post — those matter most
+        # (the HR copy a bundle re-ships). Separate from member_ids, which is
+        # the NEWLY-imported subset feeding the ImportResult contract.
+        member_record_ids: set[int] = set()
+        # Per-outcome tally so the "no images" reason names the ACTUAL cause
+        # (#718): nested-archive packs, all-deduped (benign), unsupported formats,
+        # or failed/corrupt members — instead of one catch-all string.
+        counts = {
+            "media": 0, "deduped": 0, "unsupported": 0,
+            "failed": 0, "nested": 0, "nested_rejected": 0,
+        }
+        self._collect_archive_members(
+            source, attribution=source, source_row=source_row,
+            depth=0, member_ids=member_ids, counts=counts,
+            member_record_ids=member_record_ids,
+        )
         # Preserve the archive itself (links to the same Post/Artist).
         self._capture_attachment(
             source, post=post, artist=artist_use, resolved=True
         )
+        # Stamp each member's provenance row for THIS post with the archive it
+        # came out of (milestone #87). Done as a post-pass rather than threaded
+        # through _import_media/_apply_sidecar so the many dedup/supersede
+        # branches stay untouched. NULL-only so a re-extract never re-stamps and
+        # the backfill (reextract task → this same path) is idempotent. Nested
+        # members link to this OUTER archive — the only one stored as a blob.
+        self._stamp_member_archive(
+            post.id if post is not None else None, source, member_record_ids,
+        )
         if member_ids:
             return ImportResult(
                 status="imported", image_id=member_ids[0],
                 member_image_ids=member_ids,
             )
-        # No images landed — surface WHY so a post showing "no images" beside an
-        # archive is diagnosable instead of silent. Zero members usually means
-        # the extractor backend is missing/failed (unar for rar, py7zr for 7z)
-        # or the file is corrupt; non-zero-but-no-images means it held only
-        # non-media files.
-        reason = (
-            "archive extracted but held no supported image/video members"
-            if member_total
-            else "archive yielded no members (unsupported/corrupt, or the "
-            "extractor backend failed)"
-        )
+        # No NEW images landed. Name WHY precisely so a post showing "no images"
+        # beside an archive is diagnosable — and DON'T flag the benign all-deduped
+        # case as a problem: those images already exist in the library and were
+        # re-linked to this post (enrich-on-duplicate), so the post DOES show them.
+        if counts["media"] == 0 and counts["nested"] == 0:
+            reason = ("archive yielded no members (unsupported/corrupt, or the "
+                      "extractor backend failed)")
+        elif counts["deduped"] and not (
+            counts["unsupported"] or counts["failed"] or counts["nested_rejected"]
+        ):
+            log.info(
+                "%s: all %d image member(s) already in library (deduped + linked "
+                "to this post) — archive preserved, no new import",
+                source.name, counts["deduped"],
+            )
+            return ImportResult(status="attached")
+        else:
+            reason = (
+                f"no new images — media={counts['media']} deduped={counts['deduped']} "
+                f"unsupported/non-media={counts['unsupported']} failed={counts['failed']} "
+                f"nested={counts['nested']} nested-rejected={counts['nested_rejected']}"
+            )
         log.warning("%s: %s", source.name, reason)
         return ImportResult(status="attached", error=reason)
 
+    def _collect_archive_members(
+        self, archive_path: Path, *, attribution: Path,
+        source_row: Source | None, depth: int,
+        member_ids: list[int], counts: dict,
+        member_record_ids: set[int],
+    ) -> None:
+        """Extract `archive_path` and import its image/video members, RECURSING
+        into nested archives (#718). Members attribute to `attribution` — the
+        OUTER archive's path, so its sidecar resolves them to the right Post even
+        when they came from a nested archive. Depth-capped + bomb-probed per
+        nested level. Mutates `member_ids` and `counts` in place.
+
+        extract_archive is fail-soft, but each level is wrapped so one bad nested
+        archive can't abort the whole import."""
+        try:
+            with extract_archive(archive_path) as members:
+                for _name, member_path in members:
+                    if not member_path.is_file():
+                        continue  # directory entries etc. — not media
+                    if is_archive(member_path):
+                        counts["nested"] += 1
+                        if depth + 1 > _ARCHIVE_MAX_DEPTH:
+                            counts["nested_rejected"] += 1
+                            log.warning(
+                                "nested archive past depth cap %d, skipped: %s",
+                                _ARCHIVE_MAX_DEPTH, member_path.name,
+                            )
+                            continue
+                        probe = safe_probe.probe_archive(member_path)
+                        if not probe.ok:
+                            counts["nested_rejected"] += 1
+                            log.warning(
+                                "nested archive rejected (%s): %s",
+                                probe.reason, member_path.name,
+                            )
+                            continue
+                        self._collect_archive_members(
+                            member_path, attribution=attribution,
+                            source_row=source_row, depth=depth + 1,
+                            member_ids=member_ids, counts=counts,
+                            member_record_ids=member_record_ids,
+                        )
+                        continue
+                    counts["media"] += 1
+                    if not is_supported(member_path):
+                        counts["unsupported"] += 1
+                        continue  # non-media preserved via the stored archive
+                    res = self._import_media(
+                        member_path, attribution, explicit_source=source_row,
+                    )
+                    if res.status in ("imported", "superseded") and res.image_id:
+                        member_ids.append(res.image_id)
+                        member_record_ids.add(res.image_id)
+                    elif res.status == "skipped" and res.skip_reason in (
+                        SkipReason.duplicate_hash, SkipReason.duplicate_phash
+                    ):
+                        counts["deduped"] += 1
+                        # A deduped member still links provenance to this post
+                        # (enrich-on-duplicate); record it so its archive origin
+                        # gets stamped too.
+                        if res.image_id:
+                            member_record_ids.add(res.image_id)
+                    else:
+                        counts["failed"] += 1
+        except Exception as exc:  # noqa: BLE001 — defensive per level; keep going
+            log.warning(
+                "archive extraction failed for %s (depth %d): %s",
+                archive_path.name, depth, exc,
+            )
+
+    def _stamp_member_archive(
+        self, post_id: int | None, archive_source: Path, member_record_ids: set[int],
+    ) -> None:
+        """Record which archive each extracted member came from (milestone #87).
+
+        Resolves the archive's own PostAttachment (by post + sha — it was just
+        captured) and stamps from_attachment_id on every member's provenance row
+        FOR THIS POST. NULL-only, so re-extracting the same archive (the backfill
+        path) never overwrites and stays idempotent. No-op when the archive isn't
+        post-attached (filesystem import with no post) or yielded no members.
+        """
+        if post_id is None or not member_record_ids:
+            return
+        sha = _sha256_of(archive_source)
+        att_id = self.session.execute(
+            select(PostAttachment.id).where(
+                PostAttachment.post_id == post_id,
+                PostAttachment.sha256 == sha,
+            )
+        ).scalar_one_or_none()
+        if att_id is None:
+            return
+        self.session.execute(
+            update(ImageProvenance)
+            .where(
+                ImageProvenance.image_record_id.in_(member_record_ids),
+                ImageProvenance.post_id == post_id,
+                ImageProvenance.from_attachment_id.is_(None),
+            )
+            .values(from_attachment_id=att_id)
+        )
+        self.session.commit()
+
+    @staticmethod
+    def _video_aspect_matches(w, h, cw, ch) -> bool:
+        """True when two (w,h) pairs share an aspect ratio within tolerance.
+        Missing dims → don't block (duration is the primary signal)."""
+        if not (w and h and cw and ch):
+            return True
+        return abs((w / h) - (cw / ch)) <= _VIDEO_DUP_ASPECT_TOL
+
+    def _find_similar_video(
+        self, duration: float | None, width: int | None, height: int | None,
+        artist_id: int | None,
+    ) -> tuple[str, int | None]:
+        """Tier-1 video near-dup (#871). Find a same-artist video whose duration
+        matches within tolerance AND whose aspect ratio matches — treat it as the
+        same content (a different encode/remux). Returns find_similar's contract:
+        ("none"|"larger_exists"|"smaller_exists", id), quality judged by pixel
+        dimensions so a higher-res copy supersedes a smaller one."""
+        if duration is None or artist_id is None:
+            return ("none", None)
+        lo = duration - _VIDEO_DUP_DURATION_TOL_SECONDS
+        hi = duration + _VIDEO_DUP_DURATION_TOL_SECONDS
+        rows = self.session.execute(
+            select(ImageRecord.id, ImageRecord.width, ImageRecord.height)
+            .where(
+                ImageRecord.artist_id == artist_id,
+                ImageRecord.mime.like("video/%"),
+                ImageRecord.duration_seconds.is_not(None),
+                ImageRecord.duration_seconds >= lo,
+                ImageRecord.duration_seconds <= hi,
+            )
+        ).all()
+        w, h = width or 0, height or 0
+        for cid, cw, ch in rows:
+            if not self._video_aspect_matches(width, height, cw, ch):
+                continue
+            cw0, ch0 = cw or 0, ch or 0
+            if cw0 >= w and ch0 >= h:
+                return ("larger_exists", cid)
+            if w > cw0 or h > ch0:
+                return ("smaller_exists", cid)
+        return ("none", None)
+
     def _import_media(
         self, source: Path, attribution_path: Path,
         *, explicit_source: Source | None = None,
@@ -506,13 +730,13 @@ class Importer:
 
         # Compute file dimensions (images only) and apply filters.
         width = height = None
+        duration = None  # video container duration (Tier-1 near-dup key, #871)
         has_alpha = False
         if is_video(source):
             # Layer-3 isolation: validate the container via ffprobe (a
             # separate process) before the rest of the pipeline touches
             # it. A corrupt video that would crash a decoder is rejected
-            # cleanly here, and we capture width/height for free (the
-            # importer didn't previously record video dimensions).
+            # cleanly here, and we capture width/height + duration for free.
             probe = safe_probe.probe_video(source)
             if not probe.ok:
                 if probe.crashed:
@@ -524,7 +748,7 @@ class Importer:
                     status="skipped", skip_reason=SkipReason.invalid_image,
                     error=probe.reason,
                 )
-            width, height = probe.width, probe.height
+            width, height, duration = probe.width, probe.height, probe.duration
         else:
             try:
                 with Image.open(source) as im:
@@ -566,6 +790,15 @@ class Importer:
                         error=f"{pct:.2%} transparent",
                     )
 
+        # Artist anchored to the attribution path (folder→artist), resolved
+        # UP-FRONT so the enrich-on-duplicate branches link provenance with the
+        # right artist even when the sidecar carries none — which is now the norm
+        # for archive members under post-first (the per-media sidecar is minimal).
+        # Without this a cross-posted / re-packed archive image deduped and was
+        # left UNLINKED from the new post = a post showing "no images" (#718).
+        _artist_name = derive_top_level_artist(attribution_path, self.import_root)
+        path_artist = self._upsert_artist(_artist_name) if _artist_name else None
+
         # Hash dedup (exact).
         sha = _sha256_of(source)
         existing_stmt = select(ImageRecord).where(ImageRecord.sha256 == sha)
@@ -575,9 +808,9 @@ class Importer:
                 # Enrich-on-duplicate (parity with attach_in_place): a re-scanned
                 # file that is a byte-dup of an existing image still links its
                 # post via _apply_sidecar, so a cross-posted image shows on every
-                # post. Artist is derived from the sidecar here (not yet resolved).
+                # post.
                 self._apply_sidecar(
-                    existing, attribution_path, None,
+                    existing, attribution_path, path_artist,
                     explicit_source=explicit_source,
                 )
                 self.session.commit()
@@ -611,7 +844,7 @@ class Importer:
                     larger = self.session.get(ImageRecord, match_id)
                     if larger is not None:
                         self._apply_sidecar(
-                            larger, attribution_path, None,
+                            larger, attribution_path, path_artist,
                             explicit_source=explicit_source,
                         )
                         self.session.commit()
@@ -627,6 +860,32 @@ class Importer:
                     return ImportResult(
                         status="superseded", image_id=match_id
                     )
+        elif duration is not None:
+            # Tier-1 video near-dup (#871): same-artist, matching duration+aspect
+            # → same content across re-encodes. Mirror the image flow.
+            rel, match_id = self._find_similar_video(
+                duration, width, height,
+                path_artist.id if path_artist else None,
+            )
+            if rel == "larger_exists":
+                larger = self.session.get(ImageRecord, match_id)
+                if larger is not None:
+                    self._apply_sidecar(
+                        larger, attribution_path, path_artist,
+                        explicit_source=explicit_source,
+                    )
+                    self.session.commit()
+                return ImportResult(
+                    status="skipped", skip_reason=SkipReason.duplicate_phash,
+                    image_id=match_id,
+                    error="video near-duplicate (duration+aspect) of existing equal/larger video",
+                )
+            if rel == "smaller_exists":
+                target = self.session.get(ImageRecord, match_id)
+                self._supersede(
+                    target, source, sha, None, width, height, duration=duration,
+                )
+                return ImportResult(status="superseded", image_id=match_id)
 
         dest = self._copy_to_library(source, sha, attribution_path)
 
@@ -638,6 +897,7 @@ class Importer:
             mime=_mime_for(source),
             width=width,
             height=height,
+            duration_seconds=duration,
             origin="imported_filesystem",
             integrity_status="unknown",
         )
@@ -645,13 +905,11 @@ class Importer:
         self.session.flush()
         self._phash_cache_append(phash, width, height, record.id)
 
-        # Folder→artist (anchored to attribution_path).
-        artist = None
-        artist_name = derive_top_level_artist(
-            attribution_path, self.import_root
-        )
-        if artist_name:
-            artist = self._attach_artist(record, artist_name)
+        # Folder→artist (path_artist resolved up-front above); bind it to the
+        # new record so it carries the canonical Artist.
+        if path_artist is not None and record.artist_id is None:
+            record.artist_id = path_artist.id
+            self.session.flush()
 
         # Sidecar provenance (best-effort; never fails the import).
         # explicit_source lets the FC-3c download path bind the new
@@ -661,7 +919,7 @@ class Importer:
         # subscription-downloaded zip previously lost subscription
         # linkage if the on-disk layout didn't match assumptions.
         self._apply_sidecar(
-            record, attribution_path, artist, explicit_source=explicit_source,
+            record, attribution_path, path_artist, explicit_source=explicit_source,
         )
 
         # Thumbnail is queued separately by the calling task; the importer
@@ -710,6 +968,79 @@ class Importer:
         self.session.commit()
         return ImportResult(status="refreshed", image_id=existing.id)
 
+    def _apply_post_fields(self, post: Post, sd) -> None:
+        """Write a parsed sidecar's post-level fields onto a Post — the SINGLE
+        predicate shared by BOTH ingest paths: the per-media path (_apply_sidecar)
+        and the post-record path (upsert_post_record). Fill-with-non-empty:
+        parse_sidecar yields None for empty values, so a None field is left
+        untouched (an empty feed body never wipes a populated one). raw_metadata +
+        external-link sync always run (latest snapshot). Keeping ONE copy stops
+        the two paths from diverging on how a post body/links get stored — they
+        were verbatim duplicates (#842 DRY pass; see [[feedback_preview_apply_parity]]).
+        """
+        if sd.post_url is not None:
+            post.post_url = sd.post_url
+        if sd.post_title is not None:
+            post.post_title = sd.post_title
+        if sd.post_date is not None:
+            post.post_date = sd.post_date
+        if sd.description is not None:
+            post.description = sd.description
+        if sd.attachment_count is not None:
+            post.attachment_count = sd.attachment_count
+        post.raw_metadata = sd.raw
+        self._sync_external_links(post)
+
+    def upsert_post_record(
+        self, sidecar: Path, *, artist: Artist | None = None,
+        source: Source | None = None,
+    ) -> bool:
+        """Upsert the Post for a post-ONLY sidecar (a media-less post), so the
+        artist archive includes text posts (their body + external links).
+
+        Reuses `_find_or_create_post` (keyed on external_post_id) so it UPDATES
+        the SAME Post a media import would create — never doubles. Fields are
+        FILLED, never clobbered with empty: parse_sidecar yields None for empty
+        values, and a None field is left untouched (an empty feed body never
+        wipes a populated one). Returns True if a Post was upserted, False if the
+        sidecar was unusable (parse failure / no artist)."""
+        try:
+            data = json.loads(sidecar.read_text("utf-8"))
+            if not isinstance(data, dict):
+                raise ValueError("sidecar JSON is not an object")
+        except Exception as exc:
+            log.warning("post-record sidecar parse failed for %s: %s", sidecar, exc)
+            return False
+
+        sd = parse_sidecar(data)
+
+        if artist is None:
+            name = self._sidecar_artist_name(data)
+            artist = self._upsert_artist(name) if name else None
+        if artist is None:
+            log.warning("post-record sidecar %s has no artist; skipping", sidecar)
+            return False
+
+        if source is not None:
+            src = source
+        else:
+            platform = sd.platform or "unknown"
+            src = self._lookup_source_for_sidecar(
+                artist_id=artist.id, platform=platform,
+            )
+
+        epid = sd.external_post_id or sidecar.stem
+        post = self._find_or_create_post(
+            source_id=src.id if src else None,
+            external_post_id=epid,
+            artist_id=artist.id,
+        )
+        if post.artist_id is None:
+            post.artist_id = artist.id
+        self._apply_post_fields(post, sd)
+        self.session.commit()
+        return True
+
     def attach_in_place(
         self,
         path: Path,
@@ -760,6 +1091,7 @@ class Importer:
 
         # Format / dimension / transparency filters (mirror _import_media).
         width = height = None
+        duration = None  # video container duration (Tier-1 near-dup key, #871)
         has_alpha = False
         if not is_video(path):
             try:
@@ -791,6 +1123,14 @@ class Importer:
                         status="skipped", skip_reason=SkipReason.too_transparent,
                         error=f"{pct:.2%} transparent",
                     )
+        else:
+            # Best-effort probe for dims + duration so downloaded videos can dedup
+            # (#871). LENIENT: unlike _import_media this path does not reject on a
+            # probe failure — preserve the existing "import the downloaded video"
+            # behavior; we just skip dedup when we can't read the duration.
+            vp = safe_probe.probe_video(path)
+            if vp.ok:
+                width, height, duration = vp.width, vp.height, vp.duration
 
         # Hash dedup
         sha = _sha256_of(path)
@@ -849,6 +1189,30 @@ class Importer:
                         new_path=path, artist=artist, source_row=source,
                     )
                     return ImportResult(status="superseded", image_id=match_id)
+        elif duration is not None:
+            # Tier-1 video near-dup (#871): same content re-downloaded from another
+            # source/encode. artist is the subscription artist (passed explicitly).
+            rel, match_id = self._find_similar_video(
+                duration, width, height, artist.id if artist else None,
+            )
+            if rel == "larger_exists":
+                larger = self.session.get(ImageRecord, match_id)
+                if larger is not None:
+                    self._apply_sidecar(larger, path, artist, explicit_source=source)
+                    self.session.commit()
+                return ImportResult(
+                    status="skipped", skip_reason=SkipReason.duplicate_phash,
+                    image_id=match_id,
+                    error="video near-duplicate (duration+aspect) of existing equal/larger video",
+                )
+            if rel == "smaller_exists":
+                target = self.session.get(ImageRecord, match_id)
+                self._supersede(
+                    target, path, sha, None, width, height,
+                    new_path=path, artist=artist, source_row=source,
+                    duration=duration,
+                )
+                return ImportResult(status="superseded", image_id=match_id)
 
         # Create record — the path IS where the file lives.
         record = ImageRecord(
@@ -859,6 +1223,7 @@ class Importer:
             mime=_mime_for(path),
             width=width,
             height=height,
+            duration_seconds=duration,
             origin="downloaded",
             integrity_status="unknown",
         )
@@ -887,17 +1252,6 @@ class Importer:
             self.session.flush()
         return artist
 
-    def _attach_artist(self, record: ImageRecord, artist_name: str) -> Artist:
-        """Upsert the Artist and set it as the image's canonical artist.
-        Artist-kind tags were retired in FC-2d-vii-c — the Artist row is
-        the single source of truth. Returns the Artist (sidecar
-        provenance attaches its Source to it)."""
-        artist = self._upsert_artist(artist_name)
-        if record.artist_id is None:
-            record.artist_id = artist.id
-            self.session.flush()
-        return artist
-
     @staticmethod
     def _sidecar_artist_name(data: dict) -> str | None:
         for k in ("artist", "author_name"):
@@ -915,6 +1269,35 @@ class Importer:
                         return sv.strip()
         return None
 
+    def relink_source_filehash(
+        self, path: Path, source_url: str, *, artist: Artist | None = None,
+    ) -> bool:
+        """#830 recapture: backfill source_url + source_filehash on an EXISTING
+        on-disk image's ImageRecord (matched by sha256) so its post-body inline
+        `<img src=CDN>` remaps to the local copy at render time — WITHOUT
+        re-downloading and WITHOUT unlinking the file (unlike the import path's
+        duplicate-hash branch). NULL-only, mirroring _apply_sidecar: never
+        clobbers an already-set filehash. Returns True only when a row was
+        updated. No-op when the file is gone, the url has no filehash, no record
+        matches the bytes, or the record already carries a filehash.
+        """
+        fh = filehash_from_url(source_url)
+        if not fh:
+            return False
+        try:
+            sha = _sha256_of(path)
+        except OSError:
+            return False
+        record = self.session.execute(
+            select(ImageRecord).where(ImageRecord.sha256 == sha)
+        ).scalar_one_or_none()
+        if record is None or record.source_filehash is not None:
+            return False
+        record.source_url = source_url
+        record.source_filehash = fh
+        self.session.commit()
+        return True
+
     def _apply_sidecar(
         self,
         record: ImageRecord,
@@ -957,6 +1340,14 @@ class Importer:
         if record.artist_id is None:
             record.artist_id = artist.id
 
+        # #830 Phase 2: persist the file's CDN source identity (NULL-only, so a
+        # re-import / enrich-on-duplicate never clobbers it) — the filehash is
+        # the join key that lets post_feed_service remap the body's inline
+        # `<img src=CDN>` to this local copy at render time.
+        if sd.source_url and record.source_filehash is None:
+            record.source_url = sd.source_url
+            record.source_filehash = filehash_from_url(sd.source_url)
+
         if explicit_source is not None:
             src = explicit_source
         else:
@@ -971,17 +1362,14 @@ class Importer:
             external_post_id=epid,
             artist_id=artist.id,
         )
-        if sd.post_url is not None:
-            post.post_url = sd.post_url
-        if sd.post_title is not None:
-            post.post_title = sd.post_title
-        if sd.post_date is not None:
-            post.post_date = sd.post_date
-        if sd.description is not None:
-            post.description = sd.description
-        if sd.attachment_count is not None:
-            post.attachment_count = sd.attachment_count
-        post.raw_metadata = sd.raw
+        # Post-first (#856): on the native path the post-record owns the body/
+        # links/metadata, so the per-media import must NOT write post fields — its
+        # minimal sidecar carries no body, and applying it would clobber
+        # raw_metadata + re-run link-sync off empty data. The image still links
+        # provenance + localization below. gallery-dl (post_first False) is
+        # unchanged: its sidecar IS the only body source, so it still applies.
+        if not self.post_first:
+            self._apply_post_fields(post, sd)
 
         # Race-safe (image_record_id, post_id) upsert — mirrors the
         # _find_or_create_source/post savepoint pattern. The plain
@@ -1051,6 +1439,7 @@ class Importer:
         *, new_path: Path | None = None,
         artist: Artist | None = None,
         source_row: Source | None = None,
+        duration: float | None = None,
     ) -> None:
         """Replace `existing`'s file with the larger `source`, keeping the
         row id (so tags/series/curation stay attached). ML is cleared so
@@ -1083,9 +1472,9 @@ class Importer:
         existing.mime = _mime_for(source)
         existing.width = width
         existing.height = height
+        existing.duration_seconds = duration  # #871: keep the kept copy's duration
         existing.thumbnail_path = None
         existing.integrity_status = "unknown"
-        existing.tagger_predictions = None
         existing.tagger_model_version = None
         existing.siglip_embedding = None
         existing.siglip_model_version = None

backend/app/services/ingest_core.py

@@ -36,6 +36,7 @@ from sqlalchemy import delete, func, select, text
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 
 from .gallery_dl import DownloadResult, ErrorType, make_run_stats
+from .native_ingest_common import NativeAuthError, NativeDriftError
 
 log = logging.getLogger(__name__)
 
@@ -57,6 +58,19 @@ _ERROR_MAX = 1000
 # page-tied update would lurch). Trivial churn (~one single-row UPDATE / 5s).
 _LIVE_PROGRESS_INTERVAL = 5.0
 
+# Post-body schema-drift canary (#862). Patreon's body lives in
+# content/content_json_string with NO post_type gate, so a field rename (as
+# content→content_json_string already was) zeroes EVERY body at once — across
+# every artist, every walk. If a native walk records at least this many posts
+# and extracts a body from NONE of them, treat it as that break (fail the run
+# API_DRIFT) rather than silently archiving empties. A *fraction* threshold would
+# false-positive on gallery/art creators who legitimately post images with no
+# caption, so the gate is "zero across a minimum sample": a real creator nearly
+# always has SOME text across this many posts, a broken parser has none. Set high
+# enough that a small tick (a few new posts) can't trip it — only a backfill /
+# recapture (the operator's schema-test flow) reaches the sample.
+_CANARY_MIN_SAMPLE = 30
+
 
 class Ingester:
     """Generic native-ingest orchestration. Subclass with a platform adapter
@@ -75,6 +89,7 @@ class Ingester:
         ledger_key: Callable[[object], str],
         platform: str,
         error_base: type[Exception],
+        drift_label: str | None = None,
     ):
         self.client = client
         self.downloader = downloader
@@ -86,6 +101,10 @@ class Ingester:
         self._ledger_key = ledger_key
         self._platform = platform
         self._error_base = error_base
+        # Human label for the API_DRIFT message ("<label> changed — ingester needs
+        # update"). Defaults to the platform name; adapters pass a richer phrase
+        # (e.g. "Patreon API", "SubscribeStar markup").
+        self._drift_label = drift_label or platform
 
     # -- public ------------------------------------------------------------
 
@@ -105,30 +124,73 @@ class Ingester:
     ) -> DownloadResult:
         """Walk + download for one source, returning a gallery-dl-shaped result.
 
-        `mode` is "tick" | "backfill" | "recovery". Recovery bypasses the tier-1
-        seen-ledger AND the dead-letter ledger (tier-2 disk still skips kept
-        files). The walk stops on:
+        `mode` is "tick" | "backfill" | "recovery" | "recapture". Recovery
+        bypasses the tier-1 seen-ledger AND the dead-letter ledger (tier-2 disk
+        still skips kept files). Recapture (#830) is the cheap "re-grab post
+        text" walk: it bypasses the post-record gate (so EVERY post's body +
+        external links are re-captured / detail-fetched) but KEEPS the media
+        seen-ledger — on-disk media is NOT re-downloaded, only surfaced so its
+        ImageRecord's source_filehash can be backfilled for inline-image
+        localization. The walk stops on:
           - budget exhaustion (time_budget_seconds)         → TIMEOUT / PARTIAL
           - tick early-out (seen_threshold contiguous seen)  → success
           - reaching the bottom of the feed                  → success (rc 0)
         A client-level failure (drift / auth / network) fails the whole run loud.
         """
         bypass_seen = mode == "recovery"
+        recapture = mode == "recapture"
+        # Both recovery and recapture re-capture EVERY post's body + links — they
+        # bypass the post-record seen-gate (recovery via bypass_seen, recapture
+        # explicitly). A plain backfill stays gated (capture once per post).
+        recapture_records = bypass_seen or recapture
         # Only deep walks checkpoint their cursor mid-flight (plan #705 #6); a
         # tick has no resumable backfill state.
-        checkpoint = mode in ("backfill", "recovery")
+        checkpoint = mode in ("backfill", "recovery", "recapture")
         ledger_key = self._ledger_key
+        # POST-FIRST CONTRACT (milestone #67): these two optional seams make a
+        # platform "post-first" on the native core ingester — the post-record is
+        # the single authoritative writer of the post body/links/metadata, and the
+        # per-media sidecar carries image identity only (download_service flips
+        # importer.post_first via uses_native_ingester, so the import side follows
+        # automatically). A platform migrating off gallery-dl onto the native core
+        # adopts post-first by implementing BOTH:
+        #   client.post_record_key(post)   -> (ledger_key, post_id) | None  (gate)
+        #   downloader.write_post_record(post, artist_slug) -> PostRecordOutcome
+        # Absent on stub clients/downloaders (unit tests) and on not-yet-migrated
+        # platforms → media-less posts are skipped as before and the body still
+        # comes from the per-media sidecar (gallery-dl path). See [[post-first-ingest-contract]].
+        post_record_key = getattr(self.client, "post_record_key", None)
+        write_post_record = getattr(self.downloader, "write_post_record", None)
+        # #874: optional client seam — skip tier-gated posts (the account can't
+        # view them, so Patreon serves only blurred locked-preview media) ENTIRELY:
+        # no media download, no post-record stub. Absent on stub/not-yet-migrated
+        # clients → nothing is ever treated as gated.
+        post_is_gated = getattr(self.client, "post_is_gated", None)
         start = time.monotonic()
         last_live = start  # plan #709: last live-progress write timestamp
         log_lines: list[str] = []
         written: list[str] = []
+        post_records: list[str] = []
         quarantined_paths: list[str] = []
+        # #830 recapture: (on-disk path, CDN source_url) pairs for already-present
+        # media, so phase 3 can backfill the ImageRecord's source_filehash WITHOUT
+        # re-downloading or unlinking the file. Empty outside recapture mode.
+        relink: list[tuple[str, str]] = []
         downloaded = 0
         errors = 0
         quarantined = 0
         dead_lettered = 0
         skipped_count = 0
         posts_processed = 0
+        # Post-body schema-drift canary counters (#862, native ingester only —
+        # gallery-dl walks never enter the post-record block below so these stay 0
+        # and the canary can't fire there). posts_recorded = post-records attempted
+        # this walk; posts_with_body = how many yielded a non-empty body.
+        posts_recorded = 0
+        posts_with_body = 0
+        # Tier-gated posts skipped entirely this walk (#874) — surfaced in the
+        # run summary for diagnostics ("a lot of these now").
+        gated_skipped = 0
         # Net-new posts THIS chunk for the live progress badge (plan #704 #5);
         # excludes the re-walked resume page so _backfill_posts stays a monotonic
         # absolute across chunks instead of an inflating sum. posts_processed
@@ -158,6 +220,8 @@ class Ingester:
                 files_quarantined=quarantined,
                 quarantined_paths=list(quarantined_paths),
                 written_paths=written,
+                post_record_paths=list(post_records),
+                relink_source_paths=list(relink),
                 stdout="\n".join(log_lines),
                 stderr="",
                 return_code=return_code,
@@ -176,6 +240,15 @@ class Ingester:
                 ),
             )
 
+        # #899 L1: emit run milestones through the real logger (not only the
+        # in-memory log_lines → DownloadResult.stdout, which is persisted to the
+        # DownloadEvent ONLY at phase 3). A worker SIGKILL/OOM/hard-time-limit
+        # mid-walk would otherwise leave NO trace; these land in the container log
+        # in real time regardless of whether the event gets finalized.
+        log.info(
+            "%s ingest START (%s): source=%s campaign=%s resume_cursor=%s",
+            self._platform, mode, source_id, campaign_id, resume_cursor,
+        )
         try:
             for post, included, page_cursor in self.client.iter_posts(
                 campaign_id, cursor=resume_cursor
@@ -185,6 +258,15 @@ class Ingester:
                 # after it. Carried as DownloadResult.cursor (plan #704).
                 if page_cursor and page_cursor != emitted_cursor:
                     emitted_cursor = page_cursor
+                    # #899 L1: a per-page breadcrumb in the container log (pages can
+                    # be minutes apart on image-dense backfills) — survives a worker
+                    # kill so the operator sees how far a since-died walk got.
+                    log.info(
+                        "%s ingest progress (%s, source=%s): posts=%d downloaded=%d "
+                        "skipped=%d errors=%d quarantined=%d gated=%d cursor=%s",
+                        self._platform, mode, source_id, posts_processed, downloaded,
+                        skipped_count, errors, quarantined, gated_skipped, emitted_cursor,
+                    )
                     # plan #705 #6: persist the cursor at each page boundary so a
                     # worker SIGKILL mid-chunk resumes near the crash, not the
                     # chunk start. (phase 3 still writes the final cursor — same
@@ -220,6 +302,56 @@ class Ingester:
                 # resume_cursor None, so everything counts.
                 if not (resume_cursor and page_cursor == resume_cursor):
                     chunk_new_posts += 1
+                # Tier-gated post (#874): the account can't fully view it, so
+                # Patreon serves only blurred locked-preview media. Skip it
+                # ENTIRELY — no media download AND no post-record stub (operator
+                # decision: gated content leaves no trace; a later walk re-ingests
+                # it for real once access is gained). Skipped BEFORE the
+                # post-record block so gated posts never inflate the #862 body
+                # canary's sample. post_is_gated gates only on an explicit
+                # current_user_can_view=False (missing/None → viewable).
+                if post_is_gated and post_is_gated(post):
+                    gated_skipped += 1
+                    log_lines.append(
+                        f"  post {post.get('id')} — gated (skipped, no access)"
+                    )
+                    continue
+                # Capture the post body + external links ONCE per post (gated by
+                # the synthetic post key in the seen-ledger), for EVERY post —
+                # whether or not it has downloadable media. This is what makes a
+                # backfill/recovery re-walk RECAPTURE bodies + links for posts
+                # whose media is already on disk: re-downloading existing media
+                # never fills links the system never had, so the body recapture
+                # has to ride the walk itself. Detail-fetch (for an empty feed
+                # body) happens at most once per post — the gate then spares it on
+                # later walks. bypass_seen (recovery) re-captures unconditionally.
+                if post_record_key and write_post_record:
+                    rk = post_record_key(post)
+                    if rk is not None:
+                        pkey, ppid = rk
+                        already = (
+                            set() if recapture_records
+                            else self._seen_keys(source_id, [pkey])
+                        )
+                        if pkey not in already:
+                            rec = write_post_record(post, artist_slug)
+                            posts_recorded += 1
+                            if rec.body_chars:
+                                posts_with_body += 1
+                            if rec.path is not None:
+                                post_records.append(str(rec.path))
+                                self._mark_seen(source_id, [(pkey, ppid)])
+                            # Per-post handling line in the run stdout (the existing
+                            # "Raw stdout" panel) — the downloader already read the
+                            # post; we only format its outcome here. post_type beside
+                            # a 0-char body is the "why is this one empty" answer.
+                            log_lines.append(
+                                f"  post {ppid} [{rec.post_type or '?'}] "
+                                f"body: {rec.body_chars} chars"
+                                + ("" if rec.body_chars else " — EMPTY")
+                                + (f" — {rec.title}" if rec.title else "")
+                            )
+
                 media = self.client.extract_media(post, included)
                 if not media:
                     continue
@@ -245,6 +377,7 @@ class Ingester:
                 outcomes = self.downloader.download_post(
                     post, media, artist_slug, is_seen=_is_skip,
                     should_stop=lambda: time.monotonic() - start >= time_budget_seconds,
+                    recapture=recapture,
                 )
 
                 to_mark: list[tuple[str, str]] = []
@@ -270,6 +403,12 @@ class Ingester:
                         to_clear.append(key)
                         skipped_count += 1
                         consecutive_seen += 1
+                        # #830 recapture: surface (on-disk path, CDN url) so phase
+                        # 3 can backfill source_filehash for inline-image
+                        # localization — a SEPARATE non-deleting channel, never the
+                        # import list (which would unlink the file, per above).
+                        if recapture and outcome.path is not None:
+                            relink.append((str(outcome.path), media_item.url))
                     elif outcome.status == "skipped_seen":
                         skipped_count += 1
                         consecutive_seen += 1
@@ -329,6 +468,10 @@ class Ingester:
         # posts), so don't re-write them — return PARTIAL (reads as "ok/progress",
         # the lifecycle no-ops since state is gone) instead of a false "complete".
         if stopped:
+            log.info(
+                "%s ingest STOPPED by operator (%s, source=%s): %d file(s) this chunk",
+                self._platform, mode, source_id, downloaded,
+            )
             return _result(
                 success=False, return_code=-1,
                 error_type=ErrorType.PARTIAL,
@@ -346,14 +489,23 @@ class Ingester:
             log_lines.append(f"{quarantined} media item(s) quarantined (invalid)")
         if dead_lettered:
             log_lines.append(f"{dead_lettered} media item(s) skipped (dead-lettered)")
-        log_lines.append(
+        summary = (
             f"{self._platform} ingest ({mode}): {downloaded} downloaded, "
             f"{skipped_count} skipped, {quarantined} quarantined, "
             f"{dead_lettered} dead-lettered, {errors} error(s), "
-            f"{posts_processed} post(s)"
+            f"{posts_processed} post(s), {len(post_records)} post-record(s), "
+            f"{len(relink)} relinked"
+            # Body-capture health (#862): even below the canary's red-alarm
+            # threshold, surfacing the ratio makes a partial extraction regression
+            # visible in the Raw stdout (e.g. "bodies 3/180" reads as off).
+            + (f", bodies {posts_with_body}/{posts_recorded}" if posts_recorded else "")
+            + (f", {gated_skipped} gated-skipped" if gated_skipped else "")
             + (", reached end" if reached_bottom else "")
             + (", time-boxed" if budget_hit else "")
         )
+        log_lines.append(summary)
+        # #899 L1: also to the container log (survives event-finalization failure).
+        log.info("%s (source=%s)", summary, source_id)
 
         if budget_hit:
             # A chunk that hit its time-box but made forward progress is a
@@ -376,6 +528,27 @@ class Ingester:
                 error_message="Chunk timed out with no progress",
             )
 
+        # Post-body schema-drift canary (#862): a native walk recorded a
+        # meaningful sample of posts but extracted a body from NONE of them. Since
+        # the body field has no post_type gate, that's the signature of Patreon
+        # renaming/restructuring the body field (as content→content_json_string
+        # already was) — fail RED (API_DRIFT: "fix is the field-set/parser, not
+        # creds") so the breakage screams instead of silently archiving empties.
+        # Only reached on an otherwise-clean walk (timeout/stop/error returned
+        # above), so it never masks a more specific failure.
+        if posts_recorded >= _CANARY_MIN_SAMPLE and posts_with_body == 0:
+            msg = (
+                f"Post-body canary: extracted a body from 0 of {posts_recorded} "
+                "posts — Patreon's body field shape likely changed; the ingester "
+                "needs a field-set/parser update."
+            )
+            log_lines.append(msg)
+            log.error("%s (artist=%s)", msg, artist_slug)
+            return _result(
+                success=False, return_code=-1,
+                error_type=ErrorType.API_DRIFT, error_message=msg,
+            )
+
         # Normal success: reached the bottom, or a tick that early-outed. rc 0 +
         # error_type None is REQUIRED for a backfill/recovery walk that reached
         # the bottom to be marked COMPLETE by
@@ -415,6 +588,10 @@ class Ingester:
         sample: list[dict] = []
         unset = object()
         last_page: object = unset
+        # #874: same gated-post gate as run() — the preview must not count
+        # blurred locked-preview media as "new", or it would overstate a gated
+        # source's backlog (preview/apply parity, rule 93).
+        post_is_gated = getattr(self.client, "post_is_gated", None)
         for post, included, page_cursor in self.client.iter_posts(
             campaign_id, cursor=None
         ):
@@ -426,6 +603,8 @@ class Ingester:
                     pages_scanned = page_limit
                     break
             posts_scanned += 1
+            if post_is_gated and post_is_gated(post):
+                continue
             media = self.client.extract_media(post, included)
             if not media:
                 continue
@@ -453,13 +632,41 @@ class Ingester:
     # -- failure mapping (adapter overrides) -------------------------------
 
     def _failure_result(self, exc: Exception, _result) -> DownloadResult:
-        """Map a platform client-error to a typed failed DownloadResult. The base
-        gives a safe default; adapters override with their exception taxonomy."""
-        log.warning("%s ingest failed: %s", self._platform, exc)
-        return _result(
+        """Map a platform client-error to a loud, typed failed DownloadResult —
+        NEVER a silent zero-download "success". The mapping is shared across
+        platforms via the NativeAuthError/NativeDriftError taxonomy (the platform
+        client raises subclasses), so a new platform gets it for free:
+          - NativeAuthError  → AUTH_ERROR   (rotate the credential)
+          - NativeDriftError → API_DRIFT    (the ingester/scraper needs updating)
+          - HTTP 429 / 404   → RATE_LIMITED / NOT_FOUND
+          - other HTTP status→ HTTP_ERROR;  transport failure → NETWORK_ERROR
+        Auth/Drift are matched first (they also carry a status_code in some paths).
+        """
+        message = str(exc)
+        if isinstance(exc, NativeAuthError):
+            error_type = ErrorType.AUTH_ERROR
+        elif isinstance(exc, NativeDriftError):
+            error_type = ErrorType.API_DRIFT
+            message = f"{self._drift_label} changed — ingester needs update: {message}"
+        else:
+            status = getattr(exc, "status_code", None)
+            if status == 429:
+                error_type = ErrorType.RATE_LIMITED
+            elif status == 404:
+                error_type = ErrorType.NOT_FOUND
+            elif status is not None:
+                error_type = ErrorType.HTTP_ERROR
+            else:
+                error_type = ErrorType.NETWORK_ERROR
+        log.warning("%s ingest failed (%s): %s", self._platform, error_type.value, message)
+        result = _result(
             success=False, return_code=1,
-            error_type=ErrorType.UNKNOWN_ERROR, error_message=str(exc),
+            error_type=error_type, error_message=message,
         )
+        # plan #708 B1: carry the server's Retry-After up to the cooldown.
+        if error_type == ErrorType.RATE_LIMITED:
+            result.retry_after_seconds = getattr(exc, "retry_after", None)
+        return result
 
     # -- seen-ledger (short-lived sessions) --------------------------------
 

backend/app/services/link_extract.py

@@ -0,0 +1,105 @@
+"""Extract off-platform file-host links from a post body.
+
+Pure (no I/O) so it's unit-testable and reusable by EVERY in-house downloader's
+import path — every platform stores its body in `Post.description`, so running
+this there covers them all. Finds links to the supported external file hosts in
+a post's HTML body (both `<a href="...">` anchors and bare URLs in text),
+unwraps a Patreon outbound-redirect wrapper, and preserves the FULL url
+including the `#fragment` (mega.nz puts the decryption key there) and the query
+string — without those a mega download is impossible.
+"""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from html import unescape
+from urllib.parse import parse_qs, urlsplit
+
+# Supported file-host enum values (kept in sync with the external_link CHECK).
+SUPPORTED_HOSTS = ("mega", "gdrive", "mediafire", "dropbox", "pixeldrain")
+
+# Bare-domain suffix → canonical host label. Matched against the URL netloc
+# (exact or as a dotted suffix, so www./dl. subdomains resolve too).
+_HOST_MAP = {
+    "mega.nz": "mega",
+    "mega.co.nz": "mega",
+    "drive.google.com": "gdrive",
+    "mediafire.com": "mediafire",
+    "dropbox.com": "dropbox",
+    "pixeldrain.com": "pixeldrain",
+}
+
+# `<a href="...">label</a>` — DOTALL so a label spanning tags/newlines is caught.
+_HREF_RE = re.compile(
+    r"""<a\b[^>]*?\bhref=["']([^"']+)["'][^>]*>(.*?)</a>""",
+    re.IGNORECASE | re.DOTALL,
+)
+_TAG_RE = re.compile(r"<[^>]+>")
+# Bare http(s) URL in text. Stops at whitespace, quotes, angle brackets, and
+# closing brackets — but KEEPS `#`, `&`, `?`, `=` so fragments/queries survive.
+_URL_RE = re.compile(r"""https?://[^\s"'<>)\]}]+""", re.IGNORECASE)
+
+
+@dataclass(frozen=True)
+class ExtractedLink:
+    host: str            # one of SUPPORTED_HOSTS
+    url: str             # full url incl. query + #fragment
+    label: str | None    # visible anchor text, when present
+
+
+def _netloc(url: str) -> str:
+    # Lowercase host without credentials or port.
+    return urlsplit(url).netloc.lower().split("@")[-1].split(":")[0]
+
+
+def host_for(url: str) -> str | None:
+    """Canonical host label for a url, or None if it's not a supported host."""
+    netloc = _netloc(url)
+    for suffix, host in _HOST_MAP.items():
+        if netloc == suffix or netloc.endswith("." + suffix):
+            return host
+    return None
+
+
+def _unwrap(url: str) -> str:
+    """Unwrap a Patreon outbound-redirect wrapper to its inner target, so a
+    wrapped mega/gdrive link resolves to the real host. Patreon has used both
+    `www.patreon.com/...?url=<encoded>` and the `l.patreon.com` shim; check the
+    common target-param names. Non-Patreon urls pass through untouched."""
+    netloc = _netloc(url)
+    if not (netloc == "patreon.com" or netloc.endswith(".patreon.com")):
+        return url
+    qs = parse_qs(urlsplit(url).query)
+    for key in ("url", "u", "ext_url", "redirect", "target"):
+        vals = qs.get(key)
+        if vals and vals[0]:
+            return unescape(vals[0]).strip()
+    return url
+
+
+def extract_external_links(html: str | None) -> list[ExtractedLink]:
+    """All supported-host links in `html`, de-duplicated by url (first wins, so
+    an anchor's label is kept over a later bare sighting of the same url)."""
+    if not html:
+        return []
+    found: dict[str, ExtractedLink] = {}
+
+    # 1) Anchors first — they carry a human label ("Mega - Streamable").
+    for raw_href, inner in _HREF_RE.findall(html):
+        url = _unwrap(unescape(raw_href).strip())
+        host = host_for(url)
+        if host is None:
+            continue
+        label = unescape(_TAG_RE.sub("", inner)).strip() or None
+        found.setdefault(url, ExtractedLink(host=host, url=url, label=label))
+
+    # 2) Bare URLs pasted in text (no label). Trailing prose punctuation is
+    #    trimmed; the href values already captured above de-dup away here.
+    for raw in _URL_RE.findall(html):
+        url = _unwrap(unescape(raw).strip().rstrip(".,;"))
+        host = host_for(url)
+        if host is None:
+            continue
+        found.setdefault(url, ExtractedLink(host=host, url=url, label=None))
+
+    return list(found.values())

backend/app/services/ml/aliases.py

@@ -1,7 +1,7 @@
 """Alias resolution + CRUD.
 
 A tag_alias maps (model_name, model_category) -> canonical Tag. Resolution
-happens at suggestion-read time so raw tagger_predictions stay unmolested.
+happens at suggestion-read time so the raw image_prediction rows stay unmolested.
 """
 
 from collections.abc import Sequence
@@ -81,6 +81,31 @@ class AliasService:
             .where(TagAlias.alias_category == alias_category)
         )
 
+    async def list_for_tag(self, canonical_tag_id: int) -> Sequence[AliasRow]:
+        """Aliases that resolve TO this tag — drives the tag-side 'Aliases'
+        view (see/remove the model keys that fold into a tag)."""
+        stmt = (
+            select(
+                TagAlias.alias_string,
+                TagAlias.alias_category,
+                TagAlias.canonical_tag_id,
+                Tag.name,
+            )
+            .join(Tag, Tag.id == TagAlias.canonical_tag_id)
+            .where(TagAlias.canonical_tag_id == canonical_tag_id)
+            .order_by(TagAlias.alias_string.asc())
+        )
+        rows = (await self.session.execute(stmt)).all()
+        return [
+            AliasRow(
+                alias_string=r[0],
+                alias_category=r[1],
+                canonical_tag_id=r[2],
+                canonical_tag_name=r[3],
+            )
+            for r in rows
+        ]
+
     async def list_all(self) -> Sequence[AliasRow]:
         stmt = (
             select(

backend/app/services/ml/allowlist.py

@@ -5,11 +5,18 @@ image_tag AND to tag_allowlist; per-image removal/dismiss writes a rejection.
 from collections.abc import Sequence
 from dataclasses import dataclass
 
-from sqlalchemy import delete, select
+from sqlalchemy import and_, delete, distinct, func, or_, select
 from sqlalchemy.dialects.postgresql import insert
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from ...models import MLSettings, Tag, TagAllowlist, TagSuggestionRejection
+from ...models import (
+    ImagePrediction,
+    MLSettings,
+    Tag,
+    TagAlias,
+    TagAllowlist,
+    TagSuggestionRejection,
+)
 from ...models.tag import image_tag
 from .aliases import AliasService
 
@@ -20,6 +27,8 @@ class AllowlistRow:
     tag_name: str
     tag_kind: str
     min_confidence: float
+    applied_count: int      # image_tag rows currently carrying this tag
+    coverage_count: int     # images a sweep WOULD cover at min_confidence
 
 
 class AllowlistService:
@@ -80,6 +89,12 @@ class AllowlistService:
         )
         await self.session.execute(stmt)
 
+    async def undismiss(self, image_id: int, tag_id: int) -> None:
+        """Undo a per-image dismissal — drop the TagSuggestionRejection so the
+        suggestion reverts to a live (un-rejected) state. Backs the rail's
+        one-click reject-recovery (operator-asked 2026-06-27)."""
+        await self._clear_rejection(image_id, tag_id)
+
     async def reject_applied_tag(self, image_id: int, tag_id: int) -> None:
         """Operator removed an applied tag from an image. Remove the
         image_tag row AND record a rejection so the allowlist won't
@@ -116,6 +131,44 @@ class AllowlistService:
             delete(TagAllowlist).where(TagAllowlist.tag_id == tag_id)
         )
 
+    async def _coverage_match(self, tag: Tag):
+        """The predicate over image_prediction rows that resolve to `tag`,
+        mirroring tasks.ml._confidence_for_tag's resolution: a prediction whose
+        raw_name equals the tag name (any category), OR an alias maps
+        (raw_name, category) -> this tag. Returns a SQLAlchemy boolean clause.
+        """
+        alias_rows = (
+            await self.session.execute(
+                select(TagAlias.alias_string, TagAlias.alias_category).where(
+                    TagAlias.canonical_tag_id == tag.id
+                )
+            )
+        ).all()
+        name_clause = ImagePrediction.raw_name == tag.name
+        alias_clauses = [
+            and_(
+                ImagePrediction.raw_name == a,
+                ImagePrediction.category == c,
+            )
+            for a, c in alias_rows
+        ]
+        return or_(name_clause, *alias_clauses) if alias_clauses else name_clause
+
+    async def coverage(self, tag_id: int, threshold: float) -> int:
+        """How many distinct images a sweep WOULD cover for this tag at
+        `threshold`: images with a resolving prediction scoring >= threshold.
+        The gross candidate pool (NOT minus already-applied/rejected) — it's
+        the tuning signal for "lower the threshold and ~N more images qualify".
+        """
+        tag = await self.session.get(Tag, tag_id)
+        if tag is None:
+            return 0
+        match = await self._coverage_match(tag)
+        stmt = select(
+            func.count(distinct(ImagePrediction.image_record_id))
+        ).where(ImagePrediction.score >= threshold, match)
+        return (await self.session.execute(stmt)).scalar_one()
+
     async def list_all(self) -> Sequence[AllowlistRow]:
         stmt = (
             select(
@@ -128,12 +181,33 @@ class AllowlistService:
             .order_by(Tag.name.asc())
         )
         rows = (await self.session.execute(stmt)).all()
-        return [
-            AllowlistRow(
-                tag_id=r[0],
-                tag_name=r[1],
-                tag_kind=r[2].value if hasattr(r[2], "value") else str(r[2]),
-                min_confidence=r[3],
+        tag_ids = [r[0] for r in rows]
+
+        # Applied counts in ONE grouped query (vs N per-row counts).
+        applied: dict[int, int] = {}
+        if tag_ids:
+            applied = dict(
+                (
+                    await self.session.execute(
+                        select(image_tag.c.tag_id, func.count())
+                        .where(image_tag.c.tag_id.in_(tag_ids))
+                        .group_by(image_tag.c.tag_id)
+                    )
+                ).all()
             )
-            for r in rows
-        ]
+
+        result = []
+        for r in rows:
+            # Coverage is per-tag (alias set differs); allowlist is small.
+            cov = await self.coverage(r[0], r[3])
+            result.append(
+                AllowlistRow(
+                    tag_id=r[0],
+                    tag_name=r[1],
+                    tag_kind=r[2].value if hasattr(r[2], "value") else str(r[2]),
+                    min_confidence=r[3],
+                    applied_count=applied.get(r[0], 0),
+                    coverage_count=cov,
+                )
+            )
+        return result

backend/app/services/ml/ccip.py

@@ -0,0 +1,180 @@
+"""CCIP few-shot character matcher (#114) — server-side, numpy on stored vectors.
+
+CCIP is a FROZEN identity embedding; we don't train it. Instead the operator's
+tagged characters become reference prototypes: a character tag's references are
+the CCIP vectors of figure/face regions on images carrying that tag. To suggest
+characters for a new image, we compare its figure-region CCIP vectors to every
+character's references (multi-prototype: best match over a character's examples)
+and surface the ones that clear a similarity threshold. No GPU here — the agent
+already produced the vectors; this is cosine matching on what's stored.
+
+v1 uses cosine similarity on the raw CCIP vectors with a tunable threshold; the
+exact CCIP difference metric/threshold gets validated against the model during
+the hands-on eval. numpy is imported lazily (API worker has it via pgvector).
+"""
+
+from sqlalchemy import func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from ...models import ImageRegion, MLSettings, Tag, TagKind
+from ...models.tag import image_tag
+
+# Cosine-similarity floor to call a figure the same character. The live setting
+# (ml_settings.ccip_match_threshold) drives it; this is only the fallback when no
+# threshold is supplied AND no settings row exists.
+DEFAULT_SIM_THRESHOLD = 0.85
+_FIGURE_KINDS = ("face", "figure")
+
+
+async def _settings_threshold(session: AsyncSession) -> float:
+    val = (
+        await session.execute(
+            select(MLSettings.ccip_match_threshold).where(MLSettings.id == 1)
+        )
+    ).scalar_one_or_none()
+    return float(val) if val is not None else DEFAULT_SIM_THRESHOLD
+
+
+def _l2norm(mat, np):
+    n = np.linalg.norm(mat, axis=1, keepdims=True)
+    n[n == 0] = 1.0
+    return mat / n
+
+
+# Single-shot cache of the (expensive) reference load, keyed on a cheap
+# signature that changes exactly when references could: a character tag added/
+# removed (n_char_tags) or a figure embedded (max/ n of ccip regions). Shared by
+# the live matcher (every modal open) and the auto-apply sweep.
+_REF_CACHE: dict = {"sig": None, "refs": None}
+
+
+def _single_character_images():
+    """Subquery of image ids carrying EXACTLY ONE character tag. References come
+    only from these — on a multi-character image the tag is image-level, so every
+    figure would otherwise pollute each character's prototype set (a 2-character
+    image tagged 'Velma' would make Daphne's figure a Velma reference)."""
+    return (
+        select(image_tag.c.image_record_id)
+        .join(Tag, Tag.id == image_tag.c.tag_id)
+        .where(Tag.kind == TagKind.character)
+        .group_by(image_tag.c.image_record_id)
+        .having(func.count() == 1)
+    )
+
+
+async def _ref_signature(session: AsyncSession) -> tuple:
+    n_tags = (
+        await session.execute(
+            select(func.count())
+            .select_from(image_tag)
+            .join(Tag, Tag.id == image_tag.c.tag_id)
+            .where(Tag.kind == TagKind.character)
+        )
+    ).scalar_one()
+    n_regs, max_id = (
+        await session.execute(
+            select(func.count(), func.max(ImageRegion.id)).where(
+                ImageRegion.kind.in_(_FIGURE_KINDS),
+                ImageRegion.ccip_embedding.is_not(None),
+            )
+        )
+    ).one()
+    return (n_tags, n_regs, max_id)
+
+
+async def character_references(session: AsyncSession) -> dict[int, list]:
+    """Per character-tag CCIP reference vectors: figure/face-region CCIP
+    embeddings on UNAMBIGUOUS (single-character) images carrying that tag.
+    Multi-prototype — several vectors per character. Cached on a cheap signature."""
+    sig = await _ref_signature(session)
+    if _REF_CACHE["sig"] == sig and _REF_CACHE["refs"] is not None:
+        return _REF_CACHE["refs"]
+    rows = (
+        await session.execute(
+            select(image_tag.c.tag_id, ImageRegion.ccip_embedding)
+            .select_from(ImageRegion)
+            .join(
+                image_tag,
+                image_tag.c.image_record_id == ImageRegion.image_record_id,
+            )
+            .join(Tag, Tag.id == image_tag.c.tag_id)
+            .where(Tag.kind == TagKind.character)
+            .where(ImageRegion.kind.in_(_FIGURE_KINDS))
+            .where(ImageRegion.ccip_embedding.is_not(None))
+            .where(ImageRegion.image_record_id.in_(_single_character_images()))
+        )
+    ).all()
+    refs: dict[int, list] = {}
+    for tag_id, vec in rows:
+        refs.setdefault(tag_id, []).append(vec)
+    _REF_CACHE.update(sig=sig, refs=refs)
+    return refs
+
+
+async def _tag_names(session: AsyncSession, tag_ids: list[int]) -> dict[int, str]:
+    if not tag_ids:
+        return {}
+    return dict(
+        (
+            await session.execute(
+                select(Tag.id, Tag.name).where(Tag.id.in_(tag_ids))
+            )
+        ).all()
+    )
+
+
+async def match_image(
+    session: AsyncSession, image_id: int, threshold: float | None = None
+) -> list[dict]:
+    """Character suggestions for one image from its figure-region CCIP vectors:
+    [{tag_id, name, category:'character', score, source:'ccip'}], ranked.
+    Already-applied character tags are excluded. Empty if the image has no figure
+    CCIP vectors or no character references exist yet. threshold defaults to the
+    live ml_settings.ccip_match_threshold."""
+    import numpy as np
+
+    if threshold is None:
+        threshold = await _settings_threshold(session)
+
+    qvecs = (
+        await session.execute(
+            select(ImageRegion.ccip_embedding).where(
+                ImageRegion.image_record_id == image_id,
+                ImageRegion.kind.in_(_FIGURE_KINDS),
+                ImageRegion.ccip_embedding.is_not(None),
+            )
+        )
+    ).scalars().all()
+    if not qvecs:
+        return []
+    refs = await character_references(session)
+    if not refs:
+        return []
+    applied = set(
+        (
+            await session.execute(
+                select(image_tag.c.tag_id).where(
+                    image_tag.c.image_record_id == image_id
+                )
+            )
+        ).scalars()
+    )
+    names = await _tag_names(session, [t for t in refs if t not in applied])
+
+    Q = _l2norm(np.vstack([np.asarray(v, dtype=np.float32) for v in qvecs]), np)
+    out = []
+    for tag_id, vecs in refs.items():
+        if tag_id in applied:
+            continue
+        R = _l2norm(np.vstack([np.asarray(v, dtype=np.float32) for v in vecs]), np)
+        best = float((Q @ R.T).max())   # best (query figure, reference) cosine
+        if best >= threshold:
+            out.append({
+                "tag_id": tag_id,
+                "name": names.get(tag_id, str(tag_id)),
+                "category": "character",
+                "score": round(best, 4),
+                "source": "ccip",
+            })
+    out.sort(key=lambda d: d["score"], reverse=True)
+    return out

backend/app/services/ml/crops.py

@@ -0,0 +1,73 @@
+"""Shared crop primitive for the region/crop pipeline (#114).
+
+One model- and transport-agnostic function sits at the trunk of both crop jobs:
+- CCIP characters: a face/figure detector proposes regions → crop → CCIP-embed.
+- SigLIP concepts: head-guided / saliency proposes regions → crop → SigLIP-embed.
+Only the PROPOSER (where to crop) and the EMBEDDER (what to run) differ; the crop
+itself — including the lower-bound size floor below which a region is too small to
+embed reliably — is identical, so it lives here and both jobs call it.
+
+The actual detector + embedders run in the GPU agent; this is pure Pillow so it's
+importable + testable anywhere (and the agent imports it for the crop step).
+"""
+
+from __future__ import annotations
+
+from PIL import Image
+
+# Size floor: a region must be at least this big on its SHORTER edge to be worth
+# embedding — a smaller crop is a blurry upscale carrying little real signal, and
+# unbounded tiny crops would explode the bag. Expressed as BOTH a fraction of the
+# image's short side and an absolute pixel floor; the larger of the two wins.
+MIN_CROP_FRACTION = 0.10
+MIN_CROP_PX = 64
+
+
+def _to_pixels(bbox: tuple[float, float, float, float], w: int, h: int):
+    """Normalized (x, y, w, h) in [0,1] → pixel (x, y, w, h)."""
+    x, y, bw, bh = bbox
+    return x * w, y * h, bw * w, bh * h
+
+
+def crop_region(
+    img: Image.Image,
+    bbox: tuple[float, float, float, float],
+    *,
+    pad: float = 0.0,
+    min_fraction: float = MIN_CROP_FRACTION,
+    min_px: int = MIN_CROP_PX,
+    out_size: int | None = None,
+) -> Image.Image | None:
+    """Crop a NORMALIZED bbox (x, y, w, h in [0,1]) from img.
+
+    - pad: grow the box by this fraction on each side (e.g. 0.15 = +15% context),
+      clamped to the image bounds.
+    - Returns None when the resulting region is below the size floor (too small to
+      embed reliably) — the caller skips embedding it.
+    - out_size: if given, resize the crop to out_size×out_size; otherwise return
+      the raw crop and let the embedder do its own preprocessing.
+    """
+    iw, ih = img.size
+    px, py, pw, ph = _to_pixels(bbox, iw, ih)
+
+    if pad:
+        px -= pw * pad / 2.0
+        py -= ph * pad / 2.0
+        pw *= (1.0 + pad)
+        ph *= (1.0 + pad)
+
+    left = max(0, int(round(px)))
+    top = max(0, int(round(py)))
+    right = min(iw, int(round(px + pw)))
+    bottom = min(ih, int(round(py + ph)))
+    if right <= left or bottom <= top:
+        return None
+
+    floor = max(min_px, int(min_fraction * min(iw, ih)))
+    if min(right - left, bottom - top) < floor:
+        return None
+
+    crop = img.crop((left, top, right, bottom)).convert("RGB")
+    if out_size:
+        crop = crop.resize((out_size, out_size))
+    return crop

backend/app/services/ml/embedder.py

@@ -1,8 +1,8 @@
 """SigLIP SO400M image-embedding wrapper (PyTorch CPU).
 
-Direct port of ImageRepo's siglip.py. torch/transformers are imported
-lazily inside load() so this module can be imported in the web container
-(which never runs inference) without paying the torch import cost.
+torch/transformers are imported lazily inside load() so this module can be
+imported in the web container (which never runs inference) without paying the
+torch import cost.
 """
 
 import os
@@ -13,6 +13,11 @@ from PIL import Image, ImageFile
 
 ImageFile.LOAD_TRUNCATED_IMAGES = True
 
+# Cap torch's intra-op threads so each ml-worker replica is a bounded core
+# consumer on a shared node (torch otherwise uses all cores). Keep
+# N_replicas × this within the cores allotted to ML to avoid oversubscription.
+_INTRA_OP_THREADS = 4
+
 MODEL_NAME = os.environ.get(
     "SIGLIP_MODEL_NAME", "google/siglip-so400m-patch14-384"
 )
@@ -37,6 +42,9 @@ class Embedder:
         from transformers import AutoModel, SiglipImageProcessor
 
         self._torch = torch
+        # Bound torch's CPU thread pool (see _INTRA_OP_THREADS) so each replica
+        # stays a predictable core consumer on a shared node.
+        torch.set_num_threads(_INTRA_OP_THREADS)
         # FC's embedder only does IMAGE inference — never text. AutoProcessor
         # loads the full processor including SiglipTokenizer, which requires
         # the sentencepiece library at import time even if we never call it.

backend/app/services/ml/gpu_jobs.py

@@ -0,0 +1,177 @@
+"""GPU-job queue engine (#114): enqueue / lease / heartbeat / complete / fail
+/ release / recover_orphaned.
+
+Backs the HTTP API the desktop agent pulls work from. The lease claims pending
+OR expired-leased jobs with FOR UPDATE SKIP LOCKED, so concurrent agents/workers
+never grab the same job. Orphan recovery is three-layered: a graceful agent stop
+calls release() to hand its in-flight jobs back instantly; a hard crash is caught
+by recover_orphaned() (a 60s beat sweep) which resets expired leases to pending;
+and the lease itself reclaims expired leases as a final backstop. Result-writing
+(regions) is done by the API handler via RegionService; complete() just closes.
+"""
+
+from datetime import UTC, datetime, timedelta
+
+from sqlalchemy import and_, or_, select, update
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from ...models import GpuJob
+
+# Lease window. Kept comfortably above any single job (a capped-frame video embed
+# is tens of seconds) so a live, heartbeating worker is never falsely expired,
+# but short enough that a hard crash recovers fast once the sweep fires.
+DEFAULT_LEASE_TTL = 180     # seconds an agent holds a job before it can be re-leased
+DEFAULT_BATCH = 8
+MAX_ATTEMPTS = 3
+
+
+class GpuJobService:
+    def __init__(self, session: AsyncSession):
+        self.session = session
+
+    async def enqueue(self, image_id: int, task: str) -> GpuJob | None:
+        """Queue a (image, task) job. Idempotent: returns None if one is already
+        pending/leased for the same pair (no duplicate work)."""
+        dup = (
+            await self.session.execute(
+                select(GpuJob.id).where(
+                    GpuJob.image_record_id == image_id,
+                    GpuJob.task == task,
+                    GpuJob.status.in_(["pending", "leased"]),
+                )
+            )
+        ).first()
+        if dup:
+            return None
+        job = GpuJob(image_record_id=image_id, task=task, status="pending")
+        self.session.add(job)
+        await self.session.flush()
+        return job
+
+    async def lease(
+        self, token: str, batch_size: int = DEFAULT_BATCH, ttl: int = DEFAULT_LEASE_TTL
+    ) -> list[GpuJob]:
+        """Claim up to batch_size pending (or expired-leased) jobs for `token`."""
+        now = datetime.now(UTC)
+        picked = (
+            await self.session.execute(
+                select(GpuJob.id)
+                .where(
+                    or_(
+                        GpuJob.status == "pending",
+                        and_(
+                            GpuJob.status == "leased",
+                            GpuJob.lease_expires_at < now,
+                        ),
+                    )
+                )
+                .order_by(GpuJob.id)
+                .limit(batch_size)
+                .with_for_update(skip_locked=True)
+            )
+        ).scalars().all()
+        if not picked:
+            return []
+        await self.session.execute(
+            update(GpuJob)
+            .where(GpuJob.id.in_(picked))
+            .values(
+                status="leased", lease_token=token, leased_at=now,
+                lease_expires_at=now + timedelta(seconds=ttl),
+                attempts=GpuJob.attempts + 1, updated_at=now,
+            )
+        )
+        # populate_existing: overwrite identity-map copies with the post-UPDATE
+        # values so the returned jobs reflect the new lease/attempts, not stale
+        # pre-lease state.
+        return list(
+            (
+                await self.session.execute(
+                    select(GpuJob)
+                    .where(GpuJob.id.in_(picked))
+                    .order_by(GpuJob.id)
+                    .execution_options(populate_existing=True)
+                )
+            ).scalars()
+        )
+
+    async def heartbeat(
+        self, token: str, job_ids: list[int], ttl: int = DEFAULT_LEASE_TTL
+    ) -> int:
+        """Extend the lease on the agent's in-flight jobs. Returns rows touched."""
+        now = datetime.now(UTC)
+        res = await self.session.execute(
+            update(GpuJob)
+            .where(
+                GpuJob.id.in_(job_ids),
+                GpuJob.lease_token == token,
+                GpuJob.status == "leased",
+            )
+            .values(lease_expires_at=now + timedelta(seconds=ttl), updated_at=now)
+        )
+        return res.rowcount or 0
+
+    async def complete(self, token: str, job_id: int) -> bool:
+        """Close a leased job (after its results were stored). False if the job
+        isn't leased by this token (a stale/expired submit)."""
+        job = await self.session.get(GpuJob, job_id)
+        if job is None or job.status != "leased" or job.lease_token != token:
+            return False
+        job.status = "done"
+        job.lease_token = None
+        job.lease_expires_at = None
+        job.error = None
+        job.updated_at = datetime.now(UTC)
+        return True
+
+    async def fail(self, token: str, job_id: int, error: str) -> bool:
+        """Report a failure: re-queue (pending) until MAX_ATTEMPTS, then 'error'."""
+        job = await self.session.get(GpuJob, job_id)
+        if job is None or job.lease_token != token:
+            return False
+        if job.attempts >= MAX_ATTEMPTS:
+            job.status = "error"
+        else:
+            job.status = "pending"
+            job.lease_token = None
+            job.lease_expires_at = None
+        job.error = (error or "")[:1000]
+        job.updated_at = datetime.now(UTC)
+        return True
+
+    async def release(self, token: str, job_ids: list[int]) -> int:
+        """Hand the agent's still-leased jobs back to pending NOW (graceful stop),
+        so another worker picks them up immediately instead of waiting out the
+        lease. Scoped to the token's own leases. Returns rows released."""
+        if not job_ids:
+            return 0
+        now = datetime.now(UTC)
+        res = await self.session.execute(
+            update(GpuJob)
+            .where(
+                GpuJob.id.in_(job_ids),
+                GpuJob.lease_token == token,
+                GpuJob.status == "leased",
+            )
+            .values(
+                status="pending", lease_token=None, leased_at=None,
+                lease_expires_at=None, updated_at=now,
+            )
+        )
+        return res.rowcount or 0
+
+    async def recover_orphaned(self) -> int:
+        """Reset every expired lease back to pending — catches agents that died
+        mid-job (no graceful release). Run on a short beat so the queue recovers
+        + reads honestly even when no worker is actively leasing. Returns rows
+        recovered."""
+        now = datetime.now(UTC)
+        res = await self.session.execute(
+            update(GpuJob)
+            .where(GpuJob.status == "leased", GpuJob.lease_expires_at < now)
+            .values(
+                status="pending", lease_token=None, leased_at=None,
+                lease_expires_at=None, updated_at=now,
+            )
+        )
+        return res.rowcount or 0

backend/app/services/ml/heads.py

@@ -0,0 +1,490 @@
+"""Production heads: train + score the per-concept classifiers (#114).
+
+The eval (#1130, tag_eval.py) proved the spine; this is its production form.
+- TRAIN (sync, ml worker — needs scikit-learn): for every general/character tag
+  with enough labelled positives, fit a logistic-regression head on the FROZEN
+  SigLIP embeddings (positives + negatives = rejections + sampled unlabeled),
+  derive an honest suggest threshold + earned-auto-apply point from CROSS-
+  VALIDATED scores, and upsert a TagHead row. Reuses tag_eval's proven data
+  loaders + metric helpers so production heads match the eval's measured numbers.
+- SCORE (async, API worker — numpy via pgvector, NO scikit-learn): score one
+  image's embedding against all current heads → the suggestions the rail shows,
+  REPLACING Camie predictions + per-tag centroids.
+
+scikit-learn is imported lazily inside the train path so the API worker can still
+import this module to enqueue training + to score (scoring needs only numpy).
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import UTC, datetime
+from typing import Any
+
+from sqlalchemy import delete, func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+from sqlalchemy.orm import Session
+
+from ...models import (
+    HeadAutoApplyRun,
+    HeadTrainingRun,
+    ImageRecord,
+    ImageRegion,
+    MLSettings,
+    Tag,
+    TagHead,
+    TagKind,
+    TagSuggestionRejection,
+)
+from ...models.tag import image_tag
+from .tag_eval import (
+    _auto_apply_point,
+    _ids_with_tag,
+    _l2norm,
+    _load_embeddings,
+    _metrics_from_scores,
+    _rejected_ids,
+    _safe_folds,
+    _sample_unlabeled,
+)
+
+log = logging.getLogger(__name__)
+
+DEFAULT_NEG_RATIO = 3
+DEFAULT_CV_FOLDS = 5
+MIN_POSITIVES_FLOOR = 8     # hard floor; settings.head_min_positives can raise it
+_UNLABELED_POOL = 4000
+_EXAMPLES_MIN = 8           # need at least this many embedded +/- to fit a head
+
+# Only these tag kinds get heads (the surfaced suggestion categories).
+_HEAD_KINDS = (TagKind.general, TagKind.character)
+# tag.kind -> the suggestion category the rail groups under.
+_CATEGORY = {TagKind.general: "general", TagKind.character: "character"}
+
+
+class HeadTrainingAlreadyRunning(Exception):
+    """Raised by start_head_training_run when a run is already in flight."""
+
+
+def start_head_training_run(session: Session, params: dict[str, Any]) -> int:
+    """Create a HeadTrainingRun (status='running') + dispatch the ml-queue task.
+    Returns the run id. One training run at a time (light guard)."""
+    existing = session.execute(
+        select(HeadTrainingRun.id).where(HeadTrainingRun.status == "running")
+    ).scalar_one_or_none()
+    if existing is not None:
+        raise HeadTrainingAlreadyRunning(existing)
+    norm = _normalize_params(session, params)
+    run = HeadTrainingRun(
+        params=norm, status="running", last_progress_at=datetime.now(UTC)
+    )
+    session.add(run)
+    session.flush()
+    run_id = run.id
+    from ...tasks.ml import train_heads as _task
+    _task.delay(run_id)
+    return run_id
+
+
+def _settings(session: Session) -> MLSettings:
+    return session.execute(
+        select(MLSettings).where(MLSettings.id == 1)
+    ).scalar_one()
+
+
+def _normalize_params(session: Session, params: dict[str, Any] | None) -> dict[str, Any]:
+    params = params or {}
+    s = _settings(session)
+    try:
+        min_pos = max(MIN_POSITIVES_FLOOR, int(params.get("min_positives", s.head_min_positives)))
+    except (TypeError, ValueError):
+        min_pos = max(MIN_POSITIVES_FLOOR, s.head_min_positives)
+    try:
+        neg_ratio = max(1, int(params.get("neg_ratio", DEFAULT_NEG_RATIO)))
+    except (TypeError, ValueError):
+        neg_ratio = DEFAULT_NEG_RATIO
+    try:
+        cv_folds = max(2, int(params.get("cv_folds", DEFAULT_CV_FOLDS)))
+    except (TypeError, ValueError):
+        cv_folds = DEFAULT_CV_FOLDS
+    try:
+        precision_target = min(max(float(params.get("precision_target", s.head_auto_apply_precision)), 0.5), 0.999)
+    except (TypeError, ValueError):
+        precision_target = s.head_auto_apply_precision
+    return {
+        "min_positives": min_pos,
+        "neg_ratio": neg_ratio,
+        "cv_folds": cv_folds,
+        "precision_target": round(precision_target, 4),
+    }
+
+
+def _embedder_version(session: Session) -> str:
+    return _settings(session).embedder_model_version
+
+
+def _eligible_tag_ids(session: Session, min_pos: int) -> list[int]:
+    """Concept tags (general/character) with >= min_pos labelled images — the
+    set that gets a head. Counts all sources; source-aware filtering (#1133) is
+    a separate, optional refinement."""
+    rows = session.execute(
+        select(Tag.id)
+        .join(image_tag, image_tag.c.tag_id == Tag.id)
+        .where(Tag.kind.in_(_HEAD_KINDS))
+        .group_by(Tag.id)
+        .having(func.count(image_tag.c.image_record_id) >= min_pos)
+    ).all()
+    return [r[0] for r in rows]
+
+
+def train_all_heads(
+    session: Session, params: dict[str, Any], run: HeadTrainingRun | None = None
+) -> dict[str, int]:
+    """(Re)train a head for every eligible concept; prune heads whose tag is no
+    longer eligible. Commits per head so a SIGKILL leaves trained heads durable
+    (training is idempotent). Returns {n_trained, n_skipped}."""
+    import numpy as np
+
+    cfg = _normalize_params(session, params)
+    embedding_version = _embedder_version(session)
+    eligible = _eligible_tag_ids(session, cfg["min_positives"])
+    eligible_set = set(eligible)
+    trained = 0
+    skipped = 0
+    for i, tag_id in enumerate(eligible):
+        try:
+            ok = train_head(session, tag_id, embedding_version, cfg, np)
+        except Exception:
+            log.exception("train_head failed for tag %d", tag_id)
+            ok = False
+        session.commit()
+        trained += int(ok)
+        skipped += int(not ok)
+        if run is not None and i % 10 == 0:
+            run.last_progress_at = datetime.now(UTC)
+            session.commit()
+    # Retire heads whose concept dropped out of the eligible set (lost its
+    # positives, or the tag was re-kinded) so stale heads can't keep suggesting.
+    if eligible_set:
+        session.execute(delete(TagHead).where(TagHead.tag_id.not_in(eligible_set)))
+    else:
+        session.execute(delete(TagHead))
+    session.commit()
+    return {"n_trained": trained, "n_skipped": skipped}
+
+
+def train_head(
+    session: Session, tag_id: int, embedding_version: str, cfg: dict, np
+) -> bool:
+    """Fit + upsert one head. Returns True if a head was written, False if the
+    concept had too few usable examples to train (the row is then removed)."""
+    from sklearn.linear_model import LogisticRegression
+    from sklearn.model_selection import StratifiedKFold, cross_val_predict
+
+    pos_ids = _ids_with_tag(session, tag_id)
+    if len(pos_ids) < cfg["min_positives"]:
+        session.execute(delete(TagHead).where(TagHead.tag_id == tag_id))
+        return False
+
+    pos_set = set(pos_ids)
+    rejected = [i for i in _rejected_ids(session, tag_id) if i not in pos_set]
+    want_neg = max(len(pos_ids) * cfg["neg_ratio"], _EXAMPLES_MIN * 4)
+    sampled = _sample_unlabeled(
+        session, pos_set | set(rejected), min(_UNLABELED_POOL, want_neg)
+    )
+    neg_ids = rejected + [i for i in sampled if i not in pos_set]
+
+    emb = _load_embeddings(session, pos_ids + neg_ids)
+    pos = [emb[i] for i in pos_ids if i in emb]
+    neg = [emb[i] for i in neg_ids if i in emb]
+    if len(pos) < _EXAMPLES_MIN or len(neg) < _EXAMPLES_MIN:
+        session.execute(delete(TagHead).where(TagHead.tag_id == tag_id))
+        return False
+
+    X = np.vstack(pos + neg).astype(np.float32)
+    y = np.array([1] * len(pos) + [0] * len(neg))
+    Xn = _l2norm(X, np)
+
+    clf = LogisticRegression(max_iter=1000, class_weight="balanced")
+    cv = StratifiedKFold(
+        n_splits=_safe_folds(y, cfg["cv_folds"], np), shuffle=True, random_state=0
+    )
+    # Honest thresholds from out-of-fold scores; deployable weights from a final
+    # fit on ALL the data.
+    cv_probs = cross_val_predict(clf, Xn, y, cv=cv, method="predict_proba")[:, 1]
+    metrics = _metrics_from_scores(y, cv_probs, np)
+    auto = _auto_apply_point(y, cv_probs, cfg["precision_target"], np)
+    clf.fit(Xn, y)
+
+    head = session.get(TagHead, tag_id)
+    if head is None:
+        head = TagHead(tag_id=tag_id)
+        session.add(head)
+    head.embedding_version = embedding_version
+    head.weights = clf.coef_[0].astype(np.float32).tolist()
+    head.bias = float(clf.intercept_[0])
+    head.suggest_threshold = float(metrics["threshold"])
+    head.auto_apply_threshold = float(auto["threshold"]) if auto else None
+    head.n_pos = len(pos)
+    head.n_neg = len(neg)
+    head.ap = float(metrics["ap"])
+    head.precision_cv = float(metrics["precision"])
+    head.recall = float(metrics["recall"])
+    head.trained_at = datetime.now(UTC)
+    head.metrics = {"f1": metrics["f1"], "auto_apply": auto}
+    return True
+
+
+# --- Scoring (async, API worker) -----------------------------------------
+# Score one image against every current head to produce the rail's suggestions.
+# A tiny in-process cache holds the stacked weight matrix keyed on (count,
+# max(trained_at)) so a retrain invalidates it without per-request weight loads.
+_HEADS_CACHE: dict[str, Any] = {"key": None, "heads": None}
+
+
+async def _current_heads(session: AsyncSession, embedding_version: str):
+    """Stacked (W, b, thresholds, tag_id/name/category) for heads matching the
+    current embedding, cached until the next retrain."""
+    import numpy as np
+
+    sig = (
+        await session.execute(
+            select(func.count(), func.max(TagHead.trained_at)).where(
+                TagHead.embedding_version == embedding_version
+            )
+        )
+    ).one()
+    key = f"{embedding_version}:{sig[0]}:{sig[1].isoformat() if sig[1] else '-'}"
+    cached = _HEADS_CACHE.get("heads")
+    if cached is not None and _HEADS_CACHE.get("key") == key:
+        return cached
+    rows = (
+        await session.execute(
+            select(
+                TagHead.tag_id, Tag.name, Tag.kind,
+                TagHead.weights, TagHead.bias,
+                TagHead.suggest_threshold, TagHead.auto_apply_threshold,
+            )
+            .join(Tag, Tag.id == TagHead.tag_id)
+            .where(TagHead.embedding_version == embedding_version)
+        )
+    ).all()
+    if not rows:
+        loaded = {"W": None, "rows": []}
+    else:
+        W = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in rows])
+        b = np.asarray([r.bias for r in rows], dtype=np.float32)
+        thr = np.asarray([r.suggest_threshold for r in rows], dtype=np.float32)
+        meta = [
+            {
+                "tag_id": r.tag_id,
+                "name": r.name,
+                "category": _CATEGORY.get(r.kind, "general"),
+                "auto_apply_threshold": r.auto_apply_threshold,
+            }
+            for r in rows
+        ]
+        loaded = {"W": W, "b": b, "thr": thr, "meta": meta}
+    _HEADS_CACHE["key"] = key
+    _HEADS_CACHE["heads"] = loaded
+    return loaded
+
+
+async def score_image(
+    session: AsyncSession, image_id: int, threshold_override: float | None = None,
+) -> list[dict]:
+    """Suggestions for one image from the trained heads: [{tag_id, name,
+    category, score}], ranked. A concept surfaces when its score clears the
+    head's own suggest_threshold — or, when threshold_override is given (the
+    typed-dropdown "show everything" mode), that flat floor instead (0 → every
+    head). Empty if the image has no embedding or no heads exist yet.
+
+    MAX-OVER-BAG: the image is scored as a BAG of embeddings — the whole-image
+    vector PLUS every concept-region crop the agent embedded (same model
+    version) — and each head takes its MAX score across the bag. A small/local
+    concept (glasses, a stomach bulge) that the whole-image vector washes out
+    can still surface from the crop where it dominates. The whole-image vector is
+    always in the bag, so this can never score lower than whole-image alone."""
+    import numpy as np
+
+    img = await session.get(ImageRecord, image_id)
+    if img is None or img.siglip_embedding is None:
+        return []
+    settings = await _settings_async(session)
+    heads = await _current_heads(session, settings.embedder_model_version)
+    if heads["W"] is None:
+        return []
+
+    bag = [np.asarray(img.siglip_embedding, dtype=np.float32)]
+    region_vecs = (
+        await session.execute(
+            select(ImageRegion.siglip_embedding)
+            .where(ImageRegion.image_record_id == image_id)
+            .where(ImageRegion.siglip_embedding.is_not(None))
+            .where(ImageRegion.embedding_version == settings.embedder_model_version)
+        )
+    ).all()
+    for (vec,) in region_vecs:
+        if vec is not None:
+            bag.append(np.asarray(vec, dtype=np.float32))
+
+    X = np.vstack(bag)                                  # (B, D)
+    norms = np.linalg.norm(X, axis=1, keepdims=True)
+    norms[norms == 0] = 1.0
+    Xn = X / norms
+    Z = Xn @ heads["W"].T + heads["b"]                  # (B, H)
+    probs = (1.0 / (1.0 + np.exp(-Z))).max(axis=0)      # (H,) best over the bag
+    out = []
+    for i, p in enumerate(probs):
+        cut = threshold_override if threshold_override is not None else heads["thr"][i]
+        if p >= cut:
+            m = heads["meta"][i]
+            out.append({
+                "tag_id": m["tag_id"],
+                "name": m["name"],
+                "category": m["category"],
+                "score": float(p),
+            })
+    out.sort(key=lambda d: d["score"], reverse=True)
+    return out
+
+
+async def _settings_async(session: AsyncSession) -> MLSettings:
+    return (
+        await session.execute(select(MLSettings).where(MLSettings.id == 1))
+    ).scalar_one()
+
+
+# --- Earned auto-apply (sync, ml worker) ---------------------------------
+# A graduated head can apply its tag to images it scores above the head's
+# auto_apply_threshold, without a human. Gated by a master switch + a support
+# floor so a precise-looking but under-supported head can't spray tags.
+
+_AUTO_APPLY_CHUNK = 5000
+
+
+class HeadAutoApplyAlreadyRunning(Exception):
+    """Raised when an auto-apply sweep is already in flight."""
+
+
+class HeadAutoApplyDisabled(Exception):
+    """Raised when a real (non-dry-run) sweep is requested but the master
+    switch (head_auto_apply_enabled) is off."""
+
+
+def start_head_auto_apply_run(session: Session, params: dict[str, Any]) -> int:
+    """Create a HeadAutoApplyRun + dispatch the ml-queue sweep. dry_run previews
+    (writes nothing); a real sweep needs the master switch on. One run at a time."""
+    dry_run = bool((params or {}).get("dry_run", False))
+    existing = session.execute(
+        select(HeadAutoApplyRun.id).where(HeadAutoApplyRun.status == "running")
+    ).scalar_one_or_none()
+    if existing is not None:
+        raise HeadAutoApplyAlreadyRunning(existing)
+    if not dry_run and not _settings(session).head_auto_apply_enabled:
+        raise HeadAutoApplyDisabled()
+    run = HeadAutoApplyRun(
+        dry_run=dry_run, params={"dry_run": dry_run}, status="running",
+        last_progress_at=datetime.now(UTC),
+    )
+    session.add(run)
+    session.flush()
+    run_id = run.id
+    from ...tasks.ml import apply_head_tags as _task
+    _task.delay(run_id)
+    return run_id
+
+
+def _auto_apply_heads(session: Session, embedding_version: str, min_pos: int):
+    """Eligible heads to fire: graduated (auto_apply_threshold set), enough
+    support, current embedding. Returns the row list (tag_id/name/weights/...)."""
+    return session.execute(
+        select(
+            TagHead.tag_id, Tag.name, TagHead.weights, TagHead.bias,
+            TagHead.auto_apply_threshold,
+        )
+        .join(Tag, Tag.id == TagHead.tag_id)
+        .where(TagHead.embedding_version == embedding_version)
+        .where(TagHead.auto_apply_threshold.is_not(None))
+        .where(TagHead.n_pos >= min_pos)
+    ).all()
+
+
+def auto_apply_sweep(
+    session: Session, run: HeadAutoApplyRun, dry_run: bool
+) -> dict[str, Any]:
+    """Score every embedded image against the eligible heads and apply (or, for
+    dry_run, just count) each head's tag where score >= its auto_apply_threshold
+    and the tag isn't already applied or rejected on that image. Streams
+    embeddings in chunks; commits per chunk on a real run. Returns
+    {n_applied, concepts:[{tag_id,name,applied,scanned,threshold}]}."""
+    import numpy as np
+    from sqlalchemy.dialects.postgresql import insert as pg_insert
+
+    settings = _settings(session)
+    rows = _auto_apply_heads(
+        session, settings.embedder_model_version,
+        settings.head_auto_apply_min_positives,
+    )
+    if not rows:
+        return {"n_applied": 0, "concepts": []}
+
+    W = np.vstack([np.asarray(r.weights, dtype=np.float32) for r in rows])
+    b = np.asarray([r.bias for r in rows], dtype=np.float32)
+    thr = np.asarray([r.auto_apply_threshold for r in rows], dtype=np.float32)
+    tag_ids = [r.tag_id for r in rows]
+    names = [r.name for r in rows]
+
+    # Skip images that already carry, or have rejected, each tag.
+    skip = {tid: set() for tid in tag_ids}
+    for tid in tag_ids:
+        for (iid,) in session.execute(
+            select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tid)
+        ):
+            skip[tid].add(iid)
+        for (iid,) in session.execute(
+            select(TagSuggestionRejection.image_record_id).where(
+                TagSuggestionRejection.tag_id == tid
+            )
+        ):
+            skip[tid].add(iid)
+
+    applied = [0] * len(rows)
+    scanned = 0
+    all_ids = list(session.execute(
+        select(ImageRecord.id).where(ImageRecord.siglip_embedding.is_not(None))
+    ).scalars())
+    for start in range(0, len(all_ids), _AUTO_APPLY_CHUNK):
+        chunk = all_ids[start:start + _AUTO_APPLY_CHUNK]
+        emb = _load_embeddings(session, chunk)
+        cids = [i for i in chunk if i in emb]
+        if not cids:
+            continue
+        Xn = _l2norm(np.vstack([emb[i] for i in cids]).astype(np.float32), np)
+        probs = 1.0 / (1.0 + np.exp(-(Xn @ W.T + b)))   # (N, H)
+        scanned += len(cids)
+        for h in range(len(rows)):
+            tid = tag_ids[h]
+            for idx in np.where(probs[:, h] >= thr[h])[0]:
+                iid = cids[int(idx)]
+                if iid in skip[tid]:
+                    continue
+                skip[tid].add(iid)
+                applied[h] += 1
+                if not dry_run:
+                    session.execute(
+                        pg_insert(image_tag)
+                        .values(image_record_id=iid, tag_id=tid, source="head_auto")
+                        .on_conflict_do_nothing()
+                    )
+        if not dry_run:
+            session.commit()
+        run.last_progress_at = datetime.now(UTC)
+        session.commit()
+
+    concepts = [
+        {"tag_id": tag_ids[h], "name": names[h], "applied": applied[h],
+         "scanned": scanned, "threshold": float(thr[h])}
+        for h in range(len(rows))
+    ]
+    return {"n_applied": sum(applied), "concepts": concepts}

backend/app/services/ml/regions.py

@@ -0,0 +1,59 @@
+"""Region read/write for the crop pipeline (#114).
+
+The GPU agent's results endpoint calls replace_regions() to store a freshly
+detected/embedded set; the character matcher + concept-bag scorer read via
+get_regions(). Replacement is scoped BY KIND so the figure pipeline and the
+concept pipeline don't clobber each other.
+"""
+
+from typing import Any
+
+from sqlalchemy import delete, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from ...models import ImageRegion
+
+
+class RegionService:
+    def __init__(self, session: AsyncSession):
+        self.session = session
+
+    async def get_regions(
+        self, image_id: int, kinds: list[str] | None = None
+    ) -> list[ImageRegion]:
+        stmt = select(ImageRegion).where(ImageRegion.image_record_id == image_id)
+        if kinds:
+            stmt = stmt.where(ImageRegion.kind.in_(kinds))
+        return list(
+            (await self.session.execute(stmt.order_by(ImageRegion.id))).scalars()
+        )
+
+    async def replace_regions(
+        self, image_id: int, kinds: list[str], regions: list[dict[str, Any]]
+    ) -> int:
+        """Replace this image's regions OF THE GIVEN KINDS with `regions` (a
+        re-detect/re-propose supersedes the prior set without touching other
+        kinds). Each region dict: {kind, bbox:(x,y,w,h), score?, detector_version?,
+        crop_version?, embedding_version?, ccip_embedding?, siglip_embedding?}.
+        Returns the number inserted."""
+        await self.session.execute(
+            delete(ImageRegion)
+            .where(ImageRegion.image_record_id == image_id)
+            .where(ImageRegion.kind.in_(kinds))
+        )
+        n = 0
+        for r in regions:
+            rx, ry, rw, rh = r["bbox"]
+            self.session.add(ImageRegion(
+                image_record_id=image_id, kind=r["kind"],
+                frame_time=r.get("frame_time"),
+                rx=rx, ry=ry, rw=rw, rh=rh,
+                score=r.get("score"),
+                detector_version=r.get("detector_version"),
+                crop_version=r.get("crop_version"),
+                embedding_version=r.get("embedding_version"),
+                ccip_embedding=r.get("ccip_embedding"),
+                siglip_embedding=r.get("siglip_embedding"),
+            ))
+            n += 1
+        return n

backend/app/services/ml/suggestions.py

@@ -1,24 +1,23 @@
-"""The suggestion read-path: raw predictions + centroids -> alias-resolved,
-threshold-filtered, category-grouped, ranked suggestions for one image.
+"""The suggestion read-path: trained HEADS score one image's frozen embedding
+into alias-resolved, category-grouped, ranked suggestions.
+
+Tagging-v2 (#114): suggestions now come from the per-concept heads that LEARN
+from the operator's tags (services/ml/heads.py) — the Camie prediction source
+and the per-tag SigLIP centroid have been REMOVED. A head exists only for an
+existing concept tag, so every suggestion is a canonical tag (no raw model key,
+no alias remap, no creates-new). Rejected tags stay in the list FLAGGED (not
+dropped) so the rail can show + reverse a dismissal.
 """
 
 from dataclasses import dataclass, field
 
-from sqlalchemy import func, select
+from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from ...models import (
-    ImagePrediction,
-    ImageRecord,
-    MLSettings,
-    Tag,
-    TagSuggestionRejection,
-)
+from ...models import ImageRecord, TagSuggestionRejection
 from ...models.tag import image_tag
-from .aliases import AliasService
-from .centroids import CentroidService
-from .tag_name import normalize as normalize_tag_name
-from .tagger import SURFACED_CATEGORIES
+from .ccip import match_image as ccip_match_image
+from .heads import score_image
 
 
 @dataclass(frozen=True)
@@ -29,8 +28,21 @@ class Suggestion:
     display_name: str
     category: str
     score: float
-    source: str  # 'tagger' | 'centroid' | 'both'
+    source: str  # 'head' | 'ccip' | 'both' (Camie tagger/centroid removed in v2)
     creates_new_tag: bool
+    # raw_name = the booru model vocab key behind this suggestion. It's the key
+    # an alias MUST be stored under (resolution looks up the raw key), so the
+    # modal needs it to author an alias correctly. None for centroid-only hits
+    # (no underlying prediction → nothing to alias).
+    raw_name: str | None = None
+    # via_alias = this suggestion was surfaced because an operator alias remapped
+    # the raw prediction to this canonical tag. Lets the UI mark it + offer undo.
+    via_alias: bool = False
+    # rejected = the operator dismissed this tag for this image (a stored
+    # TagSuggestionRejection). It stays in the list — flagged, not dropped — so
+    # the rejection is VISIBLE and REVERSIBLE in the rail (misclick recovery,
+    # operator-asked 2026-06-27) instead of silently vanishing or re-suggesting.
+    rejected: bool = False
 
 
 @dataclass
@@ -41,67 +53,24 @@ class SuggestionList:
 class SuggestionService:
     def __init__(self, session: AsyncSession):
         self.session = session
-        self.aliases = AliasService(session)
-        self.centroids = CentroidService(session)
-
-    async def _settings(self) -> MLSettings:
-        return (
-            await self.session.execute(select(MLSettings).where(MLSettings.id == 1))
-        ).scalar_one()
-
-    async def _load_predictions(self, image_id: int) -> dict:
-        """Predictions for one image from the normalized image_prediction
-        table (#768), in the {raw_name: {category, confidence}} shape the rest
-        of this service consumed from the old JSON column — so all downstream
-        threshold/alias/merge logic is unchanged."""
-        rows = (
-            await self.session.execute(
-                select(
-                    ImagePrediction.raw_name,
-                    ImagePrediction.category,
-                    ImagePrediction.score,
-                ).where(ImagePrediction.image_record_id == image_id)
-            )
-        ).all()
-        return {
-            r.raw_name: {"category": r.category, "confidence": r.score}
-            for r in rows
-        }
-
-    def _threshold_for(
-        self, s: MLSettings, category: str, override: float | None = None,
-    ) -> float:
-        # 'artist' (FC-2d-vii-c) and 'copyright' (2026-06-01) retired;
-        # both fall through to the 1.01 "never surfaces" default like any
-        # unsurfaced category.
-        # override (the typed-dropdown "show everything the model saw" mode)
-        # applies to the surfaced categories only — unsurfaced ones are already
-        # skipped before the threshold check, so they can't leak in.
-        if override is not None:
-            return override
-        return {
-            "character": s.suggestion_threshold_character,
-            "general": s.suggestion_threshold_general,
-        }.get(category, 1.01)
 
     async def for_image(
-        self, image_id: int, *, threshold_override: float | None = None,
+        self, image_id: int, threshold_override: float | None = None,
     ) -> SuggestionList:
-        """Ranked suggestions for one image.
+        """Head-scored suggestions for one image, grouped by category and ranked.
 
-        threshold_override surfaces EVERY stored tagger prediction (down to the
-        ingest STORE_FLOOR) regardless of the configured per-category suggestion
-        thresholds — backs the tag-input dropdown's "search all of the model's
-        predictions, including low-confidence ones, in the canonical formatting"
-        mode (operator-asked 2026-06-09). The Suggestions panel still calls with
-        no override so it stays the curated above-threshold list."""
+        Each trained head scores the image's frozen embedding; a concept surfaces
+        when its score clears the head's own suggest threshold. threshold_override
+        (used by the typed tag-input dropdown's "show everything" mode) replaces
+        that per-head cut with a flat floor (0 → every head), so a low-scoring
+        concept can still be typed + picked in canonical formatting.
+
+        Already-applied tags are dropped; rejected tags stay FLAGGED and sink to
+        the bottom of their category so a dismissal is visible + reversible."""
         img = await self.session.get(ImageRecord, image_id)
         if img is None:
             return SuggestionList()
 
-        settings = await self._settings()
-        predictions: dict = await self._load_predictions(image_id)
-
         applied = set(
             (
                 await self.session.execute(
@@ -121,137 +90,50 @@ class SuggestionService:
             ).scalars().all()
         )
 
-        # --- Camie predictions ---
-        # candidates carry (raw_name, display_name, category, confidence).
-        # raw_name = the booru-formatted vocab key, kept for alias_map
-        # lookup since alias rows are hand-curated against raw keys.
-        # display_name = normalize_tag_name(raw_name) — what the operator
-        # sees AND what gets written to tag.name on Accept.
-        candidates: list[tuple[str, str, str, float]] = []
-        for name, p in predictions.items():
-            category = p.get("category", "general")
-            if category not in SURFACED_CATEGORIES:
-                continue
-            conf = float(p.get("confidence", 0.0))
-            if conf < self._threshold_for(settings, category, threshold_override):
-                continue
-            display = normalize_tag_name(name)
-            if display is None:
-                # emoticon / pure-punctuation vocab entry — drop entirely
-                continue
-            candidates.append((name, display, category, conf))
-
-        alias_map = await self.aliases.resolve_many(
-            [(raw, c) for raw, _disp, c, _conf in candidates]
+        hits = await score_image(
+            self.session, image_id, threshold_override=threshold_override
         )
+        # CCIP character matches OVERLAY the SigLIP character heads — a
+        # complementary, identity-specialized signal with different failure modes
+        # (CCIP needs a detected figure; heads work whole-image). Merged by tag:
+        # 'both' when they corroborate, taking the higher score.
+        ccip_hits = await ccip_match_image(self.session, image_id)
 
-        merged: dict[object, Suggestion] = {}
-
-        def _merge(key, sug: Suggestion):
-            existing = merged.get(key)
-            if existing is None:
-                merged[key] = sug
-            elif sug.score > existing.score:
-                merged[key] = Suggestion(
-                    canonical_tag_id=existing.canonical_tag_id,
-                    display_name=existing.display_name,
-                    category=existing.category,
-                    score=sug.score,
-                    source="both"
-                    if existing.source != sug.source
-                    else existing.source,
-                    creates_new_tag=existing.creates_new_tag,
-                )
-
-        for raw, display, category, conf in candidates:
-            canonical = alias_map.get((raw, category))
-            if canonical is not None:
-                if canonical.id in applied or canonical.id in rejected:
-                    continue
-                _merge(
-                    canonical.id,
-                    Suggestion(
-                        canonical_tag_id=canonical.id,
-                        display_name=canonical.name,
-                        category=category,
-                        score=conf,
-                        source="tagger",
-                        creates_new_tag=False,
-                    ),
-                )
+        merged: dict[tuple[str, int], dict] = {}
+        for h in hits:
+            merged[(h["category"], h["tag_id"])] = {
+                "name": h["name"], "score": h["score"], "source": "head",
+            }
+        for c in ccip_hits:
+            key = ("character", c["tag_id"])
+            ex = merged.get(key)
+            if ex is not None:
+                ex["source"] = "both"
+                ex["score"] = max(ex["score"], c["score"])
             else:
-                # Case-insensitive match on BOTH the raw camie key AND
-                # the normalized form — covers legacy underscore-named
-                # Tag rows accepted before normalization shipped, AND
-                # any tag the operator created with the human form.
-                existing_tag = (
-                    await self.session.execute(
-                        select(Tag).where(
-                            func.lower(Tag.name).in_(
-                                [raw.lower(), display.lower()]
-                            )
-                        )
-                    )
-                ).scalars().first()
-                if existing_tag is not None:
-                    if (
-                        existing_tag.id in applied
-                        or existing_tag.id in rejected
-                    ):
-                        continue
-                    _merge(
-                        existing_tag.id,
-                        Suggestion(
-                            canonical_tag_id=existing_tag.id,
-                            display_name=existing_tag.name,
-                            category=category,
-                            score=conf,
-                            source="tagger",
-                            creates_new_tag=False,
-                        ),
-                    )
-                else:
-                    _merge(
-                        f"raw:{display}:{category}",
-                        Suggestion(
-                            canonical_tag_id=None,
-                            display_name=display,
-                            category=category,
-                            score=conf,
-                            source="tagger",
-                            creates_new_tag=True,
-                        ),
-                    )
-
-        # --- Centroid augmentation ---
-        hits = await self.centroids.find_similar_tags(image_id, limit=30)
-        for hit in hits:
-            if hit.similarity < settings.centroid_similarity_threshold:
-                continue
-            if hit.tag_id in applied or hit.tag_id in rejected:
-                continue
-            tag = await self.session.get(Tag, hit.tag_id)
-            if tag is None:
-                continue
-            cat = tag.kind.value if hasattr(tag.kind, "value") else str(tag.kind)
-            display_cat = cat if cat in SURFACED_CATEGORIES else "general"
-            _merge(
-                tag.id,
-                Suggestion(
-                    canonical_tag_id=tag.id,
-                    display_name=tag.name,
-                    category=display_cat,
-                    score=hit.similarity,
-                    source="centroid",
-                    creates_new_tag=False,
-                ),
-            )
+                merged[key] = {
+                    "name": c["name"], "score": c["score"], "source": "ccip",
+                }
 
         result = SuggestionList()
-        for sug in merged.values():
-            result.by_category.setdefault(sug.category, []).append(sug)
+        for (cat, tag_id), m in merged.items():
+            if tag_id in applied:
+                continue
+            result.by_category.setdefault(cat, []).append(
+                Suggestion(
+                    canonical_tag_id=tag_id,
+                    display_name=m["name"],
+                    category=cat,
+                    score=m["score"],
+                    source=m["source"],
+                    creates_new_tag=False,
+                    rejected=tag_id in rejected,
+                )
+            )
         for cat in result.by_category:
-            result.by_category[cat].sort(key=lambda s: s.score, reverse=True)
+            # Live suggestions first (by score), rejected ones sink to the
+            # bottom of the category — visible for recovery, out of the way.
+            result.by_category[cat].sort(key=lambda s: (s.rejected, -s.score))
         return result
 
     async def for_selection(
@@ -278,6 +160,11 @@ class SuggestionService:
                 for s in items:
                     if s.canonical_tag_id is None or s.creates_new_tag:
                         continue
+                    # for_image keeps rejected tags (flagged) for the rail;
+                    # bulk consensus must still ignore them — a tag dismissed on
+                    # an image isn't a suggestion for that image.
+                    if s.rejected:
+                        continue
                     st = stats.get(s.canonical_tag_id)
                     if st is None:
                         st = {

backend/app/services/ml/tag_eval.py

@@ -0,0 +1,430 @@
+"""Head-vs-centroid tagging eval (#1130, milestone #114 slice 1).
+
+Proves the "frozen embedding + small trained head (with negatives)" spine on the
+operator's OWN data, reusing the SigLIP embeddings already stored on
+image_record. For each concept tag it compares:
+  - CENTROID baseline (the old approach): cosine to the mean of positive vectors.
+  - HEAD (the new approach): logistic regression trained on positives + negatives.
+and reports cross-validated precision/recall/AP for both, a LEARNING CURVE
+(accuracy as the number of tagged positives grows), and example image ids to
+eyeball.
+
+numpy + scikit-learn are imported LAZILY inside run_eval so the API worker (base
+image, no ML stack) can still import start_tag_eval_run to enqueue the ml-queue
+task — the heavy compute only runs on the ml worker.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import UTC, datetime
+from typing import Any
+
+from sqlalchemy import func, select
+from sqlalchemy.orm import Session
+
+from ...models import (
+    ImageRecord,
+    Tag,
+    TagEvalRun,
+    TagKind,
+    TagPositiveConfirmation,
+    TagSuggestionRejection,
+)
+from ...models.tag import image_tag
+
+log = logging.getLogger(__name__)
+
+# The operator's real concept list (mix of whole-ish + small/local cues). The
+# admin trigger can override; this is the default eval set.
+DEFAULT_CONCEPTS = [
+    "glasses", "cat", "dog", "horse", "goblin",
+    "cum", "lactation", "fellatio", "xray", "stomach bulge",
+]
+DEFAULT_CURVE_POINTS = [10, 30, 100, 300]
+DEFAULT_NEG_RATIO = 3       # negatives per positive (rejections + sampled unlabeled)
+DEFAULT_CV_FOLDS = 5
+MIN_POSITIVES = 8           # below this, a concept can't be evaluated meaningfully
+_UNLABELED_POOL = 4000      # cap on sampled unlabeled rows pulled per concept
+_EXAMPLES_K = 12
+
+
+def start_tag_eval_run(session: Session, params: dict[str, Any]) -> int:
+    """Create a TagEvalRun (status='running') and dispatch the ml-queue task.
+    Returns the new run id. Light guard: one running eval at a time."""
+    existing = session.execute(
+        select(TagEvalRun.id).where(TagEvalRun.status == "running")
+    ).scalar_one_or_none()
+    if existing is not None:
+        raise EvalAlreadyRunning(existing)
+    norm = _normalize_params(params)
+    run = TagEvalRun(params=norm, status="running", last_progress_at=datetime.now(UTC))
+    session.add(run)
+    session.flush()
+    run_id = run.id
+    # Same enqueue-by-import pattern api/suggestions.py uses for ml tasks; the
+    # commit happens in the API handler so row + dispatch are visible together.
+    from ...tasks.ml import tag_eval_run as _task
+    _task.delay(run_id)
+    return run_id
+
+
+class EvalAlreadyRunning(Exception):
+    """Raised by start_tag_eval_run when an eval is already in flight."""
+
+
+def _normalize_params(params: dict[str, Any] | None) -> dict[str, Any]:
+    params = params or {}
+    concepts = [str(c).strip() for c in (params.get("concepts") or []) if str(c).strip()]
+    try:
+        neg_ratio = max(1, int(params.get("neg_ratio", DEFAULT_NEG_RATIO)))
+    except (TypeError, ValueError):
+        neg_ratio = DEFAULT_NEG_RATIO
+    try:
+        cv_folds = max(2, int(params.get("cv_folds", DEFAULT_CV_FOLDS)))
+    except (TypeError, ValueError):
+        cv_folds = DEFAULT_CV_FOLDS
+    try:
+        auto_top_n = min(max(int(params.get("auto_top_n", 0) or 0), 0), 200)
+    except (TypeError, ValueError):
+        auto_top_n = 0
+    try:
+        precision_target = min(max(float(params.get("precision_target", 0.97)), 0.5), 0.999)
+    except (TypeError, ValueError):
+        precision_target = 0.97
+    # No explicit concepts and auto-discovery off → fall back to the hand list.
+    if not concepts and not auto_top_n:
+        concepts = list(DEFAULT_CONCEPTS)
+    curve = params.get("curve_points") or DEFAULT_CURVE_POINTS
+    curve = sorted({int(n) for n in curve if int(n) > 0})
+    return {
+        "concepts": concepts,
+        "neg_ratio": neg_ratio,
+        "cv_folds": cv_folds,
+        "auto_top_n": auto_top_n,
+        "precision_target": round(precision_target, 4),
+        "curve_points": curve,
+    }
+
+
+def _top_general_concepts(session: Session, n: int, min_count: int) -> list[str]:
+    """The n most-tagged general (concept) tags with >= min_count images — a fast
+    server-side way to broaden the eval beyond the hand-picked list (counts all
+    sources; source-aware filtering is a separate concern)."""
+    rows = session.execute(
+        select(Tag.name)
+        .join(image_tag, image_tag.c.tag_id == Tag.id)
+        .where(Tag.kind == TagKind.general)
+        .group_by(Tag.id)
+        .having(func.count(image_tag.c.image_record_id) >= min_count)
+        .order_by(func.count(image_tag.c.image_record_id).desc())
+        .limit(n)
+    ).all()
+    return [r[0] for r in rows]
+
+
+def _resolve_tag_id(session: Session, name: str) -> int | None:
+    """Case-insensitive tag-name match; if several share a name, take the one
+    applied to the most images (the one the operator actually uses)."""
+    rows = session.execute(
+        select(Tag.id, func.count(image_tag.c.image_record_id))
+        .outerjoin(image_tag, image_tag.c.tag_id == Tag.id)
+        .where(func.lower(Tag.name) == name.lower())
+        .group_by(Tag.id)
+        .order_by(func.count(image_tag.c.image_record_id).desc())
+    ).all()
+    return rows[0][0] if rows else None
+
+
+def _ids_with_tag(session: Session, tag_id: int) -> list[int]:
+    return [
+        r[0] for r in session.execute(
+            select(image_tag.c.image_record_id).where(image_tag.c.tag_id == tag_id)
+        ).all()
+    ]
+
+
+def _rejected_ids(session: Session, tag_id: int) -> list[int]:
+    return [
+        r[0] for r in session.execute(
+            select(TagSuggestionRejection.image_record_id)
+            .where(TagSuggestionRejection.tag_id == tag_id)
+        ).all()
+    ]
+
+
+def _confirmed_ids(session: Session, tag_id: int) -> set[int]:
+    """Positives the operator explicitly affirmed ('keep') — excluded from the
+    doubts list so confirmed-correct images don't resurface every run."""
+    return {
+        r[0] for r in session.execute(
+            select(TagPositiveConfirmation.image_record_id)
+            .where(TagPositiveConfirmation.tag_id == tag_id)
+        ).all()
+    }
+
+
+def _sample_unlabeled(session: Session, exclude: set[int], limit: int) -> list[int]:
+    """Random image ids (with an embedding) NOT carrying the tag. Concepts are
+    sparse, so an untagged image is almost always a true negative."""
+    stmt = (
+        select(ImageRecord.id)
+        .where(ImageRecord.siglip_embedding.is_not(None))
+        .order_by(func.random())
+        .limit(limit)
+    )
+    if exclude:
+        stmt = stmt.where(ImageRecord.id.not_in(exclude))
+    return [r[0] for r in session.execute(stmt).all()]
+
+
+def _load_embeddings(session: Session, ids: list[int]) -> dict[int, Any]:
+    import numpy as np
+
+    out: dict[int, Any] = {}
+    if not ids:
+        return out
+    # Chunk the IN list to stay well under psycopg's parameter ceiling.
+    for i in range(0, len(ids), 2000):
+        chunk = ids[i:i + 2000]
+        for rid, emb in session.execute(
+            select(ImageRecord.id, ImageRecord.siglip_embedding)
+            .where(ImageRecord.id.in_(chunk))
+            .where(ImageRecord.siglip_embedding.is_not(None))
+        ).all():
+            out[rid] = np.asarray(emb, dtype=np.float32)
+    return out
+
+
+def run_eval(session: Session, params: dict[str, Any]) -> dict[str, Any]:
+    """Compute the full report. Per-concept failures are captured, not fatal."""
+    import numpy as np
+
+    cfg = _normalize_params(params)
+    # Auto-discovery: union the explicit concepts with the top-N most-tagged
+    # general tags (server-side, fast) so the eval can broaden itself.
+    concepts = list(cfg["concepts"])
+    if cfg["auto_top_n"]:
+        seen = {c.lower() for c in concepts}
+        for name in _top_general_concepts(session, cfg["auto_top_n"], MIN_POSITIVES):
+            if name.lower() not in seen:
+                concepts.append(name)
+                seen.add(name.lower())
+    cfg["concepts"] = concepts
+    concepts_out = []
+    for name in cfg["concepts"]:
+        try:
+            concepts_out.append(_eval_concept(session, name, cfg, np))
+        except Exception as exc:  # one bad concept shouldn't kill the run
+            log.exception("tag-eval concept %r failed", name)
+            concepts_out.append({"name": name, "skipped": f"error: {exc}"})
+    return {
+        "generated_at": datetime.now(UTC).isoformat(),
+        "params": cfg,
+        "concepts": concepts_out,
+    }
+
+
+def _eval_concept(session: Session, name: str, cfg: dict, np) -> dict[str, Any]:
+    tag_id = _resolve_tag_id(session, name)
+    if tag_id is None:
+        return {"name": name, "skipped": "no such tag"}
+    pos_ids = _ids_with_tag(session, tag_id)
+    if len(pos_ids) < MIN_POSITIVES:
+        return {"name": name, "tag_id": tag_id, "n_pos": len(pos_ids),
+                "skipped": f"too few positives (<{MIN_POSITIVES})"}
+
+    neg_ratio = cfg["neg_ratio"]
+    pos_set = set(pos_ids)
+    rejected = [i for i in _rejected_ids(session, tag_id) if i not in pos_set]
+    want_neg = max(len(pos_ids) * neg_ratio, _EXAMPLES_K * 4)
+    sampled = _sample_unlabeled(session, pos_set | set(rejected),
+                                min(_UNLABELED_POOL, want_neg))
+    neg_ids = rejected + [i for i in sampled if i not in pos_set]
+
+    emb = _load_embeddings(session, pos_ids + neg_ids)
+    pos = [(i, emb[i]) for i in pos_ids if i in emb]
+    neg = [(i, emb[i]) for i in neg_ids if i in emb]
+    if len(pos) < MIN_POSITIVES or len(neg) < MIN_POSITIVES:
+        return {"name": name, "tag_id": tag_id, "n_pos": len(pos),
+                "n_neg": len(neg), "skipped": "too few embedded examples"}
+
+    ids = np.array([i for i, _ in pos] + [i for i, _ in neg])
+    X = np.vstack([v for _, v in pos] + [v for _, v in neg]).astype(np.float32)
+    y = np.array([1] * len(pos) + [0] * len(neg))
+    Xn = _l2norm(X, np)
+
+    head = _eval_head(Xn, y, cfg["cv_folds"], cfg["precision_target"], np)
+    centroid = _eval_centroid(Xn, y, cfg["cv_folds"], np)
+    curve = _learning_curve(Xn, y, cfg["curve_points"], neg_ratio, np)
+    confirmed = _confirmed_ids(session, tag_id)
+    examples = _examples(session, Xn, y, ids, np, set(rejected), confirmed)
+
+    return {
+        "name": name, "tag_id": tag_id,
+        "n_pos": len(pos), "n_neg": len(neg),
+        "n_rejected": len(rejected),
+        "head": head, "centroid": centroid,
+        "curve": curve, "examples": examples,
+    }
+
+
+def _l2norm(X, np):
+    n = np.linalg.norm(X, axis=1, keepdims=True)
+    n[n == 0] = 1.0
+    return X / n
+
+
+def _metrics_from_scores(y, scores, np) -> dict[str, float]:
+    from sklearn.metrics import average_precision_score, precision_recall_curve
+
+    ap = float(average_precision_score(y, scores))
+    prec, rec, thr = precision_recall_curve(y, scores)
+    f1 = (2 * prec * rec) / np.clip(prec + rec, 1e-9, None)
+    best = int(np.argmax(f1))
+    # thr has len = len(prec)-1; map best index safely.
+    t = float(thr[min(best, len(thr) - 1)]) if len(thr) else 0.5
+    return {
+        "ap": round(ap, 4),
+        "precision": round(float(prec[best]), 4),
+        "recall": round(float(rec[best]), 4),
+        "f1": round(float(f1[best]), 4),
+        "threshold": round(t, 4),
+    }
+
+
+def _safe_folds(y, folds, np) -> int:
+    minority = int(min(np.bincount(y)))
+    return max(2, min(folds, minority))
+
+
+def _eval_head(Xn, y, folds, target, np) -> dict[str, float]:
+    from sklearn.linear_model import LogisticRegression
+    from sklearn.model_selection import StratifiedKFold, cross_val_predict
+
+    clf = LogisticRegression(max_iter=1000, class_weight="balanced")
+    cv = StratifiedKFold(n_splits=_safe_folds(y, folds, np), shuffle=True,
+                         random_state=0)
+    probs = cross_val_predict(clf, Xn, y, cv=cv, method="predict_proba")[:, 1]
+    m = _metrics_from_scores(y, probs, np)
+    m["auto_apply"] = _auto_apply_point(y, probs, target, np)
+    return m
+
+
+def _auto_apply_point(y, scores, target, np) -> dict | None:
+    """The auto-apply operating point: the threshold that yields the MOST recall
+    while holding precision >= target. This answers 'could this concept fire
+    without a human, and how much would it catch?' Returns None if no threshold
+    reaches the precision target (concept not auto-apply-ready)."""
+    from sklearn.metrics import precision_recall_curve
+
+    prec, rec, thr = precision_recall_curve(y, scores)
+    best = None  # (threshold, precision, recall) maximizing recall s.t. prec>=target
+    for i in range(len(thr)):  # thr[i] corresponds to prec[i], rec[i]
+        if prec[i] >= target and (best is None or rec[i] > best[2]):
+            best = (float(thr[i]), float(prec[i]), float(rec[i]))
+    if best is None:
+        return None
+    return {
+        "target": round(float(target), 4),
+        "threshold": round(best[0], 4),
+        "precision": round(best[1], 4),
+        "recall": round(best[2], 4),
+    }
+
+
+def _eval_centroid(Xn, y, folds, np) -> dict[str, float]:
+    """Cross-validated cosine-to-positive-mean — the OLD method's quality."""
+    from sklearn.model_selection import StratifiedKFold
+
+    cv = StratifiedKFold(n_splits=_safe_folds(y, folds, np), shuffle=True,
+                         random_state=0)
+    scores = np.zeros(len(y), dtype=np.float32)
+    for train, test in cv.split(Xn, y):
+        c = Xn[train][y[train] == 1].mean(axis=0)
+        cn = c / (np.linalg.norm(c) or 1.0)
+        scores[test] = Xn[test] @ cn
+    return _metrics_from_scores(y, scores, np)
+
+
+def _learning_curve(Xn, y, points, neg_ratio, np) -> list[dict[str, float]]:
+    """Hold out a fixed test split; train the head on a growing number of
+    positives and watch AP/F1 climb — answers 'does tagging more sharpen it?'"""
+    from sklearn.linear_model import LogisticRegression
+    from sklearn.model_selection import train_test_split
+
+    rng = np.random.default_rng(0)
+    idx = np.arange(len(y))
+    try:
+        tr, te = train_test_split(idx, test_size=0.3, stratify=y, random_state=0)
+    except ValueError:
+        return []
+    tr_pos = tr[y[tr] == 1]
+    tr_neg = tr[y[tr] == 0]
+    out = []
+    for n in points:
+        if n > len(tr_pos):
+            break
+        sp = rng.choice(tr_pos, size=n, replace=False)
+        nn = min(len(tr_neg), n * neg_ratio)
+        sn = rng.choice(tr_neg, size=nn, replace=False)
+        sub = np.concatenate([sp, sn])
+        clf = LogisticRegression(max_iter=1000, class_weight="balanced")
+        clf.fit(Xn[sub], y[sub])
+        prob = clf.predict_proba(Xn[te])[:, 1]
+        m = _metrics_from_scores(y[te], prob, np)
+        out.append({"n_pos": int(n), "ap": m["ap"], "f1": m["f1"]})
+    return out
+
+
+def _examples(session, Xn, y, ids, np, rejected_set, confirmed_set) -> dict[str, list[dict]]:
+    """Train on all data, then surface: top-scoring negatives the operator has
+    NOT already rejected (= fresh suggestions) and lowest-scoring POSITIVES the
+    operator has NOT already confirmed (= unreviewed doubts). Excluding rejected
+    ids stops an adjudicated near-miss from resurfacing in 'would suggest';
+    excluding confirmed ids stops a 'kept' correct positive from resurfacing in
+    'head doubts' every run. Resolves thumbnail urls for a self-contained report."""
+    from sklearn.linear_model import LogisticRegression
+
+    clf = LogisticRegression(max_iter=1000, class_weight="balanced")
+    clf.fit(Xn, y)
+    s = clf.predict_proba(Xn)[:, 1]
+    neg_idx = np.where(y == 0)[0]
+    pos_idx = np.where(y == 1)[0]
+    top_neg = []
+    for i in neg_idx[np.argsort(s[neg_idx])[::-1]]:  # high score → low
+        rid = int(ids[i])
+        if rid in rejected_set:
+            continue  # already told the head 'no' — don't re-suggest it
+        top_neg.append(rid)
+        if len(top_neg) >= _EXAMPLES_K:
+            break
+    low_pos = []
+    for i in pos_idx[np.argsort(s[pos_idx])]:  # low score → high
+        rid = int(ids[i])
+        if rid in confirmed_set:
+            continue  # already kept/confirmed — don't re-doubt it
+        low_pos.append(rid)
+        if len(low_pos) >= _EXAMPLES_K:
+            break
+    thumbs = _resolve_thumbs(session, top_neg + low_pos)
+    return {
+        "head_would_suggest": [thumbs[i] for i in top_neg if i in thumbs],
+        "head_doubts_positive": [thumbs[i] for i in low_pos if i in thumbs],
+    }
+
+
+def _resolve_thumbs(session, ids: list[int]) -> dict[int, dict]:
+    from ..gallery_service import thumbnail_url
+
+    out: dict[int, dict] = {}
+    if not ids:
+        return out
+    for rid, tp, sha, mime in session.execute(
+        select(
+            ImageRecord.id, ImageRecord.thumbnail_path,
+            ImageRecord.sha256, ImageRecord.mime,
+        ).where(ImageRecord.id.in_(ids))
+    ).all():
+        out[rid] = {"id": rid, "thumbnail_url": thumbnail_url(tp, sha, mime)}
+    return out

backend/app/services/ml/tagger.py

@@ -1,8 +1,8 @@
-"""Camie-tagger-v2 ONNX wrapper.
+"""Camie-tagger-v2 ONNX wrapper (CPU).
 
-CPU-only, single-image at a time. Loaded lazily inside the ml-worker
-process; NOT thread-safe — the ml queue worker must run --concurrency=1
-(set by the FC-1 entrypoint).
+Single-image at a time. Loaded lazily inside the ml-worker process; NOT
+thread-safe — the ml queue worker runs --concurrency=1 per process (scale ML by
+running multiple worker replicas, not threads).
 
 v2 layout reference: HuggingFace Camais03/camie-tagger-v2 root has
 camie-tagger-v2.onnx (789 MB) + camie-tagger-v2-metadata.json (7.77 MB)
@@ -19,6 +19,11 @@ from pathlib import Path
 import numpy as np
 from PIL import Image, ImageFile
 
+# Cap inference threads (see Tagger.load) so each ml-worker replica is a bounded
+# core consumer on a shared node — keep N_replicas × this within the cores
+# allotted to ML so replicas don't oversubscribe the box / starve the DB.
+_INTRA_OP_THREADS = 4
+
 # onnxruntime lives in requirements-ml.txt only — it is NOT installed in the
 # lean web image or in CI. Imported lazily inside Tagger.load() so this module
 # imports fine without it (the suggestion service imports SURFACED_CATEGORIES
@@ -117,8 +122,15 @@ class Tagger:
         # without onnxruntime (CI / lean web image).
         import onnxruntime as ort
 
+        # Cap the intra-op thread pool. ONNX Runtime otherwise sizes it to ALL
+        # host cores, so on a shared node each ml-worker replica would grab every
+        # core and oversubscribe (and starve the co-located DB/web). Bounding it
+        # makes each replica a predictable core consumer — run N replicas where
+        # N × _INTRA_OP_THREADS stays within the cores you allot to ML.
+        opts = ort.SessionOptions()
+        opts.intra_op_num_threads = _INTRA_OP_THREADS
         session = ort.InferenceSession(
-            str(model_path), providers=["CPUExecutionProvider"]
+            str(model_path), sess_options=opts, providers=["CPUExecutionProvider"],
         )
         self._input_name = session.get_inputs()[0].name
         # Assign sentinels last so a partial load isn't observable.

backend/app/services/native_ingest_common.py

@@ -0,0 +1,345 @@
+"""Shared primitives for the native-ingest platform adapters (Patreon,
+SubscribeStar, …) — the single home for logic the per-platform client/downloader
+modules would otherwise each copy.
+
+DRY pass 2026-06-17 (#899): these used to live in `patreon_*` with the
+SubscribeStar modules importing patreon privates (wrong owner + sibling-coupling).
+They're platform-agnostic, so they live here and both adapters import them. The
+per-platform modules keep only what genuinely differs (feed parsing, the media
+shape, Patreon's Mux/yt-dlp video branch + detail-fetch enrichment).
+
+FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import http.cookiejar
+import json
+import logging
+import os
+import time
+from dataclasses import dataclass
+from datetime import datetime
+from pathlib import Path
+from urllib.parse import urlsplit
+
+import requests
+
+from ..utils.paths import filehash_from_url, safe_ext
+from .file_validator import is_validatable, quarantine_file, validate_file
+
+log = logging.getLogger(__name__)
+
+_USER_AGENT = (
+    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
+)
+
+# 429 backoff (plan #703): ride out a transient API rate-limit instead of failing
+# the whole walk. Honor the server's Retry-After; else exponential, capped.
+_MAX_429_RETRIES = 3
+_BACKOFF_BASE_SECONDS = 2.0
+_BACKOFF_CAP_SECONDS = 30.0
+
+# Media-download tuning (shared by every platform downloader).
+_TIMEOUT_SECONDS = 120.0
+_CHUNK = 1 << 16
+_MAX_MEDIA_RETRIES = 3
+_TRANSIENT_TRANSPORT_EXC = (
+    requests.ConnectionError,
+    requests.Timeout,
+    requests.exceptions.ChunkedEncodingError,
+)
+
+_TITLE_MAX = 40
+# Windows/gallery-dl path-restrict forbidden set + path separators.
+_FORBIDDEN = set('<>:"/\\|?*')
+
+
+# -- shared exception taxonomy --------------------------------------------
+# Every native client raises one of these (platform subclasses keep an
+# isinstance-distinct platform name AND the semantic Auth/Drift class), so the
+# base Ingester._failure_result can map them platform-agnostically.
+
+class NativeIngestError(Exception):
+    """Base for a native-ingest client failure. `status_code` carries the HTTP
+    status when the failure was an HTTP response (None for transport/parse);
+    `retry_after` carries the server's 429 Retry-After hint so the cooldown can
+    match it (plan #708 B1)."""
+
+    def __init__(
+        self,
+        message: str,
+        *,
+        status_code: int | None = None,
+        retry_after: float | None = None,
+    ):
+        super().__init__(message)
+        self.status_code = status_code
+        self.retry_after = retry_after
+
+
+class NativeAuthError(NativeIngestError):
+    """Authentication/authorization failure — expired/missing credential or an
+    insufficient tier. The fix is rotating the credential, NOT updating the
+    ingester. Maps to error_type 'auth_error'."""
+
+
+class NativeDriftError(NativeIngestError):
+    """A response did not match the shape the ingester depends on (JSON:API field
+    set, or scraped HTML structure). Fail loud so the import step flags 'the
+    platform changed' instead of silently importing nothing. Maps to API_DRIFT."""
+
+
+# -- HTTP session ----------------------------------------------------------
+
+def make_session(
+    cookies_path: str | Path | None,
+    *,
+    accept: str = "*/*",
+    extra_headers: dict | None = None,
+) -> requests.Session:
+    """Build a requests.Session loaded with the Netscape cookies.txt
+    CredentialService materializes. `accept` sets the Accept header (the JSON:API
+    vs HTML feed differ); `extra_headers` adds platform headers (e.g.
+    X-Requested-With). Missing/unparseable cookies log a warning, never fail."""
+    session = requests.Session()
+    headers = {"User-Agent": _USER_AGENT, "Accept": accept}
+    if extra_headers:
+        headers.update(extra_headers)
+    session.headers.update(headers)
+    if cookies_path and os.path.isfile(str(cookies_path)):
+        try:
+            jar = http.cookiejar.MozillaCookieJar(str(cookies_path))
+            jar.load(ignore_discard=True, ignore_expires=True)
+            session.cookies = jar  # type: ignore[assignment]
+        except (OSError, http.cookiejar.LoadError) as exc:
+            log.warning("Could not load cookies from %s: %s", cookies_path, exc)
+    return session
+
+
+def retry_after_seconds(
+    resp: requests.Response,
+    attempt: int,
+    *,
+    base: float = _BACKOFF_BASE_SECONDS,
+    cap: float = _BACKOFF_CAP_SECONDS,
+) -> float:
+    """Backoff for a 429: the numeric Retry-After header if present, else
+    exponential base·2^(attempt-1), both capped."""
+    header = resp.headers.get("Retry-After")
+    if header:
+        try:
+            return min(float(header), cap)
+        except (TypeError, ValueError):
+            pass
+    return min(base * (2 ** max(0, attempt - 1)), cap)
+
+
+# -- filename / path helpers -----------------------------------------------
+
+def sanitize_segment(name: str) -> str:
+    """Make `name` safe for one filesystem path segment: replace separators, the
+    Windows-forbidden set, and control chars with `_`; strip trailing dots/spaces
+    (gallery-dl path-restrict). Never empty (falls back to `_`)."""
+    out = ["_" if (ch in _FORBIDDEN or ord(ch) < 32) else ch for ch in name]
+    cleaned = "".join(out).rstrip(". ")
+    return cleaned or "_"
+
+
+def basename_from_url(url: str) -> str:
+    """Derive a sane filename from a URL when the media has no name: path basename
+    with a junk-extension guard (safe_ext), bounded stem; falls back to the URL's
+    content hash, then "file"."""
+    path = urlsplit(url).path
+    base = os.path.basename(path)
+    if base:
+        ext = safe_ext(base)
+        stem = base[: -len(Path(base).suffix)] if Path(base).suffix else base
+        stem = stem[:120] or "file"
+        return f"{stem}{ext}"
+    return filehash_from_url(url) or "file"
+
+
+def post_dir_name(post: dict) -> str:
+    """`<YYYY-MM-DD>_<post_id>_<title40>` matching gallery-dl's layout (date prefix
+    omitted when published_at is missing/unparseable; title is empty for platforms
+    with no title field). Accepts both ISO and trailing-`Z` published_at."""
+    post_id = str(post.get("id") or "")
+    attrs = post.get("attributes") or {}
+    title = attrs.get("title")
+    title40 = (title if isinstance(title, str) else "")[:_TITLE_MAX]
+    published = attrs.get("published_at")
+    date_prefix = None
+    if isinstance(published, str) and published:
+        s = published.strip()
+        if s.endswith("Z"):
+            s = s[:-1] + "+00:00"
+        try:
+            date_prefix = f"{datetime.fromisoformat(s):%Y-%m-%d}"
+        except ValueError:
+            date_prefix = None
+    raw = f"{date_prefix}_{post_id}_{title40}" if date_prefix else f"{post_id}_{title40}"
+    return sanitize_segment(raw)
+
+
+# -- per-item download outcomes (shared dataclasses) -----------------------
+
+@dataclass
+class MediaOutcome:
+    """Per-media result of a download_post pass. status ∈ downloaded /
+    skipped_seen / skipped_disk / quarantined / error. `path` is the on-disk file
+    (downloaded / skipped_disk), the quarantine dest (quarantined), or None;
+    `error` is the failure/validation reason (error/quarantined) else None."""
+
+    media: object
+    status: str
+    path: Path | None
+    error: str | None
+
+
+@dataclass
+class PostRecordOutcome:
+    """Result of write_post_record — mirrors the MediaOutcome contract so the core
+    reports per-post handling. `path` is the _post.json sidecar (None when the post
+    had no id); the rest is the captured body's shape for the run log."""
+
+    path: Path | None
+    post_type: str | None
+    title: str | None
+    body_chars: int
+
+
+# -- base downloader (shared fetch/validate plumbing) ----------------------
+
+class BaseNativeDownloader:
+    """Shared download plumbing for native-platform downloaders: the streaming
+    GET (transient-retry + Range-resume) and file validation/quarantine. Platform
+    downloaders subclass this and implement `download_post` / `write_post_record`
+    / the per-media sidecar (and any platform-specific fetch, e.g. Patreon's
+    Mux/yt-dlp video branch). PURE: no DB; the seen-skip is an injected predicate.
+    """
+
+    def __init__(
+        self,
+        images_root: Path,
+        cookies_path: str | None = None,
+        *,
+        platform: str,
+        validate: bool = True,
+        rate_limit: float = 0.0,
+        session: requests.Session | None = None,
+    ):
+        self.images_root = Path(images_root)
+        self.cookies_path = str(cookies_path) if cookies_path else None
+        self.platform = platform
+        self._validate = validate
+        self._rate_limit = rate_limit or 0.0
+        self.session = session if session is not None else make_session(cookies_path)
+
+    # -- download seams ----------------------------------------------------
+
+    def _fetch_get(self, url: str, dest: Path) -> Path:
+        """Stream `url` to a .part then atomic-rename to `dest`."""
+        part = dest.with_name(dest.name + ".part")
+        try:
+            self._fetch_to_file(url, part)
+        except Exception:
+            with contextlib.suppress(OSError):
+                part.unlink()
+            raise
+        os.replace(part, dest)
+        return dest
+
+    def _fetch_to_file(self, url: str, dest: Path) -> None:
+        """Stream a URL to `dest`, retrying TRANSIENT failures (transport blips,
+        429 honoring Retry-After, 5xx) with backoff + resume-from-disk (Range);
+        failing fast on permanent 4xx (404/403). Resume: a retry with bytes on
+        disk asks `Range: bytes=<have>-`; 206 → append, 200 → restart clean, 416 →
+        already complete. The caller stages into a `.part` so a non-range server
+        never corrupts the output."""
+        attempt = 0
+        while True:
+            have = dest.stat().st_size if dest.exists() else 0
+            headers = {"Range": f"bytes={have}-"} if have > 0 else None
+            try:
+                resp = self.session.get(
+                    url, stream=True, timeout=_TIMEOUT_SECONDS, headers=headers,
+                )
+                if (resp.status_code == 429 or resp.status_code >= 500) \
+                        and attempt < _MAX_MEDIA_RETRIES:
+                    attempt += 1
+                    delay = retry_after_seconds(resp, attempt)
+                    log.warning(
+                        "%s media transient HTTP %d (%s) — backing off %.1fs "
+                        "(retry %d/%d)",
+                        self.platform, resp.status_code, url, delay, attempt,
+                        _MAX_MEDIA_RETRIES,
+                    )
+                    time.sleep(delay)
+                    continue
+                if have > 0 and resp.status_code == 416:
+                    return
+                resp.raise_for_status()
+                mode = "ab" if (have > 0 and resp.status_code == 206) else "wb"
+                with open(dest, mode) as fh:
+                    for chunk in resp.iter_content(chunk_size=_CHUNK):
+                        if chunk:
+                            fh.write(chunk)
+                return
+            except _TRANSIENT_TRANSPORT_EXC as exc:
+                if attempt >= _MAX_MEDIA_RETRIES:
+                    raise
+                attempt += 1
+                delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
+                log.warning(
+                    "%s media transport error (%s) — backing off %.1fs "
+                    "(retry %d/%d): %s",
+                    self.platform, url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
+                )
+                time.sleep(delay)
+
+    # -- validation --------------------------------------------------------
+
+    def _validate_path(
+        self, path: Path, artist_slug: str, source_url: str | None = None
+    ) -> tuple[str | None, Path | None]:
+        """Validate a freshly-written file; quarantine if bad (shared
+        file_validator move + provenance sidecar). Returns (reason,
+        quarantine_dest) when quarantined, else (None, None). Logs the quarantine
+        so a corrupt file is visible in the worker logs, not just counted (#899
+        L2)."""
+        if not self._validate or not is_validatable(path):
+            return None, None
+        try:
+            result = validate_file(path)
+        except Exception as exc:
+            log.warning("Validator raised on %s: %s", path, exc)
+            return None, None
+        if result.ok:
+            return None, None
+        dest = quarantine_file(
+            self.images_root, path, artist_slug, self.platform,
+            url=source_url, result=result,
+        )
+        reason = result.reason or "validation failed"
+        log.warning(
+            "%s quarantined %s (%s) — %s",
+            self.platform, dest or path, artist_slug, reason,
+        )
+        return reason, (dest or path)
+
+    # -- sidecar (per-media, minimal) --------------------------------------
+
+    def _write_minimal_sidecar(
+        self, post: dict, media_path: Path, *, source_url: str | None = None
+    ) -> Path:
+        """Post-first per-media sidecar (#856): image identity ONLY
+        (category/id/source_url). The post body/links live solely in _post.json."""
+        data: dict = {"category": self.platform, "id": str(post.get("id") or "")}
+        if source_url:
+            data["source_url"] = source_url
+        sidecar_path = media_path.with_suffix(".json")
+        sidecar_path.write_text(json.dumps(data, indent=2))
+        return sidecar_path

backend/app/services/patreon_client.py

@@ -26,9 +26,7 @@ FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
 
 from __future__ import annotations
 
-import http.cookiejar
 import logging
-import os
 import re
 import time
 from collections.abc import Iterator
@@ -39,93 +37,51 @@ from urllib.parse import parse_qs, urlsplit
 
 import requests
 
-from ..utils.paths import safe_ext
+from ..utils.paths import filehash_from_url
+from ..utils.prosemirror import post_body_html
+from .native_ingest_common import (
+    _MAX_429_RETRIES,
+    NativeAuthError,
+    NativeDriftError,
+    NativeIngestError,
+    basename_from_url,
+    make_session,
+    retry_after_seconds,
+)
 
 log = logging.getLogger(__name__)
 
 _POSTS_URL = "https://www.patreon.com/api/posts"
-_USER_AGENT = (
-    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
-    "(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
-)
 _TIMEOUT_SECONDS = 30.0
 
-# 429 backoff (plan #703): ride out a transient API rate-limit instead of
-# failing the whole walk (which would stamp RATE_LIMITED → platform-wide
-# cooldown → every Patreon source dark). Honor the server's `Retry-After`;
-# otherwise exponential base·2^(n-1), capped. Only after the retries are
-# exhausted does the 429 propagate as terminal RATE_LIMITED.
-_MAX_429_RETRIES = 3
-_BACKOFF_BASE_SECONDS = 2.0
-_BACKOFF_CAP_SECONDS = 30.0
-
-
-def _retry_after_seconds(
-    resp: requests.Response,
-    attempt: int,
-    *,
-    base: float = _BACKOFF_BASE_SECONDS,
-    cap: float = _BACKOFF_CAP_SECONDS,
-) -> float:
-    """Backoff delay for a 429: the `Retry-After` seconds header if present and
-    numeric, else exponential `base·2^(attempt-1)`, both capped. (HTTP-date form
-    of Retry-After is rare here and falls through to exponential.)"""
-    header = resp.headers.get("Retry-After")
-    if header:
-        try:
-            return min(float(header), cap)
-        except (TypeError, ValueError):
-            pass
-    return min(base * (2 ** max(0, attempt - 1)), cap)
-
 # JSON:API request contract (observed from real traffic — see module plan).
 _INCLUDE = (
     "campaign,access_rules,attachments,attachments_media,audio,images,media,"
     "native_video_insights,user,user_defined_tags,ti_checks"
 )
 _FIELDS_POST = (
-    "content,post_file,image,post_type,published_at,title,url,patreon_url,"
-    "current_user_can_view"
+    # `content` is the legacy flat-HTML body (Patreon now returns it null);
+    # `content_json_string` is the current ProseMirror-doc body. Request BOTH —
+    # post_body_html() prefers content (old posts) and falls back to converting
+    # content_json_string (current posts). #842.
+    "content,content_json_string,post_file,image,post_type,published_at,title,"
+    "url,patreon_url,current_user_can_view"
 )
 _FIELDS_MEDIA = "id,image_urls,download_url,metadata,file_name"
 _FIELDS_CAMPAIGN = "name,url"
 
-# A CDN download URL embeds a 32-char hex (MD5) path segment; that segment is
-# Patreon's stable per-file identity and is what we dedup + ledger against.
-# Same role gallery-dl's _filehash plays. Match the FIRST 32-hex run anywhere
-# in the URL (path or query); real Patreon CDN URLs carry exactly one.
-_FILEHASH_RE = re.compile(r"([0-9a-fA-F]{32})")
-
 # Inline post `content` is HTML; images are emitted as <img ... src="...">.
 # Pull every src; downstream dedup collapses any that duplicate a gallery item
 # by filehash. Tolerant of attribute ordering and single/double quotes.
 _CONTENT_IMG_RE = re.compile(r"<img\b[^>]*?\bsrc=[\"']([^\"']+)[\"']", re.IGNORECASE)
 
 
-class PatreonAPIError(Exception):
-    """Base for native Patreon client failures.
-
-    `status_code` carries the HTTP status when the failure was an HTTP response
-    (None for transport-level / parse failures), so the ingester can map it to a
-    DownloadResult.error_type (429 → rate_limited, 404 → not_found, …).
-    `retry_after` carries the server's `Retry-After` seconds on a terminal 429, so
-    the platform cooldown can match the server's hint instead of a flat default
-    (plan #708 B1).
-    """
-
-    def __init__(
-        self,
-        message: str,
-        *,
-        status_code: int | None = None,
-        retry_after: float | None = None,
-    ):
-        super().__init__(message)
-        self.status_code = status_code
-        self.retry_after = retry_after
+class PatreonAPIError(NativeIngestError):
+    """Base for native Patreon client failures. status_code / retry_after are
+    inherited from NativeIngestError (HTTP status; 429 Retry-After hint)."""
 
 
-class PatreonAuthError(PatreonAPIError):
+class PatreonAuthError(PatreonAPIError, NativeAuthError):
     """Authentication / authorization failure — missing or expired session
     cookies, an insufficient pledge tier, or an HTML login/challenge page served
     where JSON was expected. DISTINCT from drift: the fix is rotating the
@@ -133,7 +89,7 @@ class PatreonAuthError(PatreonAPIError):
     """
 
 
-class PatreonDriftError(PatreonAPIError):
+class PatreonDriftError(PatreonAPIError, NativeDriftError):
     """A JSON response did not match the JSON:API shape we depend on.
 
     Raised for: a missing top-level `data` list, `data` not a list, or a media
@@ -169,48 +125,10 @@ class MediaItem:
     post_id: str
 
 
-def _load_session(cookies_path: str | Path | None) -> requests.Session:
-    session = requests.Session()
-    session.headers.update(
-        {
-            "User-Agent": _USER_AGENT,
-            "Accept": "application/vnd.api+json",
-        }
-    )
-    if cookies_path and os.path.isfile(str(cookies_path)):
-        try:
-            jar = http.cookiejar.MozillaCookieJar(str(cookies_path))
-            jar.load(ignore_discard=True, ignore_expires=True)
-            session.cookies = jar  # type: ignore[assignment]
-        except (OSError, http.cookiejar.LoadError) as exc:
-            log.warning("Could not load Patreon cookies from %s: %s", cookies_path, exc)
-    return session
-
-
 def _filehash(url: str) -> str | None:
-    if not url:
-        return None
-    match = _FILEHASH_RE.search(url)
-    return match.group(1).lower() if match else None
-
-
-def _basename_from_url(url: str) -> str:
-    """Derive a sane filename from a URL when the media has no file_name.
-
-    Strips query/fragment, takes the path basename, and drops a junk
-    extension (the importer._safe_ext gotcha) so we never write base64 noise
-    as a name. Falls back to the filehash, then to "file".
-    """
-    path = urlsplit(url).path
-    base = os.path.basename(path)
-    if base:
-        ext = safe_ext(base)
-        stem = base[: -len(Path(base).suffix)] if Path(base).suffix else base
-        # Keep the stem bounded; URL-encoded stems can be enormous.
-        stem = stem[:120] or "file"
-        return f"{stem}{ext}"
-    fh = _filehash(url)
-    return fh or "file"
+    # Delegate to the shared extractor (utils.paths) so capture-time persistence
+    # and render-time inline-image matching use the EXACT same identity.
+    return filehash_from_url(url)
 
 
 def parse_cursor_from_url(url: str | None) -> str | None:
@@ -240,7 +158,7 @@ class PatreonClient:
         max_retries: int = _MAX_429_RETRIES,
     ):
         self.cookies_path = str(cookies_path) if cookies_path else None
-        self._session = _load_session(cookies_path)
+        self._session = make_session(cookies_path, accept="application/vnd.api+json")
         # Politeness: seconds to sleep before each /api/posts page fetch (paces
         # the rate-limited API endpoint). 0 = no pacing. plan #703.
         self._request_sleep = request_sleep or 0.0
@@ -285,7 +203,7 @@ class PatreonClient:
             # through to the terminal RATE_LIMITED raise below.
             if resp.status_code == 429 and attempt < self._max_retries:
                 attempt += 1
-                delay = _retry_after_seconds(resp, attempt)
+                delay = retry_after_seconds(resp, attempt)
                 log.warning(
                     "Patreon 429 (campaign_id=%s) — backing off %.1fs (retry %d/%d)",
                     campaign_id, delay, attempt, self._max_retries,
@@ -418,7 +336,7 @@ class PatreonClient:
         # uses. A genuine schema change shows up as no URL (above) or a media id
         # absent from `included` (caller), not a missing name.
         file_name = attrs.get("file_name")
-        filename = file_name if isinstance(file_name, str) and file_name else _basename_from_url(url)
+        filename = file_name if isinstance(file_name, str) and file_name else basename_from_url(url)
         return MediaItem(
             url=url,
             filename=filename,
@@ -464,7 +382,7 @@ class PatreonClient:
                 items.append(
                     MediaItem(
                         url=large_url,
-                        filename=_basename_from_url(large_url),
+                        filename=basename_from_url(large_url),
                         kind="image_large",
                         filehash=_filehash(large_url),
                         post_id=post_id,
@@ -483,7 +401,7 @@ class PatreonClient:
                 filename = (
                     pf_name
                     if isinstance(pf_name, str) and pf_name
-                    else _basename_from_url(pf_url)
+                    else basename_from_url(pf_url)
                 )
                 items.append(
                     MediaItem(
@@ -505,7 +423,7 @@ class PatreonClient:
                 items.append(
                     MediaItem(
                         url=src,
-                        filename=_basename_from_url(src),
+                        filename=basename_from_url(src),
                         kind="content",
                         filehash=_filehash(src),
                         post_id=post_id,
@@ -526,6 +444,36 @@ class PatreonClient:
             "date": published if isinstance(published, str) else None,
         }
 
+    @staticmethod
+    def post_is_gated(post: dict) -> bool:
+        """True when the authenticated account CANNOT view this post's content
+        (#874). Patreon serves only BLURRED locked-preview thumbnails for
+        paywalled / insufficient-tier posts, and `current_user_can_view` on the
+        post attributes is the access flag (it IS in `_FIELDS_POST`). The walk
+        skips a gated post ENTIRELY — no media, no post-record stub — so those
+        unusable previews never get downloaded.
+
+        Gate ONLY on an explicit `current_user_can_view == False`. A missing /
+        None flag (older posts, a sparse fieldset, or API drift) is treated as
+        viewable, so we never over-filter accessible posts on an absent field.
+        Part of the client contract the core consumes via getattr — an optional
+        seam, so stub clients / not-yet-migrated platforms simply never gate."""
+        attrs = post.get("attributes") or {}
+        return attrs.get("current_user_can_view") is False
+
+    @staticmethod
+    def post_record_key(post: dict) -> tuple[str, str] | None:
+        """`(ledger_key, post_id)` for a media-less post's seen-ledger entry, or
+        None when the post has no id. The synthetic `post:<id>` key lets the
+        generic core gate text-post capture through the SAME seen-ledger as media
+        — so a text post's body is detail-fetched + recorded ONCE, not re-fetched
+        every tick. Part of the client contract the core uses via getattr."""
+        pid = post.get("id")
+        pid = str(pid) if pid is not None else ""
+        if not pid:
+            return None
+        return (f"post:{pid}", pid)
+
     # -- iteration ---------------------------------------------------------
 
     def iter_posts(
@@ -562,6 +510,57 @@ class PatreonClient:
                 return
             current_cursor = next_cursor
 
+    # -- detail (full body enrichment) -------------------------------------
+
+    def fetch_post_detail_content(self, post_id: str) -> str | None:
+        """Best-effort fetch of a post's body (as HTML) from the per-post DETAIL
+        endpoint (`/api/posts/{id}`).
+
+        Patreon deprecated the flat `content` HTML field — it returns null on the
+        feed AND the detail endpoint, for every post type. The real body now lives
+        in `content_json_string` (a ProseMirror doc), and is only returned under
+        the DEFAULT post fieldset: a sparse `fields[post]=content` request OMITS
+        it (confirmed against the live API 2026-06-15). So we request the default
+        fieldset (no `fields[post]`) and resolve the body via `post_body_html`
+        (content → else convert content_json_string). The downloader calls this to
+        enrich a post whose feed body was empty before writing the sidecar.
+
+        Best-effort BY DESIGN: a body we can't fetch must never fail the walk.
+        Every failure path returns None rather than raising — distinct from the
+        loud drift/auth raises on the feed path, which gate real downloads.
+        """
+        if not post_id:
+            return None
+        if self._request_sleep > 0:
+            time.sleep(self._request_sleep)  # pace the API endpoint (plan #703)
+        # No `fields[post]` — the default fieldset is the only shape that returns
+        # content_json_string (the body). A sparse fieldset nulls it out.
+        try:
+            resp = self._session.get(f"{_POSTS_URL}/{post_id}", timeout=_TIMEOUT_SECONDS)
+        except requests.RequestException as exc:
+            log.warning("Patreon post-detail fetch failed (post %s): %s", post_id, exc)
+            return None
+        if resp.status_code != 200:
+            log.warning(
+                "Patreon post-detail fetch HTTP %s (post %s)", resp.status_code, post_id
+            )
+            return None
+        try:
+            payload = resp.json()
+        except ValueError:
+            return None
+        data = payload.get("data") if isinstance(payload, dict) else None
+        attrs = data.get("attributes") if isinstance(data, dict) else None
+        body = post_body_html(attrs)
+        if body and body.strip():
+            log.info("post-detail: fetched %d chars (post %s)", len(body), post_id)
+            return body
+        ptype = attrs.get("post_type") if isinstance(attrs, dict) else None
+        log.info(
+            "post-detail: no body (post %s, post_type=%s)", post_id, ptype,
+        )
+        return None
+
     # -- verify ------------------------------------------------------------
 
     def verify_auth(self, campaign_id: str) -> tuple[bool | None, str]:

backend/app/services/patreon_downloader.py

@@ -27,45 +27,33 @@ FC runs on a plain-HTTP homelab; nothing here uses a secure-context Web API.
 
 from __future__ import annotations
 
-import contextlib
 import json
 import logging
 import os
 import subprocess
 import time
 from collections.abc import Callable
-from dataclasses import dataclass
-from datetime import datetime
 from pathlib import Path
 from urllib.parse import urlsplit
 
 import requests
 
-from .file_validator import is_validatable, quarantine_file, validate_file
-from .patreon_client import (
+from ..utils.prosemirror import post_body_html
+from .native_ingest_common import (
     _BACKOFF_CAP_SECONDS,
-    _load_session,
-    _retry_after_seconds,
+    _MAX_MEDIA_RETRIES,
+    BaseNativeDownloader,
+    MediaOutcome,
+    PostRecordOutcome,
+    post_dir_name,
+    sanitize_segment,
 )
 
 log = logging.getLogger(__name__)
 
-_TITLE_MAX = 40
+# yt-dlp subprocess wall-clock per attempt (video only; the shared HTTP fetch
+# budgets live in BaseNativeDownloader).
 _TIMEOUT_SECONDS = 120.0
-_CHUNK = 1 << 16
-# Retry a media GET that hits a TRANSIENT failure within the same pass (plan
-# #705 #8): a transport blip (connection reset / timeout / truncated stream), a
-# 429, or a 5xx. PERMANENT failures (404 gone, 403 forbidden) fail fast straight
-# to the error/dead-letter path — no point re-fetching them. Keeps a momentary
-# network hiccup from becoming a per-item error that waits for the next walk.
-_MAX_MEDIA_RETRIES = 3
-# requests transport errors worth retrying (vs. an HTTPError, which is a real
-# server response and is classified by status code).
-_TRANSIENT_TRANSPORT_EXC = (
-    requests.ConnectionError,
-    requests.Timeout,
-    requests.exceptions.ChunkedEncodingError,
-)
 
 # Referer/Origin yt-dlp must send for Mux-hosted Patreon video. Mux's JWT
 # playback policy checks Referer/Origin on every request, so yt-dlp must send
@@ -77,26 +65,6 @@ _VIDEO_HEADERS = {
     "Origin": "https://www.patreon.com",
 }
 
-# Characters Windows/gallery-dl path-restrict forbids, plus path separators.
-_FORBIDDEN = set('<>:"/\\|?*')
-
-
-def _sanitize(name: str) -> str:
-    """Make `name` safe for a single filesystem path segment.
-
-    Replaces path separators, the Windows-forbidden set <>:"/\\|?* and control
-    characters with `_`, then strips trailing dots/spaces (gallery-dl
-    path-restrict behavior). Never returns empty (falls back to "_").
-    """
-    out = []
-    for ch in name:
-        if ch in _FORBIDDEN or ord(ch) < 32:
-            out.append("_")
-        else:
-            out.append(ch)
-    cleaned = "".join(out).rstrip(". ")
-    return cleaned or "_"
-
 
 def _is_video_url(url: str) -> bool:
     parts = urlsplit(url)
@@ -105,59 +73,12 @@ def _is_video_url(url: str) -> bool:
     return parts.path.lower().endswith(".m3u8")
 
 
-def _post_dir_name(post: dict) -> str:
-    """Build the post directory name matching gallery-dl's layout."""
-    post_id = str(post.get("id") or "")
-    attrs = post.get("attributes") or {}
-    title = attrs.get("title")
-    title = title if isinstance(title, str) else ""
-    title40 = title[:_TITLE_MAX]
-
-    published = attrs.get("published_at")
-    date_prefix = None
-    if isinstance(published, str) and published:
-        s = published.strip()
-        if s.endswith("Z"):
-            s = s[:-1] + "+00:00"
-        try:
-            dt = datetime.fromisoformat(s)
-        except ValueError:
-            dt = None
-        if dt is not None:
-            date_prefix = f"{dt:%Y-%m-%d}"
-
-    if date_prefix:
-        raw = f"{date_prefix}_{post_id}_{title40}"
-    else:
-        raw = f"{post_id}_{title40}"
-    return _sanitize(raw)
-
-
-@dataclass
-class MediaOutcome:
-    """Per-media result of a download_post pass.
-
-    status is one of: "downloaded", "skipped_seen", "skipped_disk",
-    "quarantined", "error". `path` is the final on-disk path for "downloaded"
-    (the actual yt-dlp output for video), the path that already existed for
-    "skipped_disk", or the _quarantine destination for "quarantined"; None for
-    "skipped_seen" and (usually) "error". `error` carries the failure/validation
-    reason for "error"/"quarantined", else None.
-    """
-
-    media: object  # MediaItem (avoid importing the name for a bare annotation)
-    status: str
-    path: Path | None
-    error: str | None
-
-
-class PatreonDownloader:
-    """Download resolved Patreon media to gallery-dl's on-disk layout.
-
-    PURE: no DB. The HTTP session and the yt-dlp invocation are injectable seams
-    so tests run without network or a real subprocess:
-      - pass `session=` to stub `session.get`, or monkeypatch `_fetch_to_file`.
-      - monkeypatch `_run_ytdlp` to avoid spawning yt-dlp.
+class PatreonDownloader(BaseNativeDownloader):
+    """Download resolved Patreon media to gallery-dl's on-disk layout. Subclasses
+    BaseNativeDownloader for the shared streaming GET (transient-retry +
+    Range-resume) and validation/quarantine; adds the Mux/HLS yt-dlp video branch
+    and the detail-fetch body enrichment. PURE: no DB. `_run_ytdlp` is
+    monkeypatchable and the HTTP session is the injectable `session=` seam.
     """
 
     def __init__(
@@ -168,18 +89,18 @@ class PatreonDownloader:
         validate: bool = True,
         rate_limit: float = 0.0,
         session: requests.Session | None = None,
+        content_fetcher: Callable[[str], str | None] | None = None,
     ):
-        self.images_root = Path(images_root)
-        self.cookies_path = str(cookies_path) if cookies_path else None
-        self._validate = validate
-        # Politeness: seconds to sleep before each actual media download (paces
-        # the CDN; honors ImportSettings.download_rate_limit_seconds, the same
-        # value gallery-dl used as its between-downloads `sleep`). 0 = no pacing.
-        # Applied only to real downloads, not to seen/disk skips. plan #703.
-        self._rate_limit = rate_limit or 0.0
-        # Build a cookie-loaded session the same way patreon_client does, so the
-        # CDN GETs carry the creator's auth.
-        self.session = session if session is not None else _load_session(cookies_path)
+        super().__init__(
+            images_root, cookies_path, platform="patreon",
+            validate=validate, rate_limit=rate_limit, session=session,
+        )
+        # Best-effort enrichment seam: (post_id) -> full HTML body, or None. The
+        # feed endpoint often omits `content`; the adapter wires this to
+        # PatreonClient.fetch_post_detail_content so the sidecar captures the
+        # real body (formatting + inline <img> + external <a href> links).
+        # None in unit tests / when enrichment isn't wanted.
+        self._content_fetcher = content_fetcher
 
     # -- public ------------------------------------------------------------
 
@@ -191,6 +112,7 @@ class PatreonDownloader:
         *,
         is_seen: Callable[[object], bool] = lambda m: False,
         should_stop: Callable[[], bool] = lambda: False,
+        recapture: bool = False,
     ) -> list[MediaOutcome]:
         """Download every media item of one post; return per-item outcomes.
 
@@ -205,8 +127,13 @@ class PatreonDownloader:
         re-checks the budget between posts), so we honour the deadline mid-post
         and return the items done so far — the rest re-fetch next chunk (they
         were never marked seen). Bounds chunk overrun to one media download.
+
+        `recapture` (#830): don't re-download already-present media, but DO surface
+        on-disk media as `skipped_disk` (with its path) even when the seen-ledger
+        would tier-1 skip it — so the engine can backfill source_filehash for
+        inline-image localization. Genuinely-missing seen media is NOT refetched.
         """
-        post_dir = self.images_root / artist_slug / "patreon" / _post_dir_name(post)
+        post_dir = self.images_root / artist_slug / "patreon" / post_dir_name(post)
         outcomes: list[MediaOutcome] = []
 
         for i, media in enumerate(media_items, start=1):
@@ -214,7 +141,10 @@ class PatreonDownloader:
                 break
             try:
                 outcomes.append(
-                    self._download_one(post, media, post_dir, artist_slug, i, is_seen)
+                    self._download_one(
+                        post, media, post_dir, artist_slug, i, is_seen,
+                        recapture=recapture,
+                    )
                 )
             except Exception as exc:  # resilient: isolate one item's failure
                 log.warning(
@@ -236,13 +166,19 @@ class PatreonDownloader:
         artist_slug: str,
         index: int,
         is_seen: Callable[[object], bool],
+        *,
+        recapture: bool = False,
     ) -> MediaOutcome:
-        # tier-1: seen ledger (injected; no DB here).
-        if is_seen(media):
+        # tier-1: seen ledger (injected; no DB here). In recapture mode we DON'T
+        # short-circuit here — we fall through to the disk check so an on-disk
+        # seen file is surfaced as skipped_disk (with its path) for source_filehash
+        # backfill; a seen file that's NOT on disk is left alone (not refetched).
+        seen = is_seen(media)
+        if seen and not recapture:
             return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
 
         nn = f"{index:02d}"
-        final_name = _sanitize(f"{nn}_{media.filename}")
+        final_name = sanitize_segment(f"{nn}_{media.filename}")
         media_path = post_dir / final_name
 
         # tier-2: already on disk.
@@ -258,6 +194,12 @@ class PatreonDownloader:
                     media=media, status="skipped_disk", path=existing, error=None
                 )
 
+        # recapture: a seen item that isn't on disk is NOT re-downloaded (that's
+        # recovery's job) — recapture only re-grabs post text + localizes existing
+        # files. Returns skipped_seen so the run-of-seen / counts stay consistent.
+        if seen:
+            return MediaOutcome(media=media, status="skipped_seen", path=None, error=None)
+
         post_dir.mkdir(parents=True, exist_ok=True)
 
         # Pace real downloads only (the skips above already returned). plan #703.
@@ -287,90 +229,12 @@ class PatreonDownloader:
                     path=quarantine_dest, error=reason,
                 )
 
-        self._write_sidecar(post, out_path)
+        self._write_sidecar(post, out_path, source_url=media.url)
         return MediaOutcome(media=media, status="downloaded", path=out_path, error=None)
 
-    # -- download seams ----------------------------------------------------
-
-    def _fetch_get(self, url: str, dest: Path) -> Path:
-        """Stream `url` to a .part file then atomic-rename to `dest`.
-
-        Thin wrapper over `_fetch_to_file` so tests can stub either the whole
-        GET path (`_fetch_to_file`) or just `session.get`.
-        """
-        part = dest.with_name(dest.name + ".part")
-        try:
-            self._fetch_to_file(url, part)
-        except Exception:
-            with contextlib.suppress(OSError):
-                part.unlink()
-            raise
-        os.replace(part, dest)
-        return dest
-
-    def _fetch_to_file(self, url: str, dest: Path) -> None:
-        """Stream a non-video URL to `dest` via the (stubbable) session, retrying
-        TRANSIENT failures within the same pass (plan #705 #8) and RESUMING from
-        the bytes already on disk via a Range request when a retry follows a
-        mid-download cut (plan #708 B5).
-
-        Retried (backoff): transport blips (connection reset / timeout /
-        truncated stream — incl. mid-download), HTTP 429 (honoring Retry-After),
-        and 5xx. Failed fast (no retry → HTTPError → per-item error → dead-letter
-        path): 4xx other than 429 (404 gone, 403 forbidden) — re-fetching a
-        permanent failure is pointless.
-
-        Resume: on a retry, if bytes already landed in `dest`, ask for the rest
-        with `Range: bytes=<have>-`. A 206 means the server honored it → append; a
-        200 means it ignored it (served the whole file) → start clean. The caller
-        (_fetch_get) stages into a `.part`, so a non-range server never corrupts
-        the output — the worst case is re-downloading from zero, as before.
-        """
-        attempt = 0
-        while True:
-            have = dest.stat().st_size if dest.exists() else 0
-            headers = {"Range": f"bytes={have}-"} if have > 0 else None
-            try:
-                resp = self.session.get(
-                    url, stream=True, timeout=_TIMEOUT_SECONDS, headers=headers,
-                )
-                if (resp.status_code == 429 or resp.status_code >= 500) \
-                        and attempt < _MAX_MEDIA_RETRIES:
-                    attempt += 1
-                    delay = _retry_after_seconds(resp, attempt)
-                    log.warning(
-                        "Patreon media transient HTTP %d (%s) — backing off "
-                        "%.1fs (retry %d/%d)",
-                        resp.status_code, url, delay, attempt, _MAX_MEDIA_RETRIES,
-                    )
-                    time.sleep(delay)
-                    continue
-                # A Range that starts at/past EOF (we already have the whole file)
-                # comes back 416 — the bytes we kept ARE the file.
-                if have > 0 and resp.status_code == 416:
-                    return
-                # 2xx → ok; 4xx-non-429 (or an exhausted 429/5xx) → HTTPError
-                # (permanent for this pass) → not caught below → per-item error.
-                resp.raise_for_status()
-                # 206 → server honored the Range; append after the kept bytes.
-                # Anything else (200) → it served the whole file → start clean.
-                mode = "ab" if (have > 0 and resp.status_code == 206) else "wb"
-                with open(dest, mode) as fh:
-                    for chunk in resp.iter_content(chunk_size=_CHUNK):
-                        if chunk:
-                            fh.write(chunk)
-                return
-            except _TRANSIENT_TRANSPORT_EXC as exc:
-                if attempt >= _MAX_MEDIA_RETRIES:
-                    raise  # exhausted → terminal error outcome
-                attempt += 1
-                delay = min(2.0 * (2 ** (attempt - 1)), _BACKOFF_CAP_SECONDS)
-                log.warning(
-                    "Patreon media transport error (%s) — backing off %.1fs "
-                    "(retry %d/%d): %s",
-                    url, delay, attempt, _MAX_MEDIA_RETRIES, exc,
-                )
-                time.sleep(delay)
+    # -- video (Mux/HLS via yt-dlp) ----------------------------------------
+    # The plain-GET streaming path (_fetch_get / _fetch_to_file) and
+    # _validate_path are inherited from BaseNativeDownloader.
 
     def _run_ytdlp(self, url: str, dest: Path, headers: dict) -> Path | None:
         """Invoke yt-dlp to fetch a Mux/HLS stream to (around) `dest`.
@@ -452,49 +316,61 @@ class PatreonDownloader:
                 return cand
         return None
 
-    # -- validation --------------------------------------------------------
-
-    def _validate_path(
-        self, path: Path, artist_slug: str, source_url: str | None = None
-    ) -> tuple[str | None, Path | None]:
-        """Validate a freshly-written file; quarantine if bad.
-
-        Uses the shared `file_validator.quarantine_file` — same move + provenance
-        sidecar gallery-dl writes (the native path used to skip the sidecar; that
-        parity gap is closed here). Returns `(reason, quarantine_dest)` when
-        quarantined (dest is the original path if the move itself failed), else
-        `(None, None)` (ok / not validatable / disabled). plan #704: the dest is
-        surfaced so the run reports a real quarantined-paths list.
-        """
-        if not self._validate or not is_validatable(path):
-            return None, None
-        try:
-            result = validate_file(path)
-        except Exception as exc:
-            log.warning("Validator raised on %s: %s", path, exc)
-            return None, None
-        if result.ok:
-            return None, None
-        dest = quarantine_file(
-            self.images_root, path, artist_slug, "patreon",
-            url=source_url, result=result,
-        )
-        return (result.reason or "validation failed"), (dest or path)
-
     # -- sidecar -----------------------------------------------------------
 
-    def _write_sidecar(self, post: dict, media_path: Path) -> Path:
-        """Write the importer-consumed sidecar next to `media_path`.
+    def _write_sidecar(
+        self, post: dict, media_path: Path, *, source_url: str | None = None
+    ) -> Path:
+        """Write the per-media sidecar next to `media_path` — post-first (#856).
 
-        Patreon uses base-default sidecar keys (parse_sidecar maps
-        category->platform, id->external_post_id, title->post_title,
-        content->description, published_at->post_date, url->post_url). Patreon
-        registers no derive_post_url, so `url` is trusted as the permalink — we
-        pass the post's attributes.url.
+        On the native ingester the POST-RECORD (`write_post_record` → `_post.json`)
+        is the sole writer of the post body/links/metadata, captured once per post
+        BEFORE its media in the walk. So the per-media sidecar carries ONLY
+        image-specific identity: `category` (platform) + `id` (external_post_id, to
+        link provenance to the right Post) + this file's `source_url` (its CDN URL,
+        #830 Phase 2 — the importer persists its filehash so the body's inline
+        `<img src>` remaps to the local copy at render time). No body: writing it
+        next to every image duplicated the post body N+1× and risked divergence
+        (milestone #67). The importer skips post fields for these (post_first).
         """
+        return self._write_sidecar_data(
+            post, media_path.with_suffix(".json"), source_url=source_url,
+            minimal=True,
+        )
+
+    def _write_sidecar_data(
+        self, post: dict, sidecar_path: Path, *, source_url: str | None = None,
+        minimal: bool = False,
+    ) -> Path:
+        """Serialize the post's metadata to `sidecar_path`. The post-only record
+        (`write_post_record`) writes the FULL post (body/title/date/url); the
+        per-media sidecar (`_write_sidecar`, minimal=True) writes only image
+        identity (category/id/source_url) — post-first (#856). `source_url` is set
+        only for the per-media sidecar — a media-less post has no source file."""
+        if minimal:
+            data = {"category": "patreon", "id": str(post.get("id") or "")}
+            if source_url:
+                data["source_url"] = source_url
+            sidecar_path.write_text(json.dumps(data, indent=2))
+            return sidecar_path
         attrs = post.get("attributes") or {}
         title = attrs.get("title")
-        content = attrs.get("content")
+        # Resolve the body HTML from the feed attrs: legacy flat `content`, else
+        # convert the current `content_json_string` ProseMirror doc (#842).
+        content = post_body_html(attrs)
+        # The feed/list endpoint frequently returns an empty body; the full body
+        # only comes from the per-post detail endpoint. Enrich on first write for
+        # this post and MEMOIZE the RESOLVED HTML by mutating the shared `post`
+        # dict — so a multi-image post fetches detail at most once, the post-record
+        # body-length read reuses it, and a fully-seen post (no fresh download → no
+        # sidecar write) never pays the extra GET.
+        if (not content or not content.strip()) and self._content_fetcher:
+            fetched = self._content_fetcher(str(post.get("id") or ""))
+            if fetched:
+                content = fetched
+        if isinstance(content, str) and content.strip():
+            attrs["content"] = content
+            post["attributes"] = attrs
         published = attrs.get("published_at")
         url = attrs.get("url")
         data = {
@@ -505,6 +381,38 @@ class PatreonDownloader:
             "published_at": published if isinstance(published, str) else None,
             "url": url if isinstance(url, str) else None,
         }
-        sidecar_path = media_path.with_suffix(".json")
+        if source_url:
+            data["source_url"] = source_url
         sidecar_path.write_text(json.dumps(data, indent=2))
         return sidecar_path
+
+    def write_post_record(self, post: dict, artist_slug: str) -> PostRecordOutcome:
+        """Write a post-ONLY sidecar (no media file) for a media-less post, so
+        the importer can still upsert the Post + its body — text posts often hold
+        the only copy of an external <a href> link. Named `_post.json`: the
+        leading underscore keeps it from colliding with a media sidecar
+        (`<NN>_<stem>.json`) and from being resolved as some media file's sidecar
+        by find_sidecar.
+
+        Returns a PostRecordOutcome (path None when the post has no id) carrying
+        the captured body's shape — post_type + final char count — so the engine
+        can log per-post handling without re-reading the post itself.
+        """
+        attrs = post.get("attributes") or {}
+        title = attrs.get("title") if isinstance(attrs.get("title"), str) else None
+        post_type = attrs.get("post_type") if isinstance(attrs.get("post_type"), str) else None
+        pid = str(post.get("id") or "")
+        if not pid:
+            return PostRecordOutcome(
+                path=None, post_type=post_type, title=title, body_chars=0,
+            )
+        post_dir = self.images_root / artist_slug / "patreon" / post_dir_name(post)
+        post_dir.mkdir(parents=True, exist_ok=True)
+        path = self._write_sidecar_data(post, post_dir / "_post.json")
+        # _write_sidecar_data has by now memoized any detail-fetched body onto
+        # post["attributes"]["content"], so re-read it for the FINAL char count.
+        body = (post.get("attributes") or {}).get("content")
+        body_chars = len(body) if isinstance(body, str) else 0
+        return PostRecordOutcome(
+            path=path, post_type=post_type, title=title, body_chars=body_chars,
+        )

backend/app/services/patreon_ingester.py

@@ -35,15 +35,8 @@ from collections.abc import Callable
 from pathlib import Path
 
 from ..models import PatreonFailedMedia, PatreonSeenMedia
-from .gallery_dl import DownloadResult, ErrorType
 from .ingest_core import DEAD_LETTER_THRESHOLD, Ingester
-from .patreon_client import (
-    MediaItem,
-    PatreonAPIError,
-    PatreonAuthError,
-    PatreonClient,
-    PatreonDriftError,
-)
+from .patreon_client import MediaItem, PatreonAPIError, PatreonClient
 from .patreon_downloader import PatreonDownloader
 from .patreon_resolver import extract_vanity, resolve_campaign_id_for_source
 
@@ -112,7 +105,10 @@ class PatreonIngester(Ingester):
             downloader
             if downloader is not None
             else PatreonDownloader(
-                self.images_root, cookies_path, validate=validate, rate_limit=rate_limit
+                self.images_root, cookies_path, validate=validate, rate_limit=rate_limit,
+                # Enrich empty feed bodies from the per-post detail endpoint, via
+                # the SAME client (shares its cookie session + request pacing).
+                content_fetcher=resolved_client.fetch_post_detail_content,
             )
         )
         super().__init__(
@@ -126,51 +122,11 @@ class PatreonIngester(Ingester):
             ledger_key=_ledger_key,
             platform="patreon",
             error_base=PatreonAPIError,
+            # API_DRIFT message phrasing; the base Ingester._failure_result owns
+            # the auth/drift/HTTP→error_type mapping now (shared across platforms).
+            drift_label="Patreon API",
         )
 
-    # -- failure mapping (Patreon exception taxonomy) ----------------------
-
-    def _failure_result(self, exc: Exception, _result) -> DownloadResult:
-        """Map a client-level exception to a loud, typed failed DownloadResult.
-
-        We NEVER return a silent zero-download "success" — the whole point of the
-        native ingester is to fail RED when Patreon's API shape or our auth
-        changes. The typed mapping lets FailingSourcesCard render the right chip
-        and tells the operator what to do:
-          - PatreonAuthError  → AUTH_ERROR  (rotate cookies)
-          - PatreonDriftError → API_DRIFT   (ingester field-set/parser needs update)
-          - HTTP 429 / 404    → RATE_LIMITED / NOT_FOUND
-          - other HTTP status → HTTP_ERROR;  transport failure → NETWORK_ERROR
-
-        PatreonAuthError and PatreonDriftError both subclass PatreonAPIError, so
-        they must be matched before the generic HTTP/transport fallthrough.
-        """
-        message = str(exc)
-        if isinstance(exc, PatreonAuthError):
-            error_type = ErrorType.AUTH_ERROR
-        elif isinstance(exc, PatreonDriftError):
-            error_type = ErrorType.API_DRIFT
-            message = f"Patreon API changed — ingester needs update: {message}"
-        else:  # generic PatreonAPIError: HTTP non-2xx (status_code set) or transport
-            status = getattr(exc, "status_code", None)
-            if status == 429:
-                error_type = ErrorType.RATE_LIMITED
-            elif status == 404:
-                error_type = ErrorType.NOT_FOUND
-            elif status is not None:
-                error_type = ErrorType.HTTP_ERROR
-            else:
-                error_type = ErrorType.NETWORK_ERROR
-        log.warning("Patreon ingest failed (%s): %s", error_type.value, message)
-        result = _result(
-            success=False, return_code=1,
-            error_type=error_type, error_message=message,
-        )
-        # plan #708 B1: carry the server's Retry-After up to the cooldown.
-        if error_type == ErrorType.RATE_LIMITED:
-            result.retry_after_seconds = getattr(exc, "retry_after", None)
-        return result
-
 
 async def verify_patreon_credential(
     url: str,

backend/app/services/platform_lock.py

@@ -21,9 +21,10 @@ from ..config import get_config
 log = logging.getLogger(__name__)
 
 # Platforms walked one-at-a-time. gallery-dl platforms are intentionally NOT
-# here: each runs as a self-pacing subprocess and they're lower-volume. Add a
-# platform here to cap it to a single concurrent walk.
-SERIALIZED_PLATFORMS = frozenset({"patreon"})
+# here: each runs as a self-pacing subprocess and they're lower-volume. The
+# native-ingester platforms are serialized (one paced scrape/API walk at a time).
+# Add a platform here to cap it to a single concurrent walk.
+SERIALIZED_PLATFORMS = frozenset({"patreon", "subscribestar"})
 
 _LOCK_PREFIX = "fc:download_lock:"
 

backend/app/services/post_feed_service.py

@@ -11,19 +11,28 @@ attachments from PostAttachment) so the API layer can jsonify directly.
 """
 from __future__ import annotations
 
+from html import unescape
+
 from sqlalchemy import and_, func, or_, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from ..models import (
     Artist,
+    ExternalLink,
     ImageProvenance,
     ImageRecord,
     Post,
     PostAttachment,
     Source,
 )
+from ..utils.html_sanitize import (
+    extract_img_srcs,
+    rewrite_img_srcs,
+    sanitize_post_html,
+)
+from ..utils.paths import filehash_from_url
 from ..utils.text import html_to_plain, truncate_at_word
-from .gallery_service import thumbnail_url
+from .gallery_service import image_url, thumbnail_url
 from .pagination import decode_cursor, encode_cursor
 
 DESCRIPTION_LIMIT = 280
@@ -45,6 +54,7 @@ class PostFeedService:
         cursor: str | None = None,
         artist_id: int | None = None,
         platform: str | None = None,
+        q: str | None = None,
         limit: int = 24,
         direction: str = "older",
     ) -> dict:
@@ -52,7 +62,12 @@ class PostFeedService:
         time (default, infinite-scroll down); direction='newer' walks forward
         (scroll up in an anchored view). Items are always returned in feed
         (descending) order; `next_cursor` points to the far edge in the
-        requested direction (null when exhausted)."""
+        requested direction (null when exhausted).
+
+        `q` is a free-text filter (ILIKE substring over post_title OR
+        description) applied INSIDE the artist/platform scope, so a search
+        from the Browse bar stays within whatever artist is filtered in
+        view (operator-asked 2026-06-11)."""
         if limit < 1 or limit > 100:
             raise ValueError("limit must be between 1 and 100")
         if direction not in ("older", "newer"):
@@ -74,6 +89,12 @@ class PostFeedService:
             stmt = stmt.where(Post.artist_id == artist_id)
         if platform is not None:
             stmt = stmt.where(Source.platform == platform)
+        if q:
+            like = f"%{q}%"
+            stmt = stmt.where(or_(
+                Post.post_title.ilike(like),
+                Post.description.ilike(like),
+            ))
         if cursor:
             cur_ts, cur_id = decode_cursor(cursor)
             if direction == "older":
@@ -124,6 +145,7 @@ class PostFeedService:
         post_id: int,
         artist_id: int | None = None,
         platform: str | None = None,
+        q: str | None = None,
         limit: int = 12,
     ) -> dict | None:
         """A window centered on `post_id`: up to `limit` newer posts + the
@@ -143,11 +165,11 @@ class PostFeedService:
 
         older = await self.scroll(
             cursor=anchor_cursor, artist_id=artist_id, platform=platform,
-            limit=limit, direction="older",
+            q=q, limit=limit, direction="older",
         )
         newer = await self.scroll(
             cursor=anchor_cursor, artist_id=artist_id, platform=platform,
-            limit=limit, direction="newer",
+            q=q, limit=limit, direction="newer",
         )
         thumbs_map = await self._thumbnails_for([anchor_post.id])
         atts_map = await self._attachments_for([anchor_post.id])
@@ -178,8 +200,71 @@ class PostFeedService:
         atts_map = await self._attachments_for([post.id])
         item = self._to_dict(post, artist, source, thumbs_map, atts_map)
         item["description_full"] = html_to_plain(post.description)
+        # Sanitized HTML body for faithful (semantic) rendering in the post view;
+        # detail-only (the feed list stays lightweight plain text). None when the
+        # post has no body. Inline `<img>` sources are remapped to locally-served
+        # copies (#830 Phase 2) so the body never hotlinks the public CDN.
+        item["description_html"] = await self._localize_inline_images(
+            sanitize_post_html(post.description), post.artist_id,
+        )
+        item["external_links"] = await self._external_links_for(post.id)
         return item
 
+    async def _localize_inline_images(
+        self, html: str | None, artist_id: int | None,
+    ) -> str | None:
+        """Rewrite a post body's inline `<img src=CDN>` to locally-served copies.
+
+        The join key is the CDN filehash the downloader persisted on each
+        ImageRecord (source_filehash): for every body image whose filehash maps
+        to a stored image of THIS artist, swap the src to /images/<path>. Images
+        we never captured (or pre-Phase-2 rows with no filehash) are left as-is —
+        they keep hotlinking, which is the prior behavior. Scoped to the post's
+        artist so one creator's body never resolves to another's file."""
+        if not html or artist_id is None:
+            return html
+        srcs = extract_img_srcs(html)
+        if not srcs:
+            return html
+        # filehash -> the raw (as-in-HTML) src strings carrying it. A body can
+        # repeat the same image; keep every raw form so each is substituted.
+        by_hash: dict[str, list[str]] = {}
+        for raw in srcs:
+            fh = filehash_from_url(unescape(raw))
+            if fh:
+                by_hash.setdefault(fh, []).append(raw)
+        if not by_hash:
+            return html
+        rows = (await self.session.execute(
+            select(ImageRecord.source_filehash, ImageRecord.path)
+            .where(
+                ImageRecord.artist_id == artist_id,
+                ImageRecord.source_filehash.in_(list(by_hash)),
+            )
+        )).all()
+        replace: dict[str, str] = {}
+        for fh, path in rows:
+            for raw in by_hash.get(fh, ()):
+                replace[raw] = image_url(path)
+        return rewrite_img_srcs(html, replace)
+
+    async def _external_links_for(self, post_id: int) -> list[dict]:
+        """Off-platform file-host links recorded for a post (detail-only). Each
+        carries its host, full url, label, and download status so the post view
+        can surface them (and, later, a retry/download affordance)."""
+        rows = (await self.session.execute(
+            select(ExternalLink)
+            .where(ExternalLink.post_id == post_id)
+            .order_by(ExternalLink.id.asc())
+        )).scalars().all()
+        return [
+            {
+                "id": e.id, "host": e.host, "url": e.url,
+                "label": e.label, "status": e.status,
+            }
+            for e in rows
+        ]
+
     # --- composition helpers ---------------------------------------------
 
     async def _thumbnails_for(

backend/app/services/provenance_service.py

@@ -66,6 +66,10 @@ class ProvenanceService:
         ).scalars().all()
         return [_attachment_dict(a) for a in rows]
 
+    async def _attachment_by_id(self, attachment_id: int) -> list[dict]:
+        att = await self.session.get(PostAttachment, attachment_id)
+        return [_attachment_dict(att)] if att is not None else []
+
     async def for_image(self, image_id: int) -> dict | None:
         rec = await self.session.get(ImageRecord, image_id)
         if rec is None:
@@ -85,7 +89,33 @@ class ProvenanceService:
         )
         rows = (await self.session.execute(stmt)).all()
         post_ids = [ip.post_id for ip, _p, _s, _a in rows]
-        attachments = await self._attachments_for_posts(post_ids)
+        # Prefer the EXACT archive this file came out of (milestone #87): if the
+        # originating post's provenance row records from_attachment_id, the image
+        # was extracted from that one .zip/.rar, so show only it — not the dozens
+        # of unrelated archives a "High Resolution Files" bundle post carries.
+        from_att_id = next(
+            (
+                ip.from_attachment_id
+                for ip, _p, _s, _a in rows
+                if ip.post_id == rec.primary_post_id
+                and ip.from_attachment_id is not None
+            ),
+            None,
+        )
+        if from_att_id is not None:
+            attachments = await self._attachment_by_id(from_att_id)
+        else:
+            # No recorded containing archive (loose download, or pre-backfill):
+            # scope to the originating post only, not every pHash-linked post.
+            # primary_post_id is the post this file was actually captured from;
+            # fall back to all linked posts when it's unset (older rows /
+            # filesystem imports).
+            attach_post_ids = (
+                [rec.primary_post_id]
+                if rec.primary_post_id is not None
+                else post_ids
+            )
+            attachments = await self._attachments_for_posts(attach_post_ids)
         return {
             "image_id": image_id,
             "provenance": [

backend/app/services/series_match_service.py

@@ -18,7 +18,6 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from ..models import ImageRecord, Post, Tag, TagKind
-from ..models.series_chapter import SeriesChapter
 from ..models.series_page import SeriesPage
 from ..models.series_suggestion import SeriesSuggestion
 from ..models.tag import image_tag
@@ -144,9 +143,11 @@ class SeriesMatchService:
         return [t for t in rows if t]
 
     async def _series_max_stated_end(self, series_tag_id: int) -> int | None:
+        # Highest printed page across the series' pages (FC-6.x: stated pages
+        # live on series_page now; chapters are cosmetic dividers).
         return await self.session.scalar(
-            select(func.max(SeriesChapter.stated_page_end)).where(
-                SeriesChapter.series_tag_id == series_tag_id
+            select(func.max(SeriesPage.stated_page)).where(
+                SeriesPage.series_tag_id == series_tag_id
             )
         )
 
@@ -288,7 +289,7 @@ class SeriesMatchService:
             raise SeriesError(f"suggestion {suggestion_id} not found")
         if s.status != "pending":
             raise SeriesError(f"suggestion {suggestion_id} is already {s.status}")
-        out = await SeriesService(self.session).add_post_as_chapter(
+        out = await SeriesService(self.session).add_post(
             s.series_tag_id, s.post_id
         )
         s.status = "added"