fix: system health is a Settings tab, not a page only the dot reached

The surface shipped at /system with no nav entry, reachable only by
clicking the health dot beside the brand — a target you have to already
suspect something is wrong to go looking for. Operator-flagged: it needs
a path someone can walk to.

Settings is where you go to ask the instance about itself, so the view
becomes a tab there, beside Activity — Activity answers "what is the
queue doing", System answers "is anything left to do it".

- SystemView.vue moves to components/settings/SystemHealthTab.vue; the
  content is unchanged apart from shedding its own container and h1.
- SettingsView adopts useTabQuery (the composable Browse and
  Subscriptions already use) so a tab can be linked TO. The health dot
  now points at ?tab=system, and /system redirects there so the previous
  build's link and any bookmark still land.
- The tab drops its own 10s poll. v-window keeps a visited item mounted
  rather than destroyed, so that timer would have gone on firing behind
  Maintenance — and TopNav already polls the same store every 15s for
  the dot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

.env.example

@@ -34,6 +34,34 @@ DB_PASSWORD=
 SECRET_KEY=
 
 
+# ---------------------------------------------------------------------------
+# FIRST BOOT ONLY — then delete this line
+# ---------------------------------------------------------------------------
+
+# FabledCurator encrypts your stored platform credentials with a Fernet key it
+# keeps at /images/secrets/credential_key.b64 — inside the ./images bind mount,
+# so it outlives the container. On a brand-new install that file does not exist
+# yet, and the app REFUSES TO START rather than quietly create one:
+#
+#     MissingCredentialKey: Fernet key file not found at
+#     /images/secrets/credential_key.b64
+#
+# That refusal is deliberate. Auto-creating a key is indistinguishable from the
+# disaster case — a restore that brought the database back but lost
+# ./images/secrets — and there it would mint a key that cannot decrypt anything,
+# leaving an instance that looks healthy while every paywalled download fails.
+# So the choice is yours to make explicitly, once.
+#
+# Set this for your first `up`, watch the container come up, then DELETE THE
+# LINE. Leaving it set disarms the protection permanently, on an instance that
+# by then has credentials worth protecting.
+#
+# BACK UP ./images/secrets/ ALONGSIDE YOUR DATABASE. The key is the only thing
+# that can read your stored credentials; a database restored without it needs
+# every credential re-entered by hand.
+CURATOR_BOOTSTRAP_NEW_KEY=1
+
+
 # ---------------------------------------------------------------------------
 # Optional — defaults are fine
 # ---------------------------------------------------------------------------

.forgejo/workflows/build.yml

@@ -44,6 +44,10 @@ on:
         description: 'Rebuild every image even if the published revision matches'
         type: boolean
         default: false
+      refresh:
+        description: 'Behave as the weekly base refresh: build main against fresh bases, publish through the candidate tag'
+        type: boolean
+        default: false
 
   # The base-image refresh (milestone 326 step 4, #3154).
   #
@@ -72,8 +76,43 @@ on:
 # Deriving it per job invites the two halves to disagree: sign-extension would
 # derive dev's extension version while build-web bundled main's, and the
 # release download would 404 on a version that exists perfectly well.
+# IS THIS A BASE REFRESH? Asked in five places and previously spelled five
+# ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in
+# one shell, an `EVENT:` env passed into another, and a bare expression on
+# `pull:`. Five spellings of one fact is how half of them come to disagree
+# after somebody adds a sixth trigger.
+#
+# The `refresh` dispatch input is here so this path can be EXERCISED. A weekly
+# cron is otherwise testable once a week, which is not a cadence anything can
+# be developed against — the same reason `force_build` exists (#3252, added to
+# confirm #3190 was gone rather than wait for it to recur). It is also what
+# makes the milestone-362 gate verifiable at all: a gate has to be watched
+# rejecting something before anyone can believe it is wired up.
+#
+# The input is normalised through `format()` before it is compared, and that
+# is not defensive styling — the direct comparison is WRONG and fails silently.
+#
+# `type: boolean` delivers a real boolean, and GitHub expression semantics cast
+# operands to numbers when their types differ: `true == 'true'` compares 1
+# against NaN and is FALSE. Measured on run 5270, whose own log says it —
+#
+#   expression '(github.event_name == 'schedule'
+#                || github.event.inputs.refresh == 'true') && 'true' || 'false''
+#     evaluated to '%!t(string=false)'
+#   trigger: raw inputs refresh='true'
+#
+# — the input arrived as `true` and the expression still said false. The run
+# then went green with every step skipped, because a refresh that evaluates
+# false behaves exactly like an ordinary push. That is the whole hazard: the
+# failure has no symptom.
+#
+# `force_build` never hit this because it never compares in an expression. It
+# passes the raw value into an env var and tests it in the shell, where
+# everything is already a string. `format('{0}', x)` buys the same thing here,
+# where a step-level `if:` needs the answer before any shell runs.
 env:
-  BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
+  IS_REFRESH: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'true' || 'false' }}
+  BUILD_REF: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'main' || github.ref }}
 
 # Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
 #   - write:package, read:package  (for docker push to git.fabledsword.com)
@@ -143,7 +182,7 @@ jobs:
       # evaluate — this file already gates steps on it — so the guard cannot
       # be disabled by the same uncertainty it exists to cover.
       - name: Guard — a scheduled run must have checked out main
-        if: github.event_name == 'schedule'
+        if: env.IS_REFRESH == 'true'
         run: |
           set -eu
           BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -413,6 +452,18 @@ jobs:
        # to. Same source of truth; no double-store.
 
   build-web:
+    # Consumed by smoke-web's job-level `if:`. It cannot read `env` — the env
+    # context is available to STEP `if:` and step bodies, never to a job's own
+    # condition, and an unresolvable context there is empty rather than an
+    # error. `smoke-web` skipped silently on run 5290 for exactly that reason.
+    #
+    # Keying off the reuse step's own output is better than re-deriving the
+    # trigger anyway: it is the same single decision the build, the XPI
+    # download and the promote all take (build.yml's "one decision drives
+    # everything downstream"), and it says the thing smoke-web actually needs
+    # to know — a candidate was published — rather than restating why.
+    outputs:
+      candidate: ${{ steps.reuse.outputs.promote }}
     # A plain `needs` — no `always()`. That expression existed to let a
     # SKIPPED sign-extension through on a tag push while still blocking a
     # FAILED one. With no tag trigger, sign-extension always runs, so the
@@ -437,7 +488,7 @@ jobs:
 
       # See sign-extension's copy for why this guard exists.
       - name: Guard — a scheduled run must have checked out main
-        if: github.event_name == 'schedule'
+        if: env.IS_REFRESH == 'true'
         run: |
           set -eu
           BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -470,8 +521,18 @@ jobs:
       #     the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
       #   * dev and main derive the same values for the same source.
       - name: Report the derived artifact version
+        env:
+          # Diagnostic for the trigger normalisation. `refresh` is reported RAW
+          # as well as normalised, because the two disagreeing is the whole
+          # failure mode: a dispatch input whose type does not compare the way
+          # the expression assumes evaluates to false silently, and the only
+          # symptom is a refresh that quietly behaves like an ordinary push.
+          RAW_REFRESH: ${{ github.event.inputs.refresh }}
+          RAW_FORCE: ${{ github.event.inputs.force_build }}
         run: |
           set -u
+          echo "trigger: event=$GITHUB_EVENT_NAME IS_REFRESH='${IS_REFRESH:-<unset>}' BUILD_REF='${BUILD_REF:-<unset>}'"
+          echo "trigger: raw inputs refresh='${RAW_REFRESH:-<unset>}' force_build='${RAW_FORCE:-<unset>}'"
           A=web
           V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
           R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
@@ -528,7 +589,7 @@ jobs:
           # Checked BEFORE the ref test, not after: a scheduled run's
           # GITHUB_REF is the default branch (dev), so the main test would
           # never fire on it.
-          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+          if [ "${IS_REFRESH:-}" = "true" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           elif [ "${GITHUB_REF##*/}" = "main" ]; then
@@ -628,7 +689,6 @@ jobs:
           # A scheduled refresh has to bypass reuse by construction: it
           # rebuilds the SAME source, so fc.revision always matches and the
           # check would skip every refresh there has ever been.
-          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision web)
@@ -638,11 +698,60 @@ jobs:
           # adds no variability the reuse check would have to account for.
           echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT"
 
+          # The build clock, pinned to the same commit (#3265). Without it
+          # buildkit stamps the image config with the wall clock of the build,
+          # so identical layers republish under a new config blob and the
+          # channel tag gets a new manifest digest for no reason. Derived from
+          # `newest()` like revision and version, so all three name one commit
+          # and cannot drift apart.
+          echo "epoch=$(sh scripts/artifacts.sh epoch web)" >> "$GITHUB_OUTPUT"
+
           # The moving tag for this channel. Which tag we ask IS the channel —
           # that is why the revision needs no -main/-dev qualifier any more.
           if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
           echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
 
+          # WHERE THE BUILD PUBLISHES, which is not always the channel — and
+          # whether the channel then has to be written separately.
+          #
+          # On a push the build writes the channel tag directly: the bytes came
+          # from a commit, and a commit is the thing CI tests. Nothing to hold
+          # it behind.
+          #
+          # On the scheduled refresh it writes a CANDIDATE tag instead. A
+          # refresh rebuilds against freshly resolved base images, and the web
+          # image's runtime is a line of UNPINNED Debian packages (ffmpeg,
+          # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
+          # Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
+          # install requirements.txt, and a base bump changes neither. So
+          # refreshed bytes have to be proven before :latest names them, and
+          # proving needs a moment between "built" and "published" to occupy.
+          # This is that moment; :latest goes on naming the build that works
+          # until something says otherwise.
+          #
+          # `:refresh-candidate` is one moving ref per image, overwritten in
+          # place, holding a build nobody is told to pull — the shape rule 145
+          # already allows for :buildcache, not the per-build tag family that
+          # milestone 318 withdrew.
+          #
+          # Decided HERE, beside `hit`, for the reason the force/schedule
+          # branch below gives: one step decides what this job does. A
+          # condition derived independently could disagree with the tag the
+          # build actually wrote.
+          #
+          # build-web additionally exposes this as `outputs.candidate`, which is
+          # what gates the `promote` job — a job's `if:` cannot read `env`, and
+          # one flag is enough because all three derive it from the same
+          # IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
+          # reads is the kind of thing that later reads as load-bearing.
+          if [ "${IS_REFRESH:-}" = "true" ]; then
+            echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
+            echo "promote=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
+            echo "promote=false" >> "$GITHUB_OUTPUT"
+          fi
+
           # Compare VALUES, never exit codes. Measured on buildx v0.36.1
           # (run 4732): a missing key returns an empty string and exits 0, so
           # branching on the exit code would read "no label yet" as success.
@@ -675,7 +784,7 @@ jobs:
           if [ "${FORCE:-false}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: force_build set — building regardless"
-          elif [ "${EVENT:-}" = "schedule" ]; then
+          elif [ "${IS_REFRESH:-}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: scheduled base refresh — building regardless"
           elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
@@ -764,6 +873,12 @@ jobs:
 
       - name: Build and push web image
         if: steps.reuse.outputs.hit != 'true'
+        # Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
+        # it normalises the image config's `created` field and the history
+        # timestamps rather than being consumed by the Dockerfile. See #3265
+        # and the reuse step's `epoch` output.
+        env:
+          SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
         uses: docker/build-push-action@v5
         with:
           context: .
@@ -776,20 +891,17 @@ jobs:
           # invalidates, and the image genuinely rebuilds.
           #
           # MEASURED on the first real fire, run 4934 (#3265): when the base
-          # did NOT move, the build is ~13s and every content step reports
+          # did NOT move, the build was ~13s with every content step CACHED —
-          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # and the channel tag STILL got a new manifest digest, because
-          # buildkit mints a fresh image config each run, so identical layers
+          # buildkit stamps a fresh image config per run and republishes the
-          # are republished under a new config blob. All three images moved
+          # identical layers under it. All three images moved that way on
-          # that way on 2026-08-30 with nothing whatsoever changed in them.
+          # 2026-08-30 with nothing whatsoever changed in them.
           #
-          # So a refresh currently rewrites :latest every Sunday whether or
+          # SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
-          # not there is anything new in it, and :c-<sha> is handed a new
+          # content came from, the config is byte-identical across runs, so
-          # manifest to diverge from on the same cadence. Layers are shared,
+          # the manifest digest is too and the push is a registry no-op. A
-          # so the storage cost is a config blob; the cost that matters is
+          # digest change means the content changed again, which is the only
-          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # thing a digest is any use for.
-          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
-          # make "same source, same bytes" true and turn the no-op case into
-          # a genuine no-op.
           #
           # What `pull` does NOT catch either: a Debian package update inside
           # the `apt-get install` layer while the base tag itself stands
@@ -799,14 +911,14 @@ jobs:
           # churn #3265 is about.
           #
           # Only on the schedule. An ordinary push wants the cached base.
-          pull: ${{ github.event_name == 'schedule' }}
+          pull: ${{ env.IS_REFRESH == 'true' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
           # out of a local image store a registry-direct build never filled —
           # #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
           # published perfectly well.
-          tags: ${{ steps.reuse.outputs.channel_ref }}
+          tags: ${{ steps.reuse.outputs.build_ref }}
           # The reuse key. Read back off the channel tag on the next push to
           # decide whether that push needs to build at all, so this is not
           # decoration — an unstamped image is one that will always rebuild.
@@ -937,6 +1049,282 @@ jobs:
           docker buildx imagetools create $ARGS "$SOURCE"
           echo "repointed from $SOURCE:$ARGS"
 
+  # Does the image a refresh just built still work?
+  #
+  # This is the gate the base refresh never had. `ci.yml` cannot be it: its
+  # lanes run on ci-python:3.14 and install requirements.txt, and a base bump
+  # changes neither — all five stay green through a refresh that breaks the
+  # product. What a refresh re-resolves is the Dockerfile's apt layer (ffmpeg,
+  # unar, libpq5, postgresql-client, zstd, megatools, libjpeg62-turbo,
+  # libwebp7, libpng16-16), unpinned, every build.
+  #
+  # So this runs the CANDIDATE IMAGE, against real Postgres and Redis. Not the
+  # source tree, and not a static inspection: `ffmpeg -version` exiting 0 would
+  # pass while a codec removal broke every thumbnail in the library.
+  #
+  # Refresh-only. On a push the bytes came from a commit, and a commit is what
+  # ci.yml already tests.
+  #
+  # Reports a verdict; it does not yet gate the promote (milestone 362 step 4).
+  # Landing the gate and the thing it gates in one change would mean the first
+  # time anyone saw this job run would also be the first time it could stop a
+  # publish.
+  smoke-web:
+    needs: [build-web]
+    if: needs.build-web.outputs.candidate == 'true'
+    runs-on: python-ci
+    container:
+      image: git.fabledsword.com/bvandeusen/ci-python:3.14
+    env:
+      DB_USER: fabledcurator
+      DB_PASSWORD: ci_smoke
+      DB_PORT: "5432"
+      DB_NAME: fabledcurator_smoke
+      SECRET_KEY: ci_smoke_placeholder
+      IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        env:
+          POSTGRES_USER: fabledcurator
+          POSTGRES_PASSWORD: ci_smoke
+          POSTGRES_DB: fabledcurator_smoke
+        options: >-
+          --health-cmd "pg_isready -U fabledcurator"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 10
+      redis:
+        image: redis:7-alpine
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 10
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # The same ref the image was built from, so the smoke script matches
+          # the code inside the candidate.
+          ref: ${{ env.BUILD_REF }}
+
+      - name: Smoke the candidate image
+        env:
+          TOKEN: ${{ secrets.RELEASE_TOKEN }}
+          ACTOR: ${{ github.actor }}
+        run: |
+          set -eux
+          # Service discovery mirrors ci.yml's integration lane: these jobs run
+          # in a container against a mounted docker socket, so the services are
+          # SIBLINGS reachable by IP, not by hostname.
+          PG=$(docker ps --filter "name=smoke" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
+          RD=$(docker ps --filter "name=smoke" --filter "ancestor=redis:7-alpine" -q | head -n1)
+          test -n "$PG" && test -n "$RD"
+          PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
+          RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
+          test -n "$PG_IP" && test -n "$RD_IP"
+
+          # Socket probe in python, not bash's /dev/tcp — these steps run under
+          # `sh -e`, where that path does not exist. Same fix and reasoning as
+          # ci.yml's integration job; see the comment there.
+          pg_ready=""
+          for i in $(seq 1 60); do
+            if python -c "import socket,sys; s=socket.socket(); s.settimeout(2); sys.exit(0 if s.connect_ex(('$PG_IP', 5432)) == 0 else 1)"; then
+              pg_ready=1
+              break
+            fi
+            sleep 2
+          done
+          if [ -z "$pg_ready" ]; then
+            echo "postgres at $PG_IP:5432 did not accept a connection within 120s"
+            exit 1
+          fi
+
+          echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
+          CANDIDATE="$IMAGE:refresh-candidate"
+          docker pull "$CANDIDATE"
+
+          ENVOPTS="-e DB_USER=$DB_USER -e DB_PASSWORD=$DB_PASSWORD -e DB_HOST=$PG_IP"
+          ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY"
+          ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0"
+          ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0"
+          # A throwaway CI instance IS first-time setup, which is the one case
+          # credential_crypto allows a key to be minted in. Without it the web
+          # role refuses to boot — deliberately, since silently generating a
+          # key on a restored-DB-but-lost-secrets deployment would leave every
+          # Credential row undecryptable (the 2026-06-02 audit). Discovered by
+          # this job on its first real run; see #3422 for the fact that no
+          # user-facing file mentions this variable at all.
+          ENVOPTS="$ENVOPTS -e CURATOR_BOOTSTRAP_NEW_KEY=1"
+
+          # 1. The schema builds from empty, using the image's OWN libpq and
+          #    psycopg. This is the same call entrypoint.sh makes before it
+          #    serves anything, so a failure here is a failure to boot.
+          echo "smoke: alembic upgrade head"
+          docker run --rm $ENVOPTS "$CANDIDATE" alembic upgrade head
+
+          # 2. The apt layer's binaries and the app's own thumbnail path, run
+          #    inside the image. Piped over stdin rather than bind-mounted: the
+          #    workspace is a docker VOLUME belonging to this job's container,
+          #    so a host bind of $PWD would not resolve for a sibling.
+          echo "smoke: image-internal checks"
+          docker run --rm -i $ENVOPTS "$CANDIDATE" shell -c 'python3 -' < scripts/smoke_image.py
+
+          # 3. It actually serves. `docker run -d` then poll the container's own
+          #    IP — no port publishing, because the job container reaches
+          #    siblings directly and a published port would collide with
+          #    whatever else the runner is hosting.
+          echo "smoke: web boots and answers /api/health"
+          CID=$(docker run -d $ENVOPTS "$CANDIDATE" web)
+          # Clean up the container however this ends, and dump its log ONLY
+          # on failure — a boot that never answers must fail with the reason
+          # visible rather than as a bare timeout (rule 156), while a green run
+          # has nothing to say. `exit $rc` preserves the real status, which a
+          # trap that ends on a successful `docker rm` would otherwise mask.
+          trap 'rc=$?; [ $rc -eq 0 ] || docker logs "$CID" 2>&1 | tail -40; docker rm -f "$CID" >/dev/null 2>&1 || true; exit $rc' EXIT
+          WEB_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$CID")
+          test -n "$WEB_IP"
+          healthy=""
+          for i in $(seq 1 60); do
+            if curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health" >/dev/null 2>&1; then
+              healthy=1
+              break
+            fi
+            # A container that has EXITED will never answer, so stop asking.
+            # Without this the loop spent 3m35s polling a dead container on
+            # this job's first run, and — because docker recycles the IP — got
+            # a confusing mix of connection-refused and 5s timeouts from
+            # whatever took the address next. The trap's log dump had the real
+            # answer the whole time; this just stops burying it.
+            if [ "$(docker inspect -f '{{.State.Running}}' "$CID" 2>/dev/null)" != "true" ]; then
+              echo "smoke: FAILED — the web container exited during boot." >&2
+              echo "smoke: its log follows; entrypoint runs alembic BEFORE" >&2
+              echo "smoke: serving, so a startup exception lands here." >&2
+              exit 1
+            fi
+            sleep 2
+          done
+          if [ -z "$healthy" ]; then
+            # 60 iterations of (up to 5s connect + 2s sleep) — up to ~7min, not
+            # the 120s an earlier version of this message claimed.
+            echo "smoke: FAILED — web is running but never answered" >&2
+            echo "smoke: /api/health. It is up, so look at hypercorn and the" >&2
+            echo "smoke: python base rather than at startup." >&2
+            exit 1
+          fi
+          curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health"
+          echo
+
+          echo "smoke: all checks passed against $CANDIDATE"
+
+  # Move the channel tags — the whole point of the gate.
+  #
+  # Lives in its own job because the verdict it depends on cannot exist until
+  # after build-web has finished, and the promote used to run INSIDE build-web.
+  #
+  # `needs` on smoke-web is the gate. A failed smoke skips this job, so a
+  # refresh that broke something leaves :latest naming the build that works —
+  # "the refresh failed" and "production is broken" must not be the same event.
+  # A SKIPPED smoke also skips this job, which is the behaviour that matters
+  # most: on run 5290 the gate silently skipped itself, and a design where only
+  # a FAILED gate blocks would have published unverified images while reporting
+  # success. Not running is not the same as passing.
+  #
+  # All three images promote TOGETHER, or none do. They are one stack: build.yml
+  # already refuses to publish a :dev web image beside a stale :dev ml, because
+  # the mismatch only shows up as a runtime failure. A refresh that published ml
+  # and withheld web would be that same trap, arrived at through the gate.
+  #
+  # The gate covers the web image only (milestone 362 step 3 scoped it there),
+  # so ml and agent are being held to web's verdict rather than their own. That
+  # is deliberate and it is the conservative direction — they ship together, so
+  # the weakest evidence should govern all three — but it is not the same as
+  # having smoked them, and it should not be read as if it were.
+  promote:
+    needs: [build-web, build-ml, build-agent, smoke-web]
+    # Only a refresh publishes through a candidate; a push writes its channel
+    # tag directly from the build. Reads the same reuse-step decision the build
+    # took, via a job output — a job's `if:` cannot see the `env` context.
+    if: needs.build-web.outputs.candidate == 'true'
+    runs-on: python-ci
+    container:
+      image: git.fabledsword.com/bvandeusen/ci-python:3.14
+    steps:
+      - name: Point the channel tags at the smoked candidates
+        env:
+          TOKEN: ${{ secrets.RELEASE_TOKEN }}
+          ACTOR: ${{ github.actor }}
+        run: |
+          set -eu
+          # `latest` is not a guess: a refresh always builds `main` (BUILD_REF),
+          # and the "must have checked out main" guard in every build job fails
+          # the run if that did not hold. So the channel is main's.
+          TAG=latest
+          FAILED=""
+
+          for NAME in fabledcurator fabledcurator-ml fabledcurator-agent; do
+            REPO="bvandeusen/$NAME"
+            echo "promote: $REPO"
+
+            # Registry auth is its own token exchange — `docker login`
+            # authenticates the docker client, not curl. Deadline on every call
+            # (rule 156): a registry that stops answering must fail this step,
+            # not hang the weekly refresh until the job times out.
+            BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
+              "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
+              | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
+
+            # Ask for the IMAGE manifest media types only. Offering the index
+            # types too would let the registry hand back an index if one ever
+            # existed at this tag, and we would faithfully copy the thing this
+            # whole approach exists to avoid creating.
+            ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
+            CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
+                   -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
+                   "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
+                 && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
+            test -n "$CT"
+            SRC=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
+            echo "promote:   candidate $SRC ($CT)"
+
+            # NOT `imagetools create`. That wraps its source in an INDEX, and
+            # `.Image.Config.Labels` does not resolve through one — the
+            # fc.revision the reuse check reads off the channel tag would come
+            # back empty, every later push would miss and rebuild, and nothing
+            # would go red (#3183, run 4751). A manifest PUT is what "make this
+            # tag name that image" means at the registry: same bytes, same media
+            # type, same digest, no layer transfer.
+            curl -fsS --max-time 120 -X PUT \
+              -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
+              --data-binary @manifest.json \
+              "https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
+
+            # Read it back. A PUT that returned 2xx but landed something else is
+            # exactly the silent-and-plausible failure this pipeline keeps
+            # producing, and the check costs one request.
+            NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
+                    -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
+                    "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
+                  | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
+            if [ "$NOW" != "$SRC" ]; then
+              echo "promote:   FAILED — $NAME:$TAG is $NOW, expected $SRC" >&2
+              FAILED="$FAILED $NAME"
+              continue
+            fi
+            echo "promote:   $NAME:$TAG now names $NOW"
+          done
+
+          if [ -n "$FAILED" ]; then
+            echo "" >&2
+            echo "promote: FAILED for:$FAILED" >&2
+            echo "promote: the channel tags are now INCONSISTENT — some images" >&2
+            echo "promote: moved and some did not. Re-run this refresh; the" >&2
+            echo "promote: candidates are still published and the promote is" >&2
+            echo "promote: idempotent." >&2
+            exit 1
+          fi
+          echo "promote: all three channel tags moved"
+
   build-ml:
     runs-on: python-ci
     container:
@@ -957,7 +1345,7 @@ jobs:
 
       # See sign-extension's copy for why this guard exists.
       - name: Guard — a scheduled run must have checked out main
-        if: github.event_name == 'schedule'
+        if: env.IS_REFRESH == 'true'
         run: |
           set -eu
           BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -990,8 +1378,18 @@ jobs:
       #     the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
       #   * dev and main derive the same values for the same source.
       - name: Report the derived artifact version
+        env:
+          # Diagnostic for the trigger normalisation. `refresh` is reported RAW
+          # as well as normalised, because the two disagreeing is the whole
+          # failure mode: a dispatch input whose type does not compare the way
+          # the expression assumes evaluates to false silently, and the only
+          # symptom is a refresh that quietly behaves like an ordinary push.
+          RAW_REFRESH: ${{ github.event.inputs.refresh }}
+          RAW_FORCE: ${{ github.event.inputs.force_build }}
         run: |
           set -u
+          echo "trigger: event=$GITHUB_EVENT_NAME IS_REFRESH='${IS_REFRESH:-<unset>}' BUILD_REF='${BUILD_REF:-<unset>}'"
+          echo "trigger: raw inputs refresh='${RAW_REFRESH:-<unset>}' force_build='${RAW_FORCE:-<unset>}'"
           A=ml
           V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
           R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
@@ -1008,7 +1406,7 @@ jobs:
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
           # Mirrors build-web's tag list and its schedule handling; see
           # the comments there.
-          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+          if [ "${IS_REFRESH:-}" = "true" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           elif [ "${GITHUB_REF##*/}" = "main" ]; then
@@ -1091,17 +1489,63 @@ jobs:
           # A scheduled refresh has to bypass reuse by construction: it
           # rebuilds the SAME source, so fc.revision always matches and the
           # check would skip every refresh there has ever been.
-          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision ml)
           echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
 
+          # The build clock, pinned to the same commit (#3265). Without it
+          # buildkit stamps the image config with the wall clock of the build,
+          # so identical layers republish under a new config blob and the
+          # channel tag gets a new manifest digest for no reason. Derived from
+          # `newest()` like revision and version, so all three name one commit
+          # and cannot drift apart.
+          echo "epoch=$(sh scripts/artifacts.sh epoch ml)" >> "$GITHUB_OUTPUT"
+
           # The moving tag for this channel. Which tag we ask IS the channel —
           # that is why the revision needs no -main/-dev qualifier any more.
           if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
           echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
 
+          # WHERE THE BUILD PUBLISHES, which is not always the channel — and
+          # whether the channel then has to be written separately.
+          #
+          # On a push the build writes the channel tag directly: the bytes came
+          # from a commit, and a commit is the thing CI tests. Nothing to hold
+          # it behind.
+          #
+          # On the scheduled refresh it writes a CANDIDATE tag instead. A
+          # refresh rebuilds against freshly resolved base images, and the web
+          # image's runtime is a line of UNPINNED Debian packages (ffmpeg,
+          # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
+          # Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
+          # install requirements.txt, and a base bump changes neither. So
+          # refreshed bytes have to be proven before :latest names them, and
+          # proving needs a moment between "built" and "published" to occupy.
+          # This is that moment; :latest goes on naming the build that works
+          # until something says otherwise.
+          #
+          # `:refresh-candidate` is one moving ref per image, overwritten in
+          # place, holding a build nobody is told to pull — the shape rule 145
+          # already allows for :buildcache, not the per-build tag family that
+          # milestone 318 withdrew.
+          #
+          # Decided HERE, beside `hit`, for the reason the force/schedule
+          # branch below gives: one step decides what this job does. A
+          # condition derived independently could disagree with the tag the
+          # build actually wrote.
+          #
+          # build-web additionally exposes this as `outputs.candidate`, which is
+          # what gates the `promote` job — a job's `if:` cannot read `env`, and
+          # one flag is enough because all three derive it from the same
+          # IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
+          # reads is the kind of thing that later reads as load-bearing.
+          if [ "${IS_REFRESH:-}" = "true" ]; then
+            echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
+          else
+            echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
+          fi
+
           # Compare VALUES, never exit codes. Measured on buildx v0.36.1
           # (run 4732): a missing key returns an empty string and exits 0, so
           # branching on the exit code would read "no label yet" as success.
@@ -1134,7 +1578,7 @@ jobs:
           if [ "${FORCE:-false}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: force_build set — building regardless"
-          elif [ "${EVENT:-}" = "schedule" ]; then
+          elif [ "${IS_REFRESH:-}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: scheduled base refresh — building regardless"
           elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
@@ -1147,6 +1591,12 @@ jobs:
 
       - name: Build and push ml image
         if: steps.reuse.outputs.hit != 'true'
+        # Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
+        # it normalises the image config's `created` field and the history
+        # timestamps rather than being consumed by the Dockerfile. See #3265
+        # and the reuse step's `epoch` output.
+        env:
+          SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
         uses: docker/build-push-action@v5
         with:
           context: .
@@ -1159,20 +1609,17 @@ jobs:
           # invalidates, and the image genuinely rebuilds.
           #
           # MEASURED on the first real fire, run 4934 (#3265): when the base
-          # did NOT move, the build is ~13s and every content step reports
+          # did NOT move, the build was ~13s with every content step CACHED —
-          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # and the channel tag STILL got a new manifest digest, because
-          # buildkit mints a fresh image config each run, so identical layers
+          # buildkit stamps a fresh image config per run and republishes the
-          # are republished under a new config blob. All three images moved
+          # identical layers under it. All three images moved that way on
-          # that way on 2026-08-30 with nothing whatsoever changed in them.
+          # 2026-08-30 with nothing whatsoever changed in them.
           #
-          # So a refresh currently rewrites :latest every Sunday whether or
+          # SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
-          # not there is anything new in it, and :c-<sha> is handed a new
+          # content came from, the config is byte-identical across runs, so
-          # manifest to diverge from on the same cadence. Layers are shared,
+          # the manifest digest is too and the push is a registry no-op. A
-          # so the storage cost is a config blob; the cost that matters is
+          # digest change means the content changed again, which is the only
-          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # thing a digest is any use for.
-          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
-          # make "same source, same bytes" true and turn the no-op case into
-          # a genuine no-op.
           #
           # What `pull` does NOT catch either: a Debian package update inside
           # the `apt-get install` layer while the base tag itself stands
@@ -1182,14 +1629,14 @@ jobs:
           # churn #3265 is about.
           #
           # Only on the schedule. An ordinary push wants the cached base.
-          pull: ${{ github.event_name == 'schedule' }}
+          pull: ${{ env.IS_REFRESH == 'true' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
           # out of a local image store a registry-direct build never filled —
           # #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
           # published perfectly well.
-          tags: ${{ steps.reuse.outputs.channel_ref }}
+          tags: ${{ steps.reuse.outputs.build_ref }}
           # The reuse key. Read back off the channel tag on the next push to
           # decide whether that push needs to build at all, so this is not
           # decoration — an unstamped image is one that will always rebuild.
@@ -1331,7 +1778,7 @@ jobs:
 
       # See sign-extension's copy for why this guard exists.
       - name: Guard — a scheduled run must have checked out main
-        if: github.event_name == 'schedule'
+        if: env.IS_REFRESH == 'true'
         run: |
           set -eu
           BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -1364,8 +1811,18 @@ jobs:
       #     the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
       #   * dev and main derive the same values for the same source.
       - name: Report the derived artifact version
+        env:
+          # Diagnostic for the trigger normalisation. `refresh` is reported RAW
+          # as well as normalised, because the two disagreeing is the whole
+          # failure mode: a dispatch input whose type does not compare the way
+          # the expression assumes evaluates to false silently, and the only
+          # symptom is a refresh that quietly behaves like an ordinary push.
+          RAW_REFRESH: ${{ github.event.inputs.refresh }}
+          RAW_FORCE: ${{ github.event.inputs.force_build }}
         run: |
           set -u
+          echo "trigger: event=$GITHUB_EVENT_NAME IS_REFRESH='${IS_REFRESH:-<unset>}' BUILD_REF='${BUILD_REF:-<unset>}'"
+          echo "trigger: raw inputs refresh='${RAW_REFRESH:-<unset>}' force_build='${RAW_FORCE:-<unset>}'"
           A=agent
           V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
           R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
@@ -1377,7 +1834,7 @@ jobs:
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
           # Mirrors build-web's tag list and its schedule handling; see
           # the comments there.
-          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+          if [ "${IS_REFRESH:-}" = "true" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           elif [ "${GITHUB_REF##*/}" = "main" ]; then
@@ -1460,17 +1917,63 @@ jobs:
           # A scheduled refresh has to bypass reuse by construction: it
           # rebuilds the SAME source, so fc.revision always matches and the
           # check would skip every refresh there has ever been.
-          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision agent)
           echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
 
+          # The build clock, pinned to the same commit (#3265). Without it
+          # buildkit stamps the image config with the wall clock of the build,
+          # so identical layers republish under a new config blob and the
+          # channel tag gets a new manifest digest for no reason. Derived from
+          # `newest()` like revision and version, so all three name one commit
+          # and cannot drift apart.
+          echo "epoch=$(sh scripts/artifacts.sh epoch agent)" >> "$GITHUB_OUTPUT"
+
           # The moving tag for this channel. Which tag we ask IS the channel —
           # that is why the revision needs no -main/-dev qualifier any more.
           if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
           echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
 
+          # WHERE THE BUILD PUBLISHES, which is not always the channel — and
+          # whether the channel then has to be written separately.
+          #
+          # On a push the build writes the channel tag directly: the bytes came
+          # from a commit, and a commit is the thing CI tests. Nothing to hold
+          # it behind.
+          #
+          # On the scheduled refresh it writes a CANDIDATE tag instead. A
+          # refresh rebuilds against freshly resolved base images, and the web
+          # image's runtime is a line of UNPINNED Debian packages (ffmpeg,
+          # libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
+          # Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
+          # install requirements.txt, and a base bump changes neither. So
+          # refreshed bytes have to be proven before :latest names them, and
+          # proving needs a moment between "built" and "published" to occupy.
+          # This is that moment; :latest goes on naming the build that works
+          # until something says otherwise.
+          #
+          # `:refresh-candidate` is one moving ref per image, overwritten in
+          # place, holding a build nobody is told to pull — the shape rule 145
+          # already allows for :buildcache, not the per-build tag family that
+          # milestone 318 withdrew.
+          #
+          # Decided HERE, beside `hit`, for the reason the force/schedule
+          # branch below gives: one step decides what this job does. A
+          # condition derived independently could disagree with the tag the
+          # build actually wrote.
+          #
+          # build-web additionally exposes this as `outputs.candidate`, which is
+          # what gates the `promote` job — a job's `if:` cannot read `env`, and
+          # one flag is enough because all three derive it from the same
+          # IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
+          # reads is the kind of thing that later reads as load-bearing.
+          if [ "${IS_REFRESH:-}" = "true" ]; then
+            echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
+          else
+            echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
+          fi
+
           # Compare VALUES, never exit codes. Measured on buildx v0.36.1
           # (run 4732): a missing key returns an empty string and exits 0, so
           # branching on the exit code would read "no label yet" as success.
@@ -1503,7 +2006,7 @@ jobs:
           if [ "${FORCE:-false}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: force_build set — building regardless"
-          elif [ "${EVENT:-}" = "schedule" ]; then
+          elif [ "${IS_REFRESH:-}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: scheduled base refresh — building regardless"
           elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
@@ -1516,6 +2019,12 @@ jobs:
 
       - name: Build and push agent image
         if: steps.reuse.outputs.hit != 'true'
+        # Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
+        # it normalises the image config's `created` field and the history
+        # timestamps rather than being consumed by the Dockerfile. See #3265
+        # and the reuse step's `epoch` output.
+        env:
+          SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
         uses: docker/build-push-action@v5
         with:
           context: agent
@@ -1528,20 +2037,17 @@ jobs:
           # invalidates, and the image genuinely rebuilds.
           #
           # MEASURED on the first real fire, run 4934 (#3265): when the base
-          # did NOT move, the build is ~13s and every content step reports
+          # did NOT move, the build was ~13s with every content step CACHED —
-          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # and the channel tag STILL got a new manifest digest, because
-          # buildkit mints a fresh image config each run, so identical layers
+          # buildkit stamps a fresh image config per run and republishes the
-          # are republished under a new config blob. All three images moved
+          # identical layers under it. All three images moved that way on
-          # that way on 2026-08-30 with nothing whatsoever changed in them.
+          # 2026-08-30 with nothing whatsoever changed in them.
           #
-          # So a refresh currently rewrites :latest every Sunday whether or
+          # SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
-          # not there is anything new in it, and :c-<sha> is handed a new
+          # content came from, the config is byte-identical across runs, so
-          # manifest to diverge from on the same cadence. Layers are shared,
+          # the manifest digest is too and the push is a registry no-op. A
-          # so the storage cost is a config blob; the cost that matters is
+          # digest change means the content changed again, which is the only
-          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # thing a digest is any use for.
-          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
-          # make "same source, same bytes" true and turn the no-op case into
-          # a genuine no-op.
           #
           # What `pull` does NOT catch either: a Debian package update inside
           # the `apt-get install` layer while the base tag itself stands
@@ -1551,14 +2057,14 @@ jobs:
           # churn #3265 is about.
           #
           # Only on the schedule. An ordinary push wants the cached base.
-          pull: ${{ github.event_name == 'schedule' }}
+          pull: ${{ env.IS_REFRESH == 'true' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
           # out of a local image store a registry-direct build never filled —
           # #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
           # published perfectly well.
-          tags: ${{ steps.reuse.outputs.channel_ref }}
+          tags: ${{ steps.reuse.outputs.build_ref }}
           # The reuse key. Read back off the channel tag on the next push to
           # decide whether that push needs to build at all, so this is not
           # decoration — an unstamped image is one that will always rebuild.

README.md

@@ -1,5 +1,6 @@
 # FabledCurator
 
+<!-- overview:start -->
 Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven
 downloading in one application. Part of the FabledSword family.
 
@@ -19,13 +20,15 @@ through afterwards.
   characters, find near-duplicates and power similarity search. Suggestions are
   reviewable — it proposes, you confirm, and it learns which proposals you keep
   rejecting.
-- **Importing.** Ingests an existing library from disk, deduplicates it by
+- **Deduplication and provenance.** Everything that arrives is hashed and
-  content hash, and reads metadata sidecars.
+  deduplicated by content, metadata sidecars are read wherever the source
+  writes them, and every file keeps a record of where it came from.
 - **Maintenance.** Backups, library audits, thumbnail and embedding backfills,
   orphan cleanup — all from the UI, all as background jobs you can watch.
 
 Everything is configured from the Settings UI and stored in the database. There
 is no config file to edit beyond a handful of bootstrap environment variables.
+<!-- overview:end -->
 
 ## Before you expose it
 
@@ -87,7 +90,31 @@ If you forget it, the symptom is a long build instead of a quick pull.
 The database schema is created automatically on first start — the web container
 runs its migrations before serving. Nothing to initialise by hand.
 
-A few things are worth knowing about the first few minutes:
+**One thing does need a deliberate act, and the app will not start without it.**
+FabledCurator encrypts your stored platform credentials with a key it keeps at
+`./images/secrets/credential_key.b64`. On a brand-new install that file does not
+exist, and rather than quietly creating one the app stops:
+
+```
+MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64
+```
+
+Set `CURATOR_BOOTSTRAP_NEW_KEY=1` in your `.env` for the first `up`, then delete
+the line once the container is running. `.env.example` ships it with that
+instruction attached.
+
+The refusal is deliberate, and worth understanding rather than working around:
+auto-creating a key is indistinguishable from the disaster case — a restore that
+brought the database back but lost `./images/secrets` — where it would mint a key
+that cannot decrypt anything, leaving an instance that looks healthy while every
+paywalled download fails. Making you say so once, on an empty install, is the
+price of that not happening silently later.
+
+**Which means: back up `./images/secrets/` alongside your database.** It is the
+only thing that can read your stored credentials. A database restored without it
+needs every credential entered again by hand.
+
+A few other things are worth knowing about the first few minutes:
 
 - **The ML worker downloads its model weights on first boot**, several GB from
   HuggingFace into `./models`. Until that finishes, tagging is queued rather

alembic/versions/0090_service_seen.py

@@ -0,0 +1,64 @@
+"""service_seen — the learned roster that makes a stopped part observable.
+
+Milestone 365. Nothing in FabledCurator knew what was SUPPOSED to be running:
+`celery inspect` reports the workers that answer, so a dead worker was a
+shorter list rather than a red light, and the only surface that could tell an
+operator otherwise was Portainer. This table is the memory that turns an
+absence into something the app can see.
+
+Keyed on the queue set for a celery role and on agent_id for the GPU agent —
+NOT on the celery worker name, which here is `celery@<container id>` and is
+minted fresh on every deploy. See the model docstring for why that choice is
+the whole design.
+
+## First migration on the collapsed baseline
+
+0089 is the single generated baseline that replaced revisions 0001..0089
+(milestone 328). This is the first revision written on top of it, so it is
+also the first evidence that the chain steps forward from the collapse rather
+than merely reproducing the schema — which nothing had demonstrated yet.
+
+An existing install is at 0089 because it ran the real 0089; a fresh one is at
+0089 because it ran the baseline. Both arrive here identically, which was the
+property the collapse was designed around.
+
+Revision ID: 0090
+Revises: 0089
+Create Date: 2026-09-02
+
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0090"
+down_revision: Union[str, None] = "0089"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "service_seen",
+        sa.Column("key", sa.String(length=128), nullable=False),
+        sa.Column("kind", sa.String(length=16), nullable=False),
+        sa.Column("display_name", sa.String(length=64), nullable=False),
+        sa.Column(
+            "first_seen_at", sa.DateTime(timezone=True),
+            server_default=sa.text("now()"), nullable=False,
+        ),
+        sa.Column(
+            "last_seen_at", sa.DateTime(timezone=True),
+            server_default=sa.text("now()"), nullable=False,
+        ),
+        sa.Column("details", sa.JSON(), nullable=False),
+        sa.PrimaryKeyConstraint("key", name=op.f("pk_service_seen")),
+    )
+    # No secondary indexes, deliberately: one row per moving part means every
+    # read is a handful of rows and an index would be write cost buying
+    # nothing (#3301 removed seven of exactly that shape).
+
+
+def downgrade() -> None:
+    op.drop_table("service_seen")

backend/app/api/__init__.py

@@ -38,6 +38,7 @@ def all_blueprints() -> list[Blueprint]:
     from .suggestions import suggestions_bp
     from .system_activity import system_activity_bp
     from .system_backup import system_backup_bp
+    from .system_health import system_health_bp
     from .tags import tags_bp
     from .thumbnails import thumbnails_bp
     return [
@@ -51,6 +52,7 @@ def all_blueprints() -> list[Blueprint]:
         showcase_bp,
         settings_bp,
         system_activity_bp,
+        system_health_bp,
         system_backup_bp,
         admin_bp,
         cleanup_bp,

backend/app/api/gpu.py

@@ -21,6 +21,7 @@ from ..services.gallery_service import image_url
 from ..services.ml.gpu_jobs import GpuJobService, error_dedupe_statements
 from ..services.ml.gpu_triage import classify_reason, recover_defective_image
 from ..services.ml.regions import RegionService
+from ..services.service_roster import touch_service
 
 gpu_bp = Blueprint("gpu", __name__, url_prefix="/api/gpu")
 
@@ -256,6 +257,18 @@ async def lease():
         if not await _agent_authed(session):
             return jsonify({"error": "unauthorized"}), 401
         jobs = await GpuJobService(session).lease(agent_id, batch_size=batch)
+        # The agent cannot be polled — it is HTTP-only and pulls from here, so
+        # web never dials it. A lease IS the check-in, and until milestone 365
+        # it was thrown away: an agent sitting idle with nothing to lease left
+        # no trace at all and was indistinguishable from one switched off a
+        # week ago. Recorded on the call that was already happening.
+        await touch_service(
+            session,
+            key=f"agent:{agent_id}",
+            kind="agent",
+            display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})",
+            details={"agent_id": agent_id, "last_call": "lease", "leased": len(jobs)},
+        )
         ml = await MLSettings.load(session)
         # image rows for url/mime in one shot
         ids = [j.image_record_id for j in jobs]
@@ -329,6 +342,13 @@ async def heartbeat():
         if not await _agent_authed(session):
             return jsonify({"error": "unauthorized"}), 401
         n = await GpuJobService(session).heartbeat(agent_id, job_ids)
+        await touch_service(
+            session,
+            key=f"agent:{agent_id}",
+            kind="agent",
+            display_name="GPU agent" if agent_id == "agent" else f"GPU agent ({agent_id})",
+            details={"agent_id": agent_id, "last_call": "heartbeat", "extended": n},
+        )
         await session.commit()
     return jsonify({"extended": n})
 

backend/app/api/system_health.py

@@ -0,0 +1,192 @@
+"""Is every part of FabledCurator running? One verdict, one endpoint.
+
+Milestone 365. The nav indicator and the System page both read this and
+nothing else — composing a verdict is this module's job, not the UI's.
+
+## Two kinds of part, answered two different ways
+
+**Learned** — celery roles and the GPU agent, from `service_seen`. The
+question is "how long since it checked in", and these are the parts that can
+be ABSENT, which is the whole point: `celery inspect` alone reports presence,
+so a dead worker is a shorter list rather than a red light.
+
+**Probed live** — Postgres and Redis. Always expected, never learned, and a
+last-seen for them would be actively misleading: that Redis answered thirty
+seconds ago says nothing about now.
+
+## This endpoint must never fail because something it checks has failed
+
+The inversion is easy to write by accident and it destroys the feature exactly
+when it is needed — a 500 when Redis is down, instead of `redis: down`. Every
+probe is wrapped, every wait has a deadline (rule 156), and the roster refresh
+swallows its own errors. The worst case is a part reported `unknown`, which is
+a true statement.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from datetime import UTC, datetime
+
+from quart import Blueprint, jsonify
+from sqlalchemy import select, text
+
+from ..config import get_config
+from ..extensions import get_session
+from ..models import ServiceSeen
+from ..services.service_roster import refresh_if_stale
+
+log = logging.getLogger(__name__)
+
+system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system")
+
+# How long a learned part may go quiet before it is doubted, then disbelieved.
+#
+# These are deliberately generous, and the reason is a deploy rather than a
+# worker: `docker compose up -d` rolls start-first, so a role is briefly served
+# by two containers and then by neither while the old one drains. Thresholds
+# tight enough to catch a crash in seconds would paint the page red every time
+# the stack is updated, and an alarm that cries wolf on every deploy is one
+# nobody reads. Tune down only after watching a real deploy pass through.
+STALE_AFTER_SECONDS = 90
+DOWN_AFTER_SECONDS = 300
+
+# Probes cross a process boundary, so they carry deadlines. A hung Postgres
+# must make this endpoint say "postgres: down", not hang alongside it.
+PROBE_TIMEOUT_SECONDS = 2.0
+
+_OK, _STALE, _DOWN, _UNKNOWN = "ok", "stale", "down", "unknown"
+
+# Worst-first, so an overall verdict is just the max.
+_SEVERITY = {_OK: 0, _UNKNOWN: 1, _STALE: 2, _DOWN: 3}
+
+
+def _age_state(age_seconds: float) -> str:
+    if age_seconds >= DOWN_AFTER_SECONDS:
+        return _DOWN
+    if age_seconds >= STALE_AFTER_SECONDS:
+        return _STALE
+    return _OK
+
+
+def _describe_learned(name: str, state: str, age: float, details: dict) -> str:
+    """Say what the state MEANS. A red chip tells an operator less than a
+    sentence does at the moment they are deciding whether to go and look."""
+    if state == _OK:
+        replicas = details.get("replicas")
+        if replicas and replicas > 1:
+            return f"{name} is running ({replicas} replicas)"
+        return f"{name} is running"
+    mins = int(age // 60)
+    ago = f"{mins} min" if mins else f"{int(age)}s"
+    if state == _STALE:
+        return f"{name} has not checked in for {ago}"
+    return f"{name} has not checked in for {ago} — treat it as stopped"
+
+
+async def _probe_postgres(session) -> dict:
+    started = time.monotonic()
+    try:
+        await asyncio.wait_for(
+            session.execute(text("SELECT 1")), timeout=PROBE_TIMEOUT_SECONDS
+        )
+    except Exception as exc:  # noqa: BLE001 — a probe reports, it never raises
+        return {
+            "key": "postgres", "kind": "datastore", "name": "PostgreSQL",
+            "state": _DOWN, "detail": f"not answering: {type(exc).__name__}",
+        }
+    return {
+        "key": "postgres", "kind": "datastore", "name": "PostgreSQL", "state": _OK,
+        "detail": "answering", "latency_ms": round((time.monotonic() - started) * 1000, 1),
+    }
+
+
+def _ping_redis_sync() -> None:
+    import redis  # local import; mirrors system_activity's pattern
+
+    client = redis.Redis.from_url(
+        get_config().celery_broker_url,
+        socket_connect_timeout=PROBE_TIMEOUT_SECONDS,
+        socket_timeout=PROBE_TIMEOUT_SECONDS,
+    )
+    client.ping()
+
+
+async def _probe_redis() -> dict:
+    started = time.monotonic()
+    try:
+        await asyncio.wait_for(
+            asyncio.to_thread(_ping_redis_sync), timeout=PROBE_TIMEOUT_SECONDS * 2
+        )
+    except Exception as exc:  # noqa: BLE001
+        return {
+            "key": "redis", "kind": "datastore", "name": "Redis",
+            "state": _DOWN,
+            "detail": f"not answering: {type(exc).__name__} — queues and workers "
+                      f"cannot be reached either",
+        }
+    return {
+        "key": "redis", "kind": "datastore", "name": "Redis", "state": _OK,
+        "detail": "answering", "latency_ms": round((time.monotonic() - started) * 1000, 1),
+    }
+
+
+@system_health_bp.route("/health", methods=["GET"])
+async def system_health():
+    """Every part, its state, and one overall verdict.
+
+    Response: {overall, parts: [{key, kind, name, state, detail, last_seen_at,
+    …}], checked_at}
+    """
+    parts: list[dict] = []
+    now = datetime.now(UTC)
+
+    async with get_session() as session:
+        # Postgres first, and if it is unreachable nothing else can be read —
+        # say so rather than failing, because "the database is down" is the
+        # single most useful thing this endpoint can ever report.
+        pg = await _probe_postgres(session)
+        parts.append(pg)
+
+        if pg["state"] == _OK:
+            # Rate-limited inside; see service_roster on why the web process
+            # is the right observer.
+            try:
+                await refresh_if_stale(session)
+                await session.commit()
+            except Exception:  # noqa: BLE001
+                log.warning("system health: roster refresh failed", exc_info=True)
+
+            rows = (
+                await session.execute(select(ServiceSeen).order_by(ServiceSeen.display_name))
+            ).scalars().all()
+            for row in rows:
+                age = (now - row.last_seen_at).total_seconds()
+                state = _age_state(age)
+                parts.append({
+                    "key": row.key,
+                    "kind": row.kind,
+                    "name": row.display_name,
+                    "state": state,
+                    "detail": _describe_learned(row.display_name, state, age, row.details or {}),
+                    "last_seen_at": row.last_seen_at.isoformat(),
+                    "first_seen_at": row.first_seen_at.isoformat(),
+                    **{k: v for k, v in (row.details or {}).items() if k != "agent_id"},
+                })
+
+    parts.append(await _probe_redis())
+
+    overall = max((p["state"] for p in parts), key=lambda s: _SEVERITY[s], default=_UNKNOWN)
+    return jsonify({
+        "overall": overall,
+        "parts": sorted(parts, key=lambda p: (-_SEVERITY[p["state"]], p["name"])),
+        "checked_at": now.isoformat(),
+        # So the UI can explain a `stale` without hard-coding the same numbers
+        # in a second place.
+        "thresholds": {
+            "stale_after_seconds": STALE_AFTER_SECONDS,
+            "down_after_seconds": DOWN_AFTER_SECONDS,
+        },
+    })

backend/app/models/__init__.py

@@ -32,6 +32,7 @@ from .presentation_review import PresentationReview
 from .series_chapter import SeriesChapter
 from .series_page import SeriesPage
 from .series_suggestion import SeriesSuggestion
+from .service_seen import ServiceSeen
 from .source import Source
 from .subscribestar_failed_media import SubscribeStarFailedMedia
 from .subscribestar_seen_media import SubscribeStarSeenMedia
@@ -63,6 +64,7 @@ __all__ = [
     "SeriesChapter",
     "SeriesPage",
     "SeriesSuggestion",
+    "ServiceSeen",
     "ImageRecord",
     "ImageProvenance",
     "ImageRegion",

backend/app/models/service_seen.py

@@ -0,0 +1,88 @@
+"""service_seen — the learned roster of FabledCurator's own moving parts.
+
+Nothing else in this application knows what is SUPPOSED to be running.
+`celery inspect` reports the workers that answer, so a stopped worker is a
+shorter list rather than a red light, and Postgres and Redis have no
+representation at all. That is why the only place an operator could see a
+dead service was Portainer, which knows the intended set (milestone 365).
+
+This table is the memory that makes an absence observable: every part that
+has ever checked in, and when it last did. A row that stops advancing is a
+part that stopped.
+
+## Why the key is not the hostname
+
+`_read_workers_sync()` returns celery's worker names, which here are
+`celery@<container id>`. Those are minted fresh on every deploy. Keyed on
+them, this table would record a death and a birth every time the stack is
+updated — and a status page that goes red on every deploy is a status page
+nobody reads, which is worse than not having one.
+
+So a celery role is keyed on its **queue set**, which is assigned per role in
+docker-compose.yml (`CELERY_QUEUES`) and survives container replacement:
+
+    default,import,thumbnail,download  ->  worker
+    maintenance,scan                   ->  scheduler   (celery worker --beat)
+    ml                                 ->  ml-worker
+
+Two replicas of one role share a queue set and are therefore ONE row — which
+is right, because the question being answered is "is that role being served",
+not "how many containers exist". The replica count and their hostnames go in
+`details`, where they can change without the identity changing.
+
+The GPU agent is keyed on its `agent_id`, the identity its lease protocol
+already uses (`api/gpu.py`).
+
+## What is NOT in here
+
+Postgres and Redis. They are always expected and never learned, and a
+last-seen for them would be actively misleading — that one answered thirty
+seconds ago says nothing about now. They are probed live at request time.
+
+## kind
+
+Plain `String`, not a Postgres ENUM and not CHECK-gated, matching
+`gpu_job.status` and `backup_run.status`. The value set here is expected to
+grow as parts are added, and a constraint swap per new kind (rule 36) would
+be cost with no invariant behind it.
+
+    celery  — a worker role, keyed on its queue set
+    agent   — a GPU agent, keyed on its agent_id
+"""
+
+from datetime import datetime
+
+from sqlalchemy import JSON, DateTime, String, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class ServiceSeen(Base):
+    __tablename__ = "service_seen"
+
+    # No indexes beyond the primary key, deliberately. This table holds one row
+    # per moving part — a handful, forever — so every query against it is a
+    # full read of a few rows and an index would be write cost buying nothing
+    # (the lesson of #3301, which removed seven redundant ones).
+    key: Mapped[str] = mapped_column(String(128), primary_key=True)
+    kind: Mapped[str] = mapped_column(String(16), nullable=False)
+
+    # What to call it in the UI. Derived from the queue set where it is
+    # recognised, and falling back to the raw queue list where it is not — a
+    # deployment that slices its queues differently should still show something
+    # true rather than a name this code invented for it.
+    display_name: Mapped[str] = mapped_column(String(64), nullable=False)
+
+    first_seen_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now(),
+    )
+    last_seen_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now(),
+    )
+
+    # The parts that change without changing identity: replica hostnames,
+    # active task counts, the queues actually being served. Kept as a blob
+    # because it is displayed and never queried — giving it columns would
+    # invite filtering on it, which is what the activity endpoints are for.
+    details: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)

backend/app/services/service_roster.py

@@ -0,0 +1,175 @@
+"""The learned roster: which of FabledCurator's parts have checked in, and when.
+
+Milestone 365. `celery inspect` answers "who is here"; this answers "who is
+missing", which nothing in the application could do before — see
+`models/service_seen.py` for why the identity is a queue set and not a
+worker hostname.
+
+## Who does the observing, and why it is the web process
+
+Three candidates, and the choice matters more than the code:
+
+* **A celery beat sweep.** Rejected. If the scheduler dies, the sweep stops,
+  every row goes stale, and the page reports that everything is down when one
+  thing is. An alarm that cannot distinguish "one part died" from "the
+  observer died" is worse than no alarm.
+* **A background task in web.** Rejected on a detail of how this deploys:
+  hypercorn runs `--workers 4`, so a `before_serving` loop would be FOUR
+  concurrent inspect loops hammering the broker, forever, per container.
+* **Refresh on demand, rate-limited by the data itself.** Taken. Whichever web
+  process happens to serve a health request refreshes the roster if it is
+  older than REFRESH_TTL, and otherwise reads what is already there.
+
+The third has the property the other two lack: **the observer is the thing
+serving the page.** If web is down you get a browser error rather than a
+confidently green page, which is the honest failure. It also self-limits
+without coordination — the TTL lives in the row everybody can see.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+
+from sqlalchemy import func, select
+from sqlalchemy.dialects.postgresql import insert as pg_insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from ..models import ServiceSeen
+
+log = logging.getLogger(__name__)
+
+# How stale the roster may be before a health request refreshes it. Comfortably
+# under the staleness thresholds that decide a service is missing, so the
+# verdict is never limited by how often anyone looked.
+REFRESH_TTL_SECONDS = 20.0
+
+# celery inspect is a broker round trip and this sits on a request path, so it
+# gets a deadline (rule 156). A broker that has stopped answering must make the
+# roster stale — which is a true statement about the system — not hang the one
+# page that exists to explain it.
+INSPECT_TIMEOUT_SECONDS = 2.0
+
+# Queue set -> the name an operator recognises. Sorted-tuple keys, because the
+# order celery reports them in is not guaranteed.
+#
+# A deployment that slices CELERY_QUEUES differently falls through to the raw
+# queue list rather than being given a name this table invented for it: a
+# wrong-but-confident label on a status page is worse than an ugly true one.
+ROLE_NAMES: dict[tuple[str, ...], str] = {
+    ("default", "download", "import", "thumbnail"): "Worker",
+    ("maintenance", "scan"): "Scheduler",
+    ("ml",): "ML worker",
+}
+
+
+def role_display_name(queues: tuple[str, ...]) -> str:
+    known = ROLE_NAMES.get(queues)
+    if known:
+        return known
+    return "Worker (" + ", ".join(queues) + ")"
+
+
+def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
+    """celery inspect, grouped by queue set rather than by worker.
+
+    Returns {queue_set: {"hostnames": [...], "active": int}}. Two replicas of
+    one role collapse into one entry on purpose — the question is whether the
+    role is being served, not how many containers exist.
+    """
+    from ..celery_app import celery as celery_app
+
+    insp = celery_app.control.inspect(timeout=INSPECT_TIMEOUT_SECONDS)
+    active_queues = insp.active_queues() or {}
+    active_tasks = insp.active() or {}
+
+    grouped: dict[tuple[str, ...], dict] = {}
+    for hostname, queues in active_queues.items():
+        key = tuple(sorted({q["name"] for q in queues}))
+        entry = grouped.setdefault(key, {"hostnames": [], "active": 0})
+        entry["hostnames"].append(hostname)
+        entry["active"] += len(active_tasks.get(hostname, []))
+    for entry in grouped.values():
+        entry["hostnames"].sort()
+    return grouped
+
+
+async def touch_service(
+    session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict
+) -> None:
+    """Record that a part checked in just now.
+
+    Upsert rather than read-modify-write: several web processes and several
+    agents can be doing this at once, and the last writer is simply the most
+    recent sighting. `first_seen_at` is deliberately NOT updated — it is the
+    one field that answers "has this ever run", which the learned-roster design
+    depends on.
+    """
+    stmt = pg_insert(ServiceSeen).values(
+        key=key, kind=kind, display_name=display_name, details=details,
+    )
+    stmt = stmt.on_conflict_do_update(
+        index_elements=[ServiceSeen.key],
+        set_={
+            "kind": stmt.excluded.kind,
+            "display_name": stmt.excluded.display_name,
+            "details": stmt.excluded.details,
+            "last_seen_at": func.now(),
+        },
+    )
+    await session.execute(stmt)
+
+
+async def refresh_celery_roster(session: AsyncSession) -> None:
+    """Inspect the broker and record what answered. Never raises.
+
+    A failure here means the roster does not advance, and the rows going stale
+    is then a TRUE report about a broker nobody can reach. Letting the
+    exception out would instead break the health endpoint, which is the one
+    thing that must keep answering when the stack is unwell.
+    """
+    try:
+        grouped = await asyncio.wait_for(
+            asyncio.to_thread(_inspect_celery_sync),
+            timeout=INSPECT_TIMEOUT_SECONDS * 2,
+        )
+    except Exception:
+        log.warning("service roster: celery inspect failed; roster not refreshed", exc_info=True)
+        return
+
+    for queues, entry in grouped.items():
+        await touch_service(
+            session,
+            key="celery:" + ",".join(queues),
+            kind="celery",
+            display_name=role_display_name(queues),
+            details={
+                "queues": list(queues),
+                "hostnames": entry["hostnames"],
+                "replicas": len(entry["hostnames"]),
+                "active": entry["active"],
+            },
+        )
+
+
+async def refresh_if_stale(session: AsyncSession) -> None:
+    """Refresh the celery roster if nobody has for REFRESH_TTL_SECONDS.
+
+    Rate-limited by the data rather than by a lock: the gate is the newest
+    last_seen_at across the celery rows, which every web process can see. Two
+    processes racing through the gate costs one redundant inspect and writes
+    the same values twice, so the benign outcome needs no coordination to
+    prevent.
+    """
+    newest = (
+        await session.execute(
+            select(func.max(ServiceSeen.last_seen_at)).where(ServiceSeen.kind == "celery")
+        )
+    ).scalar_one_or_none()
+
+    if newest is not None:
+        age = (await session.execute(select(func.now()))).scalar_one() - newest
+        if age.total_seconds() < REFRESH_TTL_SECONDS:
+            return
+
+    await refresh_celery_roster(session)

ci-requirements.md

@@ -198,14 +198,21 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
     refresh from being undone.
   - **`pull: true` on the scheduled path only** is the mechanism: a moved base
     tag changes the `FROM` layer's cache key and everything above it rebuilds.
-    **It does not currently make the unmoved case free.** Measured on the first
+    It did not always make the unmoved case free. Measured on the first real
-    real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
+    fire (run 4934, 2026-08-30): every content step reported `CACHED` and the
-    the bases resolved to unchanged digests, yet all three `:latest` tags got a
+    bases resolved to unchanged digests, yet all three `:latest` tags got a NEW
-    NEW manifest digest, because buildkit mints a fresh image config per run and
+    manifest digest, because buildkit stamps a fresh image config per run and
-    republishes identical layers under it. So `:latest` is rewritten weekly
+    republishes identical layers under it — so `:latest` was rewritten weekly
-    whether or not anything changed, and `:c-<sha>` is handed a new manifest to
+    whether or not anything changed, and a digest change stopped meaning
-    diverge from on the same cadence — a digest change stops meaning anything.
+    anything (#3265).
-    Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
+  - **`SOURCE_DATE_EPOCH` is what makes it free.** Set on each build step from
+    `artifacts.sh epoch <artifact>` — the unix timestamp of the same commit
+    `revision` and `version` name, so all three are views of one `newest()`
+    lookup and cannot drift into disagreeing. With the config's `created` field
+    and history timestamps pinned to the content rather than to the wall clock,
+    identical source produces an identical manifest digest and the push is a
+    registry no-op. That restores the property the whole scheme rests on: a
+    channel tag's digest changes when, and only when, its content does.
     Separately not caught: a Debian package update inside the `apt-get install`
     layer while the base tag stands still — a lag rather than a hole, since the
     official python/cuda images rebuild with those updates baked in.

docker-compose.yml

@@ -133,6 +133,20 @@ services:
       CELERY_RESULT_BACKEND: redis://redis:6379/0
       SECRET_KEY: ${SECRET_KEY:-dev_secret_key_not_for_production_change_me}
       LOG_LEVEL: ${LOG_LEVEL:-INFO}
+      # First boot only. FabledCurator refuses to start until the credential
+      # encryption key at /images/secrets/credential_key.b64 exists, and
+      # refuses to create one unless told to — auto-creating is
+      # indistinguishable from a restore that lost ./images/secrets, where it
+      # would mint a key that decrypts nothing and leave an instance that looks
+      # healthy while every paywalled download fails.
+      #
+      # Passed through EXPLICITLY because a variable in `.env` is only used for
+      # ${...} interpolation; it does not reach the container unless it is
+      # named here. Defaulted to empty so the refusal stands for everyone who
+      # has not opted in — the app tests for exactly "1".
+      #
+      # Set it in .env for one `up`, then remove it. See .env.example.
+      CURATOR_BOOTSTRAP_NEW_KEY: ${CURATOR_BOOTSTRAP_NEW_KEY:-}
     volumes:
       - ./images:/images
       - ./import:/import

frontend/src/components/TopNav.vue

@@ -5,9 +5,13 @@
         <img src="/favicon.svg" alt="" class="fc-brand__glyph" width="22" height="22" />
         <span class="fc-brand__text">FabledCurator</span>
       </RouterLink>
-      <span class="fc-health" :title="health.label">
+      <RouterLink
+        :to="{ name: 'settings', query: { tab: 'system' } }"
+        class="fc-health" :title="health.label"
+        :aria-label="`System health: ${health.label}`"
+      >
         <v-icon size="x-small" :color="health.color">{{ health.icon }}</v-icon>
-      </span>
+      </RouterLink>
       <PipelineStatusChip />
     </div>
 
@@ -64,13 +68,15 @@
 </template>
 
 <script setup>
-import { computed, onBeforeUnmount, onMounted, ref } from 'vue'
+import { computed, onBeforeUnmount, onMounted, onUnmounted, ref } from 'vue'
 import { useRoute } from 'vue-router'
 import router, { FRONT_DOOR } from '../router.js'
 import { useSystemStore } from '../stores/system.js'
+import { useSystemHealthStore } from '../stores/systemHealth.js'
 import PipelineStatusChip from './PipelineStatusChip.vue'
 
 const system = useSystemStore()
+const healthStore = useSystemHealthStore()
 
 // Publish the nav's REAL height as --fc-nav-h so full-height workspaces
 // (Explore/Subscriptions) and sticky sub-headers pin to it exactly instead of a
@@ -116,15 +122,55 @@ const settingsRoute = computed(() =>
   navRoutes.value.find(r => r.name === 'settings') || null
 )
 
+// The dot beside the brand, and the only ambient signal that something in the
+// stack has stopped (milestone 365).
+//
+// It used to read /api/health — a no-DB liveness check that proves the WEB
+// container is serving and nothing else. Green there while the worker was dead
+// is exactly what it looked like, and a green dot next to the product name is
+// read as "everything is fine". It now reflects the whole-stack verdict.
+//
+// Deliberately re-using this element rather than adding a second indicator:
+// there were already three partial surfaces (this, the pipeline chip, the
+// Settings Activity tab) and a fourth would have made the question harder to
+// answer, not easier. This is the one that already occupied the slot.
 const health = computed(() => {
-  if (system.healthy === null) {
+  const overall = healthStore.overall
+  if (overall === null) {
     return { icon: 'mdi-circle-outline', color: 'on-surface', label: 'checking…' }
   }
-  if (system.healthy === true) {
+  if (overall === 'ok') {
-    return { icon: 'mdi-circle', color: 'success', label: 'healthy' }
+    return { icon: 'mdi-circle', color: 'success', label: 'All parts running' }
   }
-  return { icon: 'mdi-alert-circle', color: 'error', label: 'unreachable' }
+  // Name what is wrong in the tooltip. "Something is unhealthy" sends someone
+  // hunting; "Scheduler has not checked in for 6 min" does not.
+  const worst = healthStore.problems[0]
+  const others = healthStore.problems.length - 1
+  const suffix = others > 0 ? ` (+${others} more)` : ''
+  if (overall === 'down') {
+    return {
+      icon: 'mdi-alert-circle', color: 'error',
+      label: (worst?.detail || 'A part has stopped') + suffix,
+    }
+  }
+  if (overall === 'stale') {
+    return {
+      icon: 'mdi-alert', color: 'warning',
+      label: (worst?.detail || 'A part is quiet') + suffix,
+    }
+  }
+  return { icon: 'mdi-help-circle-outline', color: 'on-surface', label: 'Health unknown' }
 })
+
+const HEALTH_POLL_MS = 15_000
+let healthTimer = null
+onMounted(() => {
+  healthStore.refresh()
+  healthTimer = setInterval(() => {
+    if (!document.hidden) healthStore.refresh()
+  }, HEALTH_POLL_MS)
+})
+onUnmounted(() => { if (healthTimer) clearInterval(healthTimer) })
 </script>
 
 <style scoped>
@@ -237,7 +283,14 @@ const health = computed(() => {
   display: flex;
   align-items: center;
   flex-shrink: 0;
+  /* A RouterLink since milestone 365 — it is the path to the Settings System
+     tab, not just an indicator. Reset the anchor so turning a span into a
+     link changed nothing about how the nav reads. */
+  text-decoration: none;
+  color: inherit;
+  border-radius: 50%;
 }
+.fc-health:hover { background: rgb(var(--v-theme-on-surface) / 0.12); }
 .fc-nav-right {
   flex: 1 1 0;
   min-width: 0;

frontend/src/components/settings/SystemHealthTab.vue

@@ -0,0 +1,128 @@
+<template>
+  <!-- A Settings tab, not a page of its own (operator 2026-09-02): the first
+       cut hung this off the health dot alone, which is a target you have to
+       already suspect something to look for. Settings is where someone goes
+       to ask the instance about itself, so it lives beside Activity. -->
+  <div>
+    <div class="d-flex align-center mb-1">
+      <v-spacer />
+      <span class="fc-sys__checked">
+        {{ store.checkedAt ? `checked ${formatRelative(store.checkedAt)}` : 'checking…' }}
+      </span>
+    </div>
+
+    <p class="fc-sys__lede text-body-2 mb-5">
+      Every moving part of FabledCurator and whether it is still checking in.
+      Parts are learned as they appear, so anything that has run at least once
+      stays listed — that is what lets a stopped one be noticed rather than
+      simply vanishing.
+    </p>
+
+    <v-alert
+      v-if="store.lastError" type="error" variant="tonal" density="compact" class="mb-4"
+    >
+      Could not reach FabledCurator: {{ store.lastError }}
+    </v-alert>
+
+    <v-card v-else variant="flat" class="fc-sys__card">
+      <div v-if="!store.parts.length" class="pa-6 text-center fc-sys__muted">
+        Still gathering — this fills in on the first check.
+      </div>
+
+      <div
+        v-for="part in store.parts" :key="part.key"
+        class="fc-sys__row" :class="`fc-sys__row--${part.state}`"
+      >
+        <span class="fc-sys__dot" :class="`fc-sys__dot--${part.state}`" />
+
+        <div class="fc-sys__body">
+          <div class="fc-sys__name">
+            {{ part.name }}
+            <span class="fc-sys__kind">{{ kindLabel(part.kind) }}</span>
+          </div>
+          <!-- The sentence, not just a chip. At the moment someone is deciding
+               whether to go and open Portainer, "has not checked in for 6 min"
+               is the thing that answers them. -->
+          <div class="fc-sys__detail">{{ part.detail }}</div>
+        </div>
+
+        <div class="fc-sys__meta">
+          <div v-if="part.last_seen_at" :title="part.last_seen_at">
+            seen {{ formatRelative(part.last_seen_at) }}
+          </div>
+          <div v-if="part.latency_ms != null">{{ part.latency_ms }} ms</div>
+          <div v-if="part.queues?.length" class="fc-sys__queues">{{ part.queues.join(', ') }}</div>
+        </div>
+      </div>
+    </v-card>
+
+    <p v-if="store.thresholds" class="fc-sys__foot text-caption mt-4">
+      A part is called stale after
+      {{ Math.round(store.thresholds.stale_after_seconds / 60) }} min without a
+      check-in and treated as stopped after
+      {{ Math.round(store.thresholds.down_after_seconds / 60) }} min. The window
+      is deliberately wide: a rolling deploy briefly runs two of a service and
+      then neither, and an indicator that reddened on every update would stop
+      being read.
+    </p>
+  </div>
+</template>
+
+<script setup>
+import { onMounted } from 'vue'
+
+import { useSystemHealthStore } from '../../stores/systemHealth.js'
+import { formatRelative } from '../../utils/date.js'
+
+const store = useSystemHealthStore()
+
+function kindLabel(kind) {
+  if (kind === 'celery') return 'background worker'
+  if (kind === 'agent') return 'GPU agent'
+  if (kind === 'datastore') return 'datastore'
+  return kind
+}
+
+// No timer of its own. TopNav already polls this same pinia store every 15s
+// for the health dot, and it is mounted on every route this tab is reachable
+// from — a second interval here would just double the request rate for a 5s
+// freshness gain. v-window keeps a visited item MOUNTED (hidden, not
+// destroyed), so a local timer would also have kept firing behind Maintenance.
+// One refresh on open, so arriving at the tab doesn't wait out the nav's tick.
+onMounted(() => { store.refresh() })
+</script>
+
+<style scoped>
+.fc-sys__lede, .fc-sys__muted, .fc-sys__checked, .fc-sys__foot {
+  color: rgb(var(--v-theme-on-surface) / 0.66);
+}
+.fc-sys__checked { font-size: 0.78rem; }
+.fc-sys__card { background: rgb(var(--v-theme-on-surface) / 0.04); }
+
+.fc-sys__row {
+  display: flex; align-items: center; gap: 12px;
+  padding: 12px 16px;
+  border-bottom: 1px solid rgb(var(--v-theme-on-surface) / 0.08);
+}
+.fc-sys__row:last-child { border-bottom: 0; }
+
+.fc-sys__dot { width: 9px; height: 9px; border-radius: 50%; flex: 0 0 auto; }
+.fc-sys__dot--ok      { background: rgb(var(--v-theme-success)); }
+.fc-sys__dot--stale   { background: rgb(var(--v-theme-warning)); }
+.fc-sys__dot--down    { background: rgb(var(--v-theme-error)); }
+.fc-sys__dot--unknown { background: rgb(var(--v-theme-on-surface) / 0.35); }
+
+.fc-sys__body { min-width: 0; flex: 1 1 auto; }
+.fc-sys__name { font-weight: 600; }
+.fc-sys__kind {
+  margin-left: 8px; font-weight: 400; font-size: 0.72rem; text-transform: uppercase;
+  letter-spacing: 0.04em; color: rgb(var(--v-theme-on-surface) / 0.5);
+}
+.fc-sys__detail { font-size: 0.82rem; color: rgb(var(--v-theme-on-surface) / 0.72); }
+
+.fc-sys__meta {
+  text-align: right; font-size: 0.75rem; flex: 0 0 auto;
+  font-variant-numeric: tabular-nums; color: rgb(var(--v-theme-on-surface) / 0.6);
+}
+.fc-sys__queues { opacity: 0.75; }
+</style>

frontend/src/router.js

@@ -45,6 +45,13 @@ const routes = [
 
   // Settings — config, pinned to the right of the nav (TopNav special-cases it).
   { path: '/settings', name: 'settings', component: SettingsView, meta: { title: 'Settings', stickyChrome: true } },
+  // System health is a Settings TAB, not a route of its own (operator
+  // 2026-09-02). It first shipped as /system reachable only from the health
+  // dot, which is a target you have to already suspect something to go
+  // looking for. Settings is where someone goes to ask the instance about
+  // itself. The path stays as a redirect so the dot's old link, and any
+  // bookmark from that build, still land somewhere real.
+  { path: '/system', name: 'system', redirect: () => ({ name: 'settings', query: { tab: 'system' } }) },
 
   // The old standalone paths now redirect into the Browse hub, preserving any
   // deep-link query (e.g. /posts?post_id=N → /browse?tab=posts&post_id=N). The

frontend/src/stores/system.js

@@ -4,7 +4,12 @@ import { useApi } from '../composables/useApi.js'
 
 export const useSystemStore = defineStore('system', () => {
   const api = useApi()
-  const healthy = ref(null)  // null=unknown, true=ok, false=down
+  // NOT what the nav dot reads any more (milestone 365): that is the
+  // whole-stack verdict in systemHealth.js. /api/health only proves the web
+  // container is serving, which is why a green dot here sat happily beside a
+  // dead worker. refreshHealth() is still called — it is also how build/version
+  // info arrives — so this stays as its by-product rather than its purpose.
+  const healthy = ref(null)
   // What the instance says it is. Since milestone 318 stopped publishing
   // version image tags, this is the only answer to "which build is this?" —
   // there is no registry name left to check it against.

frontend/src/stores/systemHealth.js

@@ -0,0 +1,49 @@
+import { defineStore } from 'pinia'
+import { computed, ref } from 'vue'
+
+import { useApi } from '../composables/useApi.js'
+
+// Whole-stack health: is every part of FabledCurator running (milestone 365)?
+//
+// Distinct from `system.js`, which polls /api/health — a no-DB liveness check
+// that only proves the web container is serving. That endpoint answers "can I
+// reach the API"; this one answers "is anything broken", which is the question
+// a green dot beside the brand was already being read as answering.
+//
+// Also distinct from `systemActivity.js`, which is about what the pipeline is
+// DOING — queue depths, running tasks, failures. Running and alive are
+// different questions and they fail independently: a perfectly idle stack with
+// a dead worker looks identical to a healthy one on the activity surfaces.
+export const useSystemHealthStore = defineStore('systemHealth', () => {
+  const api = useApi()
+
+  const overall = ref(null)      // null until the first answer: unknown ≠ ok
+  const parts = ref([])
+  const checkedAt = ref(null)
+  const thresholds = ref(null)   // server-owned, so the UI keeps no second copy
+  const lastError = ref(null)
+
+  async function refresh() {
+    try {
+      const body = await api.get('/api/system/health')
+      overall.value = body.overall
+      parts.value = body.parts || []
+      checkedAt.value = body.checked_at
+      thresholds.value = body.thresholds || null
+      lastError.value = null
+    } catch (e) {
+      // The endpoint is built never to fail because a dependency failed, so a
+      // throw here means the API itself is unreachable — which is its own kind
+      // of unhealthy and must not be shown as "ok".
+      lastError.value = e.message
+      overall.value = 'unknown'
+    }
+    return overall.value
+  }
+
+  // The parts worth naming in a tooltip — everything that is not ok, worst
+  // first. The endpoint already sorts that way.
+  const problems = computed(() => parts.value.filter(p => p.state !== 'ok'))
+
+  return { overall, parts, checkedAt, thresholds, lastError, problems, refresh }
+})

frontend/src/views/SettingsView.vue

@@ -14,6 +14,7 @@
       style="position: sticky; top: var(--fc-nav-h, 64px); z-index: 4;"
     >
       <v-tab value="overview">Overview</v-tab>
+      <v-tab value="system">System</v-tab>
       <v-tab value="activity">Activity</v-tab>
       <v-tab value="cleanup">Cleanup</v-tab>
       <v-tab value="maintenance">Maintenance</v-tab>
@@ -42,6 +43,13 @@
         </v-alert>
       </v-window-item>
 
+      <!-- Is every part of the stack still running (milestone 365). Sits
+           beside Activity deliberately: Activity answers "what is the queue
+           doing", this answers "is anything left to do it". -->
+      <v-window-item value="system">
+        <SystemHealthTab />
+      </v-window-item>
+
       <v-window-item value="activity">
         <SystemActivityTab @open-maintenance="tab = 'maintenance'" />
       </v-window-item>
@@ -73,18 +81,24 @@
 </template>
 
 <script setup>
-import { onMounted, onUnmounted, ref, watch } from 'vue'
+import { onMounted, onUnmounted, watch } from 'vue'
 import { useSystemStore } from '../stores/system.js'
 import SystemStatsCards from '../components/settings/SystemStatsCards.vue'
 import SystemActivitySummary from '../components/settings/SystemActivitySummary.vue'
 import SystemActivityTab from '../components/settings/SystemActivityTab.vue'
+import SystemHealthTab from '../components/settings/SystemHealthTab.vue'
 import GpuActivityPanel from '../components/settings/GpuActivityPanel.vue'
 import DownloadsActivityPanel from '../components/settings/DownloadsActivityPanel.vue'
 import MaintenancePanel from '../components/settings/MaintenancePanel.vue'
 import CleanupView from './CleanupView.vue'
+import { useTabQuery } from '../composables/useTabQuery.js'
 import { useMLStore } from '../stores/ml.js'
 
-const tab = ref('overview')
+// ?tab= sync (the same composable Browse/Subscriptions use) so a tab can be
+// linked TO — the health dot beside the brand points at ?tab=system, and the
+// old /system path redirects there.
+const VALID_TABS = ['overview', 'system', 'activity', 'cleanup', 'maintenance']
+const { tab } = useTabQuery(VALID_TABS, 'overview')
 const system = useSystemStore()
 const mlStore = useMLStore()
 

frontend/test/router.spec.js

@@ -40,6 +40,14 @@ describe('router', () => {
     expect(router.currentRoute.value.query.post_id).toBe('7')
   })
 
+  it('/system redirects into the Settings System tab', async () => {
+    // It shipped as a standalone page for one build; the health dot and any
+    // bookmark from it must still land on the surface, which is now a tab.
+    await router.push('/system')
+    expect(router.currentRoute.value.name).toBe('settings')
+    expect(router.currentRoute.value.query.tab).toBe('system')
+  })
+
   it('series-read is an immersive route', () => {
     const r = router.resolve('/series/5/read')
     expect(r.name).toBe('series-read')

scripts/artifacts.sh

@@ -90,7 +90,7 @@ DERIVER='scripts/artifacts.sh'
 
 
 usage() {
-  echo "usage: artifacts.sh {paths|revision|version} {web|ml|agent|extension}" >&2
+  echo "usage: artifacts.sh {paths|revision|version|epoch} {web|ml|agent|extension}" >&2
   exit 2
 }
 
@@ -149,6 +149,26 @@ cmd_revision() {
   echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
 }
 
+# The BUILD CLOCK: the same commit's unix timestamp, for SOURCE_DATE_EPOCH.
+#
+# buildkit stamps the image config's `created` field and every history entry
+# with the wall clock of the build unless this is set, so two builds of
+# identical source produce different config blobs and therefore different
+# manifest digests. That is #3265: the weekly refresh republished all three
+# `:latest` tags on 2026-08-30 with every content step CACHED and the bases
+# resolved to unchanged digests — nothing was different, and the digest moved
+# anyway. A digest that changes on a calendar cannot also mean "the content
+# changed", which is the only thing anyone wants it for.
+#
+# It is the same commit `revision` and `version` name — deliberately, and this
+# is the point of routing it through `newest()` rather than taking git's word
+# separately. Three values derived from three lookups can disagree; three
+# views of one lookup cannot. Note #3127 §2 is the record of what a second
+# clock costs.
+cmd_epoch() {
+  echo "$(newest "$1")" | cut -d' ' -f1
+}
+
 # The VERSION: `YYYY.MM.DD.HHMM`, zero-padded, UTC. One shape across the whole
 # family (note #3127 §1, rule 148) — the number an instance reports about
 # itself, and, with a `v` in front, the release tag naming the same build.
@@ -197,5 +217,6 @@ case "$1" in
   paths)    cmd_paths "$2" ;;
   revision) cmd_revision "$2" ;;
   version)  cmd_version "$2" ;;
+  epoch)    cmd_epoch "$2" ;;
   *)        usage ;;
 esac

scripts/release_notes.py

@@ -33,6 +33,32 @@ history. Ancestry is immune to the shape change, and it is also the more honest
 question: "what is in this that was not in the last one" IS a reachability
 question.
 
+Ancestry alone is not enough, though, and milestone 328 is where that showed.
+The 28 `v26.*` tags are still in the repo — the operator kept them as history
+when their releases were deleted — so `--match v*` walks straight back to
+`v26.06.04.0` and reports 533 commits. That span is not a changelog: nobody has
+run `v26.06.04.0`, its release page no longer exists to compare against, and
+the 200 lines that survive truncation are precisely the internal build-out that
+milestone 328 exists to stop shipping. So the match is `v[0-9][0-9][0-9][0-9].*`
+— rule 148's four-digit-year shape — which is exactly the set of tags that name
+a release a reader could have been running. A pre-convention tag is history,
+not a predecessor.
+
+## The first release has no changelog, and should not pretend to
+
+Once the match is narrowed, the first rule-148 tag reaches no predecessor at
+all, and the old fallback — diff against the whole history — is worse than the
+problem it replaced. The honest content for a release nobody has a previous
+version of is what the thing IS.
+
+So a release with no reachable predecessor renders the product overview instead
+of a commit list. It is read out of README.md between `<!-- overview:start -->`
+and `<!-- overview:end -->` rather than written here, for the same reason the
+changelog is derived: two hand-maintained descriptions of one product drift,
+and nothing ever catches it. The release page and the repo front page are one
+source. Every later release goes back to being a changelog, which is what §5 of
+note #3127 says a release is for.
+
 ## Re-runs update, they do not fall through
 
 Note #3127 §6.7: a publisher that POSTs and recovers the id from a `409` never
@@ -91,18 +117,45 @@ def git_ok(*args: str) -> str | None:
 
 
 def previous_tag(ref: str, tag: str | None) -> str | None:
-    """The most recent `v*` tag reachable from `ref`, excluding `tag` itself.
+    """The most recent rule-148 tag reachable from `ref`, excluding `tag` itself.
 
     `--exclude` rather than `<ref>^` so this is the same call whether or not
     `ref` is the tag being released — and so it does not blow up on a root
     commit that has no parent to walk to.
+
+    The glob deliberately does NOT match the old `v26.*` tags. They are kept as
+    history and their releases are gone, so naming one as the predecessor emits
+    a span nobody can look up. See the module docstring.
     """
-    args = ["describe", "--tags", "--abbrev=0", "--match", "v*"]
+    args = ["describe", "--tags", "--abbrev=0", "--match", "v[0-9][0-9][0-9][0-9].*"]
     if tag:
         args += ["--exclude", tag]
     return git_ok(*args, ref)
 
 
+def product_overview() -> str | None:
+    """The product description, lifted verbatim from README.md.
+
+    Returns None if the markers are absent or empty — a missing overview is
+    reported as a note and the release still publishes, on the same reasoning
+    as cross_checks(): the release is the useful object even when one part of
+    the derivation could not run.
+    """
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    try:
+        with open(os.path.join(root, "README.md"), encoding="utf-8") as fh:
+            readme = fh.read()
+    except OSError:
+        return None
+    match = re.search(
+        r"<!--\s*overview:start\s*-->(.*?)<!--\s*overview:end\s*-->",
+        readme, re.S,
+    )
+    if not match:
+        return None
+    return match.group(1).strip() or None
+
+
 def commits(previous: str | None, ref: str) -> list[str]:
     """The subjects between the previous release and this one.
 
@@ -125,7 +178,10 @@ def truncate(log: list[str]) -> tuple[list[str], str | None]:
     )
 
 
-def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list[str]) -> str:
+def render(
+    tag: str, sha: str, previous: str | None, log: list[str], notes: list[str],
+    overview: str | None,
+) -> str:
     short = sha[:7]
     parts = []
 
@@ -135,6 +191,25 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
         # is the failure this whole milestone is about.
         parts.append("\n".join(f"> **Note:** {n}" for n in notes))
 
+    # No predecessor means nobody reading this has run an earlier one, so the
+    # release describes the product rather than a diff. The overview is
+    # README.md's own words — see the module docstring on why it is not
+    # written here.
+    if previous is None and overview:
+        parts.append(overview)
+        parts.append(
+            "## Installing\n\n"
+            "```\ncurl -O https://git.fabledsword.com/bvandeusen/FabledCurator/raw/"
+            f"tag/{tag}/docker-compose.yml\ncurl -O https://git.fabledsword.com/"
+            f"bvandeusen/FabledCurator/raw/tag/{tag}/.env.example\n"
+            "mv .env.example .env   # then set SECRET_KEY, DB_PASSWORD\n"
+            "docker compose -f docker-compose.yml up -d\n```\n\n"
+            "**Read \"Before you expose it\" in the README first.** FabledCurator "
+            "has no login, and it stores live platform session cookies for "
+            "accounts that usually have a payment method attached. Bind it to a "
+            "network you trust."
+        )
+
     parts.append(
         f"Built from `{short}`. The rollback unit is the immutable `:c-` tag "
         f"(rule 145) — these three move together:\n\n```\n"
@@ -142,7 +217,19 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
         + "\n```"
     )
 
-    heading = f"## Changes since {previous}" if previous else "## Changes"
+    if previous is None:
+        # Deliberately NOT a commit list. The alternative is the whole history
+        # truncated to MAX_COMMITS, which is 200 lines of internal build-out
+        # presented to someone who has never seen this project.
+        parts.append(
+            "---\n\n_First release under rule 148's `vYYYY.MM.DD.HHMM` shape, so "
+            "there is no predecessor to diff against and no changelog to derive. "
+            "The description above is README.md's, quoted at publish time. Later "
+            "releases carry the commits since the previous one._"
+        )
+        return "\n\n".join(parts)
+
+    heading = f"## Changes since {previous}"
     if log:
         parts.append(heading + "\n\n" + "\n".join(f"- {line}" for line in log))
     else:
@@ -152,10 +239,9 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
             "names the same source under a new name._"
         )
 
-    span = f"{previous}..{tag}" if previous else tag
     parts.append(
-        f"---\n\n_Derived at publish time from `git log --no-merges {span}`. "
+        f"---\n\n_Derived at publish time from "
-        f"Nothing here is hand-maintained._"
+        f"`git log --no-merges {previous}..{tag}`. Nothing here is hand-maintained._"
     )
     return "\n\n".join(parts)
 
@@ -289,13 +375,31 @@ def main() -> None:
     for note in notes:
         print(f"release: NOTE {note}")
 
+    # A first release renders the overview instead of a changelog, so the
+    # commit walk is skipped entirely rather than computed and discarded —
+    # `commits(None, ref)` is the whole history and there is no reason to ask
+    # for it.
+    overview = None
+    log: list[str] = []
+    if previous is None:
+        overview = product_overview()
+        if overview is None:
+            note = (
+                "No `<!-- overview:start -->` block found in README.md, so this "
+                "first release has no product description. Published anyway; add "
+                "the markers and re-run the workflow to fill it in."
+            )
+            print(f"release: NOTE {note}")
+            notes.append(note)
+        print("release: no rule-148 predecessor — rendering the product overview")
+    else:
         log = commits(previous, ref)
         print(f"release: {len(log)} non-merge commits in the span")
         log, overflow = truncate(log)
         if overflow:
             print(f"release: NOTE {overflow}")
             notes.append(overflow)
-    body = render(tag or ref, sha, previous, log, notes)
+    body = render(tag or ref, sha, previous, log, notes, overview)
 
     if args.dry_run or not tag:
         print("--- body ---")

scripts/smoke_image.py

@@ -0,0 +1,155 @@
+"""Prove a freshly built image can still do the things its OS packages provide.
+
+Run INSIDE the image, not against the source tree. That distinction is the
+entire reason this file exists.
+
+`ci.yml`'s lanes run on `ci-python:3.14` and install `requirements.txt`. A base
+refresh changes neither, so all five lanes stay green through a base bump that
+breaks the product. What a refresh actually re-resolves is this, from the
+Dockerfile:
+
+    RUN apt-get update && apt-get install -y --no-install-recommends \
+            ffmpeg unar libpq5 postgresql-client zstd megatools \
+            libjpeg62-turbo libwebp7 libpng16-16 ca-certificates
+
+Unpinned, every build. Nothing else in this repo looks at it.
+
+So the checks below run the APPLICATION'S OWN code — `Thumbnailer`, which needs
+no database and no app context — against whatever Pillow and ffmpeg have
+become. `ffmpeg -version` exiting 0 would pass while a codec removal or an
+soname bump broke every thumbnail in the library; producing a thumbnail would
+not.
+
+Every failure names the package it implicates. This fires on a Sunday,
+unattended, about a change nobody made deliberately — "assertion failed" a week
+later teaches nobody anything.
+
+Usage:  docker run --rm -i <image> shell -c 'python3 -' < scripts/smoke_image.py
+"""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+import tempfile
+from pathlib import Path
+
+try:
+    from PIL import Image
+
+    from backend.app.services.thumbnailer import Thumbnailer
+except Exception as exc:  # noqa: BLE001 — a smoke test reports, it never raises
+    print(f"smoke: FAILED — could not import the thumbnail path at all: {exc}")
+    print("  Implicates Pillow or its shared libraries (libjpeg62-turbo,")
+    print("  libpng16-16, libwebp7), or the python base image itself.")
+    raise SystemExit(1) from exc
+
+
+# Binary → what stops working without it. Listed individually because
+# `--no-install-recommends` means any one of them can vanish on its own when a
+# dependency chain higher up changes.
+REQUIRED_BINARIES = {
+    "ffmpeg": "video thumbnails and transcoding (Dockerfile: ffmpeg)",
+    "unar": "archive import — cbz/zip/rar members (Dockerfile: unar)",
+    "pg_dump": "database backup (Dockerfile: postgresql-client)",
+    "zstd": "backup compression, pg_dump | tar --zstd (Dockerfile: zstd)",
+    "megatools": "mega.nz public-link downloads, #830 (Dockerfile: megatools)",
+}
+
+
+def check_jpeg(thumbs: Thumbnailer, src: Path) -> None:
+    path = src / "flat.jpg"
+    Image.new("RGB", (900, 400), (30, 90, 160)).save(path, "JPEG")
+    result = thumbs.generate_image_thumbnail(path, "a" * 64)
+    assert result.mime == "image/jpeg", f"mime was {result.mime}"
+    assert result.path.stat().st_size > 0, "no bytes written"
+    # Re-open it. A file that writes but cannot be read back is the shape a
+    # half-broken codec produces, and size alone would not catch it.
+    with Image.open(result.path) as im:
+        im.load()
+
+
+def check_png_alpha(thumbs: Thumbnailer, src: Path) -> None:
+    path = src / "alpha.png"
+    Image.new("RGBA", (400, 900), (200, 40, 40, 128)).save(path, "PNG")
+    result = thumbs.generate_image_thumbnail(path, "b" * 64)
+    assert result.mime == "image/png", f"mime was {result.mime}"
+    with Image.open(result.path) as im:
+        im.load()
+        assert im.mode in ("RGBA", "LA", "P"), f"alpha lost, mode={im.mode}"
+
+
+def check_webp(thumbs: Thumbnailer, src: Path) -> None:
+    path = src / "sample.webp"
+    Image.new("RGB", (500, 500), (10, 140, 70)).save(path, "WEBP")
+    result = thumbs.generate_image_thumbnail(path, "c" * 64)
+    assert result.path.stat().st_size > 0, "no bytes written"
+
+
+def check_video(thumbs: Thumbnailer, src: Path) -> None:
+    # Synthesised rather than committed as a fixture: a checked-in video is a
+    # binary blob nobody can review, and lavfi ships with every ffmpeg build.
+    #
+    # 3 seconds, not 2. The seek lands at max(1.0, duration * 0.05) = 1.0s, and
+    # a clip barely longer than its own seek is how #1231 produced zero frames.
+    # This check exists to exercise ffmpeg, not to re-litigate that edge.
+    clip = src / "clip.mp4"
+    subprocess.run(
+        ["ffmpeg", "-nostdin", "-f", "lavfi", "-i", "testsrc=size=640x360:rate=10",
+         "-t", "3", "-pix_fmt", "yuv420p", "-y", str(clip)],
+        check=True, capture_output=True, timeout=120,
+    )
+    result = thumbs.generate_video_thumbnail(clip, "d" * 64, duration_seconds=3.0)
+    assert result.path.stat().st_size > 0, "no bytes written"
+    with Image.open(result.path) as im:
+        im.load()
+
+
+CHECKS = (
+    ("JPEG thumbnail", "libjpeg62-turbo / Pillow", check_jpeg),
+    ("PNG thumbnail (alpha)", "libpng16-16 / Pillow", check_png_alpha),
+    ("WebP decode", "libwebp7 / Pillow", check_webp),
+    ("video thumbnail", "ffmpeg", check_video),
+)
+
+
+def main() -> int:
+    failures: list[str] = []
+
+    print("smoke: binaries the apt layer provides")
+    for binary, purpose in REQUIRED_BINARIES.items():
+        if shutil.which(binary) is None:
+            print(f"  FAIL  {binary}: not on PATH")
+            failures.append(f"{binary} — {purpose}")
+        else:
+            print(f"  ok    {binary}")
+
+    print("smoke: the application's own thumbnail path, against this image's libraries")
+    with tempfile.TemporaryDirectory() as tmp:
+        root = Path(tmp)
+        src = root / "src"
+        src.mkdir()
+        thumbs = Thumbnailer(root)
+        for name, implicates, fn in CHECKS:
+            try:
+                fn(thumbs, src)
+                print(f"  ok    {name}")
+            except Exception as exc:  # noqa: BLE001 — report every check, then fail once
+                print(f"  FAIL  {name}: {exc}")
+                failures.append(f"{name} — {implicates}")
+
+    if failures:
+        print(f"\nsmoke: FAILED — {len(failures)} check(s)")
+        for failure in failures:
+            print(f"  - {failure}")
+        print("\nThis image was built against freshly resolved base layers. The")
+        print("named packages are where to look: compare this build's apt versions")
+        print("against the previous :latest before assuming the app changed.")
+        return 1
+
+    print("\nsmoke: all checks passed")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/test_artifact_identity.py

@@ -234,3 +234,56 @@ def test_version_and_revision_describe_the_same_commit(artifact):
             f"in AMO_UNPADDED may differ here."
         )
     assert sha.startswith(revision(artifact))
+
+
+@pytest.mark.parametrize("artifact", ARTIFACTS)
+def test_epoch_is_the_same_commit_the_version_names(artifact):
+    """The build clock and the version must be one lookup, not two.
+
+    `epoch` feeds SOURCE_DATE_EPOCH, which decides the image config's bytes and
+    therefore the manifest digest; `version` is what the instance reports about
+    itself. If they could name different commits, an image would be stamped
+    reproducibly against one commit while claiming to be another — and both
+    values would look perfectly well-formed, exactly like the divergence the
+    test above guards.
+
+    They cannot, because `cmd_epoch` and `cmd_version` are two fields of one
+    `newest()` result. This pins that they stay that way: a future refactor
+    that gave epoch its own `git log` would pass every other test here.
+    """
+    epoch = artifacts("epoch", artifact).strip()
+    assert epoch.isdigit(), f"{artifact} epoch is {epoch!r}, not a unix timestamp"
+
+    sha = newest_by_commit_time(artifact)
+    committed = subprocess.run(
+        ["git", "show", "-s", "--format=%ct", sha],
+        capture_output=True, text=True, check=True, cwd=ROOT,
+    ).stdout.strip()
+    assert epoch == committed, (
+        f"{artifact} derives epoch {epoch}, but its newest shipped commit "
+        f"{sha[:12]} was committed at {committed}. SOURCE_DATE_EPOCH would "
+        f"pin the image config to a commit the version does not name."
+    )
+
+    # And the two renderings must agree, which is the property that actually
+    # matters at build time: same commit in, same digest and same reported
+    # version out.
+    rendered = subprocess.run(
+        ["git", "show", "-s", "--format=%cd", "--date=format-local:%Y.%m.%d.%H%M", sha],
+        capture_output=True, text=True, check=True, cwd=ROOT,
+        env={"TZ": "UTC", "PATH": os.environ.get("PATH", "")},
+    ).stdout.strip()
+    assert segments(artifacts("version", artifact).strip()) == segments(rendered)
+
+
+def test_epoch_is_stable_across_calls():
+    """SOURCE_DATE_EPOCH's entire job is to be the same on the next build.
+
+    A value that moved between two invocations on one unchanged checkout would
+    reintroduce #3265 through the very mechanism meant to close it, and the
+    symptom would be indistinguishable: a digest that changes for no reason.
+    """
+    for artifact in ARTIFACTS:
+        first = artifacts("epoch", artifact).strip()
+        second = artifacts("epoch", artifact).strip()
+        assert first == second, f"{artifact} epoch moved: {first} then {second}"

tests/test_release_notes.py

@@ -24,12 +24,36 @@ SCRIPT = ROOT / "scripts" / "release_notes.py"
 
 
 def notes(*args: str, cwd: Path | None = None) -> str:
+    """Run the script the way release.yml does.
+
+    A synthetic repo runs its OWN copy of the script, because the overview is
+    read relative to `__file__` rather than to the cwd — which is right in
+    production (release.yml checks out the tag, so the script IS the tagged
+    tree's copy) and would otherwise make every synthetic repo silently quote
+    FabledCurator's real README.
+    """
+    root = cwd or ROOT
+    script = root / "scripts" / "release_notes.py"
     return subprocess.run(
-        ["python3", str(SCRIPT), "--dry-run", *args],
+        ["python3", str(script if script.exists() else SCRIPT), "--dry-run", *args],
-        capture_output=True, text=True, check=True, cwd=cwd or ROOT,
+        capture_output=True, text=True, check=True, cwd=root,
     ).stdout
 
 
+OVERVIEW_TEXT = "A synthetic product, described once."
+
+
+def install_script(repo: Path, *, overview: bool = True) -> None:
+    """Give a synthetic repo the script and a README to quote."""
+    (repo / "scripts").mkdir(exist_ok=True)
+    (repo / "scripts" / "release_notes.py").write_text(SCRIPT.read_text())
+    (repo / "scripts" / "artifacts.sh").write_text("#!/bin/sh\nexit 1\n")
+    readme = "# Synthetic\n\n"
+    if overview:
+        readme += f"<!-- overview:start -->\n{OVERVIEW_TEXT}\n<!-- overview:end -->\n"
+    (repo / "README.md").write_text(readme)
+
+
 def body_of(out: str) -> str:
     assert "--- body ---" in out, f"no body was rendered:\n{out}"
     return out.split("--- body ---", 1)[1]
@@ -56,9 +80,10 @@ def shaped_history(tmp_path: Path) -> Path:
     repo = tmp_path / "shaped"
     repo.mkdir()
     git(repo, "init", "-q", "-b", "main")
+    install_script(repo)
     for i, tag in enumerate(("v26.06.04.0", "v2026.08.28.2208", "v2026.08.29.1000")):
         (repo / "f.txt").write_text(f"{i}\n")
-        git(repo, "add", "f.txt")
+        git(repo, "add", "-A")
         git(repo, "commit", "-q", "-m", f"work landing in {tag}")
         git(repo, "tag", tag)
     # One more commit and a merge, so the merge-exclusion test has something to
@@ -109,12 +134,57 @@ def test_merges_are_excluded_so_the_list_is_the_work(shaped_history):
     assert "Merge pull request #999" not in body
 
 
-def test_the_first_release_still_renders_with_nothing_behind_it(shaped_history):
+def test_a_pre_convention_tag_is_history_not_a_predecessor(shaped_history):
-    """No previous tag is reachable from the oldest one. That is a real state,
+    """The defect milestone 328 hit, and the reason the match glob narrowed.
-    not an error, and it must not take the release down with it."""
+
-    out = notes("v26.06.04.0", cwd=shaped_history)
+    The 28 `v26.*` tags are kept as history while their releases were deleted.
+    Ancestry alone happily names `v26.06.04.0` as the predecessor of the first
+    rule-148 tag — and then the body offers "changes since" a release that no
+    longer exists, over a span (533 commits in the real repo) that is the
+    internal build-out this milestone exists to stop publishing.
+
+    Reachable is not the same as comparable. Only a `vYYYY.` tag names a
+    release a reader could have been running.
+    """
+    out = notes("v2026.08.28.2208", cwd=shaped_history)
     assert "previous=<none>" in out
-    assert "## Changes" in body_of(out)
+    assert "v26.06.04.0" not in body_of(out)
+
+
+def test_the_first_release_describes_the_product_instead_of_diffing(shaped_history):
+    """No predecessor means nobody reading has run an earlier version, so a
+    changelog has no referent. The alternative the script used to take — diff
+    against the whole history, truncated — puts 200 lines of internal build-out
+    in front of someone meeting the project for the first time."""
+    body = body_of(notes("v2026.08.28.2208", cwd=shaped_history))
+    assert OVERVIEW_TEXT in body
+    assert "## Changes" not in body
+    assert not [ln for ln in body.split("\n") if ln.startswith("- work landing")]
+
+
+def test_the_overview_is_readmes_words_not_a_second_copy(shaped_history):
+    """Two hand-maintained descriptions of one product drift and nothing
+    catches it. The release page quotes README.md so there is one source."""
+    readme = (shaped_history / "README.md").read_text()
+    assert OVERVIEW_TEXT in readme
+    assert OVERVIEW_TEXT in body_of(notes("v2026.08.28.2208", cwd=shaped_history))
+
+
+def test_a_missing_overview_block_is_reported_and_still_publishes(tmp_path):
+    """Same reasoning as cross_checks(): the release is the useful object even
+    when part of the derivation could not run. Say what is missing, publish
+    anyway — do not leave the operator with a tag and no release."""
+    repo = tmp_path / "no-markers"
+    repo.mkdir()
+    git(repo, "init", "-q", "-b", "main")
+    install_script(repo, overview=False)
+    git(repo, "add", "-A")
+    git(repo, "commit", "-q", "-m", "first")
+    git(repo, "tag", "v2026.09.01.1200")
+
+    out = notes("v2026.09.01.1200", cwd=repo)
+    assert "No `<!-- overview:start -->` block found" in out
+    assert "No `<!-- overview:start -->` block found" in body_of(out)
 
 
 def test_a_non_tag_ref_renders_but_refuses_to_claim_it_published():
@@ -135,15 +205,29 @@ def test_the_rollback_refs_name_all_three_images():
         assert f"bvandeusen/{image}:c-" in body, f"{image} missing from the rollback refs"
 
 
-def test_an_unbounded_span_is_truncated_and_says_so():
+def test_a_long_span_between_two_releases_is_truncated_and_says_so(tmp_path):
-    """With no reachable previous tag the span is the whole history. Emitting
+    """The cap is still reachable, just not by the route it used to be.
-    eleven hundred lines would bury the one line explaining why there are
+
-    eleven hundred of them, so the cap is part of the message, not a silent
+    It no longer fires on "no predecessor" — that renders the overview now.
-    slice."""
+    What it still guards is two real releases far enough apart that the list
-    out = notes("HEAD")
+    stops being something anyone reads, which is the ordinary case for a
-    if "previous=<none>" not in out:
+    project that cuts a bookmark twice a year. The cap is part of the message,
-        pytest.skip("a previous tag is reachable from HEAD in this checkout")
+    not a silent slice.
+    """
+    repo = tmp_path / "long"
+    repo.mkdir()
+    git(repo, "init", "-q", "-b", "main")
+    install_script(repo)
+    git(repo, "add", "-A")
+    git(repo, "commit", "-q", "-m", "scaffold")
+    git(repo, "tag", "v2026.01.01.0000")
+    for i in range(205):
+        git(repo, "commit", "-q", "--allow-empty", "-m", f"fix: change {i}")
+    git(repo, "tag", "v2026.07.01.0000")
+
+    out = notes("v2026.07.01.0000", cwd=repo)
+    assert "previous=v2026.01.01.0000" in out
     body = body_of(out)
     listed = [ln for ln in body.split("\n") if ln.startswith("- ")]
-    assert len(listed) <= 200
+    assert len(listed) == 200
     assert "more than a changelog is for" in body