ci: emit the chain schema dump for local reconciliation work (#3275)

Reconciling the models against the deployed schema needs the actual
pg_dump, not an inference from the unified diff. Parsing table context
out of diff hunks drops every table whose CREATE TABLE line falls
outside a hunk — it under-reported 81 columns across 13 tables when the
real figure spans more, missing artist, gpu_job, download_event and
external_link entirely.

Same checksummed-base64 transport as the candidate baseline, for the
same reason: a plain cat of a file this size was silently truncated
mid-line by the runner on run 4964.

.forgejo/workflows/baseline.yml

@@ -0,0 +1,230 @@
+
+# TEMPORARY — milestone 328 steps 1-2. Delete once the baseline is stamped.
+#
+# Squashing 87 alembic revisions into one baseline has exactly one dangerous
+# failure: the generated baseline does not reproduce the schema the chain
+# produced, `alembic stamp` writes a version string anyway (it validates
+# NOTHING), and the divergence surfaces on the next real migration against the
+# operator's live data.
+#
+# So this workflow does the comparison in CI, where a pgvector Postgres already
+# gets built from the chain on every integration run, and nothing is at risk.
+# It answers one question: does `upgrade head` on the collapsed chain produce a
+# byte-identical schema to `upgrade head` on the 87-revision chain?
+#
+# The chain is read from git rather than from the working tree, so this keeps
+# working AFTER the old revisions are deleted — `chain_ref` names a commit that
+# still has them. That is what makes this the proof for step 1 and the
+# pre-flight for step 2, rather than a one-shot script.
+#
+# While the chain is still present it also autogenerates a candidate baseline
+# from the models and prints it. That is a starting point, NOT the answer:
+# autogenerate reads SQLAlchemy metadata, and three things here do not live
+# there —
+#   * CREATE EXTENSION vector           (0001)
+#   * CREATE EXTENSION tsm_system_rows  (0004)
+#   * the HNSW index on image_record.siglip_embedding, which is raw SQL
+#     because alembic's create_index cannot express `USING hnsw (...)`  (0036)
+# plus any CHECK constraint or server_default that a migration added without
+# the model declaring it. Those must be hand-added, and the diff below is what
+# proves none were missed.
+name: Alembic baseline
+
+on:
+  workflow_dispatch:
+    inputs:
+      chain_ref:
+        description: 'Commit/tag that still carries the full 0001..0087 chain'
+        type: string
+        default: '0a5bbe8'
+
+jobs:
+  compare:
+    runs-on: python-ci
+    container:
+      image: git.fabledsword.com/bvandeusen/ci-python:3.14
+    env:
+      DB_USER: fabledcurator
+      DB_PASSWORD: ci_integration
+      DB_PORT: "5432"
+      DB_NAME: fabledcurator_test
+      SECRET_KEY: ci_integration_placeholder
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        env:
+          POSTGRES_USER: fabledcurator
+          POSTGRES_PASSWORD: ci_integration
+          POSTGRES_DB: fabledcurator_test
+        options: >-
+          --health-cmd "pg_isready -U fabledcurator"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 10
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # Full history is the point: `chain_ref` is read out of git, so a
+          # shallow clone would not have the revisions to compare against.
+          fetch-depth: 0
+
+      - name: Resolve the Postgres service and install deps
+        run: |
+          set -eux
+          # Same service-IP dance as ci.yml's integration job; see the long
+          # comment there for why the job name must stay separator-free.
+          PG=$(docker ps --filter "name=compare" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
+          test -n "$PG"
+          PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
+          test -n "$PG_IP"
+          echo "PG_CONTAINER=$PG" >> "$GITHUB_ENV"
+          echo "DB_HOST=$PG_IP" >> "$GITHUB_ENV"
+          for i in $(seq 1 60); do
+            (echo > "/dev/tcp/$PG_IP/5432") >/dev/null 2>&1 && break
+            sleep 2
+          done
+          if command -v uv >/dev/null 2>&1; then
+            uv pip install --system -r requirements.txt
+          else
+            pip install -r requirements.txt
+          fi
+
+      # DB 1: the 87-revision chain, read out of git at `chain_ref`.
+      #
+      # A git worktree rather than a checkout, so the current tree — which is
+      # what we are testing — is left completely alone.
+      - name: Build the schema the OLD chain produces
+        env:
+          CHAIN_REF: ${{ github.event.inputs.chain_ref }}
+        run: |
+          set -eux
+          docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_chain
+          git worktree add /tmp/chain "$CHAIN_REF"
+          ls /tmp/chain/alembic/versions/*.py | wc -l
+          cd /tmp/chain
+          DB_NAME=fc_chain alembic upgrade head
+          cd -
+          docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
+            --no-owner --no-privileges -d fc_chain > chain.sql
+          wc -l chain.sql
+          # Emit the dump itself, checksummed, for local analysis. Reconciling
+          # the models against the deployed schema (#3275) needs the ACTUAL
+          # schema, not an inference from a diff — parsing table context out of
+          # unified-diff hunks drops every table whose CREATE TABLE line falls
+          # outside a hunk, which silently under-reports.
+          #
+          # base64 + sha256 for the same reason as the candidate: a plain cat
+          # of a file this size was truncated mid-line by the runner with the
+          # step still green (run 4964).
+          set +x
+          B64=$(base64 -w 120 chain.sql)
+          echo "===== BEGIN CHAIN SCHEMA (base64) ====="
+          echo "$B64"
+          echo "===== END CHAIN SCHEMA ====="
+          echo "chain-sha256: $(sha256sum chain.sql | cut -d' ' -f1)"
+          echo "chain-bytes:  $(wc -c < chain.sql)"
+          set -x
+
+      # A candidate baseline, autogenerated from the models against an EMPTY
+      # database so every table shows up as a create. Printed for a human to
+      # finish — it will be missing the three raw-SQL items named at the top.
+      #
+      # Gated on the TREE, not on a workflow input. A `type: boolean` input
+      # read back as `github.event.inputs.generate == 'true'` silently
+      # evaluated false on this runner (run 4960 skipped this step entirely
+      # with no diagnostic) — the same `github.event.inputs` typing quirk
+      # build.yml already works around. The file count is the real question
+      # anyway: there is nothing to generate once the chain is collapsed.
+      - name: Autogenerate a candidate baseline
+        run: |
+          set -eux
+          if [ "$(ls alembic/versions/*.py | wc -l)" -le 1 ]; then
+            echo "already collapsed — nothing to generate"
+            exit 0
+          fi
+          docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_gen
+          # Hide the existing revisions so alembic sees an empty history and
+          # emits the whole schema rather than a delta.
+          mkdir -p /tmp/versions_held
+          mv alembic/versions/*.py /tmp/versions_held/ 2>/dev/null || true
+          DB_NAME=fc_gen alembic revision --autogenerate -m "baseline" || true
+          # Printed rather than uploaded: ci-requirements.md records that this
+          # runner cannot do actions/upload-artifact@v4+, and the repo dropped
+          # the action entirely in 2026-05, so the job log is the retrieval
+          # channel actually proven here.
+          #
+          # base64, not the raw file. A plain `cat` of the ~33KB candidate was
+          # TRUNCATED MID-LINE by the runner on run 4964 — it stopped inside
+          # `sa.Column('mime', sa.String(length=128)` and carried straight on
+          # to the next traced command, with the step still green. A silent
+          # cut in the middle of a schema definition is the worst possible
+          # failure here, because the truncated text still looks like a
+          # plausible file.
+          #
+          # base64 at a fixed narrow width gives many short lines instead of
+          # few long ones, and — the actual point — a checksum and a line
+          # count that make truncation DETECTABLE rather than invisible.
+          set +x
+          F=$(ls alembic/versions/*.py | head -1)
+          B64=$(base64 -w 120 "$F")
+          echo "===== BEGIN CANDIDATE BASELINE (base64) ====="
+          echo "$B64"
+          echo "===== END CANDIDATE BASELINE ====="
+          echo "candidate-sha256: $(sha256sum "$F" | cut -d' ' -f1)"
+          echo "candidate-bytes:  $(wc -c < "$F")"
+          echo "candidate-b64-lines: $(echo "$B64" | wc -l)"
+          set -x
+          # Put the tree back exactly as it was; this job never mutates state.
+          rm -f alembic/versions/*.py
+          mv /tmp/versions_held/*.py alembic/versions/ 2>/dev/null || true
+
+      # DB 2: whatever the CURRENT tree's alembic/versions produces. Before the
+      # squash that is the same 87 revisions and the diff is trivially clean —
+      # which is worth running once as a control, so a clean diff after the
+      # squash means something.
+      - name: Build the schema the CURRENT tree produces
+        run: |
+          set -eux
+          docker exec "$PG_CONTAINER" createdb -U fabledcurator fc_base
+          ls alembic/versions/*.py | wc -l
+          DB_NAME=fc_base alembic upgrade head
+          docker exec "$PG_CONTAINER" pg_dump -U fabledcurator --schema-only \
+            --no-owner --no-privileges -d fc_base > baseline.sql
+          wc -l baseline.sql
+
+      # The verdict.
+      #
+      # pg_dump orders dumpable objects by name within type, not by creation
+      # order, so two schemas built by different routes are directly
+      # comparable. Normalisation is deliberately minimal, because a filter
+      # that hides a real difference is the one way this check passes when it
+      # should fail — blank lines, SQL comments, trailing whitespace, and:
+      #
+      #   \restrict / \unrestrict — a per-invocation RANDOM NONCE that newer
+      #   pg_dump emits to fence the dump against injection during restore. It
+      #   differs on every run by construction, so it is noise by definition,
+      #   not a schema difference. Measured on run 4960, the control: two dumps
+      #   of the SAME schema came back 1123 lines each and differed on exactly
+      #   these two lines and nothing else. That control is what licenses this
+      #   filter — it was observed to be the only false positive, rather than
+      #   assumed to be one.
+      - name: Diff
+        run: |
+          set -eu
+          norm() {
+            grep -vE '^\s*(--|$)' "$1" \
+              | grep -vE '^\\(un)?restrict ' \
+              | sed 's/[[:space:]]*$//'
+          }
+          norm chain.sql > a.txt
+          norm baseline.sql > b.txt
+          echo "normalised: chain=$(wc -l < a.txt) lines, current=$(wc -l < b.txt) lines"
+          if diff -u a.txt b.txt > schema.diff; then
+            echo "SCHEMAS IDENTICAL — the collapsed chain reproduces the old one."
+          else
+            echo "SCHEMAS DIFFER — $(grep -cE '^[+-]' schema.diff) changed lines:"
+            cat schema.diff
+            echo
+            echo "The baseline is wrong, not the database. Do not stamp."
+            exit 1
+          fi

.forgejo/workflows/build.yml

@@ -25,6 +25,56 @@ on:
     # Releases still happen (rule 148, on explicit request per rule 2). They
     # produce a changelog, not an image.
 
+  # The escape hatch for the one thing skip-if-exists makes untestable: a
+  # build that WOULD be skipped. `agent/` has not changed since 2026-07-17, so
+  # every push since has correctly declined to build it — which also means the
+  # agent build path has not run in six weeks and cannot be exercised on
+  # demand. #3190 lives on exactly that path.
+  #
+  # Editing build.yml does not force one either, and that is deliberate: the
+  # workflow is not shipped bytes, so it is in no artifact's path set. Putting
+  # it in one would re-version every artifact for a comment change.
+  #
+  # ONE input, not one per artifact. Forcing all three is cheap once the
+  # registry cache is warm (#3114), and three booleans is an interface nobody
+  # remembers the meaning of.
+  workflow_dispatch:
+    inputs:
+      force_build:
+        description: 'Rebuild every image even if the published revision matches'
+        type: boolean
+        default: false
+
+  # The base-image refresh (milestone 326 step 4, #3154).
+  #
+  # Skip-if-exists is keyed on OUR source, so an artifact whose source stops
+  # moving stops picking up base-image updates. `agent/` last changed
+  # 2026-07-17; every push since has correctly declined to rebuild it, which
+  # also means it will serve that day's `nvidia/cuda` layers forever. Nothing
+  # is wrong until it has been unchanged for months, which is precisely why
+  # this is a calendar trigger and not a condition on the push path.
+  #
+  # Weekly, Sunday 06:00 UTC. Away from CI-runner's Monday security sweep so
+  # the two are never diagnosing each other, and on the quietest day so a
+  # surprise rebuild is not competing with a push.
+  schedule:
+    - cron: '0 6 * * 0'
+
+# Which branch a run BUILDS, as opposed to which one triggered it.
+#
+# They are the same thing on every trigger but `schedule`. Forgejo registers a
+# cron from the DEFAULT branch — `dev` here — so a scheduled run arrives with
+# `github.ref` pointing at dev, and a refresh that rebuilt `:dev` would be
+# refreshing the one channel that gets rebuilt constantly anyway. Production is
+# `main` (rule 147), and `:latest` is the tag that goes stale.
+#
+# So the ref is decided once, here, and every checkout in the file takes it.
+# Deriving it per job invites the two halves to disagree: sign-extension would
+# derive dev's extension version while build-web bundled main's, and the
+# release download would 404 on a version that exists perfectly well.
+env:
+  BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
+
 # Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
 #   - write:package, read:package  (for docker push to git.fabledsword.com)
 #   - write:release                (for ext-<version> release asset cache)
@@ -68,12 +118,43 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history is load-bearing, not a convenience: the version this
           # job signs is derived from the commit TIME of the newest packaged
           # extension change. A depth-1 clone sees one commit and derives a
           # wrong, too-low value rather than failing (ci-requirements.md).
           fetch-depth: 0
 
+      # BUILD_REF is what makes a scheduled run build `main` rather than the
+      # branch its cron fired from — and it is read through the `env` context
+      # inside `with:`, which this runner is NOT known to evaluate. If it does
+      # not, checkout silently falls back to the triggering ref and the weekly
+      # refresh publishes DEV's source to `:latest`, which is production.
+      # Every lane would stay green; the first sign of it would be production
+      # running code that was never merged.
+      #
+      # So assert the checkout instead of trusting the expression. A red
+      # weekly job is a fine outcome. Shipping dev to production is not.
+      #
+      # `if:` reads the `github` context, which the runner demonstrably does
+      # evaluate — this file already gates steps on it — so the guard cannot
+      # be disabled by the same uncertainty it exists to cover.
+      - name: Guard — a scheduled run must have checked out main
+        if: github.event_name == 'schedule'
+        run: |
+          set -eu
+          BRANCH=$(git rev-parse --abbrev-ref HEAD)
+          echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
+          if [ "$BRANCH" != "main" ]; then
+            echo "schedule: expected main, got '$BRANCH'." >&2
+            echo "schedule: BUILD_REF was not honoured by the runner." >&2
+            echo "schedule: refusing to publish a channel tag from it." >&2
+            exit 1
+          fi
+
       # The version is DERIVED, not read from the repo (milestone 271 step 4,
       # cut over 2026-08-27). `packaging.sh version` returns `YYYY.M.D.HHMM`
       # UTC — the commit TIME of the newest change to a PACKAGED extension
@@ -344,12 +425,30 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history: this job RE-DERIVES the extension version rather than
           # being handed it, and a depth-1 clone derives a wrong, too-low value
           # rather than failing — which would 404 the download of a release
           # that exists perfectly well under its real name.
           fetch-depth: 0
 
+      # See sign-extension's copy for why this guard exists.
+      - name: Guard — a scheduled run must have checked out main
+        if: github.event_name == 'schedule'
+        run: |
+          set -eu
+          BRANCH=$(git rev-parse --abbrev-ref HEAD)
+          echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
+          if [ "$BRANCH" != "main" ]; then
+            echo "schedule: expected main, got '$BRANCH'." >&2
+            echo "schedule: BUILD_REF was not honoured by the runner." >&2
+            echo "schedule: refusing to publish a channel tag from it." >&2
+            exit 1
+          fi
+
       # --- derived values, one line (milestone 313) ------------------------
       # These stopped being shadow output at step 3. `revision` decides
       # whether the build below runs at all and `version` is what the image
@@ -409,8 +508,30 @@ jobs:
           # everywhere). Operator-flagged 2026-06-01 after the first :c-<sha>
           # main-push build failed at this step.
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
-          # Mirrors build-web's tag list; see the comment there.
+
-          if [ "${GITHUB_REF##*/}" = "main" ]; then
+          # A scheduled refresh publishes the CHANNEL and nothing else
+          # (#3154). :c-<sha> for main's HEAD already exists and names the
+          # bytes that commit actually built; re-pushing it over refreshed
+          # base layers would break the one tag rule 145 makes immutable —
+          # and it is the rollback unit, so the breakage would surface on the
+          # day somebody needed it.
+          #
+          # The accepted consequence: between a refresh and the next main
+          # push, :latest and :c-<sha> point at different manifests. That is
+          # the design, not drift. They RE-CONVERGE on that push — it hits
+          # reuse (a refresh does not move fc.revision, because it does not
+          # touch the source), and the repoint step then writes the new
+          # :c-<sha> from the refreshed :latest. So the rollback unit ends up
+          # naming the bytes production is actually running, which is the
+          # property that matters.
+          #
+          # Checked BEFORE the ref test, not after: a scheduled run's
+          # GITHUB_REF is the default branch (dev), so the main test would
+          # never fire on it.
+          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
+            echo "channel=main" >> "$GITHUB_OUTPUT"
+          elif [ "${GITHUB_REF##*/}" = "main" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           else
@@ -443,6 +564,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -481,6 +619,16 @@ jobs:
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
           CHANNEL: ${{ steps.tag.outputs.channel }}
+          # Empty on a push; the string "true" only from a workflow_dispatch
+          # that asked for it. `github.event.inputs` rather than the `inputs`
+          # context — release.yml already uses that form, and it is the one
+          # this runner is known to evaluate. Read through env rather than
+          # interpolated into the run block, same rule as release.yml's TAG.
+          FORCE: ${{ github.event.inputs.force_build }}
+          # A scheduled refresh has to bypass reuse by construction: it
+          # rebuilds the SAME source, so fc.revision always matches and the
+          # check would skip every refresh there has ever been.
+          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision web)
@@ -520,7 +668,17 @@ jobs:
             echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
           fi
 
-          if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
+          # FORCE is checked here rather than in the build step's `if:`, so
+          # that one decision drives everything downstream. The repoint step
+          # keys off `hit` too, and a force that bypassed only the build would
+          # leave the two disagreeing about what just happened.
+          if [ "${FORCE:-false}" = "true" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: force_build set — building regardless"
+          elif [ "${EVENT:-}" = "schedule" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: scheduled base refresh — building regardless"
+          elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
             echo "hit=true" >> "$GITHUB_OUTPUT"
             echo "reuse: already published — skipping the build"
           else
@@ -611,6 +769,37 @@ jobs:
           context: .
           file: Dockerfile
           push: true
+          # Re-resolve the FROM references against the registry instead of
+          # trusting whatever digest the cache was built against. This is the
+          # whole mechanism of the scheduled refresh (#3154): if the base tag
+          # moved, the FROM layer's cache key changes, every layer above it
+          # invalidates, and the image genuinely rebuilds.
+          #
+          # MEASURED on the first real fire, run 4934 (#3265): when the base
+          # did NOT move, the build is ~13s and every content step reports
+          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # buildkit mints a fresh image config each run, so identical layers
+          # are republished under a new config blob. All three images moved
+          # that way on 2026-08-30 with nothing whatsoever changed in them.
+          #
+          # So a refresh currently rewrites :latest every Sunday whether or
+          # not there is anything new in it, and :c-<sha> is handed a new
+          # manifest to diverge from on the same cadence. Layers are shared,
+          # so the storage cost is a config blob; the cost that matters is
+          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
+          # make "same source, same bytes" true and turn the no-op case into
+          # a genuine no-op.
+          #
+          # What `pull` does NOT catch either: a Debian package update inside
+          # the `apt-get install` layer while the base tag itself stands
+          # still. The official python/cuda images rebuild with those updates
+          # baked in, so this is a lag rather than a hole; closing it needs
+          # `no-cache: true`, which is a much larger version of the same
+          # churn #3265 is about.
+          #
+          # Only on the schedule. An ordinary push wants the cached base.
+          pull: ${{ github.event_name == 'schedule' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
@@ -623,6 +812,40 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
+          # The ONLY cache this driver can have. `docker-container` gets a
+          # FRESH buildkit instance per job, so unlike the default docker
+          # driver it has no local layer store to fall back on — measured on
+          # run 4896, the first builds after the driver change: web 3m44s
+          # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
+          # driver change ALONE is a regression; this is the other half of it.
+          #
+          # mode=max so intermediate stages cache too. web's frontend-builder
+          # stage and the agent's two ~150s pip layers are the whole cost, and
+          # they are exactly what a min-mode cache would drop.
+          #
+          # A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
+          # Rule 145 narrowed against names NOTHING reads; this one is read by
+          # every build that runs, is one moving ref per image rather than one
+          # per build, holds cache blobs rather than a shippable artifact, and
+          # is overwritten in place rather than accumulating. It is closer to
+          # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
+          cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache
+          cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache,mode=max
           # Only the web image carries these: it is the one with a UI and an
           # HTTP surface to report them on. The ml and agent images have
           # nothing to tell.
@@ -699,6 +922,12 @@ jobs:
             ARGS="$ARGS -t $t"
           done
           unset IFS
+          #
+          # This is also the whole of the scheduled refresh's tag handling
+          # (#3154): a refresh's tag list is the channel tag alone, so SOURCE
+          # is the only entry, it gets excluded, and this step correctly does
+          # nothing. No `if:` on the step and no schedule special-case —
+          # excluding the source was already the right rule.
           if [ -z "$ARGS" ]; then
             echo "repoint: $SOURCE is the only tag for this channel and"
             echo "repoint: already holds this revision — nothing to write."
@@ -715,6 +944,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history: this job derives its artifact's version from the
           # commit its shipped files last changed in (milestone 313). A
           # depth-1 clone cannot see that commit — it either derives a wrong,
@@ -722,6 +955,20 @@ jobs:
           # the build would otherwise notice.
           fetch-depth: 0
 
+      # See sign-extension's copy for why this guard exists.
+      - name: Guard — a scheduled run must have checked out main
+        if: github.event_name == 'schedule'
+        run: |
+          set -eu
+          BRANCH=$(git rev-parse --abbrev-ref HEAD)
+          echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
+          if [ "$BRANCH" != "main" ]; then
+            echo "schedule: expected main, got '$BRANCH'." >&2
+            echo "schedule: BUILD_REF was not honoured by the runner." >&2
+            echo "schedule: refusing to publish a channel tag from it." >&2
+            exit 1
+          fi
+
       # --- derived values, one line (milestone 313) ------------------------
       # These stopped being shadow output at step 3. `revision` decides
       # whether the build below runs at all and `version` is what the image
@@ -759,8 +1006,12 @@ jobs:
           # everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
           # main-push build failed at this step.
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
-          # Mirrors build-web's tag list; see the comment there.
+          # Mirrors build-web's tag list and its schedule handling; see
-          if [ "${GITHUB_REF##*/}" = "main" ]; then
+          # the comments there.
+          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
+            echo "channel=main" >> "$GITHUB_OUTPUT"
+          elif [ "${GITHUB_REF##*/}" = "main" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           else
@@ -776,6 +1027,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -814,6 +1082,16 @@ jobs:
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
           CHANNEL: ${{ steps.tag.outputs.channel }}
+          # Empty on a push; the string "true" only from a workflow_dispatch
+          # that asked for it. `github.event.inputs` rather than the `inputs`
+          # context — release.yml already uses that form, and it is the one
+          # this runner is known to evaluate. Read through env rather than
+          # interpolated into the run block, same rule as release.yml's TAG.
+          FORCE: ${{ github.event.inputs.force_build }}
+          # A scheduled refresh has to bypass reuse by construction: it
+          # rebuilds the SAME source, so fc.revision always matches and the
+          # check would skip every refresh there has ever been.
+          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision ml)
@@ -849,7 +1127,17 @@ jobs:
             echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
           fi
 
-          if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
+          # FORCE is checked here rather than in the build step's `if:`, so
+          # that one decision drives everything downstream. The repoint step
+          # keys off `hit` too, and a force that bypassed only the build would
+          # leave the two disagreeing about what just happened.
+          if [ "${FORCE:-false}" = "true" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: force_build set — building regardless"
+          elif [ "${EVENT:-}" = "schedule" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: scheduled base refresh — building regardless"
+          elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
             echo "hit=true" >> "$GITHUB_OUTPUT"
             echo "reuse: already published — skipping the build"
           else
@@ -864,6 +1152,37 @@ jobs:
           context: .
           file: Dockerfile.ml
           push: true
+          # Re-resolve the FROM references against the registry instead of
+          # trusting whatever digest the cache was built against. This is the
+          # whole mechanism of the scheduled refresh (#3154): if the base tag
+          # moved, the FROM layer's cache key changes, every layer above it
+          # invalidates, and the image genuinely rebuilds.
+          #
+          # MEASURED on the first real fire, run 4934 (#3265): when the base
+          # did NOT move, the build is ~13s and every content step reports
+          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # buildkit mints a fresh image config each run, so identical layers
+          # are republished under a new config blob. All three images moved
+          # that way on 2026-08-30 with nothing whatsoever changed in them.
+          #
+          # So a refresh currently rewrites :latest every Sunday whether or
+          # not there is anything new in it, and :c-<sha> is handed a new
+          # manifest to diverge from on the same cadence. Layers are shared,
+          # so the storage cost is a config blob; the cost that matters is
+          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
+          # make "same source, same bytes" true and turn the no-op case into
+          # a genuine no-op.
+          #
+          # What `pull` does NOT catch either: a Debian package update inside
+          # the `apt-get install` layer while the base tag itself stands
+          # still. The official python/cuda images rebuild with those updates
+          # baked in, so this is a lag rather than a hole; closing it needs
+          # `no-cache: true`, which is a much larger version of the same
+          # churn #3265 is about.
+          #
+          # Only on the schedule. An ordinary push wants the cached base.
+          pull: ${{ github.event_name == 'schedule' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
@@ -876,6 +1195,40 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
+          # The ONLY cache this driver can have. `docker-container` gets a
+          # FRESH buildkit instance per job, so unlike the default docker
+          # driver it has no local layer store to fall back on — measured on
+          # run 4896, the first builds after the driver change: web 3m44s
+          # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
+          # driver change ALONE is a regression; this is the other half of it.
+          #
+          # mode=max so intermediate stages cache too. web's frontend-builder
+          # stage and the agent's two ~150s pip layers are the whole cost, and
+          # they are exactly what a min-mode cache would drop.
+          #
+          # A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
+          # Rule 145 narrowed against names NOTHING reads; this one is read by
+          # every build that runs, is one moving ref per image rather than one
+          # per build, holds cache blobs rather than a shippable artifact, and
+          # is overwritten in place rather than accumulating. It is closer to
+          # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
+          cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
+          cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
 
       # Every tag but the channel's own is written HERE, registry-side,
       # whether or not a build ran. Each -t becomes another reference to the
@@ -965,6 +1318,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history: this job derives its artifact's version from the
           # commit its shipped files last changed in (milestone 313). A
           # depth-1 clone cannot see that commit — it either derives a wrong,
@@ -972,6 +1329,20 @@ jobs:
           # the build would otherwise notice.
           fetch-depth: 0
 
+      # See sign-extension's copy for why this guard exists.
+      - name: Guard — a scheduled run must have checked out main
+        if: github.event_name == 'schedule'
+        run: |
+          set -eu
+          BRANCH=$(git rev-parse --abbrev-ref HEAD)
+          echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
+          if [ "$BRANCH" != "main" ]; then
+            echo "schedule: expected main, got '$BRANCH'." >&2
+            echo "schedule: BUILD_REF was not honoured by the runner." >&2
+            echo "schedule: refusing to publish a channel tag from it." >&2
+            exit 1
+          fi
+
       # --- derived values, one line (milestone 313) ------------------------
       # These stopped being shadow output at step 3. `revision` decides
       # whether the build below runs at all and `version` is what the image
@@ -1004,8 +1375,12 @@ jobs:
         id: tag
         run: |
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
-          # Mirrors build-web's tag list; see the comment there.
+          # Mirrors build-web's tag list and its schedule handling; see
-          if [ "${GITHUB_REF##*/}" = "main" ]; then
+          # the comments there.
+          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
+            echo "channel=main" >> "$GITHUB_OUTPUT"
+          elif [ "${GITHUB_REF##*/}" = "main" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           else
@@ -1021,6 +1396,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -1059,6 +1451,16 @@ jobs:
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
           CHANNEL: ${{ steps.tag.outputs.channel }}
+          # Empty on a push; the string "true" only from a workflow_dispatch
+          # that asked for it. `github.event.inputs` rather than the `inputs`
+          # context — release.yml already uses that form, and it is the one
+          # this runner is known to evaluate. Read through env rather than
+          # interpolated into the run block, same rule as release.yml's TAG.
+          FORCE: ${{ github.event.inputs.force_build }}
+          # A scheduled refresh has to bypass reuse by construction: it
+          # rebuilds the SAME source, so fc.revision always matches and the
+          # check would skip every refresh there has ever been.
+          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision agent)
@@ -1094,7 +1496,17 @@ jobs:
             echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
           fi
 
-          if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
+          # FORCE is checked here rather than in the build step's `if:`, so
+          # that one decision drives everything downstream. The repoint step
+          # keys off `hit` too, and a force that bypassed only the build would
+          # leave the two disagreeing about what just happened.
+          if [ "${FORCE:-false}" = "true" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: force_build set — building regardless"
+          elif [ "${EVENT:-}" = "schedule" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: scheduled base refresh — building regardless"
+          elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
             echo "hit=true" >> "$GITHUB_OUTPUT"
             echo "reuse: already published — skipping the build"
           else
@@ -1109,6 +1521,37 @@ jobs:
           context: agent
           file: agent/Dockerfile
           push: true
+          # Re-resolve the FROM references against the registry instead of
+          # trusting whatever digest the cache was built against. This is the
+          # whole mechanism of the scheduled refresh (#3154): if the base tag
+          # moved, the FROM layer's cache key changes, every layer above it
+          # invalidates, and the image genuinely rebuilds.
+          #
+          # MEASURED on the first real fire, run 4934 (#3265): when the base
+          # did NOT move, the build is ~13s and every content step reports
+          # CACHED — but the channel tag STILL gets a new manifest digest.
+          # buildkit mints a fresh image config each run, so identical layers
+          # are republished under a new config blob. All three images moved
+          # that way on 2026-08-30 with nothing whatsoever changed in them.
+          #
+          # So a refresh currently rewrites :latest every Sunday whether or
+          # not there is anything new in it, and :c-<sha> is handed a new
+          # manifest to diverge from on the same cadence. Layers are shared,
+          # so the storage cost is a config blob; the cost that matters is
+          # that a digest change no longer MEANS anything. Tracked in #3265 —
+          # the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
+          # make "same source, same bytes" true and turn the no-op case into
+          # a genuine no-op.
+          #
+          # What `pull` does NOT catch either: a Debian package update inside
+          # the `apt-get install` layer while the base tag itself stands
+          # still. The official python/cuda images rebuild with those updates
+          # baked in, so this is a lag rather than a hole; closing it needs
+          # `no-cache: true`, which is a much larger version of the same
+          # churn #3265 is about.
+          #
+          # Only on the schedule. An ordinary push wants the cached base.
+          pull: ${{ github.event_name == 'schedule' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
@@ -1121,6 +1564,40 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
+          # The ONLY cache this driver can have. `docker-container` gets a
+          # FRESH buildkit instance per job, so unlike the default docker
+          # driver it has no local layer store to fall back on — measured on
+          # run 4896, the first builds after the driver change: web 3m44s
+          # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
+          # driver change ALONE is a regression; this is the other half of it.
+          #
+          # mode=max so intermediate stages cache too. web's frontend-builder
+          # stage and the agent's two ~150s pip layers are the whole cost, and
+          # they are exactly what a min-mode cache would drop.
+          #
+          # A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
+          # Rule 145 narrowed against names NOTHING reads; this one is read by
+          # every build that runs, is one moving ref per image rather than one
+          # per build, holds cache blobs rather than a shippable artifact, and
+          # is overwritten in place rather than accumulating. It is closer to
+          # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
+          cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
+          cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
 
       # Every tag but the channel's own is written HERE, registry-side,
       # whether or not a build ran. Each -t becomes another reference to the

ci-requirements.md

@@ -133,6 +133,82 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
   already ran, so both paths now share one proven route. The cost: `:c-<sha>`
   is an index rather than a plain image, so `fc.revision` does not resolve
   through it — nothing reads it there, and the index names the same manifest.
+- **The image builds run on a `docker-container` buildx builder, and
+  `provenance`/`sbom` are explicitly OFF** (milestone 326 step 1). The builder
+  is what makes a registry layer cache possible at all — the default `docker`
+  driver cannot export one (#3114) — and it is #3190's leading suspect, since
+  it is the driver that resolves image metadata against a local store a
+  registry-direct push never fills. **The attestation flags are load-bearing,
+  not tidiness:** on the container driver `build-push-action@v5` defaults
+  `provenance` to true when pushing, an attestation manifest makes the pushed
+  tag a manifest INDEX, and config labels do not resolve through an index — so
+  leaving them on would make every push read `fc.revision=<none>`, miss, and
+  rebuild forever with every lane green. Same failure as #3183, different door.
+  These jobs run inside a container against a mounted docker socket, so the
+  buildkit container is a sibling rather than a child.
+- **All three images import and export a registry layer cache**
+  (`<image>:buildcache`, `mode=max`). This is not an optimisation bolted onto
+  the driver change — it is the other half of it. The `docker-container`
+  driver gets a fresh buildkit instance per job and therefore has **no local
+  layer store at all**, where the old `docker` driver at least reused whatever
+  the runner's dockerd happened to hold. Measured on run 4896, the first builds
+  after the driver moved: web 3m44s (was 2m23s), ml 3m49s (was 3m20s), agent
+  11m12s (was 9m26s) — every one slower. A `:buildcache` tag is read by every
+  build that runs, is one moving ref per image, holds cache blobs rather than a
+  shippable artifact, and is overwritten in place, so it is not a return of the
+  per-version tags milestone 318 withdrew (#3114).
+- **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which
+  bypasses the reuse check for all three images. It exists because
+  skip-if-exists makes its own build path untestable: `agent/` has not changed
+  since 2026-07-17, so the agent build has not run in six weeks and cannot be
+  exercised on demand — and #3190 lives on exactly that path. Editing
+  `build.yml` does not force a build either, deliberately: the workflow is not
+  shipped bytes and is in no artifact's path set. The flag is read through
+  `github.event.inputs` into an env var rather than interpolated into a run
+  block, and it is checked inside the reuse step so that one decision drives
+  both the build and the repoint.
+- **A weekly `schedule` rebuilds all three images against fresh base layers**
+  (Sunday 06:00 UTC, milestone 326 step 4, #3154). Skip-if-exists is keyed on
+  OUR source, so an artifact whose source stops moving stops picking up base
+  updates — `agent/` has not changed since 2026-07-17 and would otherwise serve
+  that day's `nvidia/cuda` layers forever. Four things make it work:
+  - It **builds `main`, not the branch that triggered it.** Forgejo registers a
+    cron from the DEFAULT branch (`dev` here), so a scheduled run arrives with
+    `github.ref` on dev. The ref is decided once in a top-level `env:
+    BUILD_REF` that every checkout in the file takes, rather than per job —
+    otherwise `sign-extension` would derive dev's extension version while
+    `build-web` bundled main's, and the release download would 404 on a version
+    that exists perfectly well. Every job then ASSERTS its checkout is `main`
+    before doing anything, because `env` inside `with:` is not a context this
+    runner is known to evaluate — if it silently resolved to empty, checkout
+    would fall back to the triggering ref and the refresh would publish dev's
+    source to `:latest` with every lane green.
+  - It **publishes only `:latest`.** `:c-<sha>` for main's HEAD already names
+    the bytes that commit built; re-pushing it over refreshed layers would
+    break the one tag rule 145 makes immutable, and it is the rollback unit.
+    The repoint step needs no schedule case for this — the tag list is the
+    channel tag alone, so SOURCE is the only entry, it is excluded as always,
+    and the step correctly does nothing.
+  - **`:latest` and `:c-<sha>` therefore diverge between a refresh and the next
+    `main` push, by design.** They re-converge on that push: it hits reuse (a
+    refresh does not move `fc.revision`, because it does not touch the source),
+    and the repoint writes the NEW `:c-<sha>` from the refreshed `:latest`. The
+    push path needed no change for this, because the repoint already excluded
+    the source tag — the same rule that keeps the label readable also keeps a
+    refresh from being undone.
+  - **`pull: true` on the scheduled path only** is the mechanism: a moved base
+    tag changes the `FROM` layer's cache key and everything above it rebuilds.
+    **It does not currently make the unmoved case free.** Measured on the first
+    real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
+    the bases resolved to unchanged digests, yet all three `:latest` tags got a
+    NEW manifest digest, because buildkit mints a fresh image config per run and
+    republishes identical layers under it. So `:latest` is rewritten weekly
+    whether or not anything changed, and `:c-<sha>` is handed a new manifest to
+    diverge from on the same cadence — a digest change stops meaning anything.
+    Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
+    Separately not caught: a Debian package update inside the `apt-get install`
+    layer while the base tag stands still — a lag rather than a hole, since the
+    official python/cuda images rebuild with those updates baked in.
 - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
   `build.yml` passes them to the web image only — the ml and agent images have
   nothing to report them to. `/api/health` returns both, the foot of Settings

tests/test_artifact_identity.py

@@ -34,6 +34,15 @@ is genuinely the commit those paths last changed in.
 emit. Two nearly-identical formats are more dangerous than two obviously
 different ones, and the only thing keeping them identical is a test.
 
+**The extension is the one exception, and it is a rendering exception only.**
+AMO's version grammar forbids a leading zero, so the extension emits the same
+numbers unpadded — `2026.8.29.201` where the family says `2026.08.29.0201`
+(#3138, milestone 318 step 8). Rule 148 defines comparison as numeric per
+dot-segment, under which the two are equal, so this is pinned in both
+directions below: the extension must satisfy AMO's grammar, and every artifact
+must derive the same NUMBERS its own commit stamps. An exception left as "the
+extension is different" would drift into being differently different.
+
 The identity-TAG tests this file used to hold are gone with the tag. There is
 no longer a `CHANNELLED` list to drift (the channel is which tag you inspect),
 and no `identity` subcommand to refuse an unqualified call.
@@ -57,6 +66,26 @@ _REVISION = re.compile(r"^[0-9a-f]{12}$")
 # YYYY.MM.DD.HHMM, every segment zero-padded to its full width.
 _VERSION = re.compile(r"^\d{4}\.\d{2}\.\d{2}\.\d{4}$")
 
+# The artifacts that cannot use the padded rendering. Exactly one, and the
+# reason is external: `packaging.sh` derives the extension's version and AMO
+# refuses to sign a padded one.
+AMO_UNPADDED = frozenset({"extension"})
+
+# Mozilla's published grammar for addons.mozilla.org, transcribed from MDN's
+# manifest.json/version page. A segment is the single digit `0` or starts 1-9,
+# and there are at most four. This is the constraint the exception exists for,
+# so it is what the exception is tested against — `2026.08.29.0201` fails it.
+_AMO = re.compile(r"^(0|[1-9][0-9]{0,8})(\.(0|[1-9][0-9]{0,8})){0,3}$")
+
+# YYYY.M.D.HHMM — four segments, none of them zero-padded.
+_UNPADDED = re.compile(r"^\d{4}(\.(0|[1-9]\d*)){3}$")
+
+
+def segments(value: str) -> tuple[int, ...]:
+    """A version as the numbers it denotes, which is how rule 148 says to
+    compare one. `2026.08.29.0201` and `2026.8.29.201` are one value here."""
+    return tuple(int(part) for part in value.split("."))
+
 
 # Everything here goes through artifacts.sh rather than importing a sibling
 # test module. That is the interface build.yml actually calls, so the tests
@@ -126,7 +155,7 @@ def test_revision_is_a_legal_label_value_and_is_stable(artifact):
     assert first == revision(artifact), "revision is not stable across calls"
 
 
-@pytest.mark.parametrize("artifact", ARTIFACTS)
+@pytest.mark.parametrize("artifact", sorted(set(ARTIFACTS) - AMO_UNPADDED))
 def test_version_is_zero_padded_calver(artifact):
     """The family shape, pinned.
 
@@ -148,6 +177,31 @@ def test_version_is_zero_padded_calver(artifact):
     )
 
 
+@pytest.mark.parametrize("artifact", sorted(AMO_UNPADDED))
+def test_the_unpadded_artifacts_derive_something_amo_will_sign(artifact):
+    """The other half of the family shape: the documented exception, tested
+    against the constraint that justifies it rather than against itself.
+
+    A padded value passes `_UNPADDED` on any date with no leading zeros, so
+    that pattern alone would let a regression sit unnoticed until the first
+    single-digit month — at which point the failure is a burned AMO version,
+    not a red lane. AMO's grammar is the assertion that fires immediately.
+    """
+    value = artifacts("version", artifact).strip()
+    assert _AMO.match(value), (
+        f"{artifact} derives {value!r}, which AMO refuses: a segment must be "
+        f"the single digit `0` or start 1-9, and there are at most four. "
+        f"Almost certainly a zero-padded segment — the family pads and this "
+        f"artifact must not (#3138). AMO 409s on re-signing, so a version it "
+        f"rejects is burned."
+    )
+    assert _UNPADDED.match(value), (
+        f"{artifact} derives {value!r}, which is not YYYY.M.D.HHMM. AMO would "
+        f"also accept the pre-318 `1.0.<minutes>`, and that orders below every "
+        f"ext-2026.* release already signed."
+    )
+
+
 @pytest.mark.parametrize("artifact", ARTIFACTS)
 def test_version_and_revision_describe_the_same_commit(artifact):
     """They are derived independently and must not be able to disagree.
@@ -162,5 +216,21 @@ def test_version_and_revision_describe_the_same_commit(artifact):
         capture_output=True, text=True, check=True, cwd=ROOT,
         env={"TZ": "UTC", "PATH": os.environ.get("PATH", "")},
     ).stdout.strip()
-    assert artifacts("version", artifact).strip() == stamped
+    derived = artifacts("version", artifact).strip()
+
+    # Compared as NUMBERS, which is how rule 148 defines comparison and the
+    # only way one assertion can cover both renderings. This is what makes the
+    # extension's exception cosmetic rather than semantic: it must denote
+    # exactly the value its own commit stamps, whatever the padding.
+    assert segments(derived) == segments(stamped), (
+        f"{artifact} derives {derived!r}, but its newest shipped commit "
+        f"{sha[:12]} is {stamped!r}. The instance would name one commit while "
+        f"carrying another's bytes."
+    )
+    if artifact not in AMO_UNPADDED:
+        assert derived == stamped, (
+            f"{artifact} derives {derived!r} where the family shape is "
+            f"{stamped!r} — same numbers, wrong rendering. Only the artifacts "
+            f"in AMO_UNPADDED may differ here."
+        )
     assert sha.startswith(revision(artifact))