ci: a weekly base-image refresh on the channel tags (milestone 326 step 4)

Skip-if-exists is keyed on our own source, so an artifact whose source
stops moving stops picking up base-image updates. `agent/` last changed
2026-07-17; every push since has correctly declined to rebuild it, which
also means it will serve that day's nvidia/cuda layers indefinitely.

A `schedule:` trigger, Sunday 06:00 UTC, away from CI-runner's Monday
security sweep so the two are never diagnosing each other.

#3154's blocking open question is dissolved rather than answered. It was
written when the identity was a `r-<revision>` TAG, and asked how the
next ordinary push could avoid repointing :latest back off the refresh.
Milestone 318 replaced that tag with a LABEL, and #3183 made the repoint
step exclude its source tag so the label stays readable. Excluding the
source is what also keeps a refresh from being undone: on the next main
push the reuse check hits, :latest is not rewritten, and the new :c-<sha>
is written FROM the refreshed :latest. To be verified by digest, not by
this argument.

Four decisions, each commented where it lives:

* It builds `main`, not the branch that triggered it. Forgejo registers a
  cron from the default branch — `dev` here — so a scheduled run arrives
  with github.ref on dev, and a refresh of :dev would be refreshing the
  one channel that is rebuilt constantly anyway. The ref is decided once
  in a top-level `env: BUILD_REF` that all four checkouts take. Deriving
  it per job would let the halves disagree: sign-extension would derive
  dev's extension version while build-web bundled main's, and the release
  download would 404 on a version that exists perfectly well.

* It publishes only the channel tag. :c-<sha> for main's HEAD already
  names the bytes that commit built; re-pushing it over refreshed layers
  would break the one tag rule 145 makes immutable, and it is the
  rollback unit — so the breakage would surface on the day somebody
  needed it. The repoint step needs no schedule case: the tag list is the
  channel tag alone, SOURCE is the only entry, it is excluded as always,
  and the step correctly does nothing.

* It bypasses reuse by construction, since it rebuilds the same source
  and fc.revision always matches. Checked in the reuse step beside
  force_build, so one decision still drives both the build and the
  repoint.

* `pull: true`, on the scheduled path only, is the actual mechanism. A
  moved base tag changes the FROM layer's cache key and everything above
  it rebuilds; an unmoved one is satisfied by the registry cache and the
  refresh is a ~13s no-op that republishes nothing. That no-op is the
  point — :latest should change when there is something new in it, not
  every Sunday. The known lag, left deliberately: an apt package update
  while the base tag stands still is not caught, and closing it needs
  no-cache: true, which buys weekly churn for it.

.forgejo/workflows/build.yml

@@ -45,6 +45,36 @@ on:
         type: boolean
         default: false
 
+  # The base-image refresh (milestone 326 step 4, #3154).
+  #
+  # Skip-if-exists is keyed on OUR source, so an artifact whose source stops
+  # moving stops picking up base-image updates. `agent/` last changed
+  # 2026-07-17; every push since has correctly declined to rebuild it, which
+  # also means it will serve that day's `nvidia/cuda` layers forever. Nothing
+  # is wrong until it has been unchanged for months, which is precisely why
+  # this is a calendar trigger and not a condition on the push path.
+  #
+  # Weekly, Sunday 06:00 UTC. Away from CI-runner's Monday security sweep so
+  # the two are never diagnosing each other, and on the quietest day so a
+  # surprise rebuild is not competing with a push.
+  schedule:
+    - cron: '0 6 * * 0'
+
+# Which branch a run BUILDS, as opposed to which one triggered it.
+#
+# They are the same thing on every trigger but `schedule`. Forgejo registers a
+# cron from the DEFAULT branch — `dev` here — so a scheduled run arrives with
+# `github.ref` pointing at dev, and a refresh that rebuilt `:dev` would be
+# refreshing the one channel that gets rebuilt constantly anyway. Production is
+# `main` (rule 147), and `:latest` is the tag that goes stale.
+#
+# So the ref is decided once, here, and every checkout in the file takes it.
+# Deriving it per job invites the two halves to disagree: sign-extension would
+# derive dev's extension version while build-web bundled main's, and the
+# release download would 404 on a version that exists perfectly well.
+env:
+  BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
+
 # Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
 #   - write:package, read:package  (for docker push to git.fabledsword.com)
 #   - write:release                (for ext-<version> release asset cache)
@@ -88,6 +118,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history is load-bearing, not a convenience: the version this
           # job signs is derived from the commit TIME of the newest packaged
           # extension change. A depth-1 clone sees one commit and derives a
@@ -364,6 +398,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history: this job RE-DERIVES the extension version rather than
           # being handed it, and a depth-1 clone derives a wrong, too-low value
           # rather than failing — which would 404 the download of a release
@@ -429,8 +467,30 @@ jobs:
           # everywhere). Operator-flagged 2026-06-01 after the first :c-<sha>
           # main-push build failed at this step.
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
-          # Mirrors build-web's tag list; see the comment there.
+
-          if [ "${GITHUB_REF##*/}" = "main" ]; then
+          # A scheduled refresh publishes the CHANNEL and nothing else
+          # (#3154). :c-<sha> for main's HEAD already exists and names the
+          # bytes that commit actually built; re-pushing it over refreshed
+          # base layers would break the one tag rule 145 makes immutable —
+          # and it is the rollback unit, so the breakage would surface on the
+          # day somebody needed it.
+          #
+          # The accepted consequence: between a refresh and the next main
+          # push, :latest and :c-<sha> point at different manifests. That is
+          # the design, not drift. They RE-CONVERGE on that push — it hits
+          # reuse (a refresh does not move fc.revision, because it does not
+          # touch the source), and the repoint step then writes the new
+          # :c-<sha> from the refreshed :latest. So the rollback unit ends up
+          # naming the bytes production is actually running, which is the
+          # property that matters.
+          #
+          # Checked BEFORE the ref test, not after: a scheduled run's
+          # GITHUB_REF is the default branch (dev), so the main test would
+          # never fire on it.
+          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
+            echo "channel=main" >> "$GITHUB_OUTPUT"
+          elif [ "${GITHUB_REF##*/}" = "main" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           else
@@ -524,6 +584,10 @@ jobs:
           # this runner is known to evaluate. Read through env rather than
           # interpolated into the run block, same rule as release.yml's TAG.
           FORCE: ${{ github.event.inputs.force_build }}
+          # A scheduled refresh has to bypass reuse by construction: it
+          # rebuilds the SAME source, so fc.revision always matches and the
+          # check would skip every refresh there has ever been.
+          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision web)
@@ -570,6 +634,9 @@ jobs:
           if [ "${FORCE:-false}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: force_build set — building regardless"
+          elif [ "${EVENT:-}" = "schedule" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: scheduled base refresh — building regardless"
           elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
             echo "hit=true" >> "$GITHUB_OUTPUT"
             echo "reuse: already published — skipping the build"
@@ -661,6 +728,30 @@ jobs:
           context: .
           file: Dockerfile
           push: true
+          # Re-resolve the FROM references against the registry instead of
+          # trusting whatever digest the cache was built against. This is the
+          # whole mechanism of the scheduled refresh (#3154): if the base tag
+          # moved, the FROM layer's cache key changes, every layer above it
+          # invalidates, and the image genuinely rebuilds. If it did not move,
+          # the registry cache satisfies the entire graph and the refresh is a
+          # ~13s no-op that republishes nothing.
+          #
+          # That no-op is the POINT, not a shortfall: :latest should change
+          # when there is something new in it and not otherwise. A refresh
+          # that rewrote the image weekly regardless would churn the registry
+          # and hand :c-<sha> a new manifest to diverge from every Sunday, for
+          # no gain.
+          #
+          # What it therefore does NOT catch: a Debian package update inside
+          # the `apt-get install` layer while the base tag itself stands
+          # still. The official python/cuda images rebuild with those updates
+          # baked in, so this is a lag rather than a hole — but closing it
+          # would take `no-cache: true` on the scheduled path, which is the
+          # weekly-churn trade above. Left as the cheaper of the two on
+          # purpose.
+          #
+          # Only on the schedule. An ordinary push wants the cached base.
+          pull: ${{ github.event_name == 'schedule' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
@@ -783,6 +874,12 @@ jobs:
             ARGS="$ARGS -t $t"
           done
           unset IFS
+          #
+          # This is also the whole of the scheduled refresh's tag handling
+          # (#3154): a refresh's tag list is the channel tag alone, so SOURCE
+          # is the only entry, it gets excluded, and this step correctly does
+          # nothing. No `if:` on the step and no schedule special-case —
+          # excluding the source was already the right rule.
           if [ -z "$ARGS" ]; then
             echo "repoint: $SOURCE is the only tag for this channel and"
             echo "repoint: already holds this revision — nothing to write."
@@ -799,6 +896,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history: this job derives its artifact's version from the
           # commit its shipped files last changed in (milestone 313). A
           # depth-1 clone cannot see that commit — it either derives a wrong,
@@ -843,8 +944,12 @@ jobs:
           # everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
           # main-push build failed at this step.
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
-          # Mirrors build-web's tag list; see the comment there.
+          # Mirrors build-web's tag list and its schedule handling; see
-          if [ "${GITHUB_REF##*/}" = "main" ]; then
+          # the comments there.
+          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
+            echo "channel=main" >> "$GITHUB_OUTPUT"
+          elif [ "${GITHUB_REF##*/}" = "main" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           else
@@ -921,6 +1026,10 @@ jobs:
           # this runner is known to evaluate. Read through env rather than
           # interpolated into the run block, same rule as release.yml's TAG.
           FORCE: ${{ github.event.inputs.force_build }}
+          # A scheduled refresh has to bypass reuse by construction: it
+          # rebuilds the SAME source, so fc.revision always matches and the
+          # check would skip every refresh there has ever been.
+          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision ml)
@@ -963,6 +1072,9 @@ jobs:
           if [ "${FORCE:-false}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: force_build set — building regardless"
+          elif [ "${EVENT:-}" = "schedule" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: scheduled base refresh — building regardless"
           elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
             echo "hit=true" >> "$GITHUB_OUTPUT"
             echo "reuse: already published — skipping the build"
@@ -978,6 +1090,30 @@ jobs:
           context: .
           file: Dockerfile.ml
           push: true
+          # Re-resolve the FROM references against the registry instead of
+          # trusting whatever digest the cache was built against. This is the
+          # whole mechanism of the scheduled refresh (#3154): if the base tag
+          # moved, the FROM layer's cache key changes, every layer above it
+          # invalidates, and the image genuinely rebuilds. If it did not move,
+          # the registry cache satisfies the entire graph and the refresh is a
+          # ~13s no-op that republishes nothing.
+          #
+          # That no-op is the POINT, not a shortfall: :latest should change
+          # when there is something new in it and not otherwise. A refresh
+          # that rewrote the image weekly regardless would churn the registry
+          # and hand :c-<sha> a new manifest to diverge from every Sunday, for
+          # no gain.
+          #
+          # What it therefore does NOT catch: a Debian package update inside
+          # the `apt-get install` layer while the base tag itself stands
+          # still. The official python/cuda images rebuild with those updates
+          # baked in, so this is a lag rather than a hole — but closing it
+          # would take `no-cache: true` on the scheduled path, which is the
+          # weekly-churn trade above. Left as the cheaper of the two on
+          # purpose.
+          #
+          # Only on the schedule. An ordinary push wants the cached base.
+          pull: ${{ github.event_name == 'schedule' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,
@@ -1113,6 +1249,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
         with:
+          # Not the triggering ref — see the `env:` block at the top. On a
+          # scheduled refresh this is `main`; on everything else it is the ref
+          # that fired, so this is a no-op on every ordinary path.
+          ref: ${{ env.BUILD_REF }}
           # Full history: this job derives its artifact's version from the
           # commit its shipped files last changed in (milestone 313). A
           # depth-1 clone cannot see that commit — it either derives a wrong,
@@ -1152,8 +1292,12 @@ jobs:
         id: tag
         run: |
           SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
-          # Mirrors build-web's tag list; see the comment there.
+          # Mirrors build-web's tag list and its schedule handling; see
-          if [ "${GITHUB_REF##*/}" = "main" ]; then
+          # the comments there.
+          if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
+            echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
+            echo "channel=main" >> "$GITHUB_OUTPUT"
+          elif [ "${GITHUB_REF##*/}" = "main" ]; then
             echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
             echo "channel=main" >> "$GITHUB_OUTPUT"
           else
@@ -1230,6 +1374,10 @@ jobs:
           # this runner is known to evaluate. Read through env rather than
           # interpolated into the run block, same rule as release.yml's TAG.
           FORCE: ${{ github.event.inputs.force_build }}
+          # A scheduled refresh has to bypass reuse by construction: it
+          # rebuilds the SAME source, so fc.revision always matches and the
+          # check would skip every refresh there has ever been.
+          EVENT: ${{ github.event_name }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision agent)
@@ -1272,6 +1420,9 @@ jobs:
           if [ "${FORCE:-false}" = "true" ]; then
             echo "hit=false" >> "$GITHUB_OUTPUT"
             echo "reuse: force_build set — building regardless"
+          elif [ "${EVENT:-}" = "schedule" ]; then
+            echo "hit=false" >> "$GITHUB_OUTPUT"
+            echo "reuse: scheduled base refresh — building regardless"
           elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
             echo "hit=true" >> "$GITHUB_OUTPUT"
             echo "reuse: already published — skipping the build"
@@ -1287,6 +1438,30 @@ jobs:
           context: agent
           file: agent/Dockerfile
           push: true
+          # Re-resolve the FROM references against the registry instead of
+          # trusting whatever digest the cache was built against. This is the
+          # whole mechanism of the scheduled refresh (#3154): if the base tag
+          # moved, the FROM layer's cache key changes, every layer above it
+          # invalidates, and the image genuinely rebuilds. If it did not move,
+          # the registry cache satisfies the entire graph and the refresh is a
+          # ~13s no-op that republishes nothing.
+          #
+          # That no-op is the POINT, not a shortfall: :latest should change
+          # when there is something new in it and not otherwise. A refresh
+          # that rewrote the image weekly regardless would churn the registry
+          # and hand :c-<sha> a new manifest to diverge from every Sunday, for
+          # no gain.
+          #
+          # What it therefore does NOT catch: a Debian package update inside
+          # the `apt-get install` layer while the base tag itself stands
+          # still. The official python/cuda images rebuild with those updates
+          # baked in, so this is a lag rather than a hole — but closing it
+          # would take `no-cache: true` on the scheduled path, which is the
+          # weekly-churn trade above. Left as the cheaper of the two on
+          # purpose.
+          #
+          # Only on the schedule. An ordinary push wants the cached base.
+          pull: ${{ github.event_name == 'schedule' }}
           # ONE tag, the channel's. Every other tag is written by the step
           # below, registry-side. buildx here pushes the first tag to the
           # registry and then re-pushes the rest through the DOCKER driver,

ci-requirements.md

@@ -167,6 +167,41 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
   `github.event.inputs` into an env var rather than interpolated into a run
   block, and it is checked inside the reuse step so that one decision drives
   both the build and the repoint.
+- **A weekly `schedule` rebuilds all three images against fresh base layers**
+  (Sunday 06:00 UTC, milestone 326 step 4, #3154). Skip-if-exists is keyed on
+  OUR source, so an artifact whose source stops moving stops picking up base
+  updates — `agent/` has not changed since 2026-07-17 and would otherwise serve
+  that day's `nvidia/cuda` layers forever. Four things make it work:
+  - It **builds `main`, not the branch that triggered it.** Forgejo registers a
+    cron from the DEFAULT branch (`dev` here), so a scheduled run arrives with
+    `github.ref` on dev. The ref is decided once in a top-level `env:
+    BUILD_REF` that every checkout in the file takes, rather than per job —
+    otherwise `sign-extension` would derive dev's extension version while
+    `build-web` bundled main's, and the release download would 404 on a version
+    that exists perfectly well.
+  - It **publishes only `:latest`.** `:c-<sha>` for main's HEAD already names
+    the bytes that commit built; re-pushing it over refreshed layers would
+    break the one tag rule 145 makes immutable, and it is the rollback unit.
+    The repoint step needs no schedule case for this — the tag list is the
+    channel tag alone, so SOURCE is the only entry, it is excluded as always,
+    and the step correctly does nothing.
+  - **`:latest` and `:c-<sha>` therefore diverge between a refresh and the next
+    `main` push, by design.** They re-converge on that push: it hits reuse (a
+    refresh does not move `fc.revision`, because it does not touch the source),
+    and the repoint writes the NEW `:c-<sha>` from the refreshed `:latest`. The
+    push path needed no change for this, because the repoint already excluded
+    the source tag — the same rule that keeps the label readable also keeps a
+    refresh from being undone.
+  - **`pull: true` on the scheduled path only** is the actual mechanism. If a
+    base tag moved, the `FROM` layer's cache key changes and everything above
+    it rebuilds; if it did not, the registry cache satisfies the whole graph
+    and the refresh is a ~13s no-op that republishes nothing. That no-op is the
+    point — `:latest` should change when there is something new in it, not
+    every Sunday. The known lag: a Debian package update inside the `apt-get
+    install` layer while the base tag stands still is not caught. Closing it
+    needs `no-cache: true`, which buys weekly churn for it; the official
+    python/cuda images rebuild with those updates baked in, so this is a lag
+    rather than a hole.
 - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
   `build.yml` passes them to the web image only — the ml and agent images have
   nothing to report them to. `/api/health` returns both, the foot of Settings