fix(ci): the build pushes one tag; the rest are written registry-side (#3190)

buildx on this runner pushes the first tag to the registry and then
re-pushes the remaining ones through the DOCKER driver, reading them out
of a local image store that a registry-direct build never populated:

  #27 pushing …/fabledcurator:latest           DONE 15.8s
  #28 pushing …/fabledcurator:c-0e15c44 with docker
  #28 ERROR: tag does not exist: …:c-0e15c44

It is intermittent — build-ml made the identical two-tag push seconds
later in the same run and succeeded — and the consequence is worse than
the red job suggests. `:latest` had already published, so production was
correct while the immutable rollback tag rule 145 requires of every main
push simply did not exist. Nothing else would ever have noticed: a
missing :c-<sha> has no consumer that fails, so it surfaces at the moment
somebody needs to roll back, which is the worst time to learn a rollback
target was never written.

So the build now pushes exactly one ref — the channel's — and the
existing repoint step, which already excluded the source tag and already
ran on every reuse, now runs on the build path too and owns every other
tag. `imagetools create` is a registry-side manifest copy: no local
daemon, nothing that can be absent. This adds no new code path; it puts
the build case onto the one that was already proven.

Chosen over the alternative of asserting each tag resolves after the
build, which would have made the failure loud without making it rarer.

The cost, accepted: `imagetools create` wraps its source in an index, so
:c-<sha> is an index rather than a plain image and fc.revision does not
resolve through it. Nothing reads that label off :c-<sha> — the reuse
check only ever inspects the CHANNEL tag — and the index names the same
manifest, so a pull is byte-identical. The reuse path already produced
:c-<sha> this way; this only makes it uniform.

`build_tags` goes with it — the tag list now has exactly one consumer.

.forgejo/workflows/build.yml

@@ -476,7 +476,6 @@ jobs:
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
           CHANNEL: ${{ steps.tag.outputs.channel }}
-          TAGS: ${{ steps.tag.outputs.tags }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision web)
@@ -485,7 +484,6 @@ jobs:
           # A pure function of the revision — same commit, same string — so it
           # adds no variability the reuse check would have to account for.
           echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT"
-          echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
 
           # The moving tag for this channel. Which tag we ask IS the channel —
           # that is why the revision needs no -main/-dev qualifier any more.
@@ -608,7 +606,13 @@ jobs:
           context: .
           file: Dockerfile
           push: true
-          tags: ${{ steps.reuse.outputs.build_tags }}
+          # ONE tag, the channel's. Every other tag is written by the step
+          # below, registry-side. buildx here pushes the first tag to the
+          # registry and then re-pushes the rest through the DOCKER driver,
+          # out of a local image store a registry-direct build never filled —
+          # #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
+          # published perfectly well.
+          tags: ${{ steps.reuse.outputs.channel_ref }}
           # The reuse key. Read back off the channel tag on the next push to
           # decide whether that push needs to build at all, so this is not
           # decoration — an unstamped image is one that will always rebuild.
@@ -621,17 +625,40 @@ jobs:
             FC_CHANNEL=${{ steps.tag.outputs.channel }}
             FC_VERSION=${{ steps.reuse.outputs.version }}
 
-      # Registry-side manifest copy: no layer transfer, no local daemon, no
-      # rebuild. Each -t becomes another reference to the SAME manifest the
-      # channel tag already holds, so :c-<sha> is byte-identical to what is
-      # published rather than a lookalike rebuild.
+      # Every tag but the channel's own is written HERE, registry-side,
+      # whether or not a build ran. Each -t becomes another reference to the
+      # SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
+      # what is published rather than a lookalike rebuild.
       #
-      # Runs on EVERY reuse, which is what keeps family rule 146 true: a
-      # rolling channel refreshes itself, so skipping a build must never mean
-      # leaving :dev or :latest pointing at something older than the commit
-      # that was just pushed.
-      - name: Repoint the tags at the published image (reuse)
-        if: steps.reuse.outputs.hit == 'true'
+      # Owning the build path too is #3190's fix, not a tidy-up:
+      #
+      #   #27 pushing …/fabledcurator:latest           DONE 15.8s
+      #   #28 pushing …/fabledcurator:c-0e15c44 with docker
+      #   #28 ERROR: tag does not exist: …:c-0e15c44
+      #
+      # Intermittent — build-ml made the identical two-tag push seconds later
+      # and succeeded — and worse than it looks. `:latest` had already
+      # published, so production was correct while the immutable rollback tag
+      # rule 145 requires of every main push simply did not exist. Nothing but
+      # the red job would ever have noticed: a missing :c-<sha> has no
+      # consumer that fails, so it surfaces when somebody needs to roll back.
+      #
+      # `imagetools create` is a registry-side manifest copy — no layer
+      # transfer, no local daemon, nothing that can be absent. The reuse case
+      # has always gone this way, so this puts the build case on the code that
+      # was already proven rather than on a second path.
+      #
+      # Running on every path also keeps family rule 146 true: a rolling
+      # channel refreshes itself, so skipping a build must never leave :dev or
+      # :latest pointing at something older than the commit just pushed.
+      #
+      # The cost, accepted knowingly: `imagetools create` wraps its source in
+      # an index, so :c-<sha> becomes an index and fc.revision does not
+      # resolve through it. Nothing reads that label off :c-<sha> — the reuse
+      # check only ever inspects the CHANNEL tag — and the index names the
+      # same manifest, so a pull is byte-identical. The reuse path already
+      # produced :c-<sha> this way; this only makes it uniform.
+      - name: Write the remaining tags from the published image
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
           SOURCE: ${{ steps.reuse.outputs.channel_ref }}
@@ -650,15 +677,16 @@ jobs:
           # ml:dev reported fc.revision=<none> one push after run 4749 had read
           # a7e626a67a79 off it. Nothing failed; the savings just evaporated.
           #
-          # Excluding the source means the channel tag is only ever written by
-          # a real build, so it stays a plain image and stays readable. On dev
-          # that leaves nothing to do — :dev already points at the right
-          # content, which is what the hit established. On main it leaves
-          # :c-<sha>, which rule 145 requires of every main push whether or not
-          # a build ran.
+          # Excluding the source means the channel tag is only ever written
+          # by a real build, so it stays a plain image and stays readable.
+          # On dev that leaves nothing to do either way: the build pushed :dev
+          # itself, or the hit established it was already right. On main it
+          # leaves :c-<sha>, which rule 145 requires of every main push whether
+          # or not a build ran.
           #
-          # steps.tag emits ONE comma-separated list, because that is the shape
-          # docker/build-push-action takes; imagetools wants a -t per ref.
+          # steps.tag emits ONE comma-separated list; imagetools wants a -t per
+          # ref. (That list used to feed docker/build-push-action directly —
+          # which is exactly what #3190 made unsafe.)
           ARGS=""
           IFS=,
           for t in $TAGS; do
@@ -667,8 +695,8 @@ jobs:
           done
           unset IFS
           if [ -z "$ARGS" ]; then
-            echo "repoint: $SOURCE already carries this revision and is the"
-            echo "repoint: only tag for this channel — nothing to write."
+            echo "repoint: $SOURCE is the only tag for this channel and"
+            echo "repoint: already holds this revision — nothing to write."
             exit 0
           fi
           # shellcheck disable=SC2086
@@ -781,12 +809,10 @@ jobs:
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
           CHANNEL: ${{ steps.tag.outputs.channel }}
-          TAGS: ${{ steps.tag.outputs.tags }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision ml)
           echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
-          echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
 
           # The moving tag for this channel. Which tag we ask IS the channel —
           # that is why the revision needs no -main/-dev qualifier any more.
@@ -833,24 +859,53 @@ jobs:
           context: .
           file: Dockerfile.ml
           push: true
-          tags: ${{ steps.reuse.outputs.build_tags }}
+          # ONE tag, the channel's. Every other tag is written by the step
+          # below, registry-side. buildx here pushes the first tag to the
+          # registry and then re-pushes the rest through the DOCKER driver,
+          # out of a local image store a registry-direct build never filled —
+          # #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
+          # published perfectly well.
+          tags: ${{ steps.reuse.outputs.channel_ref }}
           # The reuse key. Read back off the channel tag on the next push to
           # decide whether that push needs to build at all, so this is not
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
 
-      # Registry-side manifest copy: no layer transfer, no local daemon, no
-      # rebuild. Each -t becomes another reference to the SAME manifest the
-      # channel tag already holds, so :c-<sha> is byte-identical to what is
-      # published rather than a lookalike rebuild.
+      # Every tag but the channel's own is written HERE, registry-side,
+      # whether or not a build ran. Each -t becomes another reference to the
+      # SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
+      # what is published rather than a lookalike rebuild.
       #
-      # Runs on EVERY reuse, which is what keeps family rule 146 true: a
-      # rolling channel refreshes itself, so skipping a build must never mean
-      # leaving :dev or :latest pointing at something older than the commit
-      # that was just pushed.
-      - name: Repoint the tags at the published image (reuse)
-        if: steps.reuse.outputs.hit == 'true'
+      # Owning the build path too is #3190's fix, not a tidy-up:
+      #
+      #   #27 pushing …/fabledcurator:latest           DONE 15.8s
+      #   #28 pushing …/fabledcurator:c-0e15c44 with docker
+      #   #28 ERROR: tag does not exist: …:c-0e15c44
+      #
+      # Intermittent — build-ml made the identical two-tag push seconds later
+      # and succeeded — and worse than it looks. `:latest` had already
+      # published, so production was correct while the immutable rollback tag
+      # rule 145 requires of every main push simply did not exist. Nothing but
+      # the red job would ever have noticed: a missing :c-<sha> has no
+      # consumer that fails, so it surfaces when somebody needs to roll back.
+      #
+      # `imagetools create` is a registry-side manifest copy — no layer
+      # transfer, no local daemon, nothing that can be absent. The reuse case
+      # has always gone this way, so this puts the build case on the code that
+      # was already proven rather than on a second path.
+      #
+      # Running on every path also keeps family rule 146 true: a rolling
+      # channel refreshes itself, so skipping a build must never leave :dev or
+      # :latest pointing at something older than the commit just pushed.
+      #
+      # The cost, accepted knowingly: `imagetools create` wraps its source in
+      # an index, so :c-<sha> becomes an index and fc.revision does not
+      # resolve through it. Nothing reads that label off :c-<sha> — the reuse
+      # check only ever inspects the CHANNEL tag — and the index names the
+      # same manifest, so a pull is byte-identical. The reuse path already
+      # produced :c-<sha> this way; this only makes it uniform.
+      - name: Write the remaining tags from the published image
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
           SOURCE: ${{ steps.reuse.outputs.channel_ref }}
@@ -869,15 +924,16 @@ jobs:
           # ml:dev reported fc.revision=<none> one push after run 4749 had read
           # a7e626a67a79 off it. Nothing failed; the savings just evaporated.
           #
-          # Excluding the source means the channel tag is only ever written by
-          # a real build, so it stays a plain image and stays readable. On dev
-          # that leaves nothing to do — :dev already points at the right
-          # content, which is what the hit established. On main it leaves
-          # :c-<sha>, which rule 145 requires of every main push whether or not
-          # a build ran.
+          # Excluding the source means the channel tag is only ever written
+          # by a real build, so it stays a plain image and stays readable.
+          # On dev that leaves nothing to do either way: the build pushed :dev
+          # itself, or the hit established it was already right. On main it
+          # leaves :c-<sha>, which rule 145 requires of every main push whether
+          # or not a build ran.
           #
-          # steps.tag emits ONE comma-separated list, because that is the shape
-          # docker/build-push-action takes; imagetools wants a -t per ref.
+          # steps.tag emits ONE comma-separated list; imagetools wants a -t per
+          # ref. (That list used to feed docker/build-push-action directly —
+          # which is exactly what #3190 made unsafe.)
           ARGS=""
           IFS=,
           for t in $TAGS; do
@@ -886,8 +942,8 @@ jobs:
           done
           unset IFS
           if [ -z "$ARGS" ]; then
-            echo "repoint: $SOURCE already carries this revision and is the"
-            echo "repoint: only tag for this channel — nothing to write."
+            echo "repoint: $SOURCE is the only tag for this channel and"
+            echo "repoint: already holds this revision — nothing to write."
             exit 0
           fi
           # shellcheck disable=SC2086
@@ -998,12 +1054,10 @@ jobs:
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
           CHANNEL: ${{ steps.tag.outputs.channel }}
-          TAGS: ${{ steps.tag.outputs.tags }}
         run: |
           set -eu
           DERIVED=$(sh scripts/artifacts.sh revision agent)
           echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
-          echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
 
           # The moving tag for this channel. Which tag we ask IS the channel —
           # that is why the revision needs no -main/-dev qualifier any more.
@@ -1050,24 +1104,53 @@ jobs:
           context: agent
           file: agent/Dockerfile
           push: true
-          tags: ${{ steps.reuse.outputs.build_tags }}
+          # ONE tag, the channel's. Every other tag is written by the step
+          # below, registry-side. buildx here pushes the first tag to the
+          # registry and then re-pushes the rest through the DOCKER driver,
+          # out of a local image store a registry-direct build never filled —
+          # #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
+          # published perfectly well.
+          tags: ${{ steps.reuse.outputs.channel_ref }}
           # The reuse key. Read back off the channel tag on the next push to
           # decide whether that push needs to build at all, so this is not
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
 
-      # Registry-side manifest copy: no layer transfer, no local daemon, no
-      # rebuild. Each -t becomes another reference to the SAME manifest the
-      # channel tag already holds, so :c-<sha> is byte-identical to what is
-      # published rather than a lookalike rebuild.
+      # Every tag but the channel's own is written HERE, registry-side,
+      # whether or not a build ran. Each -t becomes another reference to the
+      # SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
+      # what is published rather than a lookalike rebuild.
       #
-      # Runs on EVERY reuse, which is what keeps family rule 146 true: a
-      # rolling channel refreshes itself, so skipping a build must never mean
-      # leaving :dev or :latest pointing at something older than the commit
-      # that was just pushed.
-      - name: Repoint the tags at the published image (reuse)
-        if: steps.reuse.outputs.hit == 'true'
+      # Owning the build path too is #3190's fix, not a tidy-up:
+      #
+      #   #27 pushing …/fabledcurator:latest           DONE 15.8s
+      #   #28 pushing …/fabledcurator:c-0e15c44 with docker
+      #   #28 ERROR: tag does not exist: …:c-0e15c44
+      #
+      # Intermittent — build-ml made the identical two-tag push seconds later
+      # and succeeded — and worse than it looks. `:latest` had already
+      # published, so production was correct while the immutable rollback tag
+      # rule 145 requires of every main push simply did not exist. Nothing but
+      # the red job would ever have noticed: a missing :c-<sha> has no
+      # consumer that fails, so it surfaces when somebody needs to roll back.
+      #
+      # `imagetools create` is a registry-side manifest copy — no layer
+      # transfer, no local daemon, nothing that can be absent. The reuse case
+      # has always gone this way, so this puts the build case on the code that
+      # was already proven rather than on a second path.
+      #
+      # Running on every path also keeps family rule 146 true: a rolling
+      # channel refreshes itself, so skipping a build must never leave :dev or
+      # :latest pointing at something older than the commit just pushed.
+      #
+      # The cost, accepted knowingly: `imagetools create` wraps its source in
+      # an index, so :c-<sha> becomes an index and fc.revision does not
+      # resolve through it. Nothing reads that label off :c-<sha> — the reuse
+      # check only ever inspects the CHANNEL tag — and the index names the
+      # same manifest, so a pull is byte-identical. The reuse path already
+      # produced :c-<sha> this way; this only makes it uniform.
+      - name: Write the remaining tags from the published image
         env:
           IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
           SOURCE: ${{ steps.reuse.outputs.channel_ref }}
@@ -1086,15 +1169,16 @@ jobs:
           # ml:dev reported fc.revision=<none> one push after run 4749 had read
           # a7e626a67a79 off it. Nothing failed; the savings just evaporated.
           #
-          # Excluding the source means the channel tag is only ever written by
-          # a real build, so it stays a plain image and stays readable. On dev
-          # that leaves nothing to do — :dev already points at the right
-          # content, which is what the hit established. On main it leaves
-          # :c-<sha>, which rule 145 requires of every main push whether or not
-          # a build ran.
+          # Excluding the source means the channel tag is only ever written
+          # by a real build, so it stays a plain image and stays readable.
+          # On dev that leaves nothing to do either way: the build pushed :dev
+          # itself, or the hit established it was already right. On main it
+          # leaves :c-<sha>, which rule 145 requires of every main push whether
+          # or not a build ran.
           #
-          # steps.tag emits ONE comma-separated list, because that is the shape
-          # docker/build-push-action takes; imagetools wants a -t per ref.
+          # steps.tag emits ONE comma-separated list; imagetools wants a -t per
+          # ref. (That list used to feed docker/build-push-action directly —
+          # which is exactly what #3190 made unsafe.)
           ARGS=""
           IFS=,
           for t in $TAGS; do
@@ -1103,8 +1187,8 @@ jobs:
           done
           unset IFS
           if [ -z "$ARGS" ]; then
-            echo "repoint: $SOURCE already carries this revision and is the"
-            echo "repoint: only tag for this channel — nothing to write."
+            echo "repoint: $SOURCE is the only tag for this channel and"
+            echo "repoint: already holds this revision — nothing to write."
             exit 0
           fi
           # shellcheck disable=SC2086

ci-requirements.md

@@ -108,6 +108,18 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
   source tag — `imagetools create` wraps its source in a manifest index, and
   config labels do not resolve through an index, so writing the channel tag
   from itself destroys the label the next run reads (#3183).
+- **The build pushes exactly ONE tag — the channel's — and every other tag is
+  written registry-side afterwards** (#3190). buildx on this runner pushes the
+  first tag to the registry and then re-pushes the rest through the docker
+  driver, out of a local image store that a registry-direct build never fills;
+  it fails intermittently with `tag does not exist`. On `dev` that only reddens
+  a job, but on `main` it silently skips `:c-<sha>` while `:latest` publishes
+  fine — a missing rollback tag has no consumer that fails, so nothing but the
+  red job would notice until somebody needs to roll back. `imagetools create`
+  has no local store to be absent from, and it is the code the reuse path
+  already ran, so both paths now share one proven route. The cost: `:c-<sha>`
+  is an index rather than a plain image, so `fc.revision` does not resolve
+  through it — nothing reads it there, and the index names the same manifest.
 - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
   `build.yml` passes them to the web image only — the ml and agent images have
   nothing to report them to. `/api/health` returns both, the foot of Settings