ci: build on a real buildx driver, attestations off (milestone 326 step 1)

Adds `docker/setup-buildx-action@v3` to build-web, build-ml and build-agent,
and sets `provenance: false` / `sbom: false` on all three build-push steps.

Two open issues share one root, which is why this is one change:

* #3114 — the agent rebuilds a ~6.3 GB CUDA + torch image whenever the
  runner's local cache is cold, 9m26s against 7s warm. The default `docker`
  driver cannot export a registry cache at all, so the fix is unavailable
  until the driver moves. The cache itself is step 2, deliberately not here.
* #3190 — build-agent goes red AFTER a successful push, `No such image` from
  the local daemon. The leading candidate is the docker driver resolving
  image metadata against a local store a registry-direct push never filled.

The attestation flags are the load-bearing part. On the default driver they
were no-ops; on the container driver, build-push-action@v5 defaults provenance
to TRUE when pushing. Provenance attaches an attestation manifest, that makes
the pushed tag a manifest INDEX, and `.Image.Config.Labels` does not resolve
through an index — so the `fc.revision` label the reuse check reads off the
channel tag would come back `<none>` on every push. Every image would rebuild
forever, every lane would stay green, and the only symptom would be the bill.
Same failure as #3183, through a different door; note #3127 §4 records the
same shape for `platforms:`.

Unverified until CI says otherwise: these jobs run INSIDE a container against
a mounted docker socket, so the buildkit container is a sibling of the job
container rather than a child. That works over a socket mount and has never
been tried on this runner.

The gate is the SECOND dev push, not this one. The images currently published
were built by the old driver, so one `reuse: NOTE ... no readable fc.revision`
is expected now; what must not appear is a second one.

Scribe #3249.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.forgejo/workflows/build.yml

@@ -443,6 +443,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -623,6 +640,21 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
           # Only the web image carries these: it is the one with a UI and an
           # HTTP surface to report them on. The ml and agent images have
           # nothing to tell.
@@ -776,6 +808,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -876,6 +925,21 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
 
       # Every tag but the channel's own is written HERE, registry-side,
       # whether or not a build ran. Each -t becomes another reference to the
@@ -1021,6 +1085,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -1121,6 +1202,21 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
 
       # Every tag but the channel's own is written HERE, registry-side,
       # whether or not a build ran. Each -t becomes another reference to the

ci-requirements.md

@@ -133,6 +133,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
   already ran, so both paths now share one proven route. The cost: `:c-<sha>`
   is an index rather than a plain image, so `fc.revision` does not resolve
   through it — nothing reads it there, and the index names the same manifest.
+- **The image builds run on a `docker-container` buildx builder, and
+  `provenance`/`sbom` are explicitly OFF** (milestone 326 step 1). The builder
+  is what makes a registry layer cache possible at all — the default `docker`
+  driver cannot export one (#3114) — and it is #3190's leading suspect, since
+  it is the driver that resolves image metadata against a local store a
+  registry-direct push never fills. **The attestation flags are load-bearing,
+  not tidiness:** on the container driver `build-push-action@v5` defaults
+  `provenance` to true when pushing, an attestation manifest makes the pushed
+  tag a manifest INDEX, and config labels do not resolve through an index — so
+  leaving them on would make every push read `fc.revision=<none>`, miss, and
+  rebuild forever with every lane green. Same failure as #3183, different door.
+  These jobs run inside a container against a mounted docker socket, so the
+  buildkit container is a sibling rather than a child.
 - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
   `build.yml` passes them to the web image only — the ml and agent images have
   nothing to report them to. `/api/health` returns both, the foot of Settings