.forgejo/workflows/build.yml

@@ -443,6 +443,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -623,6 +640,21 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
           # Only the web image carries these: it is the one with a UI and an
           # HTTP surface to report them on. The ml and agent images have
           # nothing to tell.
@@ -776,6 +808,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -876,6 +925,21 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
 
       # Every tag but the channel's own is written HERE, registry-side,
       # whether or not a build ran. Each -t becomes another reference to the
@@ -1021,6 +1085,23 @@ jobs:
           ACTOR: ${{ github.actor }}
         run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
 
+      # A REAL buildx builder, not the default `docker` driver (#3114, #3190).
+      #
+      # The default driver builds through the local dockerd. It cannot export a
+      # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
+      # + torch image from scratch whenever the runner's local cache is cold,
+      # measured at 9m26s against 7s warm. It is also #3190's leading suspect:
+      # after a registry-direct push it resolves image metadata against a local
+      # store the push never filled, and reports `No such image` on an image
+      # that published perfectly well three seconds earlier.
+      #
+      # These jobs run INSIDE a container against a mounted docker socket, so
+      # the buildkit container this starts is a SIBLING of the job container,
+      # not a child. That works over the socket mount; it had never been tried
+      # here before milestone 326 step 1.
+      - name: Set up buildx
+        uses: docker/setup-buildx-action@v3
+
       # --- reuse-if-published (milestone 313, step 4) ----------------------
       # Does the image the channel tag already points at carry THIS commit's
       # revision? If so the bytes this job would produce are already published
@@ -1121,6 +1202,21 @@ jobs:
           # decoration — an unstamped image is one that will always rebuild.
           labels: |
             fc.revision=${{ steps.reuse.outputs.revision }}
+          # LOAD-BEARING, not a preference. On the default docker driver these
+          # were no-ops; on the docker-container driver above,
+          # build-push-action@v5 defaults provenance to TRUE when pushing.
+          # Provenance attaches an attestation manifest, which makes the pushed
+          # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
+          # through an index.
+          #
+          # The label directly above IS the reuse key. Wrap the channel tag in
+          # an index and the next push reads fc.revision=<none>, misses, and
+          # rebuilds. Then so does the one after that, forever. Nothing fails,
+          # nothing goes red, and the only symptom is the bill. That is #3183
+          # arriving through a different door, and note #3127 §4 records the
+          # same shape for `platforms:`.
+          provenance: false
+          sbom: false
 
       # Every tag but the channel's own is written HERE, registry-side,
       # whether or not a build ran. Each -t becomes another reference to the

ci-requirements.md

@@ -133,6 +133,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
   already ran, so both paths now share one proven route. The cost: `:c-<sha>`
   is an index rather than a plain image, so `fc.revision` does not resolve
   through it — nothing reads it there, and the index names the same manifest.
+- **The image builds run on a `docker-container` buildx builder, and
+  `provenance`/`sbom` are explicitly OFF** (milestone 326 step 1). The builder
+  is what makes a registry layer cache possible at all — the default `docker`
+  driver cannot export one (#3114) — and it is #3190's leading suspect, since
+  it is the driver that resolves image metadata against a local store a
+  registry-direct push never fills. **The attestation flags are load-bearing,
+  not tidiness:** on the container driver `build-push-action@v5` defaults
+  `provenance` to true when pushing, an attestation manifest makes the pushed
+  tag a manifest INDEX, and config labels do not resolve through an index — so
+  leaving them on would make every push read `fc.revision=<none>`, miss, and
+  rebuild forever with every lane green. Same failure as #3183, different door.
+  These jobs run inside a container against a mounted docker socket, so the
+  buildkit container is a sibling rather than a child.
 - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
   `build.yml` passes them to the web image only — the ml and agent images have
   nothing to report them to. `/api/health` returns both, the foot of Settings