From 6b3ec98fa8fbea81ffaab851b8ba68a729fe53be Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sat, 29 Aug 2026 18:36:35 -0400 Subject: [PATCH] ci: build on a real buildx driver, attestations off (milestone 326 step 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `docker/setup-buildx-action@v3` to build-web, build-ml and build-agent, and sets `provenance: false` / `sbom: false` on all three build-push steps. Two open issues share one root, which is why this is one change: * #3114 — the agent rebuilds a ~6.3 GB CUDA + torch image whenever the runner's local cache is cold, 9m26s against 7s warm. The default `docker` driver cannot export a registry cache at all, so the fix is unavailable until the driver moves. The cache itself is step 2, deliberately not here. * #3190 — build-agent goes red AFTER a successful push, `No such image` from the local daemon. The leading candidate is the docker driver resolving image metadata against a local store a registry-direct push never filled. The attestation flags are the load-bearing part. On the default driver they were no-ops; on the container driver, build-push-action@v5 defaults provenance to TRUE when pushing. Provenance attaches an attestation manifest, that makes the pushed tag a manifest INDEX, and `.Image.Config.Labels` does not resolve through an index — so the `fc.revision` label the reuse check reads off the channel tag would come back `` on every push. Every image would rebuild forever, every lane would stay green, and the only symptom would be the bill. Same failure as #3183, through a different door; note #3127 §4 records the same shape for `platforms:`. Unverified until CI says otherwise: these jobs run INSIDE a container against a mounted docker socket, so the buildkit container is a sibling of the job container rather than a child. That works over a socket mount and has never been tried on this runner. The gate is the SECOND dev push, not this one. The images currently published were built by the old driver, so one `reuse: NOTE ... no readable fc.revision` is expected now; what must not appear is a second one. Scribe #3249. Co-Authored-By: Claude Opus 5 --- .forgejo/workflows/build.yml | 96 ++++++++++++++++++++++++++++++++++++ ci-requirements.md | 13 +++++ 2 files changed, 109 insertions(+) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 42a9a72..18a45bb 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -443,6 +443,23 @@ jobs: ACTOR: ${{ github.actor }} run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin + # A REAL buildx builder, not the default `docker` driver (#3114, #3190). + # + # The default driver builds through the local dockerd. It cannot export a + # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA + # + torch image from scratch whenever the runner's local cache is cold, + # measured at 9m26s against 7s warm. It is also #3190's leading suspect: + # after a registry-direct push it resolves image metadata against a local + # store the push never filled, and reports `No such image` on an image + # that published perfectly well three seconds earlier. + # + # These jobs run INSIDE a container against a mounted docker socket, so + # the buildkit container this starts is a SIBLING of the job container, + # not a child. That works over the socket mount; it had never been tried + # here before milestone 326 step 1. + - name: Set up buildx + uses: docker/setup-buildx-action@v3 + # --- reuse-if-published (milestone 313, step 4) ---------------------- # Does the image the channel tag already points at carry THIS commit's # revision? If so the bytes this job would produce are already published @@ -623,6 +640,21 @@ jobs: # decoration — an unstamped image is one that will always rebuild. labels: | fc.revision=${{ steps.reuse.outputs.revision }} + # LOAD-BEARING, not a preference. On the default docker driver these + # were no-ops; on the docker-container driver above, + # build-push-action@v5 defaults provenance to TRUE when pushing. + # Provenance attaches an attestation manifest, which makes the pushed + # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve + # through an index. + # + # The label directly above IS the reuse key. Wrap the channel tag in + # an index and the next push reads fc.revision=, misses, and + # rebuilds. Then so does the one after that, forever. Nothing fails, + # nothing goes red, and the only symptom is the bill. That is #3183 + # arriving through a different door, and note #3127 §4 records the + # same shape for `platforms:`. + provenance: false + sbom: false # Only the web image carries these: it is the one with a UI and an # HTTP surface to report them on. The ml and agent images have # nothing to tell. @@ -776,6 +808,23 @@ jobs: ACTOR: ${{ github.actor }} run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin + # A REAL buildx builder, not the default `docker` driver (#3114, #3190). + # + # The default driver builds through the local dockerd. It cannot export a + # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA + # + torch image from scratch whenever the runner's local cache is cold, + # measured at 9m26s against 7s warm. It is also #3190's leading suspect: + # after a registry-direct push it resolves image metadata against a local + # store the push never filled, and reports `No such image` on an image + # that published perfectly well three seconds earlier. + # + # These jobs run INSIDE a container against a mounted docker socket, so + # the buildkit container this starts is a SIBLING of the job container, + # not a child. That works over the socket mount; it had never been tried + # here before milestone 326 step 1. + - name: Set up buildx + uses: docker/setup-buildx-action@v3 + # --- reuse-if-published (milestone 313, step 4) ---------------------- # Does the image the channel tag already points at carry THIS commit's # revision? If so the bytes this job would produce are already published @@ -876,6 +925,21 @@ jobs: # decoration — an unstamped image is one that will always rebuild. labels: | fc.revision=${{ steps.reuse.outputs.revision }} + # LOAD-BEARING, not a preference. On the default docker driver these + # were no-ops; on the docker-container driver above, + # build-push-action@v5 defaults provenance to TRUE when pushing. + # Provenance attaches an attestation manifest, which makes the pushed + # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve + # through an index. + # + # The label directly above IS the reuse key. Wrap the channel tag in + # an index and the next push reads fc.revision=, misses, and + # rebuilds. Then so does the one after that, forever. Nothing fails, + # nothing goes red, and the only symptom is the bill. That is #3183 + # arriving through a different door, and note #3127 §4 records the + # same shape for `platforms:`. + provenance: false + sbom: false # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the @@ -1021,6 +1085,23 @@ jobs: ACTOR: ${{ github.actor }} run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin + # A REAL buildx builder, not the default `docker` driver (#3114, #3190). + # + # The default driver builds through the local dockerd. It cannot export a + # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA + # + torch image from scratch whenever the runner's local cache is cold, + # measured at 9m26s against 7s warm. It is also #3190's leading suspect: + # after a registry-direct push it resolves image metadata against a local + # store the push never filled, and reports `No such image` on an image + # that published perfectly well three seconds earlier. + # + # These jobs run INSIDE a container against a mounted docker socket, so + # the buildkit container this starts is a SIBLING of the job container, + # not a child. That works over the socket mount; it had never been tried + # here before milestone 326 step 1. + - name: Set up buildx + uses: docker/setup-buildx-action@v3 + # --- reuse-if-published (milestone 313, step 4) ---------------------- # Does the image the channel tag already points at carry THIS commit's # revision? If so the bytes this job would produce are already published @@ -1121,6 +1202,21 @@ jobs: # decoration — an unstamped image is one that will always rebuild. labels: | fc.revision=${{ steps.reuse.outputs.revision }} + # LOAD-BEARING, not a preference. On the default docker driver these + # were no-ops; on the docker-container driver above, + # build-push-action@v5 defaults provenance to TRUE when pushing. + # Provenance attaches an attestation manifest, which makes the pushed + # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve + # through an index. + # + # The label directly above IS the reuse key. Wrap the channel tag in + # an index and the next push reads fc.revision=, misses, and + # rebuilds. Then so does the one after that, forever. Nothing fails, + # nothing goes red, and the only symptom is the bill. That is #3183 + # arriving through a different door, and note #3127 §4 records the + # same shape for `platforms:`. + provenance: false + sbom: false # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the diff --git a/ci-requirements.md b/ci-requirements.md index 2744b40..643cde6 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -133,6 +133,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". already ran, so both paths now share one proven route. The cost: `:c-` is an index rather than a plain image, so `fc.revision` does not resolve through it — nothing reads it there, and the index names the same manifest. +- **The image builds run on a `docker-container` buildx builder, and + `provenance`/`sbom` are explicitly OFF** (milestone 326 step 1). The builder + is what makes a registry layer cache possible at all — the default `docker` + driver cannot export one (#3114) — and it is #3190's leading suspect, since + it is the driver that resolves image metadata against a local store a + registry-direct push never fills. **The attestation flags are load-bearing, + not tidiness:** on the container driver `build-push-action@v5` defaults + `provenance` to true when pushing, an attestation manifest makes the pushed + tag a manifest INDEX, and config labels do not resolve through an index — so + leaving them on would make every push read `fc.revision=`, miss, and + rebuild forever with every lane green. Same failure as #3183, different door. + These jobs run inside a container against a mounted docker socket, so the + buildkit container is a sibling rather than a child. - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** `build.yml` passes them to the web image only — the ml and agent images have nothing to report them to. `/api/health` returns both, the foot of Settings