From 6b3ec98fa8fbea81ffaab851b8ba68a729fe53be Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sat, 29 Aug 2026 18:36:35 -0400 Subject: [PATCH 1/3] ci: build on a real buildx driver, attestations off (milestone 326 step 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `docker/setup-buildx-action@v3` to build-web, build-ml and build-agent, and sets `provenance: false` / `sbom: false` on all three build-push steps. Two open issues share one root, which is why this is one change: * #3114 — the agent rebuilds a ~6.3 GB CUDA + torch image whenever the runner's local cache is cold, 9m26s against 7s warm. The default `docker` driver cannot export a registry cache at all, so the fix is unavailable until the driver moves. The cache itself is step 2, deliberately not here. * #3190 — build-agent goes red AFTER a successful push, `No such image` from the local daemon. The leading candidate is the docker driver resolving image metadata against a local store a registry-direct push never filled. The attestation flags are the load-bearing part. On the default driver they were no-ops; on the container driver, build-push-action@v5 defaults provenance to TRUE when pushing. Provenance attaches an attestation manifest, that makes the pushed tag a manifest INDEX, and `.Image.Config.Labels` does not resolve through an index — so the `fc.revision` label the reuse check reads off the channel tag would come back `` on every push. Every image would rebuild forever, every lane would stay green, and the only symptom would be the bill. Same failure as #3183, through a different door; note #3127 §4 records the same shape for `platforms:`. Unverified until CI says otherwise: these jobs run INSIDE a container against a mounted docker socket, so the buildkit container is a sibling of the job container rather than a child. That works over a socket mount and has never been tried on this runner. The gate is the SECOND dev push, not this one. The images currently published were built by the old driver, so one `reuse: NOTE ... no readable fc.revision` is expected now; what must not appear is a second one. Scribe #3249. Co-Authored-By: Claude Opus 5 --- .forgejo/workflows/build.yml | 96 ++++++++++++++++++++++++++++++++++++ ci-requirements.md | 13 +++++ 2 files changed, 109 insertions(+) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 42a9a72..18a45bb 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -443,6 +443,23 @@ jobs: ACTOR: ${{ github.actor }} run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin + # A REAL buildx builder, not the default `docker` driver (#3114, #3190). + # + # The default driver builds through the local dockerd. It cannot export a + # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA + # + torch image from scratch whenever the runner's local cache is cold, + # measured at 9m26s against 7s warm. It is also #3190's leading suspect: + # after a registry-direct push it resolves image metadata against a local + # store the push never filled, and reports `No such image` on an image + # that published perfectly well three seconds earlier. + # + # These jobs run INSIDE a container against a mounted docker socket, so + # the buildkit container this starts is a SIBLING of the job container, + # not a child. That works over the socket mount; it had never been tried + # here before milestone 326 step 1. + - name: Set up buildx + uses: docker/setup-buildx-action@v3 + # --- reuse-if-published (milestone 313, step 4) ---------------------- # Does the image the channel tag already points at carry THIS commit's # revision? If so the bytes this job would produce are already published @@ -623,6 +640,21 @@ jobs: # decoration — an unstamped image is one that will always rebuild. labels: | fc.revision=${{ steps.reuse.outputs.revision }} + # LOAD-BEARING, not a preference. On the default docker driver these + # were no-ops; on the docker-container driver above, + # build-push-action@v5 defaults provenance to TRUE when pushing. + # Provenance attaches an attestation manifest, which makes the pushed + # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve + # through an index. + # + # The label directly above IS the reuse key. Wrap the channel tag in + # an index and the next push reads fc.revision=, misses, and + # rebuilds. Then so does the one after that, forever. Nothing fails, + # nothing goes red, and the only symptom is the bill. That is #3183 + # arriving through a different door, and note #3127 §4 records the + # same shape for `platforms:`. + provenance: false + sbom: false # Only the web image carries these: it is the one with a UI and an # HTTP surface to report them on. The ml and agent images have # nothing to tell. @@ -776,6 +808,23 @@ jobs: ACTOR: ${{ github.actor }} run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin + # A REAL buildx builder, not the default `docker` driver (#3114, #3190). + # + # The default driver builds through the local dockerd. It cannot export a + # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA + # + torch image from scratch whenever the runner's local cache is cold, + # measured at 9m26s against 7s warm. It is also #3190's leading suspect: + # after a registry-direct push it resolves image metadata against a local + # store the push never filled, and reports `No such image` on an image + # that published perfectly well three seconds earlier. + # + # These jobs run INSIDE a container against a mounted docker socket, so + # the buildkit container this starts is a SIBLING of the job container, + # not a child. That works over the socket mount; it had never been tried + # here before milestone 326 step 1. + - name: Set up buildx + uses: docker/setup-buildx-action@v3 + # --- reuse-if-published (milestone 313, step 4) ---------------------- # Does the image the channel tag already points at carry THIS commit's # revision? If so the bytes this job would produce are already published @@ -876,6 +925,21 @@ jobs: # decoration — an unstamped image is one that will always rebuild. labels: | fc.revision=${{ steps.reuse.outputs.revision }} + # LOAD-BEARING, not a preference. On the default docker driver these + # were no-ops; on the docker-container driver above, + # build-push-action@v5 defaults provenance to TRUE when pushing. + # Provenance attaches an attestation manifest, which makes the pushed + # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve + # through an index. + # + # The label directly above IS the reuse key. Wrap the channel tag in + # an index and the next push reads fc.revision=, misses, and + # rebuilds. Then so does the one after that, forever. Nothing fails, + # nothing goes red, and the only symptom is the bill. That is #3183 + # arriving through a different door, and note #3127 §4 records the + # same shape for `platforms:`. + provenance: false + sbom: false # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the @@ -1021,6 +1085,23 @@ jobs: ACTOR: ${{ github.actor }} run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin + # A REAL buildx builder, not the default `docker` driver (#3114, #3190). + # + # The default driver builds through the local dockerd. It cannot export a + # registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA + # + torch image from scratch whenever the runner's local cache is cold, + # measured at 9m26s against 7s warm. It is also #3190's leading suspect: + # after a registry-direct push it resolves image metadata against a local + # store the push never filled, and reports `No such image` on an image + # that published perfectly well three seconds earlier. + # + # These jobs run INSIDE a container against a mounted docker socket, so + # the buildkit container this starts is a SIBLING of the job container, + # not a child. That works over the socket mount; it had never been tried + # here before milestone 326 step 1. + - name: Set up buildx + uses: docker/setup-buildx-action@v3 + # --- reuse-if-published (milestone 313, step 4) ---------------------- # Does the image the channel tag already points at carry THIS commit's # revision? If so the bytes this job would produce are already published @@ -1121,6 +1202,21 @@ jobs: # decoration — an unstamped image is one that will always rebuild. labels: | fc.revision=${{ steps.reuse.outputs.revision }} + # LOAD-BEARING, not a preference. On the default docker driver these + # were no-ops; on the docker-container driver above, + # build-push-action@v5 defaults provenance to TRUE when pushing. + # Provenance attaches an attestation manifest, which makes the pushed + # tag a manifest INDEX — and `.Image.Config.Labels` does not resolve + # through an index. + # + # The label directly above IS the reuse key. Wrap the channel tag in + # an index and the next push reads fc.revision=, misses, and + # rebuilds. Then so does the one after that, forever. Nothing fails, + # nothing goes red, and the only symptom is the bill. That is #3183 + # arriving through a different door, and note #3127 §4 records the + # same shape for `platforms:`. + provenance: false + sbom: false # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the diff --git a/ci-requirements.md b/ci-requirements.md index 2744b40..643cde6 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -133,6 +133,19 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". already ran, so both paths now share one proven route. The cost: `:c-` is an index rather than a plain image, so `fc.revision` does not resolve through it — nothing reads it there, and the index names the same manifest. +- **The image builds run on a `docker-container` buildx builder, and + `provenance`/`sbom` are explicitly OFF** (milestone 326 step 1). The builder + is what makes a registry layer cache possible at all — the default `docker` + driver cannot export one (#3114) — and it is #3190's leading suspect, since + it is the driver that resolves image metadata against a local store a + registry-direct push never fills. **The attestation flags are load-bearing, + not tidiness:** on the container driver `build-push-action@v5` defaults + `provenance` to true when pushing, an attestation manifest makes the pushed + tag a manifest INDEX, and config labels do not resolve through an index — so + leaving them on would make every push read `fc.revision=`, miss, and + rebuild forever with every lane green. Same failure as #3183, different door. + These jobs run inside a container against a mounted docker socket, so the + buildkit container is a sibling rather than a child. - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** `build.yml` passes them to the web image only — the ml and agent images have nothing to report them to. `/api/health` returns both, the foot of Settings From e21c9fdd341f82f6df596f1a937ffaeb46b2d4e8 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sat, 29 Aug 2026 18:38:29 -0400 Subject: [PATCH 2/3] ci: a force_build escape hatch for the path skip-if-exists hides (326 step 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `workflow_dispatch` with a `force_build` boolean, honoured inside each of the three reuse steps. It exists because skip-if-exists made its own build path untestable. `agent/` has not changed since 2026-07-17, so the agent build has correctly declined to run on every push since — which also means #3190, whose whole symptom lives on that path, cannot be reproduced on demand. Editing build.yml does not force a build either, and that is deliberate: the workflow is not shipped bytes, so it is in no artifact's path set, and putting it in one would re-version every artifact for a comment change. That is also why this lands before step 2 rather than after. Step 1 moved the builds onto a container driver and turned attestations off; the claim that `fc.revision` still reads back cannot be checked until something actually builds under that driver. Run 4887 confirmed only the cheaper half — `Set up buildx` succeeded on all three jobs, so the buildkit sibling container does start against the mounted socket. Details worth keeping: * FORCE is checked in the reuse step, not in the build step's `if:`. The repoint step keys off `hit` too, and a force that bypassed only the build would leave the two disagreeing about what had happened. * `github.event.inputs`, not the `inputs` context — release.yml already uses that form and it is the one this runner is known to evaluate. Read through env rather than interpolated into the run block, same as release.yml's TAG. * One input, not one per artifact. Three booleans is an interface nobody remembers. Scribe #3252, #3249. Co-Authored-By: Claude Opus 5 --- .forgejo/workflows/build.yml | 65 ++++++++++++++++++++++++++++++++++-- ci-requirements.md | 10 ++++++ 2 files changed, 72 insertions(+), 3 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 18a45bb..6864b08 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -25,6 +25,26 @@ on: # Releases still happen (rule 148, on explicit request per rule 2). They # produce a changelog, not an image. + # The escape hatch for the one thing skip-if-exists makes untestable: a + # build that WOULD be skipped. `agent/` has not changed since 2026-07-17, so + # every push since has correctly declined to build it — which also means the + # agent build path has not run in six weeks and cannot be exercised on + # demand. #3190 lives on exactly that path. + # + # Editing build.yml does not force one either, and that is deliberate: the + # workflow is not shipped bytes, so it is in no artifact's path set. Putting + # it in one would re-version every artifact for a comment change. + # + # ONE input, not one per artifact. Forcing all three is cheap once the + # registry cache is warm (#3114), and three booleans is an interface nobody + # remembers the meaning of. + workflow_dispatch: + inputs: + force_build: + description: 'Rebuild every image even if the published revision matches' + type: boolean + default: false + # Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes: # - write:package, read:package (for docker push to git.fabledsword.com) # - write:release (for ext- release asset cache) @@ -498,6 +518,12 @@ jobs: env: IMAGE: git.fabledsword.com/bvandeusen/fabledcurator CHANNEL: ${{ steps.tag.outputs.channel }} + # Empty on a push; the string "true" only from a workflow_dispatch + # that asked for it. `github.event.inputs` rather than the `inputs` + # context — release.yml already uses that form, and it is the one + # this runner is known to evaluate. Read through env rather than + # interpolated into the run block, same rule as release.yml's TAG. + FORCE: ${{ github.event.inputs.force_build }} run: | set -eu DERIVED=$(sh scripts/artifacts.sh revision web) @@ -537,7 +563,14 @@ jobs: echo "reuse: NOTE tag is being index-wrapped and reuse is dead." fi - if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then + # FORCE is checked here rather than in the build step's `if:`, so + # that one decision drives everything downstream. The repoint step + # keys off `hit` too, and a force that bypassed only the build would + # leave the two disagreeing about what just happened. + if [ "${FORCE:-false}" = "true" ]; then + echo "hit=false" >> "$GITHUB_OUTPUT" + echo "reuse: force_build set — building regardless" + elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then echo "hit=true" >> "$GITHUB_OUTPUT" echo "reuse: already published — skipping the build" else @@ -863,6 +896,12 @@ jobs: env: IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml CHANNEL: ${{ steps.tag.outputs.channel }} + # Empty on a push; the string "true" only from a workflow_dispatch + # that asked for it. `github.event.inputs` rather than the `inputs` + # context — release.yml already uses that form, and it is the one + # this runner is known to evaluate. Read through env rather than + # interpolated into the run block, same rule as release.yml's TAG. + FORCE: ${{ github.event.inputs.force_build }} run: | set -eu DERIVED=$(sh scripts/artifacts.sh revision ml) @@ -898,7 +937,14 @@ jobs: echo "reuse: NOTE tag is being index-wrapped and reuse is dead." fi - if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then + # FORCE is checked here rather than in the build step's `if:`, so + # that one decision drives everything downstream. The repoint step + # keys off `hit` too, and a force that bypassed only the build would + # leave the two disagreeing about what just happened. + if [ "${FORCE:-false}" = "true" ]; then + echo "hit=false" >> "$GITHUB_OUTPUT" + echo "reuse: force_build set — building regardless" + elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then echo "hit=true" >> "$GITHUB_OUTPUT" echo "reuse: already published — skipping the build" else @@ -1140,6 +1186,12 @@ jobs: env: IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent CHANNEL: ${{ steps.tag.outputs.channel }} + # Empty on a push; the string "true" only from a workflow_dispatch + # that asked for it. `github.event.inputs` rather than the `inputs` + # context — release.yml already uses that form, and it is the one + # this runner is known to evaluate. Read through env rather than + # interpolated into the run block, same rule as release.yml's TAG. + FORCE: ${{ github.event.inputs.force_build }} run: | set -eu DERIVED=$(sh scripts/artifacts.sh revision agent) @@ -1175,7 +1227,14 @@ jobs: echo "reuse: NOTE tag is being index-wrapped and reuse is dead." fi - if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then + # FORCE is checked here rather than in the build step's `if:`, so + # that one decision drives everything downstream. The repoint step + # keys off `hit` too, and a force that bypassed only the build would + # leave the two disagreeing about what just happened. + if [ "${FORCE:-false}" = "true" ]; then + echo "hit=false" >> "$GITHUB_OUTPUT" + echo "reuse: force_build set — building regardless" + elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then echo "hit=true" >> "$GITHUB_OUTPUT" echo "reuse: already published — skipping the build" else diff --git a/ci-requirements.md b/ci-requirements.md index 643cde6..3fd321c 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -146,6 +146,16 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". rebuild forever with every lane green. Same failure as #3183, different door. These jobs run inside a container against a mounted docker socket, so the buildkit container is a sibling rather than a child. +- **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which + bypasses the reuse check for all three images. It exists because + skip-if-exists makes its own build path untestable: `agent/` has not changed + since 2026-07-17, so the agent build has not run in six weeks and cannot be + exercised on demand — and #3190 lives on exactly that path. Editing + `build.yml` does not force a build either, deliberately: the workflow is not + shipped bytes and is in no artifact's path set. The flag is read through + `github.event.inputs` into an env var rather than interpolated into a run + block, and it is checked inside the reuse step so that one decision drives + both the build and the repoint. - **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.** `build.yml` passes them to the web image only — the ml and agent images have nothing to report them to. `/api/health` returns both, the foot of Settings From 5e72076298490502ec6e9c5bb22cb87ea87d42a1 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sat, 29 Aug 2026 18:56:13 -0400 Subject: [PATCH 3/3] ci: registry-backed layer cache for all three images (milestone 326 step 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `cache-from`/`cache-to` on `:buildcache`, `mode=max`, on all three build steps. Closes the half of the driver change that step 1 left open. Step 1 measured worse, not better, and that was expected but is worth stating with numbers. Run 4896, first builds after moving to `docker-container`: build-web 3m44s (cold baseline 2m23s) build-ml 3m49s (cold baseline 3m20s) build-agent 11m12s (cold baseline 9m26s) The container driver gets a FRESH buildkit instance per job, so it has no local layer store to fall back on — where the old docker driver at least reused whatever the runner's dockerd happened to hold. That is why a registry cache is the only cache this driver can have, and why step 1 on its own is a regression rather than a win. `mode=max` so intermediate stages cache too. The agent's two ~150s pip layers and web's frontend-builder stage are the entire cost, and a min-mode cache would drop exactly those. A `:buildcache` tag is not the withdrawn tag scheme returning. Rule 145 narrowed against names NOTHING reads; this one is read by every build that runs, is one moving ref per image rather than one per build, holds cache blobs rather than a shippable artifact, and is overwritten in place rather than accumulating. Closer to `:dev` than to the `:2026.8.28` tags 318 deleted — and said so in the workflow, so it is not "cleaned up" by a later reader. Expect the next build to be slower again, once: it is still cold AND now pays the cache export. The measurement that matters is the one after that. Scribe #3114. Co-Authored-By: Claude Opus 5 --- .forgejo/workflows/build.yml | 57 ++++++++++++++++++++++++++++++++++++ ci-requirements.md | 11 +++++++ 2 files changed, 68 insertions(+) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 6864b08..eb4299b 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -688,6 +688,25 @@ jobs: # same shape for `platforms:`. provenance: false sbom: false + # The ONLY cache this driver can have. `docker-container` gets a + # FRESH buildkit instance per job, so unlike the default docker + # driver it has no local layer store to fall back on — measured on + # run 4896, the first builds after the driver change: web 3m44s + # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The + # driver change ALONE is a regression; this is the other half of it. + # + # mode=max so intermediate stages cache too. web's frontend-builder + # stage and the agent's two ~150s pip layers are the whole cost, and + # they are exactly what a min-mode cache would drop. + # + # A `:buildcache` tag is NOT the withdrawn tag scheme coming back. + # Rule 145 narrowed against names NOTHING reads; this one is read by + # every build that runs, is one moving ref per image rather than one + # per build, holds cache blobs rather than a shippable artifact, and + # is overwritten in place rather than accumulating. It is closer to + # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.) + cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache + cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache,mode=max # Only the web image carries these: it is the one with a UI and an # HTTP surface to report them on. The ml and agent images have # nothing to tell. @@ -986,6 +1005,25 @@ jobs: # same shape for `platforms:`. provenance: false sbom: false + # The ONLY cache this driver can have. `docker-container` gets a + # FRESH buildkit instance per job, so unlike the default docker + # driver it has no local layer store to fall back on — measured on + # run 4896, the first builds after the driver change: web 3m44s + # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The + # driver change ALONE is a regression; this is the other half of it. + # + # mode=max so intermediate stages cache too. web's frontend-builder + # stage and the agent's two ~150s pip layers are the whole cost, and + # they are exactly what a min-mode cache would drop. + # + # A `:buildcache` tag is NOT the withdrawn tag scheme coming back. + # Rule 145 narrowed against names NOTHING reads; this one is read by + # every build that runs, is one moving ref per image rather than one + # per build, holds cache blobs rather than a shippable artifact, and + # is overwritten in place rather than accumulating. It is closer to + # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.) + cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache + cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the @@ -1276,6 +1314,25 @@ jobs: # same shape for `platforms:`. provenance: false sbom: false + # The ONLY cache this driver can have. `docker-container` gets a + # FRESH buildkit instance per job, so unlike the default docker + # driver it has no local layer store to fall back on — measured on + # run 4896, the first builds after the driver change: web 3m44s + # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The + # driver change ALONE is a regression; this is the other half of it. + # + # mode=max so intermediate stages cache too. web's frontend-builder + # stage and the agent's two ~150s pip layers are the whole cost, and + # they are exactly what a min-mode cache would drop. + # + # A `:buildcache` tag is NOT the withdrawn tag scheme coming back. + # Rule 145 narrowed against names NOTHING reads; this one is read by + # every build that runs, is one moving ref per image rather than one + # per build, holds cache blobs rather than a shippable artifact, and + # is overwritten in place rather than accumulating. It is closer to + # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.) + cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache + cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the diff --git a/ci-requirements.md b/ci-requirements.md index 3fd321c..340e277 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -146,6 +146,17 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". rebuild forever with every lane green. Same failure as #3183, different door. These jobs run inside a container against a mounted docker socket, so the buildkit container is a sibling rather than a child. +- **All three images import and export a registry layer cache** + (`:buildcache`, `mode=max`). This is not an optimisation bolted onto + the driver change — it is the other half of it. The `docker-container` + driver gets a fresh buildkit instance per job and therefore has **no local + layer store at all**, where the old `docker` driver at least reused whatever + the runner's dockerd happened to hold. Measured on run 4896, the first builds + after the driver moved: web 3m44s (was 2m23s), ml 3m49s (was 3m20s), agent + 11m12s (was 9m26s) — every one slower. A `:buildcache` tag is read by every + build that runs, is one moving ref per image, holds cache blobs rather than a + shippable artifact, and is overwritten in place, so it is not a return of the + per-version tags milestone 318 withdrew (#3114). - **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which bypasses the reuse check for all three images. It exists because skip-if-exists makes its own build path untestable: `agent/` has not changed