From 5e72076298490502ec6e9c5bb22cb87ea87d42a1 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Sat, 29 Aug 2026 18:56:13 -0400 Subject: [PATCH] ci: registry-backed layer cache for all three images (milestone 326 step 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `cache-from`/`cache-to` on `:buildcache`, `mode=max`, on all three build steps. Closes the half of the driver change that step 1 left open. Step 1 measured worse, not better, and that was expected but is worth stating with numbers. Run 4896, first builds after moving to `docker-container`: build-web 3m44s (cold baseline 2m23s) build-ml 3m49s (cold baseline 3m20s) build-agent 11m12s (cold baseline 9m26s) The container driver gets a FRESH buildkit instance per job, so it has no local layer store to fall back on — where the old docker driver at least reused whatever the runner's dockerd happened to hold. That is why a registry cache is the only cache this driver can have, and why step 1 on its own is a regression rather than a win. `mode=max` so intermediate stages cache too. The agent's two ~150s pip layers and web's frontend-builder stage are the entire cost, and a min-mode cache would drop exactly those. A `:buildcache` tag is not the withdrawn tag scheme returning. Rule 145 narrowed against names NOTHING reads; this one is read by every build that runs, is one moving ref per image rather than one per build, holds cache blobs rather than a shippable artifact, and is overwritten in place rather than accumulating. Closer to `:dev` than to the `:2026.8.28` tags 318 deleted — and said so in the workflow, so it is not "cleaned up" by a later reader. Expect the next build to be slower again, once: it is still cold AND now pays the cache export. The measurement that matters is the one after that. Scribe #3114. Co-Authored-By: Claude Opus 5 --- .forgejo/workflows/build.yml | 57 ++++++++++++++++++++++++++++++++++++ ci-requirements.md | 11 +++++++ 2 files changed, 68 insertions(+) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 6864b08..eb4299b 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -688,6 +688,25 @@ jobs: # same shape for `platforms:`. provenance: false sbom: false + # The ONLY cache this driver can have. `docker-container` gets a + # FRESH buildkit instance per job, so unlike the default docker + # driver it has no local layer store to fall back on — measured on + # run 4896, the first builds after the driver change: web 3m44s + # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The + # driver change ALONE is a regression; this is the other half of it. + # + # mode=max so intermediate stages cache too. web's frontend-builder + # stage and the agent's two ~150s pip layers are the whole cost, and + # they are exactly what a min-mode cache would drop. + # + # A `:buildcache` tag is NOT the withdrawn tag scheme coming back. + # Rule 145 narrowed against names NOTHING reads; this one is read by + # every build that runs, is one moving ref per image rather than one + # per build, holds cache blobs rather than a shippable artifact, and + # is overwritten in place rather than accumulating. It is closer to + # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.) + cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache + cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache,mode=max # Only the web image carries these: it is the one with a UI and an # HTTP surface to report them on. The ml and agent images have # nothing to tell. @@ -986,6 +1005,25 @@ jobs: # same shape for `platforms:`. provenance: false sbom: false + # The ONLY cache this driver can have. `docker-container` gets a + # FRESH buildkit instance per job, so unlike the default docker + # driver it has no local layer store to fall back on — measured on + # run 4896, the first builds after the driver change: web 3m44s + # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The + # driver change ALONE is a regression; this is the other half of it. + # + # mode=max so intermediate stages cache too. web's frontend-builder + # stage and the agent's two ~150s pip layers are the whole cost, and + # they are exactly what a min-mode cache would drop. + # + # A `:buildcache` tag is NOT the withdrawn tag scheme coming back. + # Rule 145 narrowed against names NOTHING reads; this one is read by + # every build that runs, is one moving ref per image rather than one + # per build, holds cache blobs rather than a shippable artifact, and + # is overwritten in place rather than accumulating. It is closer to + # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.) + cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache + cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the @@ -1276,6 +1314,25 @@ jobs: # same shape for `platforms:`. provenance: false sbom: false + # The ONLY cache this driver can have. `docker-container` gets a + # FRESH buildkit instance per job, so unlike the default docker + # driver it has no local layer store to fall back on — measured on + # run 4896, the first builds after the driver change: web 3m44s + # (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The + # driver change ALONE is a regression; this is the other half of it. + # + # mode=max so intermediate stages cache too. web's frontend-builder + # stage and the agent's two ~150s pip layers are the whole cost, and + # they are exactly what a min-mode cache would drop. + # + # A `:buildcache` tag is NOT the withdrawn tag scheme coming back. + # Rule 145 narrowed against names NOTHING reads; this one is read by + # every build that runs, is one moving ref per image rather than one + # per build, holds cache blobs rather than a shippable artifact, and + # is overwritten in place rather than accumulating. It is closer to + # :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.) + cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache + cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the diff --git a/ci-requirements.md b/ci-requirements.md index 3fd321c..340e277 100644 --- a/ci-requirements.md +++ b/ci-requirements.md @@ -146,6 +146,17 @@ per `docs/process.md`'s "add deps to the image when used by >1 project". rebuild forever with every lane green. Same failure as #3183, different door. These jobs run inside a container against a mounted docker socket, so the buildkit container is a sibling rather than a child. +- **All three images import and export a registry layer cache** + (`:buildcache`, `mode=max`). This is not an optimisation bolted onto + the driver change — it is the other half of it. The `docker-container` + driver gets a fresh buildkit instance per job and therefore has **no local + layer store at all**, where the old `docker` driver at least reused whatever + the runner's dockerd happened to hold. Measured on run 4896, the first builds + after the driver moved: web 3m44s (was 2m23s), ml 3m49s (was 3m20s), agent + 11m12s (was 9m26s) — every one slower. A `:buildcache` tag is read by every + build that runs, is one moving ref per image, holds cache blobs rather than a + shippable artifact, and is overwritten in place, so it is not a return of the + per-version tags milestone 318 withdrew (#3114). - **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which bypasses the reuse check for all three images. It exists because skip-if-exists makes its own build path untestable: `agent/` has not changed