ci: registry-backed layer cache for all three images (milestone 326 step 2)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 5s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 9s
Build images / build-web (push) Successful in 6s
CI / frontend-build (push) Successful in 17s
extension / lint (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 33s
CI / integration (push) Successful in 3m44s
extension / lint (pull_request) Successful in 29s

`cache-from`/`cache-to` on `<image>:buildcache`, `mode=max`, on all three
build steps. Closes the half of the driver change that step 1 left open.

Step 1 measured worse, not better, and that was expected but is worth stating
with numbers. Run 4896, first builds after moving to `docker-container`:

    build-web    3m44s   (cold baseline 2m23s)
    build-ml     3m49s   (cold baseline 3m20s)
    build-agent  11m12s  (cold baseline 9m26s)

The container driver gets a FRESH buildkit instance per job, so it has no
local layer store to fall back on — where the old docker driver at least
reused whatever the runner's dockerd happened to hold. That is why a registry
cache is the only cache this driver can have, and why step 1 on its own is a
regression rather than a win.

`mode=max` so intermediate stages cache too. The agent's two ~150s pip layers
and web's frontend-builder stage are the entire cost, and a min-mode cache
would drop exactly those.

A `:buildcache` tag is not the withdrawn tag scheme returning. Rule 145
narrowed against names NOTHING reads; this one is read by every build that
runs, is one moving ref per image rather than one per build, holds cache blobs
rather than a shippable artifact, and is overwritten in place rather than
accumulating. Closer to `:dev` than to the `:2026.8.28` tags 318 deleted — and
said so in the workflow, so it is not "cleaned up" by a later reader.

Expect the next build to be slower again, once: it is still cold AND now pays
the cache export. The measurement that matters is the one after that.

Scribe #3114.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-29 18:56:13 -04:00
co-authored by Claude Opus 5
parent e21c9fdd34
commit 5e72076298
2 changed files with 68 additions and 0 deletions
+57
View File
@@ -688,6 +688,25 @@ jobs:
# same shape for `platforms:`. # same shape for `platforms:`.
provenance: false provenance: false
sbom: false sbom: false
# The ONLY cache this driver can have. `docker-container` gets a
# FRESH buildkit instance per job, so unlike the default docker
# driver it has no local layer store to fall back on — measured on
# run 4896, the first builds after the driver change: web 3m44s
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
# driver change ALONE is a regression; this is the other half of it.
#
# mode=max so intermediate stages cache too. web's frontend-builder
# stage and the agent's two ~150s pip layers are the whole cost, and
# they are exactly what a min-mode cache would drop.
#
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
# Rule 145 narrowed against names NOTHING reads; this one is read by
# every build that runs, is one moving ref per image rather than one
# per build, holds cache blobs rather than a shippable artifact, and
# is overwritten in place rather than accumulating. It is closer to
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache,mode=max
# Only the web image carries these: it is the one with a UI and an # Only the web image carries these: it is the one with a UI and an
# HTTP surface to report them on. The ml and agent images have # HTTP surface to report them on. The ml and agent images have
# nothing to tell. # nothing to tell.
@@ -986,6 +1005,25 @@ jobs:
# same shape for `platforms:`. # same shape for `platforms:`.
provenance: false provenance: false
sbom: false sbom: false
# The ONLY cache this driver can have. `docker-container` gets a
# FRESH buildkit instance per job, so unlike the default docker
# driver it has no local layer store to fall back on — measured on
# run 4896, the first builds after the driver change: web 3m44s
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
# driver change ALONE is a regression; this is the other half of it.
#
# mode=max so intermediate stages cache too. web's frontend-builder
# stage and the agent's two ~150s pip layers are the whole cost, and
# they are exactly what a min-mode cache would drop.
#
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
# Rule 145 narrowed against names NOTHING reads; this one is read by
# every build that runs, is one moving ref per image rather than one
# per build, holds cache blobs rather than a shippable artifact, and
# is overwritten in place rather than accumulating. It is closer to
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
# Every tag but the channel's own is written HERE, registry-side, # Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the # whether or not a build ran. Each -t becomes another reference to the
@@ -1276,6 +1314,25 @@ jobs:
# same shape for `platforms:`. # same shape for `platforms:`.
provenance: false provenance: false
sbom: false sbom: false
# The ONLY cache this driver can have. `docker-container` gets a
# FRESH buildkit instance per job, so unlike the default docker
# driver it has no local layer store to fall back on — measured on
# run 4896, the first builds after the driver change: web 3m44s
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
# driver change ALONE is a regression; this is the other half of it.
#
# mode=max so intermediate stages cache too. web's frontend-builder
# stage and the agent's two ~150s pip layers are the whole cost, and
# they are exactly what a min-mode cache would drop.
#
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
# Rule 145 narrowed against names NOTHING reads; this one is read by
# every build that runs, is one moving ref per image rather than one
# per build, holds cache blobs rather than a shippable artifact, and
# is overwritten in place rather than accumulating. It is closer to
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
# Every tag but the channel's own is written HERE, registry-side, # Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the # whether or not a build ran. Each -t becomes another reference to the
+11
View File
@@ -146,6 +146,17 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
rebuild forever with every lane green. Same failure as #3183, different door. rebuild forever with every lane green. Same failure as #3183, different door.
These jobs run inside a container against a mounted docker socket, so the These jobs run inside a container against a mounted docker socket, so the
buildkit container is a sibling rather than a child. buildkit container is a sibling rather than a child.
- **All three images import and export a registry layer cache**
(`<image>:buildcache`, `mode=max`). This is not an optimisation bolted onto
the driver change — it is the other half of it. The `docker-container`
driver gets a fresh buildkit instance per job and therefore has **no local
layer store at all**, where the old `docker` driver at least reused whatever
the runner's dockerd happened to hold. Measured on run 4896, the first builds
after the driver moved: web 3m44s (was 2m23s), ml 3m49s (was 3m20s), agent
11m12s (was 9m26s) — every one slower. A `:buildcache` tag is read by every
build that runs, is one moving ref per image, holds cache blobs rather than a
shippable artifact, and is overwritten in place, so it is not a return of the
per-version tags milestone 318 withdrew (#3114).
- **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which - **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which
bypasses the reuse check for all three images. It exists because bypasses the reuse check for all three images. It exists because
skip-if-exists makes its own build path untestable: `agent/` has not changed skip-if-exists makes its own build path untestable: `agent/` has not changed