ci: buildx driver, registry layer cache, and a force_build hatch (milestone 326 steps 1-3) #242

Merged
bvandeusen merged 3 commits from dev into main 2026-08-29 22:48:47 -04:00
2 changed files with 68 additions and 0 deletions
Showing only changes of commit 5e72076298 - Show all commits
+57
View File
@@ -688,6 +688,25 @@ jobs:
# same shape for `platforms:`.
provenance: false
sbom: false
# The ONLY cache this driver can have. `docker-container` gets a
# FRESH buildkit instance per job, so unlike the default docker
# driver it has no local layer store to fall back on — measured on
# run 4896, the first builds after the driver change: web 3m44s
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
# driver change ALONE is a regression; this is the other half of it.
#
# mode=max so intermediate stages cache too. web's frontend-builder
# stage and the agent's two ~150s pip layers are the whole cost, and
# they are exactly what a min-mode cache would drop.
#
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
# Rule 145 narrowed against names NOTHING reads; this one is read by
# every build that runs, is one moving ref per image rather than one
# per build, holds cache blobs rather than a shippable artifact, and
# is overwritten in place rather than accumulating. It is closer to
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache,mode=max
# Only the web image carries these: it is the one with a UI and an
# HTTP surface to report them on. The ml and agent images have
# nothing to tell.
@@ -986,6 +1005,25 @@ jobs:
# same shape for `platforms:`.
provenance: false
sbom: false
# The ONLY cache this driver can have. `docker-container` gets a
# FRESH buildkit instance per job, so unlike the default docker
# driver it has no local layer store to fall back on — measured on
# run 4896, the first builds after the driver change: web 3m44s
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
# driver change ALONE is a regression; this is the other half of it.
#
# mode=max so intermediate stages cache too. web's frontend-builder
# stage and the agent's two ~150s pip layers are the whole cost, and
# they are exactly what a min-mode cache would drop.
#
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
# Rule 145 narrowed against names NOTHING reads; this one is read by
# every build that runs, is one moving ref per image rather than one
# per build, holds cache blobs rather than a shippable artifact, and
# is overwritten in place rather than accumulating. It is closer to
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
@@ -1276,6 +1314,25 @@ jobs:
# same shape for `platforms:`.
provenance: false
sbom: false
# The ONLY cache this driver can have. `docker-container` gets a
# FRESH buildkit instance per job, so unlike the default docker
# driver it has no local layer store to fall back on — measured on
# run 4896, the first builds after the driver change: web 3m44s
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
# driver change ALONE is a regression; this is the other half of it.
#
# mode=max so intermediate stages cache too. web's frontend-builder
# stage and the agent's two ~150s pip layers are the whole cost, and
# they are exactly what a min-mode cache would drop.
#
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
# Rule 145 narrowed against names NOTHING reads; this one is read by
# every build that runs, is one moving ref per image rather than one
# per build, holds cache blobs rather than a shippable artifact, and
# is overwritten in place rather than accumulating. It is closer to
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
+11
View File
@@ -146,6 +146,17 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
rebuild forever with every lane green. Same failure as #3183, different door.
These jobs run inside a container against a mounted docker socket, so the
buildkit container is a sibling rather than a child.
- **All three images import and export a registry layer cache**
(`<image>:buildcache`, `mode=max`). This is not an optimisation bolted onto
the driver change — it is the other half of it. The `docker-container`
driver gets a fresh buildkit instance per job and therefore has **no local
layer store at all**, where the old `docker` driver at least reused whatever
the runner's dockerd happened to hold. Measured on run 4896, the first builds
after the driver moved: web 3m44s (was 2m23s), ml 3m49s (was 3m20s), agent
11m12s (was 9m26s) — every one slower. A `:buildcache` tag is read by every
build that runs, is one moving ref per image, holds cache blobs rather than a
shippable artifact, and is overwritten in place, so it is not a return of the
per-version tags milestone 318 withdrew (#3114).
- **`build.yml` accepts a `workflow_dispatch` with `force_build`**, which
bypasses the reuse check for all three images. It exists because
skip-if-exists makes its own build path untestable: `agent/` has not changed