ci: buildx driver, registry layer cache, and a force_build hatch (milestone 326 steps 1-3) #242

Merged

bvandeusen merged 3 commits from dev into main 2026-08-29 22:48:47 -04:00

Conversation 0 Commits 3 Files Changed 2

+249 -3

bvandeusen commented 2026-08-29 22:48:05 -04:00

Owner

Copy Link

Copy Source

Milestone 326 steps 1–3. Three commits, all in .forgejo/workflows/build.yml plus ci-requirements.md.

6b3ec98 — step 1: build on a real buildx driver, attestations off

Every build ran on the default docker driver, which cannot export a registry cache and resolves image metadata against a local store. docker/setup-buildx-action@v3 after each login switches all three builds to the docker-container driver.

provenance: false / sbom: false is load-bearing, not hygiene: an attestation wraps the image in a manifest INDEX, and a config label does not resolve through an index. That would silently kill the milestone-318 fc.revision reuse check with every lane green — the same hazard imagetools create hit in #3183.

5e72076 — step 2: registry-backed layer cache

cache-from / cache-to on <image>:buildcache, mode=max, per image. This is the other half of step 1, not an enhancement on top of it: docker-container gets a fresh buildkit per job with no local layer store, so step 1 alone is a regression.

Measured, all forced builds (nothing skipped):

job	old docker driver	container, no cache	cold + export	warm cache
build-web	2m23s	3m44s	2m28s	13s
build-ml	3m20s	3m49s	2m52s	13s
build-agent	9m26s	11m12s	9m29s	13s

The 13s runs were verified to be real builds rather than reuse skips: force_build set — building regardless, then importing cache manifest, then every layer CACHED.

e21c9fd — step 3: a force_build escape hatch

workflow_dispatch input wired into all three reuse steps. The path-based skip-if-exists check has no way to be overridden, which made steps 1 and 2 unverifiable — build.yml is in no artifact's path set, so a push that touches only it skips every build and writes no tag.

Closes #3190 (No such image — did not reproduce across repeated forced builds on the container driver).
Opened along the way: #3256, an act_runner shared-action-cache race the force hatch exposed.

Merging with a plain merge commit per rule 153 — it is TREESAME on every artifact path set, so the derived CalVer versions are preserved.

Milestone 326 steps 1–3. Three commits, all in `.forgejo/workflows/build.yml` plus `ci-requirements.md`. **`6b3ec98` — step 1: build on a real buildx driver, attestations off** Every build ran on the default `docker` driver, which cannot export a registry cache and resolves image metadata against a local store. `docker/setup-buildx-action@v3` after each login switches all three builds to the `docker-container` driver. `provenance: false` / `sbom: false` is load-bearing, not hygiene: an attestation wraps the image in a manifest INDEX, and a config label does not resolve through an index. That would silently kill the milestone-318 `fc.revision` reuse check with every lane green — the same hazard `imagetools create` hit in #3183. **`5e72076` — step 2: registry-backed layer cache** `cache-from` / `cache-to` on `<image>:buildcache`, `mode=max`, per image. This is the other half of step 1, not an enhancement on top of it: `docker-container` gets a fresh buildkit per job with no local layer store, so step 1 alone is a regression. Measured, all forced builds (nothing skipped): | job | old `docker` driver | container, no cache | cold + export | warm cache | |---|---|---|---|---| | build-web | 2m23s | 3m44s | 2m28s | **13s** | | build-ml | 3m20s | 3m49s | 2m52s | **13s** | | build-agent | 9m26s | 11m12s | 9m29s | **13s** | The 13s runs were verified to be real builds rather than reuse skips: `force_build set — building regardless`, then `importing cache manifest`, then every layer `CACHED`. **`e21c9fd` — step 3: a `force_build` escape hatch** `workflow_dispatch` input wired into all three reuse steps. The path-based skip-if-exists check has no way to be overridden, which made steps 1 and 2 unverifiable — `build.yml` is in no artifact's path set, so a push that touches only it skips every build and writes no tag. **Closes** #3190 (`No such image` — did not reproduce across repeated forced builds on the container driver). **Opened along the way:** #3256, an `act_runner` shared-action-cache race the force hatch exposed. Merging with a plain merge commit per rule 153 — it is TREESAME on every artifact path set, so the derived CalVer versions are preserved.

bvandeusen added 3 commits 2026-08-29 22:48:05 -04:00

ci: build on a real buildx driver, attestations off (milestone 326 step 1) ... 6b3ec98fa8

Adds `docker/setup-buildx-action@v3` to build-web, build-ml and build-agent,
and sets `provenance: false` / `sbom: false` on all three build-push steps.

Two open issues share one root, which is why this is one change:

* #3114 — the agent rebuilds a ~6.3 GB CUDA + torch image whenever the
  runner's local cache is cold, 9m26s against 7s warm. The default `docker`
  driver cannot export a registry cache at all, so the fix is unavailable
  until the driver moves. The cache itself is step 2, deliberately not here.
* #3190 — build-agent goes red AFTER a successful push, `No such image` from
  the local daemon. The leading candidate is the docker driver resolving
  image metadata against a local store a registry-direct push never filled.

The attestation flags are the load-bearing part. On the default driver they
were no-ops; on the container driver, build-push-action@v5 defaults provenance
to TRUE when pushing. Provenance attaches an attestation manifest, that makes
the pushed tag a manifest INDEX, and `.Image.Config.Labels` does not resolve
through an index — so the `fc.revision` label the reuse check reads off the
channel tag would come back `<none>` on every push. Every image would rebuild
forever, every lane would stay green, and the only symptom would be the bill.
Same failure as #3183, through a different door; note #3127 §4 records the
same shape for `platforms:`.

Unverified until CI says otherwise: these jobs run INSIDE a container against
a mounted docker socket, so the buildkit container is a sibling of the job
container rather than a child. That works over a socket mount and has never
been tried on this runner.

The gate is the SECOND dev push, not this one. The images currently published
were built by the old driver, so one `reuse: NOTE ... no readable fc.revision`
is expected now; what must not appear is a second one.

Scribe #3249.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ci: a force_build escape hatch for the path skip-if-exists hides (326 step 3) ... e21c9fdd34

`workflow_dispatch` with a `force_build` boolean, honoured inside each of the
three reuse steps.

It exists because skip-if-exists made its own build path untestable. `agent/`
has not changed since 2026-07-17, so the agent build has correctly declined to
run on every push since — which also means #3190, whose whole symptom lives on
that path, cannot be reproduced on demand. Editing build.yml does not force a
build either, and that is deliberate: the workflow is not shipped bytes, so it
is in no artifact's path set, and putting it in one would re-version every
artifact for a comment change.

That is also why this lands before step 2 rather than after. Step 1 moved the
builds onto a container driver and turned attestations off; the claim that
`fc.revision` still reads back cannot be checked until something actually
builds under that driver. Run 4887 confirmed only the cheaper half — `Set up
buildx` succeeded on all three jobs, so the buildkit sibling container does
start against the mounted socket.

Details worth keeping:

* FORCE is checked in the reuse step, not in the build step's `if:`. The
  repoint step keys off `hit` too, and a force that bypassed only the build
  would leave the two disagreeing about what had happened.
* `github.event.inputs`, not the `inputs` context — release.yml already uses
  that form and it is the one this runner is known to evaluate. Read through
  env rather than interpolated into the run block, same as release.yml's TAG.
* One input, not one per artifact. Three booleans is an interface nobody
  remembers.

Scribe #3252, #3249.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ci: registry-backed layer cache for all three images (milestone 326 step 2) ... 5e72076298

`cache-from`/`cache-to` on `<image>:buildcache`, `mode=max`, on all three
build steps. Closes the half of the driver change that step 1 left open.

Step 1 measured worse, not better, and that was expected but is worth stating
with numbers. Run 4896, first builds after moving to `docker-container`:

    build-web    3m44s   (cold baseline 2m23s)
    build-ml     3m49s   (cold baseline 3m20s)
    build-agent  11m12s  (cold baseline 9m26s)

The container driver gets a FRESH buildkit instance per job, so it has no
local layer store to fall back on — where the old docker driver at least
reused whatever the runner's dockerd happened to hold. That is why a registry
cache is the only cache this driver can have, and why step 1 on its own is a
regression rather than a win.

`mode=max` so intermediate stages cache too. The agent's two ~150s pip layers
and web's frontend-builder stage are the entire cost, and a min-mode cache
would drop exactly those.

A `:buildcache` tag is not the withdrawn tag scheme returning. Rule 145
narrowed against names NOTHING reads; this one is read by every build that
runs, is one moving ref per image rather than one per build, holds cache blobs
rather than a shippable artifact, and is overwritten in place rather than
accumulating. Closer to `:dev` than to the `:2026.8.28` tags 318 deleted — and
said so in the workflow, so it is not "cleaned up" by a later reader.

Expect the next build to be slower again, once: it is still cold AND now pays
the cache export. The measurement that matters is the one after that.

Scribe #3114.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

bvandeusen merged commit 499720d87e into main 2026-08-29 22:48:47 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#242