Stop minting identifiers nothing reads: moving channel tags, :c-<sha> on main, and a label-keyed build cache #239

Merged

bvandeusen merged 6 commits from dev into main 2026-08-28 15:14:35 -04:00

Conversation 0 Commits 6 Files Changed 3

+453 -445

bvandeusen commented 2026-08-28 15:14:30 -04:00

Owner

Copy Link

Copy Source

Milestone 318, steps 1–4. Adopts the stance rules 145 and 148 were amended to on 2026-08-28, and partly withdraws milestone 313. Sibling change: Roundtable #3159. Reference note #3127.

What changes about published artifacts

before   main → :main  :latest  :c-<sha>  :2026.8.28        dev → :dev  :r-<rev>-dev
after    main → :latest  :c-<sha>                           dev → :dev

v* tags no longer build anything at all.

Why this is a withdrawal, not a correction

313's steps 1–4 shipped and were verified on both branches, and nothing in them was wrong. The derivation, the per-artifact path sets, the drift tests, skip-if-exists and the merge-preservation property all stay — FC is note #3127's reference implementation for them.

What changed is a premise. The date tag and the planned retention existed to serve consumers who pin builds; FC has none. Verified the way #3159 verified Roundtable's: docker-compose.yml runs :dev across five services, agent/docker-compose.yml and agent/README.md use :latest, and nothing in the repo reads :c-<sha>, :2026.8.28 or :v*. The operator confirmed the production swarm stack tracks :latest.

Rule 148 names this shape of move: "a rule can be right, and then be un-justified by a later decision without ever having been wrong."

The four steps

1 — the gate. Confirm nothing pins, including the operator-side stack. Answered before anything else ran.

2 (d9aa5aa) — the v* build trigger comes out. Not a tidy-up: a release build rebuilt a commit main had already published and re-pushed :c-<sha>, which rule 145 forbids even when the bytes match, since image configs carry timestamps. It was a standing violation waiting for the next release. Three constructs go with it — the 10-minute XPI polling loop (which existed only because the tag build raced the main build and lost), the IS_TAG_PUSH carve-out, and build-web's always() gate. −119/+76 lines, most of the additions being comments about races that can no longer happen.

3 (7e065fe, 454eb3f, b3989d0) — the reuse key moves to an image label. fc.revision is stamped on the image and read back off the moving channel tag, replacing the r-<rev> identity tags. Nothing is minted per build; the -main/-dev split and CHANNELLED disappear, because which tag you inspect is the channel. It also dissolves #3154's open question rather than deferring it, and closes a defect in what shipped: r-<rev> matched no branch of the registry's keep_pattern (#3157), so identity tags were prunable and a pruned one silently cost a rebuild.

4 (cd0b0ff) — the narrowing itself. Note the inversion: :c-<sha> stays. 313 step 6 planned to retire it on "haven't used, not important"; amended rule 145 makes it the rollback unit and, with version tags gone, the only immutable identifier left. #3139 is cancelled as superseded.

No retention job is built — one already runs (#3157), a nightly Gitea package_cleanup_rule with keep_count=10. Its keep_pattern protects any dotted-numeric tag, so the date tags this PR stops minting would have been exempt from retention and grown without bound, silently. That trap is dissolved rather than fixed.

Two things found by shipping it, both recorded

#3183 — the repoint was destroying the label it depends on. imagetools create wraps its source in a manifest index, and the repoint passed the channel tag as both source and target. After any reuse, .Image.Config.Labels stopped resolving and the next push rebuilt. Reuse worked exactly once per artifact, then paid full price forever. Nothing failed; nothing went red; the savings just stopped being real. Caught by comparing three runs: run 4749 read fc.revision=a7e626a67a79 off …-ml:dev and skipped; run 4751 read <none> off the same tag and rebuilt; the only thing to touch it in between was 4749's own repoint. 313's r-<rev> design had been immune by accident — its source was never a target.

#3190 — build-agent reports failure after a successful push. Predates this milestone (run 4658). Registry timestamps show the image landing three seconds before the error. Filed, not fixed.

Verified on dev

label miss → rebuild	run 4745, all three (<none>, the one-time migration)
label hit → skip	run 4749 (web, ml) and run 4756 (web, ml)
:dev is the only dev tag	✅

What this merge verifies, and what it costs

The main path is code-reviewed but unexercised — that is the point of merging. Watch for:

1. main publishes exactly :latest and :c-<sha>. No :main, no date tag.
2. All three artifacts rebuild once: :latest was last written before labels existed, so it carries none. That is the main-side half of the one-time migration, and it is the safe direction.
3. :latest afterwards carries fc.revision, so the next merge skips.

Existing published tags are untouched — :main, :2026.8.28, :c-* and :v* all stay as history. This changes what is minted, not what exists.

Merged with a plain merge commit per rule 153: squash or rebase rewrites %ct and would break every derived version here.

Milestone **318**, steps 1–4. Adopts the stance rules **145** and **148** were amended to on 2026-08-28, and partly **withdraws** milestone 313. Sibling change: Roundtable **#3159**. Reference note **#3127**. ## What changes about published artifacts ``` before main → :main :latest :c-<sha> :2026.8.28 dev → :dev :r-<rev>-dev after main → :latest :c-<sha> dev → :dev ``` `v*` tags no longer build anything at all. ## Why this is a withdrawal, not a correction 313's steps 1–4 shipped and were verified on both branches, and **nothing in them was wrong**. The derivation, the per-artifact path sets, the drift tests, skip-if-exists and the merge-preservation property all stay — FC is note #3127's reference implementation for them. What changed is a *premise*. The date tag and the planned retention existed to serve consumers who pin builds; **FC has none.** Verified the way #3159 verified Roundtable's: `docker-compose.yml` runs `:dev` across five services, `agent/docker-compose.yml` and `agent/README.md` use `:latest`, and nothing in the repo reads `:c-<sha>`, `:2026.8.28` or `:v*`. The operator confirmed the production swarm stack tracks `:latest`. Rule 148 names this shape of move: *"a rule can be right, and then be un-justified by a later decision without ever having been wrong."* ## The four steps **1 — the gate.** Confirm nothing pins, including the operator-side stack. Answered before anything else ran. **2 (`d9aa5aa`) — the `v*` build trigger comes out.** Not a tidy-up: a release build rebuilt a commit `main` had already published and **re-pushed `:c-<sha>`**, which rule 145 forbids even when the bytes match, since image configs carry timestamps. It was a standing violation waiting for the next release. Three constructs go with it — the 10-minute XPI polling loop (which existed only because the tag build raced the main build and lost), the `IS_TAG_PUSH` carve-out, and build-web's `always()` gate. **−119/+76 lines**, most of the additions being comments about races that can no longer happen. **3 (`7e065fe`, `454eb3f`, `b3989d0`) — the reuse key moves to an image label.** `fc.revision` is stamped on the image and read back off the moving channel tag, replacing the `r-<rev>` identity tags. Nothing is minted per build; the `-main`/`-dev` split and `CHANNELLED` disappear, because *which tag you inspect is the channel*. It also dissolves #3154's open question rather than deferring it, and closes a defect in what shipped: `r-<rev>` matched no branch of the registry's `keep_pattern` (#3157), so identity tags were prunable and a pruned one silently cost a rebuild. **4 (`cd0b0ff`) — the narrowing itself.** Note the inversion: **`:c-<sha>` stays.** 313 step 6 planned to retire it on *"haven't used, not important"*; amended rule 145 makes it the rollback unit and, with version tags gone, the only immutable identifier left. #3139 is cancelled as superseded. **No retention job is built** — one already runs (#3157), a nightly Gitea `package_cleanup_rule` with `keep_count=10`. Its `keep_pattern` protects any dotted-numeric tag, so the date tags this PR stops minting **would have been exempt from retention and grown without bound**, silently. That trap is dissolved rather than fixed. ## Two things found by shipping it, both recorded **#3183 — the repoint was destroying the label it depends on.** `imagetools create` wraps its source in a manifest *index*, and the repoint passed the channel tag as both source and target. After any reuse, `.Image.Config.Labels` stopped resolving and the next push rebuilt. **Reuse worked exactly once per artifact, then paid full price forever.** Nothing failed; nothing went red; the savings just stopped being real. Caught by comparing three runs: run 4749 read `fc.revision=a7e626a67a79` off `…-ml:dev` and skipped; run 4751 read `<none>` off the same tag and rebuilt; the only thing to touch it in between was 4749's own repoint. 313's `r-<rev>` design had been immune *by accident* — its source was never a target. **#3190 — `build-agent` reports failure after a successful push.** Predates this milestone (run 4658). Registry timestamps show the image landing three seconds *before* the error. Filed, not fixed. ## Verified on dev | | | |---|---| | label miss → rebuild | run 4745, all three (`<none>`, the one-time migration) | | label hit → skip | run 4749 (web, ml) and run 4756 (web, ml) | | `:dev` is the only dev tag | ✅ | ## What this merge verifies, and what it costs **The main path is code-reviewed but unexercised** — that is the point of merging. Watch for: 1. `main` publishes exactly `:latest` and `:c-<sha>`. No `:main`, no date tag. 2. All three artifacts **rebuild once**: `:latest` was last written before labels existed, so it carries none. That is the main-side half of the one-time migration, and it is the safe direction. 3. `:latest` afterwards carries `fc.revision`, so the *next* merge skips. Existing published tags are untouched — `:main`, `:2026.8.28`, `:c-*` and `:v*` all stay as history. This changes what is minted, not what exists. Merged with a plain merge commit per rule 153: squash or rebase rewrites `%ct` and would break every derived version here.

bvandeusen added 6 commits 2026-08-28 15:14:30 -04:00

ci: a release names a build, it does not make one (milestone 318 step 2) ... d9aa5aa832

Removes the `v*` image-build trigger. A release tag names a commit `main`
already built and published; rebuilding it produced the same source under
the same names and RE-PUSHED `:c-<sha>` — which rule 145 forbids even when
the bytes match, because image configs carry timestamps and "same source"
does not mean "same manifest". The tag build was publishing nothing new and
violating an immutability rule to do it.

Three constructs go with it, all of which existed only to manage that build:

The 10-minute XPI polling loop. A release cut fired the tag build and the
main build together; the tag build skipped sign-extension and raced straight
to the download, losing every time (operator-flagged 2026-05-27 after
v26.05.27.0). Polling was the fix for a build that should not have run. It is
now a single fetch, and a 404 means the two jobs derived different versions
for one commit — which should fail loudly rather than be slept through.

The IS_TAG_PUSH carve-out from milestone 313 step 4, which stopped a tag
build from claiming an identity tag it would have re-pointed at fresh bytes.

build-web's `always() && (success || skipped)` gate, which existed to let a
SKIPPED sign-extension through on a tag push while still blocking a failed
one. sign-extension now always runs, so a plain `needs` gives exactly the
wanted behaviour. Its own branch condition goes too: main and dev are the
only triggers, so a gate naming both matched everything, and a condition
that is always true reads as though some path avoids it.

Releases still happen — rule 148 is untouched and tags are still cut on
explicit request per rule 2. They stop building images and start carrying a
changelog (step 7).

Net 115 lines deleted, 73 added, most of that comments explaining races that
can no longer occur. Nothing in the repo referenced the tag build, so no doc
changes were needed; the broader doc pass is step 9.

ci: shadow the label-based reuse key before trusting it (318 step 3) ... dee93faa37

Additive and inert. Every build now stamps `fc.revision` as an image label,
and a diagnostic step tries to read it back off the channel tag. Nothing
downstream reads either yet — the reuse check still keys on the r-<rev> tag.

The question step 3 has to answer before committing: can the reuse check be
keyed on a LABEL read off the channel tag instead of on a separate tag? If
it can, nothing is minted per build, the -main/-dev identity split
disappears with CHANNELLED, and #3154's base-refresh clobber stops existing
rather than needing a workaround. It also closes a defect found in what
already shipped: r-<rev> tags do not match the registry's keep_pattern
(#3157), so they are prunable, and a pruned identity tag silently costs a
rebuild. A label cannot be pruned — it rides inside a tag that must exist.

Three probes rather than one, because `--format` templating over .Image
varies by buildx version and one round trip per guess is a poor trade.

Two questions, answered at different times, and conflating them would
abandon a working design:

1. Do the template mechanics work here? Answered on the first run — if the
   probes print a labels map, even an empty one, `--format` resolves on this
   buildx and this registry. That is the real gate.
2. Does a value round-trip? Only once an artifact genuinely rebuilds. This
   push touches build.yml, which is in no path set, so all three artifacts
   reuse and stamp nothing. Expect no values this run.

That second point is the migration path rather than a defect: under the real
mechanism "no label" reads as a MISS, so each artifact rebuilds once and is
stamped from then on — one extra build per artifact, ever, in the safe
direction. `imagetools create` cannot backfill it, since it copies a
manifest and config labels are not manifest annotations.

ci: key the reuse check on an image label, not a tag (318 step 3) ... 7e065fed70

The shadow (dee93fa, run 4732) answered the gate: `imagetools inspect
--format` reads `.Image.Config.Labels` against this registry on buildx
v0.36.1. So the reuse check now asks the moving channel tag whether the image
it already points at carries this commit's `fc.revision`, and the r-<rev>
identity tags stop being published.

Three things this removes rather than manages:

A name minted per build that one thing read. Rule 145's narrowing is aimed
exactly there — "a third name for the same thing is upkeep for a model we do
not run."

The -main/-dev qualifier, and the CHANNELLED list behind it. Which tag you
inspect IS the channel, so the distinction has nowhere to live. cmd_identity
goes with it.

A silent expiry nobody wrote down. r-<rev> matches no branch of the
registry's keep_pattern (#3157), so identity tags were prunable past the
newest 10 — a pruned one costs a rebuild, in the safe direction and entirely
invisibly. A label rides inside a tag that has to exist anyway.

It also dissolves #3154 instead of deferring it: a scheduled base refresh
rebuilds :latest with the same revision label, the next unrelated push sees a
match and skips, and the refreshed base survives. Under the tag scheme that
push repointed :latest back to the older base.

The measured detail that shapes the code: a missing label returns an EMPTY
STRING and exits 0. Branching on the exit code would read "no label yet" as
success and skip a build that was needed. So it compares values, and every
uncertain case — absent label, unreachable tag, older image — lands as empty,
never equals a 12-char revision, and falls through to a build.

Reading the specific key matters too. The map carries the base image's labels,
and org.opencontainers.image.version sits right beside ours reading 24.04 on
the agent — a plausible-looking wrong answer.

Expect every artifact to rebuild once on this push: nothing carries a label
yet and it cannot be backfilled, since the reuse path copies a manifest and
config labels are not manifest annotations. One rebuild per artifact, ever,
self-healing after.

test_artifact_identity.py is rewritten around what is now load-bearing. The
CHANNELLED drift test had nothing left to guard; in its place the revision is
asked of git directly, so the file fails if the derivation ever stops being
"the commit this artifact's own shipped files last changed in".

fix(tests): artifact identity tests talk to artifacts.sh, not to a sibling ... 454eb3f973

Run 4746 failed lint and pytest on the same cause: `from test_artifact_paths
import ROOT, declared_paths`. No other test module in this repo imports
another, so that was a new convention introduced for no gain — and the wrong
one, since `tests/` is a package and the bare name does not resolve.

Everything now goes through `artifacts.sh`, which is the interface build.yml
actually calls. The tests exercise the contract rather than a Python
re-implementation of it, and the duplicate `declared_paths` helper is gone
rather than copied.

Two real defects found while fixing it:

The newest commit is now computed by committer TIME, matching what
artifacts.sh means. It was `git log -1`, whose default order is
reverse-chronological only within topological constraints — so on a merged
history it can name a different commit than the newest timestamp does. Both
agree on this repo today (verified across all four artifacts), which is
exactly what makes it a flake waiting for the branch shape that separates
them.

The third test asserted the same invariant as the first in different words.
Removed rather than left as apparent coverage.

ci: publish :latest + :c-<sha> on main, :dev on dev — nothing else (318 step 4) ... cd0b0ff04a

The narrowing itself. Rule 145, narrowed 2026-08-28 once it was verified that
nothing pins: "a third name for the same thing is upkeep for a model we do
not run."

Gone:

  :2026.8.28  the per-artifact date tag from milestone 313 step 3. It shipped,
              it was verified on both branches, and its premise is gone. This
              is a withdrawal, not a correction — nothing about it was wrong.
  :main       a second moving name for whatever :latest already pointed at,
              justified by nothing but symmetry with :dev.

Kept, and note the inversion: :c-<sha> on main. Milestone 313 step 6 planned
to retire it on "haven't used, not important" — amended rule 145 makes it the
rollback unit and, with version tags gone, the only immutable identifier
left. #3139 is cancelled as superseded rather than quietly dropped.

No retention job is built. One already runs (#3157): a nightly Gitea
package_cleanup_rule with keep_count=10. Two consequences worth writing down.
Its keep_pattern protects any dotted-numeric tag, so the date tags this
commit stops minting would have been exempt from retention and accumulated
without bound — silently, in the direction of unbounded growth. That trap is
now dissolved rather than needing the pattern change #3139 planned. And
:c-<sha> IS prunable under that pattern, so the honest statement is that it
is the rollback unit for roughly the last 10 builds, not for every main
commit ever.

cmd_tag goes with its last consumer rather than being left as a dead
subcommand for a later step to notice. The report step logs version and
revision, which is the same information — the date is the first three fields
of the version.

Docs needed no change: every consumer already tracks a moving tag
(docker-compose.yml on :dev, agent/README.md on :latest), which is the
evidence the whole stance rests on.

fix(ci): the repoint was destroying the label it depends on ... b3989d0224

Reuse worked exactly once per artifact, then every later push rebuilt at full
price. Nothing failed and nothing went red — the savings simply evaporated.

`imagetools create` wraps its source manifest in an INDEX. The repoint step
passed the channel tag as both source and target, so after any reuse :dev
stopped being a plain image, `.Image.Config.Labels` no longer resolved
through it, and fc.revision read as absent on the next push.

Observed across three runs rather than reasoned about: run 4749 read
fc.revision=a7e626a67a79 off fabledcurator-ml:dev and skipped the build; run
4751 read <none> off the same tag and rebuilt. The only thing to touch it in
between was 4749's own repoint. The agent hit in 4751 precisely because its
:dev had last been written by a real build, not by a repoint — which is the
control case.

Milestone 313's r-<rev> design was immune without anyone noticing why: the
source (the identity tag) was never one of the targets. Step 3 made the
channel tag both, and inherited a bug the earlier shape had avoided by
accident.

Fix: exclude the source from the target list, so the channel tag is only ever
written by a real build and stays a plain readable image. On dev that leaves
nothing to do, which is correct — the hit already established that :dev
points at the right content. On main it leaves :c-<sha>, which rule 145
requires of every main push whether or not a build ran.

Also added a note the reuse step prints when a channel tag exists but carries
no readable label. That is expected exactly once per artifact during the
migration; if it appears on every push, the tag is being index-wrapped again
and reuse is dead. This class of failure — correct behaviour, quietly worth
less than it reads — is the third one this milestone has turned up, and it is
the one that does not announce itself.

bvandeusen merged commit 6d98dfc0ec into main 2026-08-28 15:14:35 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#239