ci: the refresh publishes only what the gate passed (#3265 milestone step 4)
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 5s
Build images / build-ml (push) Successful in 8s
Build images / build-agent (push) Successful in 9s
Build images / build-web (push) Successful in 6s
Build images / smoke-web (push) Skipped
Build images / promote (push) Skipped
extension / lint (push) Successful in 19s
CI / frontend-build (push) Successful in 21s
CI / backend-lint-and-test (push) Successful in 34s
CI / integration (push) Successful in 1m53s

The gate reported a verdict nothing consulted. Now it decides.

The promote moved out of the three build jobs into its own `promote` job,
because the verdict cannot exist until build-web has finished and the promote
used to run inside it. `needs: [build-web, build-ml, build-agent, smoke-web]`
is the whole mechanism: a failed smoke skips the promote, so a refresh that
broke something leaves :latest naming the build that works. "The refresh
failed" and "production is broken" must not be the same event.

A SKIPPED smoke also skips it, and that is the case that matters most. On run
5290 the gate silently skipped itself — job-level `if:` cannot read the env
context — and a design where only a FAILED gate blocks would have published
unverified images while reporting success. Not running is not the same as
passing, and today produced two separate bugs of exactly that shape (#3414,
and the smoke-web skip).

All three images now promote together or not at all. They are one stack:
build.yml already refuses to publish a :dev web image beside a stale :dev ml
because the mismatch only surfaces as a runtime failure, and a refresh that
published ml while withholding web would be that same trap reached through the
gate. Stated plainly in the job comment: the gate covers web only, so ml and
agent are held to web's verdict rather than their own. That is the
conservative direction, not equivalent evidence, and should not be read as if
it were.

Three near-identical promote steps collapsed into one loop. A partial failure
now says which images moved and that the state is inconsistent, rather than
leaving that to be inferred — the promote is idempotent and the candidates are
still published, so the instruction is simply to re-run.

Also removed the now-dead `promote` output from the ml and agent reuse steps.
Only build-web's is read (as outputs.candidate); two more copies nothing
consults is the kind of thing that reads as load-bearing a year later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA
This commit is contained in:
2026-09-02 16:21:27 -04:00
co-authored by Claude Opus 5
parent 86abaf0b94
commit f630e50e75
+138 -229
View File
@@ -734,10 +734,16 @@ jobs:
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
# Decided HERE, beside `hit`, for the reason the force/schedule
# branch below gives: one step decides what this job does. A
# condition derived independently could disagree with the tag the
# build actually wrote.
#
# build-web additionally exposes this as `outputs.candidate`, which is
# what gates the `promote` job — a job's `if:` cannot read `env`, and
# one flag is enough because all three derive it from the same
# IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
# reads is the kind of thing that later reads as load-bearing.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
@@ -959,77 +965,6 @@ jobs:
FC_CHANNEL=${{ steps.tag.outputs.channel }}
FC_VERSION=${{ steps.reuse.outputs.version }}
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
@@ -1281,6 +1216,114 @@ jobs:
echo
echo "smoke: all checks passed against $CANDIDATE"
# Move the channel tags — the whole point of the gate.
#
# Lives in its own job because the verdict it depends on cannot exist until
# after build-web has finished, and the promote used to run INSIDE build-web.
#
# `needs` on smoke-web is the gate. A failed smoke skips this job, so a
# refresh that broke something leaves :latest naming the build that works —
# "the refresh failed" and "production is broken" must not be the same event.
# A SKIPPED smoke also skips this job, which is the behaviour that matters
# most: on run 5290 the gate silently skipped itself, and a design where only
# a FAILED gate blocks would have published unverified images while reporting
# success. Not running is not the same as passing.
#
# All three images promote TOGETHER, or none do. They are one stack: build.yml
# already refuses to publish a :dev web image beside a stale :dev ml, because
# the mismatch only shows up as a runtime failure. A refresh that published ml
# and withheld web would be that same trap, arrived at through the gate.
#
# The gate covers the web image only (milestone 362 step 3 scoped it there),
# so ml and agent are being held to web's verdict rather than their own. That
# is deliberate and it is the conservative direction — they ship together, so
# the weakest evidence should govern all three — but it is not the same as
# having smoked them, and it should not be read as if it were.
promote:
needs: [build-web, build-ml, build-agent, smoke-web]
# Only a refresh publishes through a candidate; a push writes its channel
# tag directly from the build. Reads the same reuse-step decision the build
# took, via a job output — a job's `if:` cannot see the `env` context.
if: needs.build-web.outputs.candidate == 'true'
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- name: Point the channel tags at the smoked candidates
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
# `latest` is not a guess: a refresh always builds `main` (BUILD_REF),
# and the "must have checked out main" guard in every build job fails
# the run if that did not hold. So the channel is main's.
TAG=latest
FAILED=""
for NAME in fabledcurator fabledcurator-ml fabledcurator-agent; do
REPO="bvandeusen/$NAME"
echo "promote: $REPO"
# Registry auth is its own token exchange — `docker login`
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the IMAGE manifest media types only. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing this
# whole approach exists to avoid creating.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate $SRC ($CT)"
# NOT `imagetools create`. That wraps its source in an INDEX, and
# `.Image.Config.Labels` does not resolve through one — the
# fc.revision the reuse check reads off the channel tag would come
# back empty, every later push would miss and rebuild, and nothing
# would go red (#3183, run 4751). A manifest PUT is what "make this
# tag name that image" means at the registry: same bytes, same media
# type, same digest, no layer transfer.
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC" ]; then
echo "promote: FAILED — $NAME:$TAG is $NOW, expected $SRC" >&2
FAILED="$FAILED $NAME"
continue
fi
echo "promote: $NAME:$TAG now names $NOW"
done
if [ -n "$FAILED" ]; then
echo "" >&2
echo "promote: FAILED for:$FAILED" >&2
echo "promote: the channel tags are now INCONSISTENT — some images" >&2
echo "promote: moved and some did not. Re-run this refresh; the" >&2
echo "promote: candidates are still published and the promote is" >&2
echo "promote: idempotent." >&2
exit 1
fi
echo "promote: all three channel tags moved"
build-ml:
runs-on: python-ci
container:
@@ -1486,16 +1529,20 @@ jobs:
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
# Decided HERE, beside `hit`, for the reason the force/schedule
# branch below gives: one step decides what this job does. A
# condition derived independently could disagree with the tag the
# build actually wrote.
#
# build-web additionally exposes this as `outputs.candidate`, which is
# what gates the `promote` job — a job's `if:` cannot read `env`, and
# one flag is enough because all three derive it from the same
# IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
# reads is the kind of thing that later reads as load-bearing.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
@@ -1629,77 +1676,6 @@ jobs:
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
@@ -1981,16 +1957,20 @@ jobs:
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
# Decided HERE, beside `hit`, for the reason the force/schedule
# branch below gives: one step decides what this job does. A
# condition derived independently could disagree with the tag the
# build actually wrote.
#
# build-web additionally exposes this as `outputs.candidate`, which is
# what gates the `promote` job — a job's `if:` cannot read `env`, and
# one flag is enough because all three derive it from the same
# IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
# reads is the kind of thing that later reads as load-bearing.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
@@ -2124,77 +2104,6 @@ jobs:
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to