The weekly refresh now publishes only what a gate has proven — and a new install can start #249

Merged
bvandeusen merged 6 commits from dev into main 2026-09-02 16:31:56 -04:00
Showing only changes of commit f630e50e75 - Show all commits
+138 -229
View File
@@ -734,10 +734,16 @@ jobs:
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
# Decided HERE, beside `hit`, for the reason the force/schedule
# branch below gives: one step decides what this job does. A
# condition derived independently could disagree with the tag the
# build actually wrote.
#
# build-web additionally exposes this as `outputs.candidate`, which is
# what gates the `promote` job — a job's `if:` cannot read `env`, and
# one flag is enough because all three derive it from the same
# IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
# reads is the kind of thing that later reads as load-bearing.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
@@ -959,77 +965,6 @@ jobs:
FC_CHANNEL=${{ steps.tag.outputs.channel }}
FC_VERSION=${{ steps.reuse.outputs.version }}
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
@@ -1281,6 +1216,114 @@ jobs:
echo
echo "smoke: all checks passed against $CANDIDATE"
# Move the channel tags — the whole point of the gate.
#
# Lives in its own job because the verdict it depends on cannot exist until
# after build-web has finished, and the promote used to run INSIDE build-web.
#
# `needs` on smoke-web is the gate. A failed smoke skips this job, so a
# refresh that broke something leaves :latest naming the build that works —
# "the refresh failed" and "production is broken" must not be the same event.
# A SKIPPED smoke also skips this job, which is the behaviour that matters
# most: on run 5290 the gate silently skipped itself, and a design where only
# a FAILED gate blocks would have published unverified images while reporting
# success. Not running is not the same as passing.
#
# All three images promote TOGETHER, or none do. They are one stack: build.yml
# already refuses to publish a :dev web image beside a stale :dev ml, because
# the mismatch only shows up as a runtime failure. A refresh that published ml
# and withheld web would be that same trap, arrived at through the gate.
#
# The gate covers the web image only (milestone 362 step 3 scoped it there),
# so ml and agent are being held to web's verdict rather than their own. That
# is deliberate and it is the conservative direction — they ship together, so
# the weakest evidence should govern all three — but it is not the same as
# having smoked them, and it should not be read as if it were.
promote:
needs: [build-web, build-ml, build-agent, smoke-web]
# Only a refresh publishes through a candidate; a push writes its channel
# tag directly from the build. Reads the same reuse-step decision the build
# took, via a job output — a job's `if:` cannot see the `env` context.
if: needs.build-web.outputs.candidate == 'true'
runs-on: python-ci
container:
image: git.fabledsword.com/bvandeusen/ci-python:3.14
steps:
- name: Point the channel tags at the smoked candidates
env:
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
# `latest` is not a guess: a refresh always builds `main` (BUILD_REF),
# and the "must have checked out main" guard in every build job fails
# the run if that did not hold. So the channel is main's.
TAG=latest
FAILED=""
for NAME in fabledcurator fabledcurator-ml fabledcurator-agent; do
REPO="bvandeusen/$NAME"
echo "promote: $REPO"
# Registry auth is its own token exchange — `docker login`
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the IMAGE manifest media types only. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing this
# whole approach exists to avoid creating.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate $SRC ($CT)"
# NOT `imagetools create`. That wraps its source in an INDEX, and
# `.Image.Config.Labels` does not resolve through one — the
# fc.revision the reuse check reads off the channel tag would come
# back empty, every later push would miss and rebuild, and nothing
# would go red (#3183, run 4751). A manifest PUT is what "make this
# tag name that image" means at the registry: same bytes, same media
# type, same digest, no layer transfer.
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC" ]; then
echo "promote: FAILED — $NAME:$TAG is $NOW, expected $SRC" >&2
FAILED="$FAILED $NAME"
continue
fi
echo "promote: $NAME:$TAG now names $NOW"
done
if [ -n "$FAILED" ]; then
echo "" >&2
echo "promote: FAILED for:$FAILED" >&2
echo "promote: the channel tags are now INCONSISTENT — some images" >&2
echo "promote: moved and some did not. Re-run this refresh; the" >&2
echo "promote: candidates are still published and the promote is" >&2
echo "promote: idempotent." >&2
exit 1
fi
echo "promote: all three channel tags moved"
build-ml:
runs-on: python-ci
container:
@@ -1486,16 +1529,20 @@ jobs:
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
# Decided HERE, beside `hit`, for the reason the force/schedule
# branch below gives: one step decides what this job does. A
# condition derived independently could disagree with the tag the
# build actually wrote.
#
# build-web additionally exposes this as `outputs.candidate`, which is
# what gates the `promote` job — a job's `if:` cannot read `env`, and
# one flag is enough because all three derive it from the same
# IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
# reads is the kind of thing that later reads as load-bearing.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
@@ -1629,77 +1676,6 @@ jobs:
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
@@ -1981,16 +1957,20 @@ jobs:
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
# Decided HERE, beside `hit`, for the reason the force/schedule
# branch below gives: one step decides what this job does. A
# condition derived independently could disagree with the tag the
# build actually wrote.
#
# build-web additionally exposes this as `outputs.candidate`, which is
# what gates the `promote` job — a job's `if:` cannot read `env`, and
# one flag is enough because all three derive it from the same
# IS_REFRESH. ml and agent do not re-emit it; a second copy nothing
# reads is the kind of thing that later reads as load-bearing.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
@@ -2124,77 +2104,6 @@ jobs:
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to