fix(ci): the repoint was destroying the label it depends on
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
Build images / build-web (push) Successful in 3s
extension / lint (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 42s
Build images / build-agent (push) Failing after 2m33s
CI / integration (push) Successful in 3m44s
extension / lint (pull_request) Successful in 25s

Reuse worked exactly once per artifact, then every later push rebuilt at full
price. Nothing failed and nothing went red — the savings simply evaporated.

`imagetools create` wraps its source manifest in an INDEX. The repoint step
passed the channel tag as both source and target, so after any reuse :dev
stopped being a plain image, `.Image.Config.Labels` no longer resolved
through it, and fc.revision read as absent on the next push.

Observed across three runs rather than reasoned about: run 4749 read
fc.revision=a7e626a67a79 off fabledcurator-ml:dev and skipped the build; run
4751 read <none> off the same tag and rebuilt. The only thing to touch it in
between was 4749's own repoint. The agent hit in 4751 precisely because its
:dev had last been written by a real build, not by a repoint — which is the
control case.

Milestone 313's r-<rev> design was immune without anyone noticing why: the
source (the identity tag) was never one of the targets. Step 3 made the
channel tag both, and inherited a bug the earlier shape had avoided by
accident.

Fix: exclude the source from the target list, so the channel tag is only ever
written by a real build and stays a plain readable image. On dev that leaves
nothing to do, which is correct — the hit already established that :dev
points at the right content. On main it leaves :c-<sha>, which rule 145
requires of every main push whether or not a build ran.

Also added a note the reuse step prints when a channel tag exists but carries
no readable label. That is expected exactly once per artifact during the
migration; if it appears on every push, the tag is being index-wrapped again
and reuse is dead. This class of failure — correct behaviour, quietly worth
less than it reads — is the third one this milestone has turned up, and it is
the one that does not announce itself.
This commit is contained in:
2026-08-28 14:55:49 -04:00
parent cd0b0ff04a
commit b3989d0224
+114 -21
View File
@@ -504,6 +504,15 @@ jobs:
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
2>/dev/null || echo "")
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
# The tag resolves but carries no readable label. Expected exactly
# once per artifact, during the migration onto labels. If it recurs
# every push, something is rewriting the channel tag as a manifest
# index — see the repoint step's note.
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
echo "reuse: NOTE Fine once, while migrating. Every push means the"
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
fi
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
@@ -625,20 +634,42 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# The source tag is EXCLUDED from the targets, and that is load-
# bearing rather than an optimisation.
#
# `imagetools create` wraps the source manifest in an INDEX. Point it
# at the channel tag with that same tag as a target and the tag stops
# being a plain image — after which `.Image.Config.Labels` no longer
# resolves through it and the fc.revision label reads as absent. The
# next push then misses and rebuilds, so reuse worked exactly once
# and every subsequent push paid full price. Observed on run 4751:
# ml:dev reported fc.revision=<none> one push after run 4749 had read
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
#
# Excluding the source means the channel tag is only ever written by
# a real build, so it stays a plain image and stays readable. On dev
# that leaves nothing to do — :dev already points at the right
# content, which is what the hit established. On main it leaves
# :c-<sha>, which rule 145 requires of every main push whether or not
# a build ran.
#
# steps.tag emits ONE comma-separated list, because that is the shape
# docker/build-push-action takes; imagetools wants a -t per ref.
ARGS=""
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
for t in $TAGS; do
[ "$t" = "$SOURCE" ] && continue
ARGS="$ARGS -t $t"
done
unset IFS
# Source is the channel tag itself — the image we just confirmed
# carries this revision. On dev the only target IS that tag, so this
# is a no-op that keeps the code path uniform. On main it is what
# gives the new commit its :c-<sha>, which rule 145 requires of every
# main push whether or not a build ran.
if [ -z "$ARGS" ]; then
echo "repoint: $SOURCE already carries this revision and is the"
echo "repoint: only tag for this channel — nothing to write."
exit 0
fi
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$SOURCE"
echo "repointed to $SOURCE: $TAGS"
echo "repointed from $SOURCE:$ARGS"
build-ml:
runs-on: python-ci
@@ -775,6 +806,15 @@ jobs:
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
2>/dev/null || echo "")
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
# The tag resolves but carries no readable label. Expected exactly
# once per artifact, during the migration onto labels. If it recurs
# every push, something is rewriting the channel tag as a manifest
# index — see the repoint step's note.
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
echo "reuse: NOTE Fine once, while migrating. Every push means the"
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
fi
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
@@ -815,20 +855,42 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# The source tag is EXCLUDED from the targets, and that is load-
# bearing rather than an optimisation.
#
# `imagetools create` wraps the source manifest in an INDEX. Point it
# at the channel tag with that same tag as a target and the tag stops
# being a plain image — after which `.Image.Config.Labels` no longer
# resolves through it and the fc.revision label reads as absent. The
# next push then misses and rebuilds, so reuse worked exactly once
# and every subsequent push paid full price. Observed on run 4751:
# ml:dev reported fc.revision=<none> one push after run 4749 had read
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
#
# Excluding the source means the channel tag is only ever written by
# a real build, so it stays a plain image and stays readable. On dev
# that leaves nothing to do — :dev already points at the right
# content, which is what the hit established. On main it leaves
# :c-<sha>, which rule 145 requires of every main push whether or not
# a build ran.
#
# steps.tag emits ONE comma-separated list, because that is the shape
# docker/build-push-action takes; imagetools wants a -t per ref.
ARGS=""
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
for t in $TAGS; do
[ "$t" = "$SOURCE" ] && continue
ARGS="$ARGS -t $t"
done
unset IFS
# Source is the channel tag itself — the image we just confirmed
# carries this revision. On dev the only target IS that tag, so this
# is a no-op that keeps the code path uniform. On main it is what
# gives the new commit its :c-<sha>, which rule 145 requires of every
# main push whether or not a build ran.
if [ -z "$ARGS" ]; then
echo "repoint: $SOURCE already carries this revision and is the"
echo "repoint: only tag for this channel — nothing to write."
exit 0
fi
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$SOURCE"
echo "repointed to $SOURCE: $TAGS"
echo "repointed from $SOURCE:$ARGS"
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
# the GPU machine instead of building locally. Independent of web/ml (its own
@@ -963,6 +1025,15 @@ jobs:
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
2>/dev/null || echo "")
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
# The tag resolves but carries no readable label. Expected exactly
# once per artifact, during the migration onto labels. If it recurs
# every push, something is rewriting the channel tag as a manifest
# index — see the repoint step's note.
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
echo "reuse: NOTE Fine once, while migrating. Every push means the"
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
fi
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
@@ -1003,17 +1074,39 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
# The source tag is EXCLUDED from the targets, and that is load-
# bearing rather than an optimisation.
#
# `imagetools create` wraps the source manifest in an INDEX. Point it
# at the channel tag with that same tag as a target and the tag stops
# being a plain image — after which `.Image.Config.Labels` no longer
# resolves through it and the fc.revision label reads as absent. The
# next push then misses and rebuilds, so reuse worked exactly once
# and every subsequent push paid full price. Observed on run 4751:
# ml:dev reported fc.revision=<none> one push after run 4749 had read
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
#
# Excluding the source means the channel tag is only ever written by
# a real build, so it stays a plain image and stays readable. On dev
# that leaves nothing to do — :dev already points at the right
# content, which is what the hit established. On main it leaves
# :c-<sha>, which rule 145 requires of every main push whether or not
# a build ran.
#
# steps.tag emits ONE comma-separated list, because that is the shape
# docker/build-push-action takes; imagetools wants a -t per ref.
ARGS=""
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
for t in $TAGS; do
[ "$t" = "$SOURCE" ] && continue
ARGS="$ARGS -t $t"
done
unset IFS
# Source is the channel tag itself — the image we just confirmed
# carries this revision. On dev the only target IS that tag, so this
# is a no-op that keeps the code path uniform. On main it is what
# gives the new commit its :c-<sha>, which rule 145 requires of every
# main push whether or not a build ran.
if [ -z "$ARGS" ]; then
echo "repoint: $SOURCE already carries this revision and is the"
echo "repoint: only tag for this channel — nothing to write."
exit 0
fi
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$SOURCE"
echo "repointed to $SOURCE: $TAGS"
echo "repointed from $SOURCE:$ARGS"