fix(ci): the repoint was destroying the label it depends on
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
Build images / build-web (push) Successful in 3s
extension / lint (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 42s
Build images / build-agent (push) Failing after 2m33s
CI / integration (push) Successful in 3m44s
extension / lint (pull_request) Successful in 25s
Build images / sign-extension (push) Successful in 3s
Build images / build-ml (push) Successful in 3s
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 2s
CI / frontend-build (push) Successful in 20s
Build images / build-web (push) Successful in 3s
extension / lint (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 42s
Build images / build-agent (push) Failing after 2m33s
CI / integration (push) Successful in 3m44s
extension / lint (pull_request) Successful in 25s
Reuse worked exactly once per artifact, then every later push rebuilt at full price. Nothing failed and nothing went red — the savings simply evaporated. `imagetools create` wraps its source manifest in an INDEX. The repoint step passed the channel tag as both source and target, so after any reuse :dev stopped being a plain image, `.Image.Config.Labels` no longer resolved through it, and fc.revision read as absent on the next push. Observed across three runs rather than reasoned about: run 4749 read fc.revision=a7e626a67a79 off fabledcurator-ml:dev and skipped the build; run 4751 read <none> off the same tag and rebuilt. The only thing to touch it in between was 4749's own repoint. The agent hit in 4751 precisely because its :dev had last been written by a real build, not by a repoint — which is the control case. Milestone 313's r-<rev> design was immune without anyone noticing why: the source (the identity tag) was never one of the targets. Step 3 made the channel tag both, and inherited a bug the earlier shape had avoided by accident. Fix: exclude the source from the target list, so the channel tag is only ever written by a real build and stays a plain readable image. On dev that leaves nothing to do, which is correct — the hit already established that :dev points at the right content. On main it leaves :c-<sha>, which rule 145 requires of every main push whether or not a build ran. Also added a note the reuse step prints when a channel tag exists but carries no readable label. That is expected exactly once per artifact during the migration; if it appears on every push, the tag is being index-wrapped again and reuse is dead. This class of failure — correct behaviour, quietly worth less than it reads — is the third one this milestone has turned up, and it is the one that does not announce itself.
This commit is contained in:
+114
-21
@@ -504,6 +504,15 @@ jobs:
|
|||||||
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
||||||
2>/dev/null || echo "")
|
2>/dev/null || echo "")
|
||||||
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
||||||
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
||||||
|
# The tag resolves but carries no readable label. Expected exactly
|
||||||
|
# once per artifact, during the migration onto labels. If it recurs
|
||||||
|
# every push, something is rewriting the channel tag as a manifest
|
||||||
|
# index — see the repoint step's note.
|
||||||
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
||||||
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
||||||
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
||||||
|
fi
|
||||||
|
|
||||||
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
echo "hit=true" >> "$GITHUB_OUTPUT"
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
@@ -625,20 +634,42 @@ jobs:
|
|||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
run: |
|
run: |
|
||||||
set -euf
|
set -euf
|
||||||
|
# The source tag is EXCLUDED from the targets, and that is load-
|
||||||
|
# bearing rather than an optimisation.
|
||||||
|
#
|
||||||
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
||||||
|
# at the channel tag with that same tag as a target and the tag stops
|
||||||
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
||||||
|
# resolves through it and the fc.revision label reads as absent. The
|
||||||
|
# next push then misses and rebuilds, so reuse worked exactly once
|
||||||
|
# and every subsequent push paid full price. Observed on run 4751:
|
||||||
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
||||||
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
||||||
|
#
|
||||||
|
# Excluding the source means the channel tag is only ever written by
|
||||||
|
# a real build, so it stays a plain image and stays readable. On dev
|
||||||
|
# that leaves nothing to do — :dev already points at the right
|
||||||
|
# content, which is what the hit established. On main it leaves
|
||||||
|
# :c-<sha>, which rule 145 requires of every main push whether or not
|
||||||
|
# a build ran.
|
||||||
|
#
|
||||||
# steps.tag emits ONE comma-separated list, because that is the shape
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
# docker/build-push-action takes; imagetools wants a -t per ref.
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
ARGS=""
|
ARGS=""
|
||||||
IFS=,
|
IFS=,
|
||||||
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
for t in $TAGS; do
|
||||||
|
[ "$t" = "$SOURCE" ] && continue
|
||||||
|
ARGS="$ARGS -t $t"
|
||||||
|
done
|
||||||
unset IFS
|
unset IFS
|
||||||
# Source is the channel tag itself — the image we just confirmed
|
if [ -z "$ARGS" ]; then
|
||||||
# carries this revision. On dev the only target IS that tag, so this
|
echo "repoint: $SOURCE already carries this revision and is the"
|
||||||
# is a no-op that keeps the code path uniform. On main it is what
|
echo "repoint: only tag for this channel — nothing to write."
|
||||||
# gives the new commit its :c-<sha>, which rule 145 requires of every
|
exit 0
|
||||||
# main push whether or not a build ran.
|
fi
|
||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
docker buildx imagetools create $ARGS "$SOURCE"
|
docker buildx imagetools create $ARGS "$SOURCE"
|
||||||
echo "repointed to $SOURCE: $TAGS"
|
echo "repointed from $SOURCE:$ARGS"
|
||||||
|
|
||||||
build-ml:
|
build-ml:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
@@ -775,6 +806,15 @@ jobs:
|
|||||||
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
||||||
2>/dev/null || echo "")
|
2>/dev/null || echo "")
|
||||||
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
||||||
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
||||||
|
# The tag resolves but carries no readable label. Expected exactly
|
||||||
|
# once per artifact, during the migration onto labels. If it recurs
|
||||||
|
# every push, something is rewriting the channel tag as a manifest
|
||||||
|
# index — see the repoint step's note.
|
||||||
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
||||||
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
||||||
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
||||||
|
fi
|
||||||
|
|
||||||
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
echo "hit=true" >> "$GITHUB_OUTPUT"
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
@@ -815,20 +855,42 @@ jobs:
|
|||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
run: |
|
run: |
|
||||||
set -euf
|
set -euf
|
||||||
|
# The source tag is EXCLUDED from the targets, and that is load-
|
||||||
|
# bearing rather than an optimisation.
|
||||||
|
#
|
||||||
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
||||||
|
# at the channel tag with that same tag as a target and the tag stops
|
||||||
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
||||||
|
# resolves through it and the fc.revision label reads as absent. The
|
||||||
|
# next push then misses and rebuilds, so reuse worked exactly once
|
||||||
|
# and every subsequent push paid full price. Observed on run 4751:
|
||||||
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
||||||
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
||||||
|
#
|
||||||
|
# Excluding the source means the channel tag is only ever written by
|
||||||
|
# a real build, so it stays a plain image and stays readable. On dev
|
||||||
|
# that leaves nothing to do — :dev already points at the right
|
||||||
|
# content, which is what the hit established. On main it leaves
|
||||||
|
# :c-<sha>, which rule 145 requires of every main push whether or not
|
||||||
|
# a build ran.
|
||||||
|
#
|
||||||
# steps.tag emits ONE comma-separated list, because that is the shape
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
# docker/build-push-action takes; imagetools wants a -t per ref.
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
ARGS=""
|
ARGS=""
|
||||||
IFS=,
|
IFS=,
|
||||||
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
for t in $TAGS; do
|
||||||
|
[ "$t" = "$SOURCE" ] && continue
|
||||||
|
ARGS="$ARGS -t $t"
|
||||||
|
done
|
||||||
unset IFS
|
unset IFS
|
||||||
# Source is the channel tag itself — the image we just confirmed
|
if [ -z "$ARGS" ]; then
|
||||||
# carries this revision. On dev the only target IS that tag, so this
|
echo "repoint: $SOURCE already carries this revision and is the"
|
||||||
# is a no-op that keeps the code path uniform. On main it is what
|
echo "repoint: only tag for this channel — nothing to write."
|
||||||
# gives the new commit its :c-<sha>, which rule 145 requires of every
|
exit 0
|
||||||
# main push whether or not a build ran.
|
fi
|
||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
docker buildx imagetools create $ARGS "$SOURCE"
|
docker buildx imagetools create $ARGS "$SOURCE"
|
||||||
echo "repointed to $SOURCE: $TAGS"
|
echo "repointed from $SOURCE:$ARGS"
|
||||||
|
|
||||||
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
|
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
|
||||||
# the GPU machine instead of building locally. Independent of web/ml (its own
|
# the GPU machine instead of building locally. Independent of web/ml (its own
|
||||||
@@ -963,6 +1025,15 @@ jobs:
|
|||||||
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
||||||
2>/dev/null || echo "")
|
2>/dev/null || echo "")
|
||||||
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
||||||
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
||||||
|
# The tag resolves but carries no readable label. Expected exactly
|
||||||
|
# once per artifact, during the migration onto labels. If it recurs
|
||||||
|
# every push, something is rewriting the channel tag as a manifest
|
||||||
|
# index — see the repoint step's note.
|
||||||
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
||||||
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
||||||
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
||||||
|
fi
|
||||||
|
|
||||||
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
echo "hit=true" >> "$GITHUB_OUTPUT"
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
@@ -1003,17 +1074,39 @@ jobs:
|
|||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
run: |
|
run: |
|
||||||
set -euf
|
set -euf
|
||||||
|
# The source tag is EXCLUDED from the targets, and that is load-
|
||||||
|
# bearing rather than an optimisation.
|
||||||
|
#
|
||||||
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
||||||
|
# at the channel tag with that same tag as a target and the tag stops
|
||||||
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
||||||
|
# resolves through it and the fc.revision label reads as absent. The
|
||||||
|
# next push then misses and rebuilds, so reuse worked exactly once
|
||||||
|
# and every subsequent push paid full price. Observed on run 4751:
|
||||||
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
||||||
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
||||||
|
#
|
||||||
|
# Excluding the source means the channel tag is only ever written by
|
||||||
|
# a real build, so it stays a plain image and stays readable. On dev
|
||||||
|
# that leaves nothing to do — :dev already points at the right
|
||||||
|
# content, which is what the hit established. On main it leaves
|
||||||
|
# :c-<sha>, which rule 145 requires of every main push whether or not
|
||||||
|
# a build ran.
|
||||||
|
#
|
||||||
# steps.tag emits ONE comma-separated list, because that is the shape
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
# docker/build-push-action takes; imagetools wants a -t per ref.
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
ARGS=""
|
ARGS=""
|
||||||
IFS=,
|
IFS=,
|
||||||
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
for t in $TAGS; do
|
||||||
|
[ "$t" = "$SOURCE" ] && continue
|
||||||
|
ARGS="$ARGS -t $t"
|
||||||
|
done
|
||||||
unset IFS
|
unset IFS
|
||||||
# Source is the channel tag itself — the image we just confirmed
|
if [ -z "$ARGS" ]; then
|
||||||
# carries this revision. On dev the only target IS that tag, so this
|
echo "repoint: $SOURCE already carries this revision and is the"
|
||||||
# is a no-op that keeps the code path uniform. On main it is what
|
echo "repoint: only tag for this channel — nothing to write."
|
||||||
# gives the new commit its :c-<sha>, which rule 145 requires of every
|
exit 0
|
||||||
# main push whether or not a build ran.
|
fi
|
||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
docker buildx imagetools create $ARGS "$SOURCE"
|
docker buildx imagetools create $ARGS "$SOURCE"
|
||||||
echo "repointed to $SOURCE: $TAGS"
|
echo "repointed from $SOURCE:$ARGS"
|
||||||
|
|||||||
Reference in New Issue
Block a user