From b3989d02243d1f32573cb4fcddd2686bed68925c Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Fri, 28 Aug 2026 14:55:49 -0400 Subject: [PATCH] fix(ci): the repoint was destroying the label it depends on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reuse worked exactly once per artifact, then every later push rebuilt at full price. Nothing failed and nothing went red — the savings simply evaporated. `imagetools create` wraps its source manifest in an INDEX. The repoint step passed the channel tag as both source and target, so after any reuse :dev stopped being a plain image, `.Image.Config.Labels` no longer resolved through it, and fc.revision read as absent on the next push. Observed across three runs rather than reasoned about: run 4749 read fc.revision=a7e626a67a79 off fabledcurator-ml:dev and skipped the build; run 4751 read off the same tag and rebuilt. The only thing to touch it in between was 4749's own repoint. The agent hit in 4751 precisely because its :dev had last been written by a real build, not by a repoint — which is the control case. Milestone 313's r- design was immune without anyone noticing why: the source (the identity tag) was never one of the targets. Step 3 made the channel tag both, and inherited a bug the earlier shape had avoided by accident. Fix: exclude the source from the target list, so the channel tag is only ever written by a real build and stays a plain readable image. On dev that leaves nothing to do, which is correct — the hit already established that :dev points at the right content. On main it leaves :c-, which rule 145 requires of every main push whether or not a build ran. Also added a note the reuse step prints when a channel tag exists but carries no readable label. That is expected exactly once per artifact during the migration; if it appears on every push, the tag is being index-wrapped again and reuse is dead. This class of failure — correct behaviour, quietly worth less than it reads — is the third one this milestone has turned up, and it is the one that does not announce itself. --- .forgejo/workflows/build.yml | 135 +++++++++++++++++++++++++++++------ 1 file changed, 114 insertions(+), 21 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 0334769..47228b7 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -504,6 +504,15 @@ jobs: --format '{{ index .Image.Config.Labels "fc.revision" }}' \ 2>/dev/null || echo "") echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-}; derived=$DERIVED" + if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then + # The tag resolves but carries no readable label. Expected exactly + # once per artifact, during the migration onto labels. If it recurs + # every push, something is rewriting the channel tag as a manifest + # index — see the repoint step's note. + echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision." + echo "reuse: NOTE Fine once, while migrating. Every push means the" + echo "reuse: NOTE tag is being index-wrapped and reuse is dead." + fi if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then echo "hit=true" >> "$GITHUB_OUTPUT" @@ -625,20 +634,42 @@ jobs: TAGS: ${{ steps.tag.outputs.tags }} run: | set -euf + # The source tag is EXCLUDED from the targets, and that is load- + # bearing rather than an optimisation. + # + # `imagetools create` wraps the source manifest in an INDEX. Point it + # at the channel tag with that same tag as a target and the tag stops + # being a plain image — after which `.Image.Config.Labels` no longer + # resolves through it and the fc.revision label reads as absent. The + # next push then misses and rebuilds, so reuse worked exactly once + # and every subsequent push paid full price. Observed on run 4751: + # ml:dev reported fc.revision= one push after run 4749 had read + # a7e626a67a79 off it. Nothing failed; the savings just evaporated. + # + # Excluding the source means the channel tag is only ever written by + # a real build, so it stays a plain image and stays readable. On dev + # that leaves nothing to do — :dev already points at the right + # content, which is what the hit established. On main it leaves + # :c-, which rule 145 requires of every main push whether or not + # a build ran. + # # steps.tag emits ONE comma-separated list, because that is the shape # docker/build-push-action takes; imagetools wants a -t per ref. ARGS="" IFS=, - for t in $TAGS; do ARGS="$ARGS -t $t"; done + for t in $TAGS; do + [ "$t" = "$SOURCE" ] && continue + ARGS="$ARGS -t $t" + done unset IFS - # Source is the channel tag itself — the image we just confirmed - # carries this revision. On dev the only target IS that tag, so this - # is a no-op that keeps the code path uniform. On main it is what - # gives the new commit its :c-, which rule 145 requires of every - # main push whether or not a build ran. + if [ -z "$ARGS" ]; then + echo "repoint: $SOURCE already carries this revision and is the" + echo "repoint: only tag for this channel — nothing to write." + exit 0 + fi # shellcheck disable=SC2086 docker buildx imagetools create $ARGS "$SOURCE" - echo "repointed to $SOURCE: $TAGS" + echo "repointed from $SOURCE:$ARGS" build-ml: runs-on: python-ci @@ -775,6 +806,15 @@ jobs: --format '{{ index .Image.Config.Labels "fc.revision" }}' \ 2>/dev/null || echo "") echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-}; derived=$DERIVED" + if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then + # The tag resolves but carries no readable label. Expected exactly + # once per artifact, during the migration onto labels. If it recurs + # every push, something is rewriting the channel tag as a manifest + # index — see the repoint step's note. + echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision." + echo "reuse: NOTE Fine once, while migrating. Every push means the" + echo "reuse: NOTE tag is being index-wrapped and reuse is dead." + fi if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then echo "hit=true" >> "$GITHUB_OUTPUT" @@ -815,20 +855,42 @@ jobs: TAGS: ${{ steps.tag.outputs.tags }} run: | set -euf + # The source tag is EXCLUDED from the targets, and that is load- + # bearing rather than an optimisation. + # + # `imagetools create` wraps the source manifest in an INDEX. Point it + # at the channel tag with that same tag as a target and the tag stops + # being a plain image — after which `.Image.Config.Labels` no longer + # resolves through it and the fc.revision label reads as absent. The + # next push then misses and rebuilds, so reuse worked exactly once + # and every subsequent push paid full price. Observed on run 4751: + # ml:dev reported fc.revision= one push after run 4749 had read + # a7e626a67a79 off it. Nothing failed; the savings just evaporated. + # + # Excluding the source means the channel tag is only ever written by + # a real build, so it stays a plain image and stays readable. On dev + # that leaves nothing to do — :dev already points at the right + # content, which is what the hit established. On main it leaves + # :c-, which rule 145 requires of every main push whether or not + # a build ran. + # # steps.tag emits ONE comma-separated list, because that is the shape # docker/build-push-action takes; imagetools wants a -t per ref. ARGS="" IFS=, - for t in $TAGS; do ARGS="$ARGS -t $t"; done + for t in $TAGS; do + [ "$t" = "$SOURCE" ] && continue + ARGS="$ARGS -t $t" + done unset IFS - # Source is the channel tag itself — the image we just confirmed - # carries this revision. On dev the only target IS that tag, so this - # is a no-op that keeps the code path uniform. On main it is what - # gives the new commit its :c-, which rule 145 requires of every - # main push whether or not a build ran. + if [ -z "$ARGS" ]; then + echo "repoint: $SOURCE already carries this revision and is the" + echo "repoint: only tag for this channel — nothing to write." + exit 0 + fi # shellcheck disable=SC2086 docker buildx imagetools create $ARGS "$SOURCE" - echo "repointed to $SOURCE: $TAGS" + echo "repointed from $SOURCE:$ARGS" # The desktop GPU agent (#114) — published so the operator pulls + runs it on # the GPU machine instead of building locally. Independent of web/ml (its own @@ -963,6 +1025,15 @@ jobs: --format '{{ index .Image.Config.Labels "fc.revision" }}' \ 2>/dev/null || echo "") echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-}; derived=$DERIVED" + if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then + # The tag resolves but carries no readable label. Expected exactly + # once per artifact, during the migration onto labels. If it recurs + # every push, something is rewriting the channel tag as a manifest + # index — see the repoint step's note. + echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision." + echo "reuse: NOTE Fine once, while migrating. Every push means the" + echo "reuse: NOTE tag is being index-wrapped and reuse is dead." + fi if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then echo "hit=true" >> "$GITHUB_OUTPUT" @@ -1003,17 +1074,39 @@ jobs: TAGS: ${{ steps.tag.outputs.tags }} run: | set -euf + # The source tag is EXCLUDED from the targets, and that is load- + # bearing rather than an optimisation. + # + # `imagetools create` wraps the source manifest in an INDEX. Point it + # at the channel tag with that same tag as a target and the tag stops + # being a plain image — after which `.Image.Config.Labels` no longer + # resolves through it and the fc.revision label reads as absent. The + # next push then misses and rebuilds, so reuse worked exactly once + # and every subsequent push paid full price. Observed on run 4751: + # ml:dev reported fc.revision= one push after run 4749 had read + # a7e626a67a79 off it. Nothing failed; the savings just evaporated. + # + # Excluding the source means the channel tag is only ever written by + # a real build, so it stays a plain image and stays readable. On dev + # that leaves nothing to do — :dev already points at the right + # content, which is what the hit established. On main it leaves + # :c-, which rule 145 requires of every main push whether or not + # a build ran. + # # steps.tag emits ONE comma-separated list, because that is the shape # docker/build-push-action takes; imagetools wants a -t per ref. ARGS="" IFS=, - for t in $TAGS; do ARGS="$ARGS -t $t"; done + for t in $TAGS; do + [ "$t" = "$SOURCE" ] && continue + ARGS="$ARGS -t $t" + done unset IFS - # Source is the channel tag itself — the image we just confirmed - # carries this revision. On dev the only target IS that tag, so this - # is a no-op that keeps the code path uniform. On main it is what - # gives the new commit its :c-, which rule 145 requires of every - # main push whether or not a build ran. + if [ -z "$ARGS" ]; then + echo "repoint: $SOURCE already carries this revision and is the" + echo "repoint: only tag for this channel — nothing to write." + exit 0 + fi # shellcheck disable=SC2086 docker buildx imagetools create $ARGS "$SOURCE" - echo "repointed to $SOURCE: $TAGS" + echo "repointed from $SOURCE:$ARGS"