bvandeusen/FabledCurator
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 12 Wiki Activity

Stop minting identifiers nothing reads: moving channel tags, :c-<sha> on main, and a label-keyed build cache #239

Merged
bvandeusen merged 6 commits from dev into main 2026-08-28 15:14:35 -04:00
Conversation 0 Commits 6 Files Changed 3
+249 -441
3 changed files with 249 additions and 441 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 7e065fed70 - Show all commits
.forgejo/workflows/build.yml
+159 -267
View File
@@ -452,12 +452,21 @@ jobs:
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
# --- reuse-if-published (milestone 313, step 4) ----------------------
# The identity tag names this artifact's CONTENT — r-<revision>, the
# commit its shipped files last changed in, plus the channel for images
# that bake one in. If the registry already carries it, the bytes this
# job would produce are already published and the build is pure waste:
# the channel and date tags get repointed at the existing manifest
# instead, registry-side, in seconds.
# Does the image the channel tag already points at carry THIS commit's
# revision? If so the bytes this job would produce are already published
# and the build is pure waste: the remaining tags get repointed at that
# existing manifest instead, registry-side, in seconds.
#
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
# (milestone 318 step 3). A tag would be a name minted per build that one
# thing reads — what rule 145 narrowed against — and would be prunable
# under the registry's keep_pattern (#3157), silently expiring the cache.
# A label rides inside a tag that has to exist anyway.
#
# An image with no such label reads as a miss and rebuilds. That is the
# migration, not a fault: labels cannot be backfilled, since the reuse
# path copies a manifest and config labels are not manifest annotations.
# Each artifact pays one rebuild, once.
#
# This is what stops a push that touched only `agent/` from rebuilding
# web and ml, and a merge to main from rebuilding what dev already built.
@@ -485,88 +494,39 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -eu
ID=$(sh scripts/artifacts.sh identity web "$CHANNEL")
echo "identity=$ID" >> "$GITHUB_OUTPUT"
# The value step 3 will key on once the label mechanism is proven.
# Emitted now so the build below can stamp it — stamping is harmless
# and is what gives the shadow read something to find next push.
REV=$(sh scripts/artifacts.sh revision web)
echo "revision=$REV" >> "$GITHUB_OUTPUT"
DERIVED=$(sh scripts/artifacts.sh revision web)
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
# Every build that runs now claims the identity. The carve-out here
# existed only for tag pushes, which rebuilt an already-published
# revision and so had to be stopped from re-pointing an immutable
# tag at fresh bytes. No tag trigger, nothing to carve out.
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
# The moving tag for this channel. Which tag we ask IS the channel —
# that is why the revision needs no -main/-dev qualifier any more.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
# (run 4732): a missing key returns an empty string and exits 0, so
# branching on the exit code would read "no label yet" as success.
# An unreachable tag also lands here as empty via the `|| echo`.
# Empty never equals a 12-char revision, so every uncertain case
# falls through to a build — the safe direction, with no special
# casing for it.
#
# Read the SPECIFIC key. The map also carries whatever the base image
# set, and `org.opencontainers.image.version` sits right beside ours
# looking like a plausible answer (it reads 24.04 on the agent).
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
2>/dev/null || echo "")
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is already published — skipping the build"
echo "reuse: already published — skipping the build"
else
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is not published — building"
echo "reuse: not published — building"
fi
# --- shadow: can we read a label off the channel tag? (milestone 318 step 3)
# Informational ONLY — no `set -e`, every probe falls back to
# UNAVAILABLE, and nothing downstream reads this. It must never fail a
# build; that is the whole reason it runs before the mechanism does.
#
# The question: can the reuse check be keyed on an image LABEL read off
# the channel tag, instead of on a separate r-<rev> tag? If yes, nothing
# is minted per build, the -main/-dev identity split disappears, and the
# base-refresh clobber (#3154) stops existing.
#
# Two questions, answered at different times — worth separating, because
# reading "no value" as "the mechanism does not work" would abandon a
# working design:
#
# 1. Do the TEMPLATE MECHANICS work here? Answered on the very first
# run, by probes B and C: if they print a labels map (even an empty
# one) or a config JSON, `--format` over .Image resolves on this
# buildx and this registry. That is the actual gate for step 3.
# 2. Does a stamped value ROUND-TRIP? Only answerable once an artifact
# genuinely rebuilds. A push that changes no shipped file reuses,
# the build is skipped, and nothing gets stamped — so an artifact
# keeps reporting no label until its own source next moves.
#
# That second point is not a defect, it is the migration path: under the
# real mechanism "no label" reads as a MISS, so the artifact rebuilds
# once and is stamped from then on. One extra build per artifact, ever,
# in the safe direction. `imagetools create` cannot add the label on the
# reuse path — it copies a manifest and config labels are not manifest
# annotations — so there is no way to backfill, and no need to.
#
# Several probes on purpose: `--format` templating over .Image varies by
# buildx version, and one push should tell us which spelling works
# rather than costing a round trip per guess.
- name: Shadow — read fc.revision off the channel tag (informational)
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
CHANNEL: ${{ steps.tag.outputs.channel }}
DERIVED: ${{ steps.reuse.outputs.revision }}
run: |
set -u
# The moving tag this channel publishes. `main` is asked about
# :latest, which is what step 4 keeps; :main is on its way out.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
REF="$IMAGE:$T"
echo "label-shadow: ref=$REF derived=$DERIVED"
A=$(docker buildx imagetools inspect "$REF" \
--format '{{ index .Image.Config.Labels "fc.revision" }}' 2>&1 || echo FAILED)
echo "label-shadow: probe-index = $A"
B=$(docker buildx imagetools inspect "$REF" \
--format '{{ .Image.Config.Labels }}' 2>&1 | head -c 300 || echo FAILED)
echo "label-shadow: probe-labels = $B"
C=$(docker buildx imagetools inspect "$REF" \
--format '{{ json .Image }}' 2>&1 | head -c 400 || echo FAILED)
echo "label-shadow: probe-json = $C"
echo "label-shadow: buildx = $(docker buildx version 2>&1 || echo UNAVAILABLE)"
- name: Download signed XPI from Forgejo release asset
# dev and main each bundle the XPI their own sign-extension just
# published — the point of the channel work (milestone 271 step 6): the
@@ -651,9 +611,9 @@ jobs:
file: Dockerfile
push: true
tags: ${{ steps.reuse.outputs.build_tags }}
# Stamped now, read later. Additive and inert this step: no
# decision depends on it until the shadow above proves the
# label can actually be read back (milestone 318 step 3).
# The reuse key. Read back off the channel tag on the next push to
# decide whether that push needs to build at all, so this is not
# decoration — an unstamped image is one that will always rebuild.
labels: |
fc.revision=${{ steps.reuse.outputs.revision }}
# Only the web image carries a channel: it is the one that serves
@@ -664,8 +624,8 @@ jobs:
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# identity tag holds, so :latest and the date pin are byte-identical to
# what was published rather than a lookalike rebuild.
# channel tag already holds, so :c-<sha> and the date pin are
# byte-identical to what is published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
@@ -675,7 +635,7 @@ jobs:
if: steps.reuse.outputs.hit == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
IDENTITY: ${{ steps.reuse.outputs.identity }}
SOURCE: ${{ steps.reuse.outputs.channel_ref }}
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
@@ -685,9 +645,13 @@ jobs:
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
unset IFS
# Source is the channel tag itself — the image we just confirmed
# carries this revision. One of the targets is that same tag, which
# makes its copy a no-op; the others (:main, :c-<sha>, the date pin)
# are what this exists for.
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
docker buildx imagetools create $ARGS "$SOURCE"
echo "repointed to $SOURCE: $TAGS"
build-ml:
runs-on: python-ci
@@ -780,12 +744,21 @@ jobs:
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
# --- reuse-if-published (milestone 313, step 4) ----------------------
# The identity tag names this artifact's CONTENT — r-<revision>, the
# commit its shipped files last changed in, plus the channel for images
# that bake one in. If the registry already carries it, the bytes this
# job would produce are already published and the build is pure waste:
# the channel and date tags get repointed at the existing manifest
# instead, registry-side, in seconds.
# Does the image the channel tag already points at carry THIS commit's
# revision? If so the bytes this job would produce are already published
# and the build is pure waste: the remaining tags get repointed at that
# existing manifest instead, registry-side, in seconds.
#
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
# (milestone 318 step 3). A tag would be a name minted per build that one
# thing reads — what rule 145 narrowed against — and would be prunable
# under the registry's keep_pattern (#3157), silently expiring the cache.
# A label rides inside a tag that has to exist anyway.
#
# An image with no such label reads as a miss and rebuilds. That is the
# migration, not a fault: labels cannot be backfilled, since the reuse
# path copies a manifest and config labels are not manifest annotations.
# Each artifact pays one rebuild, once.
#
# This is what stops a push that touched only `agent/` from rebuilding
# web and ml, and a merge to main from rebuilding what dev already built.
@@ -813,88 +786,39 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -eu
ID=$(sh scripts/artifacts.sh identity ml "$CHANNEL")
echo "identity=$ID" >> "$GITHUB_OUTPUT"
# The value step 3 will key on once the label mechanism is proven.
# Emitted now so the build below can stamp it — stamping is harmless
# and is what gives the shadow read something to find next push.
REV=$(sh scripts/artifacts.sh revision ml)
echo "revision=$REV" >> "$GITHUB_OUTPUT"
DERIVED=$(sh scripts/artifacts.sh revision ml)
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
# Every build that runs now claims the identity. The carve-out here
# existed only for tag pushes, which rebuilt an already-published
# revision and so had to be stopped from re-pointing an immutable
# tag at fresh bytes. No tag trigger, nothing to carve out.
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
# The moving tag for this channel. Which tag we ask IS the channel —
# that is why the revision needs no -main/-dev qualifier any more.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
# (run 4732): a missing key returns an empty string and exits 0, so
# branching on the exit code would read "no label yet" as success.
# An unreachable tag also lands here as empty via the `|| echo`.
# Empty never equals a 12-char revision, so every uncertain case
# falls through to a build — the safe direction, with no special
# casing for it.
#
# Read the SPECIFIC key. The map also carries whatever the base image
# set, and `org.opencontainers.image.version` sits right beside ours
# looking like a plausible answer (it reads 24.04 on the agent).
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
2>/dev/null || echo "")
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is already published — skipping the build"
echo "reuse: already published — skipping the build"
else
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is not published — building"
echo "reuse: not published — building"
fi
# --- shadow: can we read a label off the channel tag? (milestone 318 step 3)
# Informational ONLY — no `set -e`, every probe falls back to
# UNAVAILABLE, and nothing downstream reads this. It must never fail a
# build; that is the whole reason it runs before the mechanism does.
#
# The question: can the reuse check be keyed on an image LABEL read off
# the channel tag, instead of on a separate r-<rev> tag? If yes, nothing
# is minted per build, the -main/-dev identity split disappears, and the
# base-refresh clobber (#3154) stops existing.
#
# Two questions, answered at different times — worth separating, because
# reading "no value" as "the mechanism does not work" would abandon a
# working design:
#
# 1. Do the TEMPLATE MECHANICS work here? Answered on the very first
# run, by probes B and C: if they print a labels map (even an empty
# one) or a config JSON, `--format` over .Image resolves on this
# buildx and this registry. That is the actual gate for step 3.
# 2. Does a stamped value ROUND-TRIP? Only answerable once an artifact
# genuinely rebuilds. A push that changes no shipped file reuses,
# the build is skipped, and nothing gets stamped — so an artifact
# keeps reporting no label until its own source next moves.
#
# That second point is not a defect, it is the migration path: under the
# real mechanism "no label" reads as a MISS, so the artifact rebuilds
# once and is stamped from then on. One extra build per artifact, ever,
# in the safe direction. `imagetools create` cannot add the label on the
# reuse path — it copies a manifest and config labels are not manifest
# annotations — so there is no way to backfill, and no need to.
#
# Several probes on purpose: `--format` templating over .Image varies by
# buildx version, and one push should tell us which spelling works
# rather than costing a round trip per guess.
- name: Shadow — read fc.revision off the channel tag (informational)
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
CHANNEL: ${{ steps.tag.outputs.channel }}
DERIVED: ${{ steps.reuse.outputs.revision }}
run: |
set -u
# The moving tag this channel publishes. `main` is asked about
# :latest, which is what step 4 keeps; :main is on its way out.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
REF="$IMAGE:$T"
echo "label-shadow: ref=$REF derived=$DERIVED"
A=$(docker buildx imagetools inspect "$REF" \
--format '{{ index .Image.Config.Labels "fc.revision" }}' 2>&1 || echo FAILED)
echo "label-shadow: probe-index = $A"
B=$(docker buildx imagetools inspect "$REF" \
--format '{{ .Image.Config.Labels }}' 2>&1 | head -c 300 || echo FAILED)
echo "label-shadow: probe-labels = $B"
C=$(docker buildx imagetools inspect "$REF" \
--format '{{ json .Image }}' 2>&1 | head -c 400 || echo FAILED)
echo "label-shadow: probe-json = $C"
echo "label-shadow: buildx = $(docker buildx version 2>&1 || echo UNAVAILABLE)"
- name: Build and push ml image
if: steps.reuse.outputs.hit != 'true'
uses: docker/build-push-action@v5
@@ -903,16 +827,16 @@ jobs:
file: Dockerfile.ml
push: true
tags: ${{ steps.reuse.outputs.build_tags }}
# Stamped now, read later. Additive and inert this step: no
# decision depends on it until the shadow above proves the
# label can actually be read back (milestone 318 step 3).
# The reuse key. Read back off the channel tag on the next push to
# decide whether that push needs to build at all, so this is not
# decoration — an unstamped image is one that will always rebuild.
labels: |
fc.revision=${{ steps.reuse.outputs.revision }}
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# identity tag holds, so :latest and the date pin are byte-identical to
# what was published rather than a lookalike rebuild.
# channel tag already holds, so :c-<sha> and the date pin are
# byte-identical to what is published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
@@ -922,7 +846,7 @@ jobs:
if: steps.reuse.outputs.hit == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
IDENTITY: ${{ steps.reuse.outputs.identity }}
SOURCE: ${{ steps.reuse.outputs.channel_ref }}
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
@@ -932,9 +856,13 @@ jobs:
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
unset IFS
# Source is the channel tag itself — the image we just confirmed
# carries this revision. One of the targets is that same tag, which
# makes its copy a no-op; the others (:main, :c-<sha>, the date pin)
# are what this exists for.
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
docker buildx imagetools create $ARGS "$SOURCE"
echo "repointed to $SOURCE: $TAGS"
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
# the GPU machine instead of building locally. Independent of web/ml (its own
@@ -1023,12 +951,21 @@ jobs:
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
# --- reuse-if-published (milestone 313, step 4) ----------------------
# The identity tag names this artifact's CONTENT — r-<revision>, the
# commit its shipped files last changed in, plus the channel for images
# that bake one in. If the registry already carries it, the bytes this
# job would produce are already published and the build is pure waste:
# the channel and date tags get repointed at the existing manifest
# instead, registry-side, in seconds.
# Does the image the channel tag already points at carry THIS commit's
# revision? If so the bytes this job would produce are already published
# and the build is pure waste: the remaining tags get repointed at that
# existing manifest instead, registry-side, in seconds.
#
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
# (milestone 318 step 3). A tag would be a name minted per build that one
# thing reads — what rule 145 narrowed against — and would be prunable
# under the registry's keep_pattern (#3157), silently expiring the cache.
# A label rides inside a tag that has to exist anyway.
#
# An image with no such label reads as a miss and rebuilds. That is the
# migration, not a fault: labels cannot be backfilled, since the reuse
# path copies a manifest and config labels are not manifest annotations.
# Each artifact pays one rebuild, once.
#
# This is what stops a push that touched only `agent/` from rebuilding
# web and ml, and a merge to main from rebuilding what dev already built.
@@ -1056,88 +993,39 @@ jobs:
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -eu
ID=$(sh scripts/artifacts.sh identity agent "$CHANNEL")
echo "identity=$ID" >> "$GITHUB_OUTPUT"
# The value step 3 will key on once the label mechanism is proven.
# Emitted now so the build below can stamp it — stamping is harmless
# and is what gives the shadow read something to find next push.
REV=$(sh scripts/artifacts.sh revision agent)
echo "revision=$REV" >> "$GITHUB_OUTPUT"
DERIVED=$(sh scripts/artifacts.sh revision agent)
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
# Every build that runs now claims the identity. The carve-out here
# existed only for tag pushes, which rebuilt an already-published
# revision and so had to be stopped from re-pointing an immutable
# tag at fresh bytes. No tag trigger, nothing to carve out.
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
# The moving tag for this channel. Which tag we ask IS the channel —
# that is why the revision needs no -main/-dev qualifier any more.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
# (run 4732): a missing key returns an empty string and exits 0, so
# branching on the exit code would read "no label yet" as success.
# An unreachable tag also lands here as empty via the `|| echo`.
# Empty never equals a 12-char revision, so every uncertain case
# falls through to a build — the safe direction, with no special
# casing for it.
#
# Read the SPECIFIC key. The map also carries whatever the base image
# set, and `org.opencontainers.image.version` sits right beside ours
# looking like a plausible answer (it reads 24.04 on the agent).
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
2>/dev/null || echo "")
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is already published — skipping the build"
echo "reuse: already published — skipping the build"
else
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: $IMAGE:$ID is not published — building"
echo "reuse: not published — building"
fi
# --- shadow: can we read a label off the channel tag? (milestone 318 step 3)
# Informational ONLY — no `set -e`, every probe falls back to
# UNAVAILABLE, and nothing downstream reads this. It must never fail a
# build; that is the whole reason it runs before the mechanism does.
#
# The question: can the reuse check be keyed on an image LABEL read off
# the channel tag, instead of on a separate r-<rev> tag? If yes, nothing
# is minted per build, the -main/-dev identity split disappears, and the
# base-refresh clobber (#3154) stops existing.
#
# Two questions, answered at different times — worth separating, because
# reading "no value" as "the mechanism does not work" would abandon a
# working design:
#
# 1. Do the TEMPLATE MECHANICS work here? Answered on the very first
# run, by probes B and C: if they print a labels map (even an empty
# one) or a config JSON, `--format` over .Image resolves on this
# buildx and this registry. That is the actual gate for step 3.
# 2. Does a stamped value ROUND-TRIP? Only answerable once an artifact
# genuinely rebuilds. A push that changes no shipped file reuses,
# the build is skipped, and nothing gets stamped — so an artifact
# keeps reporting no label until its own source next moves.
#
# That second point is not a defect, it is the migration path: under the
# real mechanism "no label" reads as a MISS, so the artifact rebuilds
# once and is stamped from then on. One extra build per artifact, ever,
# in the safe direction. `imagetools create` cannot add the label on the
# reuse path — it copies a manifest and config labels are not manifest
# annotations — so there is no way to backfill, and no need to.
#
# Several probes on purpose: `--format` templating over .Image varies by
# buildx version, and one push should tell us which spelling works
# rather than costing a round trip per guess.
- name: Shadow — read fc.revision off the channel tag (informational)
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
CHANNEL: ${{ steps.tag.outputs.channel }}
DERIVED: ${{ steps.reuse.outputs.revision }}
run: |
set -u
# The moving tag this channel publishes. `main` is asked about
# :latest, which is what step 4 keeps; :main is on its way out.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
REF="$IMAGE:$T"
echo "label-shadow: ref=$REF derived=$DERIVED"
A=$(docker buildx imagetools inspect "$REF" \
--format '{{ index .Image.Config.Labels "fc.revision" }}' 2>&1 || echo FAILED)
echo "label-shadow: probe-index = $A"
B=$(docker buildx imagetools inspect "$REF" \
--format '{{ .Image.Config.Labels }}' 2>&1 | head -c 300 || echo FAILED)
echo "label-shadow: probe-labels = $B"
C=$(docker buildx imagetools inspect "$REF" \
--format '{{ json .Image }}' 2>&1 | head -c 400 || echo FAILED)
echo "label-shadow: probe-json = $C"
echo "label-shadow: buildx = $(docker buildx version 2>&1 || echo UNAVAILABLE)"
- name: Build and push agent image
if: steps.reuse.outputs.hit != 'true'
uses: docker/build-push-action@v5
@@ -1146,16 +1034,16 @@ jobs:
file: agent/Dockerfile
push: true
tags: ${{ steps.reuse.outputs.build_tags }}
# Stamped now, read later. Additive and inert this step: no
# decision depends on it until the shadow above proves the
# label can actually be read back (milestone 318 step 3).
# The reuse key. Read back off the channel tag on the next push to
# decide whether that push needs to build at all, so this is not
# decoration — an unstamped image is one that will always rebuild.
labels: |
fc.revision=${{ steps.reuse.outputs.revision }}
# Registry-side manifest copy: no layer transfer, no local daemon, no
# rebuild. Each -t becomes another reference to the SAME manifest the
# identity tag holds, so :latest and the date pin are byte-identical to
# what was published rather than a lookalike rebuild.
# channel tag already holds, so :c-<sha> and the date pin are
# byte-identical to what is published rather than a lookalike rebuild.
#
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
# rolling channel refreshes itself, so skipping a build must never mean
@@ -1165,7 +1053,7 @@ jobs:
if: steps.reuse.outputs.hit == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
IDENTITY: ${{ steps.reuse.outputs.identity }}
SOURCE: ${{ steps.reuse.outputs.channel_ref }}
TAGS: ${{ steps.tag.outputs.tags }}
run: |
set -euf
@@ -1175,6 +1063,10 @@ jobs:
IFS=,
for t in $TAGS; do ARGS="$ARGS -t $t"; done
unset IFS
# Source is the channel tag itself — the image we just confirmed
# carries this revision. One of the targets is that same tag, which
# makes its copy a no-op; the others (:main, :c-<sha>, the date pin)
# are what this exists for.
# shellcheck disable=SC2086
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
docker buildx imagetools create $ARGS "$SOURCE"
echo "repointed to $SOURCE: $TAGS"
scripts/artifacts.sh
+12 -58
View File
@@ -56,25 +56,9 @@ ML_PATHS='Dockerfile.ml requirements-ml.txt requirements.txt backend alembic ale
# it: this is deliberately NOT `agent/`.
AGENT_PATHS='agent/Dockerfile agent/requirements.txt agent/fc_agent'
# Which artifacts bake the BUILD CHANNEL into the image, and therefore cannot
# share a content identity across channels. The web image takes FC_CHANNEL as
# a build-arg and reports it from /api/extension/manifest (milestone 271 step
# 7), so `main` and `dev` builds of one revision are genuinely different
# images — reusing the dev one on main would ship an instance that names
# itself `dev` forever.
#
# ml and agent take no build-args at all: one revision, one image, and a merge
# to main can reuse exactly what dev already built. That is not a detail, it is
# most of what step 4 saves — merges would otherwise rebuild the agent's CUDA
# image to produce bytes that already exist.
#
# Extend this list if a second artifact ever gains a build-arg;
# tests/test_artifact_identity.py reads the Dockerfiles and fails if it drifts.
CHANNELLED='web'
usage() {
echo "usage: artifacts.sh {paths|revision|version|tag} {web|ml|agent|extension}" >&2
echo " artifacts.sh identity {web|ml|agent} [channel]" >&2
exit 2
}
@@ -124,9 +108,18 @@ strip0() {
}
# The IDENTITY of an artifact's content: the commit its shipped files last
# changed in. This — not the tag — is what decides whether a build can be
# skipped, because the published tag is only day-precise and two different
# builds can share it.
# changed in. This is what decides whether a build can be skipped.
#
# It is published as the `fc.revision` LABEL on the image itself, and read
# back off the moving channel tag — not as a tag of its own (milestone 318
# step 3). A tag would be a name minted per build that only one thing reads,
# which is what rule 145 narrowed against; it would also be prunable under the
# registry's keep_pattern (#3157), so the cache would silently expire.
#
# A published image with no such label reads as a MISS and rebuilds. That is
# the migration path, not a fault: `imagetools create` copies a manifest and
# config labels are not manifest annotations, so the reuse path cannot stamp
# one and there is nothing to backfill. Each artifact pays one rebuild, once.
cmd_revision() {
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
}
@@ -157,50 +150,11 @@ cmd_tag() {
"$(strip0 "$(fmt "$sha" %d)")"
}
# The CONTENT IDENTITY of a published image: an immutable tag naming exactly
# what a build of this commit would produce. build.yml asks the registry for it
# and, on a hit, skips the build entirely and repoints the channel and date
# tags at the manifest that is already there (milestone 313 step 4).
#
# It is deliberately NOT either of the other two values:
# * the date tag is day-precise and last-one-wins, so two different builds
# share it — it cannot answer "is this content published?".
# * the commit sha moves on every push, so it would never hit, which is the
# redundant rebuild this exists to remove.
#
# The revision does both jobs: it is content-unique AND stable across pushes
# that did not touch the artifact.
cmd_identity() {
_art=$1
_chan=${2:-}
case "$_art" in
web|ml|agent) ;;
extension)
echo "artifacts.sh: the extension is cached as an ext-<version> Forgejo release, not an image tag — use \`version\`" >&2
exit 2 ;;
*) usage ;;
esac
for _c in $CHANNELLED; do
if [ "$_art" = "$_c" ]; then
# Refused rather than defaulted: an unqualified identity for a
# channelled artifact would let a dev image be reused as the main one.
if [ -z "$_chan" ]; then
echo "artifacts.sh: $_art bakes the channel into the image — identity needs one" >&2
exit 2
fi
printf 'r-%s-%s\n' "$(cmd_revision "$_art")" "$_chan"
return
fi
done
printf 'r-%s\n' "$(cmd_revision "$_art")"
}
[ $# -ge 2 ] || usage
case "$1" in
paths) cmd_paths "$2" ;;
revision) cmd_revision "$2" ;;
version) cmd_version "$2" ;;
tag) cmd_tag "$2" ;;
identity) cmd_identity "$2" "${3:-}" ;;
*) usage ;;
esac
tests/test_artifact_identity.py
+78 -116
View File
@@ -1,140 +1,102 @@
"""`artifacts.sh identity` is what decides whether a build gets skipped.
"""`artifacts.sh revision` is what decides whether a build gets skipped.
Milestone 313 step 4: build.yml asks the registry for `<image>:<identity>` and,
on a hit, publishes NO new bytes — it repoints the channel and date tags at the
manifest already there. So the identity has to be a true name for the content.
Both ways of getting it wrong are silent at build time and only surface in
production:
Milestone 318 step 3: each image carries its revision as an `fc.revision`
label, and build.yml reads that label back off the moving channel tag. Equal
to the derived revision means the bytes this push would produce are already
published, so the build is skipped.
* **too coarse** — two genuinely different images share an identity, so the
second one never gets built and its tags point at the first one's bytes. The
live case is FC_CHANNEL: a `dev` and a `main` build of one revision differ,
and collapsing them ships an instance that reports the wrong channel forever.
* **too fine** — the identity moves when the content did not, nothing ever
hits, and step 4 buys nothing. A commit sha would do exactly this.
That makes the revision load-bearing in a way a version string is not — it is
compared for equality against a value stamped into a real published artifact.
Both ways of getting it wrong are silent:
The Dockerfiles are read here rather than trusted, because the coarse direction
appears the moment someone adds a build-arg without touching `CHANNELLED`.
* **it does not identify the content** — a revision that moves when the source
did not (a HEAD-derived value, say) never matches, nothing is ever skipped,
and the mechanism quietly buys nothing while every lane stays green.
* **it identifies the wrong content** — a revision that holds still when the
source DID change matches a stale label, the build is skipped, and the
channel serves bytes that do not correspond to the commit. This is the
dangerous direction, and it is what `test_artifact_paths.py` guards from the
other side by pinning the path sets.
This module owns the narrower claim: whatever the path sets say, the revision
is genuinely the commit those paths last changed in.
The identity-TAG tests this file used to hold are gone with the tag. There is
no longer a `CHANNELLED` list to drift (the channel is which tag you inspect),
and no `identity` subcommand to refuse an unqualified call.
"""
from __future__ import annotations
import re
import subprocess
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parent.parent
from test_artifact_paths import ROOT, declared_paths
# Only image artifacts have an identity — the extension is cached as an
# ext-<version> Forgejo release, not a registry tag.
IMAGE_ARTIFACTS = {
"web": "Dockerfile",
"ml": "Dockerfile.ml",
"agent": "agent/Dockerfile",
}
ARTIFACTS = ("web", "ml", "agent", "extension")
CHANNELS = ("main", "dev")
# docker's own tag grammar: [A-Za-z0-9_][A-Za-z0-9._-]{0,127}
_TAG = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$")
# `ARG FC_CHANNEL` in a Dockerfile means build.yml passes a per-channel value
# in, so the channel is part of what the image IS.
_ARG_CHANNEL = re.compile(r"^\s*ARG\s+FC_CHANNEL\b", re.MULTILINE)
# 12 hex chars — the prefix build.yml stamps and compares.
_REVISION = re.compile(r"^[0-9a-f]{12}$")
def identity(artifact: str, channel: str | None = None) -> subprocess.CompletedProcess:
cmd = ["sh", str(ROOT / "scripts" / "artifacts.sh"), "identity", artifact]
if channel is not None:
cmd.append(channel)
return subprocess.run(cmd, capture_output=True, text=True, cwd=ROOT)
def ok(artifact: str, channel: str | None = None) -> str:
proc = identity(artifact, channel)
assert proc.returncode == 0, f"identity {artifact} {channel}: {proc.stderr}"
return proc.stdout.strip()
def bakes_the_channel(artifact: str) -> bool:
return bool(_ARG_CHANNEL.search((ROOT / IMAGE_ARTIFACTS[artifact]).read_text()))
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
def test_channel_dependence_matches_the_dockerfile(artifact):
"""The coarse direction, caught at its source.
Whether the channel belongs in the identity is not a preference — it is
dictated by whether the Dockerfile takes it as a build-arg. Adding an
`ARG FC_CHANNEL` to another image without adding it to `CHANNELLED` would
make its dev and main builds collide, and nothing else would notice.
"""
per_channel = {c: ok(artifact, c) for c in CHANNELS}
differs = len(set(per_channel.values())) > 1
if bakes_the_channel(artifact):
assert differs, (
f"{IMAGE_ARTIFACTS[artifact]} declares ARG FC_CHANNEL, so a dev "
f"build and a main build of one revision are different images — "
f"but both derive the identity {per_channel['main']!r}. The main "
f"build would reuse the dev image and report the wrong channel. "
f"Add {artifact!r} to CHANNELLED in scripts/artifacts.sh."
)
else:
assert not differs, (
f"{IMAGE_ARTIFACTS[artifact]} takes no channel build-arg, so one "
f"revision is one image and a merge to main should reuse what dev "
f"already built — but the identity differs per channel "
f"({per_channel}), so every merge rebuilds it for nothing. Remove "
f"{artifact!r} from CHANNELLED in scripts/artifacts.sh."
)
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
def test_identity_tracks_the_artifacts_own_revision(artifact):
"""The fine direction: the identity must be the revision, not the push.
`revision` is the commit this artifact's shipped files last changed in, so
it holds still across pushes that did not touch it. Anything derived from
HEAD instead would move every push and never hit the registry.
"""
rev = subprocess.run(
def revision(artifact: str) -> str:
return subprocess.run(
["sh", str(ROOT / "scripts" / "artifacts.sh"), "revision", artifact],
capture_output=True, text=True, check=True, cwd=ROOT,
).stdout.strip()
value = ok(artifact, "main")
assert rev and rev in value, (
f"identity {value!r} does not contain the {artifact} revision {rev!r}"
@pytest.mark.parametrize("artifact", ARTIFACTS)
def test_revision_is_the_commit_its_own_shipped_files_last_changed_in(artifact):
"""The claim the whole skip decision rests on.
Asked of git directly rather than of the script, so this fails if the
derivation ever stops meaning what it says — deriving from HEAD, from a
build clock, or from a path set it did not actually use.
"""
paths = declared_paths(artifact)
expected = subprocess.run(
["git", "log", "--format=%H", "-1", "HEAD", "--", *paths],
capture_output=True, text=True, check=True, cwd=ROOT,
).stdout.strip()
assert expected, (
f"no commit in this history touches the {artifact} path set — the "
f"derivation has nothing to stand on"
)
assert expected.startswith(revision(artifact)), (
f"{artifact} derives {revision(artifact)!r}, but the newest commit "
f"touching its shipped files is {expected[:12]!r}. The label stamped "
f"into the image would not identify its own content."
)
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
def test_identity_is_a_legal_docker_tag(artifact):
"""It is pushed as a tag, so an illegal one fails at the registry — after
the build has already run."""
for channel in CHANNELS:
value = ok(artifact, channel)
assert _TAG.match(value), f"{value!r} is not a valid docker tag"
@pytest.mark.parametrize("artifact", ARTIFACTS)
def test_revision_is_a_legal_label_value_and_is_stable(artifact):
"""It is stamped as a docker label and compared for string equality, so a
stray newline or a varying value breaks the comparison rather than the
build — the mechanism would simply stop hitting, silently."""
first = revision(artifact)
assert _REVISION.match(first), f"{first!r} is not a 12-char hex revision"
assert first == revision(artifact), "revision is not stable across calls"
def test_a_channelled_artifact_refuses_an_unqualified_identity():
"""Refusing beats defaulting. If `identity web` quietly returned the
unqualified `r-<rev>`, a workflow that forgot to pass the channel would
publish one image under a name both channels then reuse — the exact
collision the CHANNELLED list exists to prevent, reintroduced by an
omission rather than by an edit."""
proc = identity("web")
assert proc.returncode != 0, (
"identity web returned a value with no channel: "
f"{proc.stdout.strip()!r}"
)
def test_an_artifact_whose_paths_did_not_change_keeps_its_revision():
"""The property that makes skipping possible at all.
def test_the_extension_has_no_image_identity():
"""It is cached as an ext-<version> release asset, and its cache key is the
version. Answering with a plausible image tag would invite a second,
divergent cache."""
proc = identity("extension", "main")
assert proc.returncode != 0
assert "ext-" in proc.stderr
The agent's set is disjoint from web's, so the two must be free to differ.
Asserting they *are* different today would pin an accident of history —
what matters is that the derivation is per-artifact rather than global, so
this asserts each artifact's revision is drawn from its own path set.
"""
seen = {a: revision(a) for a in ARTIFACTS}
for artifact, rev in seen.items():
touched = subprocess.run(
["git", "log", "--format=%H", "-1", "HEAD", "--", *declared_paths(artifact)],
capture_output=True, text=True, check=True, cwd=ROOT,
).stdout.strip()
assert touched.startswith(rev), (
f"{artifact}'s revision {rev!r} is not the newest commit touching "
f"its own paths — the derivation is not per-artifact"
)