Merge pull request 'Stop minting identifiers nothing reads: moving channel tags, :c-<sha> on main, and a label-keyed build cache' (#239) from dev into main
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 34s
CI / frontend-build (push) Successful in 23s
Build images / build-ml (push) Successful in 3m1s
extension / lint (push) Successful in 21s
CI / integration (push) Successful in 4m11s
Build images / build-web (push) Successful in 3m21s
Build images / build-agent (push) Successful in 9m23s
Build images / sign-extension (push) Successful in 4s
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 3s
CI / backend-lint-and-test (push) Successful in 34s
CI / frontend-build (push) Successful in 23s
Build images / build-ml (push) Successful in 3m1s
extension / lint (push) Successful in 21s
CI / integration (push) Successful in 4m11s
Build images / build-web (push) Successful in 3m21s
Build images / build-agent (push) Successful in 9m23s
This commit was merged in pull request #239.
This commit is contained in:
+352
-253
@@ -14,13 +14,16 @@ on:
|
|||||||
# with a stale `:dev` ml or agent is a worse trap than no dev channel at
|
# with a stale `:dev` ml or agent is a worse trap than no dev channel at
|
||||||
# all, since the mismatch only shows up as a runtime failure.
|
# all, since the mismatch only shows up as a runtime failure.
|
||||||
branches: [main, dev]
|
branches: [main, dev]
|
||||||
# Tag-push triggers an immutable per-version image build (e.g.
|
#
|
||||||
# `:v26.05.26.5`) — gives a real rollback story alongside the floating
|
# NO tag trigger (milestone 318 step 2). A `v*` tag names a commit `main`
|
||||||
# `:main` / `:latest`. Layer reuse keeps the registry-storage cost
|
# already built and published; rebuilding it produces the same source under
|
||||||
# negligible per tag. Doesn't overlap with the push-to-main build (that
|
# the same names and RE-PUSHES `:c-<sha>`, which rule 145 forbids even when
|
||||||
# one publishes `:main` + `:latest`; the tag-push build publishes only
|
# the bytes match — image configs carry timestamps, so "same source" does
|
||||||
# `:<tag>`).
|
# not mean "same manifest". The release build was publishing nothing new
|
||||||
tags: ['v*']
|
# and violating an immutability rule to do it.
|
||||||
|
#
|
||||||
|
# Releases still happen (rule 148, on explicit request per rule 2). They
|
||||||
|
# produce a changelog, not an image.
|
||||||
|
|
||||||
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
|
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
|
||||||
# - write:package, read:package (for docker push to git.fabledsword.com)
|
# - write:package, read:package (for docker push to git.fabledsword.com)
|
||||||
@@ -54,10 +57,11 @@ jobs:
|
|||||||
# have hit the existing ext-1.0.11 cache and bundled MAIN's stale XPI into
|
# have hit the existing ext-1.0.11 cache and bundled MAIN's stale XPI into
|
||||||
# `:dev` — a dev channel confidently serving old code.
|
# `:dev` — a dev channel confidently serving old code.
|
||||||
#
|
#
|
||||||
# Tags stay excluded: the tag path deliberately skips signing and polls for
|
# Unconditional since milestone 318 step 2: main and dev are now the only
|
||||||
# the release instead (see build-web's race note, 2026-05-27).
|
# triggers, so the branch gate that used to exclude tag pushes matched
|
||||||
|
# everything. A condition that is always true reads as if some path avoids
|
||||||
|
# it, which is worse than no condition.
|
||||||
sign-extension:
|
sign-extension:
|
||||||
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev'
|
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
@@ -119,10 +123,9 @@ jobs:
|
|||||||
run: |
|
run: |
|
||||||
set -u
|
set -u
|
||||||
A=extension
|
A=extension
|
||||||
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
|
||||||
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
echo "derived: artifact=$A version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
- name: Guard — the derived version must never go backwards
|
- name: Guard — the derived version must never go backwards
|
||||||
env:
|
env:
|
||||||
@@ -323,12 +326,12 @@ jobs:
|
|||||||
# to. Same source of truth; no double-store.
|
# to. Same source of truth; no double-store.
|
||||||
|
|
||||||
build-web:
|
build-web:
|
||||||
|
# A plain `needs` — no `always()`. That expression existed to let a
|
||||||
|
# SKIPPED sign-extension through on a tag push while still blocking a
|
||||||
|
# FAILED one. With no tag trigger, sign-extension always runs, so the
|
||||||
|
# default behaviour is exactly what we want: a failed sign skips build-web
|
||||||
|
# rather than shipping an image without its XPI.
|
||||||
needs: [sign-extension]
|
needs: [sign-extension]
|
||||||
# sign-extension runs on main and dev, and is skipped on a tag push (which
|
|
||||||
# polls for the release instead). Either is fine to build on; a FAILED sign
|
|
||||||
# is not — this condition lets success and skipped through, so a failure
|
|
||||||
# skips build-web rather than shipping an image without the XPI.
|
|
||||||
if: always() && (needs.sign-extension.result == 'success' || needs.sign-extension.result == 'skipped')
|
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
container:
|
container:
|
||||||
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
||||||
@@ -365,71 +368,44 @@ jobs:
|
|||||||
run: |
|
run: |
|
||||||
set -u
|
set -u
|
||||||
A=web
|
A=web
|
||||||
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
|
||||||
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
echo "derived: artifact=$A version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
- name: Determine tag
|
- name: Determine tag
|
||||||
id: tag
|
id: tag
|
||||||
run: |
|
run: |
|
||||||
# Three trigger shapes:
|
# Two trigger shapes, and between them they publish three tags:
|
||||||
# refs/tags/v… → tag-push: opt-in milestone label (vYY.MM.DD,
|
# main → :latest (production, moving — rule 147: main IS production)
|
||||||
# plus `.N` when the day already carries a tag —
|
# :c-<sha> (immutable, the rollback unit — rule 145)
|
||||||
# family rule 148, amended 2026-08-24 after a
|
# dev → :dev (the rolling test channel — rule 146)
|
||||||
# same-day tag was retargeted and a release
|
#
|
||||||
# deleted to make room, note 2813).
|
# That is the whole list. No :<version>, and no :main — rule 145,
|
||||||
# Publish ONLY the immutable version tag;
|
# narrowed 2026-08-28 once it was verified that nothing pins:
|
||||||
# don't touch :latest (the main-push build
|
# "a third name for the same thing is upkeep for a model we do not
|
||||||
# for the merge commit already did that).
|
# run." The date tag published between milestone 313 step 3 and
|
||||||
# refs/heads/main → push to main: publish :main + :latest
|
# milestone 318 was exactly that; :main was a second moving name for
|
||||||
# (floating) AND :c-<short_sha> (immutable
|
# whatever :latest already pointed at.
|
||||||
# per-commit rollback substrate, per family
|
#
|
||||||
# release-posture rule "Tags are milestones,
|
# `dev` gets no :c-<sha> deliberately. On a channel whose entire
|
||||||
# not gates — commit-SHA images are the
|
# contract is that it moves, a per-push immutable tag is a rollback
|
||||||
# rollback unit"). Rollback to any commit
|
# target nobody has ever pulled, accumulating forever. The accepted
|
||||||
# becomes `docker pull …:c-<sha>` without a
|
# cost: on dev there is no rollback but the previous :dev, which is
|
||||||
# release ceremony.
|
# gone — recovery is revert-on-git plus a CI cycle.
|
||||||
# refs/heads/dev → push to dev: publish :dev, the rolling test
|
#
|
||||||
# channel (family rule 146). Rolling means it may
|
# Reinstating :<version> is a real decision, not a default. It earns
|
||||||
# carry newer contents than the :c-<sha> of the
|
# its place when something genuinely pins: a second instance held on
|
||||||
# same commit; it never writes :c-<sha> itself,
|
# a known-good build, or a deliberately frozen window. Tag at the
|
||||||
# because that is the rollback unit (rule 145).
|
# moment you decide to freeze; no back-catalogue is needed.
|
||||||
|
#
|
||||||
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
||||||
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
||||||
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
# everywhere). Operator-flagged 2026-06-01 after the first :c-<sha>
|
||||||
# main-push build failed at this step.
|
# main-push build failed at this step.
|
||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
|
# Mirrors build-web's tag list; see the comment there.
|
||||||
# THIS artifact's shipped files last changed in. Day precision is
|
if [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
# deliberate — same-day work is not something worth pinning, so a
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
||||||
# second main build the same day replaces the first rather than
|
|
||||||
# accumulating a tag nobody would roll back to.
|
|
||||||
#
|
|
||||||
# Derived per artifact, so an image whose files did not change keeps
|
|
||||||
# the tag it already had: the agent reads 2026.7.17 today while web
|
|
||||||
# reads 2026.8.27 — and the reuse step below turns that into a
|
|
||||||
# skipped build rather than a rebuild of bytes that already exist.
|
|
||||||
# `channel` is baked into the image as FC_CHANNEL and reported by
|
|
||||||
# /api/extension/manifest (milestone 271 step 7). A tag-push counts as
|
|
||||||
# `main`: a vYY.MM.DD tag is cut from main, so that image is a
|
|
||||||
# main-channel artifact wearing an immutable name.
|
|
||||||
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
|
||||||
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
|
||||||
CALVER=$(sh scripts/artifacts.sh tag web)
|
|
||||||
# Guarded, and computed only on this path. There is no `set -e` in
|
|
||||||
# this step, so a failed derivation would otherwise leave CALVER
|
|
||||||
# empty and publish the tag `fabledcurator:` — an invalid
|
|
||||||
# name, from a green step. An empty pin must never reach the
|
|
||||||
# registry.
|
|
||||||
if [ -z "$CALVER" ]; then
|
|
||||||
echo "ERROR: could not derive a web version tag" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:main,git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator:${CALVER}" >> "$GITHUB_OUTPUT"
|
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
else
|
else
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
|
||||||
@@ -462,12 +438,21 @@ jobs:
|
|||||||
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
||||||
|
|
||||||
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
||||||
# The identity tag names this artifact's CONTENT — r-<revision>, the
|
# Does the image the channel tag already points at carry THIS commit's
|
||||||
# commit its shipped files last changed in, plus the channel for images
|
# revision? If so the bytes this job would produce are already published
|
||||||
# that bake one in. If the registry already carries it, the bytes this
|
# and the build is pure waste: the remaining tags get repointed at that
|
||||||
# job would produce are already published and the build is pure waste:
|
# existing manifest instead, registry-side, in seconds.
|
||||||
# the channel and date tags get repointed at the existing manifest
|
#
|
||||||
# instead, registry-side, in seconds.
|
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
|
||||||
|
# (milestone 318 step 3). A tag would be a name minted per build that one
|
||||||
|
# thing reads — what rule 145 narrowed against — and would be prunable
|
||||||
|
# under the registry's keep_pattern (#3157), silently expiring the cache.
|
||||||
|
# A label rides inside a tag that has to exist anyway.
|
||||||
|
#
|
||||||
|
# An image with no such label reads as a miss and rebuilds. That is the
|
||||||
|
# migration, not a fault: labels cannot be backfilled, since the reuse
|
||||||
|
# path copies a manifest and config labels are not manifest annotations.
|
||||||
|
# Each artifact pays one rebuild, once.
|
||||||
#
|
#
|
||||||
# This is what stops a push that touched only `agent/` from rebuilding
|
# This is what stops a push that touched only `agent/` from rebuilding
|
||||||
# web and ml, and a merge to main from rebuilding what dev already built.
|
# web and ml, and a merge to main from rebuilding what dev already built.
|
||||||
@@ -493,54 +478,71 @@ jobs:
|
|||||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
||||||
CHANNEL: ${{ steps.tag.outputs.channel }}
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
|
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
ID=$(sh scripts/artifacts.sh identity web "$CHANNEL")
|
DERIVED=$(sh scripts/artifacts.sh revision web)
|
||||||
echo "identity=$ID" >> "$GITHUB_OUTPUT"
|
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# A tag-push builds a revision that main already published, so it
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||||
# must NOT claim the identity: image configs are not bit-reproducible
|
# that is why the revision needs no -main/-dev qualifier any more.
|
||||||
# (embedded timestamps), so re-pushing r-<rev> would point an
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||||
# immutable tag at fresh bytes — rule 145's exact prohibition. It
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
# publishes only its own :v… label and otherwise reuses.
|
|
||||||
if [ "$IS_TAG_PUSH" = "true" ]; then
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||||
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||||
else
|
# branching on the exit code would read "no label yet" as success.
|
||||||
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
|
# An unreachable tag also lands here as empty via the `|| echo`.
|
||||||
|
# Empty never equals a 12-char revision, so every uncertain case
|
||||||
|
# falls through to a build — the safe direction, with no special
|
||||||
|
# casing for it.
|
||||||
|
#
|
||||||
|
# Read the SPECIFIC key. The map also carries whatever the base image
|
||||||
|
# set, and `org.opencontainers.image.version` sits right beside ours
|
||||||
|
# looking like a plausible answer (it reads 24.04 on the agent).
|
||||||
|
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
|
||||||
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
||||||
|
2>/dev/null || echo "")
|
||||||
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
||||||
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
||||||
|
# The tag resolves but carries no readable label. Expected exactly
|
||||||
|
# once per artifact, during the migration onto labels. If it recurs
|
||||||
|
# every push, something is rewriting the channel tag as a manifest
|
||||||
|
# index — see the repoint step's note.
|
||||||
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
||||||
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
||||||
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
|
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
echo "hit=true" >> "$GITHUB_OUTPUT"
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: $IMAGE:$ID is already published — skipping the build"
|
echo "reuse: already published — skipping the build"
|
||||||
else
|
else
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: $IMAGE:$ID is not published — building"
|
echo "reuse: not published — building"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
- name: Download signed XPI from Forgejo release asset
|
- name: Download signed XPI from Forgejo release asset
|
||||||
# Fires on every trigger shape. dev and main each bundle the XPI their
|
# dev and main each bundle the XPI their own sign-extension just
|
||||||
# own sign-extension just published — that is the whole point of the
|
# published — the point of the channel work (milestone 271 step 6): the
|
||||||
# channel work (milestone 271 step 6): the dev image carries the
|
# dev image carries the extension being developed, rather than
|
||||||
# extension being developed, rather than requiring a merge to try it.
|
# requiring a merge to try it.
|
||||||
# Tag-push builds re-package the same source as the preceding main-push
|
|
||||||
# build but with an immutable version tag — they need the XPI too,
|
|
||||||
# otherwise the versioned image ships without the signed extension.
|
|
||||||
#
|
#
|
||||||
# Tag-push vs main-push race (operator-flagged 2026-05-27 after
|
# The 10-minute polling loop that used to live here is gone with the
|
||||||
# v26.05.27.0 hit it): a release cut fires BOTH workflows almost
|
# tag trigger (milestone 318 step 2). It existed for one shape only: a
|
||||||
# simultaneously. Main-push runs sign-extension (1-5min AMO round
|
# release cut fired the tag build and the main build together, the tag
|
||||||
# trip) before publishing the ext-<version> release; tag-push
|
# build skipped sign-extension and raced straight here, and it lost
|
||||||
# skips sign-extension (gated to main) and races straight to
|
# every time (operator-flagged 2026-05-27 after v26.05.27.0). Polling
|
||||||
# this download step. Tag-push lost every time. Fix: poll the
|
# was the fix for a build that should not have been running.
|
||||||
# ext-<version> release endpoint with a sleep+retry loop (30s
|
#
|
||||||
# for up to 10min total) before giving up. Main-push's signing
|
# sign-extension is a `needs` dependency and it succeeded, so the
|
||||||
# eventually wins and tag-push picks the release up on a later
|
# release exists. A single fetch is correct, and a 404 now means a real
|
||||||
# iteration.
|
# disagreement about the derived version rather than a race — which is
|
||||||
# Gated on the reuse miss as well: if the image is already published it
|
# exactly what should fail loudly instead of being slept through.
|
||||||
# already contains its XPI, so this would download (and on a tag-push,
|
#
|
||||||
# poll up to 10 minutes for) a file nothing then reads.
|
# Still gated on the reuse miss: a published image already contains its
|
||||||
if: steps.reuse.outputs.hit != 'true' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/'))
|
# XPI, so this would fetch a file nothing then reads.
|
||||||
|
if: steps.reuse.outputs.hit != 'true'
|
||||||
env:
|
env:
|
||||||
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
run: |
|
run: |
|
||||||
@@ -552,25 +554,21 @@ jobs:
|
|||||||
# didn't, this download 404s and the build fails loudly instead of
|
# didn't, this download 404s and the build fails loudly instead of
|
||||||
# shipping a stale XPI.
|
# shipping a stale XPI.
|
||||||
VERSION=$(sh extension/scripts/packaging.sh version)
|
VERSION=$(sh extension/scripts/packaging.sh version)
|
||||||
# Poll for the ext-<version> release. main-push's sign-extension
|
# One fetch, no retry. sign-extension ran to success in this same
|
||||||
# step (AMO round-trip, 1-5min) needs to finish + upload before
|
# workflow and published ext-$VERSION; both jobs derive $VERSION from
|
||||||
# tag-push can fetch. 30s * 20 = up to 10min wait, then hard-fail.
|
# the same commit, so they agree by construction. A 404 here means
|
||||||
for attempt in $(seq 1 20); do
|
# they did NOT agree, and sleeping on that would only delay the
|
||||||
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
# report.
|
||||||
-H "Authorization: token $TOKEN" \
|
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
||||||
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
-H "Authorization: token $TOKEN" \
|
||||||
if [ "$STATUS" = "200" ]; then
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
||||||
echo "Found ext-$VERSION release on attempt $attempt"
|
if [ "$STATUS" != "200" ]; then
|
||||||
break
|
echo "ERROR: ext-$VERSION release not found (HTTP $STATUS)."
|
||||||
fi
|
echo "sign-extension succeeded in this run, so it published some"
|
||||||
if [ "$attempt" = "20" ]; then
|
echo "other version — the two jobs derived different values for one"
|
||||||
echo "ERROR: ext-$VERSION release not available after 10min of polling"
|
echo "commit. Check that both checked out with fetch-depth: 0."
|
||||||
echo "Last HTTP status: $STATUS"
|
exit 1
|
||||||
exit 1
|
fi
|
||||||
fi
|
|
||||||
echo "Attempt $attempt: ext-$VERSION not yet published (HTTP $STATUS); sleeping 30s"
|
|
||||||
sleep 30
|
|
||||||
done
|
|
||||||
# Extract the .xpi asset's browser_download_url (Forgejo's
|
# Extract the .xpi asset's browser_download_url (Forgejo's
|
||||||
# /releases/assets/<id> endpoint returns ASSET METADATA, not
|
# /releases/assets/<id> endpoint returns ASSET METADATA, not
|
||||||
# the binary blob — operator-flagged 2026-05-26: my prior
|
# the binary blob — operator-flagged 2026-05-26: my prior
|
||||||
@@ -608,6 +606,11 @@ jobs:
|
|||||||
file: Dockerfile
|
file: Dockerfile
|
||||||
push: true
|
push: true
|
||||||
tags: ${{ steps.reuse.outputs.build_tags }}
|
tags: ${{ steps.reuse.outputs.build_tags }}
|
||||||
|
# The reuse key. Read back off the channel tag on the next push to
|
||||||
|
# decide whether that push needs to build at all, so this is not
|
||||||
|
# decoration — an unstamped image is one that will always rebuild.
|
||||||
|
labels: |
|
||||||
|
fc.revision=${{ steps.reuse.outputs.revision }}
|
||||||
# Only the web image carries a channel: it is the one that serves
|
# Only the web image carries a channel: it is the one that serves
|
||||||
# /api/extension/manifest. The ml and agent images have nothing to
|
# /api/extension/manifest. The ml and agent images have nothing to
|
||||||
# report it to.
|
# report it to.
|
||||||
@@ -616,8 +619,8 @@ jobs:
|
|||||||
|
|
||||||
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
||||||
# rebuild. Each -t becomes another reference to the SAME manifest the
|
# rebuild. Each -t becomes another reference to the SAME manifest the
|
||||||
# identity tag holds, so :latest and the date pin are byte-identical to
|
# channel tag already holds, so :c-<sha> and the date pin are
|
||||||
# what was published rather than a lookalike rebuild.
|
# byte-identical to what is published rather than a lookalike rebuild.
|
||||||
#
|
#
|
||||||
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
||||||
# rolling channel refreshes itself, so skipping a build must never mean
|
# rolling channel refreshes itself, so skipping a build must never mean
|
||||||
@@ -627,19 +630,46 @@ jobs:
|
|||||||
if: steps.reuse.outputs.hit == 'true'
|
if: steps.reuse.outputs.hit == 'true'
|
||||||
env:
|
env:
|
||||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
||||||
IDENTITY: ${{ steps.reuse.outputs.identity }}
|
SOURCE: ${{ steps.reuse.outputs.channel_ref }}
|
||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
run: |
|
run: |
|
||||||
set -euf
|
set -euf
|
||||||
|
# The source tag is EXCLUDED from the targets, and that is load-
|
||||||
|
# bearing rather than an optimisation.
|
||||||
|
#
|
||||||
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
||||||
|
# at the channel tag with that same tag as a target and the tag stops
|
||||||
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
||||||
|
# resolves through it and the fc.revision label reads as absent. The
|
||||||
|
# next push then misses and rebuilds, so reuse worked exactly once
|
||||||
|
# and every subsequent push paid full price. Observed on run 4751:
|
||||||
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
||||||
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
||||||
|
#
|
||||||
|
# Excluding the source means the channel tag is only ever written by
|
||||||
|
# a real build, so it stays a plain image and stays readable. On dev
|
||||||
|
# that leaves nothing to do — :dev already points at the right
|
||||||
|
# content, which is what the hit established. On main it leaves
|
||||||
|
# :c-<sha>, which rule 145 requires of every main push whether or not
|
||||||
|
# a build ran.
|
||||||
|
#
|
||||||
# steps.tag emits ONE comma-separated list, because that is the shape
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
# docker/build-push-action takes; imagetools wants a -t per ref.
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
ARGS=""
|
ARGS=""
|
||||||
IFS=,
|
IFS=,
|
||||||
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
for t in $TAGS; do
|
||||||
|
[ "$t" = "$SOURCE" ] && continue
|
||||||
|
ARGS="$ARGS -t $t"
|
||||||
|
done
|
||||||
unset IFS
|
unset IFS
|
||||||
|
if [ -z "$ARGS" ]; then
|
||||||
|
echo "repoint: $SOURCE already carries this revision and is the"
|
||||||
|
echo "repoint: only tag for this channel — nothing to write."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
|
docker buildx imagetools create $ARGS "$SOURCE"
|
||||||
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
|
echo "repointed from $SOURCE:$ARGS"
|
||||||
|
|
||||||
build-ml:
|
build-ml:
|
||||||
runs-on: python-ci
|
runs-on: python-ci
|
||||||
@@ -679,49 +709,22 @@ jobs:
|
|||||||
run: |
|
run: |
|
||||||
set -u
|
set -u
|
||||||
A=ml
|
A=ml
|
||||||
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
|
||||||
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
echo "derived: artifact=$A version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
- name: Determine tag
|
- name: Determine tag
|
||||||
id: tag
|
id: tag
|
||||||
run: |
|
run: |
|
||||||
# Mirrors build-web's three-shape logic (tag-push / main-push /
|
# Mirrors build-web's tag list; see the comment there.
|
||||||
# safety-net dev) including the per-commit :c-<short_sha> tag
|
|
||||||
# on main-push per the family release-posture rule. The -ml
|
|
||||||
# image follows the same release cadence as the web image.
|
|
||||||
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
||||||
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
||||||
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
|
||||||
# main-push build failed at this step.
|
# main-push build failed at this step.
|
||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
|
# Mirrors build-web's tag list; see the comment there.
|
||||||
# THIS artifact's shipped files last changed in. Day precision is
|
if [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
# deliberate — same-day work is not something worth pinning, so a
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
||||||
# second main build the same day replaces the first rather than
|
|
||||||
# accumulating a tag nobody would roll back to.
|
|
||||||
#
|
|
||||||
# Derived per artifact, so an image whose files did not change keeps
|
|
||||||
# the tag it already had: the agent reads 2026.7.17 today while web
|
|
||||||
# reads 2026.8.27 — and the reuse step below turns that into a
|
|
||||||
# skipped build rather than a rebuild of bytes that already exist.
|
|
||||||
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
|
||||||
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
|
||||||
CALVER=$(sh scripts/artifacts.sh tag ml)
|
|
||||||
# Guarded, and computed only on this path. There is no `set -e` in
|
|
||||||
# this step, so a failed derivation would otherwise leave CALVER
|
|
||||||
# empty and publish the tag `fabledcurator-ml:` — an invalid
|
|
||||||
# name, from a green step. An empty pin must never reach the
|
|
||||||
# registry.
|
|
||||||
if [ -z "$CALVER" ]; then
|
|
||||||
echo "ERROR: could not derive a ml version tag" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:main,git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator-ml:${CALVER}" >> "$GITHUB_OUTPUT"
|
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
else
|
else
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:dev" >> "$GITHUB_OUTPUT"
|
||||||
@@ -737,12 +740,21 @@ jobs:
|
|||||||
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
||||||
|
|
||||||
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
||||||
# The identity tag names this artifact's CONTENT — r-<revision>, the
|
# Does the image the channel tag already points at carry THIS commit's
|
||||||
# commit its shipped files last changed in, plus the channel for images
|
# revision? If so the bytes this job would produce are already published
|
||||||
# that bake one in. If the registry already carries it, the bytes this
|
# and the build is pure waste: the remaining tags get repointed at that
|
||||||
# job would produce are already published and the build is pure waste:
|
# existing manifest instead, registry-side, in seconds.
|
||||||
# the channel and date tags get repointed at the existing manifest
|
#
|
||||||
# instead, registry-side, in seconds.
|
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
|
||||||
|
# (milestone 318 step 3). A tag would be a name minted per build that one
|
||||||
|
# thing reads — what rule 145 narrowed against — and would be prunable
|
||||||
|
# under the registry's keep_pattern (#3157), silently expiring the cache.
|
||||||
|
# A label rides inside a tag that has to exist anyway.
|
||||||
|
#
|
||||||
|
# An image with no such label reads as a miss and rebuilds. That is the
|
||||||
|
# migration, not a fault: labels cannot be backfilled, since the reuse
|
||||||
|
# path copies a manifest and config labels are not manifest annotations.
|
||||||
|
# Each artifact pays one rebuild, once.
|
||||||
#
|
#
|
||||||
# This is what stops a push that touched only `agent/` from rebuilding
|
# This is what stops a push that touched only `agent/` from rebuilding
|
||||||
# web and ml, and a merge to main from rebuilding what dev already built.
|
# web and ml, and a merge to main from rebuilding what dev already built.
|
||||||
@@ -768,29 +780,48 @@ jobs:
|
|||||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
||||||
CHANNEL: ${{ steps.tag.outputs.channel }}
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
|
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
ID=$(sh scripts/artifacts.sh identity ml "$CHANNEL")
|
DERIVED=$(sh scripts/artifacts.sh revision ml)
|
||||||
echo "identity=$ID" >> "$GITHUB_OUTPUT"
|
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# A tag-push builds a revision that main already published, so it
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||||
# must NOT claim the identity: image configs are not bit-reproducible
|
# that is why the revision needs no -main/-dev qualifier any more.
|
||||||
# (embedded timestamps), so re-pushing r-<rev> would point an
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||||
# immutable tag at fresh bytes — rule 145's exact prohibition. It
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
# publishes only its own :v… label and otherwise reuses.
|
|
||||||
if [ "$IS_TAG_PUSH" = "true" ]; then
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||||
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||||
else
|
# branching on the exit code would read "no label yet" as success.
|
||||||
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
|
# An unreachable tag also lands here as empty via the `|| echo`.
|
||||||
|
# Empty never equals a 12-char revision, so every uncertain case
|
||||||
|
# falls through to a build — the safe direction, with no special
|
||||||
|
# casing for it.
|
||||||
|
#
|
||||||
|
# Read the SPECIFIC key. The map also carries whatever the base image
|
||||||
|
# set, and `org.opencontainers.image.version` sits right beside ours
|
||||||
|
# looking like a plausible answer (it reads 24.04 on the agent).
|
||||||
|
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
|
||||||
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
||||||
|
2>/dev/null || echo "")
|
||||||
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
||||||
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
||||||
|
# The tag resolves but carries no readable label. Expected exactly
|
||||||
|
# once per artifact, during the migration onto labels. If it recurs
|
||||||
|
# every push, something is rewriting the channel tag as a manifest
|
||||||
|
# index — see the repoint step's note.
|
||||||
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
||||||
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
||||||
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
|
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
echo "hit=true" >> "$GITHUB_OUTPUT"
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: $IMAGE:$ID is already published — skipping the build"
|
echo "reuse: already published — skipping the build"
|
||||||
else
|
else
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: $IMAGE:$ID is not published — building"
|
echo "reuse: not published — building"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
- name: Build and push ml image
|
- name: Build and push ml image
|
||||||
@@ -801,11 +832,16 @@ jobs:
|
|||||||
file: Dockerfile.ml
|
file: Dockerfile.ml
|
||||||
push: true
|
push: true
|
||||||
tags: ${{ steps.reuse.outputs.build_tags }}
|
tags: ${{ steps.reuse.outputs.build_tags }}
|
||||||
|
# The reuse key. Read back off the channel tag on the next push to
|
||||||
|
# decide whether that push needs to build at all, so this is not
|
||||||
|
# decoration — an unstamped image is one that will always rebuild.
|
||||||
|
labels: |
|
||||||
|
fc.revision=${{ steps.reuse.outputs.revision }}
|
||||||
|
|
||||||
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
||||||
# rebuild. Each -t becomes another reference to the SAME manifest the
|
# rebuild. Each -t becomes another reference to the SAME manifest the
|
||||||
# identity tag holds, so :latest and the date pin are byte-identical to
|
# channel tag already holds, so :c-<sha> and the date pin are
|
||||||
# what was published rather than a lookalike rebuild.
|
# byte-identical to what is published rather than a lookalike rebuild.
|
||||||
#
|
#
|
||||||
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
||||||
# rolling channel refreshes itself, so skipping a build must never mean
|
# rolling channel refreshes itself, so skipping a build must never mean
|
||||||
@@ -815,19 +851,46 @@ jobs:
|
|||||||
if: steps.reuse.outputs.hit == 'true'
|
if: steps.reuse.outputs.hit == 'true'
|
||||||
env:
|
env:
|
||||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
||||||
IDENTITY: ${{ steps.reuse.outputs.identity }}
|
SOURCE: ${{ steps.reuse.outputs.channel_ref }}
|
||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
run: |
|
run: |
|
||||||
set -euf
|
set -euf
|
||||||
|
# The source tag is EXCLUDED from the targets, and that is load-
|
||||||
|
# bearing rather than an optimisation.
|
||||||
|
#
|
||||||
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
||||||
|
# at the channel tag with that same tag as a target and the tag stops
|
||||||
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
||||||
|
# resolves through it and the fc.revision label reads as absent. The
|
||||||
|
# next push then misses and rebuilds, so reuse worked exactly once
|
||||||
|
# and every subsequent push paid full price. Observed on run 4751:
|
||||||
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
||||||
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
||||||
|
#
|
||||||
|
# Excluding the source means the channel tag is only ever written by
|
||||||
|
# a real build, so it stays a plain image and stays readable. On dev
|
||||||
|
# that leaves nothing to do — :dev already points at the right
|
||||||
|
# content, which is what the hit established. On main it leaves
|
||||||
|
# :c-<sha>, which rule 145 requires of every main push whether or not
|
||||||
|
# a build ran.
|
||||||
|
#
|
||||||
# steps.tag emits ONE comma-separated list, because that is the shape
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
# docker/build-push-action takes; imagetools wants a -t per ref.
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
ARGS=""
|
ARGS=""
|
||||||
IFS=,
|
IFS=,
|
||||||
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
for t in $TAGS; do
|
||||||
|
[ "$t" = "$SOURCE" ] && continue
|
||||||
|
ARGS="$ARGS -t $t"
|
||||||
|
done
|
||||||
unset IFS
|
unset IFS
|
||||||
|
if [ -z "$ARGS" ]; then
|
||||||
|
echo "repoint: $SOURCE already carries this revision and is the"
|
||||||
|
echo "repoint: only tag for this channel — nothing to write."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
|
docker buildx imagetools create $ARGS "$SOURCE"
|
||||||
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
|
echo "repointed from $SOURCE:$ARGS"
|
||||||
|
|
||||||
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
|
# The desktop GPU agent (#114) — published so the operator pulls + runs it on
|
||||||
# the GPU machine instead of building locally. Independent of web/ml (its own
|
# the GPU machine instead of building locally. Independent of web/ml (its own
|
||||||
@@ -870,41 +933,17 @@ jobs:
|
|||||||
run: |
|
run: |
|
||||||
set -u
|
set -u
|
||||||
A=agent
|
A=agent
|
||||||
T=$(sh scripts/artifacts.sh tag "$A" 2>&1 || echo UNAVAILABLE)
|
|
||||||
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
||||||
echo "derived: artifact=$A tag=$T version=$V revision=$R sha=$GITHUB_SHA"
|
echo "derived: artifact=$A version=$V revision=$R sha=$GITHUB_SHA"
|
||||||
|
|
||||||
- name: Determine tag
|
- name: Determine tag
|
||||||
id: tag
|
id: tag
|
||||||
run: |
|
run: |
|
||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
# The pinnable tag (milestone 313 step 3): YYYY.M.D of the commit
|
# Mirrors build-web's tag list; see the comment there.
|
||||||
# THIS artifact's shipped files last changed in. Day precision is
|
if [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
# deliberate — same-day work is not something worth pinning, so a
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
||||||
# second main build the same day replaces the first rather than
|
|
||||||
# accumulating a tag nobody would roll back to.
|
|
||||||
#
|
|
||||||
# Derived per artifact, so an image whose files did not change keeps
|
|
||||||
# the tag it already had: the agent reads 2026.7.17 today while web
|
|
||||||
# reads 2026.8.27 — and the reuse step below turns that into a
|
|
||||||
# skipped build rather than a rebuild of bytes that already exist.
|
|
||||||
if [ "${GITHUB_REF#refs/tags/}" != "${GITHUB_REF}" ]; then
|
|
||||||
TAG_NAME="${GITHUB_REF#refs/tags/}"
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:${TAG_NAME}" >> "$GITHUB_OUTPUT"
|
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
|
||||||
CALVER=$(sh scripts/artifacts.sh tag agent)
|
|
||||||
# Guarded, and computed only on this path. There is no `set -e` in
|
|
||||||
# this step, so a failed derivation would otherwise leave CALVER
|
|
||||||
# empty and publish the tag `fabledcurator-agent:` — an invalid
|
|
||||||
# name, from a green step. An empty pin must never reach the
|
|
||||||
# registry.
|
|
||||||
if [ -z "$CALVER" ]; then
|
|
||||||
echo "ERROR: could not derive a agent version tag" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:main,git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA},git.fabledsword.com/bvandeusen/fabledcurator-agent:${CALVER}" >> "$GITHUB_OUTPUT"
|
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
else
|
else
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
|
||||||
@@ -920,12 +959,21 @@ jobs:
|
|||||||
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
||||||
|
|
||||||
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
||||||
# The identity tag names this artifact's CONTENT — r-<revision>, the
|
# Does the image the channel tag already points at carry THIS commit's
|
||||||
# commit its shipped files last changed in, plus the channel for images
|
# revision? If so the bytes this job would produce are already published
|
||||||
# that bake one in. If the registry already carries it, the bytes this
|
# and the build is pure waste: the remaining tags get repointed at that
|
||||||
# job would produce are already published and the build is pure waste:
|
# existing manifest instead, registry-side, in seconds.
|
||||||
# the channel and date tags get repointed at the existing manifest
|
#
|
||||||
# instead, registry-side, in seconds.
|
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
|
||||||
|
# (milestone 318 step 3). A tag would be a name minted per build that one
|
||||||
|
# thing reads — what rule 145 narrowed against — and would be prunable
|
||||||
|
# under the registry's keep_pattern (#3157), silently expiring the cache.
|
||||||
|
# A label rides inside a tag that has to exist anyway.
|
||||||
|
#
|
||||||
|
# An image with no such label reads as a miss and rebuilds. That is the
|
||||||
|
# migration, not a fault: labels cannot be backfilled, since the reuse
|
||||||
|
# path copies a manifest and config labels are not manifest annotations.
|
||||||
|
# Each artifact pays one rebuild, once.
|
||||||
#
|
#
|
||||||
# This is what stops a push that touched only `agent/` from rebuilding
|
# This is what stops a push that touched only `agent/` from rebuilding
|
||||||
# web and ml, and a merge to main from rebuilding what dev already built.
|
# web and ml, and a merge to main from rebuilding what dev already built.
|
||||||
@@ -951,29 +999,48 @@ jobs:
|
|||||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
||||||
CHANNEL: ${{ steps.tag.outputs.channel }}
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
IS_TAG_PUSH: ${{ startsWith(github.ref, 'refs/tags/') }}
|
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
ID=$(sh scripts/artifacts.sh identity agent "$CHANNEL")
|
DERIVED=$(sh scripts/artifacts.sh revision agent)
|
||||||
echo "identity=$ID" >> "$GITHUB_OUTPUT"
|
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# A tag-push builds a revision that main already published, so it
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||||
# must NOT claim the identity: image configs are not bit-reproducible
|
# that is why the revision needs no -main/-dev qualifier any more.
|
||||||
# (embedded timestamps), so re-pushing r-<rev> would point an
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||||
# immutable tag at fresh bytes — rule 145's exact prohibition. It
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
# publishes only its own :v… label and otherwise reuses.
|
|
||||||
if [ "$IS_TAG_PUSH" = "true" ]; then
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||||
echo "build_tags=$TAGS" >> "$GITHUB_OUTPUT"
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||||
else
|
# branching on the exit code would read "no label yet" as success.
|
||||||
echo "build_tags=$TAGS,$IMAGE:$ID" >> "$GITHUB_OUTPUT"
|
# An unreachable tag also lands here as empty via the `|| echo`.
|
||||||
|
# Empty never equals a 12-char revision, so every uncertain case
|
||||||
|
# falls through to a build — the safe direction, with no special
|
||||||
|
# casing for it.
|
||||||
|
#
|
||||||
|
# Read the SPECIFIC key. The map also carries whatever the base image
|
||||||
|
# set, and `org.opencontainers.image.version` sits right beside ours
|
||||||
|
# looking like a plausible answer (it reads 24.04 on the agent).
|
||||||
|
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
|
||||||
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
||||||
|
2>/dev/null || echo "")
|
||||||
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
||||||
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
||||||
|
# The tag resolves but carries no readable label. Expected exactly
|
||||||
|
# once per artifact, during the migration onto labels. If it recurs
|
||||||
|
# every push, something is rewriting the channel tag as a manifest
|
||||||
|
# index — see the repoint step's note.
|
||||||
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
||||||
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
||||||
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if docker buildx imagetools inspect "$IMAGE:$ID" >/dev/null 2>&1; then
|
if [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
echo "hit=true" >> "$GITHUB_OUTPUT"
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: $IMAGE:$ID is already published — skipping the build"
|
echo "reuse: already published — skipping the build"
|
||||||
else
|
else
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: $IMAGE:$ID is not published — building"
|
echo "reuse: not published — building"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
- name: Build and push agent image
|
- name: Build and push agent image
|
||||||
@@ -984,11 +1051,16 @@ jobs:
|
|||||||
file: agent/Dockerfile
|
file: agent/Dockerfile
|
||||||
push: true
|
push: true
|
||||||
tags: ${{ steps.reuse.outputs.build_tags }}
|
tags: ${{ steps.reuse.outputs.build_tags }}
|
||||||
|
# The reuse key. Read back off the channel tag on the next push to
|
||||||
|
# decide whether that push needs to build at all, so this is not
|
||||||
|
# decoration — an unstamped image is one that will always rebuild.
|
||||||
|
labels: |
|
||||||
|
fc.revision=${{ steps.reuse.outputs.revision }}
|
||||||
|
|
||||||
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
# Registry-side manifest copy: no layer transfer, no local daemon, no
|
||||||
# rebuild. Each -t becomes another reference to the SAME manifest the
|
# rebuild. Each -t becomes another reference to the SAME manifest the
|
||||||
# identity tag holds, so :latest and the date pin are byte-identical to
|
# channel tag already holds, so :c-<sha> and the date pin are
|
||||||
# what was published rather than a lookalike rebuild.
|
# byte-identical to what is published rather than a lookalike rebuild.
|
||||||
#
|
#
|
||||||
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
# Runs on EVERY reuse, which is what keeps family rule 146 true: a
|
||||||
# rolling channel refreshes itself, so skipping a build must never mean
|
# rolling channel refreshes itself, so skipping a build must never mean
|
||||||
@@ -998,16 +1070,43 @@ jobs:
|
|||||||
if: steps.reuse.outputs.hit == 'true'
|
if: steps.reuse.outputs.hit == 'true'
|
||||||
env:
|
env:
|
||||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
||||||
IDENTITY: ${{ steps.reuse.outputs.identity }}
|
SOURCE: ${{ steps.reuse.outputs.channel_ref }}
|
||||||
TAGS: ${{ steps.tag.outputs.tags }}
|
TAGS: ${{ steps.tag.outputs.tags }}
|
||||||
run: |
|
run: |
|
||||||
set -euf
|
set -euf
|
||||||
|
# The source tag is EXCLUDED from the targets, and that is load-
|
||||||
|
# bearing rather than an optimisation.
|
||||||
|
#
|
||||||
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
||||||
|
# at the channel tag with that same tag as a target and the tag stops
|
||||||
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
||||||
|
# resolves through it and the fc.revision label reads as absent. The
|
||||||
|
# next push then misses and rebuilds, so reuse worked exactly once
|
||||||
|
# and every subsequent push paid full price. Observed on run 4751:
|
||||||
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
||||||
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
||||||
|
#
|
||||||
|
# Excluding the source means the channel tag is only ever written by
|
||||||
|
# a real build, so it stays a plain image and stays readable. On dev
|
||||||
|
# that leaves nothing to do — :dev already points at the right
|
||||||
|
# content, which is what the hit established. On main it leaves
|
||||||
|
# :c-<sha>, which rule 145 requires of every main push whether or not
|
||||||
|
# a build ran.
|
||||||
|
#
|
||||||
# steps.tag emits ONE comma-separated list, because that is the shape
|
# steps.tag emits ONE comma-separated list, because that is the shape
|
||||||
# docker/build-push-action takes; imagetools wants a -t per ref.
|
# docker/build-push-action takes; imagetools wants a -t per ref.
|
||||||
ARGS=""
|
ARGS=""
|
||||||
IFS=,
|
IFS=,
|
||||||
for t in $TAGS; do ARGS="$ARGS -t $t"; done
|
for t in $TAGS; do
|
||||||
|
[ "$t" = "$SOURCE" ] && continue
|
||||||
|
ARGS="$ARGS -t $t"
|
||||||
|
done
|
||||||
unset IFS
|
unset IFS
|
||||||
|
if [ -z "$ARGS" ]; then
|
||||||
|
echo "repoint: $SOURCE already carries this revision and is the"
|
||||||
|
echo "repoint: only tag for this channel — nothing to write."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
docker buildx imagetools create $ARGS "$IMAGE:$IDENTITY"
|
docker buildx imagetools create $ARGS "$SOURCE"
|
||||||
echo "repointed to $IMAGE:$IDENTITY: $TAGS"
|
echo "repointed from $SOURCE:$ARGS"
|
||||||
|
|||||||
+13
-73
@@ -56,25 +56,9 @@ ML_PATHS='Dockerfile.ml requirements-ml.txt requirements.txt backend alembic ale
|
|||||||
# it: this is deliberately NOT `agent/`.
|
# it: this is deliberately NOT `agent/`.
|
||||||
AGENT_PATHS='agent/Dockerfile agent/requirements.txt agent/fc_agent'
|
AGENT_PATHS='agent/Dockerfile agent/requirements.txt agent/fc_agent'
|
||||||
|
|
||||||
# Which artifacts bake the BUILD CHANNEL into the image, and therefore cannot
|
|
||||||
# share a content identity across channels. The web image takes FC_CHANNEL as
|
|
||||||
# a build-arg and reports it from /api/extension/manifest (milestone 271 step
|
|
||||||
# 7), so `main` and `dev` builds of one revision are genuinely different
|
|
||||||
# images — reusing the dev one on main would ship an instance that names
|
|
||||||
# itself `dev` forever.
|
|
||||||
#
|
|
||||||
# ml and agent take no build-args at all: one revision, one image, and a merge
|
|
||||||
# to main can reuse exactly what dev already built. That is not a detail, it is
|
|
||||||
# most of what step 4 saves — merges would otherwise rebuild the agent's CUDA
|
|
||||||
# image to produce bytes that already exist.
|
|
||||||
#
|
|
||||||
# Extend this list if a second artifact ever gains a build-arg;
|
|
||||||
# tests/test_artifact_identity.py reads the Dockerfiles and fails if it drifts.
|
|
||||||
CHANNELLED='web'
|
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
echo "usage: artifacts.sh {paths|revision|version|tag} {web|ml|agent|extension}" >&2
|
echo "usage: artifacts.sh {paths|revision|version} {web|ml|agent|extension}" >&2
|
||||||
echo " artifacts.sh identity {web|ml|agent} [channel]" >&2
|
|
||||||
exit 2
|
exit 2
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -124,9 +108,18 @@ strip0() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
# The IDENTITY of an artifact's content: the commit its shipped files last
|
# The IDENTITY of an artifact's content: the commit its shipped files last
|
||||||
# changed in. This — not the tag — is what decides whether a build can be
|
# changed in. This is what decides whether a build can be skipped.
|
||||||
# skipped, because the published tag is only day-precise and two different
|
#
|
||||||
# builds can share it.
|
# It is published as the `fc.revision` LABEL on the image itself, and read
|
||||||
|
# back off the moving channel tag — not as a tag of its own (milestone 318
|
||||||
|
# step 3). A tag would be a name minted per build that only one thing reads,
|
||||||
|
# which is what rule 145 narrowed against; it would also be prunable under the
|
||||||
|
# registry's keep_pattern (#3157), so the cache would silently expire.
|
||||||
|
#
|
||||||
|
# A published image with no such label reads as a MISS and rebuilds. That is
|
||||||
|
# the migration path, not a fault: `imagetools create` copies a manifest and
|
||||||
|
# config labels are not manifest annotations, so the reuse path cannot stamp
|
||||||
|
# one and there is nothing to backfill. Each artifact pays one rebuild, once.
|
||||||
cmd_revision() {
|
cmd_revision() {
|
||||||
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
|
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
|
||||||
}
|
}
|
||||||
@@ -144,63 +137,10 @@ cmd_version() {
|
|||||||
"$(strip0 "$(fmt "$sha" %H%M)")"
|
"$(strip0 "$(fmt "$sha" %H%M)")"
|
||||||
}
|
}
|
||||||
|
|
||||||
# The PUBLISHED IMAGE TAG: day precision, YYYY.M.D. Deliberately coarser than
|
|
||||||
# the ordering key, per the operator 2026-08-28 — same-day work is not
|
|
||||||
# something worth pinning, so a second build the same day replaces the first
|
|
||||||
# rather than accumulating a tag nobody would roll back to. Safe only because
|
|
||||||
# skip decisions key on cmd_revision, never on this.
|
|
||||||
cmd_tag() {
|
|
||||||
sha=$(echo "$(newest "$1")" | cut -d' ' -f2)
|
|
||||||
printf '%s.%s.%s\n' \
|
|
||||||
"$(fmt "$sha" %Y)" \
|
|
||||||
"$(strip0 "$(fmt "$sha" %m)")" \
|
|
||||||
"$(strip0 "$(fmt "$sha" %d)")"
|
|
||||||
}
|
|
||||||
|
|
||||||
# The CONTENT IDENTITY of a published image: an immutable tag naming exactly
|
|
||||||
# what a build of this commit would produce. build.yml asks the registry for it
|
|
||||||
# and, on a hit, skips the build entirely and repoints the channel and date
|
|
||||||
# tags at the manifest that is already there (milestone 313 step 4).
|
|
||||||
#
|
|
||||||
# It is deliberately NOT either of the other two values:
|
|
||||||
# * the date tag is day-precise and last-one-wins, so two different builds
|
|
||||||
# share it — it cannot answer "is this content published?".
|
|
||||||
# * the commit sha moves on every push, so it would never hit, which is the
|
|
||||||
# redundant rebuild this exists to remove.
|
|
||||||
#
|
|
||||||
# The revision does both jobs: it is content-unique AND stable across pushes
|
|
||||||
# that did not touch the artifact.
|
|
||||||
cmd_identity() {
|
|
||||||
_art=$1
|
|
||||||
_chan=${2:-}
|
|
||||||
case "$_art" in
|
|
||||||
web|ml|agent) ;;
|
|
||||||
extension)
|
|
||||||
echo "artifacts.sh: the extension is cached as an ext-<version> Forgejo release, not an image tag — use \`version\`" >&2
|
|
||||||
exit 2 ;;
|
|
||||||
*) usage ;;
|
|
||||||
esac
|
|
||||||
for _c in $CHANNELLED; do
|
|
||||||
if [ "$_art" = "$_c" ]; then
|
|
||||||
# Refused rather than defaulted: an unqualified identity for a
|
|
||||||
# channelled artifact would let a dev image be reused as the main one.
|
|
||||||
if [ -z "$_chan" ]; then
|
|
||||||
echo "artifacts.sh: $_art bakes the channel into the image — identity needs one" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
printf 'r-%s-%s\n' "$(cmd_revision "$_art")" "$_chan"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
printf 'r-%s\n' "$(cmd_revision "$_art")"
|
|
||||||
}
|
|
||||||
|
|
||||||
[ $# -ge 2 ] || usage
|
[ $# -ge 2 ] || usage
|
||||||
case "$1" in
|
case "$1" in
|
||||||
paths) cmd_paths "$2" ;;
|
paths) cmd_paths "$2" ;;
|
||||||
revision) cmd_revision "$2" ;;
|
revision) cmd_revision "$2" ;;
|
||||||
version) cmd_version "$2" ;;
|
version) cmd_version "$2" ;;
|
||||||
tag) cmd_tag "$2" ;;
|
|
||||||
identity) cmd_identity "$2" "${3:-}" ;;
|
|
||||||
*) usage ;;
|
*) usage ;;
|
||||||
esac
|
esac
|
||||||
|
|||||||
+88
-119
@@ -1,20 +1,29 @@
|
|||||||
"""`artifacts.sh identity` is what decides whether a build gets skipped.
|
"""`artifacts.sh revision` is what decides whether a build gets skipped.
|
||||||
|
|
||||||
Milestone 313 step 4: build.yml asks the registry for `<image>:<identity>` and,
|
Milestone 318 step 3: each image carries its revision as an `fc.revision`
|
||||||
on a hit, publishes NO new bytes — it repoints the channel and date tags at the
|
label, and build.yml reads that label back off the moving channel tag. Equal
|
||||||
manifest already there. So the identity has to be a true name for the content.
|
to the derived revision means the bytes this push would produce are already
|
||||||
Both ways of getting it wrong are silent at build time and only surface in
|
published, so the build is skipped.
|
||||||
production:
|
|
||||||
|
|
||||||
* **too coarse** — two genuinely different images share an identity, so the
|
That makes the revision load-bearing in a way a version string is not — it is
|
||||||
second one never gets built and its tags point at the first one's bytes. The
|
compared for equality against a value stamped into a real published artifact.
|
||||||
live case is FC_CHANNEL: a `dev` and a `main` build of one revision differ,
|
Both ways of getting it wrong are silent:
|
||||||
and collapsing them ships an instance that reports the wrong channel forever.
|
|
||||||
* **too fine** — the identity moves when the content did not, nothing ever
|
|
||||||
hits, and step 4 buys nothing. A commit sha would do exactly this.
|
|
||||||
|
|
||||||
The Dockerfiles are read here rather than trusted, because the coarse direction
|
* **it does not identify the content** — a revision that moves when the source
|
||||||
appears the moment someone adds a build-arg without touching `CHANNELLED`.
|
did not (a HEAD-derived value, say) never matches, nothing is ever skipped,
|
||||||
|
and the mechanism quietly buys nothing while every lane stays green.
|
||||||
|
* **it identifies the wrong content** — a revision that holds still when the
|
||||||
|
source DID change matches a stale label, the build is skipped, and the
|
||||||
|
channel serves bytes that do not correspond to the commit. This is the
|
||||||
|
dangerous direction, and it is what `test_artifact_paths.py` guards from the
|
||||||
|
other side by pinning the path sets.
|
||||||
|
|
||||||
|
This module owns the narrower claim: whatever the path sets say, the revision
|
||||||
|
is genuinely the commit those paths last changed in.
|
||||||
|
|
||||||
|
The identity-TAG tests this file used to hold are gone with the tag. There is
|
||||||
|
no longer a `CHANNELLED` list to drift (the channel is which tag you inspect),
|
||||||
|
and no `identity` subcommand to refuse an unqualified call.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -26,115 +35,75 @@ import pytest
|
|||||||
|
|
||||||
ROOT = Path(__file__).resolve().parent.parent
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
|
||||||
# Only image artifacts have an identity — the extension is cached as an
|
ARTIFACTS = ("web", "ml", "agent", "extension")
|
||||||
# ext-<version> Forgejo release, not a registry tag.
|
|
||||||
IMAGE_ARTIFACTS = {
|
|
||||||
"web": "Dockerfile",
|
|
||||||
"ml": "Dockerfile.ml",
|
|
||||||
"agent": "agent/Dockerfile",
|
|
||||||
}
|
|
||||||
|
|
||||||
CHANNELS = ("main", "dev")
|
# 12 hex chars — the prefix build.yml stamps and compares.
|
||||||
|
_REVISION = re.compile(r"^[0-9a-f]{12}$")
|
||||||
# docker's own tag grammar: [A-Za-z0-9_][A-Za-z0-9._-]{0,127}
|
|
||||||
_TAG = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$")
|
|
||||||
|
|
||||||
# `ARG FC_CHANNEL` in a Dockerfile means build.yml passes a per-channel value
|
|
||||||
# in, so the channel is part of what the image IS.
|
|
||||||
_ARG_CHANNEL = re.compile(r"^\s*ARG\s+FC_CHANNEL\b", re.MULTILINE)
|
|
||||||
|
|
||||||
|
|
||||||
def identity(artifact: str, channel: str | None = None) -> subprocess.CompletedProcess:
|
# Everything here goes through artifacts.sh rather than importing a sibling
|
||||||
cmd = ["sh", str(ROOT / "scripts" / "artifacts.sh"), "identity", artifact]
|
# test module. That is the interface build.yml actually calls, so the tests
|
||||||
if channel is not None:
|
# exercise the contract instead of a Python re-implementation of it — and no
|
||||||
cmd.append(channel)
|
# other test module in this repo imports another, so a cross-test import would
|
||||||
return subprocess.run(cmd, capture_output=True, text=True, cwd=ROOT)
|
# be a new convention introduced for no gain.
|
||||||
|
def artifacts(*args: str) -> str:
|
||||||
|
return subprocess.run(
|
||||||
def ok(artifact: str, channel: str | None = None) -> str:
|
["sh", str(ROOT / "scripts" / "artifacts.sh"), *args],
|
||||||
proc = identity(artifact, channel)
|
|
||||||
assert proc.returncode == 0, f"identity {artifact} {channel}: {proc.stderr}"
|
|
||||||
return proc.stdout.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def bakes_the_channel(artifact: str) -> bool:
|
|
||||||
return bool(_ARG_CHANNEL.search((ROOT / IMAGE_ARTIFACTS[artifact]).read_text()))
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
|
|
||||||
def test_channel_dependence_matches_the_dockerfile(artifact):
|
|
||||||
"""The coarse direction, caught at its source.
|
|
||||||
|
|
||||||
Whether the channel belongs in the identity is not a preference — it is
|
|
||||||
dictated by whether the Dockerfile takes it as a build-arg. Adding an
|
|
||||||
`ARG FC_CHANNEL` to another image without adding it to `CHANNELLED` would
|
|
||||||
make its dev and main builds collide, and nothing else would notice.
|
|
||||||
"""
|
|
||||||
per_channel = {c: ok(artifact, c) for c in CHANNELS}
|
|
||||||
differs = len(set(per_channel.values())) > 1
|
|
||||||
|
|
||||||
if bakes_the_channel(artifact):
|
|
||||||
assert differs, (
|
|
||||||
f"{IMAGE_ARTIFACTS[artifact]} declares ARG FC_CHANNEL, so a dev "
|
|
||||||
f"build and a main build of one revision are different images — "
|
|
||||||
f"but both derive the identity {per_channel['main']!r}. The main "
|
|
||||||
f"build would reuse the dev image and report the wrong channel. "
|
|
||||||
f"Add {artifact!r} to CHANNELLED in scripts/artifacts.sh."
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
assert not differs, (
|
|
||||||
f"{IMAGE_ARTIFACTS[artifact]} takes no channel build-arg, so one "
|
|
||||||
f"revision is one image and a merge to main should reuse what dev "
|
|
||||||
f"already built — but the identity differs per channel "
|
|
||||||
f"({per_channel}), so every merge rebuilds it for nothing. Remove "
|
|
||||||
f"{artifact!r} from CHANNELLED in scripts/artifacts.sh."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
|
|
||||||
def test_identity_tracks_the_artifacts_own_revision(artifact):
|
|
||||||
"""The fine direction: the identity must be the revision, not the push.
|
|
||||||
|
|
||||||
`revision` is the commit this artifact's shipped files last changed in, so
|
|
||||||
it holds still across pushes that did not touch it. Anything derived from
|
|
||||||
HEAD instead would move every push and never hit the registry.
|
|
||||||
"""
|
|
||||||
rev = subprocess.run(
|
|
||||||
["sh", str(ROOT / "scripts" / "artifacts.sh"), "revision", artifact],
|
|
||||||
capture_output=True, text=True, check=True, cwd=ROOT,
|
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||||
).stdout.strip()
|
).stdout
|
||||||
value = ok(artifact, "main")
|
|
||||||
assert rev and rev in value, (
|
|
||||||
f"identity {value!r} does not contain the {artifact} revision {rev!r}"
|
def revision(artifact: str) -> str:
|
||||||
|
return artifacts("revision", artifact).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def newest_by_commit_time(artifact: str) -> str:
|
||||||
|
"""The full SHA of the newest commit touching this artifact's shipped set.
|
||||||
|
|
||||||
|
Ordered by committer TIME, matching what artifacts.sh means. Deliberately
|
||||||
|
not `git log -1`: git's default order is reverse-chronological only within
|
||||||
|
topological constraints, so on a merged history it can name a different
|
||||||
|
commit than the newest timestamp does. They agree on this repo today, and
|
||||||
|
a test that silently depends on them continuing to agree would be a flake
|
||||||
|
waiting for the branch shape that separates them.
|
||||||
|
"""
|
||||||
|
paths = artifacts("paths", artifact).split()
|
||||||
|
log = subprocess.run(
|
||||||
|
["git", "log", "--format=%ct %H", "HEAD", "--", *paths],
|
||||||
|
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||||
|
).stdout.split("\n")
|
||||||
|
commits = [line.split(" ", 1) for line in log if line.strip()]
|
||||||
|
assert commits, (
|
||||||
|
f"no commit in this history touches the {artifact} path set — the "
|
||||||
|
f"derivation has nothing to stand on"
|
||||||
|
)
|
||||||
|
return max(commits, key=lambda c: int(c[0]))[1]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("artifact", ARTIFACTS)
|
||||||
|
def test_revision_is_the_commit_its_own_shipped_files_last_changed_in(artifact):
|
||||||
|
"""The claim the whole skip decision rests on.
|
||||||
|
|
||||||
|
Computed from git rather than asked of the script, so it fails if the
|
||||||
|
derivation ever stops meaning what it says — switching to HEAD, to a build
|
||||||
|
clock, or to a path set it did not actually use. Each of those still
|
||||||
|
produces a plausible 12-hex value, which is why this is worth asserting
|
||||||
|
rather than eyeballing.
|
||||||
|
"""
|
||||||
|
expected = newest_by_commit_time(artifact)
|
||||||
|
got = revision(artifact)
|
||||||
|
assert expected.startswith(got), (
|
||||||
|
f"{artifact} derives {got!r}, but the newest commit touching its "
|
||||||
|
f"shipped files is {expected[:12]!r}. The label stamped into the image "
|
||||||
|
f"would not identify its own content."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("artifact", sorted(IMAGE_ARTIFACTS))
|
@pytest.mark.parametrize("artifact", ARTIFACTS)
|
||||||
def test_identity_is_a_legal_docker_tag(artifact):
|
def test_revision_is_a_legal_label_value_and_is_stable(artifact):
|
||||||
"""It is pushed as a tag, so an illegal one fails at the registry — after
|
"""It is stamped as a docker label and compared for string equality, so a
|
||||||
the build has already run."""
|
stray newline or a varying value breaks the comparison rather than the
|
||||||
for channel in CHANNELS:
|
build — the mechanism would simply stop hitting, silently."""
|
||||||
value = ok(artifact, channel)
|
first = revision(artifact)
|
||||||
assert _TAG.match(value), f"{value!r} is not a valid docker tag"
|
assert _REVISION.match(first), f"{first!r} is not a 12-char hex revision"
|
||||||
|
assert first == revision(artifact), "revision is not stable across calls"
|
||||||
|
|
||||||
def test_a_channelled_artifact_refuses_an_unqualified_identity():
|
|
||||||
"""Refusing beats defaulting. If `identity web` quietly returned the
|
|
||||||
unqualified `r-<rev>`, a workflow that forgot to pass the channel would
|
|
||||||
publish one image under a name both channels then reuse — the exact
|
|
||||||
collision the CHANNELLED list exists to prevent, reintroduced by an
|
|
||||||
omission rather than by an edit."""
|
|
||||||
proc = identity("web")
|
|
||||||
assert proc.returncode != 0, (
|
|
||||||
"identity web returned a value with no channel: "
|
|
||||||
f"{proc.stdout.strip()!r}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_the_extension_has_no_image_identity():
|
|
||||||
"""It is cached as an ext-<version> release asset, and its cache key is the
|
|
||||||
version. Answering with a plausible image tag would invite a second,
|
|
||||||
divergent cache."""
|
|
||||||
proc = identity("extension", "main")
|
|
||||||
assert proc.returncode != 0
|
|
||||||
assert "ext-" in proc.stderr
|
|
||||||
|
|||||||
Reference in New Issue
Block a user