Make a digest mean something again, and give the refresh somewhere to stand #247

Merged
bvandeusen merged 4 commits from dev into main 2026-09-02 14:56:12 -04:00
7 changed files with 742 additions and 101 deletions
+428 -59
View File
@@ -44,6 +44,10 @@ on:
description: 'Rebuild every image even if the published revision matches'
type: boolean
default: false
refresh:
description: 'Behave as the weekly base refresh: build main against fresh bases, publish through the candidate tag'
type: boolean
default: false
# The base-image refresh (milestone 326 step 4, #3154).
#
@@ -72,8 +76,25 @@ on:
# Deriving it per job invites the two halves to disagree: sign-extension would
# derive dev's extension version while build-web bundled main's, and the
# release download would 404 on a version that exists perfectly well.
# IS THIS A BASE REFRESH? Asked in five places and previously spelled five
# ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in
# one shell, an `EVENT:` env passed into another, and a bare expression on
# `pull:`. Five spellings of one fact is how half of them come to disagree
# after somebody adds a sixth trigger.
#
# The `refresh` dispatch input is here so this path can be EXERCISED. A weekly
# cron is otherwise testable once a week, which is not a cadence anything can
# be developed against — the same reason `force_build` exists (#3252, added to
# confirm #3190 was gone rather than wait for it to recur). It is also what
# makes the milestone-362 gate verifiable at all: a gate has to be watched
# rejecting something before anyone can believe it is wired up.
#
# Note this is a STRING comparison, not a boolean. Forgejo delivers
# workflow_dispatch inputs as strings, so `inputs.refresh` is 'true'/'false'
# and `&&` on it would treat the string 'false' as truthy.
env:
BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
IS_REFRESH: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'true' || 'false' }}
BUILD_REF: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'main' || github.ref }}
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
# - write:package, read:package (for docker push to git.fabledsword.com)
@@ -143,7 +164,7 @@ jobs:
# evaluate — this file already gates steps on it — so the guard cannot
# be disabled by the same uncertainty it exists to cover.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
if: env.IS_REFRESH == 'true'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -437,7 +458,7 @@ jobs:
# See sign-extension's copy for why this guard exists.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
if: env.IS_REFRESH == 'true'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -528,7 +549,7 @@ jobs:
# Checked BEFORE the ref test, not after: a scheduled run's
# GITHUB_REF is the default branch (dev), so the main test would
# never fire on it.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
@@ -628,7 +649,6 @@ jobs:
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: |
set -eu
DERIVED=$(sh scripts/artifacts.sh revision web)
@@ -638,11 +658,54 @@ jobs:
# adds no variability the reuse check would have to account for.
echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT"
# The build clock, pinned to the same commit (#3265). Without it
# buildkit stamps the image config with the wall clock of the build,
# so identical layers republish under a new config blob and the
# channel tag gets a new manifest digest for no reason. Derived from
# `newest()` like revision and version, so all three name one commit
# and cannot drift apart.
echo "epoch=$(sh scripts/artifacts.sh epoch web)" >> "$GITHUB_OUTPUT"
# The moving tag for this channel. Which tag we ask IS the channel —
# that is why the revision needs no -main/-dev qualifier any more.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
# whether the channel then has to be written separately.
#
# On a push the build writes the channel tag directly: the bytes came
# from a commit, and a commit is the thing CI tests. Nothing to hold
# it behind.
#
# On the scheduled refresh it writes a CANDIDATE tag instead. A
# refresh rebuilds against freshly resolved base images, and the web
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
# install requirements.txt, and a base bump changes neither. So
# refreshed bytes have to be proven before :latest names them, and
# proving needs a moment between "built" and "published" to occupy.
# This is that moment; :latest goes on naming the build that works
# until something says otherwise.
#
# `:refresh-candidate` is one moving ref per image, overwritten in
# place, holding a build nobody is told to pull — the shape rule 145
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
# (run 4732): a missing key returns an empty string and exits 0, so
# branching on the exit code would read "no label yet" as success.
@@ -675,7 +738,7 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
elif [ "${IS_REFRESH:-}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
@@ -764,6 +827,12 @@ jobs:
- name: Build and push web image
if: steps.reuse.outputs.hit != 'true'
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
# it normalises the image config's `created` field and the history
# timestamps rather than being consumed by the Dockerfile. See #3265
# and the reuse step's `epoch` output.
env:
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
uses: docker/build-push-action@v5
with:
context: .
@@ -776,20 +845,17 @@ jobs:
# invalidates, and the image genuinely rebuilds.
#
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
# did NOT move, the build was ~13s with every content step CACHED —
# and the channel tag STILL got a new manifest digest, because
# buildkit stamps a fresh image config per run and republishes the
# identical layers under it. All three images moved that way on
# 2026-08-30 with nothing whatsoever changed in them.
#
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
# content came from, the config is byte-identical across runs, so
# the manifest digest is too and the push is a registry no-op. A
# digest change means the content changed again, which is the only
# thing a digest is any use for.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
@@ -799,14 +865,14 @@ jobs:
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
pull: ${{ env.IS_REFRESH == 'true' }}
# ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver,
# out of a local image store a registry-direct build never filled —
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
# published perfectly well.
tags: ${{ steps.reuse.outputs.channel_ref }}
tags: ${{ steps.reuse.outputs.build_ref }}
# The reuse key. Read back off the channel tag on the next push to
# decide whether that push needs to build at all, so this is not
# decoration — an unstamped image is one that will always rebuild.
@@ -853,6 +919,77 @@ jobs:
FC_CHANNEL=${{ steps.tag.outputs.channel }}
FC_VERSION=${{ steps.reuse.outputs.version }}
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
@@ -957,7 +1094,7 @@ jobs:
# See sign-extension's copy for why this guard exists.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
if: env.IS_REFRESH == 'true'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -1008,7 +1145,7 @@ jobs:
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list and its schedule handling; see
# the comments there.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
@@ -1091,17 +1228,59 @@ jobs:
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: |
set -eu
DERIVED=$(sh scripts/artifacts.sh revision ml)
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
# The build clock, pinned to the same commit (#3265). Without it
# buildkit stamps the image config with the wall clock of the build,
# so identical layers republish under a new config blob and the
# channel tag gets a new manifest digest for no reason. Derived from
# `newest()` like revision and version, so all three name one commit
# and cannot drift apart.
echo "epoch=$(sh scripts/artifacts.sh epoch ml)" >> "$GITHUB_OUTPUT"
# The moving tag for this channel. Which tag we ask IS the channel —
# that is why the revision needs no -main/-dev qualifier any more.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
# whether the channel then has to be written separately.
#
# On a push the build writes the channel tag directly: the bytes came
# from a commit, and a commit is the thing CI tests. Nothing to hold
# it behind.
#
# On the scheduled refresh it writes a CANDIDATE tag instead. A
# refresh rebuilds against freshly resolved base images, and the web
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
# install requirements.txt, and a base bump changes neither. So
# refreshed bytes have to be proven before :latest names them, and
# proving needs a moment between "built" and "published" to occupy.
# This is that moment; :latest goes on naming the build that works
# until something says otherwise.
#
# `:refresh-candidate` is one moving ref per image, overwritten in
# place, holding a build nobody is told to pull — the shape rule 145
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
# (run 4732): a missing key returns an empty string and exits 0, so
# branching on the exit code would read "no label yet" as success.
@@ -1134,7 +1313,7 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
elif [ "${IS_REFRESH:-}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
@@ -1147,6 +1326,12 @@ jobs:
- name: Build and push ml image
if: steps.reuse.outputs.hit != 'true'
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
# it normalises the image config's `created` field and the history
# timestamps rather than being consumed by the Dockerfile. See #3265
# and the reuse step's `epoch` output.
env:
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
uses: docker/build-push-action@v5
with:
context: .
@@ -1159,20 +1344,17 @@ jobs:
# invalidates, and the image genuinely rebuilds.
#
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
# did NOT move, the build was ~13s with every content step CACHED —
# and the channel tag STILL got a new manifest digest, because
# buildkit stamps a fresh image config per run and republishes the
# identical layers under it. All three images moved that way on
# 2026-08-30 with nothing whatsoever changed in them.
#
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
# content came from, the config is byte-identical across runs, so
# the manifest digest is too and the push is a registry no-op. A
# digest change means the content changed again, which is the only
# thing a digest is any use for.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
@@ -1182,14 +1364,14 @@ jobs:
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
pull: ${{ env.IS_REFRESH == 'true' }}
# ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver,
# out of a local image store a registry-direct build never filled —
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
# published perfectly well.
tags: ${{ steps.reuse.outputs.channel_ref }}
tags: ${{ steps.reuse.outputs.build_ref }}
# The reuse key. Read back off the channel tag on the next push to
# decide whether that push needs to build at all, so this is not
# decoration — an unstamped image is one that will always rebuild.
@@ -1230,6 +1412,77 @@ jobs:
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
@@ -1331,7 +1584,7 @@ jobs:
# See sign-extension's copy for why this guard exists.
- name: Guard — a scheduled run must have checked out main
if: github.event_name == 'schedule'
if: env.IS_REFRESH == 'true'
run: |
set -eu
BRANCH=$(git rev-parse --abbrev-ref HEAD)
@@ -1377,7 +1630,7 @@ jobs:
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list and its schedule handling; see
# the comments there.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
@@ -1460,17 +1713,59 @@ jobs:
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: |
set -eu
DERIVED=$(sh scripts/artifacts.sh revision agent)
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
# The build clock, pinned to the same commit (#3265). Without it
# buildkit stamps the image config with the wall clock of the build,
# so identical layers republish under a new config blob and the
# channel tag gets a new manifest digest for no reason. Derived from
# `newest()` like revision and version, so all three name one commit
# and cannot drift apart.
echo "epoch=$(sh scripts/artifacts.sh epoch agent)" >> "$GITHUB_OUTPUT"
# The moving tag for this channel. Which tag we ask IS the channel —
# that is why the revision needs no -main/-dev qualifier any more.
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
# whether the channel then has to be written separately.
#
# On a push the build writes the channel tag directly: the bytes came
# from a commit, and a commit is the thing CI tests. Nothing to hold
# it behind.
#
# On the scheduled refresh it writes a CANDIDATE tag instead. A
# refresh rebuilds against freshly resolved base images, and the web
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
# install requirements.txt, and a base bump changes neither. So
# refreshed bytes have to be proven before :latest names them, and
# proving needs a moment between "built" and "published" to occupy.
# This is that moment; :latest goes on naming the build that works
# until something says otherwise.
#
# `:refresh-candidate` is one moving ref per image, overwritten in
# place, holding a build nobody is told to pull — the shape rule 145
# already allows for :buildcache, not the per-build tag family that
# milestone 318 withdrew.
#
# Both values are decided HERE, beside `hit`, for the reason the
# force/schedule branch below gives: one step decides what this job
# does. A promote condition derived independently could disagree with
# the tag the build actually wrote.
if [ "${IS_REFRESH:-}" = "true" ]; then
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
echo "promote=true" >> "$GITHUB_OUTPUT"
else
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
echo "promote=false" >> "$GITHUB_OUTPUT"
fi
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
# (run 4732): a missing key returns an empty string and exits 0, so
# branching on the exit code would read "no label yet" as success.
@@ -1503,7 +1798,7 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
elif [ "${IS_REFRESH:-}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
@@ -1516,6 +1811,12 @@ jobs:
- name: Build and push agent image
if: steps.reuse.outputs.hit != 'true'
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
# it normalises the image config's `created` field and the history
# timestamps rather than being consumed by the Dockerfile. See #3265
# and the reuse step's `epoch` output.
env:
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
uses: docker/build-push-action@v5
with:
context: agent
@@ -1528,20 +1829,17 @@ jobs:
# invalidates, and the image genuinely rebuilds.
#
# MEASURED on the first real fire, run 4934 (#3265): when the base
# did NOT move, the build is ~13s and every content step reports
# CACHED — but the channel tag STILL gets a new manifest digest.
# buildkit mints a fresh image config each run, so identical layers
# are republished under a new config blob. All three images moved
# that way on 2026-08-30 with nothing whatsoever changed in them.
# did NOT move, the build was ~13s with every content step CACHED —
# and the channel tag STILL got a new manifest digest, because
# buildkit stamps a fresh image config per run and republishes the
# identical layers under it. All three images moved that way on
# 2026-08-30 with nothing whatsoever changed in them.
#
# So a refresh currently rewrites :latest every Sunday whether or
# not there is anything new in it, and :c-<sha> is handed a new
# manifest to diverge from on the same cadence. Layers are shared,
# so the storage cost is a config blob; the cost that matters is
# that a digest change no longer MEANS anything. Tracked in #3265 —
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
# make "same source, same bytes" true and turn the no-op case into
# a genuine no-op.
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
# content came from, the config is byte-identical across runs, so
# the manifest digest is too and the push is a registry no-op. A
# digest change means the content changed again, which is the only
# thing a digest is any use for.
#
# What `pull` does NOT catch either: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
@@ -1551,14 +1849,14 @@ jobs:
# churn #3265 is about.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
pull: ${{ env.IS_REFRESH == 'true' }}
# ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver,
# out of a local image store a registry-direct build never filled —
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
# published perfectly well.
tags: ${{ steps.reuse.outputs.channel_ref }}
tags: ${{ steps.reuse.outputs.build_ref }}
# The reuse key. Read back off the channel tag on the next push to
# decide whether that push needs to build at all, so this is not
# decoration — an unstamped image is one that will always rebuild.
@@ -1599,6 +1897,77 @@ jobs:
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
# Point the channel tag at the candidate the refresh just built.
#
# Unconditional TODAY, so this milestone never leaves the refresh in a
# state where it builds and publishes nothing. Step 4 wraps it in the
# smoke suite's verdict; until then the scheduled path behaves exactly
# as it did, just via two operations instead of one.
#
# NOT `imagetools create`. That wraps its source in an INDEX, and an
# indexed channel tag is the one thing this pipeline cannot survive:
# `.Image.Config.Labels` does not resolve through an index, so the
# fc.revision the reuse check reads off the channel tag would come back
# empty, every subsequent push would miss and rebuild, and nothing would
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
# and the only symptom was the bill. The repoint step below excludes its
# own source tag for precisely this reason; a promote that re-introduced
# the wrap through a different door would undo that care.
#
# A manifest PUT is what "make this tag name that image" means at the
# registry level: the same bytes under the same media type, so the digest
# is identical, the media type is preserved, and no layer moves.
- name: Promote the refresh candidate to the channel
if: steps.reuse.outputs.promote == 'true'
env:
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
TOKEN: ${{ secrets.RELEASE_TOKEN }}
ACTOR: ${{ github.actor }}
run: |
set -eu
REPO=${IMAGE#git.fabledsword.com/}
TAG=${CHANNEL_REF##*:}
# Registry auth is its own token exchange — the `docker login` above
# authenticates the docker client, not curl. Deadline on every call
# (rule 156): a registry that stops answering must fail this step,
# not hang the weekly refresh until the job times out.
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
# Ask for the image manifest media types ONLY. Offering the index
# types too would let the registry hand back an index if one ever
# existed at this tag, and we would faithfully copy the thing we are
# trying not to create.
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
test -n "$CT"
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
echo "promote: candidate is $SRC_DIGEST ($CT)"
curl -fsS --max-time 120 -X PUT \
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
--data-binary @manifest.json \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
# Read it back. A PUT that returned 2xx but landed something else is
# exactly the silent-and-plausible failure this pipeline keeps
# producing, and the check costs one request.
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
if [ "$NOW" != "$SRC_DIGEST" ]; then
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
exit 1
fi
echo "promote: $IMAGE:$TAG now names $NOW"
# Every tag but the channel's own is written HERE, registry-side,
# whether or not a build ran. Each -t becomes another reference to the
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
+5 -2
View File
@@ -1,5 +1,6 @@
# FabledCurator
<!-- overview:start -->
Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven
downloading in one application. Part of the FabledSword family.
@@ -19,13 +20,15 @@ through afterwards.
characters, find near-duplicates and power similarity search. Suggestions are
reviewable — it proposes, you confirm, and it learns which proposals you keep
rejecting.
- **Importing.** Ingests an existing library from disk, deduplicates it by
content hash, and reads metadata sidecars.
- **Deduplication and provenance.** Everything that arrives is hashed and
deduplicated by content, metadata sidecars are read wherever the source
writes them, and every file keeps a record of where it came from.
- **Maintenance.** Backups, library audits, thumbnail and embedding backfills,
orphan cleanup — all from the UI, all as background jobs you can watch.
Everything is configured from the Settings UI and stored in the database. There
is no config file to edit beyond a handful of bootstrap environment variables.
<!-- overview:end -->
## Before you expose it
+15 -8
View File
@@ -198,14 +198,21 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
refresh from being undone.
- **`pull: true` on the scheduled path only** is the mechanism: a moved base
tag changes the `FROM` layer's cache key and everything above it rebuilds.
**It does not currently make the unmoved case free.** Measured on the first
real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
the bases resolved to unchanged digests, yet all three `:latest` tags got a
NEW manifest digest, because buildkit mints a fresh image config per run and
republishes identical layers under it. So `:latest` is rewritten weekly
whether or not anything changed, and `:c-<sha>` is handed a new manifest to
diverge from on the same cadence — a digest change stops meaning anything.
Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
It did not always make the unmoved case free. Measured on the first real
fire (run 4934, 2026-08-30): every content step reported `CACHED` and the
bases resolved to unchanged digests, yet all three `:latest` tags got a NEW
manifest digest, because buildkit stamps a fresh image config per run and
republishes identical layers under it — so `:latest` was rewritten weekly
whether or not anything changed, and a digest change stopped meaning
anything (#3265).
- **`SOURCE_DATE_EPOCH` is what makes it free.** Set on each build step from
`artifacts.sh epoch <artifact>` — the unix timestamp of the same commit
`revision` and `version` name, so all three are views of one `newest()`
lookup and cannot drift into disagreeing. With the config's `created` field
and history timestamps pinned to the content rather than to the wall clock,
identical source produces an identical manifest digest and the push is a
registry no-op. That restores the property the whole scheme rests on: a
channel tag's digest changes when, and only when, its content does.
Separately not caught: a Debian package update inside the `apt-get install`
layer while the base tag stands still — a lag rather than a hole, since the
official python/cuda images rebuild with those updates baked in.
+22 -1
View File
@@ -90,7 +90,7 @@ DERIVER='scripts/artifacts.sh'
usage() {
echo "usage: artifacts.sh {paths|revision|version} {web|ml|agent|extension}" >&2
echo "usage: artifacts.sh {paths|revision|version|epoch} {web|ml|agent|extension}" >&2
exit 2
}
@@ -149,6 +149,26 @@ cmd_revision() {
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
}
# The BUILD CLOCK: the same commit's unix timestamp, for SOURCE_DATE_EPOCH.
#
# buildkit stamps the image config's `created` field and every history entry
# with the wall clock of the build unless this is set, so two builds of
# identical source produce different config blobs and therefore different
# manifest digests. That is #3265: the weekly refresh republished all three
# `:latest` tags on 2026-08-30 with every content step CACHED and the bases
# resolved to unchanged digests — nothing was different, and the digest moved
# anyway. A digest that changes on a calendar cannot also mean "the content
# changed", which is the only thing anyone wants it for.
#
# It is the same commit `revision` and `version` name — deliberately, and this
# is the point of routing it through `newest()` rather than taking git's word
# separately. Three values derived from three lookups can disagree; three
# views of one lookup cannot. Note #3127 §2 is the record of what a second
# clock costs.
cmd_epoch() {
echo "$(newest "$1")" | cut -d' ' -f1
}
# The VERSION: `YYYY.MM.DD.HHMM`, zero-padded, UTC. One shape across the whole
# family (note #3127 §1, rule 148) — the number an instance reports about
# itself, and, with a `v` in front, the release tag naming the same build.
@@ -197,5 +217,6 @@ case "$1" in
paths) cmd_paths "$2" ;;
revision) cmd_revision "$2" ;;
version) cmd_version "$2" ;;
epoch) cmd_epoch "$2" ;;
*) usage ;;
esac
+118 -14
View File
@@ -33,6 +33,32 @@ history. Ancestry is immune to the shape change, and it is also the more honest
question: "what is in this that was not in the last one" IS a reachability
question.
Ancestry alone is not enough, though, and milestone 328 is where that showed.
The 28 `v26.*` tags are still in the repo — the operator kept them as history
when their releases were deleted — so `--match v*` walks straight back to
`v26.06.04.0` and reports 533 commits. That span is not a changelog: nobody has
run `v26.06.04.0`, its release page no longer exists to compare against, and
the 200 lines that survive truncation are precisely the internal build-out that
milestone 328 exists to stop shipping. So the match is `v[0-9][0-9][0-9][0-9].*`
— rule 148's four-digit-year shape — which is exactly the set of tags that name
a release a reader could have been running. A pre-convention tag is history,
not a predecessor.
## The first release has no changelog, and should not pretend to
Once the match is narrowed, the first rule-148 tag reaches no predecessor at
all, and the old fallback — diff against the whole history — is worse than the
problem it replaced. The honest content for a release nobody has a previous
version of is what the thing IS.
So a release with no reachable predecessor renders the product overview instead
of a commit list. It is read out of README.md between `<!-- overview:start -->`
and `<!-- overview:end -->` rather than written here, for the same reason the
changelog is derived: two hand-maintained descriptions of one product drift,
and nothing ever catches it. The release page and the repo front page are one
source. Every later release goes back to being a changelog, which is what §5 of
note #3127 says a release is for.
## Re-runs update, they do not fall through
Note #3127 §6.7: a publisher that POSTs and recovers the id from a `409` never
@@ -91,18 +117,45 @@ def git_ok(*args: str) -> str | None:
def previous_tag(ref: str, tag: str | None) -> str | None:
"""The most recent `v*` tag reachable from `ref`, excluding `tag` itself.
"""The most recent rule-148 tag reachable from `ref`, excluding `tag` itself.
`--exclude` rather than `<ref>^` so this is the same call whether or not
`ref` is the tag being released — and so it does not blow up on a root
commit that has no parent to walk to.
The glob deliberately does NOT match the old `v26.*` tags. They are kept as
history and their releases are gone, so naming one as the predecessor emits
a span nobody can look up. See the module docstring.
"""
args = ["describe", "--tags", "--abbrev=0", "--match", "v*"]
args = ["describe", "--tags", "--abbrev=0", "--match", "v[0-9][0-9][0-9][0-9].*"]
if tag:
args += ["--exclude", tag]
return git_ok(*args, ref)
def product_overview() -> str | None:
"""The product description, lifted verbatim from README.md.
Returns None if the markers are absent or empty — a missing overview is
reported as a note and the release still publishes, on the same reasoning
as cross_checks(): the release is the useful object even when one part of
the derivation could not run.
"""
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
try:
with open(os.path.join(root, "README.md"), encoding="utf-8") as fh:
readme = fh.read()
except OSError:
return None
match = re.search(
r"<!--\s*overview:start\s*-->(.*?)<!--\s*overview:end\s*-->",
readme, re.S,
)
if not match:
return None
return match.group(1).strip() or None
def commits(previous: str | None, ref: str) -> list[str]:
"""The subjects between the previous release and this one.
@@ -125,7 +178,10 @@ def truncate(log: list[str]) -> tuple[list[str], str | None]:
)
def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list[str]) -> str:
def render(
tag: str, sha: str, previous: str | None, log: list[str], notes: list[str],
overview: str | None,
) -> str:
short = sha[:7]
parts = []
@@ -135,6 +191,25 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
# is the failure this whole milestone is about.
parts.append("\n".join(f"> **Note:** {n}" for n in notes))
# No predecessor means nobody reading this has run an earlier one, so the
# release describes the product rather than a diff. The overview is
# README.md's own words — see the module docstring on why it is not
# written here.
if previous is None and overview:
parts.append(overview)
parts.append(
"## Installing\n\n"
"```\ncurl -O https://git.fabledsword.com/bvandeusen/FabledCurator/raw/"
f"tag/{tag}/docker-compose.yml\ncurl -O https://git.fabledsword.com/"
f"bvandeusen/FabledCurator/raw/tag/{tag}/.env.example\n"
"mv .env.example .env # then set SECRET_KEY, DB_PASSWORD\n"
"docker compose -f docker-compose.yml up -d\n```\n\n"
"**Read \"Before you expose it\" in the README first.** FabledCurator "
"has no login, and it stores live platform session cookies for "
"accounts that usually have a payment method attached. Bind it to a "
"network you trust."
)
parts.append(
f"Built from `{short}`. The rollback unit is the immutable `:c-` tag "
f"(rule 145) — these three move together:\n\n```\n"
@@ -142,7 +217,19 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
+ "\n```"
)
heading = f"## Changes since {previous}" if previous else "## Changes"
if previous is None:
# Deliberately NOT a commit list. The alternative is the whole history
# truncated to MAX_COMMITS, which is 200 lines of internal build-out
# presented to someone who has never seen this project.
parts.append(
"---\n\n_First release under rule 148's `vYYYY.MM.DD.HHMM` shape, so "
"there is no predecessor to diff against and no changelog to derive. "
"The description above is README.md's, quoted at publish time. Later "
"releases carry the commits since the previous one._"
)
return "\n\n".join(parts)
heading = f"## Changes since {previous}"
if log:
parts.append(heading + "\n\n" + "\n".join(f"- {line}" for line in log))
else:
@@ -152,10 +239,9 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
"names the same source under a new name._"
)
span = f"{previous}..{tag}" if previous else tag
parts.append(
f"---\n\n_Derived at publish time from `git log --no-merges {span}`. "
f"Nothing here is hand-maintained._"
f"---\n\n_Derived at publish time from "
f"`git log --no-merges {previous}..{tag}`. Nothing here is hand-maintained._"
)
return "\n\n".join(parts)
@@ -289,13 +375,31 @@ def main() -> None:
for note in notes:
print(f"release: NOTE {note}")
log = commits(previous, ref)
print(f"release: {len(log)} non-merge commits in the span")
log, overflow = truncate(log)
if overflow:
print(f"release: NOTE {overflow}")
notes.append(overflow)
body = render(tag or ref, sha, previous, log, notes)
# A first release renders the overview instead of a changelog, so the
# commit walk is skipped entirely rather than computed and discarded —
# `commits(None, ref)` is the whole history and there is no reason to ask
# for it.
overview = None
log: list[str] = []
if previous is None:
overview = product_overview()
if overview is None:
note = (
"No `<!-- overview:start -->` block found in README.md, so this "
"first release has no product description. Published anyway; add "
"the markers and re-run the workflow to fill it in."
)
print(f"release: NOTE {note}")
notes.append(note)
print("release: no rule-148 predecessor — rendering the product overview")
else:
log = commits(previous, ref)
print(f"release: {len(log)} non-merge commits in the span")
log, overflow = truncate(log)
if overflow:
print(f"release: NOTE {overflow}")
notes.append(overflow)
body = render(tag or ref, sha, previous, log, notes, overview)
if args.dry_run or not tag:
print("--- body ---")
+53
View File
@@ -234,3 +234,56 @@ def test_version_and_revision_describe_the_same_commit(artifact):
f"in AMO_UNPADDED may differ here."
)
assert sha.startswith(revision(artifact))
@pytest.mark.parametrize("artifact", ARTIFACTS)
def test_epoch_is_the_same_commit_the_version_names(artifact):
"""The build clock and the version must be one lookup, not two.
`epoch` feeds SOURCE_DATE_EPOCH, which decides the image config's bytes and
therefore the manifest digest; `version` is what the instance reports about
itself. If they could name different commits, an image would be stamped
reproducibly against one commit while claiming to be another — and both
values would look perfectly well-formed, exactly like the divergence the
test above guards.
They cannot, because `cmd_epoch` and `cmd_version` are two fields of one
`newest()` result. This pins that they stay that way: a future refactor
that gave epoch its own `git log` would pass every other test here.
"""
epoch = artifacts("epoch", artifact).strip()
assert epoch.isdigit(), f"{artifact} epoch is {epoch!r}, not a unix timestamp"
sha = newest_by_commit_time(artifact)
committed = subprocess.run(
["git", "show", "-s", "--format=%ct", sha],
capture_output=True, text=True, check=True, cwd=ROOT,
).stdout.strip()
assert epoch == committed, (
f"{artifact} derives epoch {epoch}, but its newest shipped commit "
f"{sha[:12]} was committed at {committed}. SOURCE_DATE_EPOCH would "
f"pin the image config to a commit the version does not name."
)
# And the two renderings must agree, which is the property that actually
# matters at build time: same commit in, same digest and same reported
# version out.
rendered = subprocess.run(
["git", "show", "-s", "--format=%cd", "--date=format-local:%Y.%m.%d.%H%M", sha],
capture_output=True, text=True, check=True, cwd=ROOT,
env={"TZ": "UTC", "PATH": os.environ.get("PATH", "")},
).stdout.strip()
assert segments(artifacts("version", artifact).strip()) == segments(rendered)
def test_epoch_is_stable_across_calls():
"""SOURCE_DATE_EPOCH's entire job is to be the same on the next build.
A value that moved between two invocations on one unchanged checkout would
reintroduce #3265 through the very mechanism meant to close it, and the
symptom would be indistinguishable: a digest that changes for no reason.
"""
for artifact in ARTIFACTS:
first = artifacts("epoch", artifact).strip()
second = artifacts("epoch", artifact).strip()
assert first == second, f"{artifact} epoch moved: {first} then {second}"
+101 -17
View File
@@ -24,12 +24,36 @@ SCRIPT = ROOT / "scripts" / "release_notes.py"
def notes(*args: str, cwd: Path | None = None) -> str:
"""Run the script the way release.yml does.
A synthetic repo runs its OWN copy of the script, because the overview is
read relative to `__file__` rather than to the cwd — which is right in
production (release.yml checks out the tag, so the script IS the tagged
tree's copy) and would otherwise make every synthetic repo silently quote
FabledCurator's real README.
"""
root = cwd or ROOT
script = root / "scripts" / "release_notes.py"
return subprocess.run(
["python3", str(SCRIPT), "--dry-run", *args],
capture_output=True, text=True, check=True, cwd=cwd or ROOT,
["python3", str(script if script.exists() else SCRIPT), "--dry-run", *args],
capture_output=True, text=True, check=True, cwd=root,
).stdout
OVERVIEW_TEXT = "A synthetic product, described once."
def install_script(repo: Path, *, overview: bool = True) -> None:
"""Give a synthetic repo the script and a README to quote."""
(repo / "scripts").mkdir(exist_ok=True)
(repo / "scripts" / "release_notes.py").write_text(SCRIPT.read_text())
(repo / "scripts" / "artifacts.sh").write_text("#!/bin/sh\nexit 1\n")
readme = "# Synthetic\n\n"
if overview:
readme += f"<!-- overview:start -->\n{OVERVIEW_TEXT}\n<!-- overview:end -->\n"
(repo / "README.md").write_text(readme)
def body_of(out: str) -> str:
assert "--- body ---" in out, f"no body was rendered:\n{out}"
return out.split("--- body ---", 1)[1]
@@ -56,9 +80,10 @@ def shaped_history(tmp_path: Path) -> Path:
repo = tmp_path / "shaped"
repo.mkdir()
git(repo, "init", "-q", "-b", "main")
install_script(repo)
for i, tag in enumerate(("v26.06.04.0", "v2026.08.28.2208", "v2026.08.29.1000")):
(repo / "f.txt").write_text(f"{i}\n")
git(repo, "add", "f.txt")
git(repo, "add", "-A")
git(repo, "commit", "-q", "-m", f"work landing in {tag}")
git(repo, "tag", tag)
# One more commit and a merge, so the merge-exclusion test has something to
@@ -109,12 +134,57 @@ def test_merges_are_excluded_so_the_list_is_the_work(shaped_history):
assert "Merge pull request #999" not in body
def test_the_first_release_still_renders_with_nothing_behind_it(shaped_history):
"""No previous tag is reachable from the oldest one. That is a real state,
not an error, and it must not take the release down with it."""
out = notes("v26.06.04.0", cwd=shaped_history)
def test_a_pre_convention_tag_is_history_not_a_predecessor(shaped_history):
"""The defect milestone 328 hit, and the reason the match glob narrowed.
The 28 `v26.*` tags are kept as history while their releases were deleted.
Ancestry alone happily names `v26.06.04.0` as the predecessor of the first
rule-148 tag — and then the body offers "changes since" a release that no
longer exists, over a span (533 commits in the real repo) that is the
internal build-out this milestone exists to stop publishing.
Reachable is not the same as comparable. Only a `vYYYY.` tag names a
release a reader could have been running.
"""
out = notes("v2026.08.28.2208", cwd=shaped_history)
assert "previous=<none>" in out
assert "## Changes" in body_of(out)
assert "v26.06.04.0" not in body_of(out)
def test_the_first_release_describes_the_product_instead_of_diffing(shaped_history):
"""No predecessor means nobody reading has run an earlier version, so a
changelog has no referent. The alternative the script used to take — diff
against the whole history, truncated — puts 200 lines of internal build-out
in front of someone meeting the project for the first time."""
body = body_of(notes("v2026.08.28.2208", cwd=shaped_history))
assert OVERVIEW_TEXT in body
assert "## Changes" not in body
assert not [ln for ln in body.split("\n") if ln.startswith("- work landing")]
def test_the_overview_is_readmes_words_not_a_second_copy(shaped_history):
"""Two hand-maintained descriptions of one product drift and nothing
catches it. The release page quotes README.md so there is one source."""
readme = (shaped_history / "README.md").read_text()
assert OVERVIEW_TEXT in readme
assert OVERVIEW_TEXT in body_of(notes("v2026.08.28.2208", cwd=shaped_history))
def test_a_missing_overview_block_is_reported_and_still_publishes(tmp_path):
"""Same reasoning as cross_checks(): the release is the useful object even
when part of the derivation could not run. Say what is missing, publish
anyway — do not leave the operator with a tag and no release."""
repo = tmp_path / "no-markers"
repo.mkdir()
git(repo, "init", "-q", "-b", "main")
install_script(repo, overview=False)
git(repo, "add", "-A")
git(repo, "commit", "-q", "-m", "first")
git(repo, "tag", "v2026.09.01.1200")
out = notes("v2026.09.01.1200", cwd=repo)
assert "No `<!-- overview:start -->` block found" in out
assert "No `<!-- overview:start -->` block found" in body_of(out)
def test_a_non_tag_ref_renders_but_refuses_to_claim_it_published():
@@ -135,15 +205,29 @@ def test_the_rollback_refs_name_all_three_images():
assert f"bvandeusen/{image}:c-" in body, f"{image} missing from the rollback refs"
def test_an_unbounded_span_is_truncated_and_says_so():
"""With no reachable previous tag the span is the whole history. Emitting
eleven hundred lines would bury the one line explaining why there are
eleven hundred of them, so the cap is part of the message, not a silent
slice."""
out = notes("HEAD")
if "previous=<none>" not in out:
pytest.skip("a previous tag is reachable from HEAD in this checkout")
def test_a_long_span_between_two_releases_is_truncated_and_says_so(tmp_path):
"""The cap is still reachable, just not by the route it used to be.
It no longer fires on "no predecessor" — that renders the overview now.
What it still guards is two real releases far enough apart that the list
stops being something anyone reads, which is the ordinary case for a
project that cuts a bookmark twice a year. The cap is part of the message,
not a silent slice.
"""
repo = tmp_path / "long"
repo.mkdir()
git(repo, "init", "-q", "-b", "main")
install_script(repo)
git(repo, "add", "-A")
git(repo, "commit", "-q", "-m", "scaffold")
git(repo, "tag", "v2026.01.01.0000")
for i in range(205):
git(repo, "commit", "-q", "--allow-empty", "-m", f"fix: change {i}")
git(repo, "tag", "v2026.07.01.0000")
out = notes("v2026.07.01.0000", cwd=repo)
assert "previous=v2026.01.01.0000" in out
body = body_of(out)
listed = [ln for ln in body.split("\n") if ln.startswith("- ")]
assert len(listed) <= 200
assert len(listed) == 200
assert "more than a changelog is for" in body