Make a digest mean something again, and give the refresh somewhere to stand #247
+428
-59
@@ -44,6 +44,10 @@ on:
|
||||
description: 'Rebuild every image even if the published revision matches'
|
||||
type: boolean
|
||||
default: false
|
||||
refresh:
|
||||
description: 'Behave as the weekly base refresh: build main against fresh bases, publish through the candidate tag'
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
# The base-image refresh (milestone 326 step 4, #3154).
|
||||
#
|
||||
@@ -72,8 +76,25 @@ on:
|
||||
# Deriving it per job invites the two halves to disagree: sign-extension would
|
||||
# derive dev's extension version while build-web bundled main's, and the
|
||||
# release download would 404 on a version that exists perfectly well.
|
||||
# IS THIS A BASE REFRESH? Asked in five places and previously spelled five
|
||||
# ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in
|
||||
# one shell, an `EVENT:` env passed into another, and a bare expression on
|
||||
# `pull:`. Five spellings of one fact is how half of them come to disagree
|
||||
# after somebody adds a sixth trigger.
|
||||
#
|
||||
# The `refresh` dispatch input is here so this path can be EXERCISED. A weekly
|
||||
# cron is otherwise testable once a week, which is not a cadence anything can
|
||||
# be developed against — the same reason `force_build` exists (#3252, added to
|
||||
# confirm #3190 was gone rather than wait for it to recur). It is also what
|
||||
# makes the milestone-362 gate verifiable at all: a gate has to be watched
|
||||
# rejecting something before anyone can believe it is wired up.
|
||||
#
|
||||
# Note this is a STRING comparison, not a boolean. Forgejo delivers
|
||||
# workflow_dispatch inputs as strings, so `inputs.refresh` is 'true'/'false'
|
||||
# and `&&` on it would treat the string 'false' as truthy.
|
||||
env:
|
||||
BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
|
||||
IS_REFRESH: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'true' || 'false' }}
|
||||
BUILD_REF: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'main' || github.ref }}
|
||||
|
||||
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
|
||||
# - write:package, read:package (for docker push to git.fabledsword.com)
|
||||
@@ -143,7 +164,7 @@ jobs:
|
||||
# evaluate — this file already gates steps on it — so the guard cannot
|
||||
# be disabled by the same uncertainty it exists to cover.
|
||||
- name: Guard — a scheduled run must have checked out main
|
||||
if: github.event_name == 'schedule'
|
||||
if: env.IS_REFRESH == 'true'
|
||||
run: |
|
||||
set -eu
|
||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||
@@ -437,7 +458,7 @@ jobs:
|
||||
|
||||
# See sign-extension's copy for why this guard exists.
|
||||
- name: Guard — a scheduled run must have checked out main
|
||||
if: github.event_name == 'schedule'
|
||||
if: env.IS_REFRESH == 'true'
|
||||
run: |
|
||||
set -eu
|
||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||
@@ -528,7 +549,7 @@ jobs:
|
||||
# Checked BEFORE the ref test, not after: a scheduled run's
|
||||
# GITHUB_REF is the default branch (dev), so the main test would
|
||||
# never fire on it.
|
||||
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
|
||||
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
|
||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||
@@ -628,7 +649,6 @@ jobs:
|
||||
# A scheduled refresh has to bypass reuse by construction: it
|
||||
# rebuilds the SAME source, so fc.revision always matches and the
|
||||
# check would skip every refresh there has ever been.
|
||||
EVENT: ${{ github.event_name }}
|
||||
run: |
|
||||
set -eu
|
||||
DERIVED=$(sh scripts/artifacts.sh revision web)
|
||||
@@ -638,11 +658,54 @@ jobs:
|
||||
# adds no variability the reuse check would have to account for.
|
||||
echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# The build clock, pinned to the same commit (#3265). Without it
|
||||
# buildkit stamps the image config with the wall clock of the build,
|
||||
# so identical layers republish under a new config blob and the
|
||||
# channel tag gets a new manifest digest for no reason. Derived from
|
||||
# `newest()` like revision and version, so all three name one commit
|
||||
# and cannot drift apart.
|
||||
echo "epoch=$(sh scripts/artifacts.sh epoch web)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||
# that is why the revision needs no -main/-dev qualifier any more.
|
||||
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
||||
# whether the channel then has to be written separately.
|
||||
#
|
||||
# On a push the build writes the channel tag directly: the bytes came
|
||||
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
||||
# it behind.
|
||||
#
|
||||
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
||||
# refresh rebuilds against freshly resolved base images, and the web
|
||||
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
||||
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
||||
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
|
||||
# install requirements.txt, and a base bump changes neither. So
|
||||
# refreshed bytes have to be proven before :latest names them, and
|
||||
# proving needs a moment between "built" and "published" to occupy.
|
||||
# This is that moment; :latest goes on naming the build that works
|
||||
# until something says otherwise.
|
||||
#
|
||||
# `:refresh-candidate` is one moving ref per image, overwritten in
|
||||
# place, holding a build nobody is told to pull — the shape rule 145
|
||||
# already allows for :buildcache, not the per-build tag family that
|
||||
# milestone 318 withdrew.
|
||||
#
|
||||
# Both values are decided HERE, beside `hit`, for the reason the
|
||||
# force/schedule branch below gives: one step decides what this job
|
||||
# does. A promote condition derived independently could disagree with
|
||||
# the tag the build actually wrote.
|
||||
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
||||
echo "promote=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||
echo "promote=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||
# branching on the exit code would read "no label yet" as success.
|
||||
@@ -675,7 +738,7 @@ jobs:
|
||||
if [ "${FORCE:-false}" = "true" ]; then
|
||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||
echo "reuse: force_build set — building regardless"
|
||||
elif [ "${EVENT:-}" = "schedule" ]; then
|
||||
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||
echo "reuse: scheduled base refresh — building regardless"
|
||||
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||
@@ -764,6 +827,12 @@ jobs:
|
||||
|
||||
- name: Build and push web image
|
||||
if: steps.reuse.outputs.hit != 'true'
|
||||
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
||||
# it normalises the image config's `created` field and the history
|
||||
# timestamps rather than being consumed by the Dockerfile. See #3265
|
||||
# and the reuse step's `epoch` output.
|
||||
env:
|
||||
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
@@ -776,20 +845,17 @@ jobs:
|
||||
# invalidates, and the image genuinely rebuilds.
|
||||
#
|
||||
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
||||
# did NOT move, the build is ~13s and every content step reports
|
||||
# CACHED — but the channel tag STILL gets a new manifest digest.
|
||||
# buildkit mints a fresh image config each run, so identical layers
|
||||
# are republished under a new config blob. All three images moved
|
||||
# that way on 2026-08-30 with nothing whatsoever changed in them.
|
||||
# did NOT move, the build was ~13s with every content step CACHED —
|
||||
# and the channel tag STILL got a new manifest digest, because
|
||||
# buildkit stamps a fresh image config per run and republishes the
|
||||
# identical layers under it. All three images moved that way on
|
||||
# 2026-08-30 with nothing whatsoever changed in them.
|
||||
#
|
||||
# So a refresh currently rewrites :latest every Sunday whether or
|
||||
# not there is anything new in it, and :c-<sha> is handed a new
|
||||
# manifest to diverge from on the same cadence. Layers are shared,
|
||||
# so the storage cost is a config blob; the cost that matters is
|
||||
# that a digest change no longer MEANS anything. Tracked in #3265 —
|
||||
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
|
||||
# make "same source, same bytes" true and turn the no-op case into
|
||||
# a genuine no-op.
|
||||
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
||||
# content came from, the config is byte-identical across runs, so
|
||||
# the manifest digest is too and the push is a registry no-op. A
|
||||
# digest change means the content changed again, which is the only
|
||||
# thing a digest is any use for.
|
||||
#
|
||||
# What `pull` does NOT catch either: a Debian package update inside
|
||||
# the `apt-get install` layer while the base tag itself stands
|
||||
@@ -799,14 +865,14 @@ jobs:
|
||||
# churn #3265 is about.
|
||||
#
|
||||
# Only on the schedule. An ordinary push wants the cached base.
|
||||
pull: ${{ github.event_name == 'schedule' }}
|
||||
pull: ${{ env.IS_REFRESH == 'true' }}
|
||||
# ONE tag, the channel's. Every other tag is written by the step
|
||||
# below, registry-side. buildx here pushes the first tag to the
|
||||
# registry and then re-pushes the rest through the DOCKER driver,
|
||||
# out of a local image store a registry-direct build never filled —
|
||||
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
||||
# published perfectly well.
|
||||
tags: ${{ steps.reuse.outputs.channel_ref }}
|
||||
tags: ${{ steps.reuse.outputs.build_ref }}
|
||||
# The reuse key. Read back off the channel tag on the next push to
|
||||
# decide whether that push needs to build at all, so this is not
|
||||
# decoration — an unstamped image is one that will always rebuild.
|
||||
@@ -853,6 +919,77 @@ jobs:
|
||||
FC_CHANNEL=${{ steps.tag.outputs.channel }}
|
||||
FC_VERSION=${{ steps.reuse.outputs.version }}
|
||||
|
||||
# Point the channel tag at the candidate the refresh just built.
|
||||
#
|
||||
# Unconditional TODAY, so this milestone never leaves the refresh in a
|
||||
# state where it builds and publishes nothing. Step 4 wraps it in the
|
||||
# smoke suite's verdict; until then the scheduled path behaves exactly
|
||||
# as it did, just via two operations instead of one.
|
||||
#
|
||||
# NOT `imagetools create`. That wraps its source in an INDEX, and an
|
||||
# indexed channel tag is the one thing this pipeline cannot survive:
|
||||
# `.Image.Config.Labels` does not resolve through an index, so the
|
||||
# fc.revision the reuse check reads off the channel tag would come back
|
||||
# empty, every subsequent push would miss and rebuild, and nothing would
|
||||
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
|
||||
# and the only symptom was the bill. The repoint step below excludes its
|
||||
# own source tag for precisely this reason; a promote that re-introduced
|
||||
# the wrap through a different door would undo that care.
|
||||
#
|
||||
# A manifest PUT is what "make this tag name that image" means at the
|
||||
# registry level: the same bytes under the same media type, so the digest
|
||||
# is identical, the media type is preserved, and no layer moves.
|
||||
- name: Promote the refresh candidate to the channel
|
||||
if: steps.reuse.outputs.promote == 'true'
|
||||
env:
|
||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
||||
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
||||
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
run: |
|
||||
set -eu
|
||||
REPO=${IMAGE#git.fabledsword.com/}
|
||||
TAG=${CHANNEL_REF##*:}
|
||||
|
||||
# Registry auth is its own token exchange — the `docker login` above
|
||||
# authenticates the docker client, not curl. Deadline on every call
|
||||
# (rule 156): a registry that stops answering must fail this step,
|
||||
# not hang the weekly refresh until the job times out.
|
||||
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
|
||||
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
|
||||
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
|
||||
|
||||
# Ask for the image manifest media types ONLY. Offering the index
|
||||
# types too would let the registry hand back an index if one ever
|
||||
# existed at this tag, and we would faithfully copy the thing we are
|
||||
# trying not to create.
|
||||
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
|
||||
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
|
||||
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
|
||||
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
|
||||
test -n "$CT"
|
||||
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||
echo "promote: candidate is $SRC_DIGEST ($CT)"
|
||||
|
||||
curl -fsS --max-time 120 -X PUT \
|
||||
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
|
||||
--data-binary @manifest.json \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
|
||||
|
||||
# Read it back. A PUT that returned 2xx but landed something else is
|
||||
# exactly the silent-and-plausible failure this pipeline keeps
|
||||
# producing, and the check costs one request.
|
||||
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
|
||||
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
|
||||
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||
if [ "$NOW" != "$SRC_DIGEST" ]; then
|
||||
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "promote: $IMAGE:$TAG now names $NOW"
|
||||
|
||||
# Every tag but the channel's own is written HERE, registry-side,
|
||||
# whether or not a build ran. Each -t becomes another reference to the
|
||||
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
||||
@@ -957,7 +1094,7 @@ jobs:
|
||||
|
||||
# See sign-extension's copy for why this guard exists.
|
||||
- name: Guard — a scheduled run must have checked out main
|
||||
if: github.event_name == 'schedule'
|
||||
if: env.IS_REFRESH == 'true'
|
||||
run: |
|
||||
set -eu
|
||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||
@@ -1008,7 +1145,7 @@ jobs:
|
||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||
# Mirrors build-web's tag list and its schedule handling; see
|
||||
# the comments there.
|
||||
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
|
||||
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
|
||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||
@@ -1091,17 +1228,59 @@ jobs:
|
||||
# A scheduled refresh has to bypass reuse by construction: it
|
||||
# rebuilds the SAME source, so fc.revision always matches and the
|
||||
# check would skip every refresh there has ever been.
|
||||
EVENT: ${{ github.event_name }}
|
||||
run: |
|
||||
set -eu
|
||||
DERIVED=$(sh scripts/artifacts.sh revision ml)
|
||||
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# The build clock, pinned to the same commit (#3265). Without it
|
||||
# buildkit stamps the image config with the wall clock of the build,
|
||||
# so identical layers republish under a new config blob and the
|
||||
# channel tag gets a new manifest digest for no reason. Derived from
|
||||
# `newest()` like revision and version, so all three name one commit
|
||||
# and cannot drift apart.
|
||||
echo "epoch=$(sh scripts/artifacts.sh epoch ml)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||
# that is why the revision needs no -main/-dev qualifier any more.
|
||||
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
||||
# whether the channel then has to be written separately.
|
||||
#
|
||||
# On a push the build writes the channel tag directly: the bytes came
|
||||
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
||||
# it behind.
|
||||
#
|
||||
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
||||
# refresh rebuilds against freshly resolved base images, and the web
|
||||
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
||||
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
||||
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
|
||||
# install requirements.txt, and a base bump changes neither. So
|
||||
# refreshed bytes have to be proven before :latest names them, and
|
||||
# proving needs a moment between "built" and "published" to occupy.
|
||||
# This is that moment; :latest goes on naming the build that works
|
||||
# until something says otherwise.
|
||||
#
|
||||
# `:refresh-candidate` is one moving ref per image, overwritten in
|
||||
# place, holding a build nobody is told to pull — the shape rule 145
|
||||
# already allows for :buildcache, not the per-build tag family that
|
||||
# milestone 318 withdrew.
|
||||
#
|
||||
# Both values are decided HERE, beside `hit`, for the reason the
|
||||
# force/schedule branch below gives: one step decides what this job
|
||||
# does. A promote condition derived independently could disagree with
|
||||
# the tag the build actually wrote.
|
||||
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
||||
echo "promote=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||
echo "promote=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||
# branching on the exit code would read "no label yet" as success.
|
||||
@@ -1134,7 +1313,7 @@ jobs:
|
||||
if [ "${FORCE:-false}" = "true" ]; then
|
||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||
echo "reuse: force_build set — building regardless"
|
||||
elif [ "${EVENT:-}" = "schedule" ]; then
|
||||
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||
echo "reuse: scheduled base refresh — building regardless"
|
||||
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||
@@ -1147,6 +1326,12 @@ jobs:
|
||||
|
||||
- name: Build and push ml image
|
||||
if: steps.reuse.outputs.hit != 'true'
|
||||
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
||||
# it normalises the image config's `created` field and the history
|
||||
# timestamps rather than being consumed by the Dockerfile. See #3265
|
||||
# and the reuse step's `epoch` output.
|
||||
env:
|
||||
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
@@ -1159,20 +1344,17 @@ jobs:
|
||||
# invalidates, and the image genuinely rebuilds.
|
||||
#
|
||||
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
||||
# did NOT move, the build is ~13s and every content step reports
|
||||
# CACHED — but the channel tag STILL gets a new manifest digest.
|
||||
# buildkit mints a fresh image config each run, so identical layers
|
||||
# are republished under a new config blob. All three images moved
|
||||
# that way on 2026-08-30 with nothing whatsoever changed in them.
|
||||
# did NOT move, the build was ~13s with every content step CACHED —
|
||||
# and the channel tag STILL got a new manifest digest, because
|
||||
# buildkit stamps a fresh image config per run and republishes the
|
||||
# identical layers under it. All three images moved that way on
|
||||
# 2026-08-30 with nothing whatsoever changed in them.
|
||||
#
|
||||
# So a refresh currently rewrites :latest every Sunday whether or
|
||||
# not there is anything new in it, and :c-<sha> is handed a new
|
||||
# manifest to diverge from on the same cadence. Layers are shared,
|
||||
# so the storage cost is a config blob; the cost that matters is
|
||||
# that a digest change no longer MEANS anything. Tracked in #3265 —
|
||||
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
|
||||
# make "same source, same bytes" true and turn the no-op case into
|
||||
# a genuine no-op.
|
||||
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
||||
# content came from, the config is byte-identical across runs, so
|
||||
# the manifest digest is too and the push is a registry no-op. A
|
||||
# digest change means the content changed again, which is the only
|
||||
# thing a digest is any use for.
|
||||
#
|
||||
# What `pull` does NOT catch either: a Debian package update inside
|
||||
# the `apt-get install` layer while the base tag itself stands
|
||||
@@ -1182,14 +1364,14 @@ jobs:
|
||||
# churn #3265 is about.
|
||||
#
|
||||
# Only on the schedule. An ordinary push wants the cached base.
|
||||
pull: ${{ github.event_name == 'schedule' }}
|
||||
pull: ${{ env.IS_REFRESH == 'true' }}
|
||||
# ONE tag, the channel's. Every other tag is written by the step
|
||||
# below, registry-side. buildx here pushes the first tag to the
|
||||
# registry and then re-pushes the rest through the DOCKER driver,
|
||||
# out of a local image store a registry-direct build never filled —
|
||||
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
||||
# published perfectly well.
|
||||
tags: ${{ steps.reuse.outputs.channel_ref }}
|
||||
tags: ${{ steps.reuse.outputs.build_ref }}
|
||||
# The reuse key. Read back off the channel tag on the next push to
|
||||
# decide whether that push needs to build at all, so this is not
|
||||
# decoration — an unstamped image is one that will always rebuild.
|
||||
@@ -1230,6 +1412,77 @@ jobs:
|
||||
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
|
||||
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
|
||||
|
||||
# Point the channel tag at the candidate the refresh just built.
|
||||
#
|
||||
# Unconditional TODAY, so this milestone never leaves the refresh in a
|
||||
# state where it builds and publishes nothing. Step 4 wraps it in the
|
||||
# smoke suite's verdict; until then the scheduled path behaves exactly
|
||||
# as it did, just via two operations instead of one.
|
||||
#
|
||||
# NOT `imagetools create`. That wraps its source in an INDEX, and an
|
||||
# indexed channel tag is the one thing this pipeline cannot survive:
|
||||
# `.Image.Config.Labels` does not resolve through an index, so the
|
||||
# fc.revision the reuse check reads off the channel tag would come back
|
||||
# empty, every subsequent push would miss and rebuild, and nothing would
|
||||
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
|
||||
# and the only symptom was the bill. The repoint step below excludes its
|
||||
# own source tag for precisely this reason; a promote that re-introduced
|
||||
# the wrap through a different door would undo that care.
|
||||
#
|
||||
# A manifest PUT is what "make this tag name that image" means at the
|
||||
# registry level: the same bytes under the same media type, so the digest
|
||||
# is identical, the media type is preserved, and no layer moves.
|
||||
- name: Promote the refresh candidate to the channel
|
||||
if: steps.reuse.outputs.promote == 'true'
|
||||
env:
|
||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
||||
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
||||
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
run: |
|
||||
set -eu
|
||||
REPO=${IMAGE#git.fabledsword.com/}
|
||||
TAG=${CHANNEL_REF##*:}
|
||||
|
||||
# Registry auth is its own token exchange — the `docker login` above
|
||||
# authenticates the docker client, not curl. Deadline on every call
|
||||
# (rule 156): a registry that stops answering must fail this step,
|
||||
# not hang the weekly refresh until the job times out.
|
||||
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
|
||||
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
|
||||
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
|
||||
|
||||
# Ask for the image manifest media types ONLY. Offering the index
|
||||
# types too would let the registry hand back an index if one ever
|
||||
# existed at this tag, and we would faithfully copy the thing we are
|
||||
# trying not to create.
|
||||
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
|
||||
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
|
||||
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
|
||||
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
|
||||
test -n "$CT"
|
||||
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||
echo "promote: candidate is $SRC_DIGEST ($CT)"
|
||||
|
||||
curl -fsS --max-time 120 -X PUT \
|
||||
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
|
||||
--data-binary @manifest.json \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
|
||||
|
||||
# Read it back. A PUT that returned 2xx but landed something else is
|
||||
# exactly the silent-and-plausible failure this pipeline keeps
|
||||
# producing, and the check costs one request.
|
||||
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
|
||||
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
|
||||
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||
if [ "$NOW" != "$SRC_DIGEST" ]; then
|
||||
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "promote: $IMAGE:$TAG now names $NOW"
|
||||
|
||||
# Every tag but the channel's own is written HERE, registry-side,
|
||||
# whether or not a build ran. Each -t becomes another reference to the
|
||||
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
||||
@@ -1331,7 +1584,7 @@ jobs:
|
||||
|
||||
# See sign-extension's copy for why this guard exists.
|
||||
- name: Guard — a scheduled run must have checked out main
|
||||
if: github.event_name == 'schedule'
|
||||
if: env.IS_REFRESH == 'true'
|
||||
run: |
|
||||
set -eu
|
||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||
@@ -1377,7 +1630,7 @@ jobs:
|
||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||
# Mirrors build-web's tag list and its schedule handling; see
|
||||
# the comments there.
|
||||
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
|
||||
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
|
||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||
@@ -1460,17 +1713,59 @@ jobs:
|
||||
# A scheduled refresh has to bypass reuse by construction: it
|
||||
# rebuilds the SAME source, so fc.revision always matches and the
|
||||
# check would skip every refresh there has ever been.
|
||||
EVENT: ${{ github.event_name }}
|
||||
run: |
|
||||
set -eu
|
||||
DERIVED=$(sh scripts/artifacts.sh revision agent)
|
||||
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# The build clock, pinned to the same commit (#3265). Without it
|
||||
# buildkit stamps the image config with the wall clock of the build,
|
||||
# so identical layers republish under a new config blob and the
|
||||
# channel tag gets a new manifest digest for no reason. Derived from
|
||||
# `newest()` like revision and version, so all three name one commit
|
||||
# and cannot drift apart.
|
||||
echo "epoch=$(sh scripts/artifacts.sh epoch agent)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||
# that is why the revision needs no -main/-dev qualifier any more.
|
||||
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
||||
# whether the channel then has to be written separately.
|
||||
#
|
||||
# On a push the build writes the channel tag directly: the bytes came
|
||||
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
||||
# it behind.
|
||||
#
|
||||
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
||||
# refresh rebuilds against freshly resolved base images, and the web
|
||||
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
||||
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
||||
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
|
||||
# install requirements.txt, and a base bump changes neither. So
|
||||
# refreshed bytes have to be proven before :latest names them, and
|
||||
# proving needs a moment between "built" and "published" to occupy.
|
||||
# This is that moment; :latest goes on naming the build that works
|
||||
# until something says otherwise.
|
||||
#
|
||||
# `:refresh-candidate` is one moving ref per image, overwritten in
|
||||
# place, holding a build nobody is told to pull — the shape rule 145
|
||||
# already allows for :buildcache, not the per-build tag family that
|
||||
# milestone 318 withdrew.
|
||||
#
|
||||
# Both values are decided HERE, beside `hit`, for the reason the
|
||||
# force/schedule branch below gives: one step decides what this job
|
||||
# does. A promote condition derived independently could disagree with
|
||||
# the tag the build actually wrote.
|
||||
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
||||
echo "promote=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||
echo "promote=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||
# branching on the exit code would read "no label yet" as success.
|
||||
@@ -1503,7 +1798,7 @@ jobs:
|
||||
if [ "${FORCE:-false}" = "true" ]; then
|
||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||
echo "reuse: force_build set — building regardless"
|
||||
elif [ "${EVENT:-}" = "schedule" ]; then
|
||||
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||
echo "reuse: scheduled base refresh — building regardless"
|
||||
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||
@@ -1516,6 +1811,12 @@ jobs:
|
||||
|
||||
- name: Build and push agent image
|
||||
if: steps.reuse.outputs.hit != 'true'
|
||||
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
||||
# it normalises the image config's `created` field and the history
|
||||
# timestamps rather than being consumed by the Dockerfile. See #3265
|
||||
# and the reuse step's `epoch` output.
|
||||
env:
|
||||
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: agent
|
||||
@@ -1528,20 +1829,17 @@ jobs:
|
||||
# invalidates, and the image genuinely rebuilds.
|
||||
#
|
||||
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
||||
# did NOT move, the build is ~13s and every content step reports
|
||||
# CACHED — but the channel tag STILL gets a new manifest digest.
|
||||
# buildkit mints a fresh image config each run, so identical layers
|
||||
# are republished under a new config blob. All three images moved
|
||||
# that way on 2026-08-30 with nothing whatsoever changed in them.
|
||||
# did NOT move, the build was ~13s with every content step CACHED —
|
||||
# and the channel tag STILL got a new manifest digest, because
|
||||
# buildkit stamps a fresh image config per run and republishes the
|
||||
# identical layers under it. All three images moved that way on
|
||||
# 2026-08-30 with nothing whatsoever changed in them.
|
||||
#
|
||||
# So a refresh currently rewrites :latest every Sunday whether or
|
||||
# not there is anything new in it, and :c-<sha> is handed a new
|
||||
# manifest to diverge from on the same cadence. Layers are shared,
|
||||
# so the storage cost is a config blob; the cost that matters is
|
||||
# that a digest change no longer MEANS anything. Tracked in #3265 —
|
||||
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
|
||||
# make "same source, same bytes" true and turn the no-op case into
|
||||
# a genuine no-op.
|
||||
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
||||
# content came from, the config is byte-identical across runs, so
|
||||
# the manifest digest is too and the push is a registry no-op. A
|
||||
# digest change means the content changed again, which is the only
|
||||
# thing a digest is any use for.
|
||||
#
|
||||
# What `pull` does NOT catch either: a Debian package update inside
|
||||
# the `apt-get install` layer while the base tag itself stands
|
||||
@@ -1551,14 +1849,14 @@ jobs:
|
||||
# churn #3265 is about.
|
||||
#
|
||||
# Only on the schedule. An ordinary push wants the cached base.
|
||||
pull: ${{ github.event_name == 'schedule' }}
|
||||
pull: ${{ env.IS_REFRESH == 'true' }}
|
||||
# ONE tag, the channel's. Every other tag is written by the step
|
||||
# below, registry-side. buildx here pushes the first tag to the
|
||||
# registry and then re-pushes the rest through the DOCKER driver,
|
||||
# out of a local image store a registry-direct build never filled —
|
||||
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
||||
# published perfectly well.
|
||||
tags: ${{ steps.reuse.outputs.channel_ref }}
|
||||
tags: ${{ steps.reuse.outputs.build_ref }}
|
||||
# The reuse key. Read back off the channel tag on the next push to
|
||||
# decide whether that push needs to build at all, so this is not
|
||||
# decoration — an unstamped image is one that will always rebuild.
|
||||
@@ -1599,6 +1897,77 @@ jobs:
|
||||
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
|
||||
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
|
||||
|
||||
# Point the channel tag at the candidate the refresh just built.
|
||||
#
|
||||
# Unconditional TODAY, so this milestone never leaves the refresh in a
|
||||
# state where it builds and publishes nothing. Step 4 wraps it in the
|
||||
# smoke suite's verdict; until then the scheduled path behaves exactly
|
||||
# as it did, just via two operations instead of one.
|
||||
#
|
||||
# NOT `imagetools create`. That wraps its source in an INDEX, and an
|
||||
# indexed channel tag is the one thing this pipeline cannot survive:
|
||||
# `.Image.Config.Labels` does not resolve through an index, so the
|
||||
# fc.revision the reuse check reads off the channel tag would come back
|
||||
# empty, every subsequent push would miss and rebuild, and nothing would
|
||||
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
|
||||
# and the only symptom was the bill. The repoint step below excludes its
|
||||
# own source tag for precisely this reason; a promote that re-introduced
|
||||
# the wrap through a different door would undo that care.
|
||||
#
|
||||
# A manifest PUT is what "make this tag name that image" means at the
|
||||
# registry level: the same bytes under the same media type, so the digest
|
||||
# is identical, the media type is preserved, and no layer moves.
|
||||
- name: Promote the refresh candidate to the channel
|
||||
if: steps.reuse.outputs.promote == 'true'
|
||||
env:
|
||||
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
||||
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
||||
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||
ACTOR: ${{ github.actor }}
|
||||
run: |
|
||||
set -eu
|
||||
REPO=${IMAGE#git.fabledsword.com/}
|
||||
TAG=${CHANNEL_REF##*:}
|
||||
|
||||
# Registry auth is its own token exchange — the `docker login` above
|
||||
# authenticates the docker client, not curl. Deadline on every call
|
||||
# (rule 156): a registry that stops answering must fail this step,
|
||||
# not hang the weekly refresh until the job times out.
|
||||
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
|
||||
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
|
||||
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
|
||||
|
||||
# Ask for the image manifest media types ONLY. Offering the index
|
||||
# types too would let the registry hand back an index if one ever
|
||||
# existed at this tag, and we would faithfully copy the thing we are
|
||||
# trying not to create.
|
||||
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
|
||||
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
|
||||
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
|
||||
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
|
||||
test -n "$CT"
|
||||
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||
echo "promote: candidate is $SRC_DIGEST ($CT)"
|
||||
|
||||
curl -fsS --max-time 120 -X PUT \
|
||||
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
|
||||
--data-binary @manifest.json \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
|
||||
|
||||
# Read it back. A PUT that returned 2xx but landed something else is
|
||||
# exactly the silent-and-plausible failure this pipeline keeps
|
||||
# producing, and the check costs one request.
|
||||
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
|
||||
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
|
||||
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||
if [ "$NOW" != "$SRC_DIGEST" ]; then
|
||||
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "promote: $IMAGE:$TAG now names $NOW"
|
||||
|
||||
# Every tag but the channel's own is written HERE, registry-side,
|
||||
# whether or not a build ran. Each -t becomes another reference to the
|
||||
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
# FabledCurator
|
||||
|
||||
<!-- overview:start -->
|
||||
Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven
|
||||
downloading in one application. Part of the FabledSword family.
|
||||
|
||||
@@ -19,13 +20,15 @@ through afterwards.
|
||||
characters, find near-duplicates and power similarity search. Suggestions are
|
||||
reviewable — it proposes, you confirm, and it learns which proposals you keep
|
||||
rejecting.
|
||||
- **Importing.** Ingests an existing library from disk, deduplicates it by
|
||||
content hash, and reads metadata sidecars.
|
||||
- **Deduplication and provenance.** Everything that arrives is hashed and
|
||||
deduplicated by content, metadata sidecars are read wherever the source
|
||||
writes them, and every file keeps a record of where it came from.
|
||||
- **Maintenance.** Backups, library audits, thumbnail and embedding backfills,
|
||||
orphan cleanup — all from the UI, all as background jobs you can watch.
|
||||
|
||||
Everything is configured from the Settings UI and stored in the database. There
|
||||
is no config file to edit beyond a handful of bootstrap environment variables.
|
||||
<!-- overview:end -->
|
||||
|
||||
## Before you expose it
|
||||
|
||||
|
||||
+15
-8
@@ -198,14 +198,21 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
|
||||
refresh from being undone.
|
||||
- **`pull: true` on the scheduled path only** is the mechanism: a moved base
|
||||
tag changes the `FROM` layer's cache key and everything above it rebuilds.
|
||||
**It does not currently make the unmoved case free.** Measured on the first
|
||||
real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
|
||||
the bases resolved to unchanged digests, yet all three `:latest` tags got a
|
||||
NEW manifest digest, because buildkit mints a fresh image config per run and
|
||||
republishes identical layers under it. So `:latest` is rewritten weekly
|
||||
whether or not anything changed, and `:c-<sha>` is handed a new manifest to
|
||||
diverge from on the same cadence — a digest change stops meaning anything.
|
||||
Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
|
||||
It did not always make the unmoved case free. Measured on the first real
|
||||
fire (run 4934, 2026-08-30): every content step reported `CACHED` and the
|
||||
bases resolved to unchanged digests, yet all three `:latest` tags got a NEW
|
||||
manifest digest, because buildkit stamps a fresh image config per run and
|
||||
republishes identical layers under it — so `:latest` was rewritten weekly
|
||||
whether or not anything changed, and a digest change stopped meaning
|
||||
anything (#3265).
|
||||
- **`SOURCE_DATE_EPOCH` is what makes it free.** Set on each build step from
|
||||
`artifacts.sh epoch <artifact>` — the unix timestamp of the same commit
|
||||
`revision` and `version` name, so all three are views of one `newest()`
|
||||
lookup and cannot drift into disagreeing. With the config's `created` field
|
||||
and history timestamps pinned to the content rather than to the wall clock,
|
||||
identical source produces an identical manifest digest and the push is a
|
||||
registry no-op. That restores the property the whole scheme rests on: a
|
||||
channel tag's digest changes when, and only when, its content does.
|
||||
Separately not caught: a Debian package update inside the `apt-get install`
|
||||
layer while the base tag stands still — a lag rather than a hole, since the
|
||||
official python/cuda images rebuild with those updates baked in.
|
||||
|
||||
+22
-1
@@ -90,7 +90,7 @@ DERIVER='scripts/artifacts.sh'
|
||||
|
||||
|
||||
usage() {
|
||||
echo "usage: artifacts.sh {paths|revision|version} {web|ml|agent|extension}" >&2
|
||||
echo "usage: artifacts.sh {paths|revision|version|epoch} {web|ml|agent|extension}" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
@@ -149,6 +149,26 @@ cmd_revision() {
|
||||
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
|
||||
}
|
||||
|
||||
# The BUILD CLOCK: the same commit's unix timestamp, for SOURCE_DATE_EPOCH.
|
||||
#
|
||||
# buildkit stamps the image config's `created` field and every history entry
|
||||
# with the wall clock of the build unless this is set, so two builds of
|
||||
# identical source produce different config blobs and therefore different
|
||||
# manifest digests. That is #3265: the weekly refresh republished all three
|
||||
# `:latest` tags on 2026-08-30 with every content step CACHED and the bases
|
||||
# resolved to unchanged digests — nothing was different, and the digest moved
|
||||
# anyway. A digest that changes on a calendar cannot also mean "the content
|
||||
# changed", which is the only thing anyone wants it for.
|
||||
#
|
||||
# It is the same commit `revision` and `version` name — deliberately, and this
|
||||
# is the point of routing it through `newest()` rather than taking git's word
|
||||
# separately. Three values derived from three lookups can disagree; three
|
||||
# views of one lookup cannot. Note #3127 §2 is the record of what a second
|
||||
# clock costs.
|
||||
cmd_epoch() {
|
||||
echo "$(newest "$1")" | cut -d' ' -f1
|
||||
}
|
||||
|
||||
# The VERSION: `YYYY.MM.DD.HHMM`, zero-padded, UTC. One shape across the whole
|
||||
# family (note #3127 §1, rule 148) — the number an instance reports about
|
||||
# itself, and, with a `v` in front, the release tag naming the same build.
|
||||
@@ -197,5 +217,6 @@ case "$1" in
|
||||
paths) cmd_paths "$2" ;;
|
||||
revision) cmd_revision "$2" ;;
|
||||
version) cmd_version "$2" ;;
|
||||
epoch) cmd_epoch "$2" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
|
||||
+118
-14
@@ -33,6 +33,32 @@ history. Ancestry is immune to the shape change, and it is also the more honest
|
||||
question: "what is in this that was not in the last one" IS a reachability
|
||||
question.
|
||||
|
||||
Ancestry alone is not enough, though, and milestone 328 is where that showed.
|
||||
The 28 `v26.*` tags are still in the repo — the operator kept them as history
|
||||
when their releases were deleted — so `--match v*` walks straight back to
|
||||
`v26.06.04.0` and reports 533 commits. That span is not a changelog: nobody has
|
||||
run `v26.06.04.0`, its release page no longer exists to compare against, and
|
||||
the 200 lines that survive truncation are precisely the internal build-out that
|
||||
milestone 328 exists to stop shipping. So the match is `v[0-9][0-9][0-9][0-9].*`
|
||||
— rule 148's four-digit-year shape — which is exactly the set of tags that name
|
||||
a release a reader could have been running. A pre-convention tag is history,
|
||||
not a predecessor.
|
||||
|
||||
## The first release has no changelog, and should not pretend to
|
||||
|
||||
Once the match is narrowed, the first rule-148 tag reaches no predecessor at
|
||||
all, and the old fallback — diff against the whole history — is worse than the
|
||||
problem it replaced. The honest content for a release nobody has a previous
|
||||
version of is what the thing IS.
|
||||
|
||||
So a release with no reachable predecessor renders the product overview instead
|
||||
of a commit list. It is read out of README.md between `<!-- overview:start -->`
|
||||
and `<!-- overview:end -->` rather than written here, for the same reason the
|
||||
changelog is derived: two hand-maintained descriptions of one product drift,
|
||||
and nothing ever catches it. The release page and the repo front page are one
|
||||
source. Every later release goes back to being a changelog, which is what §5 of
|
||||
note #3127 says a release is for.
|
||||
|
||||
## Re-runs update, they do not fall through
|
||||
|
||||
Note #3127 §6.7: a publisher that POSTs and recovers the id from a `409` never
|
||||
@@ -91,18 +117,45 @@ def git_ok(*args: str) -> str | None:
|
||||
|
||||
|
||||
def previous_tag(ref: str, tag: str | None) -> str | None:
|
||||
"""The most recent `v*` tag reachable from `ref`, excluding `tag` itself.
|
||||
"""The most recent rule-148 tag reachable from `ref`, excluding `tag` itself.
|
||||
|
||||
`--exclude` rather than `<ref>^` so this is the same call whether or not
|
||||
`ref` is the tag being released — and so it does not blow up on a root
|
||||
commit that has no parent to walk to.
|
||||
|
||||
The glob deliberately does NOT match the old `v26.*` tags. They are kept as
|
||||
history and their releases are gone, so naming one as the predecessor emits
|
||||
a span nobody can look up. See the module docstring.
|
||||
"""
|
||||
args = ["describe", "--tags", "--abbrev=0", "--match", "v*"]
|
||||
args = ["describe", "--tags", "--abbrev=0", "--match", "v[0-9][0-9][0-9][0-9].*"]
|
||||
if tag:
|
||||
args += ["--exclude", tag]
|
||||
return git_ok(*args, ref)
|
||||
|
||||
|
||||
def product_overview() -> str | None:
|
||||
"""The product description, lifted verbatim from README.md.
|
||||
|
||||
Returns None if the markers are absent or empty — a missing overview is
|
||||
reported as a note and the release still publishes, on the same reasoning
|
||||
as cross_checks(): the release is the useful object even when one part of
|
||||
the derivation could not run.
|
||||
"""
|
||||
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
with open(os.path.join(root, "README.md"), encoding="utf-8") as fh:
|
||||
readme = fh.read()
|
||||
except OSError:
|
||||
return None
|
||||
match = re.search(
|
||||
r"<!--\s*overview:start\s*-->(.*?)<!--\s*overview:end\s*-->",
|
||||
readme, re.S,
|
||||
)
|
||||
if not match:
|
||||
return None
|
||||
return match.group(1).strip() or None
|
||||
|
||||
|
||||
def commits(previous: str | None, ref: str) -> list[str]:
|
||||
"""The subjects between the previous release and this one.
|
||||
|
||||
@@ -125,7 +178,10 @@ def truncate(log: list[str]) -> tuple[list[str], str | None]:
|
||||
)
|
||||
|
||||
|
||||
def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list[str]) -> str:
|
||||
def render(
|
||||
tag: str, sha: str, previous: str | None, log: list[str], notes: list[str],
|
||||
overview: str | None,
|
||||
) -> str:
|
||||
short = sha[:7]
|
||||
parts = []
|
||||
|
||||
@@ -135,6 +191,25 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
|
||||
# is the failure this whole milestone is about.
|
||||
parts.append("\n".join(f"> **Note:** {n}" for n in notes))
|
||||
|
||||
# No predecessor means nobody reading this has run an earlier one, so the
|
||||
# release describes the product rather than a diff. The overview is
|
||||
# README.md's own words — see the module docstring on why it is not
|
||||
# written here.
|
||||
if previous is None and overview:
|
||||
parts.append(overview)
|
||||
parts.append(
|
||||
"## Installing\n\n"
|
||||
"```\ncurl -O https://git.fabledsword.com/bvandeusen/FabledCurator/raw/"
|
||||
f"tag/{tag}/docker-compose.yml\ncurl -O https://git.fabledsword.com/"
|
||||
f"bvandeusen/FabledCurator/raw/tag/{tag}/.env.example\n"
|
||||
"mv .env.example .env # then set SECRET_KEY, DB_PASSWORD\n"
|
||||
"docker compose -f docker-compose.yml up -d\n```\n\n"
|
||||
"**Read \"Before you expose it\" in the README first.** FabledCurator "
|
||||
"has no login, and it stores live platform session cookies for "
|
||||
"accounts that usually have a payment method attached. Bind it to a "
|
||||
"network you trust."
|
||||
)
|
||||
|
||||
parts.append(
|
||||
f"Built from `{short}`. The rollback unit is the immutable `:c-` tag "
|
||||
f"(rule 145) — these three move together:\n\n```\n"
|
||||
@@ -142,7 +217,19 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
|
||||
+ "\n```"
|
||||
)
|
||||
|
||||
heading = f"## Changes since {previous}" if previous else "## Changes"
|
||||
if previous is None:
|
||||
# Deliberately NOT a commit list. The alternative is the whole history
|
||||
# truncated to MAX_COMMITS, which is 200 lines of internal build-out
|
||||
# presented to someone who has never seen this project.
|
||||
parts.append(
|
||||
"---\n\n_First release under rule 148's `vYYYY.MM.DD.HHMM` shape, so "
|
||||
"there is no predecessor to diff against and no changelog to derive. "
|
||||
"The description above is README.md's, quoted at publish time. Later "
|
||||
"releases carry the commits since the previous one._"
|
||||
)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
heading = f"## Changes since {previous}"
|
||||
if log:
|
||||
parts.append(heading + "\n\n" + "\n".join(f"- {line}" for line in log))
|
||||
else:
|
||||
@@ -152,10 +239,9 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
|
||||
"names the same source under a new name._"
|
||||
)
|
||||
|
||||
span = f"{previous}..{tag}" if previous else tag
|
||||
parts.append(
|
||||
f"---\n\n_Derived at publish time from `git log --no-merges {span}`. "
|
||||
f"Nothing here is hand-maintained._"
|
||||
f"---\n\n_Derived at publish time from "
|
||||
f"`git log --no-merges {previous}..{tag}`. Nothing here is hand-maintained._"
|
||||
)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
@@ -289,13 +375,31 @@ def main() -> None:
|
||||
for note in notes:
|
||||
print(f"release: NOTE {note}")
|
||||
|
||||
log = commits(previous, ref)
|
||||
print(f"release: {len(log)} non-merge commits in the span")
|
||||
log, overflow = truncate(log)
|
||||
if overflow:
|
||||
print(f"release: NOTE {overflow}")
|
||||
notes.append(overflow)
|
||||
body = render(tag or ref, sha, previous, log, notes)
|
||||
# A first release renders the overview instead of a changelog, so the
|
||||
# commit walk is skipped entirely rather than computed and discarded —
|
||||
# `commits(None, ref)` is the whole history and there is no reason to ask
|
||||
# for it.
|
||||
overview = None
|
||||
log: list[str] = []
|
||||
if previous is None:
|
||||
overview = product_overview()
|
||||
if overview is None:
|
||||
note = (
|
||||
"No `<!-- overview:start -->` block found in README.md, so this "
|
||||
"first release has no product description. Published anyway; add "
|
||||
"the markers and re-run the workflow to fill it in."
|
||||
)
|
||||
print(f"release: NOTE {note}")
|
||||
notes.append(note)
|
||||
print("release: no rule-148 predecessor — rendering the product overview")
|
||||
else:
|
||||
log = commits(previous, ref)
|
||||
print(f"release: {len(log)} non-merge commits in the span")
|
||||
log, overflow = truncate(log)
|
||||
if overflow:
|
||||
print(f"release: NOTE {overflow}")
|
||||
notes.append(overflow)
|
||||
body = render(tag or ref, sha, previous, log, notes, overview)
|
||||
|
||||
if args.dry_run or not tag:
|
||||
print("--- body ---")
|
||||
|
||||
@@ -234,3 +234,56 @@ def test_version_and_revision_describe_the_same_commit(artifact):
|
||||
f"in AMO_UNPADDED may differ here."
|
||||
)
|
||||
assert sha.startswith(revision(artifact))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifact", ARTIFACTS)
|
||||
def test_epoch_is_the_same_commit_the_version_names(artifact):
|
||||
"""The build clock and the version must be one lookup, not two.
|
||||
|
||||
`epoch` feeds SOURCE_DATE_EPOCH, which decides the image config's bytes and
|
||||
therefore the manifest digest; `version` is what the instance reports about
|
||||
itself. If they could name different commits, an image would be stamped
|
||||
reproducibly against one commit while claiming to be another — and both
|
||||
values would look perfectly well-formed, exactly like the divergence the
|
||||
test above guards.
|
||||
|
||||
They cannot, because `cmd_epoch` and `cmd_version` are two fields of one
|
||||
`newest()` result. This pins that they stay that way: a future refactor
|
||||
that gave epoch its own `git log` would pass every other test here.
|
||||
"""
|
||||
epoch = artifacts("epoch", artifact).strip()
|
||||
assert epoch.isdigit(), f"{artifact} epoch is {epoch!r}, not a unix timestamp"
|
||||
|
||||
sha = newest_by_commit_time(artifact)
|
||||
committed = subprocess.run(
|
||||
["git", "show", "-s", "--format=%ct", sha],
|
||||
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||
).stdout.strip()
|
||||
assert epoch == committed, (
|
||||
f"{artifact} derives epoch {epoch}, but its newest shipped commit "
|
||||
f"{sha[:12]} was committed at {committed}. SOURCE_DATE_EPOCH would "
|
||||
f"pin the image config to a commit the version does not name."
|
||||
)
|
||||
|
||||
# And the two renderings must agree, which is the property that actually
|
||||
# matters at build time: same commit in, same digest and same reported
|
||||
# version out.
|
||||
rendered = subprocess.run(
|
||||
["git", "show", "-s", "--format=%cd", "--date=format-local:%Y.%m.%d.%H%M", sha],
|
||||
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||
env={"TZ": "UTC", "PATH": os.environ.get("PATH", "")},
|
||||
).stdout.strip()
|
||||
assert segments(artifacts("version", artifact).strip()) == segments(rendered)
|
||||
|
||||
|
||||
def test_epoch_is_stable_across_calls():
|
||||
"""SOURCE_DATE_EPOCH's entire job is to be the same on the next build.
|
||||
|
||||
A value that moved between two invocations on one unchanged checkout would
|
||||
reintroduce #3265 through the very mechanism meant to close it, and the
|
||||
symptom would be indistinguishable: a digest that changes for no reason.
|
||||
"""
|
||||
for artifact in ARTIFACTS:
|
||||
first = artifacts("epoch", artifact).strip()
|
||||
second = artifacts("epoch", artifact).strip()
|
||||
assert first == second, f"{artifact} epoch moved: {first} then {second}"
|
||||
|
||||
+101
-17
@@ -24,12 +24,36 @@ SCRIPT = ROOT / "scripts" / "release_notes.py"
|
||||
|
||||
|
||||
def notes(*args: str, cwd: Path | None = None) -> str:
|
||||
"""Run the script the way release.yml does.
|
||||
|
||||
A synthetic repo runs its OWN copy of the script, because the overview is
|
||||
read relative to `__file__` rather than to the cwd — which is right in
|
||||
production (release.yml checks out the tag, so the script IS the tagged
|
||||
tree's copy) and would otherwise make every synthetic repo silently quote
|
||||
FabledCurator's real README.
|
||||
"""
|
||||
root = cwd or ROOT
|
||||
script = root / "scripts" / "release_notes.py"
|
||||
return subprocess.run(
|
||||
["python3", str(SCRIPT), "--dry-run", *args],
|
||||
capture_output=True, text=True, check=True, cwd=cwd or ROOT,
|
||||
["python3", str(script if script.exists() else SCRIPT), "--dry-run", *args],
|
||||
capture_output=True, text=True, check=True, cwd=root,
|
||||
).stdout
|
||||
|
||||
|
||||
OVERVIEW_TEXT = "A synthetic product, described once."
|
||||
|
||||
|
||||
def install_script(repo: Path, *, overview: bool = True) -> None:
|
||||
"""Give a synthetic repo the script and a README to quote."""
|
||||
(repo / "scripts").mkdir(exist_ok=True)
|
||||
(repo / "scripts" / "release_notes.py").write_text(SCRIPT.read_text())
|
||||
(repo / "scripts" / "artifacts.sh").write_text("#!/bin/sh\nexit 1\n")
|
||||
readme = "# Synthetic\n\n"
|
||||
if overview:
|
||||
readme += f"<!-- overview:start -->\n{OVERVIEW_TEXT}\n<!-- overview:end -->\n"
|
||||
(repo / "README.md").write_text(readme)
|
||||
|
||||
|
||||
def body_of(out: str) -> str:
|
||||
assert "--- body ---" in out, f"no body was rendered:\n{out}"
|
||||
return out.split("--- body ---", 1)[1]
|
||||
@@ -56,9 +80,10 @@ def shaped_history(tmp_path: Path) -> Path:
|
||||
repo = tmp_path / "shaped"
|
||||
repo.mkdir()
|
||||
git(repo, "init", "-q", "-b", "main")
|
||||
install_script(repo)
|
||||
for i, tag in enumerate(("v26.06.04.0", "v2026.08.28.2208", "v2026.08.29.1000")):
|
||||
(repo / "f.txt").write_text(f"{i}\n")
|
||||
git(repo, "add", "f.txt")
|
||||
git(repo, "add", "-A")
|
||||
git(repo, "commit", "-q", "-m", f"work landing in {tag}")
|
||||
git(repo, "tag", tag)
|
||||
# One more commit and a merge, so the merge-exclusion test has something to
|
||||
@@ -109,12 +134,57 @@ def test_merges_are_excluded_so_the_list_is_the_work(shaped_history):
|
||||
assert "Merge pull request #999" not in body
|
||||
|
||||
|
||||
def test_the_first_release_still_renders_with_nothing_behind_it(shaped_history):
|
||||
"""No previous tag is reachable from the oldest one. That is a real state,
|
||||
not an error, and it must not take the release down with it."""
|
||||
out = notes("v26.06.04.0", cwd=shaped_history)
|
||||
def test_a_pre_convention_tag_is_history_not_a_predecessor(shaped_history):
|
||||
"""The defect milestone 328 hit, and the reason the match glob narrowed.
|
||||
|
||||
The 28 `v26.*` tags are kept as history while their releases were deleted.
|
||||
Ancestry alone happily names `v26.06.04.0` as the predecessor of the first
|
||||
rule-148 tag — and then the body offers "changes since" a release that no
|
||||
longer exists, over a span (533 commits in the real repo) that is the
|
||||
internal build-out this milestone exists to stop publishing.
|
||||
|
||||
Reachable is not the same as comparable. Only a `vYYYY.` tag names a
|
||||
release a reader could have been running.
|
||||
"""
|
||||
out = notes("v2026.08.28.2208", cwd=shaped_history)
|
||||
assert "previous=<none>" in out
|
||||
assert "## Changes" in body_of(out)
|
||||
assert "v26.06.04.0" not in body_of(out)
|
||||
|
||||
|
||||
def test_the_first_release_describes_the_product_instead_of_diffing(shaped_history):
|
||||
"""No predecessor means nobody reading has run an earlier version, so a
|
||||
changelog has no referent. The alternative the script used to take — diff
|
||||
against the whole history, truncated — puts 200 lines of internal build-out
|
||||
in front of someone meeting the project for the first time."""
|
||||
body = body_of(notes("v2026.08.28.2208", cwd=shaped_history))
|
||||
assert OVERVIEW_TEXT in body
|
||||
assert "## Changes" not in body
|
||||
assert not [ln for ln in body.split("\n") if ln.startswith("- work landing")]
|
||||
|
||||
|
||||
def test_the_overview_is_readmes_words_not_a_second_copy(shaped_history):
|
||||
"""Two hand-maintained descriptions of one product drift and nothing
|
||||
catches it. The release page quotes README.md so there is one source."""
|
||||
readme = (shaped_history / "README.md").read_text()
|
||||
assert OVERVIEW_TEXT in readme
|
||||
assert OVERVIEW_TEXT in body_of(notes("v2026.08.28.2208", cwd=shaped_history))
|
||||
|
||||
|
||||
def test_a_missing_overview_block_is_reported_and_still_publishes(tmp_path):
|
||||
"""Same reasoning as cross_checks(): the release is the useful object even
|
||||
when part of the derivation could not run. Say what is missing, publish
|
||||
anyway — do not leave the operator with a tag and no release."""
|
||||
repo = tmp_path / "no-markers"
|
||||
repo.mkdir()
|
||||
git(repo, "init", "-q", "-b", "main")
|
||||
install_script(repo, overview=False)
|
||||
git(repo, "add", "-A")
|
||||
git(repo, "commit", "-q", "-m", "first")
|
||||
git(repo, "tag", "v2026.09.01.1200")
|
||||
|
||||
out = notes("v2026.09.01.1200", cwd=repo)
|
||||
assert "No `<!-- overview:start -->` block found" in out
|
||||
assert "No `<!-- overview:start -->` block found" in body_of(out)
|
||||
|
||||
|
||||
def test_a_non_tag_ref_renders_but_refuses_to_claim_it_published():
|
||||
@@ -135,15 +205,29 @@ def test_the_rollback_refs_name_all_three_images():
|
||||
assert f"bvandeusen/{image}:c-" in body, f"{image} missing from the rollback refs"
|
||||
|
||||
|
||||
def test_an_unbounded_span_is_truncated_and_says_so():
|
||||
"""With no reachable previous tag the span is the whole history. Emitting
|
||||
eleven hundred lines would bury the one line explaining why there are
|
||||
eleven hundred of them, so the cap is part of the message, not a silent
|
||||
slice."""
|
||||
out = notes("HEAD")
|
||||
if "previous=<none>" not in out:
|
||||
pytest.skip("a previous tag is reachable from HEAD in this checkout")
|
||||
def test_a_long_span_between_two_releases_is_truncated_and_says_so(tmp_path):
|
||||
"""The cap is still reachable, just not by the route it used to be.
|
||||
|
||||
It no longer fires on "no predecessor" — that renders the overview now.
|
||||
What it still guards is two real releases far enough apart that the list
|
||||
stops being something anyone reads, which is the ordinary case for a
|
||||
project that cuts a bookmark twice a year. The cap is part of the message,
|
||||
not a silent slice.
|
||||
"""
|
||||
repo = tmp_path / "long"
|
||||
repo.mkdir()
|
||||
git(repo, "init", "-q", "-b", "main")
|
||||
install_script(repo)
|
||||
git(repo, "add", "-A")
|
||||
git(repo, "commit", "-q", "-m", "scaffold")
|
||||
git(repo, "tag", "v2026.01.01.0000")
|
||||
for i in range(205):
|
||||
git(repo, "commit", "-q", "--allow-empty", "-m", f"fix: change {i}")
|
||||
git(repo, "tag", "v2026.07.01.0000")
|
||||
|
||||
out = notes("v2026.07.01.0000", cwd=repo)
|
||||
assert "previous=v2026.01.01.0000" in out
|
||||
body = body_of(out)
|
||||
listed = [ln for ln in body.split("\n") if ln.startswith("- ")]
|
||||
assert len(listed) <= 200
|
||||
assert len(listed) == 200
|
||||
assert "more than a changelog is for" in body
|
||||
|
||||
Reference in New Issue
Block a user