Make a digest mean something again, and give the refresh somewhere to stand #247
+428
-59
@@ -44,6 +44,10 @@ on:
|
|||||||
description: 'Rebuild every image even if the published revision matches'
|
description: 'Rebuild every image even if the published revision matches'
|
||||||
type: boolean
|
type: boolean
|
||||||
default: false
|
default: false
|
||||||
|
refresh:
|
||||||
|
description: 'Behave as the weekly base refresh: build main against fresh bases, publish through the candidate tag'
|
||||||
|
type: boolean
|
||||||
|
default: false
|
||||||
|
|
||||||
# The base-image refresh (milestone 326 step 4, #3154).
|
# The base-image refresh (milestone 326 step 4, #3154).
|
||||||
#
|
#
|
||||||
@@ -72,8 +76,25 @@ on:
|
|||||||
# Deriving it per job invites the two halves to disagree: sign-extension would
|
# Deriving it per job invites the two halves to disagree: sign-extension would
|
||||||
# derive dev's extension version while build-web bundled main's, and the
|
# derive dev's extension version while build-web bundled main's, and the
|
||||||
# release download would 404 on a version that exists perfectly well.
|
# release download would 404 on a version that exists perfectly well.
|
||||||
|
# IS THIS A BASE REFRESH? Asked in five places and previously spelled five
|
||||||
|
# ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in
|
||||||
|
# one shell, an `EVENT:` env passed into another, and a bare expression on
|
||||||
|
# `pull:`. Five spellings of one fact is how half of them come to disagree
|
||||||
|
# after somebody adds a sixth trigger.
|
||||||
|
#
|
||||||
|
# The `refresh` dispatch input is here so this path can be EXERCISED. A weekly
|
||||||
|
# cron is otherwise testable once a week, which is not a cadence anything can
|
||||||
|
# be developed against — the same reason `force_build` exists (#3252, added to
|
||||||
|
# confirm #3190 was gone rather than wait for it to recur). It is also what
|
||||||
|
# makes the milestone-362 gate verifiable at all: a gate has to be watched
|
||||||
|
# rejecting something before anyone can believe it is wired up.
|
||||||
|
#
|
||||||
|
# Note this is a STRING comparison, not a boolean. Forgejo delivers
|
||||||
|
# workflow_dispatch inputs as strings, so `inputs.refresh` is 'true'/'false'
|
||||||
|
# and `&&` on it would treat the string 'false' as truthy.
|
||||||
env:
|
env:
|
||||||
BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
|
IS_REFRESH: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'true' || 'false' }}
|
||||||
|
BUILD_REF: ${{ (github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'main' || github.ref }}
|
||||||
|
|
||||||
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
|
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
|
||||||
# - write:package, read:package (for docker push to git.fabledsword.com)
|
# - write:package, read:package (for docker push to git.fabledsword.com)
|
||||||
@@ -143,7 +164,7 @@ jobs:
|
|||||||
# evaluate — this file already gates steps on it — so the guard cannot
|
# evaluate — this file already gates steps on it — so the guard cannot
|
||||||
# be disabled by the same uncertainty it exists to cover.
|
# be disabled by the same uncertainty it exists to cover.
|
||||||
- name: Guard — a scheduled run must have checked out main
|
- name: Guard — a scheduled run must have checked out main
|
||||||
if: github.event_name == 'schedule'
|
if: env.IS_REFRESH == 'true'
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||||
@@ -437,7 +458,7 @@ jobs:
|
|||||||
|
|
||||||
# See sign-extension's copy for why this guard exists.
|
# See sign-extension's copy for why this guard exists.
|
||||||
- name: Guard — a scheduled run must have checked out main
|
- name: Guard — a scheduled run must have checked out main
|
||||||
if: github.event_name == 'schedule'
|
if: env.IS_REFRESH == 'true'
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||||
@@ -528,7 +549,7 @@ jobs:
|
|||||||
# Checked BEFORE the ref test, not after: a scheduled run's
|
# Checked BEFORE the ref test, not after: a scheduled run's
|
||||||
# GITHUB_REF is the default branch (dev), so the main test would
|
# GITHUB_REF is the default branch (dev), so the main test would
|
||||||
# never fire on it.
|
# never fire on it.
|
||||||
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
@@ -628,7 +649,6 @@ jobs:
|
|||||||
# A scheduled refresh has to bypass reuse by construction: it
|
# A scheduled refresh has to bypass reuse by construction: it
|
||||||
# rebuilds the SAME source, so fc.revision always matches and the
|
# rebuilds the SAME source, so fc.revision always matches and the
|
||||||
# check would skip every refresh there has ever been.
|
# check would skip every refresh there has ever been.
|
||||||
EVENT: ${{ github.event_name }}
|
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
DERIVED=$(sh scripts/artifacts.sh revision web)
|
DERIVED=$(sh scripts/artifacts.sh revision web)
|
||||||
@@ -638,11 +658,54 @@ jobs:
|
|||||||
# adds no variability the reuse check would have to account for.
|
# adds no variability the reuse check would have to account for.
|
||||||
echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT"
|
echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# The build clock, pinned to the same commit (#3265). Without it
|
||||||
|
# buildkit stamps the image config with the wall clock of the build,
|
||||||
|
# so identical layers republish under a new config blob and the
|
||||||
|
# channel tag gets a new manifest digest for no reason. Derived from
|
||||||
|
# `newest()` like revision and version, so all three name one commit
|
||||||
|
# and cannot drift apart.
|
||||||
|
echo "epoch=$(sh scripts/artifacts.sh epoch web)" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# The moving tag for this channel. Which tag we ask IS the channel —
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||||
# that is why the revision needs no -main/-dev qualifier any more.
|
# that is why the revision needs no -main/-dev qualifier any more.
|
||||||
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||||
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
||||||
|
# whether the channel then has to be written separately.
|
||||||
|
#
|
||||||
|
# On a push the build writes the channel tag directly: the bytes came
|
||||||
|
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
||||||
|
# it behind.
|
||||||
|
#
|
||||||
|
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
||||||
|
# refresh rebuilds against freshly resolved base images, and the web
|
||||||
|
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
||||||
|
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
||||||
|
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
|
||||||
|
# install requirements.txt, and a base bump changes neither. So
|
||||||
|
# refreshed bytes have to be proven before :latest names them, and
|
||||||
|
# proving needs a moment between "built" and "published" to occupy.
|
||||||
|
# This is that moment; :latest goes on naming the build that works
|
||||||
|
# until something says otherwise.
|
||||||
|
#
|
||||||
|
# `:refresh-candidate` is one moving ref per image, overwritten in
|
||||||
|
# place, holding a build nobody is told to pull — the shape rule 145
|
||||||
|
# already allows for :buildcache, not the per-build tag family that
|
||||||
|
# milestone 318 withdrew.
|
||||||
|
#
|
||||||
|
# Both values are decided HERE, beside `hit`, for the reason the
|
||||||
|
# force/schedule branch below gives: one step decides what this job
|
||||||
|
# does. A promote condition derived independently could disagree with
|
||||||
|
# the tag the build actually wrote.
|
||||||
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
|
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "promote=true" >> "$GITHUB_OUTPUT"
|
||||||
|
else
|
||||||
|
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "promote=false" >> "$GITHUB_OUTPUT"
|
||||||
|
fi
|
||||||
|
|
||||||
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||||
# (run 4732): a missing key returns an empty string and exits 0, so
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||||
# branching on the exit code would read "no label yet" as success.
|
# branching on the exit code would read "no label yet" as success.
|
||||||
@@ -675,7 +738,7 @@ jobs:
|
|||||||
if [ "${FORCE:-false}" = "true" ]; then
|
if [ "${FORCE:-false}" = "true" ]; then
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: force_build set — building regardless"
|
echo "reuse: force_build set — building regardless"
|
||||||
elif [ "${EVENT:-}" = "schedule" ]; then
|
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: scheduled base refresh — building regardless"
|
echo "reuse: scheduled base refresh — building regardless"
|
||||||
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
@@ -764,6 +827,12 @@ jobs:
|
|||||||
|
|
||||||
- name: Build and push web image
|
- name: Build and push web image
|
||||||
if: steps.reuse.outputs.hit != 'true'
|
if: steps.reuse.outputs.hit != 'true'
|
||||||
|
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
||||||
|
# it normalises the image config's `created` field and the history
|
||||||
|
# timestamps rather than being consumed by the Dockerfile. See #3265
|
||||||
|
# and the reuse step's `epoch` output.
|
||||||
|
env:
|
||||||
|
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
||||||
uses: docker/build-push-action@v5
|
uses: docker/build-push-action@v5
|
||||||
with:
|
with:
|
||||||
context: .
|
context: .
|
||||||
@@ -776,20 +845,17 @@ jobs:
|
|||||||
# invalidates, and the image genuinely rebuilds.
|
# invalidates, and the image genuinely rebuilds.
|
||||||
#
|
#
|
||||||
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
||||||
# did NOT move, the build is ~13s and every content step reports
|
# did NOT move, the build was ~13s with every content step CACHED —
|
||||||
# CACHED — but the channel tag STILL gets a new manifest digest.
|
# and the channel tag STILL got a new manifest digest, because
|
||||||
# buildkit mints a fresh image config each run, so identical layers
|
# buildkit stamps a fresh image config per run and republishes the
|
||||||
# are republished under a new config blob. All three images moved
|
# identical layers under it. All three images moved that way on
|
||||||
# that way on 2026-08-30 with nothing whatsoever changed in them.
|
# 2026-08-30 with nothing whatsoever changed in them.
|
||||||
#
|
#
|
||||||
# So a refresh currently rewrites :latest every Sunday whether or
|
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
||||||
# not there is anything new in it, and :c-<sha> is handed a new
|
# content came from, the config is byte-identical across runs, so
|
||||||
# manifest to diverge from on the same cadence. Layers are shared,
|
# the manifest digest is too and the push is a registry no-op. A
|
||||||
# so the storage cost is a config blob; the cost that matters is
|
# digest change means the content changed again, which is the only
|
||||||
# that a digest change no longer MEANS anything. Tracked in #3265 —
|
# thing a digest is any use for.
|
||||||
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
|
|
||||||
# make "same source, same bytes" true and turn the no-op case into
|
|
||||||
# a genuine no-op.
|
|
||||||
#
|
#
|
||||||
# What `pull` does NOT catch either: a Debian package update inside
|
# What `pull` does NOT catch either: a Debian package update inside
|
||||||
# the `apt-get install` layer while the base tag itself stands
|
# the `apt-get install` layer while the base tag itself stands
|
||||||
@@ -799,14 +865,14 @@ jobs:
|
|||||||
# churn #3265 is about.
|
# churn #3265 is about.
|
||||||
#
|
#
|
||||||
# Only on the schedule. An ordinary push wants the cached base.
|
# Only on the schedule. An ordinary push wants the cached base.
|
||||||
pull: ${{ github.event_name == 'schedule' }}
|
pull: ${{ env.IS_REFRESH == 'true' }}
|
||||||
# ONE tag, the channel's. Every other tag is written by the step
|
# ONE tag, the channel's. Every other tag is written by the step
|
||||||
# below, registry-side. buildx here pushes the first tag to the
|
# below, registry-side. buildx here pushes the first tag to the
|
||||||
# registry and then re-pushes the rest through the DOCKER driver,
|
# registry and then re-pushes the rest through the DOCKER driver,
|
||||||
# out of a local image store a registry-direct build never filled —
|
# out of a local image store a registry-direct build never filled —
|
||||||
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
||||||
# published perfectly well.
|
# published perfectly well.
|
||||||
tags: ${{ steps.reuse.outputs.channel_ref }}
|
tags: ${{ steps.reuse.outputs.build_ref }}
|
||||||
# The reuse key. Read back off the channel tag on the next push to
|
# The reuse key. Read back off the channel tag on the next push to
|
||||||
# decide whether that push needs to build at all, so this is not
|
# decide whether that push needs to build at all, so this is not
|
||||||
# decoration — an unstamped image is one that will always rebuild.
|
# decoration — an unstamped image is one that will always rebuild.
|
||||||
@@ -853,6 +919,77 @@ jobs:
|
|||||||
FC_CHANNEL=${{ steps.tag.outputs.channel }}
|
FC_CHANNEL=${{ steps.tag.outputs.channel }}
|
||||||
FC_VERSION=${{ steps.reuse.outputs.version }}
|
FC_VERSION=${{ steps.reuse.outputs.version }}
|
||||||
|
|
||||||
|
# Point the channel tag at the candidate the refresh just built.
|
||||||
|
#
|
||||||
|
# Unconditional TODAY, so this milestone never leaves the refresh in a
|
||||||
|
# state where it builds and publishes nothing. Step 4 wraps it in the
|
||||||
|
# smoke suite's verdict; until then the scheduled path behaves exactly
|
||||||
|
# as it did, just via two operations instead of one.
|
||||||
|
#
|
||||||
|
# NOT `imagetools create`. That wraps its source in an INDEX, and an
|
||||||
|
# indexed channel tag is the one thing this pipeline cannot survive:
|
||||||
|
# `.Image.Config.Labels` does not resolve through an index, so the
|
||||||
|
# fc.revision the reuse check reads off the channel tag would come back
|
||||||
|
# empty, every subsequent push would miss and rebuild, and nothing would
|
||||||
|
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
|
||||||
|
# and the only symptom was the bill. The repoint step below excludes its
|
||||||
|
# own source tag for precisely this reason; a promote that re-introduced
|
||||||
|
# the wrap through a different door would undo that care.
|
||||||
|
#
|
||||||
|
# A manifest PUT is what "make this tag name that image" means at the
|
||||||
|
# registry level: the same bytes under the same media type, so the digest
|
||||||
|
# is identical, the media type is preserved, and no layer moves.
|
||||||
|
- name: Promote the refresh candidate to the channel
|
||||||
|
if: steps.reuse.outputs.promote == 'true'
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
||||||
|
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
||||||
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
|
ACTOR: ${{ github.actor }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
REPO=${IMAGE#git.fabledsword.com/}
|
||||||
|
TAG=${CHANNEL_REF##*:}
|
||||||
|
|
||||||
|
# Registry auth is its own token exchange — the `docker login` above
|
||||||
|
# authenticates the docker client, not curl. Deadline on every call
|
||||||
|
# (rule 156): a registry that stops answering must fail this step,
|
||||||
|
# not hang the weekly refresh until the job times out.
|
||||||
|
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
|
||||||
|
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
|
||||||
|
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
|
||||||
|
|
||||||
|
# Ask for the image manifest media types ONLY. Offering the index
|
||||||
|
# types too would let the registry hand back an index if one ever
|
||||||
|
# existed at this tag, and we would faithfully copy the thing we are
|
||||||
|
# trying not to create.
|
||||||
|
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
|
||||||
|
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
|
||||||
|
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
|
||||||
|
test -n "$CT"
|
||||||
|
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||||
|
echo "promote: candidate is $SRC_DIGEST ($CT)"
|
||||||
|
|
||||||
|
curl -fsS --max-time 120 -X PUT \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
|
||||||
|
--data-binary @manifest.json \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
|
||||||
|
|
||||||
|
# Read it back. A PUT that returned 2xx but landed something else is
|
||||||
|
# exactly the silent-and-plausible failure this pipeline keeps
|
||||||
|
# producing, and the check costs one request.
|
||||||
|
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
|
||||||
|
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||||
|
if [ "$NOW" != "$SRC_DIGEST" ]; then
|
||||||
|
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "promote: $IMAGE:$TAG now names $NOW"
|
||||||
|
|
||||||
# Every tag but the channel's own is written HERE, registry-side,
|
# Every tag but the channel's own is written HERE, registry-side,
|
||||||
# whether or not a build ran. Each -t becomes another reference to the
|
# whether or not a build ran. Each -t becomes another reference to the
|
||||||
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
||||||
@@ -957,7 +1094,7 @@ jobs:
|
|||||||
|
|
||||||
# See sign-extension's copy for why this guard exists.
|
# See sign-extension's copy for why this guard exists.
|
||||||
- name: Guard — a scheduled run must have checked out main
|
- name: Guard — a scheduled run must have checked out main
|
||||||
if: github.event_name == 'schedule'
|
if: env.IS_REFRESH == 'true'
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||||
@@ -1008,7 +1145,7 @@ jobs:
|
|||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
# Mirrors build-web's tag list and its schedule handling; see
|
# Mirrors build-web's tag list and its schedule handling; see
|
||||||
# the comments there.
|
# the comments there.
|
||||||
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
@@ -1091,17 +1228,59 @@ jobs:
|
|||||||
# A scheduled refresh has to bypass reuse by construction: it
|
# A scheduled refresh has to bypass reuse by construction: it
|
||||||
# rebuilds the SAME source, so fc.revision always matches and the
|
# rebuilds the SAME source, so fc.revision always matches and the
|
||||||
# check would skip every refresh there has ever been.
|
# check would skip every refresh there has ever been.
|
||||||
EVENT: ${{ github.event_name }}
|
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
DERIVED=$(sh scripts/artifacts.sh revision ml)
|
DERIVED=$(sh scripts/artifacts.sh revision ml)
|
||||||
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# The build clock, pinned to the same commit (#3265). Without it
|
||||||
|
# buildkit stamps the image config with the wall clock of the build,
|
||||||
|
# so identical layers republish under a new config blob and the
|
||||||
|
# channel tag gets a new manifest digest for no reason. Derived from
|
||||||
|
# `newest()` like revision and version, so all three name one commit
|
||||||
|
# and cannot drift apart.
|
||||||
|
echo "epoch=$(sh scripts/artifacts.sh epoch ml)" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# The moving tag for this channel. Which tag we ask IS the channel —
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||||
# that is why the revision needs no -main/-dev qualifier any more.
|
# that is why the revision needs no -main/-dev qualifier any more.
|
||||||
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||||
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
||||||
|
# whether the channel then has to be written separately.
|
||||||
|
#
|
||||||
|
# On a push the build writes the channel tag directly: the bytes came
|
||||||
|
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
||||||
|
# it behind.
|
||||||
|
#
|
||||||
|
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
||||||
|
# refresh rebuilds against freshly resolved base images, and the web
|
||||||
|
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
||||||
|
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
||||||
|
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
|
||||||
|
# install requirements.txt, and a base bump changes neither. So
|
||||||
|
# refreshed bytes have to be proven before :latest names them, and
|
||||||
|
# proving needs a moment between "built" and "published" to occupy.
|
||||||
|
# This is that moment; :latest goes on naming the build that works
|
||||||
|
# until something says otherwise.
|
||||||
|
#
|
||||||
|
# `:refresh-candidate` is one moving ref per image, overwritten in
|
||||||
|
# place, holding a build nobody is told to pull — the shape rule 145
|
||||||
|
# already allows for :buildcache, not the per-build tag family that
|
||||||
|
# milestone 318 withdrew.
|
||||||
|
#
|
||||||
|
# Both values are decided HERE, beside `hit`, for the reason the
|
||||||
|
# force/schedule branch below gives: one step decides what this job
|
||||||
|
# does. A promote condition derived independently could disagree with
|
||||||
|
# the tag the build actually wrote.
|
||||||
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
|
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "promote=true" >> "$GITHUB_OUTPUT"
|
||||||
|
else
|
||||||
|
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "promote=false" >> "$GITHUB_OUTPUT"
|
||||||
|
fi
|
||||||
|
|
||||||
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||||
# (run 4732): a missing key returns an empty string and exits 0, so
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||||
# branching on the exit code would read "no label yet" as success.
|
# branching on the exit code would read "no label yet" as success.
|
||||||
@@ -1134,7 +1313,7 @@ jobs:
|
|||||||
if [ "${FORCE:-false}" = "true" ]; then
|
if [ "${FORCE:-false}" = "true" ]; then
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: force_build set — building regardless"
|
echo "reuse: force_build set — building regardless"
|
||||||
elif [ "${EVENT:-}" = "schedule" ]; then
|
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: scheduled base refresh — building regardless"
|
echo "reuse: scheduled base refresh — building regardless"
|
||||||
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
@@ -1147,6 +1326,12 @@ jobs:
|
|||||||
|
|
||||||
- name: Build and push ml image
|
- name: Build and push ml image
|
||||||
if: steps.reuse.outputs.hit != 'true'
|
if: steps.reuse.outputs.hit != 'true'
|
||||||
|
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
||||||
|
# it normalises the image config's `created` field and the history
|
||||||
|
# timestamps rather than being consumed by the Dockerfile. See #3265
|
||||||
|
# and the reuse step's `epoch` output.
|
||||||
|
env:
|
||||||
|
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
||||||
uses: docker/build-push-action@v5
|
uses: docker/build-push-action@v5
|
||||||
with:
|
with:
|
||||||
context: .
|
context: .
|
||||||
@@ -1159,20 +1344,17 @@ jobs:
|
|||||||
# invalidates, and the image genuinely rebuilds.
|
# invalidates, and the image genuinely rebuilds.
|
||||||
#
|
#
|
||||||
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
||||||
# did NOT move, the build is ~13s and every content step reports
|
# did NOT move, the build was ~13s with every content step CACHED —
|
||||||
# CACHED — but the channel tag STILL gets a new manifest digest.
|
# and the channel tag STILL got a new manifest digest, because
|
||||||
# buildkit mints a fresh image config each run, so identical layers
|
# buildkit stamps a fresh image config per run and republishes the
|
||||||
# are republished under a new config blob. All three images moved
|
# identical layers under it. All three images moved that way on
|
||||||
# that way on 2026-08-30 with nothing whatsoever changed in them.
|
# 2026-08-30 with nothing whatsoever changed in them.
|
||||||
#
|
#
|
||||||
# So a refresh currently rewrites :latest every Sunday whether or
|
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
||||||
# not there is anything new in it, and :c-<sha> is handed a new
|
# content came from, the config is byte-identical across runs, so
|
||||||
# manifest to diverge from on the same cadence. Layers are shared,
|
# the manifest digest is too and the push is a registry no-op. A
|
||||||
# so the storage cost is a config blob; the cost that matters is
|
# digest change means the content changed again, which is the only
|
||||||
# that a digest change no longer MEANS anything. Tracked in #3265 —
|
# thing a digest is any use for.
|
||||||
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
|
|
||||||
# make "same source, same bytes" true and turn the no-op case into
|
|
||||||
# a genuine no-op.
|
|
||||||
#
|
#
|
||||||
# What `pull` does NOT catch either: a Debian package update inside
|
# What `pull` does NOT catch either: a Debian package update inside
|
||||||
# the `apt-get install` layer while the base tag itself stands
|
# the `apt-get install` layer while the base tag itself stands
|
||||||
@@ -1182,14 +1364,14 @@ jobs:
|
|||||||
# churn #3265 is about.
|
# churn #3265 is about.
|
||||||
#
|
#
|
||||||
# Only on the schedule. An ordinary push wants the cached base.
|
# Only on the schedule. An ordinary push wants the cached base.
|
||||||
pull: ${{ github.event_name == 'schedule' }}
|
pull: ${{ env.IS_REFRESH == 'true' }}
|
||||||
# ONE tag, the channel's. Every other tag is written by the step
|
# ONE tag, the channel's. Every other tag is written by the step
|
||||||
# below, registry-side. buildx here pushes the first tag to the
|
# below, registry-side. buildx here pushes the first tag to the
|
||||||
# registry and then re-pushes the rest through the DOCKER driver,
|
# registry and then re-pushes the rest through the DOCKER driver,
|
||||||
# out of a local image store a registry-direct build never filled —
|
# out of a local image store a registry-direct build never filled —
|
||||||
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
||||||
# published perfectly well.
|
# published perfectly well.
|
||||||
tags: ${{ steps.reuse.outputs.channel_ref }}
|
tags: ${{ steps.reuse.outputs.build_ref }}
|
||||||
# The reuse key. Read back off the channel tag on the next push to
|
# The reuse key. Read back off the channel tag on the next push to
|
||||||
# decide whether that push needs to build at all, so this is not
|
# decide whether that push needs to build at all, so this is not
|
||||||
# decoration — an unstamped image is one that will always rebuild.
|
# decoration — an unstamped image is one that will always rebuild.
|
||||||
@@ -1230,6 +1412,77 @@ jobs:
|
|||||||
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
|
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache
|
||||||
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
|
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max
|
||||||
|
|
||||||
|
# Point the channel tag at the candidate the refresh just built.
|
||||||
|
#
|
||||||
|
# Unconditional TODAY, so this milestone never leaves the refresh in a
|
||||||
|
# state where it builds and publishes nothing. Step 4 wraps it in the
|
||||||
|
# smoke suite's verdict; until then the scheduled path behaves exactly
|
||||||
|
# as it did, just via two operations instead of one.
|
||||||
|
#
|
||||||
|
# NOT `imagetools create`. That wraps its source in an INDEX, and an
|
||||||
|
# indexed channel tag is the one thing this pipeline cannot survive:
|
||||||
|
# `.Image.Config.Labels` does not resolve through an index, so the
|
||||||
|
# fc.revision the reuse check reads off the channel tag would come back
|
||||||
|
# empty, every subsequent push would miss and rebuild, and nothing would
|
||||||
|
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
|
||||||
|
# and the only symptom was the bill. The repoint step below excludes its
|
||||||
|
# own source tag for precisely this reason; a promote that re-introduced
|
||||||
|
# the wrap through a different door would undo that care.
|
||||||
|
#
|
||||||
|
# A manifest PUT is what "make this tag name that image" means at the
|
||||||
|
# registry level: the same bytes under the same media type, so the digest
|
||||||
|
# is identical, the media type is preserved, and no layer moves.
|
||||||
|
- name: Promote the refresh candidate to the channel
|
||||||
|
if: steps.reuse.outputs.promote == 'true'
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml
|
||||||
|
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
||||||
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
|
ACTOR: ${{ github.actor }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
REPO=${IMAGE#git.fabledsword.com/}
|
||||||
|
TAG=${CHANNEL_REF##*:}
|
||||||
|
|
||||||
|
# Registry auth is its own token exchange — the `docker login` above
|
||||||
|
# authenticates the docker client, not curl. Deadline on every call
|
||||||
|
# (rule 156): a registry that stops answering must fail this step,
|
||||||
|
# not hang the weekly refresh until the job times out.
|
||||||
|
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
|
||||||
|
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
|
||||||
|
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
|
||||||
|
|
||||||
|
# Ask for the image manifest media types ONLY. Offering the index
|
||||||
|
# types too would let the registry hand back an index if one ever
|
||||||
|
# existed at this tag, and we would faithfully copy the thing we are
|
||||||
|
# trying not to create.
|
||||||
|
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
|
||||||
|
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
|
||||||
|
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
|
||||||
|
test -n "$CT"
|
||||||
|
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||||
|
echo "promote: candidate is $SRC_DIGEST ($CT)"
|
||||||
|
|
||||||
|
curl -fsS --max-time 120 -X PUT \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
|
||||||
|
--data-binary @manifest.json \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
|
||||||
|
|
||||||
|
# Read it back. A PUT that returned 2xx but landed something else is
|
||||||
|
# exactly the silent-and-plausible failure this pipeline keeps
|
||||||
|
# producing, and the check costs one request.
|
||||||
|
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
|
||||||
|
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||||
|
if [ "$NOW" != "$SRC_DIGEST" ]; then
|
||||||
|
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "promote: $IMAGE:$TAG now names $NOW"
|
||||||
|
|
||||||
# Every tag but the channel's own is written HERE, registry-side,
|
# Every tag but the channel's own is written HERE, registry-side,
|
||||||
# whether or not a build ran. Each -t becomes another reference to the
|
# whether or not a build ran. Each -t becomes another reference to the
|
||||||
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
||||||
@@ -1331,7 +1584,7 @@ jobs:
|
|||||||
|
|
||||||
# See sign-extension's copy for why this guard exists.
|
# See sign-extension's copy for why this guard exists.
|
||||||
- name: Guard — a scheduled run must have checked out main
|
- name: Guard — a scheduled run must have checked out main
|
||||||
if: github.event_name == 'schedule'
|
if: env.IS_REFRESH == 'true'
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
||||||
@@ -1377,7 +1630,7 @@ jobs:
|
|||||||
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
||||||
# Mirrors build-web's tag list and its schedule handling; see
|
# Mirrors build-web's tag list and its schedule handling; see
|
||||||
# the comments there.
|
# the comments there.
|
||||||
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
|
||||||
echo "channel=main" >> "$GITHUB_OUTPUT"
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
||||||
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
||||||
@@ -1460,17 +1713,59 @@ jobs:
|
|||||||
# A scheduled refresh has to bypass reuse by construction: it
|
# A scheduled refresh has to bypass reuse by construction: it
|
||||||
# rebuilds the SAME source, so fc.revision always matches and the
|
# rebuilds the SAME source, so fc.revision always matches and the
|
||||||
# check would skip every refresh there has ever been.
|
# check would skip every refresh there has ever been.
|
||||||
EVENT: ${{ github.event_name }}
|
|
||||||
run: |
|
run: |
|
||||||
set -eu
|
set -eu
|
||||||
DERIVED=$(sh scripts/artifacts.sh revision agent)
|
DERIVED=$(sh scripts/artifacts.sh revision agent)
|
||||||
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# The build clock, pinned to the same commit (#3265). Without it
|
||||||
|
# buildkit stamps the image config with the wall clock of the build,
|
||||||
|
# so identical layers republish under a new config blob and the
|
||||||
|
# channel tag gets a new manifest digest for no reason. Derived from
|
||||||
|
# `newest()` like revision and version, so all three name one commit
|
||||||
|
# and cannot drift apart.
|
||||||
|
echo "epoch=$(sh scripts/artifacts.sh epoch agent)" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# The moving tag for this channel. Which tag we ask IS the channel —
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
||||||
# that is why the revision needs no -main/-dev qualifier any more.
|
# that is why the revision needs no -main/-dev qualifier any more.
|
||||||
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
||||||
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
||||||
|
# whether the channel then has to be written separately.
|
||||||
|
#
|
||||||
|
# On a push the build writes the channel tag directly: the bytes came
|
||||||
|
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
||||||
|
# it behind.
|
||||||
|
#
|
||||||
|
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
||||||
|
# refresh rebuilds against freshly resolved base images, and the web
|
||||||
|
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
||||||
|
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
||||||
|
# Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and
|
||||||
|
# install requirements.txt, and a base bump changes neither. So
|
||||||
|
# refreshed bytes have to be proven before :latest names them, and
|
||||||
|
# proving needs a moment between "built" and "published" to occupy.
|
||||||
|
# This is that moment; :latest goes on naming the build that works
|
||||||
|
# until something says otherwise.
|
||||||
|
#
|
||||||
|
# `:refresh-candidate` is one moving ref per image, overwritten in
|
||||||
|
# place, holding a build nobody is told to pull — the shape rule 145
|
||||||
|
# already allows for :buildcache, not the per-build tag family that
|
||||||
|
# milestone 318 withdrew.
|
||||||
|
#
|
||||||
|
# Both values are decided HERE, beside `hit`, for the reason the
|
||||||
|
# force/schedule branch below gives: one step decides what this job
|
||||||
|
# does. A promote condition derived independently could disagree with
|
||||||
|
# the tag the build actually wrote.
|
||||||
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
|
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "promote=true" >> "$GITHUB_OUTPUT"
|
||||||
|
else
|
||||||
|
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "promote=false" >> "$GITHUB_OUTPUT"
|
||||||
|
fi
|
||||||
|
|
||||||
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
||||||
# (run 4732): a missing key returns an empty string and exits 0, so
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
||||||
# branching on the exit code would read "no label yet" as success.
|
# branching on the exit code would read "no label yet" as success.
|
||||||
@@ -1503,7 +1798,7 @@ jobs:
|
|||||||
if [ "${FORCE:-false}" = "true" ]; then
|
if [ "${FORCE:-false}" = "true" ]; then
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: force_build set — building regardless"
|
echo "reuse: force_build set — building regardless"
|
||||||
elif [ "${EVENT:-}" = "schedule" ]; then
|
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
||||||
echo "hit=false" >> "$GITHUB_OUTPUT"
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
||||||
echo "reuse: scheduled base refresh — building regardless"
|
echo "reuse: scheduled base refresh — building regardless"
|
||||||
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
||||||
@@ -1516,6 +1811,12 @@ jobs:
|
|||||||
|
|
||||||
- name: Build and push agent image
|
- name: Build and push agent image
|
||||||
if: steps.reuse.outputs.hit != 'true'
|
if: steps.reuse.outputs.hit != 'true'
|
||||||
|
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
||||||
|
# it normalises the image config's `created` field and the history
|
||||||
|
# timestamps rather than being consumed by the Dockerfile. See #3265
|
||||||
|
# and the reuse step's `epoch` output.
|
||||||
|
env:
|
||||||
|
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
||||||
uses: docker/build-push-action@v5
|
uses: docker/build-push-action@v5
|
||||||
with:
|
with:
|
||||||
context: agent
|
context: agent
|
||||||
@@ -1528,20 +1829,17 @@ jobs:
|
|||||||
# invalidates, and the image genuinely rebuilds.
|
# invalidates, and the image genuinely rebuilds.
|
||||||
#
|
#
|
||||||
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
||||||
# did NOT move, the build is ~13s and every content step reports
|
# did NOT move, the build was ~13s with every content step CACHED —
|
||||||
# CACHED — but the channel tag STILL gets a new manifest digest.
|
# and the channel tag STILL got a new manifest digest, because
|
||||||
# buildkit mints a fresh image config each run, so identical layers
|
# buildkit stamps a fresh image config per run and republishes the
|
||||||
# are republished under a new config blob. All three images moved
|
# identical layers under it. All three images moved that way on
|
||||||
# that way on 2026-08-30 with nothing whatsoever changed in them.
|
# 2026-08-30 with nothing whatsoever changed in them.
|
||||||
#
|
#
|
||||||
# So a refresh currently rewrites :latest every Sunday whether or
|
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
||||||
# not there is anything new in it, and :c-<sha> is handed a new
|
# content came from, the config is byte-identical across runs, so
|
||||||
# manifest to diverge from on the same cadence. Layers are shared,
|
# the manifest digest is too and the push is a registry no-op. A
|
||||||
# so the storage cost is a config blob; the cost that matters is
|
# digest change means the content changed again, which is the only
|
||||||
# that a digest change no longer MEANS anything. Tracked in #3265 —
|
# thing a digest is any use for.
|
||||||
# the likely fix is a deterministic SOURCE_DATE_EPOCH, which would
|
|
||||||
# make "same source, same bytes" true and turn the no-op case into
|
|
||||||
# a genuine no-op.
|
|
||||||
#
|
#
|
||||||
# What `pull` does NOT catch either: a Debian package update inside
|
# What `pull` does NOT catch either: a Debian package update inside
|
||||||
# the `apt-get install` layer while the base tag itself stands
|
# the `apt-get install` layer while the base tag itself stands
|
||||||
@@ -1551,14 +1849,14 @@ jobs:
|
|||||||
# churn #3265 is about.
|
# churn #3265 is about.
|
||||||
#
|
#
|
||||||
# Only on the schedule. An ordinary push wants the cached base.
|
# Only on the schedule. An ordinary push wants the cached base.
|
||||||
pull: ${{ github.event_name == 'schedule' }}
|
pull: ${{ env.IS_REFRESH == 'true' }}
|
||||||
# ONE tag, the channel's. Every other tag is written by the step
|
# ONE tag, the channel's. Every other tag is written by the step
|
||||||
# below, registry-side. buildx here pushes the first tag to the
|
# below, registry-side. buildx here pushes the first tag to the
|
||||||
# registry and then re-pushes the rest through the DOCKER driver,
|
# registry and then re-pushes the rest through the DOCKER driver,
|
||||||
# out of a local image store a registry-direct build never filled —
|
# out of a local image store a registry-direct build never filled —
|
||||||
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
||||||
# published perfectly well.
|
# published perfectly well.
|
||||||
tags: ${{ steps.reuse.outputs.channel_ref }}
|
tags: ${{ steps.reuse.outputs.build_ref }}
|
||||||
# The reuse key. Read back off the channel tag on the next push to
|
# The reuse key. Read back off the channel tag on the next push to
|
||||||
# decide whether that push needs to build at all, so this is not
|
# decide whether that push needs to build at all, so this is not
|
||||||
# decoration — an unstamped image is one that will always rebuild.
|
# decoration — an unstamped image is one that will always rebuild.
|
||||||
@@ -1599,6 +1897,77 @@ jobs:
|
|||||||
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
|
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
|
||||||
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
|
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
|
||||||
|
|
||||||
|
# Point the channel tag at the candidate the refresh just built.
|
||||||
|
#
|
||||||
|
# Unconditional TODAY, so this milestone never leaves the refresh in a
|
||||||
|
# state where it builds and publishes nothing. Step 4 wraps it in the
|
||||||
|
# smoke suite's verdict; until then the scheduled path behaves exactly
|
||||||
|
# as it did, just via two operations instead of one.
|
||||||
|
#
|
||||||
|
# NOT `imagetools create`. That wraps its source in an INDEX, and an
|
||||||
|
# indexed channel tag is the one thing this pipeline cannot survive:
|
||||||
|
# `.Image.Config.Labels` does not resolve through an index, so the
|
||||||
|
# fc.revision the reuse check reads off the channel tag would come back
|
||||||
|
# empty, every subsequent push would miss and rebuild, and nothing would
|
||||||
|
# go red. That is #3183, observed on run 4751 — reuse worked exactly once
|
||||||
|
# and the only symptom was the bill. The repoint step below excludes its
|
||||||
|
# own source tag for precisely this reason; a promote that re-introduced
|
||||||
|
# the wrap through a different door would undo that care.
|
||||||
|
#
|
||||||
|
# A manifest PUT is what "make this tag name that image" means at the
|
||||||
|
# registry level: the same bytes under the same media type, so the digest
|
||||||
|
# is identical, the media type is preserved, and no layer moves.
|
||||||
|
- name: Promote the refresh candidate to the channel
|
||||||
|
if: steps.reuse.outputs.promote == 'true'
|
||||||
|
env:
|
||||||
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
||||||
|
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
||||||
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
||||||
|
ACTOR: ${{ github.actor }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
REPO=${IMAGE#git.fabledsword.com/}
|
||||||
|
TAG=${CHANNEL_REF##*:}
|
||||||
|
|
||||||
|
# Registry auth is its own token exchange — the `docker login` above
|
||||||
|
# authenticates the docker client, not curl. Deadline on every call
|
||||||
|
# (rule 156): a registry that stops answering must fail this step,
|
||||||
|
# not hang the weekly refresh until the job times out.
|
||||||
|
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
|
||||||
|
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
|
||||||
|
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
|
||||||
|
|
||||||
|
# Ask for the image manifest media types ONLY. Offering the index
|
||||||
|
# types too would let the registry hand back an index if one ever
|
||||||
|
# existed at this tag, and we would faithfully copy the thing we are
|
||||||
|
# trying not to create.
|
||||||
|
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
|
||||||
|
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
|
||||||
|
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
|
||||||
|
test -n "$CT"
|
||||||
|
SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||||
|
echo "promote: candidate is $SRC_DIGEST ($CT)"
|
||||||
|
|
||||||
|
curl -fsS --max-time 120 -X PUT \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
|
||||||
|
--data-binary @manifest.json \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
|
||||||
|
|
||||||
|
# Read it back. A PUT that returned 2xx but landed something else is
|
||||||
|
# exactly the silent-and-plausible failure this pipeline keeps
|
||||||
|
# producing, and the check costs one request.
|
||||||
|
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
|
||||||
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
||||||
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
|
||||||
|
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
||||||
|
if [ "$NOW" != "$SRC_DIGEST" ]; then
|
||||||
|
echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "promote: $IMAGE:$TAG now names $NOW"
|
||||||
|
|
||||||
# Every tag but the channel's own is written HERE, registry-side,
|
# Every tag but the channel's own is written HERE, registry-side,
|
||||||
# whether or not a build ran. Each -t becomes another reference to the
|
# whether or not a build ran. Each -t becomes another reference to the
|
||||||
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
# FabledCurator
|
# FabledCurator
|
||||||
|
|
||||||
|
<!-- overview:start -->
|
||||||
Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven
|
Self-hosted media curation — a gallery, ML auto-tagging, and subscription-driven
|
||||||
downloading in one application. Part of the FabledSword family.
|
downloading in one application. Part of the FabledSword family.
|
||||||
|
|
||||||
@@ -19,13 +20,15 @@ through afterwards.
|
|||||||
characters, find near-duplicates and power similarity search. Suggestions are
|
characters, find near-duplicates and power similarity search. Suggestions are
|
||||||
reviewable — it proposes, you confirm, and it learns which proposals you keep
|
reviewable — it proposes, you confirm, and it learns which proposals you keep
|
||||||
rejecting.
|
rejecting.
|
||||||
- **Importing.** Ingests an existing library from disk, deduplicates it by
|
- **Deduplication and provenance.** Everything that arrives is hashed and
|
||||||
content hash, and reads metadata sidecars.
|
deduplicated by content, metadata sidecars are read wherever the source
|
||||||
|
writes them, and every file keeps a record of where it came from.
|
||||||
- **Maintenance.** Backups, library audits, thumbnail and embedding backfills,
|
- **Maintenance.** Backups, library audits, thumbnail and embedding backfills,
|
||||||
orphan cleanup — all from the UI, all as background jobs you can watch.
|
orphan cleanup — all from the UI, all as background jobs you can watch.
|
||||||
|
|
||||||
Everything is configured from the Settings UI and stored in the database. There
|
Everything is configured from the Settings UI and stored in the database. There
|
||||||
is no config file to edit beyond a handful of bootstrap environment variables.
|
is no config file to edit beyond a handful of bootstrap environment variables.
|
||||||
|
<!-- overview:end -->
|
||||||
|
|
||||||
## Before you expose it
|
## Before you expose it
|
||||||
|
|
||||||
|
|||||||
+15
-8
@@ -198,14 +198,21 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
|
|||||||
refresh from being undone.
|
refresh from being undone.
|
||||||
- **`pull: true` on the scheduled path only** is the mechanism: a moved base
|
- **`pull: true` on the scheduled path only** is the mechanism: a moved base
|
||||||
tag changes the `FROM` layer's cache key and everything above it rebuilds.
|
tag changes the `FROM` layer's cache key and everything above it rebuilds.
|
||||||
**It does not currently make the unmoved case free.** Measured on the first
|
It did not always make the unmoved case free. Measured on the first real
|
||||||
real fire (run 4934, 2026-08-30): every content step reported `CACHED` and
|
fire (run 4934, 2026-08-30): every content step reported `CACHED` and the
|
||||||
the bases resolved to unchanged digests, yet all three `:latest` tags got a
|
bases resolved to unchanged digests, yet all three `:latest` tags got a NEW
|
||||||
NEW manifest digest, because buildkit mints a fresh image config per run and
|
manifest digest, because buildkit stamps a fresh image config per run and
|
||||||
republishes identical layers under it. So `:latest` is rewritten weekly
|
republishes identical layers under it — so `:latest` was rewritten weekly
|
||||||
whether or not anything changed, and `:c-<sha>` is handed a new manifest to
|
whether or not anything changed, and a digest change stopped meaning
|
||||||
diverge from on the same cadence — a digest change stops meaning anything.
|
anything (#3265).
|
||||||
Tracked as #3265; the likely fix is a deterministic `SOURCE_DATE_EPOCH`.
|
- **`SOURCE_DATE_EPOCH` is what makes it free.** Set on each build step from
|
||||||
|
`artifacts.sh epoch <artifact>` — the unix timestamp of the same commit
|
||||||
|
`revision` and `version` name, so all three are views of one `newest()`
|
||||||
|
lookup and cannot drift into disagreeing. With the config's `created` field
|
||||||
|
and history timestamps pinned to the content rather than to the wall clock,
|
||||||
|
identical source produces an identical manifest digest and the push is a
|
||||||
|
registry no-op. That restores the property the whole scheme rests on: a
|
||||||
|
channel tag's digest changes when, and only when, its content does.
|
||||||
Separately not caught: a Debian package update inside the `apt-get install`
|
Separately not caught: a Debian package update inside the `apt-get install`
|
||||||
layer while the base tag stands still — a lag rather than a hole, since the
|
layer while the base tag stands still — a lag rather than a hole, since the
|
||||||
official python/cuda images rebuild with those updates baked in.
|
official python/cuda images rebuild with those updates baked in.
|
||||||
|
|||||||
+22
-1
@@ -90,7 +90,7 @@ DERIVER='scripts/artifacts.sh'
|
|||||||
|
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
echo "usage: artifacts.sh {paths|revision|version} {web|ml|agent|extension}" >&2
|
echo "usage: artifacts.sh {paths|revision|version|epoch} {web|ml|agent|extension}" >&2
|
||||||
exit 2
|
exit 2
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,6 +149,26 @@ cmd_revision() {
|
|||||||
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
|
echo "$(newest "$1")" | cut -d' ' -f2 | cut -c1-12
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# The BUILD CLOCK: the same commit's unix timestamp, for SOURCE_DATE_EPOCH.
|
||||||
|
#
|
||||||
|
# buildkit stamps the image config's `created` field and every history entry
|
||||||
|
# with the wall clock of the build unless this is set, so two builds of
|
||||||
|
# identical source produce different config blobs and therefore different
|
||||||
|
# manifest digests. That is #3265: the weekly refresh republished all three
|
||||||
|
# `:latest` tags on 2026-08-30 with every content step CACHED and the bases
|
||||||
|
# resolved to unchanged digests — nothing was different, and the digest moved
|
||||||
|
# anyway. A digest that changes on a calendar cannot also mean "the content
|
||||||
|
# changed", which is the only thing anyone wants it for.
|
||||||
|
#
|
||||||
|
# It is the same commit `revision` and `version` name — deliberately, and this
|
||||||
|
# is the point of routing it through `newest()` rather than taking git's word
|
||||||
|
# separately. Three values derived from three lookups can disagree; three
|
||||||
|
# views of one lookup cannot. Note #3127 §2 is the record of what a second
|
||||||
|
# clock costs.
|
||||||
|
cmd_epoch() {
|
||||||
|
echo "$(newest "$1")" | cut -d' ' -f1
|
||||||
|
}
|
||||||
|
|
||||||
# The VERSION: `YYYY.MM.DD.HHMM`, zero-padded, UTC. One shape across the whole
|
# The VERSION: `YYYY.MM.DD.HHMM`, zero-padded, UTC. One shape across the whole
|
||||||
# family (note #3127 §1, rule 148) — the number an instance reports about
|
# family (note #3127 §1, rule 148) — the number an instance reports about
|
||||||
# itself, and, with a `v` in front, the release tag naming the same build.
|
# itself, and, with a `v` in front, the release tag naming the same build.
|
||||||
@@ -197,5 +217,6 @@ case "$1" in
|
|||||||
paths) cmd_paths "$2" ;;
|
paths) cmd_paths "$2" ;;
|
||||||
revision) cmd_revision "$2" ;;
|
revision) cmd_revision "$2" ;;
|
||||||
version) cmd_version "$2" ;;
|
version) cmd_version "$2" ;;
|
||||||
|
epoch) cmd_epoch "$2" ;;
|
||||||
*) usage ;;
|
*) usage ;;
|
||||||
esac
|
esac
|
||||||
|
|||||||
+112
-8
@@ -33,6 +33,32 @@ history. Ancestry is immune to the shape change, and it is also the more honest
|
|||||||
question: "what is in this that was not in the last one" IS a reachability
|
question: "what is in this that was not in the last one" IS a reachability
|
||||||
question.
|
question.
|
||||||
|
|
||||||
|
Ancestry alone is not enough, though, and milestone 328 is where that showed.
|
||||||
|
The 28 `v26.*` tags are still in the repo — the operator kept them as history
|
||||||
|
when their releases were deleted — so `--match v*` walks straight back to
|
||||||
|
`v26.06.04.0` and reports 533 commits. That span is not a changelog: nobody has
|
||||||
|
run `v26.06.04.0`, its release page no longer exists to compare against, and
|
||||||
|
the 200 lines that survive truncation are precisely the internal build-out that
|
||||||
|
milestone 328 exists to stop shipping. So the match is `v[0-9][0-9][0-9][0-9].*`
|
||||||
|
— rule 148's four-digit-year shape — which is exactly the set of tags that name
|
||||||
|
a release a reader could have been running. A pre-convention tag is history,
|
||||||
|
not a predecessor.
|
||||||
|
|
||||||
|
## The first release has no changelog, and should not pretend to
|
||||||
|
|
||||||
|
Once the match is narrowed, the first rule-148 tag reaches no predecessor at
|
||||||
|
all, and the old fallback — diff against the whole history — is worse than the
|
||||||
|
problem it replaced. The honest content for a release nobody has a previous
|
||||||
|
version of is what the thing IS.
|
||||||
|
|
||||||
|
So a release with no reachable predecessor renders the product overview instead
|
||||||
|
of a commit list. It is read out of README.md between `<!-- overview:start -->`
|
||||||
|
and `<!-- overview:end -->` rather than written here, for the same reason the
|
||||||
|
changelog is derived: two hand-maintained descriptions of one product drift,
|
||||||
|
and nothing ever catches it. The release page and the repo front page are one
|
||||||
|
source. Every later release goes back to being a changelog, which is what §5 of
|
||||||
|
note #3127 says a release is for.
|
||||||
|
|
||||||
## Re-runs update, they do not fall through
|
## Re-runs update, they do not fall through
|
||||||
|
|
||||||
Note #3127 §6.7: a publisher that POSTs and recovers the id from a `409` never
|
Note #3127 §6.7: a publisher that POSTs and recovers the id from a `409` never
|
||||||
@@ -91,18 +117,45 @@ def git_ok(*args: str) -> str | None:
|
|||||||
|
|
||||||
|
|
||||||
def previous_tag(ref: str, tag: str | None) -> str | None:
|
def previous_tag(ref: str, tag: str | None) -> str | None:
|
||||||
"""The most recent `v*` tag reachable from `ref`, excluding `tag` itself.
|
"""The most recent rule-148 tag reachable from `ref`, excluding `tag` itself.
|
||||||
|
|
||||||
`--exclude` rather than `<ref>^` so this is the same call whether or not
|
`--exclude` rather than `<ref>^` so this is the same call whether or not
|
||||||
`ref` is the tag being released — and so it does not blow up on a root
|
`ref` is the tag being released — and so it does not blow up on a root
|
||||||
commit that has no parent to walk to.
|
commit that has no parent to walk to.
|
||||||
|
|
||||||
|
The glob deliberately does NOT match the old `v26.*` tags. They are kept as
|
||||||
|
history and their releases are gone, so naming one as the predecessor emits
|
||||||
|
a span nobody can look up. See the module docstring.
|
||||||
"""
|
"""
|
||||||
args = ["describe", "--tags", "--abbrev=0", "--match", "v*"]
|
args = ["describe", "--tags", "--abbrev=0", "--match", "v[0-9][0-9][0-9][0-9].*"]
|
||||||
if tag:
|
if tag:
|
||||||
args += ["--exclude", tag]
|
args += ["--exclude", tag]
|
||||||
return git_ok(*args, ref)
|
return git_ok(*args, ref)
|
||||||
|
|
||||||
|
|
||||||
|
def product_overview() -> str | None:
|
||||||
|
"""The product description, lifted verbatim from README.md.
|
||||||
|
|
||||||
|
Returns None if the markers are absent or empty — a missing overview is
|
||||||
|
reported as a note and the release still publishes, on the same reasoning
|
||||||
|
as cross_checks(): the release is the useful object even when one part of
|
||||||
|
the derivation could not run.
|
||||||
|
"""
|
||||||
|
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
try:
|
||||||
|
with open(os.path.join(root, "README.md"), encoding="utf-8") as fh:
|
||||||
|
readme = fh.read()
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
match = re.search(
|
||||||
|
r"<!--\s*overview:start\s*-->(.*?)<!--\s*overview:end\s*-->",
|
||||||
|
readme, re.S,
|
||||||
|
)
|
||||||
|
if not match:
|
||||||
|
return None
|
||||||
|
return match.group(1).strip() or None
|
||||||
|
|
||||||
|
|
||||||
def commits(previous: str | None, ref: str) -> list[str]:
|
def commits(previous: str | None, ref: str) -> list[str]:
|
||||||
"""The subjects between the previous release and this one.
|
"""The subjects between the previous release and this one.
|
||||||
|
|
||||||
@@ -125,7 +178,10 @@ def truncate(log: list[str]) -> tuple[list[str], str | None]:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list[str]) -> str:
|
def render(
|
||||||
|
tag: str, sha: str, previous: str | None, log: list[str], notes: list[str],
|
||||||
|
overview: str | None,
|
||||||
|
) -> str:
|
||||||
short = sha[:7]
|
short = sha[:7]
|
||||||
parts = []
|
parts = []
|
||||||
|
|
||||||
@@ -135,6 +191,25 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
|
|||||||
# is the failure this whole milestone is about.
|
# is the failure this whole milestone is about.
|
||||||
parts.append("\n".join(f"> **Note:** {n}" for n in notes))
|
parts.append("\n".join(f"> **Note:** {n}" for n in notes))
|
||||||
|
|
||||||
|
# No predecessor means nobody reading this has run an earlier one, so the
|
||||||
|
# release describes the product rather than a diff. The overview is
|
||||||
|
# README.md's own words — see the module docstring on why it is not
|
||||||
|
# written here.
|
||||||
|
if previous is None and overview:
|
||||||
|
parts.append(overview)
|
||||||
|
parts.append(
|
||||||
|
"## Installing\n\n"
|
||||||
|
"```\ncurl -O https://git.fabledsword.com/bvandeusen/FabledCurator/raw/"
|
||||||
|
f"tag/{tag}/docker-compose.yml\ncurl -O https://git.fabledsword.com/"
|
||||||
|
f"bvandeusen/FabledCurator/raw/tag/{tag}/.env.example\n"
|
||||||
|
"mv .env.example .env # then set SECRET_KEY, DB_PASSWORD\n"
|
||||||
|
"docker compose -f docker-compose.yml up -d\n```\n\n"
|
||||||
|
"**Read \"Before you expose it\" in the README first.** FabledCurator "
|
||||||
|
"has no login, and it stores live platform session cookies for "
|
||||||
|
"accounts that usually have a payment method attached. Bind it to a "
|
||||||
|
"network you trust."
|
||||||
|
)
|
||||||
|
|
||||||
parts.append(
|
parts.append(
|
||||||
f"Built from `{short}`. The rollback unit is the immutable `:c-` tag "
|
f"Built from `{short}`. The rollback unit is the immutable `:c-` tag "
|
||||||
f"(rule 145) — these three move together:\n\n```\n"
|
f"(rule 145) — these three move together:\n\n```\n"
|
||||||
@@ -142,7 +217,19 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
|
|||||||
+ "\n```"
|
+ "\n```"
|
||||||
)
|
)
|
||||||
|
|
||||||
heading = f"## Changes since {previous}" if previous else "## Changes"
|
if previous is None:
|
||||||
|
# Deliberately NOT a commit list. The alternative is the whole history
|
||||||
|
# truncated to MAX_COMMITS, which is 200 lines of internal build-out
|
||||||
|
# presented to someone who has never seen this project.
|
||||||
|
parts.append(
|
||||||
|
"---\n\n_First release under rule 148's `vYYYY.MM.DD.HHMM` shape, so "
|
||||||
|
"there is no predecessor to diff against and no changelog to derive. "
|
||||||
|
"The description above is README.md's, quoted at publish time. Later "
|
||||||
|
"releases carry the commits since the previous one._"
|
||||||
|
)
|
||||||
|
return "\n\n".join(parts)
|
||||||
|
|
||||||
|
heading = f"## Changes since {previous}"
|
||||||
if log:
|
if log:
|
||||||
parts.append(heading + "\n\n" + "\n".join(f"- {line}" for line in log))
|
parts.append(heading + "\n\n" + "\n".join(f"- {line}" for line in log))
|
||||||
else:
|
else:
|
||||||
@@ -152,10 +239,9 @@ def render(tag: str, sha: str, previous: str | None, log: list[str], notes: list
|
|||||||
"names the same source under a new name._"
|
"names the same source under a new name._"
|
||||||
)
|
)
|
||||||
|
|
||||||
span = f"{previous}..{tag}" if previous else tag
|
|
||||||
parts.append(
|
parts.append(
|
||||||
f"---\n\n_Derived at publish time from `git log --no-merges {span}`. "
|
f"---\n\n_Derived at publish time from "
|
||||||
f"Nothing here is hand-maintained._"
|
f"`git log --no-merges {previous}..{tag}`. Nothing here is hand-maintained._"
|
||||||
)
|
)
|
||||||
return "\n\n".join(parts)
|
return "\n\n".join(parts)
|
||||||
|
|
||||||
@@ -289,13 +375,31 @@ def main() -> None:
|
|||||||
for note in notes:
|
for note in notes:
|
||||||
print(f"release: NOTE {note}")
|
print(f"release: NOTE {note}")
|
||||||
|
|
||||||
|
# A first release renders the overview instead of a changelog, so the
|
||||||
|
# commit walk is skipped entirely rather than computed and discarded —
|
||||||
|
# `commits(None, ref)` is the whole history and there is no reason to ask
|
||||||
|
# for it.
|
||||||
|
overview = None
|
||||||
|
log: list[str] = []
|
||||||
|
if previous is None:
|
||||||
|
overview = product_overview()
|
||||||
|
if overview is None:
|
||||||
|
note = (
|
||||||
|
"No `<!-- overview:start -->` block found in README.md, so this "
|
||||||
|
"first release has no product description. Published anyway; add "
|
||||||
|
"the markers and re-run the workflow to fill it in."
|
||||||
|
)
|
||||||
|
print(f"release: NOTE {note}")
|
||||||
|
notes.append(note)
|
||||||
|
print("release: no rule-148 predecessor — rendering the product overview")
|
||||||
|
else:
|
||||||
log = commits(previous, ref)
|
log = commits(previous, ref)
|
||||||
print(f"release: {len(log)} non-merge commits in the span")
|
print(f"release: {len(log)} non-merge commits in the span")
|
||||||
log, overflow = truncate(log)
|
log, overflow = truncate(log)
|
||||||
if overflow:
|
if overflow:
|
||||||
print(f"release: NOTE {overflow}")
|
print(f"release: NOTE {overflow}")
|
||||||
notes.append(overflow)
|
notes.append(overflow)
|
||||||
body = render(tag or ref, sha, previous, log, notes)
|
body = render(tag or ref, sha, previous, log, notes, overview)
|
||||||
|
|
||||||
if args.dry_run or not tag:
|
if args.dry_run or not tag:
|
||||||
print("--- body ---")
|
print("--- body ---")
|
||||||
|
|||||||
@@ -234,3 +234,56 @@ def test_version_and_revision_describe_the_same_commit(artifact):
|
|||||||
f"in AMO_UNPADDED may differ here."
|
f"in AMO_UNPADDED may differ here."
|
||||||
)
|
)
|
||||||
assert sha.startswith(revision(artifact))
|
assert sha.startswith(revision(artifact))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("artifact", ARTIFACTS)
|
||||||
|
def test_epoch_is_the_same_commit_the_version_names(artifact):
|
||||||
|
"""The build clock and the version must be one lookup, not two.
|
||||||
|
|
||||||
|
`epoch` feeds SOURCE_DATE_EPOCH, which decides the image config's bytes and
|
||||||
|
therefore the manifest digest; `version` is what the instance reports about
|
||||||
|
itself. If they could name different commits, an image would be stamped
|
||||||
|
reproducibly against one commit while claiming to be another — and both
|
||||||
|
values would look perfectly well-formed, exactly like the divergence the
|
||||||
|
test above guards.
|
||||||
|
|
||||||
|
They cannot, because `cmd_epoch` and `cmd_version` are two fields of one
|
||||||
|
`newest()` result. This pins that they stay that way: a future refactor
|
||||||
|
that gave epoch its own `git log` would pass every other test here.
|
||||||
|
"""
|
||||||
|
epoch = artifacts("epoch", artifact).strip()
|
||||||
|
assert epoch.isdigit(), f"{artifact} epoch is {epoch!r}, not a unix timestamp"
|
||||||
|
|
||||||
|
sha = newest_by_commit_time(artifact)
|
||||||
|
committed = subprocess.run(
|
||||||
|
["git", "show", "-s", "--format=%ct", sha],
|
||||||
|
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||||
|
).stdout.strip()
|
||||||
|
assert epoch == committed, (
|
||||||
|
f"{artifact} derives epoch {epoch}, but its newest shipped commit "
|
||||||
|
f"{sha[:12]} was committed at {committed}. SOURCE_DATE_EPOCH would "
|
||||||
|
f"pin the image config to a commit the version does not name."
|
||||||
|
)
|
||||||
|
|
||||||
|
# And the two renderings must agree, which is the property that actually
|
||||||
|
# matters at build time: same commit in, same digest and same reported
|
||||||
|
# version out.
|
||||||
|
rendered = subprocess.run(
|
||||||
|
["git", "show", "-s", "--format=%cd", "--date=format-local:%Y.%m.%d.%H%M", sha],
|
||||||
|
capture_output=True, text=True, check=True, cwd=ROOT,
|
||||||
|
env={"TZ": "UTC", "PATH": os.environ.get("PATH", "")},
|
||||||
|
).stdout.strip()
|
||||||
|
assert segments(artifacts("version", artifact).strip()) == segments(rendered)
|
||||||
|
|
||||||
|
|
||||||
|
def test_epoch_is_stable_across_calls():
|
||||||
|
"""SOURCE_DATE_EPOCH's entire job is to be the same on the next build.
|
||||||
|
|
||||||
|
A value that moved between two invocations on one unchanged checkout would
|
||||||
|
reintroduce #3265 through the very mechanism meant to close it, and the
|
||||||
|
symptom would be indistinguishable: a digest that changes for no reason.
|
||||||
|
"""
|
||||||
|
for artifact in ARTIFACTS:
|
||||||
|
first = artifacts("epoch", artifact).strip()
|
||||||
|
second = artifacts("epoch", artifact).strip()
|
||||||
|
assert first == second, f"{artifact} epoch moved: {first} then {second}"
|
||||||
|
|||||||
+101
-17
@@ -24,12 +24,36 @@ SCRIPT = ROOT / "scripts" / "release_notes.py"
|
|||||||
|
|
||||||
|
|
||||||
def notes(*args: str, cwd: Path | None = None) -> str:
|
def notes(*args: str, cwd: Path | None = None) -> str:
|
||||||
|
"""Run the script the way release.yml does.
|
||||||
|
|
||||||
|
A synthetic repo runs its OWN copy of the script, because the overview is
|
||||||
|
read relative to `__file__` rather than to the cwd — which is right in
|
||||||
|
production (release.yml checks out the tag, so the script IS the tagged
|
||||||
|
tree's copy) and would otherwise make every synthetic repo silently quote
|
||||||
|
FabledCurator's real README.
|
||||||
|
"""
|
||||||
|
root = cwd or ROOT
|
||||||
|
script = root / "scripts" / "release_notes.py"
|
||||||
return subprocess.run(
|
return subprocess.run(
|
||||||
["python3", str(SCRIPT), "--dry-run", *args],
|
["python3", str(script if script.exists() else SCRIPT), "--dry-run", *args],
|
||||||
capture_output=True, text=True, check=True, cwd=cwd or ROOT,
|
capture_output=True, text=True, check=True, cwd=root,
|
||||||
).stdout
|
).stdout
|
||||||
|
|
||||||
|
|
||||||
|
OVERVIEW_TEXT = "A synthetic product, described once."
|
||||||
|
|
||||||
|
|
||||||
|
def install_script(repo: Path, *, overview: bool = True) -> None:
|
||||||
|
"""Give a synthetic repo the script and a README to quote."""
|
||||||
|
(repo / "scripts").mkdir(exist_ok=True)
|
||||||
|
(repo / "scripts" / "release_notes.py").write_text(SCRIPT.read_text())
|
||||||
|
(repo / "scripts" / "artifacts.sh").write_text("#!/bin/sh\nexit 1\n")
|
||||||
|
readme = "# Synthetic\n\n"
|
||||||
|
if overview:
|
||||||
|
readme += f"<!-- overview:start -->\n{OVERVIEW_TEXT}\n<!-- overview:end -->\n"
|
||||||
|
(repo / "README.md").write_text(readme)
|
||||||
|
|
||||||
|
|
||||||
def body_of(out: str) -> str:
|
def body_of(out: str) -> str:
|
||||||
assert "--- body ---" in out, f"no body was rendered:\n{out}"
|
assert "--- body ---" in out, f"no body was rendered:\n{out}"
|
||||||
return out.split("--- body ---", 1)[1]
|
return out.split("--- body ---", 1)[1]
|
||||||
@@ -56,9 +80,10 @@ def shaped_history(tmp_path: Path) -> Path:
|
|||||||
repo = tmp_path / "shaped"
|
repo = tmp_path / "shaped"
|
||||||
repo.mkdir()
|
repo.mkdir()
|
||||||
git(repo, "init", "-q", "-b", "main")
|
git(repo, "init", "-q", "-b", "main")
|
||||||
|
install_script(repo)
|
||||||
for i, tag in enumerate(("v26.06.04.0", "v2026.08.28.2208", "v2026.08.29.1000")):
|
for i, tag in enumerate(("v26.06.04.0", "v2026.08.28.2208", "v2026.08.29.1000")):
|
||||||
(repo / "f.txt").write_text(f"{i}\n")
|
(repo / "f.txt").write_text(f"{i}\n")
|
||||||
git(repo, "add", "f.txt")
|
git(repo, "add", "-A")
|
||||||
git(repo, "commit", "-q", "-m", f"work landing in {tag}")
|
git(repo, "commit", "-q", "-m", f"work landing in {tag}")
|
||||||
git(repo, "tag", tag)
|
git(repo, "tag", tag)
|
||||||
# One more commit and a merge, so the merge-exclusion test has something to
|
# One more commit and a merge, so the merge-exclusion test has something to
|
||||||
@@ -109,12 +134,57 @@ def test_merges_are_excluded_so_the_list_is_the_work(shaped_history):
|
|||||||
assert "Merge pull request #999" not in body
|
assert "Merge pull request #999" not in body
|
||||||
|
|
||||||
|
|
||||||
def test_the_first_release_still_renders_with_nothing_behind_it(shaped_history):
|
def test_a_pre_convention_tag_is_history_not_a_predecessor(shaped_history):
|
||||||
"""No previous tag is reachable from the oldest one. That is a real state,
|
"""The defect milestone 328 hit, and the reason the match glob narrowed.
|
||||||
not an error, and it must not take the release down with it."""
|
|
||||||
out = notes("v26.06.04.0", cwd=shaped_history)
|
The 28 `v26.*` tags are kept as history while their releases were deleted.
|
||||||
|
Ancestry alone happily names `v26.06.04.0` as the predecessor of the first
|
||||||
|
rule-148 tag — and then the body offers "changes since" a release that no
|
||||||
|
longer exists, over a span (533 commits in the real repo) that is the
|
||||||
|
internal build-out this milestone exists to stop publishing.
|
||||||
|
|
||||||
|
Reachable is not the same as comparable. Only a `vYYYY.` tag names a
|
||||||
|
release a reader could have been running.
|
||||||
|
"""
|
||||||
|
out = notes("v2026.08.28.2208", cwd=shaped_history)
|
||||||
assert "previous=<none>" in out
|
assert "previous=<none>" in out
|
||||||
assert "## Changes" in body_of(out)
|
assert "v26.06.04.0" not in body_of(out)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_first_release_describes_the_product_instead_of_diffing(shaped_history):
|
||||||
|
"""No predecessor means nobody reading has run an earlier version, so a
|
||||||
|
changelog has no referent. The alternative the script used to take — diff
|
||||||
|
against the whole history, truncated — puts 200 lines of internal build-out
|
||||||
|
in front of someone meeting the project for the first time."""
|
||||||
|
body = body_of(notes("v2026.08.28.2208", cwd=shaped_history))
|
||||||
|
assert OVERVIEW_TEXT in body
|
||||||
|
assert "## Changes" not in body
|
||||||
|
assert not [ln for ln in body.split("\n") if ln.startswith("- work landing")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_overview_is_readmes_words_not_a_second_copy(shaped_history):
|
||||||
|
"""Two hand-maintained descriptions of one product drift and nothing
|
||||||
|
catches it. The release page quotes README.md so there is one source."""
|
||||||
|
readme = (shaped_history / "README.md").read_text()
|
||||||
|
assert OVERVIEW_TEXT in readme
|
||||||
|
assert OVERVIEW_TEXT in body_of(notes("v2026.08.28.2208", cwd=shaped_history))
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_missing_overview_block_is_reported_and_still_publishes(tmp_path):
|
||||||
|
"""Same reasoning as cross_checks(): the release is the useful object even
|
||||||
|
when part of the derivation could not run. Say what is missing, publish
|
||||||
|
anyway — do not leave the operator with a tag and no release."""
|
||||||
|
repo = tmp_path / "no-markers"
|
||||||
|
repo.mkdir()
|
||||||
|
git(repo, "init", "-q", "-b", "main")
|
||||||
|
install_script(repo, overview=False)
|
||||||
|
git(repo, "add", "-A")
|
||||||
|
git(repo, "commit", "-q", "-m", "first")
|
||||||
|
git(repo, "tag", "v2026.09.01.1200")
|
||||||
|
|
||||||
|
out = notes("v2026.09.01.1200", cwd=repo)
|
||||||
|
assert "No `<!-- overview:start -->` block found" in out
|
||||||
|
assert "No `<!-- overview:start -->` block found" in body_of(out)
|
||||||
|
|
||||||
|
|
||||||
def test_a_non_tag_ref_renders_but_refuses_to_claim_it_published():
|
def test_a_non_tag_ref_renders_but_refuses_to_claim_it_published():
|
||||||
@@ -135,15 +205,29 @@ def test_the_rollback_refs_name_all_three_images():
|
|||||||
assert f"bvandeusen/{image}:c-" in body, f"{image} missing from the rollback refs"
|
assert f"bvandeusen/{image}:c-" in body, f"{image} missing from the rollback refs"
|
||||||
|
|
||||||
|
|
||||||
def test_an_unbounded_span_is_truncated_and_says_so():
|
def test_a_long_span_between_two_releases_is_truncated_and_says_so(tmp_path):
|
||||||
"""With no reachable previous tag the span is the whole history. Emitting
|
"""The cap is still reachable, just not by the route it used to be.
|
||||||
eleven hundred lines would bury the one line explaining why there are
|
|
||||||
eleven hundred of them, so the cap is part of the message, not a silent
|
It no longer fires on "no predecessor" — that renders the overview now.
|
||||||
slice."""
|
What it still guards is two real releases far enough apart that the list
|
||||||
out = notes("HEAD")
|
stops being something anyone reads, which is the ordinary case for a
|
||||||
if "previous=<none>" not in out:
|
project that cuts a bookmark twice a year. The cap is part of the message,
|
||||||
pytest.skip("a previous tag is reachable from HEAD in this checkout")
|
not a silent slice.
|
||||||
|
"""
|
||||||
|
repo = tmp_path / "long"
|
||||||
|
repo.mkdir()
|
||||||
|
git(repo, "init", "-q", "-b", "main")
|
||||||
|
install_script(repo)
|
||||||
|
git(repo, "add", "-A")
|
||||||
|
git(repo, "commit", "-q", "-m", "scaffold")
|
||||||
|
git(repo, "tag", "v2026.01.01.0000")
|
||||||
|
for i in range(205):
|
||||||
|
git(repo, "commit", "-q", "--allow-empty", "-m", f"fix: change {i}")
|
||||||
|
git(repo, "tag", "v2026.07.01.0000")
|
||||||
|
|
||||||
|
out = notes("v2026.07.01.0000", cwd=repo)
|
||||||
|
assert "previous=v2026.01.01.0000" in out
|
||||||
body = body_of(out)
|
body = body_of(out)
|
||||||
listed = [ln for ln in body.split("\n") if ln.startswith("- ")]
|
listed = [ln for ln in body.split("\n") if ln.startswith("- ")]
|
||||||
assert len(listed) <= 200
|
assert len(listed) == 200
|
||||||
assert "more than a changelog is for" in body
|
assert "more than a changelog is for" in body
|
||||||
|
|||||||
Reference in New Issue
Block a user