bvandeusen/FabledCurator
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 12 Wiki Activity

Schema reconciliation + index hygiene, and the weekly base-image refresh #243

Merged
bvandeusen merged 17 commits from dev into main 2026-08-31 08:34:55 -04:00
Conversation 0 Commits 17 Files Changed 34
+216 -6
2 changed files with 216 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 63e0a423d7 - Show all commits
.forgejo/workflows/build.yml
+181 -6
View File
@@ -45,6 +45,36 @@ on:
type: boolean
default: false
# The base-image refresh (milestone 326 step 4, #3154).
#
# Skip-if-exists is keyed on OUR source, so an artifact whose source stops
# moving stops picking up base-image updates. `agent/` last changed
# 2026-07-17; every push since has correctly declined to rebuild it, which
# also means it will serve that day's `nvidia/cuda` layers forever. Nothing
# is wrong until it has been unchanged for months, which is precisely why
# this is a calendar trigger and not a condition on the push path.
#
# Weekly, Sunday 06:00 UTC. Away from CI-runner's Monday security sweep so
# the two are never diagnosing each other, and on the quietest day so a
# surprise rebuild is not competing with a push.
schedule:
- cron: '0 6 * * 0'
# Which branch a run BUILDS, as opposed to which one triggered it.
#
# They are the same thing on every trigger but `schedule`. Forgejo registers a
# cron from the DEFAULT branch — `dev` here — so a scheduled run arrives with
# `github.ref` pointing at dev, and a refresh that rebuilt `:dev` would be
# refreshing the one channel that gets rebuilt constantly anyway. Production is
# `main` (rule 147), and `:latest` is the tag that goes stale.
#
# So the ref is decided once, here, and every checkout in the file takes it.
# Deriving it per job invites the two halves to disagree: sign-extension would
# derive dev's extension version while build-web bundled main's, and the
# release download would 404 on a version that exists perfectly well.
env:
BUILD_REF: ${{ github.event_name == 'schedule' && 'main' || github.ref }}
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
# - write:package, read:package (for docker push to git.fabledsword.com)
# - write:release (for ext-<version> release asset cache)
@@ -88,6 +118,10 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history is load-bearing, not a convenience: the version this
# job signs is derived from the commit TIME of the newest packaged
# extension change. A depth-1 clone sees one commit and derives a
@@ -364,6 +398,10 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history: this job RE-DERIVES the extension version rather than
# being handed it, and a depth-1 clone derives a wrong, too-low value
# rather than failing — which would 404 the download of a release
@@ -429,8 +467,30 @@ jobs:
# everywhere). Operator-flagged 2026-06-01 after the first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list; see the comment there.
if [ "${GITHUB_REF##*/}" = "main" ]; then
# A scheduled refresh publishes the CHANNEL and nothing else
# (#3154). :c-<sha> for main's HEAD already exists and names the
# bytes that commit actually built; re-pushing it over refreshed
# base layers would break the one tag rule 145 makes immutable —
# and it is the rollback unit, so the breakage would surface on the
# day somebody needed it.
#
# The accepted consequence: between a refresh and the next main
# push, :latest and :c-<sha> point at different manifests. That is
# the design, not drift. They RE-CONVERGE on that push — it hits
# reuse (a refresh does not move fc.revision, because it does not
# touch the source), and the repoint step then writes the new
# :c-<sha> from the refreshed :latest. So the rollback unit ends up
# naming the bytes production is actually running, which is the
# property that matters.
#
# Checked BEFORE the ref test, not after: a scheduled run's
# GITHUB_REF is the default branch (dev), so the main test would
# never fire on it.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
else
@@ -524,6 +584,10 @@ jobs:
# this runner is known to evaluate. Read through env rather than
# interpolated into the run block, same rule as release.yml's TAG.
FORCE: ${{ github.event.inputs.force_build }}
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: |
set -eu
DERIVED=$(sh scripts/artifacts.sh revision web)
@@ -570,6 +634,9 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: already published — skipping the build"
@@ -661,6 +728,30 @@ jobs:
context: .
file: Dockerfile
push: true
# Re-resolve the FROM references against the registry instead of
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds. If it did not move,
# the registry cache satisfies the entire graph and the refresh is a
# ~13s no-op that republishes nothing.
#
# That no-op is the POINT, not a shortfall: :latest should change
# when there is something new in it and not otherwise. A refresh
# that rewrote the image weekly regardless would churn the registry
# and hand :c-<sha> a new manifest to diverge from every Sunday, for
# no gain.
#
# What it therefore does NOT catch: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole — but closing it
# would take `no-cache: true` on the scheduled path, which is the
# weekly-churn trade above. Left as the cheaper of the two on
# purpose.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
# ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver,
@@ -783,6 +874,12 @@ jobs:
ARGS="$ARGS -t $t"
done
unset IFS
#
# This is also the whole of the scheduled refresh's tag handling
# (#3154): a refresh's tag list is the channel tag alone, so SOURCE
# is the only entry, it gets excluded, and this step correctly does
# nothing. No `if:` on the step and no schedule special-case —
# excluding the source was already the right rule.
if [ -z "$ARGS" ]; then
echo "repoint: $SOURCE is the only tag for this channel and"
echo "repoint: already holds this revision — nothing to write."
@@ -799,6 +896,10 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history: this job derives its artifact's version from the
# commit its shipped files last changed in (milestone 313). A
# depth-1 clone cannot see that commit — it either derives a wrong,
@@ -843,8 +944,12 @@ jobs:
# everywhere). Operator-flagged 2026-06-01 after first :c-<sha>
# main-push build failed at this step.
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list; see the comment there.
if [ "${GITHUB_REF##*/}" = "main" ]; then
# Mirrors build-web's tag list and its schedule handling; see
# the comments there.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-ml:latest,git.fabledsword.com/bvandeusen/fabledcurator-ml:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
else
@@ -921,6 +1026,10 @@ jobs:
# this runner is known to evaluate. Read through env rather than
# interpolated into the run block, same rule as release.yml's TAG.
FORCE: ${{ github.event.inputs.force_build }}
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: |
set -eu
DERIVED=$(sh scripts/artifacts.sh revision ml)
@@ -963,6 +1072,9 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: already published — skipping the build"
@@ -978,6 +1090,30 @@ jobs:
context: .
file: Dockerfile.ml
push: true
# Re-resolve the FROM references against the registry instead of
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds. If it did not move,
# the registry cache satisfies the entire graph and the refresh is a
# ~13s no-op that republishes nothing.
#
# That no-op is the POINT, not a shortfall: :latest should change
# when there is something new in it and not otherwise. A refresh
# that rewrote the image weekly regardless would churn the registry
# and hand :c-<sha> a new manifest to diverge from every Sunday, for
# no gain.
#
# What it therefore does NOT catch: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole — but closing it
# would take `no-cache: true` on the scheduled path, which is the
# weekly-churn trade above. Left as the cheaper of the two on
# purpose.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
# ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver,
@@ -1113,6 +1249,10 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
# Not the triggering ref — see the `env:` block at the top. On a
# scheduled refresh this is `main`; on everything else it is the ref
# that fired, so this is a no-op on every ordinary path.
ref: ${{ env.BUILD_REF }}
# Full history: this job derives its artifact's version from the
# commit its shipped files last changed in (milestone 313). A
# depth-1 clone cannot see that commit — it either derives a wrong,
@@ -1152,8 +1292,12 @@ jobs:
id: tag
run: |
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
# Mirrors build-web's tag list; see the comment there.
if [ "${GITHUB_REF##*/}" = "main" ]; then
# Mirrors build-web's tag list and its schedule handling; see
# the comments there.
if [ "${GITHUB_EVENT_NAME:-}" = "schedule" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
elif [ "${GITHUB_REF##*/}" = "main" ]; then
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
echo "channel=main" >> "$GITHUB_OUTPUT"
else
@@ -1230,6 +1374,10 @@ jobs:
# this runner is known to evaluate. Read through env rather than
# interpolated into the run block, same rule as release.yml's TAG.
FORCE: ${{ github.event.inputs.force_build }}
# A scheduled refresh has to bypass reuse by construction: it
# rebuilds the SAME source, so fc.revision always matches and the
# check would skip every refresh there has ever been.
EVENT: ${{ github.event_name }}
run: |
set -eu
DERIVED=$(sh scripts/artifacts.sh revision agent)
@@ -1272,6 +1420,9 @@ jobs:
if [ "${FORCE:-false}" = "true" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: force_build set — building regardless"
elif [ "${EVENT:-}" = "schedule" ]; then
echo "hit=false" >> "$GITHUB_OUTPUT"
echo "reuse: scheduled base refresh — building regardless"
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
echo "hit=true" >> "$GITHUB_OUTPUT"
echo "reuse: already published — skipping the build"
@@ -1287,6 +1438,30 @@ jobs:
context: agent
file: agent/Dockerfile
push: true
# Re-resolve the FROM references against the registry instead of
# trusting whatever digest the cache was built against. This is the
# whole mechanism of the scheduled refresh (#3154): if the base tag
# moved, the FROM layer's cache key changes, every layer above it
# invalidates, and the image genuinely rebuilds. If it did not move,
# the registry cache satisfies the entire graph and the refresh is a
# ~13s no-op that republishes nothing.
#
# That no-op is the POINT, not a shortfall: :latest should change
# when there is something new in it and not otherwise. A refresh
# that rewrote the image weekly regardless would churn the registry
# and hand :c-<sha> a new manifest to diverge from every Sunday, for
# no gain.
#
# What it therefore does NOT catch: a Debian package update inside
# the `apt-get install` layer while the base tag itself stands
# still. The official python/cuda images rebuild with those updates
# baked in, so this is a lag rather than a hole — but closing it
# would take `no-cache: true` on the scheduled path, which is the
# weekly-churn trade above. Left as the cheaper of the two on
# purpose.
#
# Only on the schedule. An ordinary push wants the cached base.
pull: ${{ github.event_name == 'schedule' }}
# ONE tag, the channel's. Every other tag is written by the step
# below, registry-side. buildx here pushes the first tag to the
# registry and then re-pushes the rest through the DOCKER driver,
ci-requirements.md
+35
View File
@@ -167,6 +167,41 @@ per `docs/process.md`'s "add deps to the image when used by >1 project".
`github.event.inputs` into an env var rather than interpolated into a run
block, and it is checked inside the reuse step so that one decision drives
both the build and the repoint.
- **A weekly `schedule` rebuilds all three images against fresh base layers**
(Sunday 06:00 UTC, milestone 326 step 4, #3154). Skip-if-exists is keyed on
OUR source, so an artifact whose source stops moving stops picking up base
updates — `agent/` has not changed since 2026-07-17 and would otherwise serve
that day's `nvidia/cuda` layers forever. Four things make it work:
- It **builds `main`, not the branch that triggered it.** Forgejo registers a
cron from the DEFAULT branch (`dev` here), so a scheduled run arrives with
`github.ref` on dev. The ref is decided once in a top-level `env:
BUILD_REF` that every checkout in the file takes, rather than per job —
otherwise `sign-extension` would derive dev's extension version while
`build-web` bundled main's, and the release download would 404 on a version
that exists perfectly well.
- It **publishes only `:latest`.** `:c-<sha>` for main's HEAD already names
the bytes that commit built; re-pushing it over refreshed layers would
break the one tag rule 145 makes immutable, and it is the rollback unit.
The repoint step needs no schedule case for this — the tag list is the
channel tag alone, so SOURCE is the only entry, it is excluded as always,
and the step correctly does nothing.
- **`:latest` and `:c-<sha>` therefore diverge between a refresh and the next
`main` push, by design.** They re-converge on that push: it hits reuse (a
refresh does not move `fc.revision`, because it does not touch the source),
and the repoint writes the NEW `:c-<sha>` from the refreshed `:latest`. The
push path needed no change for this, because the repoint already excluded
the source tag — the same rule that keeps the label readable also keeps a
refresh from being undone.
- **`pull: true` on the scheduled path only** is the actual mechanism. If a
base tag moved, the `FROM` layer's cache key changes and everything above
it rebuilds; if it did not, the registry cache satisfies the whole graph
and the refresh is a ~13s no-op that republishes nothing. That no-op is the
point — `:latest` should change when there is something new in it, not
every Sunday. The known lag: a Debian package update inside the `apt-get
install` layer while the base tag stands still is not caught. Closing it
needs `no-cache: true`, which buys weekly churn for it; the official
python/cuda images rebuild with those updates baked in, so this is a lag
rather than a hole.
- **`FC_CHANNEL` and `FC_VERSION` are build args, not runtime settings.**
`build.yml` passes them to the web image only — the ml and agent images have
nothing to report them to. `/api/health` returns both, the foot of Settings