bvandeusenandClaude Opus 5
e1cd629196
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 2s
CI and images / frontend-build (push) Successful in 19s
extension / lint (push) Successful in 21s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m25s
CI and images / sign-extension (push) Successful in 4s
CI and images / build-web (push) Successful in 2m1s
CI and images / smoke-web (push) Successful in 1m2s
CI and images / build-agent (push) Successful in 10m7s
CI and images / promote (push) Skipped
Operator: the agent's build string could not identify the agent. VERSION was a
literal in app.py an author was meant to bump, and nobody did — the September
image printed the same "2026-07-17.1" as the July one, so the one surface
meant to answer "did my pull work?" answered the same either way.
Nothing new was needed. scripts/artifacts.sh has derived a version per
artifact since milestone 313, and build-agent has been computing the agent's
on every run and printing it to the log. The image just never carried it.
Three values, never folded together (rule 149):
FC_VERSION YYYY.MM.DD.HHMM from the COMMIT its shipped files last changed
in — identical on dev and main for the same source, which is
what makes "am I running production's code?" answerable.
FC_CHANNEL a sibling field, never a suffix inside the name.
FC_REVISION the 12-char sha; the same string as the fc.revision LABEL, so
the image and the registry cannot disagree about which commit
this is.
The page SHOWS the version and COMPARES the revision. Those were one value
before, which is how a version acquires a second job and then cannot be
changed without breaking the reload banner. An unstamped local build reads
`unknown` and compares `local` — absent rather than empty, one spelling of
"cannot say".
scripts/artifacts.sh joins the AGENT path set in the same commit, and it had
to: a version has no backstop. A revision that is computed differently stops
matching the published label and forces a rebuild, so it self-corrects; a
version is compared against nothing, so a change to cmd_version alone would
leave the agent publishing the old format with nothing to contradict it. That
is #3202's finding, and the agent was rightly exempt only while it had no
version of its own. tests/test_artifact_paths.py pins it.
Also corrects two build.yml comments claiming agent/ had not changed since
2026-07-17. Both were already false — it changed 2026-09-23 — and one of them
is the stated rationale for the force_build escape hatch. Rewritten without
dates: how long an artifact has been quiet is a `git log` question, and its
answer in a comment is wrong the next time anyone commits (lesson #4383).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2538 lines
141 KiB
YAML
2538 lines
141 KiB
YAML
name: CI and images
|
|
|
|
on:
|
|
push:
|
|
# `:dev` builds were dropped 2026-05-26 to save a docker build per dev
|
|
# push, on the reasoning that "operator tests from `:latest` after
|
|
# merge-to-main". Restored 2026-08-27: that is testing by shipping, and
|
|
# family rules 146/147 now name it directly — `main` IS production, and a
|
|
# channel that can only be refreshed by shipping is not a channel. The
|
|
# pressure to merge in order to try something does not come from
|
|
# carelessness; it comes from `:dev` being unable to carry the build.
|
|
#
|
|
# BOTH images build on dev, deliberately: a `:dev` web image paired
|
|
# with a stale `:dev` agent is a worse trap than no dev channel at all,
|
|
# since the mismatch only shows up as a runtime failure. (It was three
|
|
# until #4311 — `fabledcurator-ml` was the same bytes as `fabledcurator`
|
|
# under a second name, and the stack that needed the second name is being
|
|
# collapsed onto the consolidated image.)
|
|
branches: [main, dev]
|
|
|
|
# Renovate opens PRs from `renovate/*` branches into `dev`. Those branches
|
|
# never push to dev/main, so the push trigger above gives them NO pre-merge
|
|
# CI — a bump could only be validated after it was already merged. Base
|
|
# `dev` only: it deliberately does NOT fire on dev→main PRs, which rely on
|
|
# the dev push run, so no duplicate runs. FC has no fork PRs (single-operator
|
|
# Forgejo repo), so secrets-on-PR is not a concern.
|
|
#
|
|
# Nothing PUBLISHES on this trigger — see the `if:` on the three publishing
|
|
# jobs below. A PR run is the lanes and nothing else.
|
|
pull_request:
|
|
branches: [dev]
|
|
#
|
|
# NO tag trigger (milestone 318 step 2). A `v*` tag names a commit `main`
|
|
# already built and published; rebuilding it produces the same source under
|
|
# the same names and RE-PUSHES `:c-<sha>`, which rule 145 forbids even when
|
|
# the bytes match — image configs carry timestamps, so "same source" does
|
|
# not mean "same manifest". The release build was publishing nothing new
|
|
# and violating an immutability rule to do it.
|
|
#
|
|
# Releases still happen (rule 148, on explicit request per rule 2). They
|
|
# produce a changelog, not an image.
|
|
|
|
# The escape hatch for the one thing skip-if-exists makes untestable: a
|
|
# build that WOULD be skipped. An artifact whose source is quiet has its
|
|
# build path correctly declined on every push — which also means that path
|
|
# is not exercised, and cannot be exercised on demand, for as long as the
|
|
# quiet lasts. #3190 lives on exactly such a path.
|
|
#
|
|
# This used to name the agent and a date ("unchanged since 2026-07-17"). It
|
|
# was already false when read on 2026-09-24 — the agent had changed the day
|
|
# before — and a rationale resting on a stale fact reads as settled
|
|
# reasoning forever after (lesson #4383). Which artifact is quiet is a
|
|
# question for `git log`, not for a comment; the hatch exists because ANY of
|
|
# them can be.
|
|
#
|
|
# Editing build.yml does not force one either, and that is deliberate: the
|
|
# workflow is not shipped bytes, so it is in no artifact's path set. Putting
|
|
# it in one would re-version every artifact for a comment change.
|
|
#
|
|
# ONE input, not one per artifact. Forcing both is cheap once the registry
|
|
# cache is warm (#3114), and a boolean per artifact is an interface nobody
|
|
# remembers the meaning of.
|
|
workflow_dispatch:
|
|
inputs:
|
|
force_build:
|
|
description: 'Rebuild every image even if the published revision matches'
|
|
type: boolean
|
|
default: false
|
|
refresh:
|
|
description: 'Behave as the weekly base refresh: build main against fresh bases, publish through the candidate tag'
|
|
type: boolean
|
|
default: false
|
|
|
|
# The base-image refresh (milestone 326 step 4, #3154).
|
|
#
|
|
# Skip-if-exists is keyed on OUR source, so an artifact whose source stops
|
|
# moving stops picking up base-image updates — it will serve the base layers
|
|
# of its last build forever. The agent is the standing example because its
|
|
# base is the heaviest (`nvidia/cuda`, ~6.3 GB) and its source the quietest:
|
|
# 48 of 1111 commits touch it, about 4% (#3114).
|
|
#
|
|
# Deliberately no date here. Nothing is wrong until an artifact has been
|
|
# unchanged for months, and "how long has it been" is a `git log` question
|
|
# whose answer in a comment is wrong the next time anyone commits — which is
|
|
# exactly how the previous version of this paragraph came to claim the agent
|
|
# had not moved since 2026-07-17 while it had moved the day before. That is
|
|
# precisely why this is a CALENDAR trigger and not a condition on the push
|
|
# path: the calendar cannot be wrong about how much time has passed.
|
|
#
|
|
# Weekly, Sunday 06:00 UTC. Away from CI-runner's Monday security sweep so
|
|
# the two are never diagnosing each other, and on the quietest day so a
|
|
# surprise rebuild is not competing with a push.
|
|
schedule:
|
|
- cron: '0 6 * * 0'
|
|
|
|
# One build.yml run per branch at a time (#4290).
|
|
#
|
|
# Without this, two pushes to one branch run in full parallel. Both read
|
|
# `fc.revision` off the channel tag before either has pushed, so both miss the
|
|
# reuse check and both build, and whichever finishes LAST owns the tag — so a
|
|
# slower older build can leave `:dev` carrying content older than the commit
|
|
# that moved it. Family rule 146 says a rolling channel refreshes itself; that
|
|
# is the case where it quietly does not.
|
|
#
|
|
# `cancel-in-progress: false` — QUEUE, never cancel. Cancelling could kill
|
|
# sign-extension mid-AMO-upload, leaving the version registered at AMO with no
|
|
# cached asset: exactly the stuck state the rollback trap in that job exists to
|
|
# prevent, reached through a different door. AMO will not release a burned
|
|
# version, so that state is unrecoverable rather than merely annoying. Waiting
|
|
# a few minutes is the cheaper end of that trade by a wide margin.
|
|
#
|
|
# Keyed on `github.ref`, so `dev` and `main` never block each other. Not on
|
|
# BUILD_REF: the group is evaluated before any job starts and cannot read the
|
|
# `env` context (the same restriction that makes a job-level `if:` unable to
|
|
# see it — see build-web's `outputs.candidate` note). The one consequence is
|
|
# that a scheduled refresh, whose ref is the default branch, shares dev's
|
|
# queue while publishing main's channel. It only ever waits, and it fires
|
|
# 06:00 Sunday precisely because nothing else is running then.
|
|
#
|
|
# UNVERIFIED AT THE TIME OF WRITING, and this file has been burned by exactly
|
|
# that before: the `format()` note below records `true == 'true'` evaluating
|
|
# FALSE on run 5270 with no symptom whatsoever — every lane green, the feature
|
|
# simply not happening. A `concurrency:` key this Gitea ignored would look
|
|
# identical: runs still overlapping, nothing red. So this is a belt, and the
|
|
# digest-pinned repoint in each build job is the braces — that one makes the
|
|
# :c-<sha> correctness property hold whether or not this key is honoured.
|
|
# Confirm by pushing twice in quick succession and reading the RUN LIST for a
|
|
# queued second run, never by reading this comment.
|
|
concurrency:
|
|
group: build-${{ github.ref }}
|
|
cancel-in-progress: false
|
|
|
|
# Which branch a run BUILDS, as opposed to which one triggered it.
|
|
#
|
|
# They are the same thing on every trigger but `schedule`. Forgejo registers a
|
|
# cron from the DEFAULT branch — `dev` here — so a scheduled run arrives with
|
|
# `github.ref` pointing at dev, and a refresh that rebuilt `:dev` would be
|
|
# refreshing the one channel that gets rebuilt constantly anyway. Production is
|
|
# `main` (rule 147), and `:latest` is the tag that goes stale.
|
|
#
|
|
# So the ref is decided once, here, and every checkout in the file takes it.
|
|
# Deriving it per job invites the two halves to disagree: sign-extension would
|
|
# derive dev's extension version while build-web bundled main's, and the
|
|
# release download would 404 on a version that exists perfectly well.
|
|
# IS THIS A BASE REFRESH? Asked in five places and previously spelled five
|
|
# ways — `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in
|
|
# one shell, an `EVENT:` env passed into another, and a bare expression on
|
|
# `pull:`. Five spellings of one fact is how half of them come to disagree
|
|
# after somebody adds a sixth trigger.
|
|
#
|
|
# The `refresh` dispatch input is here so this path can be EXERCISED. A weekly
|
|
# cron is otherwise testable once a week, which is not a cadence anything can
|
|
# be developed against — the same reason `force_build` exists (#3252, added to
|
|
# confirm #3190 was gone rather than wait for it to recur). It is also what
|
|
# makes the milestone-362 gate verifiable at all: a gate has to be watched
|
|
# rejecting something before anyone can believe it is wired up.
|
|
#
|
|
# The input is normalised through `format()` before it is compared, and that
|
|
# is not defensive styling — the direct comparison is WRONG and fails silently.
|
|
#
|
|
# `type: boolean` delivers a real boolean, and GitHub expression semantics cast
|
|
# operands to numbers when their types differ: `true == 'true'` compares 1
|
|
# against NaN and is FALSE. Measured on run 5270, whose own log says it —
|
|
#
|
|
# expression '(github.event_name == 'schedule'
|
|
# || github.event.inputs.refresh == 'true') && 'true' || 'false''
|
|
# evaluated to '%!t(string=false)'
|
|
# trigger: raw inputs refresh='true'
|
|
#
|
|
# — the input arrived as `true` and the expression still said false. The run
|
|
# then went green with every step skipped, because a refresh that evaluates
|
|
# false behaves exactly like an ordinary push. That is the whole hazard: the
|
|
# failure has no symptom.
|
|
#
|
|
# `force_build` never hit this because it never compares in an expression. It
|
|
# passes the raw value into an env var and tests it in the shell, where
|
|
# everything is already a string. `format('{0}', x)` buys the same thing here,
|
|
# where a step-level `if:` needs the answer before any shell runs.
|
|
env:
|
|
IS_REFRESH: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'true' || 'false' }}
|
|
BUILD_REF: ${{ (github.event_name == 'schedule' || format('{0}', github.event.inputs.refresh) == 'true') && 'main' || github.ref }}
|
|
|
|
# Requires repo secret RELEASE_TOKEN — a Forgejo PAT with scopes:
|
|
# - write:package, read:package (for docker push to git.fabledsword.com)
|
|
# - write:release (for ext-<version> release asset cache)
|
|
# - write:issue (for future issue-management automation)
|
|
# The injected GITHUB_TOKEN cannot be used — it lacks write:package.
|
|
|
|
jobs:
|
|
# ---------------------------------------------------------------------------
|
|
# THE LANES. Merged in from `ci.yml`, which is deleted, 2026-09-23.
|
|
#
|
|
# They were a separate workflow on the same push trigger, which meant the
|
|
# build could not see their verdict and published regardless. Run 7348 is the
|
|
# worked example: the unit lane went red on `2f8f0bc` and `build-web` pushed
|
|
# `:dev` anyway, in the same minute. Operator: *"tighten the gate so :dev
|
|
# can't publish on red tests"*, then *"I don't want failing builds to publish
|
|
# anywhere going forward."*
|
|
#
|
|
# Two workflows cannot express that. A `needs:` edge only exists inside one
|
|
# graph — so the lanes and the publish are one graph now, and the gate is the
|
|
# `needs:` on the three publishing jobs rather than anything new.
|
|
#
|
|
# The cost, accepted knowingly: this workflow QUEUES per branch and never
|
|
# cancels (see `concurrency:` above), so on two pushes in quick succession
|
|
# the second push's lint feedback waits out the first run's build. That is
|
|
# the price of the edge, and it is the right way round — a slower red is
|
|
# better than a fast red that ships.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
# Fast-fail lint lane. ruff is pre-installed in the ci-python image, so
|
|
# this runs with NO dependency install and surfaces the most common bounce
|
|
# class (lint: I001 / UP037 / ASYNC109 / W293 …) in seconds — instead of
|
|
# after the backend job's ~30-60s wheel install. ruff is static analysis,
|
|
# so no DB/secret env is needed.
|
|
lint:
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Ruff lint
|
|
# agent/ included so the GPU-agent is linted before its image is built
|
|
# (build.yml only `docker build`s it — this is where it gets checked).
|
|
# scripts/ likewise: release_notes.py runs only on a tag push, so a
|
|
# syntax or import error there would otherwise surface at the one
|
|
# moment nobody wants to debug a workflow.
|
|
run: ruff check backend/ tests/ alembic/ agent/ scripts/
|
|
- name: Agent syntax check
|
|
# The agent's runtime deps (torch/transformers/ultralytics) aren't in the
|
|
# CI image, so we can't import it — but compileall parses every module,
|
|
# catching syntax errors before the image build.
|
|
run: python -m compileall -q agent/fc_agent
|
|
|
|
# The extension version is DERIVED, not hand-maintained (milestone 271 step
|
|
# 4): build.yml computes it from the commit TIME of the newest packaged
|
|
# extension change and stamps it into manifest.json / package.json at build
|
|
# time. The guard that used to live here — "packaged files changed but nobody
|
|
# bumped the version" — was therefore checking a fact that had stopped
|
|
# existing. Worse than useless: it would have failed this lane on every real
|
|
# extension change, demanding a bump that decides nothing. Retired 2026-08-27
|
|
# rather than left running beside the new mechanism (rule 22).
|
|
#
|
|
# Two things are still worth asserting, and this is the only lane that can:
|
|
# the extension.yml suite runs on node:24-slim, which is exactly why
|
|
# version.spec.js sticks to packaging.sh's git-free subcommands.
|
|
# 1. the derivation actually resolves on this commit
|
|
# 2. the derived string is one AMO will accept, checked against Mozilla's
|
|
# own published grammar rather than a loose "digits and dots"
|
|
#
|
|
# The MAJOR.MINOR-agreement check that used to be (2) is gone with milestone
|
|
# 318 step 8: the committed version no longer seeds anything, so there is no
|
|
# hand-set part left for the two files to disagree about.
|
|
#
|
|
# Deliberately NOT checked here: that the derived value beats what has already
|
|
# been signed. That guard belongs in build.yml, where it compares against the
|
|
# real ext-* releases. Comparing against origin/main here would be wrong —
|
|
# dev legitimately derives a LOWER value whenever main is ahead on the
|
|
# extension, and a lane that fails for being behind is a lane people learn to
|
|
# ignore.
|
|
extension-version:
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
# The derivation needs real history: a depth-1 clone sees one commit
|
|
# and produces a wrong, too-low value RATHER THAN FAILING. Checking
|
|
# that here is half the point of the lane.
|
|
fetch-depth: 0
|
|
- name: Extension version derives cleanly
|
|
run: |
|
|
set -eu
|
|
# busybox sh on the act_runner — no bashisms (family rule).
|
|
VERSION=$(sh extension/scripts/packaging.sh version)
|
|
echo "derived: $VERSION"
|
|
|
|
# Mozilla's published grammar for AMO, transcribed verbatim from
|
|
# MDN's manifest.json/version page:
|
|
#
|
|
# ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$
|
|
#
|
|
# Not the looser `^[0-9]+(\.[0-9]+)*$` this lane used to carry. That
|
|
# one passes `2026.08.29.0201`, which AMO REJECTS — a segment must be
|
|
# the single digit 0 or start 1-9 — and it also passes five segments,
|
|
# where AMO allows four. Both would surface as a failed sign with the
|
|
# version already burned: AMO 409s on re-signing, so a rejected value
|
|
# cannot be reclaimed and cannot be reused. This lane is the cheap
|
|
# place to find out. (#3138, milestone 318 step 8.)
|
|
if ! echo "$VERSION" | grep -qE '^(0|[1-9][0-9]{0,8})(\.(0|[1-9][0-9]{0,8})){0,3}$'; then
|
|
echo "ERROR: derived version '$VERSION' is not a version AMO accepts."
|
|
echo "AMO's grammar: ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$"
|
|
echo "Most likely cause: a zero-padded segment (08, 0201). The rest"
|
|
echo "of the family pads; the extension must not — see packaging.sh."
|
|
exit 1
|
|
fi
|
|
|
|
# ...and the shape this project actually derives. AMO would happily
|
|
# take `1.0.3500147` too, so the grammar check alone would not notice
|
|
# a regression to the pre-318 shape — which orders BELOW everything
|
|
# signed since, and is unrecoverable once Firefox has the higher one.
|
|
if ! echo "$VERSION" | grep -qE '^20[0-9][0-9]\.[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,4}$'; then
|
|
echo "ERROR: derived version '$VERSION' is not YYYY.M.D.HHMM."
|
|
echo "Rule 148's CalVer is what build.yml signs; the old"
|
|
echo "1.0.<minutes> shape would order below every ext-2026.* release."
|
|
exit 1
|
|
fi
|
|
echo "OK: derived version $VERSION"
|
|
|
|
backend-lint-and-test:
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
env:
|
|
# DB_PASSWORD and SECRET_KEY are required by config.py at import time
|
|
# even though unit tests don't actually touch the DB or use the secret.
|
|
DB_PASSWORD: ci_unit_test_placeholder
|
|
SECRET_KEY: ci_unit_test_placeholder
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
# Full history for tests/test_artifact_identity.py, which derives
|
|
# each artifact's revision to check the identity scheme. On a
|
|
# depth-1 clone that derivation either fails or returns the tip sha
|
|
# — so the lane would go green while asserting nothing, which is
|
|
# the one outcome worse than a red one.
|
|
fetch-depth: 0
|
|
|
|
# Cache step removed 2026-05-26: act_runner's cache backend has been
|
|
# broken on this homelab runner since 2026-05-15 (first as request-
|
|
# timeout warnings, then as hard "Cannot find module .../dist/restore/
|
|
# index.js" failures that tank the whole job). The cache step targeted
|
|
# ~/.cache/pip but the install below uses `uv pip install` primarily,
|
|
# whose own cache lives at ~/.cache/uv — so the cache step's real
|
|
# benefit was marginal even when working. Cost of removal: ~30s of
|
|
# wheel downloads per job. Future re-enable: mount ~/.cache/uv as a
|
|
# docker volume at the runner level (skips actions/cache entirely),
|
|
# or fix the runner-side cache backend (clear /var/run/act/actions/*,
|
|
# pin act_runner version, etc.).
|
|
|
|
- name: Install Python deps
|
|
# ruff is pre-installed in the ci-python image (see CI-Runner/CI-python/
|
|
# Dockerfile's RUFF_VERSION). Per FabledRulebook ci-runners.md, toolchain
|
|
# versions live on the runner image, not here.
|
|
# uv: 5-10x faster wheel resolve than pip for cold caches.
|
|
# Falls back to pip install on uv-missing runners (older images).
|
|
run: |
|
|
if command -v uv >/dev/null 2>&1; then
|
|
uv pip install --system -r requirements.txt pytest pytest-asyncio
|
|
else
|
|
pip install -r requirements.txt pytest pytest-asyncio
|
|
fi
|
|
|
|
# Ruff moved to the dedicated fast `lint` job above (fails in seconds,
|
|
# no dep install). This job is now unit tests only.
|
|
- name: Pytest (unit only — integration runs in the integration job)
|
|
run: pytest tests/ -v -m "not integration"
|
|
|
|
frontend-build:
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
defaults:
|
|
run:
|
|
working-directory: frontend
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
# No package-lock.json is tracked yet (we don't run npm locally per
|
|
# feedback-no-local-runs). Using `npm install` instead of `npm ci`.
|
|
# If we want strict lockfile-based reproducibility later, commit a
|
|
# package-lock.json and flip this back to `npm ci`.
|
|
- run: npm install --no-audit --no-fund
|
|
# No type-check step: the frontend is pure JS (no .ts files, no JSDoc),
|
|
# so a type-checker has nothing to do. The vue-tsc devDep + its `check`
|
|
# script were dropped 2026-07-11 rather than bumped to v3. If we add
|
|
# TS/JSDoc later, re-add a tsconfig.json + vue-tsc + a type-check step.
|
|
- run: npm run test:unit
|
|
- run: npm run build
|
|
|
|
# Single integration job — collapsed from a 3-way shard split on 2026-06-04.
|
|
# The shards existed to parallelize ~8.5min of integration tests; once the
|
|
# throwaway Postgres runs with fsync OFF (the durability step below) the whole
|
|
# suite runs in ~45s, so the split only triplicated the ~2min fixed overhead
|
|
# (container + `uv pip install` + `alembic upgrade head`) and burned 3 of 6
|
|
# runner slots for no wall-clock gain. One job now: spin up once, install
|
|
# once, migrate once, run every integration test.
|
|
#
|
|
# The docker-ps filter scopes to THIS job's own Postgres/Redis service
|
|
# containers by job name. act_runner strips underscores from job names when
|
|
# labelling containers (`int_api` matched nothing on 2026-05-25), so the name
|
|
# stays separator-free (`integration`). The step prints `docker ps -a` first
|
|
# so a future naming-convention shift surfaces in the log without a
|
|
# guess-and-push cycle.
|
|
#
|
|
# Pre-baking requirements.txt into ci-python:3.14 is intentionally NOT done —
|
|
# per ci-requirements.md, FC is the only Python consumer of that image and the
|
|
# CI-Runner "add deps to image when used by >1 project" rule keeps it per-job.
|
|
integration:
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
env:
|
|
DB_USER: fabledcurator
|
|
DB_PASSWORD: ci_integration
|
|
DB_PORT: "5432"
|
|
DB_NAME: fabledcurator_test
|
|
SECRET_KEY: ci_integration_placeholder
|
|
services:
|
|
postgres:
|
|
image: pgvector/pgvector:pg16
|
|
env:
|
|
POSTGRES_USER: fabledcurator
|
|
POSTGRES_PASSWORD: ci_integration
|
|
POSTGRES_DB: fabledcurator_test
|
|
options: >-
|
|
--health-cmd "pg_isready -U fabledcurator"
|
|
--health-interval 10s
|
|
--health-timeout 5s
|
|
--health-retries 10
|
|
redis:
|
|
image: redis:7-alpine
|
|
options: >-
|
|
--health-cmd "redis-cli ping"
|
|
--health-interval 10s
|
|
--health-timeout 5s
|
|
--health-retries 10
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Integration suite (resolve service IPs, migrate, test)
|
|
run: |
|
|
set -eux
|
|
echo "=== container landscape (diagnostic for filter scoping) ==="
|
|
docker ps -a --format '{{.ID}} {{.Image}} -> {{.Names}}'
|
|
echo "=== end landscape ==="
|
|
PG=$(docker ps --filter "name=integration" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
|
|
RD=$(docker ps --filter "name=integration" --filter "ancestor=redis:7-alpine" -q | head -n1)
|
|
test -n "$PG" && test -n "$RD"
|
|
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
|
|
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
|
|
test -n "$PG_IP" && test -n "$RD_IP"
|
|
export DB_HOST="$PG_IP"
|
|
export CELERY_BROKER_URL="redis://$RD_IP:6379/0"
|
|
export CELERY_RESULT_BACKEND="redis://$RD_IP:6379/0"
|
|
# These steps run under `sh -e`, not bash, so bash's /dev/tcp magic
|
|
# path does not exist here — the probe this loop used to run could
|
|
# never succeed and simply burned the full 120s on every run, green
|
|
# or red, then continued without having established anything. Python
|
|
# is in the image and needs no installed package for a socket
|
|
# connect, so it is the probe. Exhausting the budget is now a named
|
|
# failure rather than a silent fall-through (rule 156): if Postgres
|
|
# is genuinely not up, that is what the log should say, instead of
|
|
# whatever the first query happens to raise two minutes later.
|
|
pg_ready=""
|
|
for i in $(seq 1 60); do
|
|
if python -c "import socket,sys; s=socket.socket(); s.settimeout(2); sys.exit(0 if s.connect_ex(('$PG_IP', 5432)) == 0 else 1)"; then
|
|
pg_ready=1
|
|
break
|
|
fi
|
|
sleep 2
|
|
done
|
|
if [ -z "$pg_ready" ]; then
|
|
echo "postgres at $PG_IP:5432 did not accept a connection within 120s"
|
|
exit 1
|
|
fi
|
|
if command -v uv >/dev/null 2>&1; then
|
|
uv pip install --system -r requirements.txt pytest pytest-asyncio
|
|
else
|
|
pip install -r requirements.txt pytest pytest-asyncio
|
|
fi
|
|
# Relax durability on the throwaway CI Postgres so the per-test
|
|
# TRUNCATE's commit-fsync — the integration teardown's dominant cost
|
|
# (~1.5-2s/test, which collapsed the suite from ~13min to ~45s) — is
|
|
# skipped. fsync/full_page_writes are sighup GUCs and synchronous_commit
|
|
# is user-context, so ALTER SYSTEM + pg_reload_conf() applies them with
|
|
# NO restart. Ephemeral DB ⇒ fsync-off is safe. Non-fatal so a perms
|
|
# surprise can't red the job; fabledcurator is the postgres image's
|
|
# bootstrap superuser.
|
|
python -c "import os,psycopg; c=psycopg.connect(host=os.environ['DB_HOST'],port=5432,user=os.environ['DB_USER'],password=os.environ['DB_PASSWORD'],dbname=os.environ['DB_NAME'],autocommit=True); [c.execute(q) for q in ('ALTER SYSTEM SET fsync=off','ALTER SYSTEM SET synchronous_commit=off','ALTER SYSTEM SET full_page_writes=off','SELECT pg_reload_conf()')]; c.close()" || echo 'WARN: durability GUC relax failed (continuing)'
|
|
alembic upgrade head
|
|
pytest tests/ -v -m integration --durations=15
|
|
|
|
# Sign-or-fetch-from-cache: signs the extension via AMO if no ext-<version>
|
|
# Forgejo release exists yet, otherwise downloads the cached signed XPI.
|
|
# Result is uploaded as an Actions artifact for build-web to consume.
|
|
#
|
|
# Why this lives in build.yml (not a separate workflow): the image a push
|
|
# publishes MUST carry the XPI. A separate sign workflow racing build.yml
|
|
# leaves that image without one for ~5min (until the commit-back triggers
|
|
# another build). Inline ordering eliminates the race.
|
|
# Cache strategy: Forgejo Release Assets — picked 2026-05-25 over Generic
|
|
# Packages (cleaner API surface) and commit-back-to-side-branch (no extra
|
|
# branch to manage). AMO blocks re-signing the same version (returns 409),
|
|
# so signing is intentionally one-shot per version.
|
|
#
|
|
# BOTH branches sign (milestone 271 step 6, 2026-08-27). Not two signatures:
|
|
# the version is the commit TIME of the newest packaged-extension change, so
|
|
# dev and main derive the SAME number for the same extension source. A dev
|
|
# push that changes the extension signs it; the merge to main then finds the
|
|
# ext-<version> release already there, hits the cache, and bundles the
|
|
# byte-identical XPI into `:latest` with no second AMO call. One signature
|
|
# per extension CHANGE, shared by both channels — that is what makes two
|
|
# channels affordable, and it is why step 4 (derived version) had to land
|
|
# first. Ungating this while the version was still the hand-set 1.0.11 would
|
|
# have hit the existing ext-1.0.11 cache and bundled MAIN's stale XPI into
|
|
# `:dev` — a dev channel confidently serving old code.
|
|
#
|
|
# Unconditional since milestone 318 step 2: main and dev are now the only
|
|
# triggers, so the branch gate that used to exclude tag pushes matched
|
|
# everything. A condition that is always true reads as if some path avoids
|
|
# it, which is worse than no condition.
|
|
sign-extension:
|
|
# THE GATE (2026-09-23). Every lane above must have PASSED before this job
|
|
# exists at all — so a red suite does not produce an image, let alone push
|
|
# one. Nothing here is a new mechanism: it is the same `needs:` edge that
|
|
# has gated `promote` since milestone 362 step 4, and it carries that
|
|
# step's hardest-won property unchanged — **not running is not the same as
|
|
# passing.** `needs` treats a SKIPPED dependency as unsatisfied, so a lane
|
|
# that silently skips itself blocks the publish exactly as a failing one
|
|
# does. Run 5290 is why that is worth saying out loud: `smoke-web` skipped
|
|
# itself through a job-level `if:` that could not read `env`, and a design
|
|
# where only a FAILED gate blocks would have published unverified images
|
|
# while reporting success.
|
|
needs: [lint, extension-version, backend-lint-and-test, frontend-build, integration]
|
|
# A pull_request run is the lanes and nothing else. This is the ONLY thing
|
|
# separating "validate a Renovate bump" from "publish a Renovate bump", so
|
|
# it is stated on each publishing job rather than inferred from a `needs`
|
|
# chain that a later edit could quietly break.
|
|
if: github.event_name != 'pull_request'
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
# Not the triggering ref — see the `env:` block at the top. On a
|
|
# scheduled refresh this is `main`; on everything else it is the ref
|
|
# that fired, so this is a no-op on every ordinary path.
|
|
ref: ${{ env.BUILD_REF }}
|
|
# Full history is load-bearing, not a convenience: the version this
|
|
# job signs is derived from the commit TIME of the newest packaged
|
|
# extension change. A depth-1 clone sees one commit and derives a
|
|
# wrong, too-low value rather than failing (ci-requirements.md).
|
|
fetch-depth: 0
|
|
|
|
# BUILD_REF is what makes a scheduled run build `main` rather than the
|
|
# branch its cron fired from — and it is read through the `env` context
|
|
# inside `with:`, which this runner is NOT known to evaluate. If it does
|
|
# not, checkout silently falls back to the triggering ref and the weekly
|
|
# refresh publishes DEV's source to `:latest`, which is production.
|
|
# Every lane would stay green; the first sign of it would be production
|
|
# running code that was never merged.
|
|
#
|
|
# So assert the checkout instead of trusting the expression. A red
|
|
# weekly job is a fine outcome. Shipping dev to production is not.
|
|
#
|
|
# `if:` reads the `github` context, which the runner demonstrably does
|
|
# evaluate — this file already gates steps on it — so the guard cannot
|
|
# be disabled by the same uncertainty it exists to cover.
|
|
- name: Guard — a scheduled run must have checked out main
|
|
if: env.IS_REFRESH == 'true'
|
|
run: |
|
|
set -eu
|
|
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
|
echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
|
|
if [ "$BRANCH" != "main" ]; then
|
|
echo "schedule: expected main, got '$BRANCH'." >&2
|
|
echo "schedule: BUILD_REF was not honoured by the runner." >&2
|
|
echo "schedule: refusing to publish a channel tag from it." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# The version is DERIVED, not read from the repo (milestone 271 step 4,
|
|
# cut over 2026-08-27). `packaging.sh version` returns `YYYY.M.D.HHMM`
|
|
# UTC — the commit TIME of the newest change to a PACKAGED extension
|
|
# file, per family rule 148/149. Never a commit count, which orders by
|
|
# branch rather than by recency.
|
|
#
|
|
# Unpadded, and only here: AMO's grammar rejects a leading zero, so the
|
|
# extension renders rule 148's numbers without the family's padding
|
|
# (milestone 318 step 8). Same value, one character narrower per segment;
|
|
# the extension-version lane above checks the string against Mozilla's
|
|
# published regex before this job ever calls AMO.
|
|
#
|
|
# The committed "version" in manifest.json / package.json decides NOTHING
|
|
# — not even a MAJOR.MINOR prefix, which step 8 removed. The stamp step
|
|
# below overwrites it in the working tree before web-ext ever reads it,
|
|
# and it is deliberately NOT committed back: the commit carrying the bump
|
|
# would itself be a change to the extension and would move the version
|
|
# again. The repo holds the source; the build derives the label.
|
|
- name: Derive extension version
|
|
id: extver
|
|
run: |
|
|
set -eu
|
|
VERSION=$(sh extension/scripts/packaging.sh version)
|
|
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
|
echo "Derived extension version: $VERSION"
|
|
|
|
# Firefox refuses a downgrade and AMO never releases a burned version,
|
|
# so a version that moves BACKWARDS is unrecoverable: it strands every
|
|
# install that already took the higher one. Two ways it could happen —
|
|
# a checkout without full history (derives too low), or a rewritten
|
|
# history that drops the newest packaged commit.
|
|
#
|
|
# The test is `derived < highest already signed`, strictly. Equality is
|
|
# the ORDINARY case, not a fault: an unchanged extension derives the same
|
|
# version it did last build, which is exactly what lets the ext-<version>
|
|
# cache hit and holds AMO to one call per extension CHANGE. Only moving
|
|
# backwards is a failure, so this runs on every path — cache hit
|
|
# included — rather than only before a sign.
|
|
# --- shadow mode (milestone 313, step 2) -----------------------------
|
|
# Informational ONLY. Nothing reads this and it must never fail the
|
|
# build — no `set -e`, and every derivation falls back to UNAVAILABLE.
|
|
#
|
|
# What to watch across pushes, because this is what step 3 will trust:
|
|
# * a push touching only agent/ moves the agent and leaves web STILL.
|
|
# If web moves, its path set is too wide.
|
|
# * a push touching only docs moves nothing.
|
|
# * a push touching the extension moves the extension AND web, since
|
|
# web bakes in the XPI. If web does not move, its set is too narrow:
|
|
# the reuse check hits, and the channel serves a web image bundling
|
|
# the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
|
|
# * dev and main derive the same values for the same source.
|
|
- name: Shadow — derived artifact version (informational)
|
|
run: |
|
|
set -u
|
|
A=extension
|
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
|
echo "derived: artifact=$A version=$V revision=$R sha=$GITHUB_SHA"
|
|
|
|
- name: Guard — the derived version must never go backwards
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
DERIVED: ${{ steps.extver.outputs.version }}
|
|
run: |
|
|
python3 - <<'PY'
|
|
import json, os, sys, urllib.request
|
|
|
|
API = ("https://git.fabledsword.com/api/v1/repos/"
|
|
"bvandeusen/FabledCurator/releases")
|
|
headers = {"Authorization": "token " + os.environ["TOKEN"]}
|
|
|
|
# Paginated rather than first-page-only: ext-* releases share this
|
|
# list with the v* release tags, so one page would start missing them
|
|
# as those accumulate. The bound FAILS rather than silently scanning
|
|
# part of the list and calling the highest it saw the highest there is.
|
|
tags = []
|
|
for page in range(1, 21):
|
|
req = urllib.request.Request(
|
|
f"{API}?limit=50&page={page}", headers=headers)
|
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
|
batch = json.load(resp)
|
|
if not batch:
|
|
break
|
|
tags += [r.get("tag_name", "") for r in batch]
|
|
else:
|
|
sys.exit("guard: >1000 releases — pagination bound reached")
|
|
|
|
def parse(v):
|
|
try:
|
|
return tuple(int(part) for part in v.split("."))
|
|
except ValueError:
|
|
return None
|
|
|
|
derived_s = os.environ["DERIVED"]
|
|
derived = parse(derived_s)
|
|
if derived is None:
|
|
sys.exit(f"guard: derived version {derived_s!r} is not numeric")
|
|
|
|
signed = sorted(
|
|
(v, t) for t in tags if t.startswith("ext-")
|
|
for v in [parse(t[4:])] if v
|
|
)
|
|
if not signed:
|
|
print("guard: no ext-* release yet — nothing to go backwards from")
|
|
raise SystemExit(0)
|
|
|
|
hi, hi_tag = signed[-1]
|
|
print(f"guard: derived={derived_s} highest already signed={hi_tag}")
|
|
if derived < hi:
|
|
sys.exit(
|
|
f"REFUSING TO SIGN: derived {derived_s} is OLDER than the "
|
|
f"already-signed {hi_tag}. Firefox would reject it as a "
|
|
f"downgrade, and AMO will not release the burned version. "
|
|
f"First thing to check: did this job check out with "
|
|
f"fetch-depth: 0?"
|
|
)
|
|
print("guard: ok")
|
|
PY
|
|
|
|
- name: Check Forgejo release-asset cache
|
|
id: cache
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
run: |
|
|
set -eu
|
|
VERSION=${{ steps.extver.outputs.version }}
|
|
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
|
-H "Authorization: token $TOKEN" \
|
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
|
echo "Tag lookup HTTP status: $STATUS"
|
|
# JSON parsing via python (ci-python:3.14 has stdlib json; jq is
|
|
# not in the image and adding it per ci-requirements.md is not
|
|
# warranted for a single consumer — operator-flagged 2026-05-26
|
|
# after a sign job failed with `jq: not found`).
|
|
if [ "$STATUS" = "200" ]; then
|
|
ASSET_ID=$(python3 -c "import json; r=json.load(open('release.json')); xpis=[a for a in r.get('assets', []) if a.get('name','').endswith('.xpi')]; print(xpis[0]['id'] if xpis else '')")
|
|
if [ -n "$ASSET_ID" ]; then
|
|
echo "cached=true" >> "$GITHUB_OUTPUT"
|
|
echo "asset_id=$ASSET_ID" >> "$GITHUB_OUTPUT"
|
|
echo "Cached XPI exists at ext-$VERSION (asset id $ASSET_ID); skipping AMO sign"
|
|
else
|
|
echo "cached=false" >> "$GITHUB_OUTPUT"
|
|
echo "Release ext-$VERSION exists but has no .xpi asset; will re-sign + re-upload"
|
|
fi
|
|
else
|
|
echo "cached=false" >> "$GITHUB_OUTPUT"
|
|
echo "No release named ext-$VERSION; will sign via AMO and upload"
|
|
fi
|
|
|
|
# No "download cached XPI in sign-extension" step: build-web
|
|
# fetches directly from the Forgejo ext-<version> release asset
|
|
# (removed 2026-05-26 alongside the actions/upload-artifact
|
|
# removal — sign-extension's job is just to ensure the cache
|
|
# exists on Forgejo; the build-web side reads it independently).
|
|
|
|
# web-ext signs whatever manifest.json says, so the derived value has to
|
|
# reach the tree before signing. package.json is written too: the two are
|
|
# required to agree (the extension-version lane's guard), and a local
|
|
# `npm run build` reads
|
|
# it. Working tree only — never committed, per the note on the derive
|
|
# step.
|
|
- name: Stamp the derived version into manifest.json + package.json
|
|
env:
|
|
DERIVED: ${{ steps.extver.outputs.version }}
|
|
run: |
|
|
python3 - <<'PY'
|
|
import json, os
|
|
|
|
version = os.environ["DERIVED"]
|
|
for path in ("extension/manifest.json", "extension/package.json"):
|
|
with open(path) as fh:
|
|
doc = json.load(fh)
|
|
doc["version"] = version
|
|
with open(path, "w") as fh:
|
|
json.dump(doc, fh, indent=2)
|
|
fh.write("\n")
|
|
print(f"{path}: version -> {version}")
|
|
PY
|
|
|
|
- name: Sign via AMO (cache miss)
|
|
if: steps.cache.outputs.cached != 'true'
|
|
run: |
|
|
cd extension && npm install --no-save --no-audit --no-fund && npm run sign
|
|
env:
|
|
WEB_EXT_API_KEY: ${{ secrets.MOZILLA_AMO_JWT_KEY }}
|
|
WEB_EXT_API_SECRET: ${{ secrets.MOZILLA_AMO_JWT_SECRET }}
|
|
|
|
- name: Upload signed XPI to ext-<version> release (cache miss)
|
|
if: steps.cache.outputs.cached != 'true'
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
run: |
|
|
set -eux
|
|
VERSION=${{ steps.extver.outputs.version }}
|
|
# AMO renames signed XPIs with its internal addon-id-safe-string;
|
|
# canonicalize to fabledcurator-<version>.xpi so the FC server's
|
|
# whitelist (backend/app/frontend.py expects 'fabledcurator-*.xpi')
|
|
# keeps working.
|
|
SIGNED=$(ls extension/web-ext-artifacts/*.xpi | head -1)
|
|
XPI="extension/web-ext-artifacts/fabledcurator-$VERSION.xpi"
|
|
cp "$SIGNED" "$XPI"
|
|
# Find-or-create the ext-<version> release. Track whether WE
|
|
# created it so an upload failure below can roll back (don't
|
|
# leave an empty release tombstone that the next run's
|
|
# cache-check mistakes for a partial-failure state).
|
|
#
|
|
# target_commitish is the signing commit, not a branch name: since
|
|
# step 6 either branch can create this release, and hard-coding
|
|
# `main` would tag a dev-signed XPI against a main commit that may
|
|
# not even contain the extension source it was built from.
|
|
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
|
-H "Authorization: token $TOKEN" \
|
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
|
if [ "$STATUS" = "200" ]; then
|
|
CREATED_BY_US=false
|
|
else
|
|
curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
|
|
-d "{\"tag_name\":\"ext-$VERSION\",\"name\":\"Extension $VERSION (signed XPI cache)\",\"body\":\"Internal cache for the signed XPI consumed by build.yml's build-web job. Not a user-facing FC release.\",\"target_commitish\":\"$GITHUB_SHA\"}" \
|
|
-o release.json \
|
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases"
|
|
CREATED_BY_US=true
|
|
fi
|
|
RELEASE_ID=$(python3 -c "import json; print(json.load(open('release.json'))['id'])")
|
|
test -n "$RELEASE_ID"
|
|
# Rollback-on-failure: if the asset upload fails AND we just
|
|
# created the release in this run, delete it. Prevents an empty
|
|
# ext-<version> release from poisoning the next workflow run
|
|
# (operator-flagged 2026-05-26 — without rollback the next run
|
|
# saw 'release exists, no asset → cache miss → sign' which AMO
|
|
# then rejected with 409 'Version already exists').
|
|
rollback_if_we_created() {
|
|
if [ "$CREATED_BY_US" = "true" ]; then
|
|
echo "Rolling back: deleting just-created release $RELEASE_ID"
|
|
curl -s -X DELETE -H "Authorization: token $TOKEN" \
|
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/$RELEASE_ID" || true
|
|
curl -s -X DELETE -H "Authorization: token $TOKEN" \
|
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/tags/ext-$VERSION" || true
|
|
fi
|
|
}
|
|
trap 'rollback_if_we_created' EXIT
|
|
HTTP_CODE=$(curl -s -X POST -H "Authorization: token $TOKEN" \
|
|
-F "attachment=@$XPI" \
|
|
-o /dev/null -w "%{http_code}" \
|
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/$RELEASE_ID/assets?name=fabledcurator-$VERSION.xpi")
|
|
if [ "$HTTP_CODE" != "201" ] && [ "$HTTP_CODE" != "200" ]; then
|
|
echo "Asset upload failed with HTTP $HTTP_CODE"
|
|
exit 1
|
|
fi
|
|
# Upload succeeded — clear the rollback trap.
|
|
trap - EXIT
|
|
echo "Uploaded fabledcurator-$VERSION.xpi to ext-$VERSION release"
|
|
|
|
# No actions/upload-artifact step: build-web reads the signed XPI
|
|
# straight from the ext-<version> Forgejo release we just uploaded to.
|
|
# Same source of truth; no double-store. The step was dropped 2026-05-26
|
|
# because act_runner could not run upload-artifact@v4+; gitea/runner 3.x
|
|
# can (Scribe snippet #2271), but the release asset stays the better
|
|
# channel for a file build-web needs on every run.
|
|
|
|
build-web:
|
|
# Consumed by smoke-web's job-level `if:`. It cannot read `env` — the env
|
|
# context is available to STEP `if:` and step bodies, never to a job's own
|
|
# condition, and an unresolvable context there is empty rather than an
|
|
# error. `smoke-web` skipped silently on run 5290 for exactly that reason.
|
|
#
|
|
# Keying off the reuse step's own output is better than re-deriving the
|
|
# trigger anyway: it is the same single decision the build, the XPI
|
|
# download and the promote all take (build.yml's "one decision drives
|
|
# everything downstream"), and it says the thing smoke-web actually needs
|
|
# to know — a candidate was published — rather than restating why.
|
|
outputs:
|
|
candidate: ${{ steps.reuse.outputs.promote }}
|
|
# The manifest THIS run pushed, empty on a reuse hit. smoke-web addresses
|
|
# it by digest rather than by tag: a tag can move between the build and
|
|
# the smoke, and then the check reports on bytes nobody built here.
|
|
digest: ${{ steps.build.outputs.digest }}
|
|
# What the channel tag already names. This is what smoke-web checks on a
|
|
# reuse hit — deliberately NOT folded into `digest`, which the :c-<sha>
|
|
# repoint reads and which must keep meaning "what this run built" (#4290).
|
|
published_digest: ${{ steps.reuse.outputs.published_digest }}
|
|
# A plain `needs` — no `always()`. That expression existed to let a
|
|
# SKIPPED sign-extension through on a tag push while still blocking a
|
|
# FAILED one. With no tag trigger, sign-extension always runs, so the
|
|
# default behaviour is exactly what we want: a failed sign skips build-web
|
|
# rather than shipping an image without its XPI.
|
|
needs: [sign-extension, lint, extension-version, backend-lint-and-test,
|
|
frontend-build, integration]
|
|
# The lanes in that list are THE GATE (2026-09-23) — see sign-extension's
|
|
# copy of this comment for why, including the property that a SKIPPED lane
|
|
# blocks as firmly as a failing one. They are repeated here rather than
|
|
# inherited through `sign-extension`: this job is what pushes the channel
|
|
# tag, and the one place the gate must be legible is the place that
|
|
# publishes.
|
|
if: github.event_name != 'pull_request'
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
# Not the triggering ref — see the `env:` block at the top. On a
|
|
# scheduled refresh this is `main`; on everything else it is the ref
|
|
# that fired, so this is a no-op on every ordinary path.
|
|
ref: ${{ env.BUILD_REF }}
|
|
# Full history: this job RE-DERIVES the extension version rather than
|
|
# being handed it, and a depth-1 clone derives a wrong, too-low value
|
|
# rather than failing — which would 404 the download of a release
|
|
# that exists perfectly well under its real name.
|
|
fetch-depth: 0
|
|
|
|
# See sign-extension's copy for why this guard exists.
|
|
- name: Guard — a scheduled run must have checked out main
|
|
if: env.IS_REFRESH == 'true'
|
|
run: |
|
|
set -eu
|
|
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
|
echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
|
|
if [ "$BRANCH" != "main" ]; then
|
|
echo "schedule: expected main, got '$BRANCH'." >&2
|
|
echo "schedule: BUILD_REF was not honoured by the runner." >&2
|
|
echo "schedule: refusing to publish a channel tag from it." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# --- derived values, one line (milestone 313) ------------------------
|
|
# These stopped being shadow output at step 3. `revision` decides
|
|
# whether the build below runs at all and `version` is what the image
|
|
# reports about itself; the load-bearing steps each print only the one
|
|
# they use, so this is the only place the pair appears together. When a
|
|
# build is skipped, this is the line that says what the commit derived.
|
|
#
|
|
# Still diagnostic, so it still must not fail the build — no `set -e`,
|
|
# and every derivation falls back to UNAVAILABLE. A broken echo must
|
|
# never be the reason an image does not ship.
|
|
#
|
|
# What it should say:
|
|
# * a push touching only agent/ moves the agent and leaves web STILL.
|
|
# If web moves, its path set is too wide.
|
|
# * a push touching only docs moves nothing.
|
|
# * a push touching the extension moves the extension AND web, since
|
|
# web bakes in the XPI. If web does not move, its set is too narrow:
|
|
# the reuse check hits, and the channel serves a web image bundling
|
|
# the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
|
|
# * dev and main derive the same values for the same source.
|
|
- name: Report the derived artifact version
|
|
env:
|
|
# Diagnostic for the trigger normalisation. `refresh` is reported RAW
|
|
# as well as normalised, because the two disagreeing is the whole
|
|
# failure mode: a dispatch input whose type does not compare the way
|
|
# the expression assumes evaluates to false silently, and the only
|
|
# symptom is a refresh that quietly behaves like an ordinary push.
|
|
RAW_REFRESH: ${{ github.event.inputs.refresh }}
|
|
RAW_FORCE: ${{ github.event.inputs.force_build }}
|
|
run: |
|
|
set -u
|
|
echo "trigger: event=$GITHUB_EVENT_NAME IS_REFRESH='${IS_REFRESH:-<unset>}' BUILD_REF='${BUILD_REF:-<unset>}'"
|
|
echo "trigger: raw inputs refresh='${RAW_REFRESH:-<unset>}' force_build='${RAW_FORCE:-<unset>}'"
|
|
A=web
|
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
|
echo "derived: artifact=$A version=$V revision=$R sha=$GITHUB_SHA"
|
|
|
|
- name: Determine tag
|
|
id: tag
|
|
run: |
|
|
# Two trigger shapes, and between them they publish three tags:
|
|
# main → :latest (production, moving — rule 147: main IS production)
|
|
# :c-<sha> (immutable, the rollback unit — rule 145)
|
|
# dev → :dev (the rolling test channel — rule 146)
|
|
#
|
|
# That is the whole list. No :<version>, and no :main — rule 145,
|
|
# narrowed 2026-08-28 once it was verified that nothing pins:
|
|
# "a third name for the same thing is upkeep for a model we do not
|
|
# run." The date tag published between milestone 313 step 3 and
|
|
# milestone 318 was exactly that; :main was a second moving name for
|
|
# whatever :latest already pointed at.
|
|
#
|
|
# `dev` gets no :c-<sha> deliberately. On a channel whose entire
|
|
# contract is that it moves, a per-push immutable tag is a rollback
|
|
# target nobody has ever pulled, accumulating forever. The accepted
|
|
# cost: on dev there is no rollback but the previous :dev, which is
|
|
# gone — recovery is revert-on-git plus a CI cycle.
|
|
#
|
|
# Reinstating :<version> is a real decision, not a default. It earns
|
|
# its place when something genuinely pins: a second instance held on
|
|
# a known-good build, or a deliberately frozen window. Tag at the
|
|
# moment you decide to freeze; no back-catalogue is needed.
|
|
#
|
|
# POSIX-safe substring (the runner shell is dash/BusyBox sh, not
|
|
# bash — `${var:0:7}` errors with "Bad substitution"; cut works
|
|
# everywhere). Operator-flagged 2026-06-01 after the first :c-<sha>
|
|
# main-push build failed at this step.
|
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
|
|
|
# A scheduled refresh publishes the CHANNEL and nothing else
|
|
# (#3154). :c-<sha> for main's HEAD already exists and names the
|
|
# bytes that commit actually built; re-pushing it over refreshed
|
|
# base layers would break the one tag rule 145 makes immutable —
|
|
# and it is the rollback unit, so the breakage would surface on the
|
|
# day somebody needed it.
|
|
#
|
|
# The accepted consequence: between a refresh and the next main
|
|
# push, :latest and :c-<sha> point at different manifests. That is
|
|
# the design, not drift. They RE-CONVERGE on that push — it hits
|
|
# reuse (a refresh does not move fc.revision, because it does not
|
|
# touch the source), and the repoint step then writes the new
|
|
# :c-<sha> from the refreshed :latest. So the rollback unit ends up
|
|
# naming the bytes production is actually running, which is the
|
|
# property that matters.
|
|
#
|
|
# Checked BEFORE the ref test, not after: a scheduled run's
|
|
# GITHUB_REF is the default branch (dev), so the main test would
|
|
# never fire on it.
|
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest" >> "$GITHUB_OUTPUT"
|
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:latest,git.fabledsword.com/bvandeusen/fabledcurator:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator:dev" >> "$GITHUB_OUTPUT"
|
|
echo "channel=dev" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
|
|
# A shell step, not docker/login-action@v3, because the action's shared
|
|
# cache races itself (#3118). act_runner caches a remote action under one
|
|
# /root/.cache/act/<hash> per runner, and build-web, build-ml and
|
|
# build-agent all start in the same second and all want this same action.
|
|
# One job re-clones the directory — which empties and repopulates it —
|
|
# while another is walking it to copy into its container, and the walker
|
|
# lstat()s a file that has just vanished. It failed twice on 2026-08-27,
|
|
# naming a DIFFERENT missing file each time (`eslint.config.mjs`, then
|
|
# `jest.config.ts`), which is what rules out a corrupt cache and points at
|
|
# a race. The loser dies with MODULE_NOT_FOUND on dist/index.js before the
|
|
# action runs at all, so the secret is never even reached.
|
|
#
|
|
# Nothing is lost by dropping it: logging in is one command, the docker
|
|
# CLI is already in the CI image (ci-requirements.md), and the same
|
|
# reasoning as family rule 5 applies — a marketplace action buys nothing
|
|
# when the tool is baked into the image the workflow already selected.
|
|
#
|
|
# Password on stdin, never as an argument: an argument lands in the
|
|
# process table and draws docker's own deprecation warning.
|
|
- name: Login to Forgejo registry
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
ACTOR: ${{ github.actor }}
|
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
|
|
|
# A REAL buildx builder, not the default `docker` driver (#3114, #3190).
|
|
#
|
|
# The default driver builds through the local dockerd. It cannot export a
|
|
# registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
|
|
# + torch image from scratch whenever the runner's local cache is cold,
|
|
# measured at 9m26s against 7s warm. It is also #3190's leading suspect:
|
|
# after a registry-direct push it resolves image metadata against a local
|
|
# store the push never filled, and reports `No such image` on an image
|
|
# that published perfectly well three seconds earlier.
|
|
#
|
|
# These jobs run INSIDE a container against a mounted docker socket, so
|
|
# the buildkit container this starts is a SIBLING of the job container,
|
|
# not a child. That works over the socket mount; it had never been tried
|
|
# here before milestone 326 step 1.
|
|
- name: Set up buildx
|
|
uses: docker/setup-buildx-action@v3
|
|
|
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
|
# Does the image the channel tag already points at carry THIS commit's
|
|
# revision? If so the bytes this job would produce are already published
|
|
# and the build is pure waste: the remaining tags get repointed at that
|
|
# existing manifest instead, registry-side, in seconds.
|
|
#
|
|
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
|
|
# (milestone 318 step 3). A tag would be a name minted per build that one
|
|
# thing reads — what rule 145 narrowed against — and would be prunable
|
|
# under the registry's keep_pattern (#3157), silently expiring the cache.
|
|
# A label rides inside a tag that has to exist anyway.
|
|
#
|
|
# An image with no such label reads as a miss and rebuilds. That is the
|
|
# migration, not a fault: labels cannot be backfilled, since the reuse
|
|
# path copies a manifest and config labels are not manifest annotations.
|
|
# Each artifact pays one rebuild, once.
|
|
#
|
|
# This is what stops a push that touched only `agent/` from rebuilding
|
|
# web, and a merge to main from rebuilding what dev already built.
|
|
#
|
|
# The failure direction is deliberate. An inspect that errors for ANY
|
|
# reason — network, auth, a registry hiccup — reads as a miss and the
|
|
# build runs. Only a genuine 200 skips one, so there is no path here
|
|
# that skips a build that was actually needed; the worst case is paying
|
|
# for a build we could have avoided.
|
|
#
|
|
# BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops
|
|
# picking up base-image updates. Milestone 318 removed the argument this
|
|
# used to need rather than answering it — with no version tags there is
|
|
# no immutable name a refresh could contradict, and rule 145 already
|
|
# allows a rebuild with different contents to republish a MOVING tag.
|
|
# So a refresh is just a build. A scheduled channel-only one is tracked
|
|
# separately (#3154); it does not belong in the push path.
|
|
- name: Is this content already published?
|
|
id: reuse
|
|
env:
|
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
|
# Empty on a push; the string "true" only from a workflow_dispatch
|
|
# that asked for it. `github.event.inputs` rather than the `inputs`
|
|
# context — release.yml already uses that form, and it is the one
|
|
# this runner is known to evaluate. Read through env rather than
|
|
# interpolated into the run block, same rule as release.yml's TAG.
|
|
FORCE: ${{ github.event.inputs.force_build }}
|
|
# A scheduled refresh has to bypass reuse by construction: it
|
|
# rebuilds the SAME source, so fc.revision always matches and the
|
|
# check would skip every refresh there has ever been.
|
|
run: |
|
|
set -eu
|
|
DERIVED=$(sh scripts/artifacts.sh revision web)
|
|
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
|
# Baked into the web image as FC_VERSION and reported by /api/health.
|
|
# A pure function of the revision — same commit, same string — so it
|
|
# adds no variability the reuse check would have to account for.
|
|
echo "version=$(sh scripts/artifacts.sh version web)" >> "$GITHUB_OUTPUT"
|
|
|
|
# The build clock, pinned to the same commit (#3265). Without it
|
|
# buildkit stamps the image config with the wall clock of the build,
|
|
# so identical layers republish under a new config blob and the
|
|
# channel tag gets a new manifest digest for no reason. Derived from
|
|
# `newest()` like revision and version, so all three name one commit
|
|
# and cannot drift apart.
|
|
echo "epoch=$(sh scripts/artifacts.sh epoch web)" >> "$GITHUB_OUTPUT"
|
|
|
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
|
# that is why the revision needs no -main/-dev qualifier any more.
|
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
|
|
|
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
|
# whether the channel then has to be written separately.
|
|
#
|
|
# On a push the build writes the channel tag directly: the bytes came
|
|
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
|
# it behind.
|
|
#
|
|
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
|
# refresh rebuilds against freshly resolved base images, and the web
|
|
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
|
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
|
# No verification LANE can see that: they run on ci-python:3.14 and
|
|
# install requirements.txt, and a base bump changes neither. So
|
|
# refreshed bytes have to be proven before :latest names them, and
|
|
# proving needs a moment between "built" and "published" to occupy.
|
|
# This is that moment; :latest goes on naming the build that works
|
|
# until something says otherwise.
|
|
#
|
|
# `:refresh-candidate` is one moving ref per image, overwritten in
|
|
# place, holding a build nobody is told to pull — the shape rule 145
|
|
# already allows for :buildcache, not the per-build tag family that
|
|
# milestone 318 withdrew.
|
|
#
|
|
# Decided HERE, beside `hit`, for the reason the force/schedule
|
|
# branch below gives: one step decides what this job does. A
|
|
# condition derived independently could disagree with the tag the
|
|
# build actually wrote.
|
|
#
|
|
# build-web additionally exposes this as `outputs.candidate`, which is
|
|
# what gates the `promote` job — a job's `if:` cannot read `env`, and
|
|
# one flag is enough because both jobs derive it from the same
|
|
# IS_REFRESH. build-agent does not re-emit it; a second copy nothing
|
|
# reads is the kind of thing that later reads as load-bearing.
|
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
|
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
|
echo "promote=true" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
|
echo "promote=false" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
|
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
|
# branching on the exit code would read "no label yet" as success.
|
|
# An unreachable tag also lands here as empty via the `|| echo`.
|
|
# Empty never equals a 12-char revision, so every uncertain case
|
|
# falls through to a build — the safe direction, with no special
|
|
# casing for it.
|
|
#
|
|
# Read the SPECIFIC key. The map also carries whatever the base image
|
|
# set, and `org.opencontainers.image.version` sits right beside ours
|
|
# looking like a plausible answer (it reads 24.04 on the agent).
|
|
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
|
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
|
2>/dev/null || echo "")
|
|
|
|
# The manifest the channel tag names RIGHT NOW. Exported so the
|
|
# smoke has something to check on a reuse hit, when this job builds
|
|
# nothing and emits no digest of its own (#4323). Resolved here
|
|
# because this step is already resolving the tag to read its label —
|
|
# one lookup, one answer, rather than a second one that could name a
|
|
# different manifest if anything moved the tag in between.
|
|
PUBLISHED_DIGEST=$(docker buildx imagetools inspect "$IMAGE:$T" \
|
|
--format '{{ .Manifest.Digest }}' 2>/dev/null || echo "")
|
|
echo "published_digest=$PUBLISHED_DIGEST" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
|
# The tag resolves but carries no readable label. Expected exactly
|
|
# once per artifact, during the migration onto labels. If it recurs
|
|
# every push, something is rewriting the channel tag as a manifest
|
|
# index — see the repoint step's note.
|
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
|
fi
|
|
|
|
# FORCE is checked here rather than in the build step's `if:`, so
|
|
# that one decision drives everything downstream. The repoint step
|
|
# keys off `hit` too, and a force that bypassed only the build would
|
|
# leave the two disagreeing about what just happened.
|
|
if [ "${FORCE:-false}" = "true" ]; then
|
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: force_build set — building regardless"
|
|
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: scheduled base refresh — building regardless"
|
|
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: already published — skipping the build"
|
|
else
|
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: not published — building"
|
|
fi
|
|
|
|
- name: Download signed XPI from Forgejo release asset
|
|
# dev and main each bundle the XPI their own sign-extension just
|
|
# published — the point of the channel work (milestone 271 step 6): the
|
|
# dev image carries the extension being developed, rather than
|
|
# requiring a merge to try it.
|
|
#
|
|
# The 10-minute polling loop that used to live here is gone with the
|
|
# tag trigger (milestone 318 step 2). It existed for one shape only: a
|
|
# release cut fired the tag build and the main build together, the tag
|
|
# build skipped sign-extension and raced straight here, and it lost
|
|
# every time (operator-flagged 2026-05-27 after v26.05.27.0). Polling
|
|
# was the fix for a build that should not have been running.
|
|
#
|
|
# sign-extension is a `needs` dependency and it succeeded, so the
|
|
# release exists. A single fetch is correct, and a 404 now means a real
|
|
# disagreement about the derived version rather than a race — which is
|
|
# exactly what should fail loudly instead of being slept through.
|
|
#
|
|
# Still gated on the reuse miss: a published image already contains its
|
|
# XPI, so this would fetch a file nothing then reads.
|
|
if: steps.reuse.outputs.hit != 'true'
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
run: |
|
|
set -eux
|
|
# Re-derived, not read from the repo: sign-extension published
|
|
# ext-<derived>, and the committed version has been inert since
|
|
# milestone 271 step 4. Both jobs run `packaging.sh version` over the
|
|
# same commit, so they agree by construction — and if they ever
|
|
# didn't, this download 404s and the build fails loudly instead of
|
|
# shipping a stale XPI.
|
|
VERSION=$(sh extension/scripts/packaging.sh version)
|
|
# One fetch, no retry. sign-extension ran to success in this same
|
|
# workflow and published ext-$VERSION; both jobs derive $VERSION from
|
|
# the same commit, so they agree by construction. A 404 here means
|
|
# they did NOT agree, and sleeping on that would only delay the
|
|
# report.
|
|
STATUS=$(curl -s -o release.json -w "%{http_code}" \
|
|
-H "Authorization: token $TOKEN" \
|
|
"https://git.fabledsword.com/api/v1/repos/bvandeusen/FabledCurator/releases/tags/ext-$VERSION" || echo 000)
|
|
if [ "$STATUS" != "200" ]; then
|
|
echo "ERROR: ext-$VERSION release not found (HTTP $STATUS)."
|
|
echo "sign-extension succeeded in this run, so it published some"
|
|
echo "other version — the two jobs derived different values for one"
|
|
echo "commit. Check that both checked out with fetch-depth: 0."
|
|
exit 1
|
|
fi
|
|
# Extract the .xpi asset's browser_download_url (Forgejo's
|
|
# /releases/assets/<id> endpoint returns ASSET METADATA, not
|
|
# the binary blob — operator-flagged 2026-05-26: my prior
|
|
# code curl'd the metadata endpoint without -f and wrote the
|
|
# resulting 404-page-not-found text into fabledcurator-*.xpi,
|
|
# which Firefox then rejected as "corrupt").
|
|
# browser_download_url is the canonical binary endpoint and
|
|
# is also publicly accessible (no token needed) but we pass
|
|
# the token anyway for symmetry with private-repo support.
|
|
DOWNLOAD_URL=$(python3 -c "import json; r=json.load(open('release.json')); xpis=[a for a in r.get('assets', []) if a.get('name','').endswith('.xpi')]; print(xpis[0]['browser_download_url'])")
|
|
test -n "$DOWNLOAD_URL"
|
|
echo "Downloading XPI from: $DOWNLOAD_URL"
|
|
mkdir -p frontend/public/extension
|
|
DEST="frontend/public/extension/fabledcurator-$VERSION.xpi"
|
|
# -f = fail on HTTP error (prevents silent corruption like the
|
|
# 2026-05-26 incident); -L = follow redirects.
|
|
curl -sfL -H "Authorization: token $TOKEN" -o "$DEST" "$DOWNLOAD_URL"
|
|
# Sanity check: the binary should start with the ZIP magic (PK\x03\x04).
|
|
# If it's anything else, the next docker build will ship a corrupt XPI.
|
|
MAGIC=$(head -c 2 "$DEST" | od -An -c | tr -d ' \n')
|
|
if [ "$MAGIC" != "PK" ]; then
|
|
echo "ERROR: downloaded XPI does not start with ZIP magic 'PK' (got '$MAGIC')"
|
|
echo "File contents preview:"
|
|
head -c 200 "$DEST"
|
|
exit 1
|
|
fi
|
|
cp "$DEST" "frontend/public/extension/fabledcurator-latest.xpi"
|
|
ls -la frontend/public/extension/
|
|
|
|
- name: Build and push web image
|
|
# `id:` so the repoint step below can read `outputs.digest` — the
|
|
# manifest THIS run published, as opposed to whatever the channel tag
|
|
# happens to name by the time that step runs (#4290).
|
|
id: build
|
|
if: steps.reuse.outputs.hit != 'true'
|
|
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
|
# it normalises the image config's `created` field and the history
|
|
# timestamps rather than being consumed by the Dockerfile. See #3265
|
|
# and the reuse step's `epoch` output.
|
|
env:
|
|
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
|
uses: docker/build-push-action@v5
|
|
with:
|
|
context: .
|
|
file: Dockerfile
|
|
push: true
|
|
# Re-resolve the FROM references against the registry instead of
|
|
# trusting whatever digest the cache was built against. This is the
|
|
# whole mechanism of the scheduled refresh (#3154): if the base tag
|
|
# moved, the FROM layer's cache key changes, every layer above it
|
|
# invalidates, and the image genuinely rebuilds.
|
|
#
|
|
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
|
# did NOT move, the build was ~13s with every content step CACHED —
|
|
# and the channel tag STILL got a new manifest digest, because
|
|
# buildkit stamps a fresh image config per run and republishes the
|
|
# identical layers under it. All three images moved that way on
|
|
# 2026-08-30 with nothing whatsoever changed in them.
|
|
#
|
|
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
|
# content came from, the config is byte-identical across runs, so
|
|
# the manifest digest is too and the push is a registry no-op. A
|
|
# digest change means the content changed again, which is the only
|
|
# thing a digest is any use for.
|
|
#
|
|
# What `pull` does NOT catch either: a Debian package update inside
|
|
# the `apt-get install` layer while the base tag itself stands
|
|
# still. The official python/cuda images rebuild with those updates
|
|
# baked in, so this is a lag rather than a hole; closing it needs
|
|
# `no-cache: true`, which is a much larger version of the same
|
|
# churn #3265 is about.
|
|
#
|
|
# Only on the schedule. An ordinary push wants the cached base.
|
|
pull: ${{ env.IS_REFRESH == 'true' }}
|
|
# ONE tag, the channel's. Every other tag is written by the step
|
|
# below, registry-side. buildx here pushes the first tag to the
|
|
# registry and then re-pushes the rest through the DOCKER driver,
|
|
# out of a local image store a registry-direct build never filled —
|
|
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
|
# published perfectly well.
|
|
tags: ${{ steps.reuse.outputs.build_ref }}
|
|
# The reuse key. Read back off the channel tag on the next push to
|
|
# decide whether that push needs to build at all, so this is not
|
|
# decoration — an unstamped image is one that will always rebuild.
|
|
labels: |
|
|
fc.revision=${{ steps.reuse.outputs.revision }}
|
|
# LOAD-BEARING, not a preference. On the default docker driver these
|
|
# were no-ops; on the docker-container driver above,
|
|
# build-push-action@v5 defaults provenance to TRUE when pushing.
|
|
# Provenance attaches an attestation manifest, which makes the pushed
|
|
# tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
|
|
# through an index.
|
|
#
|
|
# The label directly above IS the reuse key. Wrap the channel tag in
|
|
# an index and the next push reads fc.revision=<none>, misses, and
|
|
# rebuilds. Then so does the one after that, forever. Nothing fails,
|
|
# nothing goes red, and the only symptom is the bill. That is #3183
|
|
# arriving through a different door, and note #3127 §4 records the
|
|
# same shape for `platforms:`.
|
|
provenance: false
|
|
sbom: false
|
|
# The ONLY cache this driver can have. `docker-container` gets a
|
|
# FRESH buildkit instance per job, so unlike the default docker
|
|
# driver it has no local layer store to fall back on — measured on
|
|
# run 4896, the first builds after the driver change: web 3m44s
|
|
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
|
|
# driver change ALONE is a regression; this is the other half of it.
|
|
#
|
|
# mode=max so intermediate stages cache too. web's frontend-builder
|
|
# stage and the agent's two ~150s pip layers are the whole cost, and
|
|
# they are exactly what a min-mode cache would drop.
|
|
#
|
|
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
|
|
# Rule 145 narrowed against names NOTHING reads; this one is read by
|
|
# every build that runs, is one moving ref per image rather than one
|
|
# per build, holds cache blobs rather than a shippable artifact, and
|
|
# is overwritten in place rather than accumulating. It is closer to
|
|
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
|
|
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache
|
|
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator:buildcache,mode=max
|
|
# The agent carries its own copy of these (plus FC_REVISION) since
|
|
# 2026-09-24. This used to say the agent "has nothing to tell" — true
|
|
# of the ml image, which no longer exists (#4311), and never true of
|
|
# the agent, which has a control page and a /status endpoint and was
|
|
# reporting a hand-written literal on both.
|
|
build-args: |
|
|
FC_CHANNEL=${{ steps.tag.outputs.channel }}
|
|
FC_VERSION=${{ steps.reuse.outputs.version }}
|
|
|
|
# Every tag but the channel's own is written HERE, registry-side,
|
|
# whether or not a build ran. Each -t becomes another reference to the
|
|
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
|
# what is published rather than a lookalike rebuild.
|
|
#
|
|
# Owning the build path too is #3190's fix, not a tidy-up:
|
|
#
|
|
# #27 pushing …/fabledcurator:latest DONE 15.8s
|
|
# #28 pushing …/fabledcurator:c-0e15c44 with docker
|
|
# #28 ERROR: tag does not exist: …:c-0e15c44
|
|
#
|
|
# Intermittent — build-ml made the identical two-tag push seconds later
|
|
# and succeeded — and worse than it looks. `:latest` had already
|
|
# published, so production was correct while the immutable rollback tag
|
|
# rule 145 requires of every main push simply did not exist. Nothing but
|
|
# the red job would ever have noticed: a missing :c-<sha> has no
|
|
# consumer that fails, so it surfaces when somebody needs to roll back.
|
|
#
|
|
# `imagetools create` is a registry-side manifest copy — no layer
|
|
# transfer, no local daemon, nothing that can be absent. The reuse case
|
|
# has always gone this way, so this puts the build case on the code that
|
|
# was already proven rather than on a second path.
|
|
#
|
|
# Running on every path also keeps family rule 146 true: a rolling
|
|
# channel refreshes itself, so skipping a build must never leave :dev or
|
|
# :latest pointing at something older than the commit just pushed.
|
|
#
|
|
# The cost, accepted knowingly: `imagetools create` wraps its source in
|
|
# an index, so :c-<sha> becomes an index and fc.revision does not
|
|
# resolve through it. Nothing reads that label off :c-<sha> — the reuse
|
|
# check only ever inspects the CHANNEL tag — and the index names the
|
|
# same manifest, so a pull is byte-identical. The reuse path already
|
|
# produced :c-<sha> this way; this only makes it uniform.
|
|
- name: Write the remaining tags from the published image
|
|
env:
|
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
|
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
|
# Empty when no build ran this job (a reuse hit, or the step's `if:`
|
|
# skipped it). Non-empty means THIS run pushed that manifest.
|
|
BUILT_DIGEST: ${{ steps.build.outputs.digest }}
|
|
TAGS: ${{ steps.tag.outputs.tags }}
|
|
run: |
|
|
set -euf
|
|
# WHAT WE COPY FROM, which is not what we EXCLUDE (#4290).
|
|
#
|
|
# This step used to copy from the channel tag by NAME. Nothing
|
|
# serialises builds — there is no `concurrency:` key anywhere in
|
|
# .forgejo/workflows/ — so two pushes to one branch run in full
|
|
# parallel, both miss the reuse check, and both build. If the OLDER
|
|
# one finishes last it wins the channel tag; and then its repoint
|
|
# step, reading that tag by name, wrote :c-<sha> from whatever the
|
|
# other run had just published. An immutable rollback tag (rule 145)
|
|
# naming a different commit's bytes, wrong from birth — and
|
|
# immutability then guarantees nobody ever corrects it. Nothing goes
|
|
# red; it surfaces the day someone needs to roll back.
|
|
#
|
|
# So when this job built, copy from the DIGEST it pushed. Correct
|
|
# whatever a concurrent run does to the tag, and it does not depend
|
|
# on the runner honouring a `concurrency:` key — which this file has
|
|
# already been burned by once (the `format()` note at the top: an
|
|
# expression that evaluated false with no symptom at all).
|
|
#
|
|
# On a reuse hit there is no digest, and the channel tag is still the
|
|
# right source: "hit" MEANS that tag already carries this commit's
|
|
# fc.revision, which the reuse step verified by reading it.
|
|
if [ -n "${BUILT_DIGEST:-}" ]; then
|
|
SOURCE="$IMAGE@$BUILT_DIGEST"
|
|
echo "repoint: copying the digest this run published: $SOURCE"
|
|
else
|
|
SOURCE="$CHANNEL_REF"
|
|
echo "repoint: no build this run (reuse hit) — copying from $SOURCE"
|
|
fi
|
|
# The source tag is EXCLUDED from the targets, and that is load-
|
|
# bearing rather than an optimisation.
|
|
#
|
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
|
# at the channel tag with that same tag as a target and the tag stops
|
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
|
# resolves through it and the fc.revision label reads as absent. The
|
|
# next push then misses and rebuilds, so reuse worked exactly once
|
|
# and every subsequent push paid full price. Observed on run 4751:
|
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
|
#
|
|
# Excluding the source means the channel tag is only ever written
|
|
# by a real build, so it stays a plain image and stays readable.
|
|
# On dev that leaves nothing to do either way: the build pushed :dev
|
|
# itself, or the hit established it was already right. On main it
|
|
# leaves :c-<sha>, which rule 145 requires of every main push whether
|
|
# or not a build ran.
|
|
#
|
|
# steps.tag emits ONE comma-separated list; imagetools wants a -t per
|
|
# ref. (That list used to feed docker/build-push-action directly —
|
|
# which is exactly what #3190 made unsafe.)
|
|
ARGS=""
|
|
IFS=,
|
|
for t in $TAGS; do
|
|
# Keyed on CHANNEL_REF, never on SOURCE. SOURCE may now be a digest
|
|
# ref, which never equals a tag string — testing against it would
|
|
# stop excluding the channel tag, imagetools would index-wrap it,
|
|
# and `.Image.Config.Labels` would stop resolving through it. That
|
|
# kills the reuse label permanently (see the note just below).
|
|
[ "$t" = "$CHANNEL_REF" ] && continue
|
|
ARGS="$ARGS -t $t"
|
|
done
|
|
unset IFS
|
|
#
|
|
# This is also the whole of the scheduled refresh's tag handling
|
|
# (#3154): a refresh's tag list is the channel tag alone, so
|
|
# CHANNEL_REF is the only entry, it gets excluded, and this step
|
|
# correctly does nothing. No `if:` on the step and no schedule
|
|
# special-case — excluding the channel tag was already the right
|
|
# rule. (A refresh builds to :refresh-candidate, so BUILT_DIGEST is
|
|
# set here and simply goes unused; `promote` moves :latest later.)
|
|
if [ -z "$ARGS" ]; then
|
|
echo "repoint: $CHANNEL_REF is the only tag for this channel and"
|
|
echo "repoint: already holds this revision — nothing to write."
|
|
exit 0
|
|
fi
|
|
# shellcheck disable=SC2086
|
|
docker buildx imagetools create $ARGS "$SOURCE"
|
|
echo "repointed from $SOURCE:$ARGS"
|
|
|
|
# Does the image a refresh just built still work?
|
|
#
|
|
# This is the gate the base refresh never had. The five verification lanes
|
|
# cannot be it: they run on ci-python:3.14 and install requirements.txt, and
|
|
# a base bump changes neither — all five stay green through a refresh that breaks the
|
|
# product. What a refresh re-resolves is the Dockerfile's apt layer (ffmpeg,
|
|
# unar, libpq5, postgresql-client, zstd, megatools, libjpeg62-turbo,
|
|
# libwebp7, libpng16-16), unpinned, every build.
|
|
#
|
|
# So this runs the CANDIDATE IMAGE, against real Postgres and Redis. Not the
|
|
# source tree, and not a static inspection: `ffmpeg -version` exiting 0 would
|
|
# pass while a codec removal broke every thumbnail in the library.
|
|
#
|
|
# Refresh-only. On a push the bytes came from a commit, and a commit is what
|
|
# the lanes above already test — and since 2026-09-23 they gate the build, so
|
|
# those bytes could not exist without them having passed.
|
|
#
|
|
# Reports a verdict; it does not yet gate the promote (milestone 362 step 4).
|
|
# Landing the gate and the thing it gates in one change would mean the first
|
|
# time anyone saw this job run would also be the first time it could stop a
|
|
# publish.
|
|
smoke-web:
|
|
needs: [build-web]
|
|
# Every run that actually BUILT something, not just the weekly refresh.
|
|
# The egress property (rule 164) is broken by a code or Dockerfile change,
|
|
# which is a push — checking it only on the refresh would test it on the
|
|
# one trigger that changes no source.
|
|
#
|
|
# A reuse hit is skipped deliberately: those bytes are already published
|
|
# and were smoked when they were built. Re-smoking them would burn two
|
|
# minutes to re-learn a fact.
|
|
#
|
|
# HONEST LIMIT, and it is the reason #4299 exists: on a push this runs
|
|
# AFTER build-web has written the channel tag, so it detects rather than
|
|
# gates. Rule 164's verify_with asks for the check BETWEEN build and push.
|
|
# Closing that needs the push path to adopt the candidate-then-promote
|
|
# shape the refresh already has — per-channel candidate tags, promote
|
|
# learning its channel, and the :c-<sha> repoint moving after the gate.
|
|
# That is a redesign of the production publish path and is its own task.
|
|
#
|
|
# NO `if:` — this job always runs (#4323). It used to be gated on the
|
|
# build having published something, which skipped it on a reuse hit. That
|
|
# sounds like an optimisation and is a hole: this workflow file is in no
|
|
# artifact's path set (correctly — editing it changes no shipped byte), so
|
|
# a commit that touches ONLY the smoke moves no revision, hits reuse,
|
|
# emits no digest, and skips the smoke. The one commit whose purpose is
|
|
# changing this check was the one commit that could not run it.
|
|
#
|
|
# That is not hypothetical twice over. 5ca1058 added the egress sandbox
|
|
# and went green three times with this job SKIPPED. 7175ace fixed the
|
|
# bug that hid behind those greens (#4319) and needed a manual
|
|
# force_build to exercise at all. Both relied on someone remembering.
|
|
#
|
|
# It is also where the other historical failure lived: on run 5290 this
|
|
# expression read `env`, which a job `if:` cannot see, so it evaluated
|
|
# empty and skipped silently. Two skips, one expression. The expression
|
|
# goes.
|
|
#
|
|
# The cost is one pull and boot on a reuse-hit push, re-smoking bytes
|
|
# that were smoked when they were built. That is the price of a harness
|
|
# that tests itself, and it overlaps the lanes above, so little wall-clock
|
|
# moves. Not running is not the same as passing.
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
env:
|
|
DB_USER: fabledcurator
|
|
DB_PASSWORD: ci_smoke
|
|
DB_PORT: "5432"
|
|
DB_NAME: fabledcurator_smoke
|
|
SECRET_KEY: ci_smoke_placeholder
|
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator
|
|
services:
|
|
postgres:
|
|
image: pgvector/pgvector:pg16
|
|
env:
|
|
POSTGRES_USER: fabledcurator
|
|
POSTGRES_PASSWORD: ci_smoke
|
|
POSTGRES_DB: fabledcurator_smoke
|
|
options: >-
|
|
--health-cmd "pg_isready -U fabledcurator"
|
|
--health-interval 10s
|
|
--health-timeout 5s
|
|
--health-retries 10
|
|
redis:
|
|
image: redis:7-alpine
|
|
options: >-
|
|
--health-cmd "redis-cli ping"
|
|
--health-interval 10s
|
|
--health-timeout 5s
|
|
--health-retries 10
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
# The same ref the image was built from, so the smoke script matches
|
|
# the code inside the candidate.
|
|
ref: ${{ env.BUILD_REF }}
|
|
|
|
- name: Smoke the candidate image
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
ACTOR: ${{ github.actor }}
|
|
BUILT_DIGEST: ${{ needs.build-web.outputs.digest }}
|
|
PUBLISHED_DIGEST: ${{ needs.build-web.outputs.published_digest }}
|
|
IS_CANDIDATE: ${{ needs.build-web.outputs.candidate }}
|
|
run: |
|
|
set -eux
|
|
# Service discovery mirrors the integration lane above: these jobs run
|
|
# in a container against a mounted docker socket, so the services are
|
|
# SIBLINGS reachable by IP, not by hostname.
|
|
PG=$(docker ps --filter "name=smoke" --filter "ancestor=pgvector/pgvector:pg16" -q | head -n1)
|
|
RD=$(docker ps --filter "name=smoke" --filter "ancestor=redis:7-alpine" -q | head -n1)
|
|
test -n "$PG" && test -n "$RD"
|
|
PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$PG")
|
|
RD_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$RD")
|
|
test -n "$PG_IP" && test -n "$RD_IP"
|
|
|
|
# Socket probe in python, not bash's /dev/tcp — these steps run under
|
|
# `sh -e`, where that path does not exist. Same fix and reasoning as
|
|
# the integration lane above; see the comment there.
|
|
pg_ready=""
|
|
for i in $(seq 1 60); do
|
|
if python -c "import socket,sys; s=socket.socket(); s.settimeout(2); sys.exit(0 if s.connect_ex(('$PG_IP', 5432)) == 0 else 1)"; then
|
|
pg_ready=1
|
|
break
|
|
fi
|
|
sleep 2
|
|
done
|
|
if [ -z "$pg_ready" ]; then
|
|
echo "postgres at $PG_IP:5432 did not accept a connection within 120s"
|
|
exit 1
|
|
fi
|
|
|
|
echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
|
# WHAT GETS SMOKED, in order of preference — always a DIGEST, never
|
|
# a tag: a tag can move between the build and this job, and then the
|
|
# check reports on bytes nobody here decided to ship (#4290).
|
|
#
|
|
# 1. what this run built, when it built;
|
|
# 2. what the channel tag already names, on a reuse hit — the
|
|
# digest the reuse step resolved while reading its label. This
|
|
# is the case that makes the job able to verify a change to
|
|
# ITSELF (#4323), since such a change rebuilds nothing.
|
|
#
|
|
# A refresh always lands in (1); the candidate tag stays only as the
|
|
# last resort for one, and says so rather than being a silent else.
|
|
DIGEST="${BUILT_DIGEST:-}"
|
|
[ -n "$DIGEST" ] || DIGEST="${PUBLISHED_DIGEST:-}"
|
|
if [ -n "$DIGEST" ]; then
|
|
CANDIDATE="$IMAGE@$DIGEST"
|
|
elif [ "${IS_CANDIDATE:-}" = "true" ]; then
|
|
CANDIDATE="$IMAGE:refresh-candidate"
|
|
else
|
|
# Nothing built and nothing published. There is no artifact this
|
|
# job could honestly report on, so it fails rather than passing
|
|
# quietly — a guard with nothing to check is not a passing guard.
|
|
echo "smoke: FAILED — no image to smoke. build-web neither built" >&2
|
|
echo "smoke: one nor resolved a published digest, so there is" >&2
|
|
echo "smoke: nothing here to verify." >&2
|
|
exit 1
|
|
fi
|
|
docker pull "$CANDIDATE"
|
|
|
|
# --- EGRESS BLOCKED from here (rule 164) ---------------------------
|
|
#
|
|
# Rule 164 requires a deployed instance to start and serve its full
|
|
# UI with NO outbound internet, and says to verify it by removing the
|
|
# network rather than by reading the code. Until now this job proved
|
|
# the image WORKS; it never proved it works OFFLINE, because every
|
|
# container below ran on the runner's default network with the
|
|
# internet one hop away.
|
|
#
|
|
# That gap became load-bearing at milestone 422 step 6. The ML role
|
|
# used to run `download_models` before celery started — a boot that
|
|
# reached HuggingFace for ~3.5GB — and that fetch moved to a task
|
|
# enqueued when the lane is enabled. This check is what proves it
|
|
# actually moved, rather than proving it on the machine that built it
|
|
# where the model is already cached.
|
|
#
|
|
# `--internal` is the mechanism rule 164's own verify_with names, and
|
|
# `--network none` is explicitly the WRONG check here: it would only
|
|
# prove the app fails without a database, which proves nothing about
|
|
# egress. An internal network blocks the default route while leaving
|
|
# container-to-container traffic and embedded DNS intact, so Postgres
|
|
# and Redis stay reachable and nothing else is.
|
|
#
|
|
# The service containers are SIBLINGS created by the runner, so they
|
|
# are attached to the internal network rather than created on it.
|
|
# They keep their original network too — that is fine, since what
|
|
# must be offline is the APP container, and it is created with only
|
|
# this network.
|
|
# Named for the RUN, not for `$$`. The shell's pid is deterministic
|
|
# in this runner — every execution of this step got 157 — so `$$`
|
|
# produced one shared name, and the second run died on "network with
|
|
# name smoke-noegress-157 already exists". A pid is unique among
|
|
# LIVE processes, which is not the same as unique over time, and in
|
|
# a fresh container it is neither.
|
|
# The date fallback matters: if the runner does not set these,
|
|
# a literal default would put every run back on one shared name
|
|
# — the bug, with different letters.
|
|
NET="smoke-noegress-${GITHUB_RUN_ID:-$(date +%s)}-${GITHUB_RUN_ATTEMPT:-0}"
|
|
|
|
# Sweep anything an earlier run left behind. Needed because the
|
|
# cleanup below used to be destroyed before it could fire (see the
|
|
# trap note), so every execution leaked its network. `rm` on one
|
|
# still in use fails, and `|| true` keeps that harmless — so a
|
|
# concurrent run's network survives this.
|
|
docker network ls --filter name=^smoke-noegress- -q \
|
|
| while read -r stale; do
|
|
docker network rm "$stale" >/dev/null 2>&1 || true
|
|
done
|
|
|
|
docker network create --internal "$NET"
|
|
|
|
# ONE exit trap, for everything. `trap ... EXIT` REPLACES the
|
|
# previous handler rather than adding to it, so the network's own
|
|
# trap used to be silently discarded the moment the container's was
|
|
# installed further down — and the network was never removed. That
|
|
# is invisible in a passing run and only ever surfaces on the NEXT
|
|
# one, as a name collision.
|
|
#
|
|
# CID is empty until the app container exists, so this is safe to
|
|
# arm now and still covers a failure before that point.
|
|
CID=""
|
|
CID_ALL=""
|
|
cleanup() {
|
|
rc=$?
|
|
for c in $CID $CID_ALL; do
|
|
# The log ONLY on failure — a boot that never answered must fail
|
|
# with the reason visible rather than as a bare timeout (rule
|
|
# 156), while a green run has nothing to say.
|
|
[ $rc -eq 0 ] || docker logs "$c" 2>&1 | tail -40
|
|
docker rm -f "$c" >/dev/null 2>&1 || true
|
|
done
|
|
docker network rm "$NET" >/dev/null 2>&1 || true
|
|
# Preserve the real status, which a trap ending on a successful
|
|
# `docker rm` would otherwise mask.
|
|
exit $rc
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
docker network connect "$NET" "$PG"
|
|
docker network connect "$NET" "$RD"
|
|
# Re-read the addresses ON THIS NETWORK. The IPs discovered above
|
|
# belong to the runner's default bridge and are not routable from a
|
|
# container that is only on the internal one.
|
|
PG_IP=$(docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" "$PG")
|
|
RD_IP=$(docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" "$RD")
|
|
test -n "$PG_IP" && test -n "$RD_IP"
|
|
|
|
ENVOPTS="--network $NET"
|
|
ENVOPTS="$ENVOPTS -e DB_USER=$DB_USER -e DB_PASSWORD=$DB_PASSWORD -e DB_HOST=$PG_IP"
|
|
ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY"
|
|
ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0"
|
|
ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0"
|
|
# A throwaway CI instance IS first-time setup, which is the one case
|
|
# credential_crypto allows a key to be minted in. Without it the web
|
|
# role refuses to boot — deliberately, since silently generating a
|
|
# key on a restored-DB-but-lost-secrets deployment would leave every
|
|
# Credential row undecryptable (the 2026-06-02 audit). Discovered by
|
|
# this job on its first real run; see #3422 for the fact that no
|
|
# user-facing file mentions this variable at all.
|
|
ENVOPTS="$ENVOPTS -e CURATOR_BOOTSTRAP_NEW_KEY=1"
|
|
|
|
# 0. PROVE the network is actually blocking egress. Without this the
|
|
# rest is theatre: if `--internal` silently stopped working, or
|
|
# the app container picked up a second network, every check below
|
|
# would pass with the internet available and report an offline
|
|
# boot that never happened. A guard that cannot fail is not a
|
|
# guard (rule 167).
|
|
echo "smoke: confirming the sandbox has no route out"
|
|
if docker run --rm --network "$NET" "$CANDIDATE" shell -c \
|
|
'python3 -c "import socket,sys; s=socket.socket(); s.settimeout(5); sys.exit(0 if s.connect_ex((\"1.1.1.1\", 443)) == 0 else 1)"'; then
|
|
echo "smoke: FAILED — the sandbox reached 1.1.1.1:443." >&2
|
|
echo "smoke: the network is NOT internal, so nothing below would" >&2
|
|
echo "smoke: have tested the offline property (rule 164)." >&2
|
|
exit 1
|
|
fi
|
|
echo "smoke: no route out, as required"
|
|
|
|
# 1. The schema builds from empty, using the image's OWN libpq and
|
|
# psycopg. This is the same call entrypoint.sh makes before it
|
|
# serves anything, so a failure here is a failure to boot.
|
|
echo "smoke: alembic upgrade head"
|
|
docker run --rm $ENVOPTS "$CANDIDATE" alembic upgrade head
|
|
|
|
# 2. The apt layer's binaries and the app's own thumbnail path, run
|
|
# inside the image. Piped over stdin rather than bind-mounted: the
|
|
# workspace is a docker VOLUME belonging to this job's container,
|
|
# so a host bind of $PWD would not resolve for a sibling.
|
|
echo "smoke: image-internal checks"
|
|
docker run --rm -i $ENVOPTS "$CANDIDATE" shell -c 'python3 -' < scripts/smoke_image.py
|
|
|
|
# 3. It actually serves. `docker run -d` then poll the container's own
|
|
# IP — no port publishing, because the job container reaches
|
|
# siblings directly and a published port would collide with
|
|
# whatever else the runner is hosting.
|
|
echo "smoke: web boots and answers /api/health"
|
|
# Assigning CID is all that is needed — the single EXIT trap armed
|
|
# beside the network already covers the container, and it checks CID
|
|
# for emptiness precisely so it can be installed before this line.
|
|
# Installing a second trap here is what used to discard the first.
|
|
CID=$(docker run -d $ENVOPTS "$CANDIDATE" web)
|
|
WEB_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$CID")
|
|
test -n "$WEB_IP"
|
|
|
|
# The probe runs INSIDE the sandbox, like every check above it.
|
|
#
|
|
# It has to. Docker gives an `--internal` network isolation rules
|
|
# that DROP traffic entering it from any other interface, and this
|
|
# job's own container sits on the runner's default bridge — so a
|
|
# curl from here to $WEB_IP is discarded before it arrives. Because
|
|
# the packets are dropped rather than refused, every attempt burns
|
|
# the full --max-time and the job reports a web container that
|
|
# "never answered" while the app is running perfectly. That is how
|
|
# this read on its first real execution (run 7282): a false failure
|
|
# blaming the application for the harness's own blind spot.
|
|
#
|
|
# Steps 0-2 were already right by accident — each runs a container
|
|
# ON $NET. Only this one reached in from outside, and it was the
|
|
# only one that could not work.
|
|
#
|
|
# Same shape as the egress probe above: the image's own python3 over
|
|
# `shell -c`, since the runtime stage ships no curl.
|
|
probe() {
|
|
docker run --rm --network "$NET" "$CANDIDATE" shell -c \
|
|
"python3 -c \"import urllib.request; urllib.request.urlopen('http://$WEB_IP:8080/api/health', timeout=5)\"" \
|
|
>/dev/null 2>&1
|
|
}
|
|
|
|
healthy=""
|
|
for i in $(seq 1 60); do
|
|
if probe; then
|
|
healthy=1
|
|
break
|
|
fi
|
|
# A container that has EXITED will never answer, so stop asking.
|
|
# Without this the loop spent 3m35s polling a dead container on
|
|
# this job's first run, and — because docker recycles the IP — got
|
|
# a confusing mix of connection-refused and 5s timeouts from
|
|
# whatever took the address next. The trap's log dump had the real
|
|
# answer the whole time; this just stops burying it.
|
|
if [ "$(docker inspect -f '{{.State.Running}}' "$CID" 2>/dev/null)" != "true" ]; then
|
|
echo "smoke: FAILED — the web container exited during boot." >&2
|
|
echo "smoke: its log follows; entrypoint runs alembic BEFORE" >&2
|
|
echo "smoke: serving, so a startup exception lands here." >&2
|
|
exit 1
|
|
fi
|
|
sleep 2
|
|
done
|
|
if [ -z "$healthy" ]; then
|
|
# 60 iterations of (up to 5s connect + 2s sleep) — up to ~7min, not
|
|
# the 120s an earlier version of this message claimed.
|
|
echo "smoke: FAILED — web is running but never answered" >&2
|
|
echo "smoke: /api/health. It is up, so look at hypercorn and the" >&2
|
|
echo "smoke: python base rather than at startup." >&2
|
|
exit 1
|
|
fi
|
|
# Print what it actually answered — from inside, for the same reason.
|
|
docker run --rm --network "$NET" "$CANDIDATE" shell -c \
|
|
"python3 -c \"import urllib.request; print(urllib.request.urlopen('http://$WEB_IP:8080/api/health', timeout=5).read().decode())\""
|
|
|
|
# 4. THE DEFAULT ROLE. Everything above boots `web` explicitly, so
|
|
# until now nothing had ever started the shape the single-
|
|
# container layout actually runs — supervisord bringing up
|
|
# hypercorn plus one celery process per lane. The milestone is
|
|
# named for that shape and CI had never once executed it.
|
|
#
|
|
# Deliberately NO command: this is `docker run <image>` with
|
|
# nothing after it, so it checks the Dockerfile's CMD and the
|
|
# entrypoint's default together with the role itself. An adopter
|
|
# who writes no `command:` gets exactly this.
|
|
#
|
|
# The image's own `healthcheck` is the right assertion, and it
|
|
# was itself never run: for the `all` role it passes only when
|
|
# hypercorn answers AND every lane in the table is answering. A web-only check would go
|
|
# green with every worker dead, which is the failure mode
|
|
# consolidation creates.
|
|
#
|
|
# Cheap because ml ships at 0 slots and disabled, so nothing
|
|
# loads a model — the lane answers `inspect` with its consumers
|
|
# cancelled, which is what "healthy" means for a disabled lane.
|
|
echo "smoke: the default role boots every lane under supervisord"
|
|
CID_ALL=$(docker run -d $ENVOPTS "$CANDIDATE")
|
|
lanes_up=""
|
|
for i in $(seq 1 60); do
|
|
if docker exec "$CID_ALL" python -m backend.app.scripts.healthcheck; then
|
|
lanes_up=1
|
|
break
|
|
fi
|
|
if [ "$(docker inspect -f '{{.State.Running}}' "$CID_ALL" 2>/dev/null)" != "true" ]; then
|
|
echo "smoke: FAILED — the all-role container exited during boot." >&2
|
|
exit 1
|
|
fi
|
|
sleep 3
|
|
done
|
|
if [ -z "$lanes_up" ]; then
|
|
echo "smoke: FAILED — supervisord came up but the composite" >&2
|
|
echo "smoke: healthcheck never passed: either hypercorn did not" >&2
|
|
echo "smoke: answer or a lane is not consuming. Its log follows." >&2
|
|
exit 1
|
|
fi
|
|
echo "smoke: every lane answered"
|
|
|
|
# PID 1 is an init, and the IMAGE provides it — no `init: true` in
|
|
# whatever runs this. Read from /proc rather than `ps`, which the
|
|
# runtime stage does not install.
|
|
PID1=$(docker exec "$CID_ALL" cat /proc/1/comm)
|
|
echo "smoke: pid 1 is $PID1"
|
|
if [ "$PID1" != "tini" ]; then
|
|
echo "smoke: FAILED — pid 1 is '$PID1', not an init." >&2
|
|
echo "smoke: orphaned gallery-dl/ffmpeg/pg_dump processes would" >&2
|
|
echo "smoke: accumulate as zombies, and the image would be back" >&2
|
|
echo "smoke: to needing init:true from every deployment." >&2
|
|
exit 1
|
|
fi
|
|
# Say WHICH processes supervisord is running, so a lane that is
|
|
# merely restart-looping is visible rather than inferred.
|
|
#
|
|
# NOT `|| true` any more. Behind that, this printed
|
|
# "Error: .ini file does not include supervisorctl section" for a
|
|
# whole run and passed — supervisord was fine and supervisorctl
|
|
# could not reach it, which is the tool an operator debugging a lane
|
|
# inside the one container reaches for first. A diagnostic allowed
|
|
# to fail silently is a diagnostic that stops being true without
|
|
# telling anyone.
|
|
docker exec "$CID_ALL" supervisorctl -c "${SUPERVISOR_CONF:-/tmp/supervisord.conf}" status
|
|
|
|
echo "smoke: all checks passed against $CANDIDATE, with egress blocked"
|
|
|
|
# Move the channel tags — the whole point of the gate.
|
|
#
|
|
# Lives in its own job because the verdict it depends on cannot exist until
|
|
# after build-web has finished, and the promote used to run INSIDE build-web.
|
|
#
|
|
# `needs` on smoke-web is the gate. A failed smoke skips this job, so a
|
|
# refresh that broke something leaves :latest naming the build that works —
|
|
# "the refresh failed" and "production is broken" must not be the same event.
|
|
# A SKIPPED smoke also skips this job, which is the behaviour that matters
|
|
# most: on run 5290 the gate silently skipped itself, and a design where only
|
|
# a FAILED gate blocks would have published unverified images while reporting
|
|
# success. Not running is not the same as passing.
|
|
#
|
|
# BOTH images promote TOGETHER, or neither does. They are one stack:
|
|
# build.yml already refuses to publish a :dev web image beside a stale :dev
|
|
# agent, because the mismatch only shows up as a runtime failure. A refresh
|
|
# that published the agent and withheld web would be that same trap, arrived
|
|
# at through the gate.
|
|
#
|
|
# The gate covers the web image only (milestone 362 step 3 scoped it there),
|
|
# so the agent is being held to web's verdict rather than its own. That is
|
|
# deliberate and it is the conservative direction — they ship together, so
|
|
# the weakest evidence should govern both — but it is not the same as having
|
|
# smoked it, and it should not be read as if it were.
|
|
promote:
|
|
needs: [build-web, build-agent, smoke-web]
|
|
# Only a refresh publishes through a candidate; a push writes its channel
|
|
# tag directly from the build. Reads the same reuse-step decision the build
|
|
# took, via a job output — a job's `if:` cannot see the `env` context.
|
|
if: needs.build-web.outputs.candidate == 'true'
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
steps:
|
|
- name: Point the channel tags at the smoked candidates
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
ACTOR: ${{ github.actor }}
|
|
run: |
|
|
set -eu
|
|
# `latest` is not a guess: a refresh always builds `main` (BUILD_REF),
|
|
# and the "must have checked out main" guard in every build job fails
|
|
# the run if that did not hold. So the channel is main's.
|
|
TAG=latest
|
|
FAILED=""
|
|
|
|
for NAME in fabledcurator fabledcurator-agent; do
|
|
REPO="bvandeusen/$NAME"
|
|
echo "promote: $REPO"
|
|
|
|
# Registry auth is its own token exchange — `docker login`
|
|
# authenticates the docker client, not curl. Deadline on every call
|
|
# (rule 156): a registry that stops answering must fail this step,
|
|
# not hang the weekly refresh until the job times out.
|
|
BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \
|
|
"https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \
|
|
| python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
|
|
|
|
# Ask for the IMAGE manifest media types only. Offering the index
|
|
# types too would let the registry hand back an index if one ever
|
|
# existed at this tag, and we would faithfully copy the thing this
|
|
# whole approach exists to avoid creating.
|
|
ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json'
|
|
CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \
|
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
|
"https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \
|
|
&& tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}')
|
|
test -n "$CT"
|
|
SRC=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
|
echo "promote: candidate $SRC ($CT)"
|
|
|
|
# NOT `imagetools create`. That wraps its source in an INDEX, and
|
|
# `.Image.Config.Labels` does not resolve through one — the
|
|
# fc.revision the reuse check reads off the channel tag would come
|
|
# back empty, every later push would miss and rebuild, and nothing
|
|
# would go red (#3183, run 4751). A manifest PUT is what "make this
|
|
# tag name that image" means at the registry: same bytes, same media
|
|
# type, same digest, no layer transfer.
|
|
curl -fsS --max-time 120 -X PUT \
|
|
-H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \
|
|
--data-binary @manifest.json \
|
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG"
|
|
|
|
# Read it back. A PUT that returned 2xx but landed something else is
|
|
# exactly the silent-and-plausible failure this pipeline keeps
|
|
# producing, and the check costs one request.
|
|
NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \
|
|
-H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \
|
|
"https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \
|
|
| tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}')
|
|
if [ "$NOW" != "$SRC" ]; then
|
|
echo "promote: FAILED — $NAME:$TAG is $NOW, expected $SRC" >&2
|
|
FAILED="$FAILED $NAME"
|
|
continue
|
|
fi
|
|
echo "promote: $NAME:$TAG now names $NOW"
|
|
done
|
|
|
|
if [ -n "$FAILED" ]; then
|
|
echo "" >&2
|
|
echo "promote: FAILED for:$FAILED" >&2
|
|
echo "promote: the channel tags are now INCONSISTENT — some images" >&2
|
|
echo "promote: moved and some did not. Re-run this refresh; the" >&2
|
|
echo "promote: candidates are still published and the promote is" >&2
|
|
echo "promote: idempotent." >&2
|
|
exit 1
|
|
fi
|
|
echo "promote: both channel tags moved"
|
|
build-agent:
|
|
# THE GATE (2026-09-23). Every lane above must have PASSED before this job
|
|
# exists at all — so a red suite does not produce an image, let alone push
|
|
# one. Nothing here is a new mechanism: it is the same `needs:` edge that
|
|
# has gated `promote` since milestone 362 step 4, and it carries that
|
|
# step's hardest-won property unchanged — **not running is not the same as
|
|
# passing.** `needs` treats a SKIPPED dependency as unsatisfied, so a lane
|
|
# that silently skips itself blocks the publish exactly as a failing one
|
|
# does. Run 5290 is why that is worth saying out loud: `smoke-web` skipped
|
|
# itself through a job-level `if:` that could not read `env`, and a design
|
|
# where only a FAILED gate blocks would have published unverified images
|
|
# while reporting success.
|
|
needs: [lint, extension-version, backend-lint-and-test, frontend-build, integration]
|
|
# A pull_request run is the lanes and nothing else. This is the ONLY thing
|
|
# separating "validate a Renovate bump" from "publish a Renovate bump", so
|
|
# it is stated on each publishing job rather than inferred from a `needs`
|
|
# chain that a later edit could quietly break.
|
|
if: github.event_name != 'pull_request'
|
|
runs-on: python-ci
|
|
container:
|
|
image: git.fabledsword.com/bvandeusen/ci-python:3.14
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
# Not the triggering ref — see the `env:` block at the top. On a
|
|
# scheduled refresh this is `main`; on everything else it is the ref
|
|
# that fired, so this is a no-op on every ordinary path.
|
|
ref: ${{ env.BUILD_REF }}
|
|
# Full history: this job derives its artifact's version from the
|
|
# commit its shipped files last changed in (milestone 313). A
|
|
# depth-1 clone cannot see that commit — it either derives a wrong,
|
|
# too-low value or finds nothing at all, and neither is a failure
|
|
# the build would otherwise notice.
|
|
fetch-depth: 0
|
|
|
|
# See sign-extension's copy for why this guard exists.
|
|
- name: Guard — a scheduled run must have checked out main
|
|
if: env.IS_REFRESH == 'true'
|
|
run: |
|
|
set -eu
|
|
BRANCH=$(git rev-parse --abbrev-ref HEAD)
|
|
echo "schedule: HEAD is $BRANCH ($(git rev-parse --short HEAD))"
|
|
if [ "$BRANCH" != "main" ]; then
|
|
echo "schedule: expected main, got '$BRANCH'." >&2
|
|
echo "schedule: BUILD_REF was not honoured by the runner." >&2
|
|
echo "schedule: refusing to publish a channel tag from it." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# --- derived values, one line (milestone 313) ------------------------
|
|
# These stopped being shadow output at step 3. `revision` decides
|
|
# whether the build below runs at all and `version` is what the image
|
|
# reports about itself; the load-bearing steps each print only the one
|
|
# they use, so this is the only place the pair appears together. When a
|
|
# build is skipped, this is the line that says what the commit derived.
|
|
#
|
|
# Still diagnostic, so it still must not fail the build — no `set -e`,
|
|
# and every derivation falls back to UNAVAILABLE. A broken echo must
|
|
# never be the reason an image does not ship.
|
|
#
|
|
# What it should say:
|
|
# * a push touching only agent/ moves the agent and leaves web STILL.
|
|
# If web moves, its path set is too wide.
|
|
# * a push touching only docs moves nothing.
|
|
# * a push touching the extension moves the extension AND web, since
|
|
# web bakes in the XPI. If web does not move, its set is too narrow:
|
|
# the reuse check hits, and the channel serves a web image bundling
|
|
# the PREVIOUS XPI while the freshly signed one is orphaned (#3156).
|
|
# * dev and main derive the same values for the same source.
|
|
- name: Report the derived artifact version
|
|
env:
|
|
# Diagnostic for the trigger normalisation. `refresh` is reported RAW
|
|
# as well as normalised, because the two disagreeing is the whole
|
|
# failure mode: a dispatch input whose type does not compare the way
|
|
# the expression assumes evaluates to false silently, and the only
|
|
# symptom is a refresh that quietly behaves like an ordinary push.
|
|
RAW_REFRESH: ${{ github.event.inputs.refresh }}
|
|
RAW_FORCE: ${{ github.event.inputs.force_build }}
|
|
run: |
|
|
set -u
|
|
echo "trigger: event=$GITHUB_EVENT_NAME IS_REFRESH='${IS_REFRESH:-<unset>}' BUILD_REF='${BUILD_REF:-<unset>}'"
|
|
echo "trigger: raw inputs refresh='${RAW_REFRESH:-<unset>}' force_build='${RAW_FORCE:-<unset>}'"
|
|
A=agent
|
|
V=$(sh scripts/artifacts.sh version "$A" 2>&1 || echo UNAVAILABLE)
|
|
R=$(sh scripts/artifacts.sh revision "$A" 2>&1 || echo UNAVAILABLE)
|
|
echo "derived: artifact=$A version=$V revision=$R sha=$GITHUB_SHA"
|
|
|
|
- name: Determine tag
|
|
id: tag
|
|
run: |
|
|
SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)
|
|
# Mirrors build-web's tag list and its schedule handling; see
|
|
# the comments there.
|
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest" >> "$GITHUB_OUTPUT"
|
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
|
elif [ "${GITHUB_REF##*/}" = "main" ]; then
|
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:latest,git.fabledsword.com/bvandeusen/fabledcurator-agent:c-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
|
echo "channel=main" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "tags=git.fabledsword.com/bvandeusen/fabledcurator-agent:dev" >> "$GITHUB_OUTPUT"
|
|
echo "channel=dev" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
|
|
# Shell step rather than docker/login-action — see build-web's note on
|
|
# the shared action-cache race (#3118).
|
|
- name: Login to Forgejo registry
|
|
env:
|
|
TOKEN: ${{ secrets.RELEASE_TOKEN }}
|
|
ACTOR: ${{ github.actor }}
|
|
run: echo "$TOKEN" | docker login git.fabledsword.com -u "$ACTOR" --password-stdin
|
|
|
|
# A REAL buildx builder, not the default `docker` driver (#3114, #3190).
|
|
#
|
|
# The default driver builds through the local dockerd. It cannot export a
|
|
# registry cache at all — which is why the agent rebuilds a ~6.3 GB CUDA
|
|
# + torch image from scratch whenever the runner's local cache is cold,
|
|
# measured at 9m26s against 7s warm. It is also #3190's leading suspect:
|
|
# after a registry-direct push it resolves image metadata against a local
|
|
# store the push never filled, and reports `No such image` on an image
|
|
# that published perfectly well three seconds earlier.
|
|
#
|
|
# These jobs run INSIDE a container against a mounted docker socket, so
|
|
# the buildkit container this starts is a SIBLING of the job container,
|
|
# not a child. That works over the socket mount; it had never been tried
|
|
# here before milestone 326 step 1.
|
|
- name: Set up buildx
|
|
uses: docker/setup-buildx-action@v3
|
|
|
|
# --- reuse-if-published (milestone 313, step 4) ----------------------
|
|
# Does the image the channel tag already points at carry THIS commit's
|
|
# revision? If so the bytes this job would produce are already published
|
|
# and the build is pure waste: the remaining tags get repointed at that
|
|
# existing manifest instead, registry-side, in seconds.
|
|
#
|
|
# Keyed on an `fc.revision` LABEL rather than on a tag of its own
|
|
# (milestone 318 step 3). A tag would be a name minted per build that one
|
|
# thing reads — what rule 145 narrowed against — and would be prunable
|
|
# under the registry's keep_pattern (#3157), silently expiring the cache.
|
|
# A label rides inside a tag that has to exist anyway.
|
|
#
|
|
# An image with no such label reads as a miss and rebuilds. That is the
|
|
# migration, not a fault: labels cannot be backfilled, since the reuse
|
|
# path copies a manifest and config labels are not manifest annotations.
|
|
# Each artifact pays one rebuild, once.
|
|
#
|
|
# This is what stops a push that touched only `agent/` from rebuilding
|
|
# web, and a merge to main from rebuilding what dev already built.
|
|
#
|
|
# The failure direction is deliberate. An inspect that errors for ANY
|
|
# reason — network, auth, a registry hiccup — reads as a miss and the
|
|
# build runs. Only a genuine 200 skips one, so there is no path here
|
|
# that skips a build that was actually needed; the worst case is paying
|
|
# for a build we could have avoided.
|
|
#
|
|
# BASE-IMAGE FRESHNESS: an artifact whose source stops moving stops
|
|
# picking up base-image updates. Milestone 318 removed the argument this
|
|
# used to need rather than answering it — with no version tags there is
|
|
# no immutable name a refresh could contradict, and rule 145 already
|
|
# allows a rebuild with different contents to republish a MOVING tag.
|
|
# So a refresh is just a build. A scheduled channel-only one is tracked
|
|
# separately (#3154); it does not belong in the push path.
|
|
- name: Is this content already published?
|
|
id: reuse
|
|
env:
|
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
|
CHANNEL: ${{ steps.tag.outputs.channel }}
|
|
# Empty on a push; the string "true" only from a workflow_dispatch
|
|
# that asked for it. `github.event.inputs` rather than the `inputs`
|
|
# context — release.yml already uses that form, and it is the one
|
|
# this runner is known to evaluate. Read through env rather than
|
|
# interpolated into the run block, same rule as release.yml's TAG.
|
|
FORCE: ${{ github.event.inputs.force_build }}
|
|
# A scheduled refresh has to bypass reuse by construction: it
|
|
# rebuilds the SAME source, so fc.revision always matches and the
|
|
# check would skip every refresh there has ever been.
|
|
run: |
|
|
set -eu
|
|
DERIVED=$(sh scripts/artifacts.sh revision agent)
|
|
echo "revision=$DERIVED" >> "$GITHUB_OUTPUT"
|
|
# Baked into the agent image as FC_VERSION and reported by its
|
|
# control page and /status. A pure function of the revision — same
|
|
# commit, same string — so it adds no variability the reuse check
|
|
# would have to account for. Added 2026-09-24: the agent's version
|
|
# used to be a hand-written literal in app.py that nobody bumped, so
|
|
# the September image reported the same string as the July one.
|
|
echo "version=$(sh scripts/artifacts.sh version agent)" >> "$GITHUB_OUTPUT"
|
|
|
|
# The build clock, pinned to the same commit (#3265). Without it
|
|
# buildkit stamps the image config with the wall clock of the build,
|
|
# so identical layers republish under a new config blob and the
|
|
# channel tag gets a new manifest digest for no reason. Derived from
|
|
# `newest()` like revision and version, so all three name one commit
|
|
# and cannot drift apart.
|
|
echo "epoch=$(sh scripts/artifacts.sh epoch agent)" >> "$GITHUB_OUTPUT"
|
|
|
|
# The moving tag for this channel. Which tag we ask IS the channel —
|
|
# that is why the revision needs no -main/-dev qualifier any more.
|
|
if [ "$CHANNEL" = "main" ]; then T=latest; else T=dev; fi
|
|
echo "channel_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
|
|
|
# WHERE THE BUILD PUBLISHES, which is not always the channel — and
|
|
# whether the channel then has to be written separately.
|
|
#
|
|
# On a push the build writes the channel tag directly: the bytes came
|
|
# from a commit, and a commit is the thing CI tests. Nothing to hold
|
|
# it behind.
|
|
#
|
|
# On the scheduled refresh it writes a CANDIDATE tag instead. A
|
|
# refresh rebuilds against freshly resolved base images, and the web
|
|
# image's runtime is a line of UNPINNED Debian packages (ffmpeg,
|
|
# libjpeg62-turbo, libpq5, megatools…) re-resolved on every build.
|
|
# No verification LANE can see that: they run on ci-python:3.14 and
|
|
# install requirements.txt, and a base bump changes neither. So
|
|
# refreshed bytes have to be proven before :latest names them, and
|
|
# proving needs a moment between "built" and "published" to occupy.
|
|
# This is that moment; :latest goes on naming the build that works
|
|
# until something says otherwise.
|
|
#
|
|
# `:refresh-candidate` is one moving ref per image, overwritten in
|
|
# place, holding a build nobody is told to pull — the shape rule 145
|
|
# already allows for :buildcache, not the per-build tag family that
|
|
# milestone 318 withdrew.
|
|
#
|
|
# Decided HERE, beside `hit`, for the reason the force/schedule
|
|
# branch below gives: one step decides what this job does. A
|
|
# condition derived independently could disagree with the tag the
|
|
# build actually wrote.
|
|
#
|
|
# build-web additionally exposes this as `outputs.candidate`, which is
|
|
# what gates the `promote` job — a job's `if:` cannot read `env`, and
|
|
# one flag is enough because both jobs derive it from the same
|
|
# IS_REFRESH. build-agent does not re-emit it; a second copy nothing
|
|
# reads is the kind of thing that later reads as load-bearing.
|
|
if [ "${IS_REFRESH:-}" = "true" ]; then
|
|
echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
|
|
# Compare VALUES, never exit codes. Measured on buildx v0.36.1
|
|
# (run 4732): a missing key returns an empty string and exits 0, so
|
|
# branching on the exit code would read "no label yet" as success.
|
|
# An unreachable tag also lands here as empty via the `|| echo`.
|
|
# Empty never equals a 12-char revision, so every uncertain case
|
|
# falls through to a build — the safe direction, with no special
|
|
# casing for it.
|
|
#
|
|
# Read the SPECIFIC key. The map also carries whatever the base image
|
|
# set, and `org.opencontainers.image.version` sits right beside ours
|
|
# looking like a plausible answer (it reads 24.04 on the agent).
|
|
PUBLISHED=$(docker buildx imagetools inspect "$IMAGE:$T" \
|
|
--format '{{ index .Image.Config.Labels "fc.revision" }}' \
|
|
2>/dev/null || echo "")
|
|
echo "reuse: $IMAGE:$T carries fc.revision=${PUBLISHED:-<none>}; derived=$DERIVED"
|
|
if [ -z "$PUBLISHED" ] && docker buildx imagetools inspect "$IMAGE:$T" >/dev/null 2>&1; then
|
|
# The tag resolves but carries no readable label. Expected exactly
|
|
# once per artifact, during the migration onto labels. If it recurs
|
|
# every push, something is rewriting the channel tag as a manifest
|
|
# index — see the repoint step's note.
|
|
echo "reuse: NOTE $IMAGE:$T exists but has no readable fc.revision."
|
|
echo "reuse: NOTE Fine once, while migrating. Every push means the"
|
|
echo "reuse: NOTE tag is being index-wrapped and reuse is dead."
|
|
fi
|
|
|
|
# FORCE is checked here rather than in the build step's `if:`, so
|
|
# that one decision drives everything downstream. The repoint step
|
|
# keys off `hit` too, and a force that bypassed only the build would
|
|
# leave the two disagreeing about what just happened.
|
|
if [ "${FORCE:-false}" = "true" ]; then
|
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: force_build set — building regardless"
|
|
elif [ "${IS_REFRESH:-}" = "true" ]; then
|
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: scheduled base refresh — building regardless"
|
|
elif [ -n "$PUBLISHED" ] && [ "$PUBLISHED" = "$DERIVED" ]; then
|
|
echo "hit=true" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: already published — skipping the build"
|
|
else
|
|
echo "hit=false" >> "$GITHUB_OUTPUT"
|
|
echo "reuse: not published — building"
|
|
fi
|
|
|
|
- name: Build and push agent image
|
|
# `id:` so the repoint step below can read `outputs.digest` — the
|
|
# manifest THIS run published, as opposed to whatever the channel tag
|
|
# happens to name by the time that step runs (#4290).
|
|
id: build
|
|
if: steps.reuse.outputs.hit != 'true'
|
|
# Read by buildx out of the ENVIRONMENT, not passed as a build-arg —
|
|
# it normalises the image config's `created` field and the history
|
|
# timestamps rather than being consumed by the Dockerfile. See #3265
|
|
# and the reuse step's `epoch` output.
|
|
env:
|
|
SOURCE_DATE_EPOCH: ${{ steps.reuse.outputs.epoch }}
|
|
uses: docker/build-push-action@v5
|
|
with:
|
|
context: agent
|
|
file: agent/Dockerfile
|
|
push: true
|
|
# Re-resolve the FROM references against the registry instead of
|
|
# trusting whatever digest the cache was built against. This is the
|
|
# whole mechanism of the scheduled refresh (#3154): if the base tag
|
|
# moved, the FROM layer's cache key changes, every layer above it
|
|
# invalidates, and the image genuinely rebuilds.
|
|
#
|
|
# MEASURED on the first real fire, run 4934 (#3265): when the base
|
|
# did NOT move, the build was ~13s with every content step CACHED —
|
|
# and the channel tag STILL got a new manifest digest, because
|
|
# buildkit stamps a fresh image config per run and republishes the
|
|
# identical layers under it. All three images moved that way on
|
|
# 2026-08-30 with nothing whatsoever changed in them.
|
|
#
|
|
# SOURCE_DATE_EPOCH (below) is the fix: pinned to the commit the
|
|
# content came from, the config is byte-identical across runs, so
|
|
# the manifest digest is too and the push is a registry no-op. A
|
|
# digest change means the content changed again, which is the only
|
|
# thing a digest is any use for.
|
|
#
|
|
# What `pull` does NOT catch either: a Debian package update inside
|
|
# the `apt-get install` layer while the base tag itself stands
|
|
# still. The official python/cuda images rebuild with those updates
|
|
# baked in, so this is a lag rather than a hole; closing it needs
|
|
# `no-cache: true`, which is a much larger version of the same
|
|
# churn #3265 is about.
|
|
#
|
|
# Only on the schedule. An ordinary push wants the cached base.
|
|
pull: ${{ env.IS_REFRESH == 'true' }}
|
|
# ONE tag, the channel's. Every other tag is written by the step
|
|
# below, registry-side. buildx here pushes the first tag to the
|
|
# registry and then re-pushes the rest through the DOCKER driver,
|
|
# out of a local image store a registry-direct build never filled —
|
|
# #3190, which cost `main` its :c-<sha> on 2026-08-29 while :latest
|
|
# published perfectly well.
|
|
tags: ${{ steps.reuse.outputs.build_ref }}
|
|
# The reuse key. Read back off the channel tag on the next push to
|
|
# decide whether that push needs to build at all, so this is not
|
|
# decoration — an unstamped image is one that will always rebuild.
|
|
labels: |
|
|
fc.revision=${{ steps.reuse.outputs.revision }}
|
|
# What the agent reports about ITSELF, as opposed to the label above,
|
|
# which is what the registry reports about it. Three values kept
|
|
# apart (rule 149). FC_REVISION is the same string as the label, so
|
|
# the image and the registry cannot disagree about which commit this
|
|
# is — a build whose self-report names a different commit from the
|
|
# tag it was published under is unrollbackable in practice.
|
|
build-args: |
|
|
FC_CHANNEL=${{ steps.tag.outputs.channel }}
|
|
FC_VERSION=${{ steps.reuse.outputs.version }}
|
|
FC_REVISION=${{ steps.reuse.outputs.revision }}
|
|
# LOAD-BEARING, not a preference. On the default docker driver these
|
|
# were no-ops; on the docker-container driver above,
|
|
# build-push-action@v5 defaults provenance to TRUE when pushing.
|
|
# Provenance attaches an attestation manifest, which makes the pushed
|
|
# tag a manifest INDEX — and `.Image.Config.Labels` does not resolve
|
|
# through an index.
|
|
#
|
|
# The label directly above IS the reuse key. Wrap the channel tag in
|
|
# an index and the next push reads fc.revision=<none>, misses, and
|
|
# rebuilds. Then so does the one after that, forever. Nothing fails,
|
|
# nothing goes red, and the only symptom is the bill. That is #3183
|
|
# arriving through a different door, and note #3127 §4 records the
|
|
# same shape for `platforms:`.
|
|
provenance: false
|
|
sbom: false
|
|
# The ONLY cache this driver can have. `docker-container` gets a
|
|
# FRESH buildkit instance per job, so unlike the default docker
|
|
# driver it has no local layer store to fall back on — measured on
|
|
# run 4896, the first builds after the driver change: web 3m44s
|
|
# (was 2m23s), ml 3m49s (was 3m20s), agent 11m12s (was 9m26s). The
|
|
# driver change ALONE is a regression; this is the other half of it.
|
|
#
|
|
# mode=max so intermediate stages cache too. web's frontend-builder
|
|
# stage and the agent's two ~150s pip layers are the whole cost, and
|
|
# they are exactly what a min-mode cache would drop.
|
|
#
|
|
# A `:buildcache` tag is NOT the withdrawn tag scheme coming back.
|
|
# Rule 145 narrowed against names NOTHING reads; this one is read by
|
|
# every build that runs, is one moving ref per image rather than one
|
|
# per build, holds cache blobs rather than a shippable artifact, and
|
|
# is overwritten in place rather than accumulating. It is closer to
|
|
# :dev than to the :2026.8.28 tags milestone 318 deleted. (#3114.)
|
|
cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache
|
|
cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max
|
|
|
|
# Every tag but the channel's own is written HERE, registry-side,
|
|
# whether or not a build ran. Each -t becomes another reference to the
|
|
# SAME manifest the channel tag holds, so :c-<sha> is byte-identical to
|
|
# what is published rather than a lookalike rebuild.
|
|
#
|
|
# Owning the build path too is #3190's fix, not a tidy-up:
|
|
#
|
|
# #27 pushing …/fabledcurator:latest DONE 15.8s
|
|
# #28 pushing …/fabledcurator:c-0e15c44 with docker
|
|
# #28 ERROR: tag does not exist: …:c-0e15c44
|
|
#
|
|
# Intermittent — build-ml made the identical two-tag push seconds later
|
|
# and succeeded — and worse than it looks. `:latest` had already
|
|
# published, so production was correct while the immutable rollback tag
|
|
# rule 145 requires of every main push simply did not exist. Nothing but
|
|
# the red job would ever have noticed: a missing :c-<sha> has no
|
|
# consumer that fails, so it surfaces when somebody needs to roll back.
|
|
#
|
|
# `imagetools create` is a registry-side manifest copy — no layer
|
|
# transfer, no local daemon, nothing that can be absent. The reuse case
|
|
# has always gone this way, so this puts the build case on the code that
|
|
# was already proven rather than on a second path.
|
|
#
|
|
# Running on every path also keeps family rule 146 true: a rolling
|
|
# channel refreshes itself, so skipping a build must never leave :dev or
|
|
# :latest pointing at something older than the commit just pushed.
|
|
#
|
|
# The cost, accepted knowingly: `imagetools create` wraps its source in
|
|
# an index, so :c-<sha> becomes an index and fc.revision does not
|
|
# resolve through it. Nothing reads that label off :c-<sha> — the reuse
|
|
# check only ever inspects the CHANNEL tag — and the index names the
|
|
# same manifest, so a pull is byte-identical. The reuse path already
|
|
# produced :c-<sha> this way; this only makes it uniform.
|
|
- name: Write the remaining tags from the published image
|
|
env:
|
|
IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent
|
|
CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }}
|
|
# Empty when no build ran this job (a reuse hit, or the step's `if:`
|
|
# skipped it). Non-empty means THIS run pushed that manifest.
|
|
BUILT_DIGEST: ${{ steps.build.outputs.digest }}
|
|
TAGS: ${{ steps.tag.outputs.tags }}
|
|
run: |
|
|
set -euf
|
|
# WHAT WE COPY FROM, which is not what we EXCLUDE (#4290).
|
|
#
|
|
# This step used to copy from the channel tag by NAME. Nothing
|
|
# serialises builds — there is no `concurrency:` key anywhere in
|
|
# .forgejo/workflows/ — so two pushes to one branch run in full
|
|
# parallel, both miss the reuse check, and both build. If the OLDER
|
|
# one finishes last it wins the channel tag; and then its repoint
|
|
# step, reading that tag by name, wrote :c-<sha> from whatever the
|
|
# other run had just published. An immutable rollback tag (rule 145)
|
|
# naming a different commit's bytes, wrong from birth — and
|
|
# immutability then guarantees nobody ever corrects it. Nothing goes
|
|
# red; it surfaces the day someone needs to roll back.
|
|
#
|
|
# So when this job built, copy from the DIGEST it pushed. Correct
|
|
# whatever a concurrent run does to the tag, and it does not depend
|
|
# on the runner honouring a `concurrency:` key — which this file has
|
|
# already been burned by once (the `format()` note at the top: an
|
|
# expression that evaluated false with no symptom at all).
|
|
#
|
|
# On a reuse hit there is no digest, and the channel tag is still the
|
|
# right source: "hit" MEANS that tag already carries this commit's
|
|
# fc.revision, which the reuse step verified by reading it.
|
|
if [ -n "${BUILT_DIGEST:-}" ]; then
|
|
SOURCE="$IMAGE@$BUILT_DIGEST"
|
|
echo "repoint: copying the digest this run published: $SOURCE"
|
|
else
|
|
SOURCE="$CHANNEL_REF"
|
|
echo "repoint: no build this run (reuse hit) — copying from $SOURCE"
|
|
fi
|
|
# The source tag is EXCLUDED from the targets, and that is load-
|
|
# bearing rather than an optimisation.
|
|
#
|
|
# `imagetools create` wraps the source manifest in an INDEX. Point it
|
|
# at the channel tag with that same tag as a target and the tag stops
|
|
# being a plain image — after which `.Image.Config.Labels` no longer
|
|
# resolves through it and the fc.revision label reads as absent. The
|
|
# next push then misses and rebuilds, so reuse worked exactly once
|
|
# and every subsequent push paid full price. Observed on run 4751:
|
|
# ml:dev reported fc.revision=<none> one push after run 4749 had read
|
|
# a7e626a67a79 off it. Nothing failed; the savings just evaporated.
|
|
#
|
|
# Excluding the source means the channel tag is only ever written
|
|
# by a real build, so it stays a plain image and stays readable.
|
|
# On dev that leaves nothing to do either way: the build pushed :dev
|
|
# itself, or the hit established it was already right. On main it
|
|
# leaves :c-<sha>, which rule 145 requires of every main push whether
|
|
# or not a build ran.
|
|
#
|
|
# steps.tag emits ONE comma-separated list; imagetools wants a -t per
|
|
# ref. (That list used to feed docker/build-push-action directly —
|
|
# which is exactly what #3190 made unsafe.)
|
|
ARGS=""
|
|
IFS=,
|
|
for t in $TAGS; do
|
|
# Keyed on CHANNEL_REF, never on SOURCE. SOURCE may now be a digest
|
|
# ref, which never equals a tag string — testing against it would
|
|
# stop excluding the channel tag, imagetools would index-wrap it,
|
|
# and `.Image.Config.Labels` would stop resolving through it. That
|
|
# kills the reuse label permanently (see the note just below).
|
|
[ "$t" = "$CHANNEL_REF" ] && continue
|
|
ARGS="$ARGS -t $t"
|
|
done
|
|
unset IFS
|
|
if [ -z "$ARGS" ]; then
|
|
echo "repoint: $CHANNEL_REF is the only tag for this channel and"
|
|
echo "repoint: already holds this revision — nothing to write."
|
|
exit 0
|
|
fi
|
|
# shellcheck disable=SC2086
|
|
docker buildx imagetools create $ARGS "$SOURCE"
|
|
echo "repointed from $SOURCE:$ARGS"
|