From 81b7b6f3089b1ad613ff62d42739d78ca08ad710 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 15:53:40 -0400 Subject: [PATCH 1/6] =?UTF-8?q?ci:=20smoke-web=20never=20ran=20=E2=80=94?= =?UTF-8?q?=20a=20job's=20`if:`=20cannot=20read=20the=20env=20context?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 5290 dispatched a refresh. Everything worked: the guard fired, the build published the candidate, the promote pointed :latest at it. And `smoke-web` reported conclusion "skipped", with no steps and no log. Its condition was `if: env.IS_REFRESH == 'true'`. The env context is available to STEP conditions and step bodies but never to a job's own `if:`, and an unresolvable context there evaluates to empty rather than erroring. So the gate skipped itself, silently, on the one run that existed to exercise it. Second silent-skip of this family today, after #3414. Same shape both times: something evaluated false, nothing failed, and the run reported success. It is worth naming the pattern — on this pipeline, "green" and "ran" are different claims, and the steps' own conclusions are the only place the difference shows. Fixed by keying off a job output rather than re-deriving the trigger: build-web now exposes the reuse step's `promote` decision as `outputs.candidate` and smoke-web consumes it. That is better than duplicating the expression: it is the same single decision the build, the XPI download and the promote all take already — build.yml's own "one decision drives everything downstream" — and it asserts the thing smoke-web actually depends on, that a candidate was published, rather than restating the reason one would be. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index a1c115f..5d3398a 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -452,6 +452,18 @@ jobs: # to. Same source of truth; no double-store. build-web: + # Consumed by smoke-web's job-level `if:`. It cannot read `env` — the env + # context is available to STEP `if:` and step bodies, never to a job's own + # condition, and an unresolvable context there is empty rather than an + # error. `smoke-web` skipped silently on run 5290 for exactly that reason. + # + # Keying off the reuse step's own output is better than re-deriving the + # trigger anyway: it is the same single decision the build, the XPI + # download and the promote all take (build.yml's "one decision drives + # everything downstream"), and it says the thing smoke-web actually needs + # to know — a candidate was published — rather than restating why. + outputs: + candidate: ${{ steps.reuse.outputs.promote }} # A plain `needs` — no `always()`. That expression existed to let a # SKIPPED sign-extension through on a tag push while still blocking a # FAILED one. With no tag trigger, sign-extension always runs, so the @@ -1123,8 +1135,8 @@ jobs: # time anyone saw this job run would also be the first time it could stop a # publish. smoke-web: - if: env.IS_REFRESH == 'true' needs: [build-web] + if: needs.build-web.outputs.candidate == 'true' runs-on: python-ci container: image: git.fabledsword.com/bvandeusen/ci-python:3.14 -- 2.54.0 From 4815040d7486a088d0f2bcfb7914b6409c53d985 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 16:00:18 -0400 Subject: [PATCH 2/6] =?UTF-8?q?ci:=20the=20smoke=20gate=20found=20a=20real?= =?UTF-8?q?=20one=20on=20its=20first=20run=20=E2=80=94=20and=20had=20two?= =?UTF-8?q?=20bugs=20of=20its=20own?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 5296 was `smoke-web`'s first genuine execution. Checks 1 and 2 passed: alembic built the schema from empty inside the image, all five apt binaries resolved, and the application's own Thumbnailer produced JPEG, PNG-with-alpha, WebP and an ffmpeg video frame against the image's libraries. Check 3 failed, and the trap's log dump said exactly why: MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. That is the product being right. credential_crypto refuses to mint a key unless someone opts in, because the 2026-06-02 audit found a partial restore (DB back, /images/secrets/ lost) silently generating a fresh one and leaving a working-looking system where every authenticated download failed AUTH_ERROR. It is also a first-run blocker for milestone 328, filed as #3422: the variable appears in no README, no .env.example and no compose file, so the install path that milestone just finished writing produces a container that exits on boot. Not fixed here — the fix trades safety against friction and is the operator's call. Two defects in the gate itself, both surfaced by the same run: - A throwaway CI instance IS first-time setup, so it now passes CURATOR_BOOTSTRAP_NEW_KEY=1. The check was asserting a condition no fresh container can satisfy. - The health loop polled a dead container for 3m35s. Docker had already recycled its IP, so the replies were a baffling mix of connection-refused and 5s timeouts from whatever took the address next. It now checks `.State.Running` each iteration and fails immediately with the container's log. The trap had the real answer the whole time; this stops burying it under four minutes of noise. Also corrected a message claiming a 120s budget: 60 iterations of up to 5s connect plus 2s sleep is nearer seven minutes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 5d3398a..f0c2424 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -1213,6 +1213,14 @@ jobs: ENVOPTS="$ENVOPTS -e DB_PORT=5432 -e DB_NAME=$DB_NAME -e SECRET_KEY=$SECRET_KEY" ENVOPTS="$ENVOPTS -e CELERY_BROKER_URL=redis://$RD_IP:6379/0" ENVOPTS="$ENVOPTS -e CELERY_RESULT_BACKEND=redis://$RD_IP:6379/0" + # A throwaway CI instance IS first-time setup, which is the one case + # credential_crypto allows a key to be minted in. Without it the web + # role refuses to boot — deliberately, since silently generating a + # key on a restored-DB-but-lost-secrets deployment would leave every + # Credential row undecryptable (the 2026-06-02 audit). Discovered by + # this job on its first real run; see #3422 for the fact that no + # user-facing file mentions this variable at all. + ENVOPTS="$ENVOPTS -e CURATOR_BOOTSTRAP_NEW_KEY=1" # 1. The schema builds from empty, using the image's OWN libpq and # psycopg. This is the same call entrypoint.sh makes before it @@ -1247,12 +1255,26 @@ jobs: healthy=1 break fi + # A container that has EXITED will never answer, so stop asking. + # Without this the loop spent 3m35s polling a dead container on + # this job's first run, and — because docker recycles the IP — got + # a confusing mix of connection-refused and 5s timeouts from + # whatever took the address next. The trap's log dump had the real + # answer the whole time; this just stops burying it. + if [ "$(docker inspect -f '{{.State.Running}}' "$CID" 2>/dev/null)" != "true" ]; then + echo "smoke: FAILED — the web container exited during boot." >&2 + echo "smoke: its log follows; entrypoint runs alembic BEFORE" >&2 + echo "smoke: serving, so a startup exception lands here." >&2 + exit 1 + fi sleep 2 done if [ -z "$healthy" ]; then - echo "smoke: FAILED — web did not answer /api/health within 120s." >&2 - echo "smoke: entrypoint runs alembic BEFORE serving, and step 1" >&2 - echo "smoke: passed, so look at hypercorn and the python base." >&2 + # 60 iterations of (up to 5s connect + 2s sleep) — up to ~7min, not + # the 120s an earlier version of this message claimed. + echo "smoke: FAILED — web is running but never answered" >&2 + echo "smoke: /api/health. It is up, so look at hypercorn and the" >&2 + echo "smoke: python base rather than at startup." >&2 exit 1 fi curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health" -- 2.54.0 From 86abaf0b94f17077004afe55deca9e9276b42b46 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 16:19:28 -0400 Subject: [PATCH 3/6] docs: a new install could not start, and nothing told anyone why (#3422) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The install path milestone 328 wrote produces a web container that exits on boot. entrypoint.sh runs alembic, then app construction raises: MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. That variable appeared in no README, no .env.example and no compose file — only in backend/. So a stranger following the documented steps got an app that does not start and an error with no context. Found by the milestone-362 smoke gate on its first real run (#3422). The product behaviour stays exactly as it is. credential_crypto refuses to mint a key because the 2026-06-02 audit found a partial restore — database back, ./images/secrets lost — silently generating a fresh one and producing a healthy-looking instance where every authenticated download failed AUTH_ERROR. Failing fast is right; not saying so is the bug. So: .env.example carries the variable in its own FIRST BOOT ONLY section with the reasoning and an instruction to delete the line afterwards, and README's First run leads with it, because "the app will not start" belongs before "the ML worker downloads weights". Both say to back up ./images/secrets/ alongside the database, which is the part that costs real data if it is learned late. **compose had to change too, and this is the part that would have shipped a second broken instruction.** A variable in `.env` is only used for ${...} interpolation — it does not reach the container unless the service names it. Telling people to set it in .env, without that, would have documented a step that does nothing. Added to the shared app_env anchor, defaulted to empty so the refusal still stands for everyone who has not opted in. Not taken: auto-bootstrapping when the credential table is empty, which would remove the manual step entirely and keep the audit's protection for restores. That is the better product and it is a code change with a predicate that has to be exactly right; this is the smallest correct fix, and #3422 stays open for the other one. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .env.example | 28 ++++++++++++++++++++++++++++ README.md | 26 +++++++++++++++++++++++++- docker-compose.yml | 14 ++++++++++++++ 3 files changed, 67 insertions(+), 1 deletion(-) diff --git a/.env.example b/.env.example index cbb94d4..fe2cb12 100644 --- a/.env.example +++ b/.env.example @@ -34,6 +34,34 @@ DB_PASSWORD= SECRET_KEY= +# --------------------------------------------------------------------------- +# FIRST BOOT ONLY — then delete this line +# --------------------------------------------------------------------------- + +# FabledCurator encrypts your stored platform credentials with a Fernet key it +# keeps at /images/secrets/credential_key.b64 — inside the ./images bind mount, +# so it outlives the container. On a brand-new install that file does not exist +# yet, and the app REFUSES TO START rather than quietly create one: +# +# MissingCredentialKey: Fernet key file not found at +# /images/secrets/credential_key.b64 +# +# That refusal is deliberate. Auto-creating a key is indistinguishable from the +# disaster case — a restore that brought the database back but lost +# ./images/secrets — and there it would mint a key that cannot decrypt anything, +# leaving an instance that looks healthy while every paywalled download fails. +# So the choice is yours to make explicitly, once. +# +# Set this for your first `up`, watch the container come up, then DELETE THE +# LINE. Leaving it set disarms the protection permanently, on an instance that +# by then has credentials worth protecting. +# +# BACK UP ./images/secrets/ ALONGSIDE YOUR DATABASE. The key is the only thing +# that can read your stored credentials; a database restored without it needs +# every credential re-entered by hand. +CURATOR_BOOTSTRAP_NEW_KEY=1 + + # --------------------------------------------------------------------------- # Optional — defaults are fine # --------------------------------------------------------------------------- diff --git a/README.md b/README.md index 365032d..2770fd4 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,31 @@ If you forget it, the symptom is a long build instead of a quick pull. The database schema is created automatically on first start — the web container runs its migrations before serving. Nothing to initialise by hand. -A few things are worth knowing about the first few minutes: +**One thing does need a deliberate act, and the app will not start without it.** +FabledCurator encrypts your stored platform credentials with a key it keeps at +`./images/secrets/credential_key.b64`. On a brand-new install that file does not +exist, and rather than quietly creating one the app stops: + +``` +MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64 +``` + +Set `CURATOR_BOOTSTRAP_NEW_KEY=1` in your `.env` for the first `up`, then delete +the line once the container is running. `.env.example` ships it with that +instruction attached. + +The refusal is deliberate, and worth understanding rather than working around: +auto-creating a key is indistinguishable from the disaster case — a restore that +brought the database back but lost `./images/secrets` — where it would mint a key +that cannot decrypt anything, leaving an instance that looks healthy while every +paywalled download fails. Making you say so once, on an empty install, is the +price of that not happening silently later. + +**Which means: back up `./images/secrets/` alongside your database.** It is the +only thing that can read your stored credentials. A database restored without it +needs every credential entered again by hand. + +A few other things are worth knowing about the first few minutes: - **The ML worker downloads its model weights on first boot**, several GB from HuggingFace into `./models`. Until that finishes, tagging is queued rather diff --git a/docker-compose.yml b/docker-compose.yml index fd68189..0bca4f2 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -133,6 +133,20 @@ services: CELERY_RESULT_BACKEND: redis://redis:6379/0 SECRET_KEY: ${SECRET_KEY:-dev_secret_key_not_for_production_change_me} LOG_LEVEL: ${LOG_LEVEL:-INFO} + # First boot only. FabledCurator refuses to start until the credential + # encryption key at /images/secrets/credential_key.b64 exists, and + # refuses to create one unless told to — auto-creating is + # indistinguishable from a restore that lost ./images/secrets, where it + # would mint a key that decrypts nothing and leave an instance that looks + # healthy while every paywalled download fails. + # + # Passed through EXPLICITLY because a variable in `.env` is only used for + # ${...} interpolation; it does not reach the container unless it is + # named here. Defaulted to empty so the refusal stands for everyone who + # has not opted in — the app tests for exactly "1". + # + # Set it in .env for one `up`, then remove it. See .env.example. + CURATOR_BOOTSTRAP_NEW_KEY: ${CURATOR_BOOTSTRAP_NEW_KEY:-} volumes: - ./images:/images - ./import:/import -- 2.54.0 From f630e50e75cb72bf45b3d9fa46a1e6b23ad620f1 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 16:21:27 -0400 Subject: [PATCH 4/6] ci: the refresh publishes only what the gate passed (#3265 milestone step 4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate reported a verdict nothing consulted. Now it decides. The promote moved out of the three build jobs into its own `promote` job, because the verdict cannot exist until build-web has finished and the promote used to run inside it. `needs: [build-web, build-ml, build-agent, smoke-web]` is the whole mechanism: a failed smoke skips the promote, so a refresh that broke something leaves :latest naming the build that works. "The refresh failed" and "production is broken" must not be the same event. A SKIPPED smoke also skips it, and that is the case that matters most. On run 5290 the gate silently skipped itself — job-level `if:` cannot read the env context — and a design where only a FAILED gate blocks would have published unverified images while reporting success. Not running is not the same as passing, and today produced two separate bugs of exactly that shape (#3414, and the smoke-web skip). All three images now promote together or not at all. They are one stack: build.yml already refuses to publish a :dev web image beside a stale :dev ml because the mismatch only surfaces as a runtime failure, and a refresh that published ml while withholding web would be that same trap reached through the gate. Stated plainly in the job comment: the gate covers web only, so ml and agent are held to web's verdict rather than their own. That is the conservative direction, not equivalent evidence, and should not be read as if it were. Three near-identical promote steps collapsed into one loop. A partial failure now says which images moved and that the state is inconsistent, rather than leaving that to be inferred — the promote is idempotent and the candidates are still published, so the instruction is simply to re-run. Also removed the now-dead `promote` output from the ml and agent reuse steps. Only build-web's is read (as outputs.candidate); two more copies nothing consults is the kind of thing that reads as load-bearing a year later. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 367 +++++++++++++---------------------- 1 file changed, 138 insertions(+), 229 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index f0c2424..4e989f7 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -734,10 +734,16 @@ jobs: # already allows for :buildcache, not the per-build tag family that # milestone 318 withdrew. # - # Both values are decided HERE, beside `hit`, for the reason the - # force/schedule branch below gives: one step decides what this job - # does. A promote condition derived independently could disagree with - # the tag the build actually wrote. + # Decided HERE, beside `hit`, for the reason the force/schedule + # branch below gives: one step decides what this job does. A + # condition derived independently could disagree with the tag the + # build actually wrote. + # + # build-web additionally exposes this as `outputs.candidate`, which is + # what gates the `promote` job — a job's `if:` cannot read `env`, and + # one flag is enough because all three derive it from the same + # IS_REFRESH. ml and agent do not re-emit it; a second copy nothing + # reads is the kind of thing that later reads as load-bearing. if [ "${IS_REFRESH:-}" = "true" ]; then echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" echo "promote=true" >> "$GITHUB_OUTPUT" @@ -959,77 +965,6 @@ jobs: FC_CHANNEL=${{ steps.tag.outputs.channel }} FC_VERSION=${{ steps.reuse.outputs.version }} - # Point the channel tag at the candidate the refresh just built. - # - # Unconditional TODAY, so this milestone never leaves the refresh in a - # state where it builds and publishes nothing. Step 4 wraps it in the - # smoke suite's verdict; until then the scheduled path behaves exactly - # as it did, just via two operations instead of one. - # - # NOT `imagetools create`. That wraps its source in an INDEX, and an - # indexed channel tag is the one thing this pipeline cannot survive: - # `.Image.Config.Labels` does not resolve through an index, so the - # fc.revision the reuse check reads off the channel tag would come back - # empty, every subsequent push would miss and rebuild, and nothing would - # go red. That is #3183, observed on run 4751 — reuse worked exactly once - # and the only symptom was the bill. The repoint step below excludes its - # own source tag for precisely this reason; a promote that re-introduced - # the wrap through a different door would undo that care. - # - # A manifest PUT is what "make this tag name that image" means at the - # registry level: the same bytes under the same media type, so the digest - # is identical, the media type is preserved, and no layer moves. - - name: Promote the refresh candidate to the channel - if: steps.reuse.outputs.promote == 'true' - env: - IMAGE: git.fabledsword.com/bvandeusen/fabledcurator - CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }} - TOKEN: ${{ secrets.RELEASE_TOKEN }} - ACTOR: ${{ github.actor }} - run: | - set -eu - REPO=${IMAGE#git.fabledsword.com/} - TAG=${CHANNEL_REF##*:} - - # Registry auth is its own token exchange — the `docker login` above - # authenticates the docker client, not curl. Deadline on every call - # (rule 156): a registry that stops answering must fail this step, - # not hang the weekly refresh until the job times out. - BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ - "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ - | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') - - # Ask for the image manifest media types ONLY. Offering the index - # types too would let the registry hand back an index if one ever - # existed at this tag, and we would faithfully copy the thing we are - # trying not to create. - ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' - CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ - -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ - && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') - test -n "$CT" - SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - echo "promote: candidate is $SRC_DIGEST ($CT)" - - curl -fsS --max-time 120 -X PUT \ - -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ - --data-binary @manifest.json \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" - - # Read it back. A PUT that returned 2xx but landed something else is - # exactly the silent-and-plausible failure this pipeline keeps - # producing, and the check costs one request. - NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ - -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ - | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - if [ "$NOW" != "$SRC_DIGEST" ]; then - echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2 - exit 1 - fi - echo "promote: $IMAGE:$TAG now names $NOW" - # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the # SAME manifest the channel tag holds, so :c- is byte-identical to @@ -1281,6 +1216,114 @@ jobs: echo echo "smoke: all checks passed against $CANDIDATE" + # Move the channel tags — the whole point of the gate. + # + # Lives in its own job because the verdict it depends on cannot exist until + # after build-web has finished, and the promote used to run INSIDE build-web. + # + # `needs` on smoke-web is the gate. A failed smoke skips this job, so a + # refresh that broke something leaves :latest naming the build that works — + # "the refresh failed" and "production is broken" must not be the same event. + # A SKIPPED smoke also skips this job, which is the behaviour that matters + # most: on run 5290 the gate silently skipped itself, and a design where only + # a FAILED gate blocks would have published unverified images while reporting + # success. Not running is not the same as passing. + # + # All three images promote TOGETHER, or none do. They are one stack: build.yml + # already refuses to publish a :dev web image beside a stale :dev ml, because + # the mismatch only shows up as a runtime failure. A refresh that published ml + # and withheld web would be that same trap, arrived at through the gate. + # + # The gate covers the web image only (milestone 362 step 3 scoped it there), + # so ml and agent are being held to web's verdict rather than their own. That + # is deliberate and it is the conservative direction — they ship together, so + # the weakest evidence should govern all three — but it is not the same as + # having smoked them, and it should not be read as if it were. + promote: + needs: [build-web, build-ml, build-agent, smoke-web] + # Only a refresh publishes through a candidate; a push writes its channel + # tag directly from the build. Reads the same reuse-step decision the build + # took, via a job output — a job's `if:` cannot see the `env` context. + if: needs.build-web.outputs.candidate == 'true' + runs-on: python-ci + container: + image: git.fabledsword.com/bvandeusen/ci-python:3.14 + steps: + - name: Point the channel tags at the smoked candidates + env: + TOKEN: ${{ secrets.RELEASE_TOKEN }} + ACTOR: ${{ github.actor }} + run: | + set -eu + # `latest` is not a guess: a refresh always builds `main` (BUILD_REF), + # and the "must have checked out main" guard in every build job fails + # the run if that did not hold. So the channel is main's. + TAG=latest + FAILED="" + + for NAME in fabledcurator fabledcurator-ml fabledcurator-agent; do + REPO="bvandeusen/$NAME" + echo "promote: $REPO" + + # Registry auth is its own token exchange — `docker login` + # authenticates the docker client, not curl. Deadline on every call + # (rule 156): a registry that stops answering must fail this step, + # not hang the weekly refresh until the job times out. + BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ + "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') + + # Ask for the IMAGE manifest media types only. Offering the index + # types too would let the registry hand back an index if one ever + # existed at this tag, and we would faithfully copy the thing this + # whole approach exists to avoid creating. + ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' + CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ + && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') + test -n "$CT" + SRC=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + echo "promote: candidate $SRC ($CT)" + + # NOT `imagetools create`. That wraps its source in an INDEX, and + # `.Image.Config.Labels` does not resolve through one — the + # fc.revision the reuse check reads off the channel tag would come + # back empty, every later push would miss and rebuild, and nothing + # would go red (#3183, run 4751). A manifest PUT is what "make this + # tag name that image" means at the registry: same bytes, same media + # type, same digest, no layer transfer. + curl -fsS --max-time 120 -X PUT \ + -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ + --data-binary @manifest.json \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" + + # Read it back. A PUT that returned 2xx but landed something else is + # exactly the silent-and-plausible failure this pipeline keeps + # producing, and the check costs one request. + NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ + -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ + "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ + | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') + if [ "$NOW" != "$SRC" ]; then + echo "promote: FAILED — $NAME:$TAG is $NOW, expected $SRC" >&2 + FAILED="$FAILED $NAME" + continue + fi + echo "promote: $NAME:$TAG now names $NOW" + done + + if [ -n "$FAILED" ]; then + echo "" >&2 + echo "promote: FAILED for:$FAILED" >&2 + echo "promote: the channel tags are now INCONSISTENT — some images" >&2 + echo "promote: moved and some did not. Re-run this refresh; the" >&2 + echo "promote: candidates are still published and the promote is" >&2 + echo "promote: idempotent." >&2 + exit 1 + fi + echo "promote: all three channel tags moved" + build-ml: runs-on: python-ci container: @@ -1486,16 +1529,20 @@ jobs: # already allows for :buildcache, not the per-build tag family that # milestone 318 withdrew. # - # Both values are decided HERE, beside `hit`, for the reason the - # force/schedule branch below gives: one step decides what this job - # does. A promote condition derived independently could disagree with - # the tag the build actually wrote. + # Decided HERE, beside `hit`, for the reason the force/schedule + # branch below gives: one step decides what this job does. A + # condition derived independently could disagree with the tag the + # build actually wrote. + # + # build-web additionally exposes this as `outputs.candidate`, which is + # what gates the `promote` job — a job's `if:` cannot read `env`, and + # one flag is enough because all three derive it from the same + # IS_REFRESH. ml and agent do not re-emit it; a second copy nothing + # reads is the kind of thing that later reads as load-bearing. if [ "${IS_REFRESH:-}" = "true" ]; then echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" - echo "promote=true" >> "$GITHUB_OUTPUT" else echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" - echo "promote=false" >> "$GITHUB_OUTPUT" fi # Compare VALUES, never exit codes. Measured on buildx v0.36.1 @@ -1629,77 +1676,6 @@ jobs: cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-ml:buildcache,mode=max - # Point the channel tag at the candidate the refresh just built. - # - # Unconditional TODAY, so this milestone never leaves the refresh in a - # state where it builds and publishes nothing. Step 4 wraps it in the - # smoke suite's verdict; until then the scheduled path behaves exactly - # as it did, just via two operations instead of one. - # - # NOT `imagetools create`. That wraps its source in an INDEX, and an - # indexed channel tag is the one thing this pipeline cannot survive: - # `.Image.Config.Labels` does not resolve through an index, so the - # fc.revision the reuse check reads off the channel tag would come back - # empty, every subsequent push would miss and rebuild, and nothing would - # go red. That is #3183, observed on run 4751 — reuse worked exactly once - # and the only symptom was the bill. The repoint step below excludes its - # own source tag for precisely this reason; a promote that re-introduced - # the wrap through a different door would undo that care. - # - # A manifest PUT is what "make this tag name that image" means at the - # registry level: the same bytes under the same media type, so the digest - # is identical, the media type is preserved, and no layer moves. - - name: Promote the refresh candidate to the channel - if: steps.reuse.outputs.promote == 'true' - env: - IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-ml - CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }} - TOKEN: ${{ secrets.RELEASE_TOKEN }} - ACTOR: ${{ github.actor }} - run: | - set -eu - REPO=${IMAGE#git.fabledsword.com/} - TAG=${CHANNEL_REF##*:} - - # Registry auth is its own token exchange — the `docker login` above - # authenticates the docker client, not curl. Deadline on every call - # (rule 156): a registry that stops answering must fail this step, - # not hang the weekly refresh until the job times out. - BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ - "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ - | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') - - # Ask for the image manifest media types ONLY. Offering the index - # types too would let the registry hand back an index if one ever - # existed at this tag, and we would faithfully copy the thing we are - # trying not to create. - ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' - CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ - -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ - && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') - test -n "$CT" - SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - echo "promote: candidate is $SRC_DIGEST ($CT)" - - curl -fsS --max-time 120 -X PUT \ - -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ - --data-binary @manifest.json \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" - - # Read it back. A PUT that returned 2xx but landed something else is - # exactly the silent-and-plausible failure this pipeline keeps - # producing, and the check costs one request. - NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ - -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ - | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - if [ "$NOW" != "$SRC_DIGEST" ]; then - echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2 - exit 1 - fi - echo "promote: $IMAGE:$TAG now names $NOW" - # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the # SAME manifest the channel tag holds, so :c- is byte-identical to @@ -1981,16 +1957,20 @@ jobs: # already allows for :buildcache, not the per-build tag family that # milestone 318 withdrew. # - # Both values are decided HERE, beside `hit`, for the reason the - # force/schedule branch below gives: one step decides what this job - # does. A promote condition derived independently could disagree with - # the tag the build actually wrote. + # Decided HERE, beside `hit`, for the reason the force/schedule + # branch below gives: one step decides what this job does. A + # condition derived independently could disagree with the tag the + # build actually wrote. + # + # build-web additionally exposes this as `outputs.candidate`, which is + # what gates the `promote` job — a job's `if:` cannot read `env`, and + # one flag is enough because all three derive it from the same + # IS_REFRESH. ml and agent do not re-emit it; a second copy nothing + # reads is the kind of thing that later reads as load-bearing. if [ "${IS_REFRESH:-}" = "true" ]; then echo "build_ref=$IMAGE:refresh-candidate" >> "$GITHUB_OUTPUT" - echo "promote=true" >> "$GITHUB_OUTPUT" else echo "build_ref=$IMAGE:$T" >> "$GITHUB_OUTPUT" - echo "promote=false" >> "$GITHUB_OUTPUT" fi # Compare VALUES, never exit codes. Measured on buildx v0.36.1 @@ -2124,77 +2104,6 @@ jobs: cache-from: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache cache-to: type=registry,ref=git.fabledsword.com/bvandeusen/fabledcurator-agent:buildcache,mode=max - # Point the channel tag at the candidate the refresh just built. - # - # Unconditional TODAY, so this milestone never leaves the refresh in a - # state where it builds and publishes nothing. Step 4 wraps it in the - # smoke suite's verdict; until then the scheduled path behaves exactly - # as it did, just via two operations instead of one. - # - # NOT `imagetools create`. That wraps its source in an INDEX, and an - # indexed channel tag is the one thing this pipeline cannot survive: - # `.Image.Config.Labels` does not resolve through an index, so the - # fc.revision the reuse check reads off the channel tag would come back - # empty, every subsequent push would miss and rebuild, and nothing would - # go red. That is #3183, observed on run 4751 — reuse worked exactly once - # and the only symptom was the bill. The repoint step below excludes its - # own source tag for precisely this reason; a promote that re-introduced - # the wrap through a different door would undo that care. - # - # A manifest PUT is what "make this tag name that image" means at the - # registry level: the same bytes under the same media type, so the digest - # is identical, the media type is preserved, and no layer moves. - - name: Promote the refresh candidate to the channel - if: steps.reuse.outputs.promote == 'true' - env: - IMAGE: git.fabledsword.com/bvandeusen/fabledcurator-agent - CHANNEL_REF: ${{ steps.reuse.outputs.channel_ref }} - TOKEN: ${{ secrets.RELEASE_TOKEN }} - ACTOR: ${{ github.actor }} - run: | - set -eu - REPO=${IMAGE#git.fabledsword.com/} - TAG=${CHANNEL_REF##*:} - - # Registry auth is its own token exchange — the `docker login` above - # authenticates the docker client, not curl. Deadline on every call - # (rule 156): a registry that stops answering must fail this step, - # not hang the weekly refresh until the job times out. - BEARER=$(curl -fsS --max-time 30 -u "$ACTOR:$TOKEN" \ - "https://git.fabledsword.com/v2/token?scope=repository:$REPO:pull,push&service=git.fabledsword.com" \ - | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])') - - # Ask for the image manifest media types ONLY. Offering the index - # types too would let the registry hand back an index if one ever - # existed at this tag, and we would faithfully copy the thing we are - # trying not to create. - ACCEPT='application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' - CT=$(curl -fsS --max-time 60 -o manifest.json -D headers.txt \ - -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/refresh-candidate" \ - && tr -d '\r' < headers.txt | awk -F': ' '/^[Cc]ontent-[Tt]ype:/{print $2}') - test -n "$CT" - SRC_DIGEST=$(tr -d '\r' < headers.txt | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - echo "promote: candidate is $SRC_DIGEST ($CT)" - - curl -fsS --max-time 120 -X PUT \ - -H "Authorization: Bearer $BEARER" -H "Content-Type: $CT" \ - --data-binary @manifest.json \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" - - # Read it back. A PUT that returned 2xx but landed something else is - # exactly the silent-and-plausible failure this pipeline keeps - # producing, and the check costs one request. - NOW=$(curl -fsS --max-time 30 -o /dev/null -D - \ - -H "Authorization: Bearer $BEARER" -H "Accept: $ACCEPT" \ - "https://git.fabledsword.com/v2/$REPO/manifests/$TAG" \ - | tr -d '\r' | awk -F': ' '/^[Dd]ocker-[Cc]ontent-[Dd]igest:/{print $2}') - if [ "$NOW" != "$SRC_DIGEST" ]; then - echo "promote: $IMAGE:$TAG is $NOW, expected $SRC_DIGEST" >&2 - exit 1 - fi - echo "promote: $IMAGE:$TAG now names $NOW" - # Every tag but the channel's own is written HERE, registry-side, # whether or not a build ran. Each -t becomes another reference to the # SAME manifest the channel tag holds, so :c- is byte-identical to -- 2.54.0 From 59d27ef76e8123ae43d0327bff08a0ed17313624 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 16:22:01 -0400 Subject: [PATCH 5/6] test: force the smoke gate to fail, to watch it block a publish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TEMPORARY, reverted in the next commit. Milestone 362's verification section requires the gate to be seen rejecting a build — a gate nobody has watched reject anything is a gate nobody knows is wired up. Every real check passes, so the rejection has to be forced. Under test is the job dependency, not the assertions: a failed smoke-web must skip the promote job, and the three :latest tags must still name the digests they named before the run. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index 4e989f7..c2fc0a5 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -1214,6 +1214,18 @@ jobs: fi curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health" echo + + # >>> TEMPORARY — REVERTED IN THE NEXT COMMIT <<< + # Milestone 362's verification says the gate must be watched + # REJECTING a build before anyone can believe it is wired up. Every + # real check above passes right now, so the only way to see a + # rejection is to force one. What is under test here is the job + # dependency, not the assertions: does a failed smoke actually stop + # the promote, and does :latest stay where it was? + echo "smoke: FORCED FAILURE — verifying the gate blocks the promote" >&2 + exit 1 + # >>> END TEMPORARY <<< + echo "smoke: all checks passed against $CANDIDATE" # Move the channel tags — the whole point of the gate. -- 2.54.0 From 131237143be17efd5af5dff0e7bd6ff5bbdada77 Mon Sep 17 00:00:00 2001 From: Bryan Van Deusen Date: Wed, 2 Sep 2026 16:23:49 -0400 Subject: [PATCH 6/6] Revert "test: force the smoke gate to fail, to watch it block a publish" The gate held. Run 5320, dispatched with the forced failure in place: build-web success (candidate published) build-ml success build-agent success smoke-web FAILED promote skipped run failure And the three channel tags did not move: fabledcurator 33d3d8332f74 -> 33d3d8332f74 fabledcurator-ml e94a5435cb45 -> e94a5435cb45 fabledcurator-agent bae27d34d811 -> bae27d34d811 So a refresh that breaks something now leaves :latest naming the build that works, which is the property milestone 362 exists to establish. The rejected candidate is still published under :refresh-candidate, so whoever reads the red job on Monday can pull the exact image that failed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA --- .forgejo/workflows/build.yml | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/.forgejo/workflows/build.yml b/.forgejo/workflows/build.yml index c2fc0a5..b37e0ba 100644 --- a/.forgejo/workflows/build.yml +++ b/.forgejo/workflows/build.yml @@ -1215,17 +1215,6 @@ jobs: curl -fsS --max-time 5 "http://$WEB_IP:8080/api/health" echo - # >>> TEMPORARY — REVERTED IN THE NEXT COMMIT <<< - # Milestone 362's verification says the gate must be watched - # REJECTING a build before anyone can believe it is wired up. Every - # real check above passes right now, so the only way to see a - # rejection is to force one. What is under test here is the job - # dependency, not the assertions: does a failed smoke actually stop - # the promote, and does :latest stay where it was? - echo "smoke: FORCED FAILURE — verifying the gate blocks the promote" >&2 - exit 1 - # >>> END TEMPORARY <<< - echo "smoke: all checks passed against $CANDIDATE" # Move the channel tags — the whole point of the gate. -- 2.54.0