The weekly refresh now publishes only what a gate has proven — and a new install can start #249

Merged

bvandeusen merged 6 commits from dev into main 2026-09-02 16:31:56 -04:00

Conversation 0 Commits 6 Files Changed 4

+244 -234

bvandeusen commented 2026-09-02 16:31:50 -04:00

Owner

Copy Link

Copy Source

Milestone 362 steps 3 and 4, and the first-run blocker the gate found.

86abaf0 — a new install could not start (#3422)

The install path milestone 328 just finished writing produces a web container that exits on boot:

MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64.
For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1.

That variable appeared in no README, no .env.example and no compose file — only in backend/. So a stranger following the documented steps got an app that does not start and an error with no context. A harder blocker than #3367: not a missing button, the app does not run.

The product behaviour is right and unchanged. credential_crypto refuses to mint a key because the 2026-06-02 audit found a partial restore — database back, ./images/secrets lost — silently generating one and producing a healthy-looking instance where every authenticated download failed AUTH_ERROR. Failing fast is correct; not saying so was the bug.

compose had to change too, and this is the part that nearly shipped a second broken instruction. A variable in .env is only used for ${...} interpolation — it does not reach the container unless the service names it. Added to the shared app_env anchor, defaulted to empty so the refusal stands for everyone who has not opted in.

Not taken: auto-bootstrapping when the credential table is empty. Better product, code change, predicate has to be exactly right. #3422 stays open for it.

bfa9fd6 … 4815040 — the smoke gate (step 3)

ci.yml cannot gate a base refresh: its lanes run on ci-python:3.14 and install requirements.txt, and a refresh changes neither. What it re-resolves is the Dockerfile's unpinned apt layer, which nothing else in the repo looks at.

smoke-web runs the candidate image against real pgvector and redis: alembic upgrade head from empty, the apt binaries, the application's own Thumbnailer (JPEG, PNG-with-alpha, WebP, an ffmpeg frame), and /api/health. Every failure names the package it implicates.

It found the credential-key blocker on its first real run, which is the gate justifying itself. Three defects of my own surfaced the same way — smoke-web skipping itself entirely (job-level if: cannot read env), a check asserting a condition no fresh container can satisfy, and a health loop polling a dead container for 3m35s because docker had recycled its IP.

f630e50 — the gate decides (step 4)

The promote moved into its own job, gated on needs: [build-web, build-ml, build-agent, smoke-web].

All three images promote together or not at all. They are one stack — build.yml already refuses to pair a fresh :dev web with a stale :dev ml — so a refresh publishing ml while withholding web would be that trap reached through the gate. The gate covers web only, so ml and agent are held to web's verdict rather than their own: conservative, and stated as such rather than left to look like equivalent evidence.

A skipped gate blocks too. needs treats skipped as not-satisfied. Run 5290's gate skipped itself silently, and a design where only a FAILED gate blocks would have published unverified images while reporting success.

Verification — both directions

Red (run 5320), with a forced failure: smoke-web failed, promote skipped, and the three channel tags did not move —

fabledcurator        33d3d8332f74 -> 33d3d8332f74
fabledcurator-ml     e94a5435cb45 -> e94a5435cb45
fabledcurator-agent  bae27d34d811 -> bae27d34d811

The rejected candidate stayed under :refresh-candidate, so the exact failing image is pullable.

Green (run 5326), after reverting: promote: all three channel tags moved, all plain image manifests, :latest still carrying fc.revision — #3183's failure mode did not occur through the new path. The promoted digests were byte-identical to the baseline, because SOURCE_DATE_EPOCH made the rebuild reproducible: the promote wrote the digest it found, which is what a refresh that changed nothing should be.

CI green on 1312371.

One pattern worth naming

Three separate bugs today were something evaluating false while the run reported success — the boolean dispatch input, the job-level env context, and the gate skipping itself. On this pipeline "green" and "ran" are different claims, and the step conclusions are the only place that difference shows.

Prod runs d01f33d and is unaffected until the next deploy.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

Milestone 362 steps 3 and 4, and the first-run blocker the gate found. ## `86abaf0` — a new install could not start (#3422) The install path milestone 328 just finished writing produces a web container that **exits on boot**: ``` MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. ``` That variable appeared in no README, no `.env.example` and no compose file — only in `backend/`. So a stranger following the documented steps got an app that does not start and an error with no context. A harder blocker than #3367: not a missing button, the app does not run. The product behaviour is right and unchanged. `credential_crypto` refuses to mint a key because the 2026-06-02 audit found a partial restore — database back, `./images/secrets` lost — silently generating one and producing a healthy-looking instance where every authenticated download failed `AUTH_ERROR`. Failing fast is correct; not saying so was the bug. **compose had to change too, and this is the part that nearly shipped a second broken instruction.** A variable in `.env` is only used for `${...}` interpolation — it does not reach the container unless the service names it. Added to the shared `app_env` anchor, defaulted to empty so the refusal stands for everyone who has not opted in. Not taken: auto-bootstrapping when the credential table is empty. Better product, code change, predicate has to be exactly right. #3422 stays open for it. ## `bfa9fd6` … `4815040` — the smoke gate (step 3) `ci.yml` cannot gate a base refresh: its lanes run on `ci-python:3.14` and install `requirements.txt`, and a refresh changes neither. What it re-resolves is the Dockerfile's unpinned apt layer, which nothing else in the repo looks at. `smoke-web` runs the **candidate image** against real pgvector and redis: `alembic upgrade head` from empty, the apt binaries, the application's own `Thumbnailer` (JPEG, PNG-with-alpha, WebP, an ffmpeg frame), and `/api/health`. Every failure names the package it implicates. It found the credential-key blocker on its first real run, which is the gate justifying itself. Three defects of my own surfaced the same way — `smoke-web` skipping itself entirely (job-level `if:` cannot read `env`), a check asserting a condition no fresh container can satisfy, and a health loop polling a dead container for 3m35s because docker had recycled its IP. ## `f630e50` — the gate decides (step 4) The promote moved into its own job, gated on `needs: [build-web, build-ml, build-agent, smoke-web]`. **All three images promote together or not at all.** They are one stack — `build.yml` already refuses to pair a fresh `:dev` web with a stale `:dev` ml — so a refresh publishing ml while withholding web would be that trap reached through the gate. The gate covers web only, so ml and agent are held to web's verdict rather than their own: conservative, and stated as such rather than left to look like equivalent evidence. **A skipped gate blocks too.** `needs` treats skipped as not-satisfied. Run 5290's gate skipped itself silently, and a design where only a FAILED gate blocks would have published unverified images while reporting success. ## Verification — both directions **Red (run 5320)**, with a forced failure: `smoke-web` failed, `promote` skipped, and the three channel tags did not move — ``` fabledcurator 33d3d8332f74 -> 33d3d8332f74 fabledcurator-ml e94a5435cb45 -> e94a5435cb45 fabledcurator-agent bae27d34d811 -> bae27d34d811 ``` The rejected candidate stayed under `:refresh-candidate`, so the exact failing image is pullable. **Green (run 5326)**, after reverting: `promote: all three channel tags moved`, all plain image manifests, `:latest` still carrying `fc.revision` — #3183's failure mode did not occur through the new path. The promoted digests were byte-identical to the baseline, because `SOURCE_DATE_EPOCH` made the rebuild reproducible: the promote wrote the digest it found, which is what a refresh that changed nothing should be. CI green on `1312371`. ## One pattern worth naming Three separate bugs today were something evaluating false while the run reported success — the boolean dispatch input, the job-level `env` context, and the gate skipping itself. On this pipeline **"green" and "ran" are different claims**, and the step conclusions are the only place that difference shows. Prod runs `d01f33d` and is unaffected until the next deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

bvandeusen added 6 commits 2026-09-02 16:31:50 -04:00

ci: smoke-web never ran — a job's if: cannot read the env context ... 81b7b6f308

Run 5290 dispatched a refresh. Everything worked: the guard fired, the build
published the candidate, the promote pointed :latest at it. And `smoke-web`
reported conclusion "skipped", with no steps and no log.

Its condition was `if: env.IS_REFRESH == 'true'`. The env context is available
to STEP conditions and step bodies but never to a job's own `if:`, and an
unresolvable context there evaluates to empty rather than erroring. So the
gate skipped itself, silently, on the one run that existed to exercise it.

Second silent-skip of this family today, after #3414. Same shape both times:
something evaluated false, nothing failed, and the run reported success. It is
worth naming the pattern — on this pipeline, "green" and "ran" are different
claims, and the steps' own conclusions are the only place the difference shows.

Fixed by keying off a job output rather than re-deriving the trigger:
build-web now exposes the reuse step's `promote` decision as `outputs.candidate`
and smoke-web consumes it. That is better than duplicating the expression:
it is the same single decision the build, the XPI download and the promote all
take already — build.yml's own "one decision drives everything downstream" —
and it asserts the thing smoke-web actually depends on, that a candidate was
published, rather than restating the reason one would be.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

ci: the smoke gate found a real one on its first run — and had two bugs of its own ... 4815040d74

Run 5296 was `smoke-web`'s first genuine execution. Checks 1 and 2 passed:
alembic built the schema from empty inside the image, all five apt binaries
resolved, and the application's own Thumbnailer produced JPEG, PNG-with-alpha,
WebP and an ffmpeg video frame against the image's libraries. Check 3 failed,
and the trap's log dump said exactly why:

  MissingCredentialKey: Fernet key file not found at
  /images/secrets/credential_key.b64. For first-time setup, set
  CURATOR_BOOTSTRAP_NEW_KEY=1.

That is the product being right. credential_crypto refuses to mint a key
unless someone opts in, because the 2026-06-02 audit found a partial restore
(DB back, /images/secrets/ lost) silently generating a fresh one and leaving a
working-looking system where every authenticated download failed AUTH_ERROR.

It is also a first-run blocker for milestone 328, filed as #3422: the variable
appears in no README, no .env.example and no compose file, so the install path
that milestone just finished writing produces a container that exits on boot.
Not fixed here — the fix trades safety against friction and is the operator's
call.

Two defects in the gate itself, both surfaced by the same run:

- A throwaway CI instance IS first-time setup, so it now passes
  CURATOR_BOOTSTRAP_NEW_KEY=1. The check was asserting a condition no fresh
  container can satisfy.

- The health loop polled a dead container for 3m35s. Docker had already
  recycled its IP, so the replies were a baffling mix of connection-refused
  and 5s timeouts from whatever took the address next. It now checks
  `.State.Running` each iteration and fails immediately with the container's
  log. The trap had the real answer the whole time; this stops burying it
  under four minutes of noise.

Also corrected a message claiming a 120s budget: 60 iterations of up to 5s
connect plus 2s sleep is nearer seven minutes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

docs: a new install could not start, and nothing told anyone why (#3422) ... 86abaf0b94

The install path milestone 328 wrote produces a web container that exits on
boot. entrypoint.sh runs alembic, then app construction raises:

  MissingCredentialKey: Fernet key file not found at
  /images/secrets/credential_key.b64. For first-time setup, set
  CURATOR_BOOTSTRAP_NEW_KEY=1.

That variable appeared in no README, no .env.example and no compose file —
only in backend/. So a stranger following the documented steps got an app
that does not start and an error with no context. Found by the milestone-362
smoke gate on its first real run (#3422).

The product behaviour stays exactly as it is. credential_crypto refuses to
mint a key because the 2026-06-02 audit found a partial restore — database
back, ./images/secrets lost — silently generating a fresh one and producing a
healthy-looking instance where every authenticated download failed AUTH_ERROR.
Failing fast is right; not saying so is the bug.

So: .env.example carries the variable in its own FIRST BOOT ONLY section with
the reasoning and an instruction to delete the line afterwards, and README's
First run leads with it, because "the app will not start" belongs before "the
ML worker downloads weights". Both say to back up ./images/secrets/ alongside
the database, which is the part that costs real data if it is learned late.

**compose had to change too, and this is the part that would have shipped a
second broken instruction.** A variable in `.env` is only used for ${...}
interpolation — it does not reach the container unless the service names it.
Telling people to set it in .env, without that, would have documented a step
that does nothing. Added to the shared app_env anchor, defaulted to empty so
the refusal still stands for everyone who has not opted in.

Not taken: auto-bootstrapping when the credential table is empty, which would
remove the manual step entirely and keep the audit's protection for restores.
That is the better product and it is a code change with a predicate that has
to be exactly right; this is the smallest correct fix, and #3422 stays open
for the other one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

ci: the refresh publishes only what the gate passed (#3265 milestone step 4) ... f630e50e75

The gate reported a verdict nothing consulted. Now it decides.

The promote moved out of the three build jobs into its own `promote` job,
because the verdict cannot exist until build-web has finished and the promote
used to run inside it. `needs: [build-web, build-ml, build-agent, smoke-web]`
is the whole mechanism: a failed smoke skips the promote, so a refresh that
broke something leaves :latest naming the build that works. "The refresh
failed" and "production is broken" must not be the same event.

A SKIPPED smoke also skips it, and that is the case that matters most. On run
5290 the gate silently skipped itself — job-level `if:` cannot read the env
context — and a design where only a FAILED gate blocks would have published
unverified images while reporting success. Not running is not the same as
passing, and today produced two separate bugs of exactly that shape (#3414,
and the smoke-web skip).

All three images now promote together or not at all. They are one stack:
build.yml already refuses to publish a :dev web image beside a stale :dev ml
because the mismatch only surfaces as a runtime failure, and a refresh that
published ml while withholding web would be that same trap reached through the
gate. Stated plainly in the job comment: the gate covers web only, so ml and
agent are held to web's verdict rather than their own. That is the
conservative direction, not equivalent evidence, and should not be read as if
it were.

Three near-identical promote steps collapsed into one loop. A partial failure
now says which images moved and that the state is inconsistent, rather than
leaving that to be inferred — the promote is idempotent and the candidates are
still published, so the instruction is simply to re-run.

Also removed the now-dead `promote` output from the ml and agent reuse steps.
Only build-web's is read (as outputs.candidate); two more copies nothing
consults is the kind of thing that reads as load-bearing a year later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

test: force the smoke gate to fail, to watch it block a publish ... 59d27ef76e

TEMPORARY, reverted in the next commit. Milestone 362's verification section
requires the gate to be seen rejecting a build — a gate nobody has watched
reject anything is a gate nobody knows is wired up. Every real check passes,
so the rejection has to be forced.

Under test is the job dependency, not the assertions: a failed smoke-web must
skip the promote job, and the three :latest tags must still name the digests
they named before the run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

Revert "test: force the smoke gate to fail, to watch it block a publish" ... 131237143b

The gate held. Run 5320, dispatched with the forced failure in place:

  build-web    success   (candidate published)
  build-ml     success
  build-agent  success
  smoke-web    FAILED
  promote      skipped
  run          failure

And the three channel tags did not move:

  fabledcurator        33d3d8332f74 -> 33d3d8332f74
  fabledcurator-ml     e94a5435cb45 -> e94a5435cb45
  fabledcurator-agent  bae27d34d811 -> bae27d34d811

So a refresh that breaks something now leaves :latest naming the build that
works, which is the property milestone 362 exists to establish. The rejected
candidate is still published under :refresh-candidate, so whoever reads the
red job on Monday can pull the exact image that failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA

bvandeusen merged commit 0421fd3109 into main 2026-09-02 16:31:56 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate-bot bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#249