The weekly refresh now publishes only what a gate has proven — and a new install can start #249

Merged

bvandeusen merged 6 commits from dev into main 2026-09-02 16:31:56 -04:00

Conversation 0 Commits 6 Files Changed 4

+244 -234

6 Commits

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	131237143b	Revert "test: force the smoke gate to fail, to watch it block a publish" ... The gate held. Run 5320, dispatched with the forced failure in place: build-web success (candidate published) build-ml success build-agent success smoke-web FAILED promote skipped run failure And the three channel tags did not move: fabledcurator 33d3d8332f74 -> 33d3d8332f74 fabledcurator-ml e94a5435cb45 -> e94a5435cb45 fabledcurator-agent bae27d34d811 -> bae27d34d811 So a refresh that breaks something now leaves :latest naming the build that works, which is the property milestone 362 exists to establish. The rejected candidate is still published under :refresh-candidate, so whoever reads the red job on Monday can pull the exact image that failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:23:49 -04:00
bvandeusenandClaude Opus 5	59d27ef76e	test: force the smoke gate to fail, to watch it block a publish ... TEMPORARY, reverted in the next commit. Milestone 362's verification section requires the gate to be seen rejecting a build — a gate nobody has watched reject anything is a gate nobody knows is wired up. Every real check passes, so the rejection has to be forced. Under test is the job dependency, not the assertions: a failed smoke-web must skip the promote job, and the three :latest tags must still name the digests they named before the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:22:01 -04:00
bvandeusenandClaude Opus 5	f630e50e75	ci: the refresh publishes only what the gate passed (#3265 milestone step 4) ... The gate reported a verdict nothing consulted. Now it decides. The promote moved out of the three build jobs into its own `promote` job, because the verdict cannot exist until build-web has finished and the promote used to run inside it. `needs: [build-web, build-ml, build-agent, smoke-web]` is the whole mechanism: a failed smoke skips the promote, so a refresh that broke something leaves :latest naming the build that works. "The refresh failed" and "production is broken" must not be the same event. A SKIPPED smoke also skips it, and that is the case that matters most. On run 5290 the gate silently skipped itself — job-level `if:` cannot read the env context — and a design where only a FAILED gate blocks would have published unverified images while reporting success. Not running is not the same as passing, and today produced two separate bugs of exactly that shape (#3414, and the smoke-web skip). All three images now promote together or not at all. They are one stack: build.yml already refuses to publish a :dev web image beside a stale :dev ml because the mismatch only surfaces as a runtime failure, and a refresh that published ml while withholding web would be that same trap reached through the gate. Stated plainly in the job comment: the gate covers web only, so ml and agent are held to web's verdict rather than their own. That is the conservative direction, not equivalent evidence, and should not be read as if it were. Three near-identical promote steps collapsed into one loop. A partial failure now says which images moved and that the state is inconsistent, rather than leaving that to be inferred — the promote is idempotent and the candidates are still published, so the instruction is simply to re-run. Also removed the now-dead `promote` output from the ml and agent reuse steps. Only build-web's is read (as outputs.candidate); two more copies nothing consults is the kind of thing that reads as load-bearing a year later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:21:27 -04:00
bvandeusenandClaude Opus 5	86abaf0b94	docs: a new install could not start, and nothing told anyone why (#3422) ... The install path milestone 328 wrote produces a web container that exits on boot. entrypoint.sh runs alembic, then app construction raises: MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. That variable appeared in no README, no .env.example and no compose file — only in backend/. So a stranger following the documented steps got an app that does not start and an error with no context. Found by the milestone-362 smoke gate on its first real run (#3422). The product behaviour stays exactly as it is. credential_crypto refuses to mint a key because the 2026-06-02 audit found a partial restore — database back, ./images/secrets lost — silently generating a fresh one and producing a healthy-looking instance where every authenticated download failed AUTH_ERROR. Failing fast is right; not saying so is the bug. So: .env.example carries the variable in its own FIRST BOOT ONLY section with the reasoning and an instruction to delete the line afterwards, and README's First run leads with it, because "the app will not start" belongs before "the ML worker downloads weights". Both say to back up ./images/secrets/ alongside the database, which is the part that costs real data if it is learned late. **compose had to change too, and this is the part that would have shipped a second broken instruction.** A variable in `.env` is only used for ${...} interpolation — it does not reach the container unless the service names it. Telling people to set it in .env, without that, would have documented a step that does nothing. Added to the shared app_env anchor, defaulted to empty so the refusal still stands for everyone who has not opted in. Not taken: auto-bootstrapping when the credential table is empty, which would remove the manual step entirely and keep the audit's protection for restores. That is the better product and it is a code change with a predicate that has to be exactly right; this is the smallest correct fix, and #3422 stays open for the other one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:19:28 -04:00
bvandeusenandClaude Opus 5	4815040d74	ci: the smoke gate found a real one on its first run — and had two bugs of its own ... Run 5296 was `smoke-web`'s first genuine execution. Checks 1 and 2 passed: alembic built the schema from empty inside the image, all five apt binaries resolved, and the application's own Thumbnailer produced JPEG, PNG-with-alpha, WebP and an ffmpeg video frame against the image's libraries. Check 3 failed, and the trap's log dump said exactly why: MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. That is the product being right. credential_crypto refuses to mint a key unless someone opts in, because the 2026-06-02 audit found a partial restore (DB back, /images/secrets/ lost) silently generating a fresh one and leaving a working-looking system where every authenticated download failed AUTH_ERROR. It is also a first-run blocker for milestone 328, filed as #3422: the variable appears in no README, no .env.example and no compose file, so the install path that milestone just finished writing produces a container that exits on boot. Not fixed here — the fix trades safety against friction and is the operator's call. Two defects in the gate itself, both surfaced by the same run: - A throwaway CI instance IS first-time setup, so it now passes CURATOR_BOOTSTRAP_NEW_KEY=1. The check was asserting a condition no fresh container can satisfy. - The health loop polled a dead container for 3m35s. Docker had already recycled its IP, so the replies were a baffling mix of connection-refused and 5s timeouts from whatever took the address next. It now checks `.State.Running` each iteration and fails immediately with the container's log. The trap had the real answer the whole time; this stops burying it under four minutes of noise. Also corrected a message claiming a 120s budget: 60 iterations of up to 5s connect plus 2s sleep is nearer seven minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:00:18 -04:00
bvandeusenandClaude Opus 5	81b7b6f308	ci: smoke-web never ran — a job's if: cannot read the env context ... Run 5290 dispatched a refresh. Everything worked: the guard fired, the build published the candidate, the promote pointed :latest at it. And `smoke-web` reported conclusion "skipped", with no steps and no log. Its condition was `if: env.IS_REFRESH == 'true'`. The env context is available to STEP conditions and step bodies but never to a job's own `if:`, and an unresolvable context there evaluates to empty rather than erroring. So the gate skipped itself, silently, on the one run that existed to exercise it. Second silent-skip of this family today, after #3414. Same shape both times: something evaluated false, nothing failed, and the run reported success. It is worth naming the pattern — on this pipeline, "green" and "ran" are different claims, and the steps' own conclusions are the only place the difference shows. Fixed by keying off a job output rather than re-deriving the trigger: build-web now exposes the reuse step's `promote` decision as `outputs.candidate` and smoke-web consumes it. That is better than duplicating the expression: it is the same single decision the build, the XPI download and the promote all take already — build.yml's own "one decision drives everything downstream" — and it asserts the thing smoke-web actually depends on, that a candidate was published, rather than restating the reason one would be. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 15:53:40 -04:00