47 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	131237143b	Revert "test: force the smoke gate to fail, to watch it block a publish" ... The gate held. Run 5320, dispatched with the forced failure in place: build-web success (candidate published) build-ml success build-agent success smoke-web FAILED promote skipped run failure And the three channel tags did not move: fabledcurator 33d3d8332f74 -> 33d3d8332f74 fabledcurator-ml e94a5435cb45 -> e94a5435cb45 fabledcurator-agent bae27d34d811 -> bae27d34d811 So a refresh that breaks something now leaves :latest naming the build that works, which is the property milestone 362 exists to establish. The rejected candidate is still published under :refresh-candidate, so whoever reads the red job on Monday can pull the exact image that failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:23:49 -04:00
bvandeusenandClaude Opus 5	59d27ef76e	test: force the smoke gate to fail, to watch it block a publish ... TEMPORARY, reverted in the next commit. Milestone 362's verification section requires the gate to be seen rejecting a build — a gate nobody has watched reject anything is a gate nobody knows is wired up. Every real check passes, so the rejection has to be forced. Under test is the job dependency, not the assertions: a failed smoke-web must skip the promote job, and the three :latest tags must still name the digests they named before the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:22:01 -04:00
bvandeusenandClaude Opus 5	f630e50e75	ci: the refresh publishes only what the gate passed (#3265 milestone step 4) ... The gate reported a verdict nothing consulted. Now it decides. The promote moved out of the three build jobs into its own `promote` job, because the verdict cannot exist until build-web has finished and the promote used to run inside it. `needs: [build-web, build-ml, build-agent, smoke-web]` is the whole mechanism: a failed smoke skips the promote, so a refresh that broke something leaves :latest naming the build that works. "The refresh failed" and "production is broken" must not be the same event. A SKIPPED smoke also skips it, and that is the case that matters most. On run 5290 the gate silently skipped itself — job-level `if:` cannot read the env context — and a design where only a FAILED gate blocks would have published unverified images while reporting success. Not running is not the same as passing, and today produced two separate bugs of exactly that shape (#3414, and the smoke-web skip). All three images now promote together or not at all. They are one stack: build.yml already refuses to publish a :dev web image beside a stale :dev ml because the mismatch only surfaces as a runtime failure, and a refresh that published ml while withholding web would be that same trap reached through the gate. Stated plainly in the job comment: the gate covers web only, so ml and agent are held to web's verdict rather than their own. That is the conservative direction, not equivalent evidence, and should not be read as if it were. Three near-identical promote steps collapsed into one loop. A partial failure now says which images moved and that the state is inconsistent, rather than leaving that to be inferred — the promote is idempotent and the candidates are still published, so the instruction is simply to re-run. Also removed the now-dead `promote` output from the ml and agent reuse steps. Only build-web's is read (as outputs.candidate); two more copies nothing consults is the kind of thing that reads as load-bearing a year later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:21:27 -04:00
bvandeusenandClaude Opus 5	4815040d74	ci: the smoke gate found a real one on its first run — and had two bugs of its own ... Run 5296 was `smoke-web`'s first genuine execution. Checks 1 and 2 passed: alembic built the schema from empty inside the image, all five apt binaries resolved, and the application's own Thumbnailer produced JPEG, PNG-with-alpha, WebP and an ffmpeg video frame against the image's libraries. Check 3 failed, and the trap's log dump said exactly why: MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. That is the product being right. credential_crypto refuses to mint a key unless someone opts in, because the 2026-06-02 audit found a partial restore (DB back, /images/secrets/ lost) silently generating a fresh one and leaving a working-looking system where every authenticated download failed AUTH_ERROR. It is also a first-run blocker for milestone 328, filed as #3422: the variable appears in no README, no .env.example and no compose file, so the install path that milestone just finished writing produces a container that exits on boot. Not fixed here — the fix trades safety against friction and is the operator's call. Two defects in the gate itself, both surfaced by the same run: - A throwaway CI instance IS first-time setup, so it now passes CURATOR_BOOTSTRAP_NEW_KEY=1. The check was asserting a condition no fresh container can satisfy. - The health loop polled a dead container for 3m35s. Docker had already recycled its IP, so the replies were a baffling mix of connection-refused and 5s timeouts from whatever took the address next. It now checks `.State.Running` each iteration and fails immediately with the container's log. The trap had the real answer the whole time; this stops burying it under four minutes of noise. Also corrected a message claiming a 120s budget: 60 iterations of up to 5s connect plus 2s sleep is nearer seven minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:00:18 -04:00
bvandeusenandClaude Opus 5	81b7b6f308	ci: smoke-web never ran — a job's if: cannot read the env context ... Run 5290 dispatched a refresh. Everything worked: the guard fired, the build published the candidate, the promote pointed :latest at it. And `smoke-web` reported conclusion "skipped", with no steps and no log. Its condition was `if: env.IS_REFRESH == 'true'`. The env context is available to STEP conditions and step bodies but never to a job's own `if:`, and an unresolvable context there evaluates to empty rather than erroring. So the gate skipped itself, silently, on the one run that existed to exercise it. Second silent-skip of this family today, after #3414. Same shape both times: something evaluated false, nothing failed, and the run reported success. It is worth naming the pattern — on this pipeline, "green" and "ran" are different claims, and the steps' own conclusions are the only place the difference shows. Fixed by keying off a job output rather than re-deriving the trigger: build-web now exposes the reuse step's `promote` decision as `outputs.candidate` and smoke-web consumes it. That is better than duplicating the expression: it is the same single decision the build, the XPI download and the promote all take already — build.yml's own "one decision drives everything downstream" — and it asserts the thing smoke-web actually depends on, that a candidate was published, rather than restating the reason one would be. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 15:53:40 -04:00
bvandeusenandClaude Opus 5	bfa9fd678b	ci: smoke the refreshed image against real Postgres and Redis ... Milestone 362 step 3. This is the gate the weekly base refresh never had. `ci.yml` cannot be that gate, and the reason matters more than the fix. Its lanes run on ci-python:3.14 and install requirements.txt — a base refresh changes neither, so all five stay green through a bump that breaks the product. What a refresh re-resolves is the Dockerfile's apt layer: ffmpeg unar libpq5 postgresql-client zstd megatools libjpeg62-turbo libwebp7 libpng16-16 ca-certificates Unpinned, every build, and nothing else in this repo looks at it. That line is the dependency creep; it is also precisely what the test suite structurally cannot observe, since the suite never runs inside the image and the image carries no tests and no pytest. So `smoke-web` runs the CANDIDATE IMAGE against real service containers: 1. `alembic upgrade head` on an empty database — the image's own libpq and psycopg, and the same call entrypoint.sh makes before it serves anything, so a failure here is a failure to boot. 2. The apt binaries, then the application's own `Thumbnailer` — JPEG, PNG with alpha, WebP, and a video frame through ffmpeg. `Thumbnailer` needs no database and no app context, so the check exercises real product code rather than a proxy for it. `ffmpeg -version` exiting 0 would pass while a codec removal broke every thumbnail in the library. 3. The web role boots and answers /api/health. Every failure names the package it implicates. This fires on a Sunday, unattended, about a change nobody made deliberately — "assertion failed" a week later teaches nobody anything. The script is piped over stdin rather than bind-mounted: the workspace is a docker volume belonging to the job's own container, so a host bind of $PWD does not resolve for a sibling. Container logs are dumped only on failure, and the trap re-exits with the real status rather than the status of `docker rm`. Deliberately NOT gating the promote yet — that is step 4. Landing the gate and the thing it gates together would mean the first time anyone saw this job run would also be the first time it could stop a publish. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 15:45:10 -04:00
bvandeusenandClaude Opus 5	24a2b70a5a	ci: a boolean input never equals the string 'true' ... The refresh lever did not work, and the way it did not work is the point. Run 5270 dispatched with refresh=true. Its log: expression '(github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'true' || 'false'' evaluated to '%!t(string=false)' trigger: event=workflow_dispatch IS_REFRESH='false' BUILD_REF='refs/heads/dev' trigger: raw inputs refresh='true' force_build='false' The input arrived as true and the comparison still said false. `type: boolean` delivers a real boolean, and GitHub expression semantics cast operands to numbers when their types differ — so `true == 'true'` compares 1 against NaN. My comment on the previous commit asserted the opposite, that Forgejo delivers inputs as strings, and asserted it without checking. The run went GREEN with every step skipped, because a refresh that evaluates false is indistinguishable from an ordinary push. A lever that silently does nothing is worse than no lever: it would have been trusted. Normalised through format(), which is representation-independent — a boolean true and a string 'true' both render 'true'. That is also why force_build was never bitten: it passes its raw value into an env var and compares in the shell, where everything is a string already. format() buys the same thing at expression level, which is where a step `if:` needs the answer. The diagnostic from the previous commit stays. It is what turned this from a guess into a measurement, and it is the only thing that would catch the same class of failure next time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:59:48 -04:00
bvandeusenandClaude Opus 5	2c88ad3efb	ci: report the raw and normalised trigger values ... The refresh dispatch on run 5265 went green with every step skipped: the main-only guard did not fire, checkout took dev, and the reuse step read IS_REFRESH as false. So both workflow-level expressions evaluated false while the identical accessor works for force_build, which compares its value in the shell rather than in an expression. That is a guess until it is measured, and the failure is silent by construction — a refresh that evaluates false behaves exactly like an ordinary push and reports success. This prints the raw input beside the normalised value in the step that already exists to say what a run derived, so the two disagreeing is visible rather than inferred. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:58:20 -04:00
bvandeusenandClaude Opus 5	bfc4f9cec9	ci: one fact for "is this a base refresh", and a lever to trigger one ... Milestone 362, enabling step 2's verification and everything after it. The weekly refresh was testable once a week. That is not a cadence anything can be developed against, and milestone 362's whole point is a gate — which has to be watched rejecting something before anyone can believe it is wired up. So `refresh` joins `force_build` as a dispatch input, on the same reasoning that added that one (#3252: confirm #3190 was gone rather than wait for it to recur). Adding it meant confronting that "is this a refresh?" was asked in five places and spelled five ways: `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in one shell, an `EVENT:` env passed into another, and a bare expression on `pull:`. Five spellings of one fact is how half of them come to disagree once somebody adds a sixth trigger — which is precisely what this commit is. So it is derived once at the top, next to BUILD_REF, which already exists for exactly this reason on exactly this question. String comparison, not boolean: Forgejo delivers dispatch inputs as strings, so `inputs.refresh` is 'true'/'false' and `&&` on it would read the string 'false' as truthy. **A constraint this makes visible, which pre-dates it.** A refresh checks out `main` (BUILD_REF) while running the workflow definition from the branch that triggered it — the cron registers from the default branch. So dev's workflow builds main's source, and dev's workflow cannot depend on anything main's tree does not have yet. It does now: the reuse step calls `artifacts.sh epoch`, which lands on main with this batch. Until then a refresh dispatch fails loudly at that call, which is the right failure — the alternative is tolerating a missing epoch and silently rebuilding #3265 into every refresh. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:41:33 -04:00
bvandeusenandClaude Opus 5	b590d25f8f	ci: the scheduled refresh builds a candidate, then names the channel ... Milestone 362 step 2. Structural: it creates a moment between "built" and "published" for step 3's gate to occupy. No behaviour change. A refresh rebuilds against freshly resolved base images, and the web image's runtime is a line of UNPINNED Debian packages — ffmpeg, libjpeg62-turbo, libpq5, megatools — re-resolved on every build. Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and install requirements.txt, and a base bump changes neither. So refreshed bytes need proving before :latest names them, and proving needs somewhere to stand. On a push nothing changes: build_ref IS channel_ref, promote is false, and the build writes the channel tag directly the way it always has. On the schedule the build writes :refresh-candidate — one moving ref per image, overwritten in place, holding a build nobody is told to pull. That is the shape rule 145 already allows for :buildcache, not the per-build tag family 318 withdrew. Both values are decided in the reuse step beside `hit`, because that step already owns "what does this job do" (build.yml's own rule, at the force branch). A promote condition derived somewhere else could disagree with the tag the build actually wrote. **The promote is a manifest PUT, not `imagetools create`.** That distinction is the whole risk in this change. `imagetools create` wraps its source in an index, and an indexed channel tag is the one thing this pipeline cannot survive: `.Image.Config.Labels` does not resolve through an index, so the fc.revision the reuse check reads back would come up empty, every later push would miss and rebuild, and nothing would go red. That is #3183, observed on run 4751 — reuse worked exactly once and the only symptom was the bill. The repoint step already excludes its own source tag for this reason; a promote that re-introduced the wrap through another door would undo that care. A manifest PUT is what "make this tag name that image" means at the registry: same bytes, same media type, identical digest, no layer transfer. It reads the result back and fails if the tag does not name what was just written — a PUT that 2xx'd and landed something else is exactly the silent-and-plausible failure this pipeline keeps producing. Every call carries a deadline (rule 156); a registry that stops answering must fail the step, not hang the weekly refresh until the job times out. Promote is UNCONDITIONAL today, deliberately. Gating it before the gate exists would leave the refresh building something and publishing nothing for as long as this milestone takes. Step 4 wraps it in the smoke suite's verdict. Not yet verified on the refresh path — that needs a scheduled run, and the lever to trigger one on demand is the next commit. This one is verified by the push path being untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:40:19 -04:00
bvandeusenandClaude Opus 5	635138b0d1	ci: pin the build clock to the commit, so an unchanged refresh publishes nothing ... Milestone 362 step 1, closing #3265's root cause. The weekly base refresh rewrote all three `:latest` tags on 2026-08-30 with nothing changed in any of them. Not a cache miss — run 4934's log shows every content step CACHED and both bases resolved to unchanged pinned digests. buildkit stamps the image config with the wall clock of the build, so identical layers get republished under a new config blob and therefore a new manifest digest. The cost is not storage, it is meaning: `:latest` moved on a calendar, so a digest change stopped being evidence that anything was different. That is the one thing a digest is any use for, and it is load-bearing here — the reuse check, the `:c-<sha>` rollback story and any future redeploy signal all rest on it. SOURCE_DATE_EPOCH normalises `created` and the history timestamps, so the same source produces the same config bytes and the same digest, and pushing it is a registry no-op. The value is routed through artifacts.sh's existing `newest()` rather than taken from git separately. `revision`, `version` and now `epoch` are three fields of ONE lookup, so they cannot drift into naming different commits — a divergence that would stamp an image reproducibly against one commit while it reported being another, with both values looking perfectly well-formed. Note #3127 §2 is the record of what a second clock costs; this adds a view, not a clock. Also corrected: the build step comment and ci-requirements.md both described the churn as current behaviour with the fix as a "likely" future. They now describe what the file does. Tests pin the property the fix depends on, not the fix: epoch is the same commit version names, in both renderings including the extension's unpadded one, and it does not move between two calls on one checkout. A future refactor that gave epoch its own `git log` would pass every other test in that file. Not yet verified end to end — proving it needs two consecutive refreshes to land on the same digest, which is the next thing, and is the step #3265 exists because nobody did last time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 13:46:16 -04:00
bvandeusen	0a5bbe81dc	docs: the scheduled refresh does NOT republish nothing (#3265) ... Step 4 asserted that when the base has not moved the refresh is "a ~13s no-op that republishes nothing", and that this no-op was the point. The first half is false and was written without being tested. Run 4934, the first real fire: every content step reported CACHED and both bases resolved to unchanged pinned digests, yet all three :latest tags took a new manifest digest anyway. fabledcurator 4ea5265ba017 -> 380e504de0fa fabledcurator-ml 6e7cfc0c09fd -> 6b2eefc301d8 fabledcurator-agent 44920e0af1f3 -> 54accbeb52ed buildkit mints a fresh image config per run, so identical layers get republished under a new config blob. Storage cost is trivial; the cost that matters is that a :latest digest change stops meaning "something is different", and :c-<sha> is handed a new manifest to diverge from every Sunday for no reason. Corrects the workflow comment (x3) and ci-requirements.md to say what actually happens. Filed as #3265 with the candidate fixes; the likely one is a deterministic SOURCE_DATE_EPOCH off the value artifacts.sh already derives, which would make "same source, same version" into "same source, same bytes". The rest of step 4 verified clean on the same run: the guard passed (HEAD is main (499720d), `git checkout -B main`) — so this runner DOES evaluate the env context inside `with:` — the tag list was :latest alone with no :c-<sha>, and the repoint step correctly found nothing to write.	2026-08-30 12:57:06 -04:00
bvandeusen	6663e06aa6	ci: assert the scheduled refresh actually checked out main ... BUILD_REF is read through the `env` context inside `with:`, which this runner is not known to evaluate. `${{ steps.* }}` and `${{ secrets.* }}` in `with:`/`env:` are proven here; `env` is not, and run 4915's checkout log (`git checkout -B dev refs/remotes/origin/dev`) cannot tell an honoured `refs/heads/dev` from an empty value falling back to the same place — the two are indistinguishable on every path except the one that matters. If it does resolve empty, the weekly refresh checks out dev and pushes its source to :latest, which is production. Every lane stays green and the first symptom is production running code that was never merged. So each of the four jobs now asserts its own checkout before doing anything, gated on `github.event_name` — the `github` context is demonstrably evaluated in `if:`, so the guard cannot be disabled by the same uncertainty it covers. A red weekly job is an acceptable outcome; shipping dev to production is not.	2026-08-29 22:57:41 -04:00
bvandeusen	63e0a423d7	ci: a weekly base-image refresh on the channel tags (milestone 326 step 4) ... Skip-if-exists is keyed on our own source, so an artifact whose source stops moving stops picking up base-image updates. `agent/` last changed 2026-07-17; every push since has correctly declined to rebuild it, which also means it will serve that day's nvidia/cuda layers indefinitely. A `schedule:` trigger, Sunday 06:00 UTC, away from CI-runner's Monday security sweep so the two are never diagnosing each other. #3154's blocking open question is dissolved rather than answered. It was written when the identity was a `r-<revision>` TAG, and asked how the next ordinary push could avoid repointing :latest back off the refresh. Milestone 318 replaced that tag with a LABEL, and #3183 made the repoint step exclude its source tag so the label stays readable. Excluding the source is what also keeps a refresh from being undone: on the next main push the reuse check hits, :latest is not rewritten, and the new :c-<sha> is written FROM the refreshed :latest. To be verified by digest, not by this argument. Four decisions, each commented where it lives: * It builds `main`, not the branch that triggered it. Forgejo registers a cron from the default branch — `dev` here — so a scheduled run arrives with github.ref on dev, and a refresh of :dev would be refreshing the one channel that is rebuilt constantly anyway. The ref is decided once in a top-level `env: BUILD_REF` that all four checkouts take. Deriving it per job would let the halves disagree: sign-extension would derive dev's extension version while build-web bundled main's, and the release download would 404 on a version that exists perfectly well. * It publishes only the channel tag. :c-<sha> for main's HEAD already names the bytes that commit built; re-pushing it over refreshed layers would break the one tag rule 145 makes immutable, and it is the rollback unit — so the breakage would surface on the day somebody needed it. The repoint step needs no schedule case: the tag list is the channel tag alone, SOURCE is the only entry, it is excluded as always, and the step correctly does nothing. * It bypasses reuse by construction, since it rebuilds the same source and fc.revision always matches. Checked in the reuse step beside force_build, so one decision still drives both the build and the repoint. * `pull: true`, on the scheduled path only, is the actual mechanism. A moved base tag changes the FROM layer's cache key and everything above it rebuilds; an unmoved one is satisfied by the registry cache and the refresh is a ~13s no-op that republishes nothing. That no-op is the point — :latest should change when there is something new in it, not every Sunday. The known lag, left deliberately: an apt package update while the base tag stands still is not caught, and closing it needs no-cache: true, which buys weekly churn for it.	2026-08-29 22:53:48 -04:00
bvandeusenandClaude Opus 5	5e72076298	ci: registry-backed layer cache for all three images (milestone 326 step 2) ... `cache-from`/`cache-to` on `<image>:buildcache`, `mode=max`, on all three build steps. Closes the half of the driver change that step 1 left open. Step 1 measured worse, not better, and that was expected but is worth stating with numbers. Run 4896, first builds after moving to `docker-container`: build-web 3m44s (cold baseline 2m23s) build-ml 3m49s (cold baseline 3m20s) build-agent 11m12s (cold baseline 9m26s) The container driver gets a FRESH buildkit instance per job, so it has no local layer store to fall back on — where the old docker driver at least reused whatever the runner's dockerd happened to hold. That is why a registry cache is the only cache this driver can have, and why step 1 on its own is a regression rather than a win. `mode=max` so intermediate stages cache too. The agent's two ~150s pip layers and web's frontend-builder stage are the entire cost, and a min-mode cache would drop exactly those. A `:buildcache` tag is not the withdrawn tag scheme returning. Rule 145 narrowed against names NOTHING reads; this one is read by every build that runs, is one moving ref per image rather than one per build, holds cache blobs rather than a shippable artifact, and is overwritten in place rather than accumulating. Closer to `:dev` than to the `:2026.8.28` tags 318 deleted — and said so in the workflow, so it is not "cleaned up" by a later reader. Expect the next build to be slower again, once: it is still cold AND now pays the cache export. The measurement that matters is the one after that. Scribe #3114. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 18:56:13 -04:00
bvandeusenandClaude Opus 5	e21c9fdd34	ci: a force_build escape hatch for the path skip-if-exists hides (326 step 3) ... `workflow_dispatch` with a `force_build` boolean, honoured inside each of the three reuse steps. It exists because skip-if-exists made its own build path untestable. `agent/` has not changed since 2026-07-17, so the agent build has correctly declined to run on every push since — which also means #3190, whose whole symptom lives on that path, cannot be reproduced on demand. Editing build.yml does not force a build either, and that is deliberate: the workflow is not shipped bytes, so it is in no artifact's path set, and putting it in one would re-version every artifact for a comment change. That is also why this lands before step 2 rather than after. Step 1 moved the builds onto a container driver and turned attestations off; the claim that `fc.revision` still reads back cannot be checked until something actually builds under that driver. Run 4887 confirmed only the cheaper half — `Set up buildx` succeeded on all three jobs, so the buildkit sibling container does start against the mounted socket. Details worth keeping: * FORCE is checked in the reuse step, not in the build step's `if:`. The repoint step keys off `hit` too, and a force that bypassed only the build would leave the two disagreeing about what had happened. * `github.event.inputs`, not the `inputs` context — release.yml already uses that form and it is the one this runner is known to evaluate. Read through env rather than interpolated into the run block, same as release.yml's TAG. * One input, not one per artifact. Three booleans is an interface nobody remembers. Scribe #3252, #3249. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 18:38:29 -04:00
bvandeusenandClaude Opus 5	6b3ec98fa8	ci: build on a real buildx driver, attestations off (milestone 326 step 1) ... Adds `docker/setup-buildx-action@v3` to build-web, build-ml and build-agent, and sets `provenance: false` / `sbom: false` on all three build-push steps. Two open issues share one root, which is why this is one change: * #3114 — the agent rebuilds a ~6.3 GB CUDA + torch image whenever the runner's local cache is cold, 9m26s against 7s warm. The default `docker` driver cannot export a registry cache at all, so the fix is unavailable until the driver moves. The cache itself is step 2, deliberately not here. * #3190 — build-agent goes red AFTER a successful push, `No such image` from the local daemon. The leading candidate is the docker driver resolving image metadata against a local store a registry-direct push never filled. The attestation flags are the load-bearing part. On the default driver they were no-ops; on the container driver, build-push-action@v5 defaults provenance to TRUE when pushing. Provenance attaches an attestation manifest, that makes the pushed tag a manifest INDEX, and `.Image.Config.Labels` does not resolve through an index — so the `fc.revision` label the reuse check reads off the channel tag would come back `<none>` on every push. Every image would rebuild forever, every lane would stay green, and the only symptom would be the bill. Same failure as #3183, through a different door; note #3127 §4 records the same shape for `platforms:`. Unverified until CI says otherwise: these jobs run INSIDE a container against a mounted docker socket, so the buildkit container is a sibling of the job container rather than a child. That works over a socket mount and has never been tried on this runner. The gate is the SECOND dev push, not this one. The images currently published were built by the old driver, so one `reuse: NOTE ... no readable fc.revision` is expected now; what must not appear is a second one. Scribe #3249. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 18:36:35 -04:00
bvandeusenandClaude Opus 5	2e01242381	feat(extension): derive the version as unpadded CalVer (milestone 318 step 8) ... `1.0.<minutes since 2020>` -> `YYYY.M.D.HHMM` UTC, from the commit time of the newest change to a packaged extension file. Same clock and same commit as before; readable instead of opaque, and the same value the rest of the family derives. The hold on this step was two questions about AMO, and Mozilla's own docs answer both: ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$ 1. four all-numeric segments -> ACCEPTED ({0,3} more after the first). 2. leading zeros -> REJECTED. A segment is the single digit `0` or starts 1-9, so `08` and `0201` are refused. MDN says it in prose too: "Non-zero numbers must not include a leading zero." So the documented fallback applies, extension only: the same numbers rendered without the family's zero-padding. `2026.08.29.0201` and `2026.8.29.201` are one value in two renderings — rule 148 defines comparison as numeric per segment, under which they are equal — so nothing already published is reordered, and left-padding each segment recovers the family string exactly. HHMM stays one segment because AMO allows at most four. The transition is safe in the other direction too: 2026 > 1, so every CalVer outranks every published 1.0.x. build.yml's downgrade guard confirms it. Also in scope: * MAJOR.MINOR is gone. `cmd_major_minor`, `cmd_patch` and VERSION_EPOCH go with it, the committed version in manifest.json / package.json is now wholly inert, and ci.yml's MAJOR.MINOR-agreement check is retired rather than left running beside a fact that stopped existing (rule 22). * ci.yml's `extension-version` lane now asserts Mozilla's regex verbatim instead of a loose `^[0-9]+(\.[0-9]+)*$` — which would have passed the padded shape. It also asserts YYYY.M.D.HHMM, because AMO would accept a regression to `1.0.<minutes>` while that orders below everything signed since. Checking here is the point: AMO 409s on re-signing, so a version it rejects is burned and cannot be reused. * `artifacts.sh version extension` delegates to packaging.sh, so the two cannot answer differently. The direction matches the existing one — artifacts.sh already asks packaging.sh for the extension's path set. #3156 is what makes this commit safe to make: packaging.sh is in web's path set, so the web revision moves with the extension version and build-web rebuilds instead of republishing an image bundling the previous XPI. Scribe #3138. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 13:43:30 -04:00
bvandeusen	41f2bec3af	fix(ci): the build pushes one tag; the rest are written registry-side (#3190) ... buildx on this runner pushes the first tag to the registry and then re-pushes the remaining ones through the DOCKER driver, reading them out of a local image store that a registry-direct build never populated: #27 pushing …/fabledcurator:latest DONE 15.8s #28 pushing …/fabledcurator:c-0e15c44 with docker #28 ERROR: tag does not exist: …:c-0e15c44 It is intermittent — build-ml made the identical two-tag push seconds later in the same run and succeeded — and the consequence is worse than the red job suggests. `:latest` had already published, so production was correct while the immutable rollback tag rule 145 requires of every main push simply did not exist. Nothing else would ever have noticed: a missing :c-<sha> has no consumer that fails, so it surfaces at the moment somebody needs to roll back, which is the worst time to learn a rollback target was never written. So the build now pushes exactly one ref — the channel's — and the existing repoint step, which already excluded the source tag and already ran on every reuse, now runs on the build path too and owns every other tag. `imagetools create` is a registry-side manifest copy: no local daemon, nothing that can be absent. This adds no new code path; it puts the build case onto the one that was already proven. Chosen over the alternative of asserting each tag resolves after the build, which would have made the failure loud without making it rarer. The cost, accepted: `imagetools create` wraps its source in an index, so :c-<sha> is an index rather than a plain image and fc.revision does not resolve through it. Nothing reads that label off :c-<sha> — the reuse check only ever inspects the CHANNEL tag — and the index names the same manifest, so a pull is byte-identical. The reuse path already produced :c-<sha> this way; this only makes it uniform. `build_tags` goes with it — the tag list now has exactly one consumer.	2026-08-29 00:38:26 -04:00
bvandeusen	d38585ed94	docs: true up the tag scheme against what the pipeline publishes (318 step 9) ... A doc describing a tag scheme the pipeline stopped using is worse than no doc — it is a confident wrong answer, and #3159 named it as the drift that had just bitten rule 148. build.yml (each edit ×3, once per build job): * The derived-values block claimed three values including a `tag` subcommand that no longer exists, and said "on dev the date tag is computed nowhere else". There are two values and no date tag. * The BASE-IMAGE FRESHNESS paragraph argued from a pin that no longer exists — "a date tag has to keep serving the bytes it served (fabledcurator:2026.7.17 still resolves to July's image)". Milestone 318 removed the argument's premise rather than answering it: with no version tags there is no immutable name a refresh could contradict, and rule 145 already allows a moving tag to be republished. Deleted the argument, kept a line saying why there is nothing to argue about (#3159's phrasing: the workflow becomes obviously correct instead of carefully correct). * The repoint comment said ":c-<sha> and the date pin". There is no date pin. * "its set is too narrow — the direction that serves stale bytes on a pin" now names the failure that actually exists: the reuse check hits and the channel serves a web image bundling the previous XPI (#3156). README gains a "Versions and tags" section — the three tags, why there is no fourth, and the fact that with no registry name carrying it, the instance's own report is the only answer to "which build is this?". Also fixes two stale claims: three workflows (four since release.yml) and "AMO signing runs on main only", which has been false since milestone 271 step 6 signed on both channels. ci-requirements.md was missing milestones 313 and 318 entirely: no artifacts.sh, no label-keyed reuse, no FC_VERSION. Added those, plus the #3156/#3202 membership test (can changing this file change the published bytes — not is it copied in), and corrected packaging.sh's "two consumers" to three now that artifacts.sh reads it. docker-compose.yml needed nothing — it names `:dev` directly and never offered a pin target. Note #3127 and rule 145's relation notes are updated in Scribe.	2026-08-29 00:09:52 -04:00
bvandeusen	bce894ba24	feat(settings): the instance reports which build it is (318 step 6) ... A dim line at the foot of Settings: `FabledCurator 2026.08.28.1249 · dev`. This is no longer a convenience. Milestone 318 stopped publishing version image tags, so an instance's own report is the ONLY answer to "which build is this?" — there is no registry name left to check it against. Note #3127 §5 says it directly: a wrong answer here has no second source to contradict it. Three states, kept distinct because collapsing any two of them lies: not asked yet render nothing asked, no version render "unknown" asked, has a version render it A blank footer reads as "no version", which is a different claim from "I cannot say". And a failed health call deliberately does NOT mark the build loaded — a network blip says nothing about the image, and presenting it as "unknown" would look like a defective build. Carried on /api/health rather than a new route: it answers at the same cost (two module constants, no I/O) and TopNav already fetches it app-wide, so a separate endpoint would mean a second request for two strings. Both fields are OMITTED when unset rather than sent empty. Absence already means "cannot say" — an image predating the field says exactly that by not having the key — so a second spelling would make every reader special-case it. The pre-existing test asserting the body is EXACTLY {"status": "ok"} is what keeps a well-meaning `or ""` default from creeping in. FC_CHANNEL now has one definition. It was read from the environment in extension.py and would have been read again here; the new build_info module holds both, and extension.py binds it as a module-level name so existing tests monkeypatch it exactly as before. Separate from config.py on purpose: those are operator settings meant to be changed, these describe the artifact. Channel sits beside the version, never inside it (rule 149), asserted from both ends. A `-dev` suffix would read as a 0 segment to the extension's parseInt comparator and make every dev build compare equal — #2993 exactly. Not hidden, per the operator and §7: the JS bundle and asset hashes fingerprint the build anyway, and "I'm on 2026.08.28.1249" is the single most useful line in a bug report.	2026-08-28 18:08:31 -04:00
bvandeusen	b3989d0224	fix(ci): the repoint was destroying the label it depends on ... Reuse worked exactly once per artifact, then every later push rebuilt at full price. Nothing failed and nothing went red — the savings simply evaporated. `imagetools create` wraps its source manifest in an INDEX. The repoint step passed the channel tag as both source and target, so after any reuse :dev stopped being a plain image, `.Image.Config.Labels` no longer resolved through it, and fc.revision read as absent on the next push. Observed across three runs rather than reasoned about: run 4749 read fc.revision=a7e626a67a79 off fabledcurator-ml:dev and skipped the build; run 4751 read <none> off the same tag and rebuilt. The only thing to touch it in between was 4749's own repoint. The agent hit in 4751 precisely because its :dev had last been written by a real build, not by a repoint — which is the control case. Milestone 313's r-<rev> design was immune without anyone noticing why: the source (the identity tag) was never one of the targets. Step 3 made the channel tag both, and inherited a bug the earlier shape had avoided by accident. Fix: exclude the source from the target list, so the channel tag is only ever written by a real build and stays a plain readable image. On dev that leaves nothing to do, which is correct — the hit already established that :dev points at the right content. On main it leaves :c-<sha>, which rule 145 requires of every main push whether or not a build ran. Also added a note the reuse step prints when a channel tag exists but carries no readable label. That is expected exactly once per artifact during the migration; if it appears on every push, the tag is being index-wrapped again and reuse is dead. This class of failure — correct behaviour, quietly worth less than it reads — is the third one this milestone has turned up, and it is the one that does not announce itself.	2026-08-28 14:55:49 -04:00
bvandeusen	cd0b0ff04a	ci: publish :latest + :c-<sha> on main, :dev on dev — nothing else (318 step 4) ... The narrowing itself. Rule 145, narrowed 2026-08-28 once it was verified that nothing pins: "a third name for the same thing is upkeep for a model we do not run." Gone: :2026.8.28 the per-artifact date tag from milestone 313 step 3. It shipped, it was verified on both branches, and its premise is gone. This is a withdrawal, not a correction — nothing about it was wrong. :main a second moving name for whatever :latest already pointed at, justified by nothing but symmetry with :dev. Kept, and note the inversion: :c-<sha> on main. Milestone 313 step 6 planned to retire it on "haven't used, not important" — amended rule 145 makes it the rollback unit and, with version tags gone, the only immutable identifier left. #3139 is cancelled as superseded rather than quietly dropped. No retention job is built. One already runs (#3157): a nightly Gitea package_cleanup_rule with keep_count=10. Two consequences worth writing down. Its keep_pattern protects any dotted-numeric tag, so the date tags this commit stops minting would have been exempt from retention and accumulated without bound — silently, in the direction of unbounded growth. That trap is now dissolved rather than needing the pattern change #3139 planned. And :c-<sha> IS prunable under that pattern, so the honest statement is that it is the rollback unit for roughly the last 10 builds, not for every main commit ever. cmd_tag goes with its last consumer rather than being left as a dead subcommand for a later step to notice. The report step logs version and revision, which is the same information — the date is the first three fields of the version. Docs needed no change: every consumer already tracks a moving tag (docker-compose.yml on :dev, agent/README.md on :latest), which is the evidence the whole stance rests on.	2026-08-28 14:45:11 -04:00
bvandeusen	7e065fed70	ci: key the reuse check on an image label, not a tag (318 step 3) ... The shadow (dee93fa, run 4732) answered the gate: `imagetools inspect --format` reads `.Image.Config.Labels` against this registry on buildx v0.36.1. So the reuse check now asks the moving channel tag whether the image it already points at carries this commit's `fc.revision`, and the r-<rev> identity tags stop being published. Three things this removes rather than manages: A name minted per build that one thing read. Rule 145's narrowing is aimed exactly there — "a third name for the same thing is upkeep for a model we do not run." The -main/-dev qualifier, and the CHANNELLED list behind it. Which tag you inspect IS the channel, so the distinction has nowhere to live. cmd_identity goes with it. A silent expiry nobody wrote down. r-<rev> matches no branch of the registry's keep_pattern (#3157), so identity tags were prunable past the newest 10 — a pruned one costs a rebuild, in the safe direction and entirely invisibly. A label rides inside a tag that has to exist anyway. It also dissolves #3154 instead of deferring it: a scheduled base refresh rebuilds :latest with the same revision label, the next unrelated push sees a match and skips, and the refreshed base survives. Under the tag scheme that push repointed :latest back to the older base. The measured detail that shapes the code: a missing label returns an EMPTY STRING and exits 0. Branching on the exit code would read "no label yet" as success and skip a build that was needed. So it compares values, and every uncertain case — absent label, unreachable tag, older image — lands as empty, never equals a 12-char revision, and falls through to a build. Reading the specific key matters too. The map carries the base image's labels, and org.opencontainers.image.version sits right beside ours reading 24.04 on the agent — a plausible-looking wrong answer. Expect every artifact to rebuild once on this push: nothing carries a label yet and it cannot be backfilled, since the reuse path copies a manifest and config labels are not manifest annotations. One rebuild per artifact, ever, self-healing after. test_artifact_identity.py is rewritten around what is now load-bearing. The CHANNELLED drift test had nothing left to guard; in its place the revision is asked of git directly, so the file fails if the derivation ever stops being "the commit this artifact's own shipped files last changed in".	2026-08-28 14:37:36 -04:00
bvandeusen	dee93faa37	ci: shadow the label-based reuse key before trusting it (318 step 3) ... Additive and inert. Every build now stamps `fc.revision` as an image label, and a diagnostic step tries to read it back off the channel tag. Nothing downstream reads either yet — the reuse check still keys on the r-<rev> tag. The question step 3 has to answer before committing: can the reuse check be keyed on a LABEL read off the channel tag instead of on a separate tag? If it can, nothing is minted per build, the -main/-dev identity split disappears with CHANNELLED, and #3154's base-refresh clobber stops existing rather than needing a workaround. It also closes a defect found in what already shipped: r-<rev> tags do not match the registry's keep_pattern (#3157), so they are prunable, and a pruned identity tag silently costs a rebuild. A label cannot be pruned — it rides inside a tag that must exist. Three probes rather than one, because `--format` templating over .Image varies by buildx version and one round trip per guess is a poor trade. Two questions, answered at different times, and conflating them would abandon a working design: 1. Do the template mechanics work here? Answered on the first run — if the probes print a labels map, even an empty one, `--format` resolves on this buildx and this registry. That is the real gate. 2. Does a value round-trip? Only once an artifact genuinely rebuilds. This push touches build.yml, which is in no path set, so all three artifacts reuse and stamp nothing. Expect no values this run. That second point is the migration path rather than a defect: under the real mechanism "no label" reads as a MISS, so each artifact rebuilds once and is stamped from then on — one extra build per artifact, ever, in the safe direction. `imagetools create` cannot backfill it, since it copies a manifest and config labels are not manifest annotations.	2026-08-28 14:00:09 -04:00
bvandeusen	d9aa5aa832	ci: a release names a build, it does not make one (milestone 318 step 2) ... Removes the `v*` image-build trigger. A release tag names a commit `main` already built and published; rebuilding it produced the same source under the same names and RE-PUSHED `:c-<sha>` — which rule 145 forbids even when the bytes match, because image configs carry timestamps and "same source" does not mean "same manifest". The tag build was publishing nothing new and violating an immutability rule to do it. Three constructs go with it, all of which existed only to manage that build: The 10-minute XPI polling loop. A release cut fired the tag build and the main build together; the tag build skipped sign-extension and raced straight to the download, losing every time (operator-flagged 2026-05-27 after v26.05.27.0). Polling was the fix for a build that should not have run. It is now a single fetch, and a 404 means the two jobs derived different versions for one commit — which should fail loudly rather than be slept through. The IS_TAG_PUSH carve-out from milestone 313 step 4, which stopped a tag build from claiming an identity tag it would have re-pointed at fresh bytes. build-web's `always() && (success || skipped)` gate, which existed to let a SKIPPED sign-extension through on a tag push while still blocking a failed one. sign-extension now always runs, so a plain `needs` gives exactly the wanted behaviour. Its own branch condition goes too: main and dev are the only triggers, so a gate naming both matched everything, and a condition that is always true reads as though some path avoids it. Releases still happen — rule 148 is untouched and tags are still cut on explicit request per rule 2. They stop building images and start carrying a changelog (step 7). Net 115 lines deleted, 73 added, most of that comments explaining races that can no longer occur. Nothing in the repo referenced the tag build, so no doc changes were needed; the broader doc pass is step 9.	2026-08-28 13:49:45 -04:00
bvandeusen	609bc82acc	ci: reuse the published image instead of rebuilding it (milestone 313 step 4) ... Before building, each job asks the registry whether this artifact's content is already published. On a hit it skips the build entirely and repoints the channel and date tags at the existing manifest with `imagetools create` — registry-side, no layer transfer, seconds. This is the step that stops a push touching only `agent/` from rebuilding web and ml, and stops a merge to main rebuilding what dev already built. The question is asked with a new `artifacts.sh identity`, not with the date tag: the date tag is day-precise and last-one-wins, so two different builds share it and it cannot answer "is this content published?". The commit sha would move on every push and never hit, which is the redundant rebuild being removed. The revision does both jobs — content-unique, and stable across pushes that did not touch the artifact. Identity is channel-qualified for web and only for web, because web is the only image that takes a build-arg: FC_CHANNEL is baked in and reported by /api/extension/manifest, so its dev and main builds of one revision are genuinely different images. ml and agent take none, which is what lets a merge reuse dev's build rather than rebuilding the agent's CUDA image to produce bytes that already exist. tests/test_artifact_identity.py reads the Dockerfiles and fails if that list drifts from the ARG declarations, in either direction — collapsing the channels ships an instance that reports the wrong one, and splitting them needlessly rebuilds every merge. Failure direction is deliberate: an inspect that errors for any reason reads as a miss and the build runs. Only a real 200 skips one. A tag-push never claims the identity. It rebuilds a revision main already published, and image configs are not bit-reproducible, so re-pushing r-<rev> would point an immutable tag at fresh bytes — rule 145's exact prohibition. It publishes only its own :v... label and otherwise reuses. Base-image freshness, decided rather than left implicit: an artifact whose source stops moving stops picking up base updates under its pinned tag. That is what a pin means, and rule 145 already says the refresh belongs on the moving tag instead. Filed as #3154 rather than folded in here, because the naive version regresses :latest on the next unrelated push. ci.yml's backend lane gains fetch-depth: 0 — the new tests derive real revisions, and on a depth-1 clone that derivation returns the tip sha or fails, so the lane would go green while asserting nothing. The three build jobs' shadow steps are renamed and re-commented: those values stopped being informational at step 3, and a step captioned "nothing reads this" beside steps that do is worse than no caption.	2026-08-28 08:24:58 -04:00
bvandeusen	7a20c55441	ci: publish a per-artifact date tag on main builds (milestone 313 step 3) ... Each main build now also publishes <image>:YYYY.M.D — the date of the commit that artifact's shipped files last changed in. Purely additive; :main, :latest, :c-<sha> and the dev path are untouched, so reverting this commit reverts the behaviour. main push -> :main, :latest, :c-<sha>, :2026.8.27 dev push -> :dev Per artifact, so an image whose files did not change keeps the tag it already had. On this commit the agent reads 2026.7.17 while web and ml read 2026.8.27 — six weeks apart, from one push. Step 4 turns that into not rebuilding it. Day precision, and a second main build the same day replaces the first. Operator's call, and the reasoning is theirs: same-day work is not something worth pinning. A rollback goes to a day, not to the fourth merge of a Tuesday afternoon. It also makes retention mean "the last N days" rather than "the last N pushes". CALVER is computed inside the main branch rather than at the top of the step, and hard-fails when empty. There is no `set -e` here, so an unconditional assignment that failed would have left it empty and published the tag `fabledcurator:` — an invalid name, from a step that still reported success. It is also simply unused on the dev and tag paths. Fixed a stale comment while in this block rather than leaving it for step 7: it claimed release tags carry "no `.N` per family release-posture rule". Rule 148 was amended on 2026-08-24 to REQUIRE the suffix, after the ban caused a same-day tag to be retargeted and a release deleted to make room (note 2813). FC's own tags already carry suffixes; only the comment was asserting the superseded rule. Verified before pushing: the derivation holds across 200 commits of real history — a derived revision always touches its own path set, the version never decreases along any parent->child edge, and web tracks all three extension-only commits in the log. That last one is the direction that would serve stale bytes on a pin.	2026-08-27 21:56:52 -04:00
bvandeusen	0c43fa3eb2	ci: shadow the per-artifact derived versions (milestone 313 step 2) ... Every build job now logs the tag, version and revision its artifact would get. Nothing reads them; no `set -e`, and each derivation falls back to UNAVAILABLE, so a broken script cannot fail a build. Same discipline as milestone 271 step 2, which is what made that cutover safe to do in one commit. Also fixes a landmine the plan named but had not checked: build-ml and build-agent were checking out at depth 1. Both now use fetch-depth: 0. That mattered more than it looks. A depth-1 clone sees one commit, so `git log HEAD -- <shipped paths>` either returns that commit's timestamp — plausible, and wrong — or returns nothing. For build-ml on this push it would have returned today's date, because HEAD touches backend/, and nothing downstream would have questioned it. For build-agent it would have returned nothing at all, since no single commit here touches agent/, and artifacts.sh exits non-zero rather than guessing. One direction is silent and one is loud; only the loud one was ever going to get noticed. What to read from the shadow lines over the next few pushes, in order of how badly each would bite: * a push touching the extension must move BOTH the extension and web, because build-web bakes the XPI in. If web does not move, its path set is too narrow and a pinned web image will serve an extension it does not name. * a push touching only agent/ must leave web and ml still. If they move, their sets are too wide and they will rebuild for changes they do not ship. * a docs-only push must move nothing. * dev and main must derive the same values for the same source. Step 3 only lets these values name a tag once those hold.	2026-08-27 21:41:14 -04:00
bvandeusen	0db38cc111	ci: log in to the registry with the docker CLI, not docker/login-action ... build-ml failed at the login step twice on a7e626a, five seconds in, with MODULE_NOT_FOUND on the action's own dist/index.js. Not the token — the secret resolved to *** and the action never ran far enough to use it. The cause is a race in act_runner's shared action cache, not corruption. A remote action is cached at one /root/.cache/act/<hash> per runner, and build-web, build-ml and build-agent all start in the same second and all want docker/login-action@v3. One job re-clones that directory — emptying and repopulating it — while another walks it to copy into its container, and the walker lstat()s a file that just vanished. The two failures named DIFFERENT missing files, eslint.config.mjs then jest.config.ts, which is what rules out a bad cache entry and points at the race: a dangling entry would name the same file every time. Re-running does not help, because the re-run starts the three jobs simultaneously again. It reproduced immediately. Dropping the action removes FC from that race for this step. Logging in is one command, the docker CLI is already in the CI image per ci-requirements.md, and the same reasoning as family rule 5 applies: a marketplace action buys nothing when the tool is baked into the image the workflow already selected. Password on stdin, never as an argument — an argument lands in the process table and draws docker's own deprecation warning. This narrows the exposure rather than closing it. All three jobs also share docker/build-push-action@v5 and can race on it the same way; that one has not lost yet, and replacing it means hand-rolling buildx invocation including the build-args and provenance handling, which is a bigger change than this failure justifies. Recorded on #3118. Live consequence being cleared: fabledcurator-ml:dev was left a commit behind fabledcurator:dev, which is the stale-pairing trap the trigger comment on 239b1ed warns about.	2026-08-27 12:08:46 -04:00
bvandeusen	a7e626a67a	feat(extension): report the channel beside the version (step 7) ... Closes the half of the ask the signing work didn't: a way to tell a dev build from a main one. FC_CHANNEL is baked into the web image at build time and /api/extension/manifest reports it as its own key, next to version — the popup banner, the toolbar tooltip and the Settings card all name it. Beside the version, never inside it. A `1.0.3499884-dev` suffix is the obvious shortcut and it is the exact failure this design comes from: versionIsNewer parses each dotted segment with parseInt, so a suffixed segment reads as 0, every dev build compares equal to every other, and "no update available" stops being distinguishable from "I cannot read this version". The comparator already degrades rather than discarding (rule 150), which is a reason not to NEED the suffix, not a licence to add one. Two tests hold the line — one backend, asserting version and channel are separate keys; one frontend, asserting the rendered version text stays the bare derived number. Optional on the read side, and absent rather than defaulted. An image built before this field says nothing by not having the key; an image built without a channel now says nothing the same way, so there is one absence to handle instead of a second spelling of "unknown". Every reader drops the label entirely when it is missing and reads exactly as it did before. Reported verbatim rather than validated against {dev, main}: if an image declares something else, showing what it claims helps whoever is debugging more than dropping it would. FC_CHANNEL is declared LAST in the Dockerfile. An ARG invalidates every layer below it, and this is the one value that differs between the dev and main builds of identical source — earlier, and the two channels could never share a cached pip install. A tag push counts as main: a vYY.MM.DD tag is cut from main, so that image is a main-channel artifact wearing an immutable name. No channel switcher, deliberately. background.js:34 already records that Firefox's static update_url cannot apply, because every FC instance is a different host — so the extension asks its configured backend, and the channel IS the instance it points at. Switching is repointing apiUrl and reinstalling from that host. A separate setting would contradict each server build shipping its own extension. This commit touches packaged extension files, so it moves the derived version and will sign a new one via AMO — the first push to exercise the extension-changed path from dev end to end.	2026-08-27 11:47:30 -04:00
bvandeusen	9eb946b21b	ci(extension): sign on dev too, and bundle the XPI into :dev (step 6) ... The step the milestone exists for. sign-extension ungates from main-only to main-or-dev, and build-web downloads the XPI on dev as well, so a dev push produces an image carrying the extension that is being developed rather than requiring a merge to try one. Not two signatures. The version is the commit TIME of the newest packaged extension change, so dev and main derive the SAME number for the same source. A dev push that changes the extension signs it; the merge to main finds the ext-<version> release already there, hits the cache, and bundles the byte-identical XPI into :latest with no second AMO call. One signature per extension CHANGE, shared by both channels. That property is what makes two channels affordable at all, and it is why step 4 had to land first: ungating this while the version was still the hand-set 1.0.11 would have found the existing ext-1.0.11 release, skipped AMO, and bundled main's stale XPI into :dev — a dev channel confidently serving old code. Tags stay excluded. The tag path deliberately skips signing and polls for the release instead (the 2026-05-27 race). The ext-<version> release's target_commitish moves from the literal "main" to $GITHUB_SHA. Either branch can create that release now, and tagging a dev-signed XPI against a main commit that need not even contain the source it was built from is a lie that costs nothing to avoid. Known, not addressed here: two concurrent builds that both derive the same unsigned version will both call AMO and the loser gets a 409. The window already existed between main and tag pushes; dev signing widens it. It fails loudly rather than shipping anything wrong, and the rollback trap cleans up the empty release. Filed separately. Also unchanged here: ci.yml's manual-bump guard is still in place and still false. It does not fire on this commit — nothing packaged changed — but it will fail the lane on the next extension change, demanding a bump that no longer decides anything. Step 5 next.	2026-08-27 10:56:58 -04:00
bvandeusen	5447a40e97	ci(extension): the derived version drives signing (milestone 271 step 4) ... Cutover. sign-extension no longer reads the version out of the repo — it runs packaging.sh version and stamps the result into manifest.json and package.json in the working tree before web-ext sees them. Never committed back: the commit carrying the bump would itself be a change to the extension and would move the version again. Shadow mode ends here, in both build.yml and ci.yml. It had one job — validate the formula at zero cost before a real AMO version was burned — and CI confirmed it on 239b1ed: shadow: manual=1.0.11 derived=1.0.3499884. build-web re-derives rather than being handed the value, so it gains fetch-depth: 0. It was the outstanding landmine: a depth-1 clone derives a WRONG, too-low version rather than failing, and would then 404 fetching a release that exists under its real name. sign-extension and extension-version already had full history. New guard, and it stays permanently: refuse to sign when the derived version is strictly OLDER than the highest ext-* release already signed. Firefox rejects a downgrade and AMO never releases a burned version, so backwards is unrecoverable — it strands every install that took the higher one. Strictly older, not older-or-equal: equality is the ordinary case, an unchanged extension deriving the same version it did last build, which is exactly what makes the ext-<version> cache hit and holds AMO to one call per extension CHANGE rather than per push. The release list is paginated because ext-* shares it with the v* tags, and the bound fails rather than calling the highest it happened to see the highest there is. First derived value is 1.0.3499884 against a highest-signed ext-1.0.10, so the backfill direction is right by six orders of magnitude. 1.0.11 sits in the repo and was never signed; nothing is stranded by skipping past it. Still main-only. Step 6 ungates sign-extension to dev, which is what actually puts an XPI on :dev. Note for step 5: ci.yml's manual-bump guard is now false. It still demands a hand bump when a packaged file changes, and that bump no longer decides anything — the derived value overwrites it at build time. Harmless but pointless, and it should be retired before the next extension change.	2026-08-27 10:45:17 -04:00
bvandeusen	239b1ed8d9	ci: build :dev images again so the dev channel can carry a build ... build.yml triggered on main and tags only. The 2026-05-26 comment gave the reason: "operator tests from :latest after merge-to-main, not from the dev branch image. Saves one full docker build per dev push." That trade has since been named as a fault. Family rule 147 — main IS production, test on :dev, never by shipping — and rule 146 — a rolling channel refreshes itself, and a channel that can only be refreshed by shipping is not a channel. 146's note on 147 describes this exact shape: the pressure to test by shipping does not come from carelessness, it comes from :dev being unable to carry the build. Two live consequences, not hypotheticals: - docker-compose.yml pins fabledcurator:dev, an image nothing has published since May. The registry-image path of the documented quick-start could not have worked. - trying an extension change required merging to main, because sign-extension is gated to main and :dev did not exist to carry an XPI. Shipping was the only way to test. All three images build on dev. Deliberate: a :dev web image paired with a stale :dev ml or agent is a worse trap than no dev channel, because the mismatch surfaces as a runtime failure rather than a missing tag. The cost the 2026-05-26 note was avoiding is real and is now paid on every dev push — layer reuse should keep ml's cost to the COPY layers, but if it bites, narrowing is a `paths:` filter away. :dev only. The dev path never writes :c-<sha>: that is the rollback unit (rule 145), and a rolling tag may legitimately carry newer contents than the :c-<sha> of the same commit. This does NOT yet put an XPI on :dev — sign-extension is still gated to main, and ungating it has to wait for the derived version to control publishing, or dev would sign the hand-set 1.0.11, hit the existing cache and ship main's stale XPI. That is the next step.	2026-08-27 09:26:58 -04:00
Claude	1c6452e10e	ci(extension): shadow the derived version + verify real XPI contents ... Milestone #271 steps 2 and 3. Neither changes what gets published. STEP 2 -- shadow mode. build.yml's sign-extension and ci.yml's extension-version guard now log the version that WOULD be derived from git history alongside the hand-maintained one. Nothing reads the derived value, and neither site can fail because of it. This exists because `web-ext sign` is one-shot per version: AMO 409s on a repeat, so a wrong formula burns a real version number that cannot be reclaimed. Comparing the two across real builds is the only way to validate it at zero cost. sign-extension runs on main only, so main pushes are the sole source of truth for whether the derived number moves exactly when the shipped extension changes -- the dev-side log is a convenience, not the evidence. sign-extension now checks out with fetch-depth: 0. The derived version is a commit count and a depth-1 clone cannot produce one. STEP 3 -- XPI content verification. Every other packaging assertion checks our declaration against itself. This is the first that asks web-ext what it ACTUALLY wrote into the archive. That assumption was both unverified and fragile: `test/**` only survives to web-ext because callers `set -f` before substituting it, so losing that quoting would silently start shipping dev files with no other signal. The step builds the XPI and asserts test/, scripts/, vitest.config.js, package.json, package-lock.json, README.md and node_modules are absent -- and, because an over-matching exclusion would break the extension at runtime rather than at build time, that manifest.json, all four lib/*.js and every UI directory are present. unzip is installed only when missing; node:24-bookworm-slim may not carry it. Refs #2399, #2400 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 16:15:12 -04:00
bvandeusenandClaude Opus 4.8	9449241fc2	ci(agent): publish the GPU agent image (build-agent job) ... Build + push fabledcurator-agent alongside web/ml (own CUDA + onnxruntime-gpu image, context=agent/, same tag cadence: main → :main/:latest/:c-<sha>, tag → :<version>). So the operator PULLS + runs it on the GPU machine instead of building locally. README switched to docker pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa	2026-06-29 14:26:03 -04:00
bvandeusen	9564d073b9	fix(ci): POSIX-safe SHORT_SHA in build.yml (runner uses dash, not bash) ... `${GITHUB_SHA:0:7}` substring expansion is bash-only; the runner executes the step via dash/BusyBox sh and errored with `Bad substitution` (act_runner workflow shell, observed on the 65386f0 main-push run). Switched to `SHORT_SHA=$(printf '%s' "$GITHUB_SHA" | cut -c1-7)` which works in both shells. Both build-web and build-ml tag-determination steps updated.	2026-06-01 07:53:33 -04:00
bvandeusen	8de7ccd07d	build(ci): per-commit :c-<short_sha> tag on main-push per family rule #46 ... Adds an immutable per-commit docker tag to every main-push build: `git.fabledsword.com/bvandeusen/fabledcurator{,-ml}:c-<short_sha>`, alongside the existing floating `:main` + `:latest`. Implements the new family release-posture rule "Tags are milestones, not gates — commit-SHA images are the rollback unit" so rollback to any commit on main is `docker pull …:c-<sha>` with no release ceremony required. Behavior change summary: - main-push: was {:main, :latest} → now {:main, :latest, :c-<short_sha>} - tag-push (opt-in vYY.MM.DD only, no .N): unchanged - safety-net dev: unchanged No code changes; the rule is about how the tag list is constructed. Tag-push workflows stay as-is — vYY.MM.DD milestone cuts can still fire them when the operator wants a labeled checkpoint.	2026-06-01 01:28:55 -04:00
bvandeusen	b8ad17c68d	fix(build): poll for ext-<version> release in tag-push build-web (race fix) ... Cutting a release fires BOTH the push-to-main workflow AND the push-to-tag workflow in parallel. main-push runs sign-extension (AMO round-trip 1-5min) then publishes the ext-<version> Forgejo release; tag-push skips sign-extension (gated to main) and races straight to build-web's Download XPI step. Tag-push lost every time — got 404 from releases/tags/ext-<version> before main-push had finished signing. v26.05.27.0 hit this: tag-push build-web died on exit 22 because the ext-1.0.4 release wasn't published yet (it arrived ~4min later). Fix: wrap the release lookup in a 20-iteration sleep+retry loop, 30s between attempts (10min total upper bound, generous for AMO). main-push's signing eventually publishes the release; tag-push picks it up on a later poll. No more manual rerun of the failed job after every release cut. Banked the trap as reference_tag_push_main_push_race.md — same shape will recur any time a tag-push workflow consumes a main-push-produced artifact.	2026-05-27 13:25:58 -04:00
bvandeusenandClaude Opus 4.7	3f0153cba5	ci(workflows): dedupe + versioned image tags ... ci.yml: drop pull_request: trigger — push: branches: [dev, main] already covers it; pull_request was duplicating ci.yml runs on every dev push with an open PR. (No fork PRs in this repo.) build.yml: drop dev from push triggers — operator doesn't use the :dev image. Add tags: ['v*'] trigger + tag-push branch in the Determine-tag logic so cutting a release tag publishes an immutable :v26.05.26.X image (rollback story) without re-publishing :latest. Extend the XPI-download step to fire on tag pushes too so the versioned image carries the signed extension. Net per hotfix cycle: 5 runs → 3 (no tag) / 4 (with tag). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 18:26:56 -04:00
bvandeusen	06a2f60c08	fix(ext-ci): use browser_download_url not /releases/assets/<id> + add -f to curl + magic-byte sanity check (operator-flagged 2026-05-26: prior build silently wrote '404 page not found' into the XPI file, Firefox rejected as corrupt) — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 00:42:35 -04:00
bvandeusen	efb142239d	fix(ext-ci): drop actions/upload-artifact (Forgejo Actions doesn't support v4+ GHES) — build-web reads XPI directly from the ext-<version> Forgejo release asset — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 23:32:58 -04:00
bvandeusen	5587a76606	fix(ext-ci): replace jq with python3 (jq not in ci-python image) + bump ext 1.0.2→1.0.3 (escape AMO 'version already exists' from prior partial-failure run) + add rollback to prevent empty cache-release tombstones — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 23:14:11 -04:00
bvandeusen	c06cbc0abe	feat(ci): inline extension sign into build.yml + Forgejo Release Assets as XPI cache (v26.05.25.5) — bump ext to 1.0.2 — Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-25 21:30:11 -04:00
bvandeusenandClaude Opus 4.7	4a09cca46e	feat(ci): pin container.image to ci-python:3.14 + ship ci-requirements.md ... Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 09:19:30 -04:00
bvandeusenandClaude Opus 4.7	ff122c55eb	ci: switch runner label from fabledcurator-ci to python-ci ... Generic python-ci runner is reusable across the family (FabledScribe, FabledSteward, NhenArchiver, StashHandler, etc.) rather than scoped to just this project. Runner image lives at CI-Runner/CI-python/ in the operator's workspace; pattern mirrors CI-Runner/CI-go and CI-Runner/CI-flutter. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 08:45:48 -04:00
bvandeusenandClaude Opus 4.7	edb3fafd4d	ci: add image-build workflow + document required runner label and RELEASE_TOKEN ... Pushes :dev on every dev push; pushes :main and :latest on main. Uses RELEASE_TOKEN (a broader-scoped Forgejo PAT covering write:package, read:package, write:release, write:issue) so the same secret can serve future release-cutting and issue-management workflows. README now documents the fabledcurator-ci runner label and the required RELEASE_TOKEN scopes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 07:52:55 -04:00