104 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	b74a4c964b	fix: the image brings its own PID 1 instead of asking for init: true (4295) ... Operator, 2026-09-23: *"it is out of the norm to require this init call we need to fix this."* Correct, and it is the same mistake as declaring the healthcheck per service — the image needing a deployment to remember a flag before it behaves correctly. PID 1 carries a duty no other process has: every orphaned process in the container reparents to it and must be reaped or it stays a zombie holding a PID slot. This app makes orphans in normal operation — six service modules shell out (gallery_dl, thumbnailer, backup_service, external_fetch, download_service, download_backends) and celery's prefork pool forks children that spawn them. Whatever the role, something that is not an init ends up as PID 1: supervisord for `all`, hypercorn for `web`, celery for a worker. `init: true` covered that, and cost correctness the moment it was forgotten or silently dropped — an older Swarm, a plain `docker run`, a compose file someone copied. No signal either way. So tini goes in the image and is the ENTRYPOINT. `docker run <image>` is correct on its own now, `init: true` comes out of docker-compose.single.yml, and nothing downstream has to know. The smoke asserts /proc/1/comm is tini, read from /proc because the runtime stage installs no `ps`. ## A correction to what I told the operator I justified `init: true` by saying supervisord "has no idea about orphans it never started". That is very likely wrong: supervisord's reaper calls waitpid(-1) and logs "reaped unknown pid" for children it did not spawn, so it does reap orphans. I asserted the mechanism without checking it, and could not check it here — `supervisor` is not installed in this environment. It does not change this commit. tini is correct whichever way that lands, and it covers the single-role containers too, where celery or hypercorn is PID 1 and the subprocess-spawning is heaviest. But the reason I gave was not a verified one and should not have been stated as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:18:47 -04:00
bvandeusenandClaude Opus 5	b09ee87255	fix: supervisorctl could not reach supervisord inside the container (4295) ... The smoke's own diagnostic line printed this for a whole run and passed, because it was behind `|| true`: Error: .ini file does not include supervisorctl section supervisord was fine. `supervisorctl` simply could not talk to it — the generated config had no `[unix_http_server]`, `[supervisorctl]` or `[rpcinterface:supervisor]`. That is the first tool anyone reaches for when a lane misbehaves in the consolidated container. `docker exec <c> supervisorctl status` to see which processes are actually up; `restart ml` to bounce one without taking the whole application down with it. Consolidation took `docker ps` away as the way to see the lanes, and this is what replaces it — so shipping without it would have left an operator with one container, five processes inside it, and no way to ask about any of them. They are about to run this in production. The test asserts the three sections AGREE on one socket path rather than merely existing: a serverurl pointing where nothing listens fails in exactly the same way and reads as configured. The smoke's line loses its `|| true`. A diagnostic allowed to fail silently is one that stops being true without telling anyone — which is precisely what happened here. It still printed the evidence that something was wrong while nothing depended on it, which is the argument for printing it at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:09:11 -04:00
bvandeusenandClaude Opus 5	efde3b188f	refactor: the image carries its own healthcheck and picks it by role (4295) ... Operator, 2026-09-23: *"why isn't the healthcheck built into the image or base on what command runs if one is passed in. why is it manually declared in the stack here."* No good reason. The container is the only thing that knows what it was asked to run, and every compose file, stack file and README had to restate it: web -> urllib /api/health worker -> celery inspect ping -d celery@$HOSTNAME all -> both, for every lane Three checks written by hand, once per service, in every file anyone ever wrote — none of them wrong until a role changed, and all of them silently wrong after. The same duplication the lane table exists to remove one level down, and I built it without noticing. `entrypoint.sh` now records the role it started. The Dockerfile declares ONE `HEALTHCHECK` that reads it and asks the right question: HTTP for web, a self-addressed celery ping for a worker lane, both-for-every-lane for `all`, and nothing for shell/alembic, which are one-shot and have no liveness to probe. `docker-compose.single.yml` and the consolidated stack declare none. A service that wants something else can still declare its own; docker prefers it, so the escape hatch is the default docker behaviour rather than a flag. Two details that are load-bearing: * The role is written ONCE, by the outermost invocation. `all` starts the other roles through this same script under supervisord, and a child overwriting the container's role would turn the composite check into a web-only one — silently, and only on the consolidated path. FC_ROLE is exported so a child sees it set and skips. * The celery ping is addressed to THIS node, not a bare ping. A bare one is answered by any worker on the broker, so in a stack with replicas a dead container would report healthy for as long as a sibling lived — the check would be measuring the cluster rather than the container it is inside. `healthcheck_all.py` is deleted; its two probes moved into the dispatcher rather than being a second copy beside it. An unrecorded role PASSES. The entrypoint always writes the file, so the only way to miss it is bypassing the entrypoint — a debugging shape, where a check that cannot tell what it is looking at must not assert the thing is broken (snippet #3969). Said on stdout rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:07:39 -04:00
bvandeusenandClaude Opus 5	43ac737516	feat: the whole application is what the image runs by default (4296) ... Operator, 2026-09-23: *"I also want to see that we remove the need for the command line of the configuration in the consolidated version."* `CMD` was `web`, so the single-container layout only worked if you knew to ask for it by name. A compose file that forgot `command: ["all"]` got a web server with nothing processing its queues — a gallery that loads, accepts an import, and never finishes one. Nothing errors; it just never progresses. Now `docker run fabledcurator` with no command starts hypercorn plus every lane under supervisord. `entrypoint.sh`'s own default moves with it, since the two are doors to the same decision and a disagreement would only show up as `--entrypoint` behaving differently from a plain run. `docker-compose.single.yml` drops its `command:` line; `["all"]` still works and still means the same thing. The multi-service stack is untouched — every service there names its role explicitly, which is what makes it the multi-service stack. ## And CI now actually boots it This is the gap I should have named when I reported milestone 422 at 7/7 and did not. Measured, not inferred: the smoke booted role `web` only (build.yml:1454), nothing in CI ran `all`, `docker-compose.single.yml` was read as TEXT by one test checking stop_grace_period and never run, and test_gen_supervisord asserts the generated config against the lane table without ever handing it to supervisord. So the shape this milestone is NAMED for had started nowhere. Steps 5-7 were marked done on evidence that did not cover it, and the operator is about to collapse their production stack onto exactly that. The smoke now boots the image with NO command — checking the Dockerfile CMD, the entrypoint default and the role together, the way an adopter gets it — and asserts `healthcheck_all`, which was itself never executed. That check passes only when hypercorn answers AND every lane in the table answers the broker; a web-only check goes green with every worker dead, which is the failure mode consolidation creates. It then prints `supervisorctl status`, so a lane that is merely restart-looping is visible rather than inferred. Cheap because ml ships at 0 slots and disabled: nothing loads a model, and the lane answers `inspect` with its consumers cancelled, which is what healthy means for a disabled lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:48:11 -04:00
bvandeusenandClaude Opus 5	2677ce020c	fix: the ml artifact was in a second list I never grepped for (4311) ... ac70f2a removed the `ml` image but CI went red on six tests: `tests/test_artifact_identity.py` parametrises over its own `ARTIFACTS = ("web", "ml", "agent", "extension")`, and every case now hits the dispatch guard that same commit added. My miss, and a specific one. I grepped for `fabledcurator-ml` and `ML_PATHS` and called the survey done — but the artifact is also named as a bare `"ml"`, which neither pattern finds. Rule 90 (grep pinned tests when changing a shared symbol) was surfaced to me while I was making the change and I ran a narrower sweep than it asks for. Lesson #4275 names the shape exactly: an absence claim is only as good as the search behind it, and a grep that matched nothing looks identical to a grep that asked the wrong question. The re-run was done by value, not by name: every occurrence of a bare `ml` in the repo, then filtering. That distinguishes the two things the token means — the celery LANE `ml` and the `backend/app/services/ml` package both stay and account for nearly every hit; only the IMAGE name went. Worth stating in the test, since the next person to grep will hit the same ambiguity. Also swept the prose the first pass left describing the old pipeline: "Four artifacts" (README, ci-requirements), "leaves web and ml alone" (×5 in build.yml, plus both docs), "CI publishes it alongside the web/ml images" (agent/README). The run 4896 build-time measurements keep their `ml` number — that was measured when ml was a real build — with a note saying the name has since gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:32:49 -04:00
bvandeusenandClaude Opus 5	ac70f2aadc	refactor: retire the fabledcurator-ml image name entirely (4311) ... Operator, 2026-09-23: *"we don't need to keep this as I'll be collapsing my stack to the same consolidated version that we're building here."* That was the only thing holding the name up. `fabledcurator-ml` has been the same bytes as `fabledcurator` since milestone 422 step 6, and 8152684 had already stopped rebuilding it — this removes the name. Gone: the whole `build-ml` job (137 more lines), its entry in promote's `needs` and in promote's tag loop, `ML_PATHS` and the `ml)` case in artifacts.sh, the `ml` artifact in test_artifact_paths, the image in release_notes.py and its test, and the README row. `docker-compose.yml`'s ml-worker service now runs `fabledcurator:latest` with `command: ["ml-worker"]` — the service still exists for anyone who wants lane separation, it just no longer needs a second image name. Prose that said "all three images" is now "both", except where it means three VALUES (revision/version/epoch) or records what happened on 2026-08-30, when there genuinely were three. Published tags are left alone. Nothing new goes to that name; what is already in the registry stays pullable, so a stack that has not moved yet keeps running rather than losing its image mid-flight. ## A latent trap this made reachable, found by running the script `artifacts.sh revision ml` did not fail. It printed the usage line to stderr and answered with **the newest commit in the whole repository** — a real-looking 12-char sha on stdout, exit 0. `newest()` inlined the path set as `git log ... -- $(cmd_paths "$1")`, and `usage` exits from the command SUBSHELL, so the substitution came back empty and `git log HEAD --` had no pathspec left to filter by. The reuse check would have compared that answer against a published label, missed, and rebuilt on every push forever with nothing going red. It could not be fixed inside `newest`, which was my first attempt: every caller wraps it in a substitution too, so its `exit` also died in a subshell — stdout went empty but the status stayed 0. The guard has to run in the main shell, so it validates the artifact name at dispatch. `newest` keeps a defensive `|| exit 2` for a future caller that reaches it another way, and the comment says which of the two is the real check. Latent while every name callers passed was valid. Removing `ml` from the set is what made a name that used to work start taking the silent path, so the test that pins it lands here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:30:47 -04:00
bvandeusenandClaude Opus 5	f0595e43e6	docs: record what the alias push actually costs — zero layer bytes (4311) ... The comment said the first run would answer whether the registry shares blobs across two repositories of one owner. Run 7300 answered it: every layer came back "Layer already exists" or "Mounted from bvandeusen/fabledcurator". The push moves a manifest and nothing else. 22 seconds, against the ~2 minutes the duplicate build cost (run 7282: build-ml 1m55s for output identical to build-web's). Also noted: the pushed manifest gets its own digest, because docker re-serialises it — same layers, same config, same fc.revision, different manifest bytes. Nothing compares the two, but the digests differing is the first thing someone will notice and misread as a second build still happening. Leaving a "we do not know yet" in the file once the run has said would make it the kind of stale note that misleads the next person to read it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 07:21:56 -04:00
bvandeusenandClaude Opus 5	815268418e	perf: stop building the same image twice under two names (4311) ... `fabledcurator-ml` has been the SAME IMAGE as `fabledcurator` since milestone 422 step 6 merged the ML layers into the one Dockerfile. `build-ml` was still running `file: Dockerfile, context: .` — byte for byte what `build-web` builds — and publishing the result under a second name. So CI did the whole build twice. Measured on run 7282, cold cache: build-web 1m54s, build-ml 1m55s, for identical output, plus a second push of a few hundred MB. Operator, 2026-09-22: *"please fix the CI so it doesn't do this superfluous work."* build-ml now publishes nothing of its own. It pulls the manifest build-web just pushed, re-tags it, and pushes that — so the second name goes on working for the operator's Swarm stack, which still references it, while CI stops compiling anything twice. 437 lines of duplicated build machinery go with it: its own buildx setup, its own reuse check, its own version derivation, its own tag repoint. It now `needs: [build-web]` rather than running in parallel. That is the one cost, and it is seconds: there is nothing to copy until build-web has decided what this run publishes, and the layers are already on the runner. Deliberately NOT `imagetools create`, which would have been the obvious one-liner. That wraps its source in an INDEX and `.Image.Config.Labels` does not resolve through one (#3183, run 4751) — and it would bite harder here than it did there, because `promote` asks the registry for IMAGE manifest media types ONLY, so an index at :refresh-candidate would fail its `test -n "$CT"` and break the weekly refresh for this name. Plain pull/tag/push keeps a plain manifest, which rule 40 also asks for. The step reads the tag back and fails if `fc.revision` stops resolving, so a future change that reintroduces index-wrapping says so instead of quietly costing the refresh. The tag list mirrors build-web's exactly, refresh included: a refresh writes :refresh-candidate alone and `promote` moves it to :latest after smoke-web passes. So this name keeps the gate it has today and `promote` needs no change — it already loops over all three image names. Source is build-web's `digest` when it built, else its `published_digest` on a reuse hit — the digest its channel tag already names, which is what a hit MEANS. With neither, the job fails with a reason rather than publishing something arbitrary. This does not retire the name; #4311 still does that, and is still gated on the operator's stack pointing ml-worker at `fabledcurator:latest` with `command: ["ml-worker"]`. Until then nothing of theirs changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 07:20:30 -04:00
bvandeusenandClaude Opus 5	187b6d2cdf	fix: the smoke always runs, so a change to the smoke can verify itself (4323) ... This workflow file is in no artifact's path set — correctly, since editing it changes no shipped byte. But smoke-web was gated on build-web having published something, so a commit touching ONLY the smoke moved no revision, hit reuse, emitted no digest, and skipped the smoke. The one commit whose purpose is changing this check was the one commit that could not run it. Twice already: 5ca1058 added the egress sandbox and went green three times with smoke-web SKIPPED; 7175ace fixed the bug hiding behind those greens (#4319) and needed a manual force_build to exercise. Both relied on someone remembering. It is also where the other historical skip lived — run 5290's `if:` read `env`, which a job condition cannot see, so it evaluated empty and skipped silently. Two skips, one expression. The expression goes. The job now smokes whichever manifest is current: the digest this run built, or — on a reuse hit — the one the channel tag already names, which the reuse step resolves anyway to read its fc.revision label and now exports as `published_digest`. Always a digest, never a tag (#4290). Kept separate from `digest`, which the :c-<sha> repoint reads and must go on meaning "what this run built". With neither available the job FAILS with a reason rather than passing quietly. Adding build.yml to WEB_PATHS would also make the smoke run, and would be wrong: fc.revision means "the commit this artifact's shipped files last changed in", so moving it for a CI edit makes the label lie and rebuilds three images for a change none of them ship. The problem was never the artifact's identity — it was when the guard runs. Two more defects in the same block, both from 5ca1058, both found by run 7288 — which only existed because the force_build above ran the check a second time: * `NET=smoke-noegress-$$` is not unique. The shell's pid is deterministic in this runner — every execution got 157 — so the second run died on "network with name smoke-noegress-157 already exists". A pid is unique among live processes, which is not unique over time. * The network's cleanup trap was destroyed before it could fire. `trap ... EXIT` REPLACES the previous handler, and the container's trap was installed further down, so every run leaked its network. Invisible in a passing run; it can only ever surface on the NEXT one. Now one EXIT handler does both, armed beside the network and tolerant of an empty CID so it covers failures before the container exists. Plus a sweep of any network earlier runs leaked, which fails harmlessly on one still in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 11:20:36 -04:00
bvandeusenandClaude Opus 5	7175ace67a	fix: the smoke's health probe has to run inside the sandbox it created (4319) ... The first real execution of the egress-blocked smoke (run 7282) failed with "web is running but never answered /api/health" — and the application was perfect. Its own log shows all four hypercorn workers serving three seconds after start and still up seven minutes later, with no internet: [entrypoint] Starting hypercorn on :8080 [14:08:58] [10] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [11] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [12] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [13] [INFO] Running on http://0.0.0.0:8080 5ca1058 put the app container on an `--internal` network. Docker gives such a network isolation rules that DROP traffic entering it from any other interface, and this job's own container sits on the runner's default bridge — so its curl to the app was discarded before arrival. Dropped rather than refused, so every attempt burned the full --max-time and the loop read as a wedged app instead of an unroutable address. Steps 0-2 were right only by accident: each already runs its check inside a container on $NET. Step 3 was the one place that reached in from outside, and so the one place that could not work. It now probes from inside too, using the image's own python3 over `shell -c` — the same shape as the egress guard above it, and necessary because the runtime stage ships no curl. Attaching the job container to $NET would also work in one line. Rejected: it puts an internet-connected container on the network whose whole purpose is being offline, and it would rest on `hostname` equalling the container id. Same family as #3374 — a CI check that could never pass, failing in a way that accuses the thing it was meant to protect. Worth stating plainly: the egress property itself PASSED on 7282 ("smoke: no route out, as required") and the schema built from empty through 0104 with no network. Only the harness's last step was broken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 10:16:17 -04:00
bvandeusenandClaude Opus 5	5ca1058fb5	test: the smoke runs with egress blocked, on every build, and proves the block (4296) ... Milestone 422 step 6's rule-164 half. THE PROPERTY WAS NEVER TESTED, not even weekly. smoke-web already booted the built image against real Postgres and Redis — but on the runner's default network, with the internet one hop away. It proved the image WORKS; it never proved it works OFFLINE, which is the thing rule 164 is about and the thing step 6 put at risk by moving download_models out of boot. Now on an `--internal` docker network, which is the mechanism rule 164's own verify_with names. `--network none` is explicitly the wrong check here: it would only prove the app fails without a database, which says nothing about egress. Internal blocks the default route while leaving container-to-container traffic and embedded DNS intact, so Postgres and Redis stay reachable and nothing else does. The service containers are runner-created siblings, so they are ATTACHED to the network rather than created on it, and their addresses are re-read on it — the bridge IPs discovered earlier are not routable from a container that is only on the internal network. A STEP THAT PROVES THE SANDBOX IS SEALED, before anything depends on it. It tries to reach 1.1.1.1:443 from inside the candidate and fails the job if it succeeds. Without it the rest is theatre: if `--internal` silently stopped working, or the container picked up a second network, every check below would pass with the internet available and report an offline boot that never happened. A guard that cannot fail is not a guard (rule 167). IT RUNS ON EVERY BUILD, not just the weekly refresh. The egress property is broken by a code or Dockerfile change — a push — so checking it only on the refresh would test it on the one trigger that changes no source. Addressed by the DIGEST build-web published rather than by a tag: a tag can move between the build and the smoke, and then the check reports on bytes nobody built here. A reuse hit is skipped, because those bytes were smoked when built. WHAT THIS STILL IS NOT, filed as #4310: on a push it runs AFTER build-web has written the channel tag, so it detects rather than gates. Rule 164 asks for the check BETWEEN build and push. Closing that means giving the push path the candidate-then-promote shape the refresh already has — per-channel candidate tags, promote learning its channel, and the :c-<sha> repoint moving after the gate. That is a redesign of the path that publishes production and it is not something to fold into a test change. Also filed #4311: retiring the fabledcurator-ml image NAME, gated on the operator moving their Swarm stack file. Same two-phase shape #406 used for pixiv, for the same reason — dropping it would not break their deploy, it would freeze it silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 09:50:25 -04:00
bvandeusenandClaude Opus 5	ffcd13096a	feat: one image for every lane, with the model fetch gated on enabling (4296) ... Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:51:44 -04:00
bvandeusenandClaude Opus 5	aa2bb665b9	fix: serialise builds per branch, and pin :c-<sha> to the digest this run built (4290) ... Two independent defects from one missing mechanism. `grep -rn concurrency .forgejo/workflows/` returned nothing, so two pushes to a branch ran build.yml in full parallel with no ordering. Both jobs read `fc.revision` off the channel tag before either has pushed, so both miss the reuse check and both build. Whichever finishes LAST owns the tag. Half 1 — the rolling tag. A slower older build leaves `:dev` carrying content older than the commit that moved it. Family rule 146 says a rolling channel refreshes itself; this is the case where it quietly does not. Self-healing on the next push. Half 2, and the reason this is not filed low — `:c-<sha>`. The repoint step wrote every non-channel tag by copying the channel tag BY NAME, so the immutable rollback tag named whatever `:latest` pointed at when that step happened to run. Lose the race and `:c-<shaA>` names the OTHER run's bytes. Rule 145 makes that tag the rollback unit and immutable, so this does not break immutability — it makes the tag wrong from birth, and immutability then guarantees nobody ever corrects it. Nothing goes red; it surfaces the day someone rolls back and gets a commit they did not choose. Not self-healing. Two fixes, deliberately both: A. A workflow-level `concurrency` group keyed on `github.ref`, so dev and main never block each other. `cancel-in-progress: false` — queue, never cancel: cancelling could kill sign-extension mid-AMO-upload, leaving the version registered at AMO with no cached asset, which is the unrecoverable stuck state that job's rollback trap exists to prevent, reached by another door. Not keyed on BUILD_REF because the group is evaluated before any job starts and cannot read the `env` context. B. Each build-push-action step gains `id: build`, and the repoint step copies from `$IMAGE@${{ steps.build.outputs.digest }}` — the manifest THIS run pushed — rather than from the channel tag by name. On a reuse hit there is no digest and the channel tag remains correct by definition: "hit" means that tag already carries this commit's fc.revision. B is not redundant with A. A depends on this Gitea honouring a key whose failure mode is silent, and this file has been burned by exactly that before (the `format()` note records `true == 'true'` evaluating FALSE on run 5270, every lane green, the feature simply not happening). B holds the :c-<sha> correctness property whether or not A is honoured. The loop's exclusion is now keyed on CHANNEL_REF rather than on SOURCE, and that is load-bearing: SOURCE may now be a digest ref, which never equals a tag string, so testing against it would stop excluding the channel tag. imagetools would index-wrap it and `.Image.Config.Labels` would stop resolving, killing the reuse label permanently — #3183 arriving again. Neither half is verifiable on a dev push: dev's tag list is the channel tag alone, so the repoint correctly does nothing there. B is observable on the next merge to main (:c-<sha>'s digest must equal the build step's), and A by pushing twice in quick succession and reading the run list for a queued second run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 22:07:53 -04:00
bvandeusenandClaude Opus 5	059e2128ee	fix(ci): stop telling readers the runner can't run upload-artifact@v4+ ... build.yml and baseline.yml both explained a missing artifact step by saying act_runner cannot run actions/upload-artifact@v4+ (and baseline.yml said ci-requirements.md records that, which it does not). The runner is now gitea/runner 3.x, which edits the action's GHES refusal out of its bundle, and stock v4+ is proven working on this forge (Scribe spike #3843). Comments only. Neither lane gains an upload step: build-web still reads the signed XPI from the release asset, and the baseline candidate is still printed to the log, because those remain the better channels. The comments now say why for the right reason. Scribe snippet #2271, milestone 395. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DwoKYuw3qJmUUYsJeNherB	2026-09-10 17:43:36 -04:00
bvandeusenandClaude Opus 5	131237143b	Revert "test: force the smoke gate to fail, to watch it block a publish" ... The gate held. Run 5320, dispatched with the forced failure in place: build-web success (candidate published) build-ml success build-agent success smoke-web FAILED promote skipped run failure And the three channel tags did not move: fabledcurator 33d3d8332f74 -> 33d3d8332f74 fabledcurator-ml e94a5435cb45 -> e94a5435cb45 fabledcurator-agent bae27d34d811 -> bae27d34d811 So a refresh that breaks something now leaves :latest naming the build that works, which is the property milestone 362 exists to establish. The rejected candidate is still published under :refresh-candidate, so whoever reads the red job on Monday can pull the exact image that failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:23:49 -04:00
bvandeusenandClaude Opus 5	59d27ef76e	test: force the smoke gate to fail, to watch it block a publish ... TEMPORARY, reverted in the next commit. Milestone 362's verification section requires the gate to be seen rejecting a build — a gate nobody has watched reject anything is a gate nobody knows is wired up. Every real check passes, so the rejection has to be forced. Under test is the job dependency, not the assertions: a failed smoke-web must skip the promote job, and the three :latest tags must still name the digests they named before the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:22:01 -04:00
bvandeusenandClaude Opus 5	f630e50e75	ci: the refresh publishes only what the gate passed (#3265 milestone step 4) ... The gate reported a verdict nothing consulted. Now it decides. The promote moved out of the three build jobs into its own `promote` job, because the verdict cannot exist until build-web has finished and the promote used to run inside it. `needs: [build-web, build-ml, build-agent, smoke-web]` is the whole mechanism: a failed smoke skips the promote, so a refresh that broke something leaves :latest naming the build that works. "The refresh failed" and "production is broken" must not be the same event. A SKIPPED smoke also skips it, and that is the case that matters most. On run 5290 the gate silently skipped itself — job-level `if:` cannot read the env context — and a design where only a FAILED gate blocks would have published unverified images while reporting success. Not running is not the same as passing, and today produced two separate bugs of exactly that shape (#3414, and the smoke-web skip). All three images now promote together or not at all. They are one stack: build.yml already refuses to publish a :dev web image beside a stale :dev ml because the mismatch only surfaces as a runtime failure, and a refresh that published ml while withholding web would be that same trap reached through the gate. Stated plainly in the job comment: the gate covers web only, so ml and agent are held to web's verdict rather than their own. That is the conservative direction, not equivalent evidence, and should not be read as if it were. Three near-identical promote steps collapsed into one loop. A partial failure now says which images moved and that the state is inconsistent, rather than leaving that to be inferred — the promote is idempotent and the candidates are still published, so the instruction is simply to re-run. Also removed the now-dead `promote` output from the ml and agent reuse steps. Only build-web's is read (as outputs.candidate); two more copies nothing consults is the kind of thing that reads as load-bearing a year later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:21:27 -04:00
bvandeusenandClaude Opus 5	4815040d74	ci: the smoke gate found a real one on its first run — and had two bugs of its own ... Run 5296 was `smoke-web`'s first genuine execution. Checks 1 and 2 passed: alembic built the schema from empty inside the image, all five apt binaries resolved, and the application's own Thumbnailer produced JPEG, PNG-with-alpha, WebP and an ffmpeg video frame against the image's libraries. Check 3 failed, and the trap's log dump said exactly why: MissingCredentialKey: Fernet key file not found at /images/secrets/credential_key.b64. For first-time setup, set CURATOR_BOOTSTRAP_NEW_KEY=1. That is the product being right. credential_crypto refuses to mint a key unless someone opts in, because the 2026-06-02 audit found a partial restore (DB back, /images/secrets/ lost) silently generating a fresh one and leaving a working-looking system where every authenticated download failed AUTH_ERROR. It is also a first-run blocker for milestone 328, filed as #3422: the variable appears in no README, no .env.example and no compose file, so the install path that milestone just finished writing produces a container that exits on boot. Not fixed here — the fix trades safety against friction and is the operator's call. Two defects in the gate itself, both surfaced by the same run: - A throwaway CI instance IS first-time setup, so it now passes CURATOR_BOOTSTRAP_NEW_KEY=1. The check was asserting a condition no fresh container can satisfy. - The health loop polled a dead container for 3m35s. Docker had already recycled its IP, so the replies were a baffling mix of connection-refused and 5s timeouts from whatever took the address next. It now checks `.State.Running` each iteration and fails immediately with the container's log. The trap had the real answer the whole time; this stops burying it under four minutes of noise. Also corrected a message claiming a 120s budget: 60 iterations of up to 5s connect plus 2s sleep is nearer seven minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 16:00:18 -04:00
bvandeusenandClaude Opus 5	81b7b6f308	ci: smoke-web never ran — a job's if: cannot read the env context ... Run 5290 dispatched a refresh. Everything worked: the guard fired, the build published the candidate, the promote pointed :latest at it. And `smoke-web` reported conclusion "skipped", with no steps and no log. Its condition was `if: env.IS_REFRESH == 'true'`. The env context is available to STEP conditions and step bodies but never to a job's own `if:`, and an unresolvable context there evaluates to empty rather than erroring. So the gate skipped itself, silently, on the one run that existed to exercise it. Second silent-skip of this family today, after #3414. Same shape both times: something evaluated false, nothing failed, and the run reported success. It is worth naming the pattern — on this pipeline, "green" and "ran" are different claims, and the steps' own conclusions are the only place the difference shows. Fixed by keying off a job output rather than re-deriving the trigger: build-web now exposes the reuse step's `promote` decision as `outputs.candidate` and smoke-web consumes it. That is better than duplicating the expression: it is the same single decision the build, the XPI download and the promote all take already — build.yml's own "one decision drives everything downstream" — and it asserts the thing smoke-web actually depends on, that a candidate was published, rather than restating the reason one would be. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 15:53:40 -04:00
bvandeusenandClaude Opus 5	bfa9fd678b	ci: smoke the refreshed image against real Postgres and Redis ... Milestone 362 step 3. This is the gate the weekly base refresh never had. `ci.yml` cannot be that gate, and the reason matters more than the fix. Its lanes run on ci-python:3.14 and install requirements.txt — a base refresh changes neither, so all five stay green through a bump that breaks the product. What a refresh re-resolves is the Dockerfile's apt layer: ffmpeg unar libpq5 postgresql-client zstd megatools libjpeg62-turbo libwebp7 libpng16-16 ca-certificates Unpinned, every build, and nothing else in this repo looks at it. That line is the dependency creep; it is also precisely what the test suite structurally cannot observe, since the suite never runs inside the image and the image carries no tests and no pytest. So `smoke-web` runs the CANDIDATE IMAGE against real service containers: 1. `alembic upgrade head` on an empty database — the image's own libpq and psycopg, and the same call entrypoint.sh makes before it serves anything, so a failure here is a failure to boot. 2. The apt binaries, then the application's own `Thumbnailer` — JPEG, PNG with alpha, WebP, and a video frame through ffmpeg. `Thumbnailer` needs no database and no app context, so the check exercises real product code rather than a proxy for it. `ffmpeg -version` exiting 0 would pass while a codec removal broke every thumbnail in the library. 3. The web role boots and answers /api/health. Every failure names the package it implicates. This fires on a Sunday, unattended, about a change nobody made deliberately — "assertion failed" a week later teaches nobody anything. The script is piped over stdin rather than bind-mounted: the workspace is a docker volume belonging to the job's own container, so a host bind of $PWD does not resolve for a sibling. Container logs are dumped only on failure, and the trap re-exits with the real status rather than the status of `docker rm`. Deliberately NOT gating the promote yet — that is step 4. Landing the gate and the thing it gates together would mean the first time anyone saw this job run would also be the first time it could stop a publish. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 15:45:10 -04:00
bvandeusenandClaude Opus 5	24a2b70a5a	ci: a boolean input never equals the string 'true' ... The refresh lever did not work, and the way it did not work is the point. Run 5270 dispatched with refresh=true. Its log: expression '(github.event_name == 'schedule' || github.event.inputs.refresh == 'true') && 'true' || 'false'' evaluated to '%!t(string=false)' trigger: event=workflow_dispatch IS_REFRESH='false' BUILD_REF='refs/heads/dev' trigger: raw inputs refresh='true' force_build='false' The input arrived as true and the comparison still said false. `type: boolean` delivers a real boolean, and GitHub expression semantics cast operands to numbers when their types differ — so `true == 'true'` compares 1 against NaN. My comment on the previous commit asserted the opposite, that Forgejo delivers inputs as strings, and asserted it without checking. The run went GREEN with every step skipped, because a refresh that evaluates false is indistinguishable from an ordinary push. A lever that silently does nothing is worse than no lever: it would have been trusted. Normalised through format(), which is representation-independent — a boolean true and a string 'true' both render 'true'. That is also why force_build was never bitten: it passes its raw value into an env var and compares in the shell, where everything is a string already. format() buys the same thing at expression level, which is where a step `if:` needs the answer. The diagnostic from the previous commit stays. It is what turned this from a guess into a measurement, and it is the only thing that would catch the same class of failure next time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:59:48 -04:00
bvandeusenandClaude Opus 5	2c88ad3efb	ci: report the raw and normalised trigger values ... The refresh dispatch on run 5265 went green with every step skipped: the main-only guard did not fire, checkout took dev, and the reuse step read IS_REFRESH as false. So both workflow-level expressions evaluated false while the identical accessor works for force_build, which compares its value in the shell rather than in an expression. That is a guess until it is measured, and the failure is silent by construction — a refresh that evaluates false behaves exactly like an ordinary push and reports success. This prints the raw input beside the normalised value in the step that already exists to say what a run derived, so the two disagreeing is visible rather than inferred. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:58:20 -04:00
bvandeusenandClaude Opus 5	bfc4f9cec9	ci: one fact for "is this a base refresh", and a lever to trigger one ... Milestone 362, enabling step 2's verification and everything after it. The weekly refresh was testable once a week. That is not a cadence anything can be developed against, and milestone 362's whole point is a gate — which has to be watched rejecting something before anyone can believe it is wired up. So `refresh` joins `force_build` as a dispatch input, on the same reasoning that added that one (#3252: confirm #3190 was gone rather than wait for it to recur). Adding it meant confronting that "is this a refresh?" was asked in five places and spelled five ways: `github.event_name == 'schedule'` in an `if:`, `$GITHUB_EVENT_NAME` in one shell, an `EVENT:` env passed into another, and a bare expression on `pull:`. Five spellings of one fact is how half of them come to disagree once somebody adds a sixth trigger — which is precisely what this commit is. So it is derived once at the top, next to BUILD_REF, which already exists for exactly this reason on exactly this question. String comparison, not boolean: Forgejo delivers dispatch inputs as strings, so `inputs.refresh` is 'true'/'false' and `&&` on it would read the string 'false' as truthy. **A constraint this makes visible, which pre-dates it.** A refresh checks out `main` (BUILD_REF) while running the workflow definition from the branch that triggered it — the cron registers from the default branch. So dev's workflow builds main's source, and dev's workflow cannot depend on anything main's tree does not have yet. It does now: the reuse step calls `artifacts.sh epoch`, which lands on main with this batch. Until then a refresh dispatch fails loudly at that call, which is the right failure — the alternative is tolerating a missing epoch and silently rebuilding #3265 into every refresh. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:41:33 -04:00
bvandeusenandClaude Opus 5	b590d25f8f	ci: the scheduled refresh builds a candidate, then names the channel ... Milestone 362 step 2. Structural: it creates a moment between "built" and "published" for step 3's gate to occupy. No behaviour change. A refresh rebuilds against freshly resolved base images, and the web image's runtime is a line of UNPINNED Debian packages — ffmpeg, libjpeg62-turbo, libpq5, megatools — re-resolved on every build. Nothing in ci.yml can see that: its lanes run on ci-python:3.14 and install requirements.txt, and a base bump changes neither. So refreshed bytes need proving before :latest names them, and proving needs somewhere to stand. On a push nothing changes: build_ref IS channel_ref, promote is false, and the build writes the channel tag directly the way it always has. On the schedule the build writes :refresh-candidate — one moving ref per image, overwritten in place, holding a build nobody is told to pull. That is the shape rule 145 already allows for :buildcache, not the per-build tag family 318 withdrew. Both values are decided in the reuse step beside `hit`, because that step already owns "what does this job do" (build.yml's own rule, at the force branch). A promote condition derived somewhere else could disagree with the tag the build actually wrote. **The promote is a manifest PUT, not `imagetools create`.** That distinction is the whole risk in this change. `imagetools create` wraps its source in an index, and an indexed channel tag is the one thing this pipeline cannot survive: `.Image.Config.Labels` does not resolve through an index, so the fc.revision the reuse check reads back would come up empty, every later push would miss and rebuild, and nothing would go red. That is #3183, observed on run 4751 — reuse worked exactly once and the only symptom was the bill. The repoint step already excludes its own source tag for this reason; a promote that re-introduced the wrap through another door would undo that care. A manifest PUT is what "make this tag name that image" means at the registry: same bytes, same media type, identical digest, no layer transfer. It reads the result back and fails if the tag does not name what was just written — a PUT that 2xx'd and landed something else is exactly the silent-and-plausible failure this pipeline keeps producing. Every call carries a deadline (rule 156); a registry that stops answering must fail the step, not hang the weekly refresh until the job times out. Promote is UNCONDITIONAL today, deliberately. Gating it before the gate exists would leave the refresh building something and publishing nothing for as long as this milestone takes. Step 4 wraps it in the smoke suite's verdict. Not yet verified on the refresh path — that needs a scheduled run, and the lever to trigger one on demand is the next commit. This one is verified by the push path being untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 14:40:19 -04:00
bvandeusenandClaude Opus 5	635138b0d1	ci: pin the build clock to the commit, so an unchanged refresh publishes nothing ... Milestone 362 step 1, closing #3265's root cause. The weekly base refresh rewrote all three `:latest` tags on 2026-08-30 with nothing changed in any of them. Not a cache miss — run 4934's log shows every content step CACHED and both bases resolved to unchanged pinned digests. buildkit stamps the image config with the wall clock of the build, so identical layers get republished under a new config blob and therefore a new manifest digest. The cost is not storage, it is meaning: `:latest` moved on a calendar, so a digest change stopped being evidence that anything was different. That is the one thing a digest is any use for, and it is load-bearing here — the reuse check, the `:c-<sha>` rollback story and any future redeploy signal all rest on it. SOURCE_DATE_EPOCH normalises `created` and the history timestamps, so the same source produces the same config bytes and the same digest, and pushing it is a registry no-op. The value is routed through artifacts.sh's existing `newest()` rather than taken from git separately. `revision`, `version` and now `epoch` are three fields of ONE lookup, so they cannot drift into naming different commits — a divergence that would stamp an image reproducibly against one commit while it reported being another, with both values looking perfectly well-formed. Note #3127 §2 is the record of what a second clock costs; this adds a view, not a clock. Also corrected: the build step comment and ci-requirements.md both described the churn as current behaviour with the fix as a "likely" future. They now describe what the file does. Tests pin the property the fix depends on, not the fix: epoch is the same commit version names, in both renderings including the extension's unpadded one, and it does not move between two calls on one checkout. A future refactor that gave epoch its own `git log` would pass every other test in that file. Not yet verified end to end — proving it needs two consecutive refreshes to land on the same digest, which is the next thing, and is the step #3265 exists because nobody did last time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 13:46:16 -04:00
bvandeusenandClaude Opus 5	3590c478f5	docs+ci: folder import stays retired, and fix a readiness probe that never probed ... Two unrelated things, both found while closing out milestone 328. **Folder import (#3367).** The operator's call, this session: the import-from-file surface was abandoned on purpose and is not coming back — "it has its own complexities that we didn't need." The README and the compose comment both described the missing button as a rough edge with a tracking issue, which promised a fix that is not coming. Both now say the retirement is the decision, name Subscriptions as the supported way to fill a new install, and describe /api/import/trigger as an unsupported escape hatch for anyone who wants to script one. **The CI readiness probe.** ci.yml's integration job and baseline.yml both waited for Postgres with `(echo > /dev/tcp/$PG_IP/5432)`. Those steps run under `sh -e` — act's default shell — where /dev/tcp is not a magic path but a filename that does not exist. The probe could therefore never succeed: run 18035, a GREEN run, spends 05:20:53 → 05:22:53 in that loop and exits it by exhaustion, not by connecting. Every integration run has been paying a flat 120s for a check that established nothing, and proceeding regardless. Replaced with a socket connect in python (present in the image, no package needed), and exhausting the budget is now a named failure instead of a silent fall-through — rule 156. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 00:51:25 -04:00
bvandeusenandClaude Opus 5	973db73221	db: collapse alembic 0001..0089 into one baseline (#3266) ... 89 files and 6,300 lines become one file of 807. Nothing about the resulting schema changes; what goes away is the requirement that a new installation replay our development history to arrive at it. revision = "0089", down_revision = None. That pairing IS the migration strategy for existing installs, not a detail of it: a deployed database already has alembic_version = '0089' from running the real 0089, so alembic reads the version table, sees head reached, and does nothing. No stamp is required — which matters, because `alembic stamp` writes a version string without validating anything about the schema it is writing it against, and a wrong stamp is indistinguishable from a right one until the next migration fails. An empty database runs the file and records 0089. Both paths converge. The next migration is 0090, as it would have been; the numbering is continuous across the collapse on purpose. Autogenerate produced nearly all of this unaided, which was NOT true of the first attempt — that one was reverted because the generator silently dropped eleven indexes and three uniqueness guarantees. #3275 put those on the models first, so the HNSW index with its opclass, the COALESCE expression index, the partial uniques, 107 server_defaults and the enum CHECKs are all emitted now. Doing the reconciliation before the squash, rather than after, is what made this work. Hand-added, because none of it can live in a model: * CREATE EXTENSION vector / tsm_system_rows (0001, 0004) — database objects, not table metadata. * The pgvector import. Autogenerate writes qualified pgvector.sqlalchemy.vector.VECTOR references without importing the package, so its own output cannot run (run 4988). * THE TWO SEED ROWS. 0002 and 0003 did not only build schema — each inserted a settings singleton, and nothing in the app ever creates them: ImportSettings.load() and MLSettings.load() are select(...).scalar_one(), which RAISES NoResultFound rather than returning None. A models-only baseline would leave both tables empty and crash a fresh install on first settings access, while baseline.yml reported a perfect schema match. Only running the app against a new database finds that. Not carried over: 0023's DELETE FROM tag and 0047's series deletes, which are historical cleanups operating on rows an empty database lacks. downgrade() raises. A baseline's downgrade is "drop every table", which is a data-loss event wearing a migration as a disguise; offering it as one invites someone to run it. Restore from a backup. Also removed, per the plan: the 10 test_migration_*.py files (they assert intermediate states and backfills that no longer exist — a test that a column exists is already the model tests' job) and backend/app/utils/artist_backfill.py, whose only importer was 0008. Verified no other consumer anywhere in backend/ or tests/. baseline.yml changes with it. chain_ref now DEFAULTS to 725bf15, since the tree no longer carries a chain to compare against — that pinned commit is the last one that does. And the CheckConstraint repair is removed, because it never fired. I added it claiming autogenerate re-doubles a constraint name on the round trip and asserted in b979062 that it was "still correct and still needed". It is not: autogenerate wraps names in op.f(), which marks them already-formatted and blocks the convention from re-applying. Tested against the real candidate line — the regex matches nothing. What actually fixed the mismatch was 0088's renames alone. The doubling is a real hazard, but of hand-writing a pre-prefixed name, not of the generator; the comment asserting otherwise was worse than the dead code under it. The header comment is rewritten for the same reason — it described 87 revisions, and claimed the HNSW index could not be expressed in a model, which #3275 disproved. It now also states plainly what this check CANNOT see: it compares schema, so a green run means the schema is right, not that the baseline is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-09-01 01:14:37 -04:00
bvandeusenandClaude Opus 5	b979062dd7	db: rename the four double-prefixed CHECK constraints (#3275) ... Run 5026 got the models-vs-chain diff to 7 lines. Three findings, and one of them reverses an assumption I made in the previous commit. The doubled CHECK names are what the DATABASE has, not what the generator invented. base.py's convention is ck_%(table_name)s_%(constraint_name)s, which — unlike uq/fk/ix — applies even to a constraint that already has a name, so four migrations that passed an already-prefixed name got it prefixed twice: ck_import_settings_ck_import_settings_singleton ck_ml_settings_ck_ml_settings_singleton ck_post_ck_post_translation_override ck_tag_ck_tag_fandom_requires_character The workflow repair added last commit is still correct and still needed — autogenerate really does re-double a name on the round trip — but it was making the MODELS side clean against a chain that is dirty. The comment in ml_settings.py claiming its bare name "matches migration 0003" was simply false; 0003 produces the doubled form. Nothing reads a CHECK constraint by name, so this has never done harm. But it is precisely the development-era residue the collapsed baseline exists to leave behind, and a public schema should not ship it — so 0088 renames the deployed constraints and all six models now declare bare names. RENAME CONSTRAINT is catalog-only: no scan, no rewrite, no revalidation, which is why this is safe on post and tag. Guarded on pg_constraint scoped by conrelid, so it is a no-op on a database built from the models. ix_tag_fandom_id showed as a difference only because chain_ref was pinned to 0a5bbe8, which predates 0088 — the comparison was measuring the models against a chain missing the migration that closes the gap. chain_ref now defaults to blank, meaning "the chain in this ref". Pin it to a commit only after the collapse, when the tree no longer carries the revisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:29:57 -04:00
bvandeusenandClaude Opus 5	573228b9da	db: finish reconciling the models with the deployed schema (#3275) ... Closes the residue the first reconciliation pass left, and corrects a factual error I put into the record. sha256 was NOT missing a uniqueness guarantee. I read `op.create_index("ix_image_record_sha256", ...)` at 0001 line 151 and concluded duplicates were possible, without reading line 149 two lines above it: sa.UniqueConstraint("sha256", name="uq_image_record_sha256"), Uniqueness has held since the initial schema. The database expresses it as a CONSTRAINT plus a separate non-unique lookup index; the model said `unique=True, index=True`, which is one UNIQUE index under a different name. Same guarantee, different objects — which is exactly why the two schemas did not line up. The model now declares both objects. No DDL. 0088's docstring, which repeated the claim, is corrected in place. Two real divergences, both the MODEL over-claiming: * source: uq_source_artist_platform_url (alembic 0010) was declared nowhere in the models — source.py had no __table_args__ at all — so autogenerate would have proposed DROPPING it. * head_metrics_snapshot.tag_id: model said NOT NULL, 0060 created it nullable. Left nullable; the FK already cascades. Seven constraints renamed to what the chain actually created, rather than what base.py's naming convention renders: uq_series_page_image, uq_series_chapter_anchor_page, fk_series_chapter_anchor_page, fk_image_record_artist_id, fk_image_provenance_from_attachment, and the two hand-shortened fk_tsr_* names from 0003. Float server_defaults now mirror their own migration, per column. The chain is MIXED: a plain string renders DEFAULT '0.90'::double precision, sa.text() renders DEFAULT 0.90, and the migrations used both. Seven columns take text(); the rest stay strings. Two literals also disagreed outright — process_{auto_apply,conflict}_threshold said 0.9/0.5 against the migration's 0.90/0.50. baseline.yml gains two things. A repair for a SECOND generator defect in the same class as the missing pgvector import: base.py's ck convention contains %(constraint_name)s, so it applies even to a NAMED CheckConstraint — autogenerate writes the already-rendered name into the migration and running it applies the convention again, yielding ck_ml_settings_ck_ml_settings_singleton. That is round-tripping damage, not a claim the models make, so it is undone rather than counted. And the diff now runs twice. Column ORDER differs permanently between a schema built by 87 ADD COLUMNs and one built in a single shot — the operator's database keeps chain order forever, a fresh install gets model order — so a check that failed on it could never pass. The second pass SORTS column lines within each CREATE TABLE instead of deleting them, which cannot hide a column present on one side only, or one whose type, nullability or default differs. Ordered diff is reported as information; the order-insensitive one is the verdict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017QHszn9H8VBvx5Ke8x1hvw	2026-08-31 00:24:00 -04:00
bvandeusen	d044e93bdb	ci: repair autogenerate's missing pgvector import before applying (#3275) ... mode: models applies the raw autogenerated candidate, and it cannot run: sa.Column('weights', pgvector.sqlalchemy.vector.VECTOR(dim=1152), ...) NameError: name 'pgvector' is not defined Alembic emits the qualified reference without emitting the import. Observed on run 4988, which turns this from a thing I predicted by reading the candidate into a thing demonstrated by executing it. Repaired in the workflow rather than counted as a schema difference: the comparison asks whether the MODELS describe the schema, and this is a defect in the generator. The same fixup has to be applied by hand to any baseline generated this way, which is why it is item 4 on the collapsed baseline's hand-written list.	2026-08-30 14:47:29 -04:00
bvandeusen	ed2b1adc2e	ci: compare the schema the MODELS produce against the migrations (#3275) ... baseline.yml only ever compared migrations against migrations. The question #3275 exists because nobody had ever asked the other one: does a database built from the MODELS match the one the chain produces? `mode: models` answers it. It applies the candidate autogenerated from the models instead of this tree's revisions, and diffs that against the chain. A clean run means --autogenerate is trustworthy again, which it demonstrably has not been: against the pre-reconciliation models it would have proposed dropping eleven indexes and two uniqueness guarantees. The two extensions are created by hand in that mode. They are database objects rather than table metadata, so no model can carry them — their absence is outside what this comparison asks about, and silently tolerating it is correct rather than a filter that hides a defect. Also declares the HNSW index on the ImageRecord model. SQLAlchemy can express an hnsw access method with an operator class (postgresql_using + postgresql_ops), so there was never a reason for it to live only in 0036. That removes the last item from the list of things a generated baseline cannot reproduce, leaving only the two extensions.	2026-08-30 14:43:26 -04:00
bvandeusen	98b56330d0	ci: emit the chain schema dump for local reconciliation work (#3275) ... Reconciling the models against the deployed schema needs the actual pg_dump, not an inference from the unified diff. Parsing table context out of diff hunks drops every table whose CREATE TABLE line falls outside a hunk — it under-reported 81 columns across 13 tables when the real figure spans more, missing artist, gpu_job, download_event and external_link entirely. Same checksummed-base64 transport as the candidate baseline, for the same reason: a plain cat of a file this size was silently truncated mid-line by the runner on run 4964.	2026-08-30 14:35:02 -04:00
bvandeusen	8f1ac0c96a	ci: transport the candidate baseline as verifiable base64 ... Run 4964 passed the control (1121 normalised lines, schemas identical) but its candidate print was silently truncated. `cat` of the ~33KB generated file stopped mid-line inside sa.Column('mime', sa.String(length=128) and the runner carried straight on to the next traced command with the step still green. The captured text was 484 lines and 29 tables, and looked entirely plausible — which is exactly what makes it dangerous: a schema definition cut in half is still syntactically suggestive, and nothing in the log says it was cut. Now emitted as base64 at a fixed 120-column width, followed by a sha256, a byte count and a base64 line count. Short lines instead of long ones, and more importantly the receiving end can PROVE it got the whole file rather than trusting that it did. Also found in that output, and the reason the candidate could never have been committed as-is: it references pgvector.sqlalchemy.vector.VECTOR(dim=1152) for head_training_run.weights and image_record.siglip_embedding, but autogenerate does not add the corresponding import. The file would die with NameError on the first run. That is the fourth item on the list of things the generator cannot be trusted with, alongside the two CREATE EXTENSIONs and the HNSW index.	2026-08-30 13:40:15 -04:00
bvandeusen	5fd171a544	ci: fix two things the baseline control run found ... Run 4960 was the control — the chain compared against itself, which must come back clean before a clean diff after the squash means anything. It did its job and failed on both counts. 1. The harness is sound. Both dumps came back 1123 normalised lines and differed on EXACTLY two, the \restrict / \unrestrict pair that newer pg_dump emits to fence a dump against injection during restore. It is a fresh random nonce per invocation, so it differs by construction and is noise by definition. Now filtered — and the control is what licenses that filter: it was OBSERVED to be the only false positive rather than assumed to be one, which matters for a check whose whole value is that its normalisation does not hide a real difference. 2. The candidate-baseline step never ran. `if: github.event.inputs .generate == 'true'` on a `type: boolean` input silently evaluated false — no diagnostic, step skipped, job carried on. The same `github.event.inputs` typing quirk build.yml already works around for force_build. Rather than fight the input typing, the gate is now the tree itself: skip if alembic/versions holds one file. That is the real question anyway — there is nothing to generate once the chain is collapsed — and it cannot be silently wrong the way an unevaluated expression can. Worth noting what the control also proved incidentally: the two schemas were byte-identical across 1123 lines despite being built by separate alembic runs into separate databases, so pg_dump's object ordering is stable enough to diff directly and no sort normalisation is needed.	2026-08-30 13:35:26 -04:00
bvandeusen	62583791d8	ci: a workflow that proves a collapsed alembic chain matches the old one ... Milestone 328 step 1 needs a baseline generated from the models, and step 2 must not stamp the operator's live database until that baseline is proven to reproduce what the 87-revision chain produced. `alembic stamp` validates nothing, so an unproven baseline fails silently now and loudly later, on real data. There is no local Python environment and rules 10/12 point away from standing one up, so the comparison runs in CI, where a pgvector Postgres is already built from the chain on every integration run and nothing is at risk. It builds two databases and diffs their pg_dump --schema-only output: one from `alembic upgrade head` on the revisions read out of git at `chain_ref`, one from the current tree. Reading the chain from git via a worktree — rather than from the working tree — is what keeps this usable AFTER the old revisions are deleted, so it is the proof for step 1 and the pre-flight for step 2 rather than a one-shot script. Both sides use `alembic upgrade head`, never metadata.create_all, per rule 82 — and that rule's reasoning is exactly the hazard here. `create_all` emits plain CREATE TABLE and skips everything else, which is why the optional autogenerated candidate CANNOT be trusted as the answer. Three things in this schema are invisible to SQLAlchemy metadata: CREATE EXTENSION vector (0001) CREATE EXTENSION tsm_system_rows (0004) the HNSW index on image_record.siglip_embedding, raw SQL because alembic's create_index cannot express USING hnsw (...) (0036) plus any CHECK constraint or server_default a migration added without the model declaring it — 4 model files declare CheckConstraints against 6 migrations that touch them. The candidate is a starting point to hand finish; the diff is what proves nothing was missed. Results are printed to the job log rather than uploaded: ci-requirements records that this runner cannot do actions/upload-artifact@v4+, and the repo dropped the action entirely in 2026-05. Run it first with the chain still present, as a control — the diff compares the chain against itself and must come back clean. A clean diff after the squash only means something if the harness was shown to be capable of producing one beforehand. Temporary. Delete once the baseline is stamped.	2026-08-30 13:31:16 -04:00
bvandeusen	0a5bbe81dc	docs: the scheduled refresh does NOT republish nothing (#3265) ... Step 4 asserted that when the base has not moved the refresh is "a ~13s no-op that republishes nothing", and that this no-op was the point. The first half is false and was written without being tested. Run 4934, the first real fire: every content step reported CACHED and both bases resolved to unchanged pinned digests, yet all three :latest tags took a new manifest digest anyway. fabledcurator 4ea5265ba017 -> 380e504de0fa fabledcurator-ml 6e7cfc0c09fd -> 6b2eefc301d8 fabledcurator-agent 44920e0af1f3 -> 54accbeb52ed buildkit mints a fresh image config per run, so identical layers get republished under a new config blob. Storage cost is trivial; the cost that matters is that a :latest digest change stops meaning "something is different", and :c-<sha> is handed a new manifest to diverge from every Sunday for no reason. Corrects the workflow comment (x3) and ci-requirements.md to say what actually happens. Filed as #3265 with the candidate fixes; the likely one is a deterministic SOURCE_DATE_EPOCH off the value artifacts.sh already derives, which would make "same source, same version" into "same source, same bytes". The rest of step 4 verified clean on the same run: the guard passed (HEAD is main (499720d), `git checkout -B main`) — so this runner DOES evaluate the env context inside `with:` — the tag list was :latest alone with no :c-<sha>, and the repoint step correctly found nothing to write.	2026-08-30 12:57:06 -04:00
bvandeusen	6663e06aa6	ci: assert the scheduled refresh actually checked out main ... BUILD_REF is read through the `env` context inside `with:`, which this runner is not known to evaluate. `${{ steps.* }}` and `${{ secrets.* }}` in `with:`/`env:` are proven here; `env` is not, and run 4915's checkout log (`git checkout -B dev refs/remotes/origin/dev`) cannot tell an honoured `refs/heads/dev` from an empty value falling back to the same place — the two are indistinguishable on every path except the one that matters. If it does resolve empty, the weekly refresh checks out dev and pushes its source to :latest, which is production. Every lane stays green and the first symptom is production running code that was never merged. So each of the four jobs now asserts its own checkout before doing anything, gated on `github.event_name` — the `github` context is demonstrably evaluated in `if:`, so the guard cannot be disabled by the same uncertainty it covers. A red weekly job is an acceptable outcome; shipping dev to production is not.	2026-08-29 22:57:41 -04:00
bvandeusen	63e0a423d7	ci: a weekly base-image refresh on the channel tags (milestone 326 step 4) ... Skip-if-exists is keyed on our own source, so an artifact whose source stops moving stops picking up base-image updates. `agent/` last changed 2026-07-17; every push since has correctly declined to rebuild it, which also means it will serve that day's nvidia/cuda layers indefinitely. A `schedule:` trigger, Sunday 06:00 UTC, away from CI-runner's Monday security sweep so the two are never diagnosing each other. #3154's blocking open question is dissolved rather than answered. It was written when the identity was a `r-<revision>` TAG, and asked how the next ordinary push could avoid repointing :latest back off the refresh. Milestone 318 replaced that tag with a LABEL, and #3183 made the repoint step exclude its source tag so the label stays readable. Excluding the source is what also keeps a refresh from being undone: on the next main push the reuse check hits, :latest is not rewritten, and the new :c-<sha> is written FROM the refreshed :latest. To be verified by digest, not by this argument. Four decisions, each commented where it lives: * It builds `main`, not the branch that triggered it. Forgejo registers a cron from the default branch — `dev` here — so a scheduled run arrives with github.ref on dev, and a refresh of :dev would be refreshing the one channel that is rebuilt constantly anyway. The ref is decided once in a top-level `env: BUILD_REF` that all four checkouts take. Deriving it per job would let the halves disagree: sign-extension would derive dev's extension version while build-web bundled main's, and the release download would 404 on a version that exists perfectly well. * It publishes only the channel tag. :c-<sha> for main's HEAD already names the bytes that commit built; re-pushing it over refreshed layers would break the one tag rule 145 makes immutable, and it is the rollback unit — so the breakage would surface on the day somebody needed it. The repoint step needs no schedule case: the tag list is the channel tag alone, SOURCE is the only entry, it is excluded as always, and the step correctly does nothing. * It bypasses reuse by construction, since it rebuilds the same source and fc.revision always matches. Checked in the reuse step beside force_build, so one decision still drives both the build and the repoint. * `pull: true`, on the scheduled path only, is the actual mechanism. A moved base tag changes the FROM layer's cache key and everything above it rebuilds; an unmoved one is satisfied by the registry cache and the refresh is a ~13s no-op that republishes nothing. That no-op is the point — :latest should change when there is something new in it, not every Sunday. The known lag, left deliberately: an apt package update while the base tag stands still is not caught, and closing it needs no-cache: true, which buys weekly churn for it.	2026-08-29 22:53:48 -04:00
bvandeusenandClaude Opus 5	5e72076298	ci: registry-backed layer cache for all three images (milestone 326 step 2) ... `cache-from`/`cache-to` on `<image>:buildcache`, `mode=max`, on all three build steps. Closes the half of the driver change that step 1 left open. Step 1 measured worse, not better, and that was expected but is worth stating with numbers. Run 4896, first builds after moving to `docker-container`: build-web 3m44s (cold baseline 2m23s) build-ml 3m49s (cold baseline 3m20s) build-agent 11m12s (cold baseline 9m26s) The container driver gets a FRESH buildkit instance per job, so it has no local layer store to fall back on — where the old docker driver at least reused whatever the runner's dockerd happened to hold. That is why a registry cache is the only cache this driver can have, and why step 1 on its own is a regression rather than a win. `mode=max` so intermediate stages cache too. The agent's two ~150s pip layers and web's frontend-builder stage are the entire cost, and a min-mode cache would drop exactly those. A `:buildcache` tag is not the withdrawn tag scheme returning. Rule 145 narrowed against names NOTHING reads; this one is read by every build that runs, is one moving ref per image rather than one per build, holds cache blobs rather than a shippable artifact, and is overwritten in place rather than accumulating. Closer to `:dev` than to the `:2026.8.28` tags 318 deleted — and said so in the workflow, so it is not "cleaned up" by a later reader. Expect the next build to be slower again, once: it is still cold AND now pays the cache export. The measurement that matters is the one after that. Scribe #3114. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 18:56:13 -04:00
bvandeusenandClaude Opus 5	e21c9fdd34	ci: a force_build escape hatch for the path skip-if-exists hides (326 step 3) ... `workflow_dispatch` with a `force_build` boolean, honoured inside each of the three reuse steps. It exists because skip-if-exists made its own build path untestable. `agent/` has not changed since 2026-07-17, so the agent build has correctly declined to run on every push since — which also means #3190, whose whole symptom lives on that path, cannot be reproduced on demand. Editing build.yml does not force a build either, and that is deliberate: the workflow is not shipped bytes, so it is in no artifact's path set, and putting it in one would re-version every artifact for a comment change. That is also why this lands before step 2 rather than after. Step 1 moved the builds onto a container driver and turned attestations off; the claim that `fc.revision` still reads back cannot be checked until something actually builds under that driver. Run 4887 confirmed only the cheaper half — `Set up buildx` succeeded on all three jobs, so the buildkit sibling container does start against the mounted socket. Details worth keeping: * FORCE is checked in the reuse step, not in the build step's `if:`. The repoint step keys off `hit` too, and a force that bypassed only the build would leave the two disagreeing about what had happened. * `github.event.inputs`, not the `inputs` context — release.yml already uses that form and it is the one this runner is known to evaluate. Read through env rather than interpolated into the run block, same as release.yml's TAG. * One input, not one per artifact. Three booleans is an interface nobody remembers. Scribe #3252, #3249. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 18:38:29 -04:00
bvandeusenandClaude Opus 5	6b3ec98fa8	ci: build on a real buildx driver, attestations off (milestone 326 step 1) ... Adds `docker/setup-buildx-action@v3` to build-web, build-ml and build-agent, and sets `provenance: false` / `sbom: false` on all three build-push steps. Two open issues share one root, which is why this is one change: * #3114 — the agent rebuilds a ~6.3 GB CUDA + torch image whenever the runner's local cache is cold, 9m26s against 7s warm. The default `docker` driver cannot export a registry cache at all, so the fix is unavailable until the driver moves. The cache itself is step 2, deliberately not here. * #3190 — build-agent goes red AFTER a successful push, `No such image` from the local daemon. The leading candidate is the docker driver resolving image metadata against a local store a registry-direct push never filled. The attestation flags are the load-bearing part. On the default driver they were no-ops; on the container driver, build-push-action@v5 defaults provenance to TRUE when pushing. Provenance attaches an attestation manifest, that makes the pushed tag a manifest INDEX, and `.Image.Config.Labels` does not resolve through an index — so the `fc.revision` label the reuse check reads off the channel tag would come back `<none>` on every push. Every image would rebuild forever, every lane would stay green, and the only symptom would be the bill. Same failure as #3183, through a different door; note #3127 §4 records the same shape for `platforms:`. Unverified until CI says otherwise: these jobs run INSIDE a container against a mounted docker socket, so the buildkit container is a sibling of the job container rather than a child. That works over a socket mount and has never been tried on this runner. The gate is the SECOND dev push, not this one. The images currently published were built by the old driver, so one `reuse: NOTE ... no readable fc.revision` is expected now; what must not appear is a second one. Scribe #3249. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 18:36:35 -04:00
bvandeusenandClaude Opus 5	2e01242381	feat(extension): derive the version as unpadded CalVer (milestone 318 step 8) ... `1.0.<minutes since 2020>` -> `YYYY.M.D.HHMM` UTC, from the commit time of the newest change to a packaged extension file. Same clock and same commit as before; readable instead of opaque, and the same value the rest of the family derives. The hold on this step was two questions about AMO, and Mozilla's own docs answer both: ^(0|[1-9][0-9]{0,8})([.](0|[1-9][0-9]{0,8})){0,3}$ 1. four all-numeric segments -> ACCEPTED ({0,3} more after the first). 2. leading zeros -> REJECTED. A segment is the single digit `0` or starts 1-9, so `08` and `0201` are refused. MDN says it in prose too: "Non-zero numbers must not include a leading zero." So the documented fallback applies, extension only: the same numbers rendered without the family's zero-padding. `2026.08.29.0201` and `2026.8.29.201` are one value in two renderings — rule 148 defines comparison as numeric per segment, under which they are equal — so nothing already published is reordered, and left-padding each segment recovers the family string exactly. HHMM stays one segment because AMO allows at most four. The transition is safe in the other direction too: 2026 > 1, so every CalVer outranks every published 1.0.x. build.yml's downgrade guard confirms it. Also in scope: * MAJOR.MINOR is gone. `cmd_major_minor`, `cmd_patch` and VERSION_EPOCH go with it, the committed version in manifest.json / package.json is now wholly inert, and ci.yml's MAJOR.MINOR-agreement check is retired rather than left running beside a fact that stopped existing (rule 22). * ci.yml's `extension-version` lane now asserts Mozilla's regex verbatim instead of a loose `^[0-9]+(\.[0-9]+)*$` — which would have passed the padded shape. It also asserts YYYY.M.D.HHMM, because AMO would accept a regression to `1.0.<minutes>` while that orders below everything signed since. Checking here is the point: AMO 409s on re-signing, so a version it rejects is burned and cannot be reused. * `artifacts.sh version extension` delegates to packaging.sh, so the two cannot answer differently. The direction matches the existing one — artifacts.sh already asks packaging.sh for the extension's path set. #3156 is what makes this commit safe to make: packaging.sh is in web's path set, so the web revision moves with the extension version and build-web rebuilds instead of republishing an image bundling the previous XPI. Scribe #3138. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 13:43:30 -04:00
bvandeusen	41f2bec3af	fix(ci): the build pushes one tag; the rest are written registry-side (#3190) ... buildx on this runner pushes the first tag to the registry and then re-pushes the remaining ones through the DOCKER driver, reading them out of a local image store that a registry-direct build never populated: #27 pushing …/fabledcurator:latest DONE 15.8s #28 pushing …/fabledcurator:c-0e15c44 with docker #28 ERROR: tag does not exist: …:c-0e15c44 It is intermittent — build-ml made the identical two-tag push seconds later in the same run and succeeded — and the consequence is worse than the red job suggests. `:latest` had already published, so production was correct while the immutable rollback tag rule 145 requires of every main push simply did not exist. Nothing else would ever have noticed: a missing :c-<sha> has no consumer that fails, so it surfaces at the moment somebody needs to roll back, which is the worst time to learn a rollback target was never written. So the build now pushes exactly one ref — the channel's — and the existing repoint step, which already excluded the source tag and already ran on every reuse, now runs on the build path too and owns every other tag. `imagetools create` is a registry-side manifest copy: no local daemon, nothing that can be absent. This adds no new code path; it puts the build case onto the one that was already proven. Chosen over the alternative of asserting each tag resolves after the build, which would have made the failure loud without making it rarer. The cost, accepted: `imagetools create` wraps its source in an index, so :c-<sha> is an index rather than a plain image and fc.revision does not resolve through it. Nothing reads that label off :c-<sha> — the reuse check only ever inspects the CHANNEL tag — and the index names the same manifest, so a pull is byte-identical. The reuse path already produced :c-<sha> this way; this only makes it uniform. `build_tags` goes with it — the tag list now has exactly one consumer.	2026-08-29 00:38:26 -04:00
bvandeusen	d38585ed94	docs: true up the tag scheme against what the pipeline publishes (318 step 9) ... A doc describing a tag scheme the pipeline stopped using is worse than no doc — it is a confident wrong answer, and #3159 named it as the drift that had just bitten rule 148. build.yml (each edit ×3, once per build job): * The derived-values block claimed three values including a `tag` subcommand that no longer exists, and said "on dev the date tag is computed nowhere else". There are two values and no date tag. * The BASE-IMAGE FRESHNESS paragraph argued from a pin that no longer exists — "a date tag has to keep serving the bytes it served (fabledcurator:2026.7.17 still resolves to July's image)". Milestone 318 removed the argument's premise rather than answering it: with no version tags there is no immutable name a refresh could contradict, and rule 145 already allows a moving tag to be republished. Deleted the argument, kept a line saying why there is nothing to argue about (#3159's phrasing: the workflow becomes obviously correct instead of carefully correct). * The repoint comment said ":c-<sha> and the date pin". There is no date pin. * "its set is too narrow — the direction that serves stale bytes on a pin" now names the failure that actually exists: the reuse check hits and the channel serves a web image bundling the previous XPI (#3156). README gains a "Versions and tags" section — the three tags, why there is no fourth, and the fact that with no registry name carrying it, the instance's own report is the only answer to "which build is this?". Also fixes two stale claims: three workflows (four since release.yml) and "AMO signing runs on main only", which has been false since milestone 271 step 6 signed on both channels. ci-requirements.md was missing milestones 313 and 318 entirely: no artifacts.sh, no label-keyed reuse, no FC_VERSION. Added those, plus the #3156/#3202 membership test (can changing this file change the published bytes — not is it copied in), and corrected packaging.sh's "two consumers" to three now that artifacts.sh reads it. docker-compose.yml needed nothing — it names `:dev` directly and never offered a pin target. Note #3127 and rule 145's relation notes are updated in Scribe.	2026-08-29 00:09:52 -04:00
bvandeusen	b6b9fd8287	ci: a release publishes a changelog, not an image (318 step 7) ... Step 2 took the build consequence away from a `v*` tag — `main` has already built and published the commit by the time anyone tags it, and rebuilding would re-push `:c-<sha>`, which rule 145 forbids even when the source matches. That left the tag with nothing to do at all. This is the job it has instead. Step 6 put the derived version in the Settings footer, so an operator can say WHICH build they are running; this says what is in it that was not in the one they ran last month. Both halves of one question (note #3127 §5). The previous release is found by walking ANCESTRY, not by sorting a list. That is load-bearing here specifically: rule 148 moved the tag shape from `v26.05.22.0` to `v2026.08.28.2208`, and lexicographically `v2026...` sorts BEFORE `v26...` — the third character is `0` against `6`. A sorted implementation would reach back past every new-shape tag to the newest old-shape one and publish months of commits as "changes since", looking entirely correct while doing it. `git describe --exclude` is immune to the shape change, and reachability is the more honest question anyway. The publisher GETs and PATCHes rather than POSTing and recovering the id from a 409 — note #3127 §6.7, which is ThoughtSync #2182's bug. A `v*` tag is created once so the conflict path is rare, but "rare" is how that one survived to be found somewhere else. Cross-checks are reported on the release, not enforced. The tag is already pushed by the time this runs, so failing would leave the operator with a tag, no release, and a red lane to explain it — while the release is still the useful object. It says so at the top when the tag names a version the web image does not report, or when the commit is not on `main` and the `:c-` rollback refs it lists were never published. Nothing runs on a schedule and nothing auto-tags on merge. Release tags are bookmarks (note #3127 §0); FC went twelve weeks without one and nothing was wrong. Also here: - `scripts/` joins the ruff lane. release_notes.py runs only on a tag push, so a syntax error there would otherwise surface at the one moment nobody wants to be debugging a workflow. - version.spec.js reads the workflow directory instead of listing three files by hand. Its own comment says the assertion should survive consumers coming and going; the hardcoded list was the part that could not, and release.yml would have joined the directory without joining the check. Tests build a synthetic history spanning the tag-shape change rather than leaning on this repo's tags, so the span assertion holds whether or not a checkout brought the tags along — a span test that quietly skips is worse than one that fails.	2026-08-28 20:57:25 -04:00
bvandeusen	bce894ba24	feat(settings): the instance reports which build it is (318 step 6) ... A dim line at the foot of Settings: `FabledCurator 2026.08.28.1249 · dev`. This is no longer a convenience. Milestone 318 stopped publishing version image tags, so an instance's own report is the ONLY answer to "which build is this?" — there is no registry name left to check it against. Note #3127 §5 says it directly: a wrong answer here has no second source to contradict it. Three states, kept distinct because collapsing any two of them lies: not asked yet render nothing asked, no version render "unknown" asked, has a version render it A blank footer reads as "no version", which is a different claim from "I cannot say". And a failed health call deliberately does NOT mark the build loaded — a network blip says nothing about the image, and presenting it as "unknown" would look like a defective build. Carried on /api/health rather than a new route: it answers at the same cost (two module constants, no I/O) and TopNav already fetches it app-wide, so a separate endpoint would mean a second request for two strings. Both fields are OMITTED when unset rather than sent empty. Absence already means "cannot say" — an image predating the field says exactly that by not having the key — so a second spelling would make every reader special-case it. The pre-existing test asserting the body is EXACTLY {"status": "ok"} is what keeps a well-meaning `or ""` default from creeping in. FC_CHANNEL now has one definition. It was read from the environment in extension.py and would have been read again here; the new build_info module holds both, and extension.py binds it as a module-level name so existing tests monkeypatch it exactly as before. Separate from config.py on purpose: those are operator settings meant to be changed, these describe the artifact. Channel sits beside the version, never inside it (rule 149), asserted from both ends. A `-dev` suffix would read as a 0 segment to the extension's parseInt comparator and make every dev build compare equal — #2993 exactly. Not hidden, per the operator and §7: the JS bundle and asset hashes fingerprint the build anyway, and "I'm on 2026.08.28.1249" is the single most useful line in a bug report.	2026-08-28 18:08:31 -04:00
bvandeusen	b3989d0224	fix(ci): the repoint was destroying the label it depends on ... Reuse worked exactly once per artifact, then every later push rebuilt at full price. Nothing failed and nothing went red — the savings simply evaporated. `imagetools create` wraps its source manifest in an INDEX. The repoint step passed the channel tag as both source and target, so after any reuse :dev stopped being a plain image, `.Image.Config.Labels` no longer resolved through it, and fc.revision read as absent on the next push. Observed across three runs rather than reasoned about: run 4749 read fc.revision=a7e626a67a79 off fabledcurator-ml:dev and skipped the build; run 4751 read <none> off the same tag and rebuilt. The only thing to touch it in between was 4749's own repoint. The agent hit in 4751 precisely because its :dev had last been written by a real build, not by a repoint — which is the control case. Milestone 313's r-<rev> design was immune without anyone noticing why: the source (the identity tag) was never one of the targets. Step 3 made the channel tag both, and inherited a bug the earlier shape had avoided by accident. Fix: exclude the source from the target list, so the channel tag is only ever written by a real build and stays a plain readable image. On dev that leaves nothing to do, which is correct — the hit already established that :dev points at the right content. On main it leaves :c-<sha>, which rule 145 requires of every main push whether or not a build ran. Also added a note the reuse step prints when a channel tag exists but carries no readable label. That is expected exactly once per artifact during the migration; if it appears on every push, the tag is being index-wrapped again and reuse is dead. This class of failure — correct behaviour, quietly worth less than it reads — is the third one this milestone has turned up, and it is the one that does not announce itself.	2026-08-28 14:55:49 -04:00
bvandeusen	cd0b0ff04a	ci: publish :latest + :c-<sha> on main, :dev on dev — nothing else (318 step 4) ... The narrowing itself. Rule 145, narrowed 2026-08-28 once it was verified that nothing pins: "a third name for the same thing is upkeep for a model we do not run." Gone: :2026.8.28 the per-artifact date tag from milestone 313 step 3. It shipped, it was verified on both branches, and its premise is gone. This is a withdrawal, not a correction — nothing about it was wrong. :main a second moving name for whatever :latest already pointed at, justified by nothing but symmetry with :dev. Kept, and note the inversion: :c-<sha> on main. Milestone 313 step 6 planned to retire it on "haven't used, not important" — amended rule 145 makes it the rollback unit and, with version tags gone, the only immutable identifier left. #3139 is cancelled as superseded rather than quietly dropped. No retention job is built. One already runs (#3157): a nightly Gitea package_cleanup_rule with keep_count=10. Two consequences worth writing down. Its keep_pattern protects any dotted-numeric tag, so the date tags this commit stops minting would have been exempt from retention and accumulated without bound — silently, in the direction of unbounded growth. That trap is now dissolved rather than needing the pattern change #3139 planned. And :c-<sha> IS prunable under that pattern, so the honest statement is that it is the rollback unit for roughly the last 10 builds, not for every main commit ever. cmd_tag goes with its last consumer rather than being left as a dead subcommand for a later step to notice. The report step logs version and revision, which is the same information — the date is the first three fields of the version. Docs needed no change: every consumer already tracks a moving tag (docker-compose.yml on :dev, agent/README.md on :latest), which is the evidence the whole stance rests on.	2026-08-28 14:45:11 -04:00
bvandeusen	7e065fed70	ci: key the reuse check on an image label, not a tag (318 step 3) ... The shadow (dee93fa, run 4732) answered the gate: `imagetools inspect --format` reads `.Image.Config.Labels` against this registry on buildx v0.36.1. So the reuse check now asks the moving channel tag whether the image it already points at carries this commit's `fc.revision`, and the r-<rev> identity tags stop being published. Three things this removes rather than manages: A name minted per build that one thing read. Rule 145's narrowing is aimed exactly there — "a third name for the same thing is upkeep for a model we do not run." The -main/-dev qualifier, and the CHANNELLED list behind it. Which tag you inspect IS the channel, so the distinction has nowhere to live. cmd_identity goes with it. A silent expiry nobody wrote down. r-<rev> matches no branch of the registry's keep_pattern (#3157), so identity tags were prunable past the newest 10 — a pruned one costs a rebuild, in the safe direction and entirely invisibly. A label rides inside a tag that has to exist anyway. It also dissolves #3154 instead of deferring it: a scheduled base refresh rebuilds :latest with the same revision label, the next unrelated push sees a match and skips, and the refreshed base survives. Under the tag scheme that push repointed :latest back to the older base. The measured detail that shapes the code: a missing label returns an EMPTY STRING and exits 0. Branching on the exit code would read "no label yet" as success and skip a build that was needed. So it compares values, and every uncertain case — absent label, unreachable tag, older image — lands as empty, never equals a 12-char revision, and falls through to a build. Reading the specific key matters too. The map carries the base image's labels, and org.opencontainers.image.version sits right beside ours reading 24.04 on the agent — a plausible-looking wrong answer. Expect every artifact to rebuild once on this push: nothing carries a label yet and it cannot be backfilled, since the reuse path copies a manifest and config labels are not manifest annotations. One rebuild per artifact, ever, self-healing after. test_artifact_identity.py is rewritten around what is now load-bearing. The CHANNELLED drift test had nothing left to guard; in its place the revision is asked of git directly, so the file fails if the derivation ever stops being "the commit this artifact's own shipped files last changed in".	2026-08-28 14:37:36 -04:00
bvandeusen	dee93faa37	ci: shadow the label-based reuse key before trusting it (318 step 3) ... Additive and inert. Every build now stamps `fc.revision` as an image label, and a diagnostic step tries to read it back off the channel tag. Nothing downstream reads either yet — the reuse check still keys on the r-<rev> tag. The question step 3 has to answer before committing: can the reuse check be keyed on a LABEL read off the channel tag instead of on a separate tag? If it can, nothing is minted per build, the -main/-dev identity split disappears with CHANNELLED, and #3154's base-refresh clobber stops existing rather than needing a workaround. It also closes a defect found in what already shipped: r-<rev> tags do not match the registry's keep_pattern (#3157), so they are prunable, and a pruned identity tag silently costs a rebuild. A label cannot be pruned — it rides inside a tag that must exist. Three probes rather than one, because `--format` templating over .Image varies by buildx version and one round trip per guess is a poor trade. Two questions, answered at different times, and conflating them would abandon a working design: 1. Do the template mechanics work here? Answered on the first run — if the probes print a labels map, even an empty one, `--format` resolves on this buildx and this registry. That is the real gate. 2. Does a value round-trip? Only once an artifact genuinely rebuilds. This push touches build.yml, which is in no path set, so all three artifacts reuse and stamp nothing. Expect no values this run. That second point is the migration path rather than a defect: under the real mechanism "no label" reads as a MISS, so each artifact rebuilds once and is stamped from then on — one extra build per artifact, ever, in the safe direction. `imagetools create` cannot backfill it, since it copies a manifest and config labels are not manifest annotations.	2026-08-28 14:00:09 -04:00