1285 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	48108a3569	fix: a lane that is OFF was not attributable to itself (4295) ... Operator, 2026-09-23: *"clean up the stale service_seen rows"*. **They were not stale.** They were phantoms, written on purpose, and they will come back on every install that turns a lane off — so the rows are the smaller half of this. A celery worker was attributed to its lane by the queues it was CONSUMING. A lane at cap 0 has its consumers cancelled, so it answers `active_queues()` with an empty list, matches no lane, and is dropped. Three consequences, all on the operator's screen at once: 1. The lanes table reported the lane **not answering** — the signal for a crashed worker, not for one the operator turned off. 2. The roster grew a phantom row named **`Worker ()`** — the empty queue set rendered as a display name — shown "running" beside the real lane's row going stale, because nothing updated it any more. 3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every lane present. ML ships at cap 0, so a fresh install was permanently unhealthy and Swarm restarts an unhealthy task forever. That third one is the severe one, and its docstring asserted the opposite of what the code did — *"a disabled lane still runs its process with its consumers cancelled, so it answers inspect and is healthy"*. It answers. It was not attributed. A comment can be right about the intent and wrong about the program, and this one had been wrong since the consolidated container shipped. `worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels with the process rather than with what it happens to be doing. `gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the information was there and nothing read it. Falls back to the queue set for a deployment that names no node, and `docker-compose.yml` now sets one per service so the multi-service stack gets it too. The roster keys on the LANE's queue set when the node resolves, which is the same string the row already had while it was consuming — so an existing row keeps updating rather than a second one appearing. Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately NOT a retention sweep: the roster never forgets on purpose, so a quiet row is what it is FOR, and only a row that cannot correspond to anything real is safe to remove. An `agent:agent` row, if one exists, is left alone — nothing here can tell an abandoned agent id from a second agent that is genuinely down, and hiding a dead GPU agent is the one thing the roster must not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 15:21:21 -04:00
bvandeusenandClaude Opus 5	a4c66601db	fix: the heartbeat guard grepped its own explanation (4295) ... Run 7375: the test that asserts the agent's heartbeat is not gated on holding leases failed — on the docstring of the fix, which quotes the construct the fix removed, because that is what a docstring explaining a fix does. assert "if ids:" not in loop A source-TEXT assertion cannot tell code from prose about code. Parsed now: the function's AST body, unparsed with the docstring node dropped, so the guard reads only what executes. Worth stating as the general shape, since this repo writes long explanatory comments on purpose: any check that greps source for the absence of a pattern is in tension with documenting why that pattern is gone. Either it excludes the prose, or the next person to explain the fix breaks the guard that protects it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:59:06 -04:00
bvandeusenandClaude Opus 5	693759f2bb	fix: an idle GPU agent could not check in, so the roster called it stopped ... Operator, 2026-09-23: *"I'm running the gpu agent on my device and it currently reads as 'offline' but it's running and has checked in recently."* It had checked in — twelve minutes ago. Two cadences that never agreed: idle lease poll ceiling 900s agent/fc_agent/worker.py (sleep mode) heartbeat while idle never gated on holding leases roster "stopped" after 300s api/system_health.py The roster records an agent check-in on `lease` and `heartbeat`. The heartbeat loop was gated on `if ids:`, so an agent holding no leases sent nothing at all — leaving the lease poll as the only check-in, and sleep mode backs that off exponentially to a 900s ceiling. 900 against 300: an IDLE agent was structurally guaranteed to read as stopped. Nothing was broken; nothing was misconfigured; the two halves simply disagreed. Not a recent regression. Sleep mode landed 2026-07-02; the roster adopted the lease as its check-in on 2026-09-02 — *"A lease IS the check-in … Recorded on the call that was already happening"* — without noticing that the call it was piggybacking on had been deliberately slowed ten weeks earlier. The heartbeat now sends whether or not it holds leases. An empty one extends nothing (`id.in_([])` matches no rows) and costs one small POST every 45s — against the 6/min lease poll sleep mode exists to avoid, that is not a cadence worth protecting, and it is what makes "is the agent alive" answerable at all. Still gated on `self._running`: a worker that has been stopped is not checking in for work, and reporting it as present would be a different lie. Two things I could NOT determine from the code, both needing the live table: whether a stale `agent:agent` row exists from an older build that omitted `agent_id` (the server defaults it), and whether changing `AGENT_ID` has ever stranded an abandoned row — nothing prunes `service_seen`, so either would sit there reading "stopped" forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:52:36 -04:00
bvandeusenandClaude Opus 5	5b6f2ba526	fix: a Postgres connection was held across every celery round trip (4295) ... Operator, 2026-09-23: *"something about changing the cap number is blocking to the website... it shouldn't be"*. Nothing here was slow in itself. A database connection was held across work that is slow, and that is why it surfaced as the whole site stalling rather than as one slow page. `lane_view` took the session and kept it open through a celery inspect whose budget is 11s. The System tab polls that endpoint every 15s — and with a lane not answering, every inspect runs to nearly its full budget, so each poll pinned a connection for most of the interval. SQLAlchemy's default pool is 5 plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing, and a cap change adding two more inspects exhausts it, and every OTHER request then waits for a connection. Split so the database work finishes before the broker work starts: - `lane_settings(session)` reads the caps and the oldest running task, then the session closes. `lane_view(settings)` does the inspect with none held. - `store_lane_cap(session, …)` validates and commits, then the session closes. `push_lane_cap(lane, …)` does the live push with none held. And a second finding while measuring it: **raising a cap now costs no broker round trip at all.** The first cut only knew on/off, so it inspected on every raise to find out whether the pool needed lowering — the control meant to be instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap so the push knows the direction; only a lowering needs to say anything. The guard is structural, not timed: `lane_view` and `push_lane_cap` must not ACCEPT a session. A timing test would be flaky, and a call-order test would pass against a version that took the session and merely used it early. `/api/system/health` has the same shape and is NOT fixed here — it is rate-limited by `refresh_if_stale` so it does not inspect on every request. Worth doing, separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:52:36 -04:00
bvandeusenandClaude Opus 5	c09ebd6639	fix: the race fix's temp name assumed one bootstrap per process (4295) ... Run 7370: six of the eight threads in the new test died with FileNotFoundError(2, 'No such file or directory') The first cut named the temp file `.credential_key.b64.<pid>.tmp`, which assumes one bootstrap per PROCESS. Eight threads share one pid, so all eight raced the same filename and six found it already unlinked by another. The assumption held for hypercorn's workers, which are processes, and would have held in production. It was still an assumption the code did not need to make, and the test found it in one run — which is the test doing its job rather than the test being wrong, so `tempfile.mkstemp` replaces the guess. Verified against the SHIPPED file, loaded by path so the package `__init__` (which wants quart) stays out of it: 25 consecutive attempts, eight threads through a barrier each time, one key, no leftover temp files. My first check of this reimplemented the logic in a scratch script, which measures a copy — rule 10 names that exact failure, and it is not evidence about the code that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 13:11:29 -04:00
bvandeusenandClaude Opus 5	895589a578	fix: first boot raced itself for the credential key (4295) ... Run 7368's smoke, on an image whose five verification lanes were all green: WARNING Generating NEW Fernet credential key at /images/secrets/credential_key.b64 ... ValueError: Fernet key must be 32 url-safe base64-encoded bytes. Nothing to do with this batch's changes — it is a first-boot race that has been there since the key file existed, and it is a RACE rather than a certainty: the same code booted cleanly on the three runs before it. hypercorn starts several worker processes and each one builds the app, so on a first boot they all reach the bootstrap together. `write_bytes` creates the file at size zero and fills it a moment later, which gives the second process an `exists()` of True and a `read_bytes()` of `b""`. A first boot that fails one time in five is worse than one that fails every time, because it looks like the deployment rather than the code — and this is the very first thing a new install does. The key is now written to a temp file and `os.link`ed into place. `os.link` is the atomic part: it either creates the name or raises FileExistsError, and it cannot expose a half-written file. Deliberately NOT `os.replace`, which would succeed — two processes that both generated a key would each think they had won, and the loser's key would overwrite the one the winner had already handed to Fernet. The losing branch reads the winner's key back rather than returning its own, which is what keeps every worker on ONE key. Tested for AGREEMENT, not for the absence of a crash: eight threads through a barrier, and all eight must end up holding the same key. A race that left each worker with its own would pass a "did it raise" check and produce a system where a credential written by one worker cannot be read by the next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 13:04:26 -04:00
bvandeusenandClaude Opus 5	61641fbba7	fix: a test still described the control the cap replaced (4295) ... Run 7367, integration lane: FAILED test_a_cap_is_stored_even_when_it_cannot_be_pushed assert True is False The code was right. Raising a cap is PERMISSION, not a request — it deliberately does not grow the pool, because that would put workers on a lane with nothing to do, and the sizing pass spends the permission on its next tick if there is work. So nothing is pushed and `applied` is vacuously true. The test was carried over from when the number meant "run this many", where every write pushed. It asserted the old control's behaviour against the new one — lesson #4338's shape again: an assertion encoding the thing that changed, failing on the change rather than on a defect. Split into the two cases that actually exist now: - raising a cap stores it and pushes nothing, reporting applied; - turning a lane OFF does push, because consumers follow the cap immediately in both directions — off must take effect when it is asked for — so with nothing answering it reports `applied: false` with a reason, and the value is still stored for the sizing pass to carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:57:33 -04:00
bvandeusenandClaude Opus 5	364e050632	fix: a spliced-out test left its tail behind (4295) ... Run 7366, in both the lint and unit lanes — one bug wearing two hats: NameError: name 'slots' is not defined tests/test_worker_lanes.py:278 Rewriting `test_worker_lane_check_constraints` — parametrised over (slots, cap, ok) — I cut the parametrize decorator and the assertion but left the two lines after it, which still referenced all three names. `ast.parse` is happy with that; only running it is not. Both failures were the same fragment, which is the tell: the lint lane's F821 and the unit lane's NameError are the same fact reported twice. Checked the rest of today's splices for the same seam rather than fixing only the one CI named — a module-level undefined-name sweep over all four test files I rewrote today comes back clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:53:27 -04:00
bvandeusenandClaude Opus 5	830394ed5e	fix: the migration's DROP CONSTRAINT names doubled their own prefix (4295) ... Run 7365, integration lane: (psycopg.errors.UndefinedObject) constraint "ck_worker_lane_ck_worker_lane_slots_within_cap" of relation "worker_lane" does not exist That is #3275 exactly, from the other direction. alembic 0088 had to RENAME four constraints CREATED with a doubled prefix; this one tried to DROP two with the same doubling. `op.drop_constraint` runs its name through Base.metadata's naming convention, which prepends `ck_worker_lane_` to a string that already carries it — `op.f()` is what marks a name as final, and 0103 used it on the way in. The model test also went red, correctly: `test_worker_lane_check_constraints` was parametrised over (slots, cap) pairs and asserted all three constraints, and two of them went with the `slots` column. It is one unparametrised test now, asserting the whole remaining set rather than a membership — a constraint left behind naming a dropped column is not a harmless leftover, it is a table the migration cannot have produced. Worth recording: **the gate worked.** Run 7365 skipped `sign-extension`, `build-web`, `smoke-web`, `promote` and `build-agent`, and `:dev` still names the previous digest. That is the red-direction verification #4339 owed, and it arrived by accident rather than by a forced failure — which is the better evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:51:27 -04:00
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	abe16aa382	feat: the System tab is one bounded table, and the dial is the switch (4295) ... Operator, 2026-09-23, on the screenshot: *"I feel that we can probably combine the two sections into a single table and to format it in such a way that it appears more bounded and less free-form or open. also there's nothing to describe what 'auto' means or why their needs to be or should be on/off toggles. almost all of it always needs to run there's only one optional piece and it is killed by moving the 'cap' to zero."* Three separate things, all correct. ## The four lanes were listed twice The roster (milestone 365) said "ML tagging is running", and four hundred pixels below it the lanes pane said "ML tagging · 1/1 busy". Two answers to one question from two endpoints, free to disagree on screen. I moved the second pane onto this tab yesterday and did not notice it duplicated the first. Now one row per part, with controls on the rows that have a lane and none on the rows that do not. The join is on the QUEUE SET, because that is what `service_roster` keys a celery part on — as a set, not as a string, so neither side has to agree about order. It lives in `utils/systemParts.js` rather than inline, and has a spec, because its failure is SILENT and is the exact thing it exists to prevent: a lane that stops matching its part does not throw, it grows a second row for the same worker. The duplication, returning through the code that removed it. ## Bounded, not free-form A real table — header, column rules, one bordered card — instead of dotted rows floating on the page background with nothing saying where the list began or what a column meant. ## The dial is the switch There was an `On` switch per lane beside the slots dial. Of four lanes, three must run for the application to work at all, so that switch offered a choice that was never real — and for the one lane that IS optional, "off" and "zero slots" were two ways of saying the same thing that could disagree with each other. So `enabled` is now DERIVED from the number: `set_lane` sets it from `slots > 0` when the caller did not say. It stays on the API and in the model — it is still the mechanism, and a drain-before-restart may still want a lane holding its process with consumers cancelled without destroying the operator's slot count to say so. Two things fell out that a test now pins: - The consumer command is sent on the CHANGE, not on the field being present. Otherwise every slots write re-sends a command that changes nothing — lesson #4183's churn, arriving through the new derivation. - The model fetch fires on the off→on TRANSITION. It used to test `enabled is True`, the field having been sent. The UI no longer sends it, so the download that makes the ML lane usable would simply never have fired and the lane would have come on to consume a queue it had no model for. ## And Auto now says what it is A legend under the table, in the operator's terms: what a slot is, that zero turns a lane off, that three of the four are not optional, what `of N` means, and that Auto lets a lane add slots by itself when its queue is backed up AND every slot is busy — with why it is off by default, since it is the only thing on the page that acts without being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:26:27 -04:00
bvandeusenandClaude Opus 5	274f7ffe21	ci: the tests gate the publish — ci.yml folds into build.yml ... Operator, 2026-09-23: *"tighten the gate so :dev can't publish on red tests"*, then *"I don't want failing builds to publish anywhere going forward."* Run 7348 is the worked example. The backend unit lane went red on `2f8f0bc` and `build-web` pushed `:dev` in the same minute, because the lanes and the build were SEPARATE WORKFLOWS on the same push trigger. Neither could see the other's verdict. `:dev` was a "it built" signal, never a "it passed" one, and nothing about that was visible from either run. Two workflows cannot express the gate. A `needs:` edge only exists inside one graph. So `ci.yml`'s five lanes move into `build.yml` and `ci.yml` is deleted; `sign-extension`, `build-web` and `build-agent` now need all five. Nothing here is a new mechanism — it is the same edge that has gated `promote` since milestone 362 step 4, and it keeps that step's hardest-won property: **not running is not the same as passing.** `needs` treats a SKIPPED dependency as unsatisfied, so a lane that silently skips itself blocks the publish exactly as a failing one does. Run 5290 is why that is worth stating. Scope, said plainly rather than implied: - Gated: every image tag (`:dev`, `:latest`, `:c-<sha>`), the weekly base refresh, and the `ext-<version>` signed-XPI release asset — `sign-extension` publishes too, so it is gated with the rest. - Not gated, deliberately: `extension.yml` publishes nothing, and `release.yml` runs on a `v*` tag, generates notes rather than an artifact, and its commit already went through main's gated build. - `pull_request` (Renovate bumps into `dev`) comes across with the lanes. Its runs are the lanes and nothing else, via an `if:` on each publishing job rather than an inference from the `needs` chain. The cost, accepted knowingly: this workflow queues per branch and never cancels, so on two pushes in quick succession the second's lint feedback waits out the first's build. A slower red beats a fast red that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:15:42 -04:00
bvandeusenandClaude Opus 5	468ff71353	fix: two lane fixtures described a container that cannot exist (4295) ... `pools={"host-a": 0}` — a celery replica running an empty pool. billiard will not run one, and `gen_supervisord` starts every lane at one process for exactly that reason, so no container has ever looked like this. It passed for as long as the code agreed with it. The floor landed, the reconcile's target for a zero-slot lane became one, and the fixture's impossible zero read as a real difference: `changed: ['ml']` on a settled lane, and run 7346 red on the commit that fixed the live bug. Both now say one, which is what ml actually runs. The two tests the fix added that asserted the same properties from the other side are gone with them — one assertion per fact, in the test that already named it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:03:13 -04:00
bvandeusenandClaude Opus 5	2f8f0bc9eb	feat: worker lanes move to the System tab, in the roster's own idiom (4295) ... Operator, 2026-09-23: *"where is the setting surface to set the number of each kind of worker? we have a system view under settings but no options to be set here"* — then *"move it to system tab and improve the view to be more inline with other UI elements."* They went looking under System and found a read-only roster. The card's own first cut argued that a control had no place on a tab about things being broken; that was wrong twice over. Half these lanes are optional and off by DESIGN, so "not running" is the normal state someone comes here to change — and a tab that only reports is a tab you leave in order to act somewhere else. So it is the roster's second half now, and built out of the same parts: the pane above says what is running, this one says how hard. Dot, name, detail, numbers on the right; the controls take the place the roster gives to latency. It was a bordered card wrapping a v-table, which is what made it read as bolted on. - The row styles move from SystemHealthTab's scoped block into app.css, so the second pane IS the first one's idiom rather than imitating it. A copy would have drifted the first time either was touched. - Lane state uses the roster's four dots. A lane that is off reads grey, never red: colouring an intended state as a fault is how an indicator stops being read. - The card owns its own polling. On Activity it rode that tab's 3s queue timer; System has none, so moving without this would have shown an empty pane to anyone who opened System first. 15s and `document.hidden`-guarded — it is a dial, not a live meter, and v-window keeps a hidden tab mounted. - The table header carried the column names, so each control now labels itself: `slots · cap N`, `auto`, `on`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:01:54 -04:00
bvandeusenandClaude Opus 5	86d6509936	fix: a lane at zero slots tried to empty a pool billiard will not empty (4295) ... Found on the operator's live deploy, not in CI: [scheduler] worker_control: ml reconciled 1 -> 0 slots [ml] pidbox command error: ValueError("Can't shrink pool. All processes busy!") ML ships at 0 stored slots and disabled, and `gen_supervisord` starts every lane at one process so `add_consumer` has something to reach. So the stored value and the running pool disagreed by one, permanently: billiard will not remove the last worker, and `set_lane_slots_sync` returns True on SENDING the control message — the refusal happens later, on the worker. The reconcile logged a successful correction and reported `changed: ['ml']` every tick, forever, on the default configuration of every install. Lesson #4183 in production: an enforcer whose target is unreachable re-does its own work on every pass and says it worked. The floor is now one PROCESS, in one place — `effective_slots()` — applied wherever a target is COMPARED as well as wherever one is sent. Comparing against the unclamped 0 sees a difference no control message can ever close, which is the same non-convergence one layer up. Zero slots still means zero WORK: the lane's consumers are cancelled, and the idle process is what the enable switch lands on. A cross-file guard ties the generator's starting concurrency to the same function, so the two ends of the floor cannot drift apart again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:58:58 -04:00
bvandeusenandClaude Opus 5	a987ca41ca	perf: the lane read is one broadcast and three targeted, not four broadcasts (4295) ... Found while fixing the roster's budget (f23ab9f) and reported to the operator rather than changed mid-deploy; they asked for it. `inspect_lanes_sync` made FOUR broadcast inspect calls — active_queues, stats, active, reserved — at 2.0s each. A broadcast with no `destination` cannot know how many replies to expect, so each waits out its whole timeout rather than returning on the last one. About eight seconds, and `lane_view` sits on Settings -> Activity -> Worker lanes, so that was the load time of that card every time it was opened. The composite healthcheck paid it too, against a 15s timeout. Now the first read discovers the nodes and the other three name them, so celery stops as soon as those nodes have answered — milliseconds, for workers in this same container. The worst case is unchanged: a node that vanishes between the broadcast and the targeted reads still costs a full timeout waiting for a reply that is not coming, which is why the bound stays four. Nothing answering now costs ONE round trip instead of four. The three later reads exist only to describe what answered, so with an empty roster they described nothing at three full timeouts. That is the broker-down case — exactly when the healthcheck and the card need an answer rather than a wait. `lane_view` also gets a deadline. It awaited `to_thread` with no bound at all, which is rule 156's shape even though every inner call has its own timeout; on expiry it now reports every lane as not answering, because a page that renders "not answering" is a better answer than one that does not render. The budget is derived the same way the roster's now is — round trips times the timeout, plus slack — and a test asserts the relation rather than the number, plus one that reads the source so a fifth call cannot quietly put the deadline back under the work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:37:20 -04:00
bvandeusenandClaude Opus 5	f23ab9f50e	fix: the roster's inspect budget was exactly the work it waited for (4295) ... From the operator's first consolidated deploy, 2026-09-23. The app is serving — showcase, thumbnails, a Patreon ingest tick, all five lanes in one container — and this repeats in the log: WARNING service roster: celery inspect failed; roster not refreshed File "service_roster.py", line 138, in refresh_celery_roster grouped = await asyncio.wait_for(...) TimeoutError The inspect calls were working. The budget was wrong. `_inspect_celery_sync` makes TWO broadcasts — `active_queues()` and `active()` — and a broadcast with no `destination` cannot know how many replies to expect, so each waits out its full timeout rather than returning on the last reply. The sync call costs ~2 x INSPECT_TIMEOUT_SECONDS. The wrapper allowed `INSPECT_TIMEOUT_SECONDS * 2`. That reads like a safety factor and is precisely the worst case with nothing left over — and this runs on a web process that was serving ninety thumbnails a second at the time, so the thread handing off through `asyncio.to_thread` need not even be scheduled inside the budget. A budget equal to the work fails under any load at all. Now derived: `INSPECT_TIMEOUT_SECONDS * INSPECT_ROUND_TRIPS + slack`, with the round-trip count named beside the calls it counts. Both tests assert the RELATION rather than the numbers, and one reads the source to check the count still matches the calls actually made — a third inspect call added later is exactly how this comes back silently. Consequence while it was broken: the roster stopped advancing and the System tab's rows went stale, with a traceback per attempt. Never an outage — `refresh_celery_roster` catches and returns, `/api/system/health` kept answering 200 throughout, which the same log shows. ## Observed, not fixed here `worker_control.inspect_lanes_sync` makes FOUR of these broadcasts (active_queues, stats, active, reserved) at 2.0s each — roughly 8s — and `lane_view` awaits it with no deadline at all. That is the Settings -> Activity -> Worker lanes card, so that card likely takes ~8s to load, and the composite healthcheck carries the same cost against its 15s timeout. Reported to the operator rather than changed: they are mid-deploy, and the fix is to cut round trips rather than raise a number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:32:09 -04:00
bvandeusenandClaude Opus 5	b74a4c964b	fix: the image brings its own PID 1 instead of asking for init: true (4295) ... Operator, 2026-09-23: *"it is out of the norm to require this init call we need to fix this."* Correct, and it is the same mistake as declaring the healthcheck per service — the image needing a deployment to remember a flag before it behaves correctly. PID 1 carries a duty no other process has: every orphaned process in the container reparents to it and must be reaped or it stays a zombie holding a PID slot. This app makes orphans in normal operation — six service modules shell out (gallery_dl, thumbnailer, backup_service, external_fetch, download_service, download_backends) and celery's prefork pool forks children that spawn them. Whatever the role, something that is not an init ends up as PID 1: supervisord for `all`, hypercorn for `web`, celery for a worker. `init: true` covered that, and cost correctness the moment it was forgotten or silently dropped — an older Swarm, a plain `docker run`, a compose file someone copied. No signal either way. So tini goes in the image and is the ENTRYPOINT. `docker run <image>` is correct on its own now, `init: true` comes out of docker-compose.single.yml, and nothing downstream has to know. The smoke asserts /proc/1/comm is tini, read from /proc because the runtime stage installs no `ps`. ## A correction to what I told the operator I justified `init: true` by saying supervisord "has no idea about orphans it never started". That is very likely wrong: supervisord's reaper calls waitpid(-1) and logs "reaped unknown pid" for children it did not spawn, so it does reap orphans. I asserted the mechanism without checking it, and could not check it here — `supervisor` is not installed in this environment. It does not change this commit. tini is correct whichever way that lands, and it covers the single-role containers too, where celery or hypercorn is PID 1 and the subprocess-spawning is heaviest. But the reason I gave was not a verified one and should not have been stated as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:18:47 -04:00
bvandeusenandClaude Opus 5	22dcbcfb74	fix: the lane/program guard compared every section, not just programs (4295) ... `test_every_lane_gets_a_program` asserted `set(cp.sections()) - {"supervisord"} == expected`, so it failed the moment the generated config grew non-program sections. b09ee87 added three — the control socket that lets `supervisorctl` reach supervisord — and the property the test exists for had not moved at all: every lane still has a program and no program is not a lane. Now compared over `program:` sections only, both directions. A guard that fires on a correct change is one people learn to edit rather than read, which costs more than it catches. I reported b09ee87 as green off runs 7328 and 7330 and did not look at 7329, which was red on this. The image itself smoked clean — that part was true — but "green" was a claim about the commit and I checked two of its three runs. Earlier in this session I had been reading all three every time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:06:36 -04:00
bvandeusenandClaude Opus 5	b2da3acce9	feat: wait for Postgres and Redis before starting work (4295) ... Operator, 2026-09-23: *"it's a single container that need to connect successfully to redis and postgres before starting work shouldn't that simply be a check (with retries) at the start of the container."* Yes, and the consolidated layout makes it necessary rather than tidy. Swarm has no ordering primitive — it ignores `depends_on` outright — so every service in a stack starts at once and this container has always raced its own database on a cold deploy. The multi-service stack hid how sharp that is: a `web` task that failed `alembic upgrade head` against a still-initialising Postgres simply died, and Swarm restarted it until it worked. Nobody ever saw a problem worth naming. Consolidation removes that safety net. Each supervisord program gets `startretries=3`, so three quick failures put the program in FATAL and leave it there — supervisord keeps running, the container keeps running, and the application never starts. It would present as a permanently unhealthy container whose image was fine and whose database merely took twenty seconds to initialise, which is a miserable thing to debug on a first deploy. A TCP connect, not a query: the same probe ci.yml's integration lane and the build smoke already use. It answers the question actually being asked — is something listening — and cannot fail for a reason that retrying will never fix. A real query would be a stronger readiness signal and a worse gate, since a wrong password or a missing database is not transient, and a loop waiting for one to heal turns a five-second misconfiguration into a two-minute timeout with a misleading message. Those belong to alembic, which runs seconds later and says exactly what is wrong. Targets are derived from the same env the application reads, so the wait cannot drift from what the app will actually connect to — a gate checking a different host than the app uses is worse than no gate. Bounded at 120s (rule 156), reporting every few attempts so `docker logs` on a waiting container says what it is waiting for. Skipped for `shell`, which exists precisely for when something else is broken and you want a prompt rather than a gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:17:58 -04:00
bvandeusenandClaude Opus 5	b09ee87255	fix: supervisorctl could not reach supervisord inside the container (4295) ... The smoke's own diagnostic line printed this for a whole run and passed, because it was behind `|| true`: Error: .ini file does not include supervisorctl section supervisord was fine. `supervisorctl` simply could not talk to it — the generated config had no `[unix_http_server]`, `[supervisorctl]` or `[rpcinterface:supervisor]`. That is the first tool anyone reaches for when a lane misbehaves in the consolidated container. `docker exec <c> supervisorctl status` to see which processes are actually up; `restart ml` to bounce one without taking the whole application down with it. Consolidation took `docker ps` away as the way to see the lanes, and this is what replaces it — so shipping without it would have left an operator with one container, five processes inside it, and no way to ask about any of them. They are about to run this in production. The test asserts the three sections AGREE on one socket path rather than merely existing: a serverurl pointing where nothing listens fails in exactly the same way and reads as configured. The smoke's line loses its `|| true`. A diagnostic allowed to fail silently is one that stops being true without telling anyone — which is precisely what happened here. It still printed the evidence that something was wrong while nothing depended on it, which is the argument for printing it at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:09:11 -04:00
bvandeusenandClaude Opus 5	efde3b188f	refactor: the image carries its own healthcheck and picks it by role (4295) ... Operator, 2026-09-23: *"why isn't the healthcheck built into the image or base on what command runs if one is passed in. why is it manually declared in the stack here."* No good reason. The container is the only thing that knows what it was asked to run, and every compose file, stack file and README had to restate it: web -> urllib /api/health worker -> celery inspect ping -d celery@$HOSTNAME all -> both, for every lane Three checks written by hand, once per service, in every file anyone ever wrote — none of them wrong until a role changed, and all of them silently wrong after. The same duplication the lane table exists to remove one level down, and I built it without noticing. `entrypoint.sh` now records the role it started. The Dockerfile declares ONE `HEALTHCHECK` that reads it and asks the right question: HTTP for web, a self-addressed celery ping for a worker lane, both-for-every-lane for `all`, and nothing for shell/alembic, which are one-shot and have no liveness to probe. `docker-compose.single.yml` and the consolidated stack declare none. A service that wants something else can still declare its own; docker prefers it, so the escape hatch is the default docker behaviour rather than a flag. Two details that are load-bearing: * The role is written ONCE, by the outermost invocation. `all` starts the other roles through this same script under supervisord, and a child overwriting the container's role would turn the composite check into a web-only one — silently, and only on the consolidated path. FC_ROLE is exported so a child sees it set and skips. * The celery ping is addressed to THIS node, not a bare ping. A bare one is answered by any worker on the broker, so in a stack with replicas a dead container would report healthy for as long as a sibling lived — the check would be measuring the cluster rather than the container it is inside. `healthcheck_all.py` is deleted; its two probes moved into the dispatcher rather than being a second copy beside it. An unrecorded role PASSES. The entrypoint always writes the file, so the only way to miss it is bypassing the entrypoint — a debugging shape, where a check that cannot tell what it is looking at must not assert the thing is broken (snippet #3969). Said on stdout rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:07:39 -04:00
bvandeusenandClaude Opus 5	828c6a5ae3	fix: each lane needs its own celery node name, or three of four vanish (4295) ... Found by the all-role smoke on its very first execution (run 7319), which is the whole argument for having added it one commit ago. Celery's default node name is `celery@<hostname>`. In the single-container layout all four lanes share one hostname, so all four registered as the SAME node. Celery says so itself: DuplicateNodenameWarning: Received multiple replies from node name: celery@72adc5b706a7 `inspect` collapses four replies into one dict key and the last one wins, so three lanes read as absent — and WHICH three varies between calls: lanes not answering: maintenance_long, ml, worker lanes not answering: maintenance_long, scheduler, worker Fatal twice over: * The composite healthcheck can never pass. In Swarm that is a container that never goes healthy — restart loop, then an automatic rollback of a deploy whose image was fine. * `pool_grow`/`pool_shrink` take a `destination` of node names. The UI dial and the autoscaler would have resized whichever lane happened to answer rather than the one asked for — silently, and differently each time. Every celery role now starts with `-n "${CELERY_NODENAME:-celery}@%h"`, and the generated supervisord config sets that per lane. The lanes become worker@<cid>, scheduler@<cid>, maintenance_long@<cid>, ml@<cid> — distinct, so inspect keeps four entries and `destination` addresses what it names. `inspect_lanes_sync` maps hostname to lane by QUEUES, so nothing there changes; it just stops having three of its four entries overwritten. Unset, it falls back to `celery` — exactly celery's own default — so every service in the multi-service stack is byte-identical to before, including the `celery@$HOSTNAME` healthcheck in docker-compose.yml and in the operator's Swarm stack. The test asserts DISTINCTNESS across the whole lane table rather than a fixed string per lane. The property that broke is that no two collide, and stating it that way keeps holding when a lane is added. This is the bug I said a live deploy was needed to find, found in CI instead for the price of one `docker run` — and it would have met the operator as a rollback loop on their first consolidated deploy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:52:56 -04:00
bvandeusenandClaude Opus 5	43ac737516	feat: the whole application is what the image runs by default (4296) ... Operator, 2026-09-23: *"I also want to see that we remove the need for the command line of the configuration in the consolidated version."* `CMD` was `web`, so the single-container layout only worked if you knew to ask for it by name. A compose file that forgot `command: ["all"]` got a web server with nothing processing its queues — a gallery that loads, accepts an import, and never finishes one. Nothing errors; it just never progresses. Now `docker run fabledcurator` with no command starts hypercorn plus every lane under supervisord. `entrypoint.sh`'s own default moves with it, since the two are doors to the same decision and a disagreement would only show up as `--entrypoint` behaving differently from a plain run. `docker-compose.single.yml` drops its `command:` line; `["all"]` still works and still means the same thing. The multi-service stack is untouched — every service there names its role explicitly, which is what makes it the multi-service stack. ## And CI now actually boots it This is the gap I should have named when I reported milestone 422 at 7/7 and did not. Measured, not inferred: the smoke booted role `web` only (build.yml:1454), nothing in CI ran `all`, `docker-compose.single.yml` was read as TEXT by one test checking stop_grace_period and never run, and test_gen_supervisord asserts the generated config against the lane table without ever handing it to supervisord. So the shape this milestone is NAMED for had started nowhere. Steps 5-7 were marked done on evidence that did not cover it, and the operator is about to collapse their production stack onto exactly that. The smoke now boots the image with NO command — checking the Dockerfile CMD, the entrypoint default and the role together, the way an adopter gets it — and asserts `healthcheck_all`, which was itself never executed. That check passes only when hypercorn answers AND every lane in the table answers the broker; a web-only check goes green with every worker dead, which is the failure mode consolidation creates. It then prints `supervisorctl status`, so a lane that is merely restart-looping is visible rather than inferred. Cheap because ml ships at 0 slots and disabled: nothing loads a model, and the lane answers `inspect` with its consumers cancelled, which is what healthy means for a disabled lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:48:11 -04:00
bvandeusenandClaude Opus 5	2677ce020c	fix: the ml artifact was in a second list I never grepped for (4311) ... ac70f2a removed the `ml` image but CI went red on six tests: `tests/test_artifact_identity.py` parametrises over its own `ARTIFACTS = ("web", "ml", "agent", "extension")`, and every case now hits the dispatch guard that same commit added. My miss, and a specific one. I grepped for `fabledcurator-ml` and `ML_PATHS` and called the survey done — but the artifact is also named as a bare `"ml"`, which neither pattern finds. Rule 90 (grep pinned tests when changing a shared symbol) was surfaced to me while I was making the change and I ran a narrower sweep than it asks for. Lesson #4275 names the shape exactly: an absence claim is only as good as the search behind it, and a grep that matched nothing looks identical to a grep that asked the wrong question. The re-run was done by value, not by name: every occurrence of a bare `ml` in the repo, then filtering. That distinguishes the two things the token means — the celery LANE `ml` and the `backend/app/services/ml` package both stay and account for nearly every hit; only the IMAGE name went. Worth stating in the test, since the next person to grep will hit the same ambiguity. Also swept the prose the first pass left describing the old pipeline: "Four artifacts" (README, ci-requirements), "leaves web and ml alone" (×5 in build.yml, plus both docs), "CI publishes it alongside the web/ml images" (agent/README). The run 4896 build-time measurements keep their `ml` number — that was measured when ml was a real build — with a note saying the name has since gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:32:49 -04:00
bvandeusenandClaude Opus 5	ac70f2aadc	refactor: retire the fabledcurator-ml image name entirely (4311) ... Operator, 2026-09-23: *"we don't need to keep this as I'll be collapsing my stack to the same consolidated version that we're building here."* That was the only thing holding the name up. `fabledcurator-ml` has been the same bytes as `fabledcurator` since milestone 422 step 6, and 8152684 had already stopped rebuilding it — this removes the name. Gone: the whole `build-ml` job (137 more lines), its entry in promote's `needs` and in promote's tag loop, `ML_PATHS` and the `ml)` case in artifacts.sh, the `ml` artifact in test_artifact_paths, the image in release_notes.py and its test, and the README row. `docker-compose.yml`'s ml-worker service now runs `fabledcurator:latest` with `command: ["ml-worker"]` — the service still exists for anyone who wants lane separation, it just no longer needs a second image name. Prose that said "all three images" is now "both", except where it means three VALUES (revision/version/epoch) or records what happened on 2026-08-30, when there genuinely were three. Published tags are left alone. Nothing new goes to that name; what is already in the registry stays pullable, so a stack that has not moved yet keeps running rather than losing its image mid-flight. ## A latent trap this made reachable, found by running the script `artifacts.sh revision ml` did not fail. It printed the usage line to stderr and answered with **the newest commit in the whole repository** — a real-looking 12-char sha on stdout, exit 0. `newest()` inlined the path set as `git log ... -- $(cmd_paths "$1")`, and `usage` exits from the command SUBSHELL, so the substitution came back empty and `git log HEAD --` had no pathspec left to filter by. The reuse check would have compared that answer against a published label, missed, and rebuilt on every push forever with nothing going red. It could not be fixed inside `newest`, which was my first attempt: every caller wraps it in a substitution too, so its `exit` also died in a subshell — stdout went empty but the status stayed 0. The guard has to run in the main shell, so it validates the artifact name at dispatch. `newest` keeps a defensive `|| exit 2` for a future caller that reaches it another way, and the comment says which of the two is the real check. Latent while every name callers passed was valid. Removing `ml` from the set is what made a name that used to work start taking the silent path, so the test that pins it lands here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:30:47 -04:00
bvandeusenandClaude Opus 5	f0595e43e6	docs: record what the alias push actually costs — zero layer bytes (4311) ... The comment said the first run would answer whether the registry shares blobs across two repositories of one owner. Run 7300 answered it: every layer came back "Layer already exists" or "Mounted from bvandeusen/fabledcurator". The push moves a manifest and nothing else. 22 seconds, against the ~2 minutes the duplicate build cost (run 7282: build-ml 1m55s for output identical to build-web's). Also noted: the pushed manifest gets its own digest, because docker re-serialises it — same layers, same config, same fc.revision, different manifest bytes. Nothing compares the two, but the digests differing is the first thing someone will notice and misread as a second build still happening. Leaving a "we do not know yet" in the file once the run has said would make it the kind of stale note that misleads the next person to read it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 07:21:56 -04:00
bvandeusenandClaude Opus 5	815268418e	perf: stop building the same image twice under two names (4311) ... `fabledcurator-ml` has been the SAME IMAGE as `fabledcurator` since milestone 422 step 6 merged the ML layers into the one Dockerfile. `build-ml` was still running `file: Dockerfile, context: .` — byte for byte what `build-web` builds — and publishing the result under a second name. So CI did the whole build twice. Measured on run 7282, cold cache: build-web 1m54s, build-ml 1m55s, for identical output, plus a second push of a few hundred MB. Operator, 2026-09-22: *"please fix the CI so it doesn't do this superfluous work."* build-ml now publishes nothing of its own. It pulls the manifest build-web just pushed, re-tags it, and pushes that — so the second name goes on working for the operator's Swarm stack, which still references it, while CI stops compiling anything twice. 437 lines of duplicated build machinery go with it: its own buildx setup, its own reuse check, its own version derivation, its own tag repoint. It now `needs: [build-web]` rather than running in parallel. That is the one cost, and it is seconds: there is nothing to copy until build-web has decided what this run publishes, and the layers are already on the runner. Deliberately NOT `imagetools create`, which would have been the obvious one-liner. That wraps its source in an INDEX and `.Image.Config.Labels` does not resolve through one (#3183, run 4751) — and it would bite harder here than it did there, because `promote` asks the registry for IMAGE manifest media types ONLY, so an index at :refresh-candidate would fail its `test -n "$CT"` and break the weekly refresh for this name. Plain pull/tag/push keeps a plain manifest, which rule 40 also asks for. The step reads the tag back and fails if `fc.revision` stops resolving, so a future change that reintroduces index-wrapping says so instead of quietly costing the refresh. The tag list mirrors build-web's exactly, refresh included: a refresh writes :refresh-candidate alone and `promote` moves it to :latest after smoke-web passes. So this name keeps the gate it has today and `promote` needs no change — it already loops over all three image names. Source is build-web's `digest` when it built, else its `published_digest` on a reuse hit — the digest its channel tag already names, which is what a hit MEANS. With neither, the job fails with a reason rather than publishing something arbitrary. This does not retire the name; #4311 still does that, and is still gated on the operator's stack pointing ml-worker at `fabledcurator:latest` with `command: ["ml-worker"]`. Until then nothing of theirs changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 07:20:30 -04:00
bvandeusenandClaude Opus 5	187b6d2cdf	fix: the smoke always runs, so a change to the smoke can verify itself (4323) ... This workflow file is in no artifact's path set — correctly, since editing it changes no shipped byte. But smoke-web was gated on build-web having published something, so a commit touching ONLY the smoke moved no revision, hit reuse, emitted no digest, and skipped the smoke. The one commit whose purpose is changing this check was the one commit that could not run it. Twice already: 5ca1058 added the egress sandbox and went green three times with smoke-web SKIPPED; 7175ace fixed the bug hiding behind those greens (#4319) and needed a manual force_build to exercise. Both relied on someone remembering. It is also where the other historical skip lived — run 5290's `if:` read `env`, which a job condition cannot see, so it evaluated empty and skipped silently. Two skips, one expression. The expression goes. The job now smokes whichever manifest is current: the digest this run built, or — on a reuse hit — the one the channel tag already names, which the reuse step resolves anyway to read its fc.revision label and now exports as `published_digest`. Always a digest, never a tag (#4290). Kept separate from `digest`, which the :c-<sha> repoint reads and must go on meaning "what this run built". With neither available the job FAILS with a reason rather than passing quietly. Adding build.yml to WEB_PATHS would also make the smoke run, and would be wrong: fc.revision means "the commit this artifact's shipped files last changed in", so moving it for a CI edit makes the label lie and rebuilds three images for a change none of them ship. The problem was never the artifact's identity — it was when the guard runs. Two more defects in the same block, both from 5ca1058, both found by run 7288 — which only existed because the force_build above ran the check a second time: * `NET=smoke-noegress-$$` is not unique. The shell's pid is deterministic in this runner — every execution got 157 — so the second run died on "network with name smoke-noegress-157 already exists". A pid is unique among live processes, which is not unique over time. * The network's cleanup trap was destroyed before it could fire. `trap ... EXIT` REPLACES the previous handler, and the container's trap was installed further down, so every run leaked its network. Invisible in a passing run; it can only ever surface on the NEXT one. Now one EXIT handler does both, armed beside the network and tolerant of an empty CID so it covers failures before the container exists. Plus a sweep of any network earlier runs leaked, which fails harmlessly on one still in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 11:20:36 -04:00
bvandeusenandClaude Opus 5	7175ace67a	fix: the smoke's health probe has to run inside the sandbox it created (4319) ... The first real execution of the egress-blocked smoke (run 7282) failed with "web is running but never answered /api/health" — and the application was perfect. Its own log shows all four hypercorn workers serving three seconds after start and still up seven minutes later, with no internet: [entrypoint] Starting hypercorn on :8080 [14:08:58] [10] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [11] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [12] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [13] [INFO] Running on http://0.0.0.0:8080 5ca1058 put the app container on an `--internal` network. Docker gives such a network isolation rules that DROP traffic entering it from any other interface, and this job's own container sits on the runner's default bridge — so its curl to the app was discarded before arrival. Dropped rather than refused, so every attempt burned the full --max-time and the loop read as a wedged app instead of an unroutable address. Steps 0-2 were right only by accident: each already runs its check inside a container on $NET. Step 3 was the one place that reached in from outside, and so the one place that could not work. It now probes from inside too, using the image's own python3 over `shell -c` — the same shape as the egress guard above it, and necessary because the runtime stage ships no curl. Attaching the job container to $NET would also work in one line. Rejected: it puts an internet-connected container on the network whose whole purpose is being offline, and it would rest on `hostname` equalling the container id. Same family as #3374 — a CI check that could never pass, failing in a way that accuses the thing it was meant to protect. Worth stating plainly: the egress property itself PASSED on 7282 ("smoke: no route out, as required") and the schema built from empty through 0104 with no network. Only the harness's last step was broken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 10:16:17 -04:00
bvandeusenandClaude Opus 5	a01165365b	feat: a saturated lane can grow itself, within the cap the operator set (4297) ... Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 10:05:43 -04:00
bvandeusenandClaude Opus 5	5ca1058fb5	test: the smoke runs with egress blocked, on every build, and proves the block (4296) ... Milestone 422 step 6's rule-164 half. THE PROPERTY WAS NEVER TESTED, not even weekly. smoke-web already booted the built image against real Postgres and Redis — but on the runner's default network, with the internet one hop away. It proved the image WORKS; it never proved it works OFFLINE, which is the thing rule 164 is about and the thing step 6 put at risk by moving download_models out of boot. Now on an `--internal` docker network, which is the mechanism rule 164's own verify_with names. `--network none` is explicitly the wrong check here: it would only prove the app fails without a database, which says nothing about egress. Internal blocks the default route while leaving container-to-container traffic and embedded DNS intact, so Postgres and Redis stay reachable and nothing else does. The service containers are runner-created siblings, so they are ATTACHED to the network rather than created on it, and their addresses are re-read on it — the bridge IPs discovered earlier are not routable from a container that is only on the internal network. A STEP THAT PROVES THE SANDBOX IS SEALED, before anything depends on it. It tries to reach 1.1.1.1:443 from inside the candidate and fails the job if it succeeds. Without it the rest is theatre: if `--internal` silently stopped working, or the container picked up a second network, every check below would pass with the internet available and report an offline boot that never happened. A guard that cannot fail is not a guard (rule 167). IT RUNS ON EVERY BUILD, not just the weekly refresh. The egress property is broken by a code or Dockerfile change — a push — so checking it only on the refresh would test it on the one trigger that changes no source. Addressed by the DIGEST build-web published rather than by a tag: a tag can move between the build and the smoke, and then the check reports on bytes nobody built here. A reuse hit is skipped, because those bytes were smoked when built. WHAT THIS STILL IS NOT, filed as #4310: on a push it runs AFTER build-web has written the channel tag, so it detects rather than gates. Rule 164 asks for the check BETWEEN build and push. Closing that means giving the push path the candidate-then-promote shape the refresh already has — per-channel candidate tags, promote learning its channel, and the :c-<sha> repoint moving after the gate. That is a redesign of the path that publishes production and it is not something to fold into a test change. Also filed #4311: retiring the fabledcurator-ml image NAME, gated on the operator moving their Swarm stack file. Same two-phase shape #406 used for pixiv, for the same reason — dropping it would not break their deploy, it would freeze it silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 09:50:25 -04:00
bvandeusenandClaude Opus 5	0f98e46200	docs: the merged image's cost, measured — and it corrects my own estimate (4296) ... I wrote in ffcd130's Dockerfile comment that merging ML in means "everyone pulls it, including the many who will never turn tagging on", framed as a real cost the milestone accepted. My working estimate behind that was ~4GB. Measured from run 7273's build-web log: torch 2.12.1+cpu wheel 192.3 MB torchvision 0.27.1+cpu 1.8 MB transformers / onnxruntime / opencv / sklearn and friends 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB largest newly-pushed layer 222.07 MB The ML code adds a few HUNDRED MB, not gigabytes. The `--index-url` CPU resolution is what makes that true — the default PyPI torch wheel carries the CUDA runtime and is ~2GB by itself, and the log confirms 2.12.1+cpu resolved, so it is working as intended rather than as intended-but-unverified. Why this matters beyond a comment being wrong: it settles the trade this step was explicitly asked to weigh and could not, and it reverses how close the call looked. Baking the weights in adds ~3.5GB to every pull for a feature many adopters never enable; shipping the code and fetching on demand adds ~350MB. An order of magnitude, where the estimate had them within 15% of each other. Off-by-default is not a judgement call here, it is arithmetic. The gigabytes were always in the MODEL, and the model is not in the image. Two things NOT measured, still: the total image size (the push only transfers layers the registry lacks, so a push log cannot give it) and the per-slot resident RAM, which stays flagged `measured=False` in the lane table and renders as "about" in the UI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 09:02:47 -04:00
bvandeusenandClaude Opus 5	f174981b07	fix: ml and web are one image, so they are one path set (4296) ... Run 7271. `test_every_copied_path_is_in_the_artifacts_path_set` — a guard I did not know this repo had — caught the merge's real consequence within minutes, and it is precisely #3202's class: ml: Dockerfile copies 'frontend/package.json' into the ml image, but no include in scripts/artifacts.sh covers it. web: Dockerfile copies 'requirements-ml.txt' into the web image, but no include covers it. Both follow from step 6 and neither was visible by reading my own diff. The merged Dockerfile carries the frontend-builder stage, so the ml image now copies frontend files; and the web image now installs the ML requirements. Left as they were, each artifact's derived version would have stopped moving when those files changed, and a pinned build would serve stale bytes — silently, since nothing fails when a version simply does not advance. The fix is structural rather than two more entries. `fabledcurator-ml` IS web's image now: same Dockerfile, same context, same bytes, published under a second name only because the operator's Swarm stack still references it. So ML_PATHS is WEB_PATHS by assignment, and `cmd_paths` gives ml the same deriver and extension append — the XPI is in those bytes too. Two lists describing one image is the duplication this milestone has been collapsing all day. It existed for about an hour and the guard found it first, which is the argument for the guard. I also predicted this failure would be the image build. It was not; the build passed and the unit lane failed. Worth noting because the prediction was confident and wrong, and reading the log took one call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:57:09 -04:00
bvandeusenandClaude Opus 5	ffcd13096a	feat: one image for every lane, with the model fetch gated on enabling (4296) ... Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:51:44 -04:00
bvandeusenandClaude Opus 5	ecbd325437	feat: an optional lane says it is optional, and what enabling it costs (4296) ... Operator, 2026-09-22: "since the ml-worker is optional it should be shown as such in the UI and have a warning about what it does and that it pulls the models and what models and their projected size and ram requirements to run." The card previously said "a few GB, once" — a number sourced from nothing, which is exactly the hand-wave I had flagged in this step's own survey log as something that should be measured rather than asserted. ONE FACT CORRECTED WHILE WRITING THE COPY. I had named the lane "ML tagging". It downloads an EMBEDDER: google/siglip-so400m-patch14-384. WD14 tagging is the GPU agent's job — celery_app.py:5 still names both, but that has been stale since B3 (#1238), when the agent took over and this lane was left as the CPU embed fallback for stacks running no agent (see MLSettings.cpu_embed_enabled). Telling someone the lane "does tagging" would have been wrong in exactly the way this request exists to prevent. The facts are structured data on the lane, not prose in a component: ModelRequirement(repo, approx_download_bytes, approx_resident_bytes, measured). The API carries them; the card renders them. Numbers come from the system, wording from the UI. ML_BYTES_PER_SLOT IS NOW DERIVED from that requirement rather than stated separately. They have to be one number: the figure quoted to the operator before they enable the lane and the figure the cap enforces. Two copies could disagree, and the UI would promise a slot the cap then refuses. `measured=False` travels with the numbers and the card renders "about". They are estimates from the checkpoint's parameter count and dtype — ~877M params at fp32 is ~3.5GB of weights — not from a build. This decides whether someone's server survives, so it is labelled rather than rounded into something that reads like a fact. A test asserts the flag is false, to be flipped in the same commit that records a real measurement. The card now shows: an "optional" chip in the row itself (someone scanning the table should not have to enable a lane to learn it was never required), and before the switch, what the lane does, that you only need it if you are NOT running the GPU agent, the repo id, the download size, the per-slot RAM, and why the ceiling is what it is — including saying plainly when a box has too little memory to run it at all. Keyed on the lane's own `optional` flag, not on the name 'ml', so a second optional lane gets the same treatment without anyone remembering to add it. A test asserts no REQUIRED lane declares a model: if one ever needs a download, it stops being required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:45:22 -04:00
bvandeusenandClaude Opus 5	e579333455	docs: record that the ml :ro loss is a ruled non-issue (4295) ... Operator, 2026-09-22: "I don't care about the :ro loss thank you for calling it out repeated but I don't care." Raised three times across the milestone body, this file's header and two reports. Recorded as settled at the point someone would rediscover it, so it is not raised a fourth time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:38:49 -04:00
bvandeusenandClaude Opus 5	172e33de9a	feat: run web and every worker lane in one container (4295) ... Milestone 422 step 5. `docker compose -f docker-compose.single.yml up -d` gives three containers — FabledCurator, Postgres, Redis — where the stack previously needed seven. THE MULTI-SERVICE STACK IS KEPT. docker-compose.yml still runs the five app services separately and remains the right shape for a Swarm deployment spread across hosts, where per-service rolling rollback and placement constraints matter. This adds a compose file; it deletes none. `entrypoint.sh all` GENERATES the supervisord config from worker_lanes.LANES and execs it as PID 1. Generated rather than checked in because a static .conf would spell out each lane's -Q list, making a FIFTH hand-kept copy of the queue names — after celery_app.task_routes and the three collapsed in steps 1, 2 and 4. Every one of those had already drifted when found. Generating gives a stronger guarantee than "they match today": a lane added to LANES gets a process, and a queue cannot end up with no consumer because someone missed a file. supervisord over s6-overlay: one pip dependency on an image already Python, with per-program stop timeouts and stopasgroup. The process-group part is not a detail — celery's prefork pool forks children, and a TERM reaching only the parent leaves them orphaned holding tasks. s6's advantage (PID-1 signal and zombie handling) comes from `init: true` instead. Nothing in FC talks to the supervisor, so the choice is reversible without touching product code. FOUR LANES, NOT FIVE. The ml lane is skipped: torch and the ML requirements live only in Dockerfile.ml until step 6 merges the images, so an `ml` program here would fail to import on every restart forever. `--with-ml` is the flag step 6 turns on. THREE BUGS FOUND BY READING IT BACK, none of which the first tests caught: 1. `environment=CELERY_QUEUES=default,import,thumbnail,download` — supervisord parses that key as a COMMA-separated list, so it reads as CELERY_QUEUES=default plus three malformed entries and the worker lane would have consumed only `default`. Silent: the worker starts, reports healthy, never picks up an import. Now quoted, and the test asserts the quoted form rather than the bare substring, which passed either way. 2. The generator emitted `entrypoint.sh <lane.name>`, but `maintenance_long` is not a role — compose runs it as the plain `worker` role with different queues. Lane now carries `entrypoint_role`, and a test reads entrypoint.sh to assert every role a lane names actually exists. 3. The `scheduler` role hardcoded --concurrency=1, ignoring CELERY_CONCURRENCY. Harmless while only compose started it and set none; with a generated value being passed, the lane would have sat at 1 until the reconcile noticed, with nothing saying why. The healthcheck asserts BOTH halves — hypercorn answers and every configured lane is answering the broker. That is the failure mode consolidation creates: docker can no longer see the lanes as separate services, so a web-only check would report a healthy container with every lane inside it dead. It deliberately ignores the `enabled` flag: a disabled lane still has a running process with its consumers cancelled, and marking the container unhealthy for turning tagging off would be wrong. stop_grace_period 200s, sized to the slowest lane (maintenance_long at 180s) rather than the average, with a test asserting no program's stopwaitsecs can exceed what compose allows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:32:27 -04:00
bvandeusenandClaude Opus 5	da48edf7da	feat: the worker-lanes card — see each lane, change its slots (4294) ... Milestone 422 step 4. Rule 27: no UI, no ship. This is where the previous three steps become usable. REACHABLE AT: Settings -> Activity -> Worker lanes, directly under the "Queues + workers" pane. Written from opening the view, not from memory of having built it — lesson #4282, and #3463 is the same trap landing inside milestone 365, where the System page shipped with no navigation to it. Under that pane deliberately, not in the System tab. System answers "is everything running", where every control would be about something broken. This is about something working that should work harder, and it belongs beside the backlog it reacts to: you watch a queue grow and give that lane another slot without leaving the pane. Per lane: queues, pending, busy, a stepper, an enable switch. - PENDING is depth + reserved, not LLEN. Celery prefetches, so LLEN alone reads 0 while a worker holds tasks in memory — the number that would make someone think a buried lane was idle. - NOT ANSWERING, never "stopped". present=false means nothing replied; saying stopped would send the operator looking for a crash that has not happened. - THE CEILING IS ON SCREEN, with "(memory)" on the ML lane. It is the one number here the operator cannot change, so it has to justify itself; a greyed stepper with no explanation reads as a bug. - `busy` is per lane, so adjusting one does not freeze the others. TWO OUTCOMES THAT MUST NOT COLLAPSE INTO ONE MESSAGE. A stored-but-unpushed change (applied:false — the lane is restarting) is information: the value is saved and the reconcile will carry it, so the card says so and invites waiting. A refused value (400) is an error and shows the endpoint's own sentence. Collapsing them would make one of the two invite the wrong action. A BUG CAUGHT BEFORE COMMIT: the card read `e.detail?.detail`, but ApiError puts the parsed body on `.body` and `.message` on the short `error` key. It would have shown the operator the bare word "refused" with no reason — the exact failure that line exists to prevent. The test would not have caught it either: `rejects.toThrow()` passes whether the sentence is reachable or not. It now asserts `err.body.detail` specifically. Also: queueOptions in SystemActivityTab was a fourth hand-kept copy of the queue list and had drifted — `maintenance_long` was missing, so activity on that lane could not be filtered for at all despite four task routes pointing there. Added, and DELIBERATELY left as a written-out list rather than derived like the other three were: this filters task_run HISTORY, so a derived list would hide the filter for any queue that has rows but no longer has a lane — precisely when someone is looking — and would be empty whenever the endpoint is down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:23:14 -04:00
bvandeusenandClaude Opus 5	5f8c63f61b	feat: reconcile every running lane back to its stored slots (4293) ... Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by its supervisor comes back at its ENV concurrency, silently below whatever the operator set, and nothing on step 2's write path would ever notice. Storing the value made it survivable; this makes it survive. A BEAT TASK, NOT A HOOK IN WEB — a deliberate deviation from the step as written, for a reason already recorded in this codebase. Step 3 said "web applies the stored values after it starts". It cannot: service_roster.py documents that hypercorn runs --workers 4, so anything in before_serving becomes four concurrent loops per container hammering the broker forever. service_roster's own answer — refresh on demand from whichever request arrives — was also rejected, because the two solve different problems. A stale ROSTER only misleads someone looking at it, so recomputing when they look is exactly right. A lane running at the wrong size is doing less work than it was told to whether or not anyone is watching, and the case that matters is a deploy at 3am followed by a backlog nobody is awake to see. So: unattended, every 5 minutes, on the quick `maintenance` lane beside the other recovery sweeps. Accepted cost — a dead scheduler stops reconciliation, but a dead scheduler already stops every other sweep and the roster reports it, so this adds no new blind spot. A BUG I WROTE AND CAUGHT BEFORE COMMITTING. The first version called set_lane_enabled_sync unconditionally, so a settled system re-sent add_consumer for every queue on every tick — forever. Harmless per call (add_consumer on an already-consumed queue does nothing), unbounded in aggregate, and completely invisible. That is lesson #4183's failure mode exactly, in the very function whose docstring cites it. Worse, my test would not have caught it: it asserted only on grew/shrank. LaneLiveState now carries `consuming` — which queues a lane is actually serving, distinct from the queues it was configured with — so the reconcile compares before acting. The test now asserts ALL FOUR control families are silent on a settled tick, plus a new case for an already-disabled lane, which is the other half of the same fixed point. An absent lane is SKIPPED, not corrected. present=False means nothing answered, not zero slots; correcting it would be a conclusion from an unswept read (snippet #3969), and there would be nothing to send the message to. One lane failing does not stop the others. One inspect serves every lane: the two setters now take an optional pre-fetched LaneLiveState, so a tick costs one broker round trip rather than one per lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:08:38 -04:00
bvandeusenandClaude Opus 5	a9c1b421a7	feat: change a lane's slots on a running system, over the broker (4292) ... Milestone 422 step 2. `GET /api/system/workers` reports every lane joined to its live pool; `POST /api/system/workers/<name>` changes it. NO DOCKER SOCKET. Milestone 365 deferred "acting on the state" because restarting a dead worker needs a socket the web container deliberately does not have. That holds for restarting a CONTAINER; it does not hold for changing how much work a RUNNING worker does. celery's pool_grow / pool_shrink / add_consumer / cancel_consumer send a message over the Redis the app already uses, and the worker resizes itself. No new privilege, no new surface, and the security question that deferred this is never raised. PERSIST AND PUSH, in one call, in that order. pool_grow is not durable — a restart drops every lane to its env concurrency — so a UI that only pushed would lose the setting on the next deploy with nothing to show for it (lesson #4202). Storing alone would describe nothing until something restarted. A failed PUSH is not a failed setting: 200 with `applied: false` and a reason, so the UI says "saved, not yet live" rather than "that didn't work". Step 3's reconcile carries it when the lane answers again. PER-REPLICA DELTAS. `pool_grow(n, destination=[...])` adds n to EACH destination, so while `worker` runs `replicas: 2` a single delta from an aggregate is wrong for both. `slots` therefore means what CELERY_CONCURRENCY means — one process's pool — and each replica is driven to it from its OWN current size, so replicas that drifted apart converge rather than moving in lockstep. I wrote this wrong first: the docstring claimed per-replica while the code computed one delta from the max across replicas. LaneLiveState now carries `pools` per hostname and exposes `pool` as a property. A replica already at the target is sent nothing at all — the reachable fixed point step 3's periodic reconcile needs, or it re-issues a grow of zero every tick forever (lesson #4183). A replica that answered inspect but not stats is NAMED in the error rather than skipped silently, since otherwise it would run at a size the UI claims it does not. `present=False` is not "zero slots", it is "nothing answered" — kept distinct throughout, because step 3 skips an absent lane rather than correcting it. /workers now also reports pool size (from `insp.stats()`) and RESERVED count. Celery prefetches, so tasks that have left the Redis list but not started are invisible to LLEN: a lane can read depth 0 with thirty tasks held in worker memory. `pending` is depth + reserved. The UI is misleading without this and step 7's autoscaler would be simply wrong. Also kills the THIRD copy of the queue list: system_activity's _QUEUE_NAMES, whose own comment admitted the coupling ("must match celery_app.task_routes") and which sat alongside task_routes and the ROLE_NAMES copy step 1 collapsed. Now derived from LANES. The rendered order changes to lane grouping, which is the better shape for a lane-oriented UI. Separate blueprint rather than folding into system_activity, which states in its first line that it is read-only and answers a different question — its /workers is keyed on celery HOSTNAME and reports which nodes answered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:01:23 -04:00
bvandeusenandClaude Opus 5	5974a1bfbc	fix: two errors in the worker-lane tests (4291) ... Both mine, both in tests/test_worker_lanes.py, neither in the code under test. Run 7242. RUFF I001 — two blank lines between the import block and the first module-level comment. Rule 102 names this exact trap ("exactly ONE blank line between imports and a module-level constant/comment/pytestmark") and I was pointed at that rule repeatedly before opening it. SIX FAILURES in test_worker_lane_check_constraints — the test asserted bare constraint names, but Base.metadata's naming_convention has already applied the `ck_worker_lane_` prefix by the time __table__.constraints is read. The failure output is worth keeping: it shows the model emits exactly the three intended constraints, prefixed once — ck_worker_lane_slots_non_negative slots >= 0 ck_worker_lane_cap_non_negative slots_cap >= 0 ck_worker_lane_slots_within_cap slots <= slots_cap — which is the model behaving correctly, and confirms the migration's op.f() names match what the ORM produces. The assertion is now an equality against the prefixed names plus an explicit check for a doubled prefix. That is strictly more valuable than what I wrote: a bare-name assertion would have passed just as happily against `ck_worker_lane_ck_worker_lane_slots_within_cap`, which is the defect alembic 0088 had to rename four constraints for (#3275). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 07:53:30 -04:00
bvandeusenandClaude Opus 5	84f13135ce	feat: worker lanes become rows — slots, a settable cap, a derived ceiling (4291) ... Milestone 422 step 1. The data model the rest of the milestone reads. No behaviour change: nothing consumes these rows yet, and every lane still boots at its CELERY_CONCURRENCY env value. Three numbers, not two, per the operator's distinction — the derived value is a cap ON the cap: slots <= slots_cap <= derived_ceiling (live) (operator) (computed) They can always lower their own cap; they cannot raise it past what the container can hold. The ceiling is never stored, so a row written on a 32GB host and later run in a 4GB container is bounded by the 4GB. `services/worker_lanes.py` is the one place that knows the lane set. `models/worker_lane.py` holds only what an operator may change. Two deviations from the step as written, both deliberate: QUEUES ARE NOT A COLUMN. The step body said the row carries its `-Q` list, but a lane's queues are decided by celery_app's task_routes, not by preference — an operator cannot move a backup off maintenance_long. Storing them would create a row that can contradict the routing table, with nothing to notice until a queue had no consumer. So queues are code, slots are data. `test_every_routed_queue_has_a_lane_that_serves_it` reads the real routing table and fails if a route is ever added without a lane. ROLE_NAMES IS NOW DERIVED, not left alone. It was a hand-kept second copy of "queue set -> display name" and had already drifted: maintenance_long is a live lane with four task routes and a dedicated worker in the operator's stack, and the roster did not know its name — so the System tab labelled it `Worker (maintenance_long)`. Adding a lane table beside it would have made three copies. The ceiling honours cgroup limits rather than the host's. `os.cpu_count()` reports the HOST's cores from inside a container, so a 4-core quota on a 32-core host would otherwise offer 32 slots — and the operator's own stack sets `cpus: '4.0'` on ml-worker, so that is real configuration, not a hypothetical. Memory reads cgroup v2 then v1, and recognises v1's PAGE_SIZE-aligned LONG_MAX sentinel by magnitude rather than treating it as petabytes. Every uncertain case fails LOW. An unreadable limit yields UNKNOWN_CEILING, never unlimited — not knowing how much memory there is must not read as plenty. A box too small to hold one model beside the web process gets an ML ceiling of 0 rather than a floor of 1: offering a slot that OOMs the container the first time it is used is exactly what this exists to prevent. ML_BYTES_PER_SLOT is 4 GiB and is UNMEASURED — flagged as such in the code, with the method for replacing it with a real figure. It decides whether a stranger's server survives enabling tagging, so it errs toward refusing a slot that would have fitted. Seeded one-of-each with ml at 0 and disabled (alembic 0103). ML off is step 6's requirement arriving early: enabling the lane is what triggers the SigLIP download, and rule 164 permits a runtime fetch only for a feature that is optional and clearly off. The seed values are literals rather than an import of LANES — a migration is a statement about one moment, and importing the live defaults would silently change what this revision does on a fresh database in 2027. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 07:48:25 -04:00
bvandeusenandClaude Opus 5	aa2bb665b9	fix: serialise builds per branch, and pin :c-<sha> to the digest this run built (4290) ... Two independent defects from one missing mechanism. `grep -rn concurrency .forgejo/workflows/` returned nothing, so two pushes to a branch ran build.yml in full parallel with no ordering. Both jobs read `fc.revision` off the channel tag before either has pushed, so both miss the reuse check and both build. Whichever finishes LAST owns the tag. Half 1 — the rolling tag. A slower older build leaves `:dev` carrying content older than the commit that moved it. Family rule 146 says a rolling channel refreshes itself; this is the case where it quietly does not. Self-healing on the next push. Half 2, and the reason this is not filed low — `:c-<sha>`. The repoint step wrote every non-channel tag by copying the channel tag BY NAME, so the immutable rollback tag named whatever `:latest` pointed at when that step happened to run. Lose the race and `:c-<shaA>` names the OTHER run's bytes. Rule 145 makes that tag the rollback unit and immutable, so this does not break immutability — it makes the tag wrong from birth, and immutability then guarantees nobody ever corrects it. Nothing goes red; it surfaces the day someone rolls back and gets a commit they did not choose. Not self-healing. Two fixes, deliberately both: A. A workflow-level `concurrency` group keyed on `github.ref`, so dev and main never block each other. `cancel-in-progress: false` — queue, never cancel: cancelling could kill sign-extension mid-AMO-upload, leaving the version registered at AMO with no cached asset, which is the unrecoverable stuck state that job's rollback trap exists to prevent, reached by another door. Not keyed on BUILD_REF because the group is evaluated before any job starts and cannot read the `env` context. B. Each build-push-action step gains `id: build`, and the repoint step copies from `$IMAGE@${{ steps.build.outputs.digest }}` — the manifest THIS run pushed — rather than from the channel tag by name. On a reuse hit there is no digest and the channel tag remains correct by definition: "hit" means that tag already carries this commit's fc.revision. B is not redundant with A. A depends on this Gitea honouring a key whose failure mode is silent, and this file has been burned by exactly that before (the `format()` note records `true == 'true'` evaluating FALSE on run 5270, every lane green, the feature simply not happening). B holds the :c-<sha> correctness property whether or not A is honoured. The loop's exclusion is now keyed on CHANNEL_REF rather than on SOURCE, and that is load-bearing: SOURCE may now be a digest ref, which never equals a tag string, so testing against it would stop excluding the channel tag. imagetools would index-wrap it and `.Image.Config.Labels` would stop resolving, killing the reuse label permanently — #3183 arriving again. Neither half is verifiable on a dev push: dev's tag list is the channel tag alone, so the repoint correctly does nothing there. B is observable on the next merge to main (:c-<sha>'s digest must equal the build step's), and A by pushing twice in quick succession and reading the run list for a queued second run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 22:07:53 -04:00
bvandeusenandClaude Opus 5	bd92fb46b3	test: pin the JS<->Py artist-pattern mirror with a shared sample table (3093) ... `PLATFORM_ARTIST_PATTERNS` (extension/lib/platforms.js) and `_PLATFORM_PATTERNS` (extension_service.py) are two hand-kept copies of one table whose only guard was the comment "keep in sync by hand; reviewers catch drift" — the same guarantee manifest.json had before #3069, where deviantart sat in the manifest for seven weeks after the product dropped it. Drift here is worse than the manifest case, because the two copies gate opposite halves of ONE interaction: the JS copy decides whether the "Add to FC" button appears, the Python copy decides whether the resulting POST is accepted. JS looser than Py shows a button that 400s; Py looser than JS silently never offers a button for a URL the backend would take. #1485 (the Patreon /c/ and /cw/ shapes) was the second of those, and its fix had to be applied to both files by hand. The two-runtimes objection to a shared SOURCE file is fair, so this tests the invariant instead of the source. `extension/test/artist-url-samples.json` is one table of 25 URL samples — match (with the expected slug) and no_match, each with a `why` — read by BOTH suites and asserted against each one's own copy of the patterns. Neither runtime imports the other; a change to one copy alone turns the other runtime's suite red. Both halves also assert their own coverage: the sample platforms must equal the platforms that actually have an artist pattern, and every platform must have samples in both directions. Without that, deleting a platform's samples would make the guard pass by testing less. Discord is deliberately in neither table — it is channel-based and has no creator page to put a button on. The no_match half asserts `_derive` RAISES rather than merely missing the platform: it tries every pattern in turn, so a nav page some other platform's pattern happened to swallow would still be accepted by the backend — the same defect wearing a different platform name. Samples live under extension/test/ because that path is excluded from both the XPI file set and the extension version derivation (packaging.sh: NOT_PACKAGED_TRACKED and NOT_VERSION_RELEVANT both carry `test/**`), so adding samples ships no bytes and forces no re-sign. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 21:04:53 -04:00
bvandeusenandClaude Opus 5	24b10d0ffa	feat: retire pixiv entirely — delete its code, its ledgers, its credential (3977, 3978, 3979) ... Milestone #406 phase 2, with issue #3980 folded in. Phase 1 (2026-09-13) unregistered pixiv so nothing could reach it; the code has sat in the tree uncalled since. DeviantArt is why the second half is not left for later — #3069 retired it in code on 2026-08-27 and its stored session was still in the database seven weeks on. Step 5 — the code. Deletes pixiv_client, pixiv_downloader, pixiv_ingester, platforms/pixiv and their three test modules and fixture, then edits out every remaining reference: the dispatch entry, the campaign-id and verify branches in download_backends, the display-name branch in extension_service, and the comments that still described pixiv as live. The consolidation check the step asked for comes back negative: native_ingest_common has seven non-pixiv callers (patreon, subscribestar, membership_reconcile, membership_roster, ingest_core), so nothing there drops to a single user. Step 6 — the data, alembic 0102. Drops pixiv_seen_media and pixiv_failed_media, and deletes credential rows whose platform is not registered. Written as "not registered" rather than "pixiv" at the step's explicit ask, which is what makes one migration cover two retirements: the pixiv OAuth refresh token and DeviantArt's leftover session (#3980). It is also the only way either row can go — the credentials UI renders one card per platform from /api/platforms and looks the credential up by key, so an unregistered platform's row has no card and no Remove button. Pixiv's Source rows are KEPT, changing the milestone's original data table on the operator's call. `platform` is stored only on Source; neither Post nor ImageRecord carries it. Both FKs are ON DELETE SET NULL, so a delete would not lose the art — but it would drop every pixiv image into the gallery's __unsourced__ bucket and strip the platform chip off every pixiv post. The rows stay disabled (0097) and unregistered, so nothing schedules or downloads through them. Keeping them costs nothing and keeps the attribution that "the art already downloaded from pixiv stays" is about. Step 7 — the guard. test_pixiv_code_and_tables_are_gone asserts absence from the module table and from Base.metadata, not from prose (snippet #3352's trap). The extension and registry negative assertions were already in place from phase 1. The final sweep found one real residue step 4 missed: extension/README.md still advertised pixiv support and carried a "Pixiv OAuth" manual-test item. Also replaces the two deleted dispatch tests with one over the whole NATIVE_INGESTER_PLATFORMS set, so adding a platform and forgetting its ingester class now fails at unit level rather than as a mid-download KeyError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 20:46:15 -04:00
bvandeusenandClaude Opus 5	5aa8e3d81b	fix: a stopped source is not a failing one, and cannot be deep-scanned (4279) ... Ebi77 sat in the "1 source is failing" banner for six days with no action available, reading `stranded by recovery sweep (no terminal status after time_limit)`. Four things lined up: 1. The membership sweep did its job — saw `former_patron`, disabled the source, cleared its failure state. Clean at 02:50. 2. Twenty minutes later a deep scan was armed on it. `/backfill` had a credential pre-flight but NO `enabled` guard, while `/check` has carried one all along. The two trigger endpoints disagreed, and the ungated one is the one that arms the long walk. 3. Without a membership the walk cannot finish, never reaches a terminal status, and the recovery sweep strands it with consecutive_failures = 1. 4. Nothing could clear that. A disabled source is never scheduled, so no successful run resets the count; `SourceService.update` clears only on an explicit disable and it was already disabled; and the banner's Retry routes to `/check`, which refuses a disabled source. The card offered a button structurally incapable of acting on the only source it was showing. `failing_sources_clause()` now means "enabled AND erroring". That also settles a disagreement its two callers already had: the scheduler's count paired it with `enabled.is_(True)` and `SourceService.list(failing=True)` did not, so one counted Ebi77 and the other did not — exactly the drift the note above that function warns about, which is why the test belongs IN the predicate rather than beside it. The scheduler's now-duplicate clause is dropped so one place decides. `/backfill` gains the guard for start/recover/recapture. `stop` stays open on a disabled source, or arming becomes a one-way door. Migration 0101 clears failure state on sources that are already disabled — the predicate fixes what the surfaces report, not what the rows carry, and the rows are why the operator had no way out (lesson #4202). It matches what `update` already does on an explicit disable, so rows disabled by any other path come into line. Enabled sources are untouched: a real failure on a live source must keep showing, which the second new test pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 19:25:42 -04:00
bvandeusenandClaude Opus 5	11a01a9686	revert: remove the placement reconciler — it manufactured the problem it solved ... Milestone #421 built a sweep that compared each image's `artist_id` to the name of the directory holding its file, and called every mismatch a misplaced image. It reported 33,789 of 63,605 as wrongly filed. That number described the comparison, not the library. What it actually was: 32,475 (97.1%) one artist's own folder, spelled differently — Telepurte/ vs telepurte/. Same artist, same art. 657 ( 2.0%) loose at the images root 328 ( 1.0%) in a folder named after a different artist And the 1% did not mean what the tool assumed either. `ImageProvenance` records the post and source every file was downloaded from — the authoritative answer, which the tool never consulted. Querying it for all 328: 144 provenance agrees with the record (move would be right) 87 provenance agrees with the FOLDER (the record is wrong; move wrong) 53 provenance names SEVERAL artists (no single correct folder) 41 no provenance at all 3 agrees with neither So the sweep would have misfiled or arbitrarily picked for ~41% of the only set it was really needed for. The system already knew where each file came from; the tool inferred it from a column and a directory name instead. Operator, 2026-09-21: *"the current system consistently records where items are and where they came from this is just complicating something works and doesn't need fixing."* Correct on both counts. Removed: the service, the tasks, the model and migration 0099's table, the /api/cleanup/layout and /placement/* endpoints, the Maintenance card and its store actions, and the tests. 0100 drops the table (rule #22 — no legacy). KEPT deliberately, per the operator: - `utils.paths.canonical_subdir` — new filesystem imports derive their directory from the artist's slug, matching what the downloader always did. Not part of this tool; removing it would be churn that fixes nothing. - The 327 files run 1 moved (InsoUwu/ -> insouwu/). Same artist either way, and the gallery renders them correctly. - Everything from #4223 (three-gate dedup, 256-bit pHash) and #4234 (backup credential exclusion). Those fixed problems that were actually reported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 18:17:13 -04:00
bvandeusenandClaude Opus 5	2dd9b956d5	feat: File placement card — survey, plan per artist, apply, put back (4246, slice 3c) ... The UI half of the reconciler, in Maintenance. Survey shows how many images sit in the wrong artist's folder and which folders they are in; each artist gets its own Plan button; each run can be reviewed, applied, and put back. Deliberately NOT using useMaintenanceTask. That composable stashes a task id in localStorage so a result survives navigate-away, which is the right answer when the only record is a Celery result. Here the runs are database rows — so a reload, another machine, or coming back tomorrow simply shows the same state, because the state IS the row. The card polls the runs endpoint instead. Copy avoids the vocabulary this work has been tripping over: "folder", "put back", "in the wrong folder" rather than artist_id, revert and canonical. The one thing the operator most needs to know — nothing here changes who an image belongs to — is what the blurb says first. Three things I had assumed and checked instead: MaintenanceTile lives in common/ not settings/; there is no generic ConfirmDialog (BackupCard uses a purpose-built modal), so this uses a plain v-dialog; and `loadArtistNames` did not exist — it does now, mapping id to name so a run row reads "Conto" rather than "#47". The name deliberately is not denormalised into the run: it belongs to the artist and would go stale on a rename. Also extracted `stubFetch` to frontend/test/stubFetch.js. The shape ledger flagged it as a byte-identical duplicate across five specs and this would have been the sixth; the others keep their copies until each is next touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 14:32:51 -04:00
bvandeusenandClaude Opus 5	a4bdbcaca4	test: import the placement task module so its names reach celery.tasks (4246) ... Three registration assertions failed: a task name only enters `celery.tasks` when its module is imported, and nothing in the test process imported `library_placement`. The API routes import it lazily inside the handlers, and the registration test runs before any handler test triggers that. `include=[...]` is what gets the module imported in a real WORKER, so production registration was never in question — the test was asserting something only observable after an import it never performed. test_tasks_admin already carries the convention verbatim (`import backend.app.tasks.admin # noqa: F401 — register tasks`); I wrote the assertion from what I meant instead of copying the idiom next to it. Same mistake shape as the commit-vs-flush bounce one commit ago: the pattern was already in the suite both times. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-21 14:21:20 -04:00