1300 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	941f1c6e07	fix: run 7464's two failures — an unused loop target and a missed call site (4392) ... Both mine, both from the same change. `identity` is unpacked in the write loop and never read there — the decision it feeds happens above it, when `conclusive` is built. flake8-bugbear is on repo-wide and B007 is exactly this. Renamed `_identity`, and `linked += status == "linked"` spelled out as the `if` it actually is. The second is worse, because it was a real assertion silently pointed at the wrong shape. `match_post` now returns `(proposed, linked)`, and when I rewrote the call sites I matched on `) == 0` — so the one comparison in the file that reads `) == 1` kept comparing a tuple to an integer. Found by walking the module's AST for every comparison against `match_post` and every use of a name assigned from it, rather than grepping again with the pattern that had already missed it once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:20:12 -04:00
bvandeusenandClaude Opus 5	b3e4491b0a	feat: the announcement card offers the auto-link switch and says why (4392) ... The card's copy had gone stale against its own backend in three places, which matters more than usual here because every line of it is a claim about what FC will do with the operator's library: * "nothing is linked until you accept one" is no longer true for a pair the creator's own file name settles. The switch that governs that is now on the card rather than only in the database. * "always needs two reasons — close in time and the post mentioning Discord" left out the shared-marker signal, and named 0.55 as the number below which no single signal can carry a pair. That number is 0.45 now. * "pairs only appear when a post lands near a drop and says it is about Discord" described the empty state as if the identity route did not exist. A queued row now shows the working name behind it when there is one, and says in the same breath why that name was not enough on its own. A review queue that cannot explain itself is one the operator learns to click through without reading, and this is the only signal that claims the two posts are the SAME piece rather than that they happened near each other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:16:07 -04:00
bvandeusenandClaude Opus 5	31de54e326	feat: FC links a conclusive pair itself instead of asking (4392) ... Operator, 2026-09-24: *"I don't want this to be manual that defeats the convenience that I'm going for."* Confirm-only was right while every signal was circumstantial. Time proximity and a body that mentions Discord can never be more than suggestive, so asking was the honest response to what FC actually knew. A shared working name is different in kind: when the creator's own name for a piece appears in exactly these two posts and nowhere else in their library, there is nothing left for the operator to adjudicate, and asking is a chore FC invented for them. AUTO_LINK_FLOOR is 1.0 and sits deliberately above IDENTITY_FLOOR's 0.75. The gap between them IS the review queue — real evidence, not certain enough for FC to act on alone. Measured on artist 8, of 15 name-sharing pairs: 11 are conclusive, 2 more propose, 2 fall short of both. Three refusals, because an auto-link is FC asserting something the operator never saw: * Exactly one candidate may be conclusive. Two is not a tie to be broken by score — one post can tease two pieces dropped separately, and then each drop carries a different name from the same teaser, each individually conclusive. Two conclusive answers to "which drop is this" means the question was wrong, so both queue and FC says nothing. * Neither end may already be claimed by an accepted link. That link is the operator's decision and reassigning its other end would overrule them silently. * The whole thing is one setting, defaulting on, reversible in the UI — an accepted link is a row they can dismiss. `match_post` now returns (proposed, linked) so a sweep can report what it did on its own rather than only what it queued. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:14:04 -04:00
bvandeusenandClaude Opus 5	81f23991e9	fix: a sweep follows the drops FC has just authored, not only recent posts (4392) ... #4392's third cause, and the only one of the three about whether a pair is SCORED AT ALL rather than how well. A drop's post_date is backdated to its first message, but FC cannot author the drop until that message has an embedding and the hourly grouper has run. So a drop created this minute lands wherever its messages were — days or weeks back in the feed. The sweep only looked at announcements published within twice the window of NOW, so by the time the drop existed its neighbours were already outside the horizon, and nothing brought the sweep back to them. Measured on the live instance: a pair scoring 0.800 with `associations: []` and an empty queue. The two scoring fixes in the previous commit would not have helped it, because nothing scored it. So the sweep now also gathers announcements sitting beside any drop whose `last_grew_at`/`downloaded_at` is inside the horizon — the drop's own clock rather than its backdated position. Capped at MAX_RECENT_DROPS so a backfill authoring thousands at once does not quietly become the full-library rescan the manual button exists for, and the id set is sorted before the loop because a sweep visiting posts in a different order each run is one whose failures cannot be reproduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:09:09 -04:00
bvandeusenandClaude Opus 5	fd214f3a08	feat: the matcher gets an identity route, so a name can link what time cannot (4392) ... #4392 measured three compounding reasons the announcement matcher could not fire for this creator. This is the first two. **Vocabulary.** `discord` is not how these creators write once the audience already knows where the server is. Across 20,558 Patreon bodies `discord` appears in 486 and `the server` in 37 — but the distribution is the point, not the totals: for artist 8, 21 of 42 posts say `discord`, 7 say `the server`, and it is the RECENT ones that say the latter. A vocabulary list written from old posts silently stops matching the posts that still need it. **Two routes.** Circumstance and identity answer different questions. Proximity and a declaration say two things happened near each other; a working name the creator uses on these two posts and nowhere else says they are the same piece. Averaging those makes the threshold uninterpretable and costs both ways — adding identity as a fourth weight dilutes the others enough that measured teaser/drop pairs an hour apart stop proposing, while capping identity at a weight means the strongest evidence available can never carry a pair alone. So identity may OVERRIDE, never dilute; below IDENTITY_FLOOR it is recorded for the operator and moves nothing, which is the conservative direction, since a wrong link asserts two different pieces are one. The circumstantial bundle is now proximity/declared/marker at .45/.35/.20. Rebalanced rather than extended: the existing guard still holds arithmetically (no single weight reaches the threshold) and the existing pairs still score where they did — a teaser and its drop an hour apart with an invite link go from 0.815 to 0.781, both well over. Measured against the live instance, artist 8: 17 candidate pairs, 14 carried by a shared name, no false positives, and no drop claimed by two posts. The operator's own example — `Anya -- 🍈🍈` and the drop 75 minutes before it — links on the token `0-k`, which is what its two files are called on either platform. Two pairs 21h apart that proximity scores 0.10 are reached, and two more (`680lc`, `cnni18x`, four posts each) are the stated cost of the floor. Both rare-token signals are artist-scoped and loaded once per artist per sweep, not per candidate pair. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:07:25 -04:00
bvandeusenandClaude Opus 5	fb7563e263	fix: ruff UP031 — percent formatting in the new naming test (4392) ... Run 7456's only failure. `"comic2_%02d.jpg" % i` is an f-string here; UP is enabled repo-wide and I wrote the one construct it forbids. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:07:25 -04:00
bvandeusenandClaude Opus 5	c67a8c313e	fix: count the POSTS a working name spans, not the files (4392) ... Counting files punished a piece for the one thing a working name is guaranteed to do — have several exports. Measured across the operator's four dual-platform artists: knuxy carries `p217` on four files spanning exactly two posts, the Patreon post and the Discord drop, and ~200 comic-page tokens have that shape. Every one scored half strength. Counting posts scores them 1.00 and still catches the habits, which span many posts rather than many files: tamadaheijun's `comic2` spans 8, conto's `seth2` 5, `maid` 4. Knuxy's ambiguous band drops from 268 tokens to 55. Three false positives found the same way — by running the module over the real library rather than reading it: * The screenshot guard matches from the start of the stem, so it never fired on the legacy `<post id>_media_<media id>_` era. `Screenshot 2025-07-27 182450ab` sailed through and contributed `2025-07-27` — the same-day date collision this module's docstring exists to refuse. * tamadaheijun names screenshots in Japanese. A guard that knows only the English word is a guard for one artist. * `timeline 3-0002` spans three unrelated posts. Keeping hyphens inside tokens for `0-k`'s sake let the number survive whole, so an identity token now has to contain a letter — which is the property behind the old bare-year and all-digits rules anyway, and leaves `0-k`, `680lc` and `p59` untouched. Also stopwords for `the`, `gif`, `main`, `patreon`, `capture`, `timeline`, each measured carrying a false match inside the admitted band. IDENTITY_FLOOR names what a shared name must reach to propose a link with no corroboration. At 0.75, 13 of artist 8's 15 name-sharing pairs clear it, including the operator's own example. The two that do not are real pairs this signal will not carry alone — the stated cost of refusing the four-post band, where conto's `illustration9` and `maid` also sit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 08:00:36 -04:00
bvandeusenandClaude Opus 5	ecbfd83029	feat: a creator's own working name links the teaser to the drop (4392) ... The Patreon teaser is a censored crop of the Discord release, which is the pair whole-image comparison handles worst — a censor bar is exactly the local edit that moves a perceptual hash and blurs an embedding. But the creator exports both from one file and the internal working name survives into both platforms untouched, so the filename answers what the pixels will not. Measured on artist 8 (520 images, 409 tokens): 17 candidate pairs, 14 of them carried by a shared name, no false positives and nothing ambiguous. Ten are byte-identical names either side — ConnFront/ConnFront, LoisLaneTB2/ LoisLaneTB2 — and the signal is orthogonal to timing, reaching a pair 23.8h apart that proximity scores 0.005. Both signals gate on rarity within ONE ARTIST's library, through one shared _rarity so they cannot drift apart again. They already had: the filename side grew a frequency cap and the text side never did, so a habitual emoji scored the same 1.00 as a marker used twice. Measured, on real proposals: the deciding evidence for one pair was 💦 (13 of 300 posts) and for another the word "like" (43 posts). Both now score zero. The cap discards exactly what it should — anya 8, riju 9, undyne 8, bea 16 are character names, and ungated every Anya post would match every Anya drop. A screenshot contributes nothing rather than the date match it could be squeezed for: that date collides across platforms by construction, since the teaser and the release go out the same day. Not yet wired into post_association_service — that is the next commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 07:55:18 -04:00
bvandeusenandClaude Opus 5	e1cd629196	feat: the agent derives its version instead of being asked to remember it (4386) ... Operator: the agent's build string could not identify the agent. VERSION was a literal in app.py an author was meant to bump, and nobody did — the September image printed the same "2026-07-17.1" as the July one, so the one surface meant to answer "did my pull work?" answered the same either way. Nothing new was needed. scripts/artifacts.sh has derived a version per artifact since milestone 313, and build-agent has been computing the agent's on every run and printing it to the log. The image just never carried it. Three values, never folded together (rule 149): FC_VERSION YYYY.MM.DD.HHMM from the COMMIT its shipped files last changed in — identical on dev and main for the same source, which is what makes "am I running production's code?" answerable. FC_CHANNEL a sibling field, never a suffix inside the name. FC_REVISION the 12-char sha; the same string as the fc.revision LABEL, so the image and the registry cannot disagree about which commit this is. The page SHOWS the version and COMPARES the revision. Those were one value before, which is how a version acquires a second job and then cannot be changed without breaking the reload banner. An unstamped local build reads `unknown` and compares `local` — absent rather than empty, one spelling of "cannot say". scripts/artifacts.sh joins the AGENT path set in the same commit, and it had to: a version has no backstop. A revision that is computed differently stops matching the published label and forces a rebuild, so it self-corrects; a version is compared against nothing, so a change to cmd_version alone would leave the agent publishing the old format with nothing to contradict it. That is #3202's finding, and the agent was rightly exempt only while it had no version of its own. tests/test_artifact_paths.py pins it. Also corrects two build.yml comments claiming agent/ had not changed since 2026-07-17. Both were already false — it changed 2026-09-23 — and one of them is the stated rationale for the force_build escape hatch. Rewritten without dates: how long an artifact has been quiet is a `git log` question, and its answer in a comment is wrong the next time anyone commits (lesson #4383). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 07:07:31 -04:00
bvandeusenandClaude Opus 5	970d17f98a	fix: a second _seed_seen shadowed the one the recapture tests call (4386) ... Run 7422 integration: two #830 tests died with `_seed_seen() got an unexpected keyword argument 'post_id'`. Not those tests' fault — I defined a second `_seed_seen` at the BOTTOM of the module, taking a list of media instead of one key, and Python's last definition wins for every call site in the file including the ones 500 lines above it. Renamed to `_seed_all_seen` and made it loop over the existing one-key helper, so there is one definition of what seeding the ledger means. The nearby failure mode is worth naming: a helper defined below the tests that use it is invisible at the point of use, and shadowing produces a TypeError in code nobody touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 06:39:49 -04:00
bvandeusenandClaude Opus 5	2b093958a4	feat: a tick keeps looking back 30 days, so an EDITED post is reached (4386) ... Operator, 2026-09-23, on a Floppystack post: "this post has been updated as he implements hot fixes — any chance we have a way to scan for or see updated posts so we can update ours to match and pull the new attachments and pictures etc." The download half already worked: extract_media reads the media list off the LIVE feed response every walk, so a newly attached hotfix build is a ledger key we have never seen. Only REACHING the post was missing — a tick stopped after 20 contiguous already-have-it items, and a post edited three days after publication sits well below twenty. Not a bug in the early-out; a count cannot express "recent". The early-out now needs BOTH conditions: the run of seen items AND a post published before the horizon. Strictly a widening — window 0 is exactly the old behaviour, and no window can make a tick stop EARLIER than it used to, so a source paused for months still walks its whole unseen backlog. The horizon is a floor on how far to look, never a ceiling. Inside the window the post-record gate is bypassed too (write_post_record revisit=True): the body is re-read from the feed response already in hand, so a revisit costs zero requests, and a body that comes back empty writes NOTHING rather than blanking one a detail-fetch had filled. Revisits are kept out of the #862 body-drift canary's sample for the same reason — an empty revisit is healthy, and counting it would walk the alarm toward firing on good ticks. The run summary names what changed ("3 post(s) updated (5 new file(s))") with a line per post; the ask was to SEE updated posts, not only to end up with their bytes. download_revisit_days is a settings row, not a constant (rule 25) — how long a creator keeps editing is a property of the creator. Default 30, 0 turns it off. Migration 0108. Also corrects two stale docstrings: both clients described post_meta as feeding an Ingester.preview that no longer calls it. It had no consumer at all until this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 19:21:06 -04:00
bvandeusenandClaude Opus 5	ffbe21098c	fix: l is an ambiguous name and ruff says so (4295) ... E741, run 7410. Two comprehensions in the new endpoint tests. Every other lane was green — backend unit, frontend and integration — and nothing published, which is the gate doing its job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 18:55:36 -04:00
bvandeusenandClaude Opus 5	45bb7044f7	feat: the System tab reads a stored sample instead of inspecting per load (4295) ... Operator: "there is a repull every time this page loads is there a reason this info isn't being tracked in the background and stored in some way?" There was a reason and it had expired, and underneath it there was plain waste. The expired one: /api/system/workers was deliberately uncached because an operator dragging the stepper must not be shown a pre-change value. That stopped being true at 1353d34, when the UI began patching its row from the write's reply instead of refetching. The waste: size_worker_lanes already inspected the broker on a timer to decide pool sizes — computing the pool, active, reserved and queue depth the page shows, using them, and discarding them. The browser then asked the broker for the same numbers four times a minute, per open tab. So one inspect now feeds three things: the sizing decision, a stored sample (worker_lane_sample, alembic 0107), and the celery roster. No request path touches the broker at all — the roster refresh comes off /api/system/health too, where it had been rate-limited to 20s and so made worker liveness a function of whether anyone had a browser open. Consequences, stated rather than hidden: - The live figures are up to one sweep old. measured_at travels with each lane and the page says how old, because a stale number presented as current is how someone watches a queue "not move" that is moving. - The sweep is the roster's only writer now, so its period and the staleness thresholds are in a relationship. 60s against a 90s stale threshold left one missed tick between normal and all-yellow — the shape of lesson #4355 — so the period is 30s, named once in worker_lanes, and system_health asserts its headroom at import with a test stating the same thing in prose. - An idle lane therefore also gives a worker back twice as fast. That is the direction asked for: "idle instances quiet down when not running". Also bounds the inspect in push_lane_cap, which was an await with no deadline (rule 156) — harmless while it ran on a request, less so now that it runs in a background task where a hang would be silent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 18:52:08 -04:00
bvandeusenandClaude Opus 5	7f1693a40d	fix: the ML dial offered slots the machine had no cores to feed (4295) ... Operator's 2026-09-23 log: embed_image taking 107-246s each, ~49 slots in flight by Little's law, and the daily CCIP sweep dying on its 1800s soft limit in a numpy matmul. The billiard/pool.py frame in that traceback is the soft-timeout signal handler, not a pool fault. Two causes, both mine. 1. `derived_ceiling` computed the ML lane from MEMORY ALONE. Meanwhile `embedder.py` carried `_INTRA_OP_THREADS = 4` beside a comment reading "keep N_replicas x this within the cores allotted to ML" — a constraint stated where nothing could act on it. A large-memory host offered ~49 slots, the operator took what the dial offered, and the lane asked the box for ~200 torch threads. The number moves onto the lane as `threads_per_slot`, the embedder reads it rather than restating it, and the ceiling is now the smaller of the two bounds. They fail differently on purpose: too little memory is honestly zero, because the first task would OOM the container; too few cores is merely slow, so it floors at one rather than making the lane unreachable on a small box. 2. `scheduled_ccip_auto_apply` scored one image per matmul, over every image in the library, on every daily run — ~119k products each too small to pay for its own BLAS setup. `char_maxima` does the same arithmetic in blocks bounded by elements, so its memory stays flat as either axis grows. Batching changes no arithmetic: a character's score for an image is a max over that image's figures AND that character's prototypes, and max does not care how it is grouped. Pinned against the old loop written out longhand, and against itself with the blocking forced to split every row. The UI copy said the ML ceiling came from memory; it says cores or memory, whichever runs out first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 16:46:22 -04:00
bvandeusenandClaude Opus 5	1353d346b3	fix: the cap dial waited out a broker round trip it did not need (4295) ... Operator: "when the number is changed the change should be queued so that it isn't blocking of the webui or the system itself. we shouldn't have to wait for the validation live." Two waits, and 5b6f2ba removed neither — it stopped a Postgres connection being HELD across them, which is what had been stalling the whole site, and left the press itself as slow as it was. 1. The store refetched after every write. GET /api/system/workers runs a celery inspect on an eleven-second budget, so the stepper stayed disabled through a round trip the press did not need. It now patches the row from the reply — cap, ceiling, enabled, the three fields that reply actually decides — and lets the 15s poll bring the live columns, which are measurements it must not invent. 2. The endpoint pushed to the broker before answering. Turning a lane off is four cancel_consumer messages; lowering a cap reads the live pool first. Now it stores the cap, answers `queued`, and hands the push to a Quart background task. Raising a cap was already free and stays free. Nothing is lost by not waiting: the stored cap is what the system obeys and the sizing pass re-reads it every minute. That sweep was already the backstop for a push that failed, which under `no_live_workers` is every push in the suite. Also closes a hole the move exposed: the model fetch was gated on the consumer change having landed, so raising ML off zero while the lane was restarting stored the cap, let the sizing pass start the consumers a minute later, and left the lane running with no model — nothing else ever asks for one. It now fires on the transition and waits in the ml queue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 15:41:32 -04:00
bvandeusenandClaude Opus 5	48108a3569	fix: a lane that is OFF was not attributable to itself (4295) ... Operator, 2026-09-23: *"clean up the stale service_seen rows"*. **They were not stale.** They were phantoms, written on purpose, and they will come back on every install that turns a lane off — so the rows are the smaller half of this. A celery worker was attributed to its lane by the queues it was CONSUMING. A lane at cap 0 has its consumers cancelled, so it answers `active_queues()` with an empty list, matches no lane, and is dropped. Three consequences, all on the operator's screen at once: 1. The lanes table reported the lane **not answering** — the signal for a crashed worker, not for one the operator turned off. 2. The roster grew a phantom row named **`Worker ()`** — the empty queue set rendered as a display name — shown "running" beside the real lane's row going stale, because nothing updated it any more. 3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every lane present. ML ships at cap 0, so a fresh install was permanently unhealthy and Swarm restarts an unhealthy task forever. That third one is the severe one, and its docstring asserted the opposite of what the code did — *"a disabled lane still runs its process with its consumers cancelled, so it answers inspect and is healthy"*. It answers. It was not attributed. A comment can be right about the intent and wrong about the program, and this one had been wrong since the consolidated container shipped. `worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels with the process rather than with what it happens to be doing. `gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the information was there and nothing read it. Falls back to the queue set for a deployment that names no node, and `docker-compose.yml` now sets one per service so the multi-service stack gets it too. The roster keys on the LANE's queue set when the node resolves, which is the same string the row already had while it was consuming — so an existing row keeps updating rather than a second one appearing. Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately NOT a retention sweep: the roster never forgets on purpose, so a quiet row is what it is FOR, and only a row that cannot correspond to anything real is safe to remove. An `agent:agent` row, if one exists, is left alone — nothing here can tell an abandoned agent id from a second agent that is genuinely down, and hiding a dead GPU agent is the one thing the roster must not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 15:21:21 -04:00
bvandeusenandClaude Opus 5	a4c66601db	fix: the heartbeat guard grepped its own explanation (4295) ... Run 7375: the test that asserts the agent's heartbeat is not gated on holding leases failed — on the docstring of the fix, which quotes the construct the fix removed, because that is what a docstring explaining a fix does. assert "if ids:" not in loop A source-TEXT assertion cannot tell code from prose about code. Parsed now: the function's AST body, unparsed with the docstring node dropped, so the guard reads only what executes. Worth stating as the general shape, since this repo writes long explanatory comments on purpose: any check that greps source for the absence of a pattern is in tension with documenting why that pattern is gone. Either it excludes the prose, or the next person to explain the fix breaks the guard that protects it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:59:06 -04:00
bvandeusenandClaude Opus 5	693759f2bb	fix: an idle GPU agent could not check in, so the roster called it stopped ... Operator, 2026-09-23: *"I'm running the gpu agent on my device and it currently reads as 'offline' but it's running and has checked in recently."* It had checked in — twelve minutes ago. Two cadences that never agreed: idle lease poll ceiling 900s agent/fc_agent/worker.py (sleep mode) heartbeat while idle never gated on holding leases roster "stopped" after 300s api/system_health.py The roster records an agent check-in on `lease` and `heartbeat`. The heartbeat loop was gated on `if ids:`, so an agent holding no leases sent nothing at all — leaving the lease poll as the only check-in, and sleep mode backs that off exponentially to a 900s ceiling. 900 against 300: an IDLE agent was structurally guaranteed to read as stopped. Nothing was broken; nothing was misconfigured; the two halves simply disagreed. Not a recent regression. Sleep mode landed 2026-07-02; the roster adopted the lease as its check-in on 2026-09-02 — *"A lease IS the check-in … Recorded on the call that was already happening"* — without noticing that the call it was piggybacking on had been deliberately slowed ten weeks earlier. The heartbeat now sends whether or not it holds leases. An empty one extends nothing (`id.in_([])` matches no rows) and costs one small POST every 45s — against the 6/min lease poll sleep mode exists to avoid, that is not a cadence worth protecting, and it is what makes "is the agent alive" answerable at all. Still gated on `self._running`: a worker that has been stopped is not checking in for work, and reporting it as present would be a different lie. Two things I could NOT determine from the code, both needing the live table: whether a stale `agent:agent` row exists from an older build that omitted `agent_id` (the server defaults it), and whether changing `AGENT_ID` has ever stranded an abandoned row — nothing prunes `service_seen`, so either would sit there reading "stopped" forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:52:36 -04:00
bvandeusenandClaude Opus 5	5b6f2ba526	fix: a Postgres connection was held across every celery round trip (4295) ... Operator, 2026-09-23: *"something about changing the cap number is blocking to the website... it shouldn't be"*. Nothing here was slow in itself. A database connection was held across work that is slow, and that is why it surfaced as the whole site stalling rather than as one slow page. `lane_view` took the session and kept it open through a celery inspect whose budget is 11s. The System tab polls that endpoint every 15s — and with a lane not answering, every inspect runs to nearly its full budget, so each poll pinned a connection for most of the interval. SQLAlchemy's default pool is 5 plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing, and a cap change adding two more inspects exhausts it, and every OTHER request then waits for a connection. Split so the database work finishes before the broker work starts: - `lane_settings(session)` reads the caps and the oldest running task, then the session closes. `lane_view(settings)` does the inspect with none held. - `store_lane_cap(session, …)` validates and commits, then the session closes. `push_lane_cap(lane, …)` does the live push with none held. And a second finding while measuring it: **raising a cap now costs no broker round trip at all.** The first cut only knew on/off, so it inspected on every raise to find out whether the pool needed lowering — the control meant to be instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap so the push knows the direction; only a lowering needs to say anything. The guard is structural, not timed: `lane_view` and `push_lane_cap` must not ACCEPT a session. A timing test would be flaky, and a call-order test would pass against a version that took the session and merely used it early. `/api/system/health` has the same shape and is NOT fixed here — it is rate-limited by `refresh_if_stale` so it does not inspect on every request. Worth doing, separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 14:52:36 -04:00
bvandeusenandClaude Opus 5	c09ebd6639	fix: the race fix's temp name assumed one bootstrap per process (4295) ... Run 7370: six of the eight threads in the new test died with FileNotFoundError(2, 'No such file or directory') The first cut named the temp file `.credential_key.b64.<pid>.tmp`, which assumes one bootstrap per PROCESS. Eight threads share one pid, so all eight raced the same filename and six found it already unlinked by another. The assumption held for hypercorn's workers, which are processes, and would have held in production. It was still an assumption the code did not need to make, and the test found it in one run — which is the test doing its job rather than the test being wrong, so `tempfile.mkstemp` replaces the guess. Verified against the SHIPPED file, loaded by path so the package `__init__` (which wants quart) stays out of it: 25 consecutive attempts, eight threads through a barrier each time, one key, no leftover temp files. My first check of this reimplemented the logic in a scratch script, which measures a copy — rule 10 names that exact failure, and it is not evidence about the code that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 13:11:29 -04:00
bvandeusenandClaude Opus 5	895589a578	fix: first boot raced itself for the credential key (4295) ... Run 7368's smoke, on an image whose five verification lanes were all green: WARNING Generating NEW Fernet credential key at /images/secrets/credential_key.b64 ... ValueError: Fernet key must be 32 url-safe base64-encoded bytes. Nothing to do with this batch's changes — it is a first-boot race that has been there since the key file existed, and it is a RACE rather than a certainty: the same code booted cleanly on the three runs before it. hypercorn starts several worker processes and each one builds the app, so on a first boot they all reach the bootstrap together. `write_bytes` creates the file at size zero and fills it a moment later, which gives the second process an `exists()` of True and a `read_bytes()` of `b""`. A first boot that fails one time in five is worse than one that fails every time, because it looks like the deployment rather than the code — and this is the very first thing a new install does. The key is now written to a temp file and `os.link`ed into place. `os.link` is the atomic part: it either creates the name or raises FileExistsError, and it cannot expose a half-written file. Deliberately NOT `os.replace`, which would succeed — two processes that both generated a key would each think they had won, and the loser's key would overwrite the one the winner had already handed to Fernet. The losing branch reads the winner's key back rather than returning its own, which is what keeps every worker on ONE key. Tested for AGREEMENT, not for the absence of a crash: eight threads through a barrier, and all eight must end up holding the same key. A race that left each worker with its own would pass a "did it raise" check and produce a system where a credential written by one worker cannot be read by the next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 13:04:26 -04:00
bvandeusenandClaude Opus 5	61641fbba7	fix: a test still described the control the cap replaced (4295) ... Run 7367, integration lane: FAILED test_a_cap_is_stored_even_when_it_cannot_be_pushed assert True is False The code was right. Raising a cap is PERMISSION, not a request — it deliberately does not grow the pool, because that would put workers on a lane with nothing to do, and the sizing pass spends the permission on its next tick if there is work. So nothing is pushed and `applied` is vacuously true. The test was carried over from when the number meant "run this many", where every write pushed. It asserted the old control's behaviour against the new one — lesson #4338's shape again: an assertion encoding the thing that changed, failing on the change rather than on a defect. Split into the two cases that actually exist now: - raising a cap stores it and pushes nothing, reporting applied; - turning a lane OFF does push, because consumers follow the cap immediately in both directions — off must take effect when it is asked for — so with nothing answering it reports `applied: false` with a reason, and the value is still stored for the sizing pass to carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:57:33 -04:00
bvandeusenandClaude Opus 5	364e050632	fix: a spliced-out test left its tail behind (4295) ... Run 7366, in both the lint and unit lanes — one bug wearing two hats: NameError: name 'slots' is not defined tests/test_worker_lanes.py:278 Rewriting `test_worker_lane_check_constraints` — parametrised over (slots, cap, ok) — I cut the parametrize decorator and the assertion but left the two lines after it, which still referenced all three names. `ast.parse` is happy with that; only running it is not. Both failures were the same fragment, which is the tell: the lint lane's F821 and the unit lane's NameError are the same fact reported twice. Checked the rest of today's splices for the same seam rather than fixing only the one CI named — a module-level undefined-name sweep over all four test files I rewrote today comes back clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:53:27 -04:00
bvandeusenandClaude Opus 5	830394ed5e	fix: the migration's DROP CONSTRAINT names doubled their own prefix (4295) ... Run 7365, integration lane: (psycopg.errors.UndefinedObject) constraint "ck_worker_lane_ck_worker_lane_slots_within_cap" of relation "worker_lane" does not exist That is #3275 exactly, from the other direction. alembic 0088 had to RENAME four constraints CREATED with a doubled prefix; this one tried to DROP two with the same doubling. `op.drop_constraint` runs its name through Base.metadata's naming convention, which prepends `ck_worker_lane_` to a string that already carries it — `op.f()` is what marks a name as final, and 0103 used it on the way in. The model test also went red, correctly: `test_worker_lane_check_constraints` was parametrised over (slots, cap) pairs and asserted all three constraints, and two of them went with the `slots` column. It is one unparametrised test now, asserting the whole remaining set rather than a membership — a constraint left behind naming a dropped column is not a harmless leftover, it is a table the migration cannot have produced. Worth recording: **the gate worked.** Run 7365 skipped `sign-extension`, `build-web`, `smoke-web`, `promote` and `build-agent`, and `:dev` still names the previous digest. That is the red-direction verification #4339 owed, and it arrived by accident rather than by a forced failure — which is the better evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:51:27 -04:00
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	abe16aa382	feat: the System tab is one bounded table, and the dial is the switch (4295) ... Operator, 2026-09-23, on the screenshot: *"I feel that we can probably combine the two sections into a single table and to format it in such a way that it appears more bounded and less free-form or open. also there's nothing to describe what 'auto' means or why their needs to be or should be on/off toggles. almost all of it always needs to run there's only one optional piece and it is killed by moving the 'cap' to zero."* Three separate things, all correct. ## The four lanes were listed twice The roster (milestone 365) said "ML tagging is running", and four hundred pixels below it the lanes pane said "ML tagging · 1/1 busy". Two answers to one question from two endpoints, free to disagree on screen. I moved the second pane onto this tab yesterday and did not notice it duplicated the first. Now one row per part, with controls on the rows that have a lane and none on the rows that do not. The join is on the QUEUE SET, because that is what `service_roster` keys a celery part on — as a set, not as a string, so neither side has to agree about order. It lives in `utils/systemParts.js` rather than inline, and has a spec, because its failure is SILENT and is the exact thing it exists to prevent: a lane that stops matching its part does not throw, it grows a second row for the same worker. The duplication, returning through the code that removed it. ## Bounded, not free-form A real table — header, column rules, one bordered card — instead of dotted rows floating on the page background with nothing saying where the list began or what a column meant. ## The dial is the switch There was an `On` switch per lane beside the slots dial. Of four lanes, three must run for the application to work at all, so that switch offered a choice that was never real — and for the one lane that IS optional, "off" and "zero slots" were two ways of saying the same thing that could disagree with each other. So `enabled` is now DERIVED from the number: `set_lane` sets it from `slots > 0` when the caller did not say. It stays on the API and in the model — it is still the mechanism, and a drain-before-restart may still want a lane holding its process with consumers cancelled without destroying the operator's slot count to say so. Two things fell out that a test now pins: - The consumer command is sent on the CHANGE, not on the field being present. Otherwise every slots write re-sends a command that changes nothing — lesson #4183's churn, arriving through the new derivation. - The model fetch fires on the off→on TRANSITION. It used to test `enabled is True`, the field having been sent. The UI no longer sends it, so the download that makes the ML lane usable would simply never have fired and the lane would have come on to consume a queue it had no model for. ## And Auto now says what it is A legend under the table, in the operator's terms: what a slot is, that zero turns a lane off, that three of the four are not optional, what `of N` means, and that Auto lets a lane add slots by itself when its queue is backed up AND every slot is busy — with why it is off by default, since it is the only thing on the page that acts without being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:26:27 -04:00
bvandeusenandClaude Opus 5	274f7ffe21	ci: the tests gate the publish — ci.yml folds into build.yml ... Operator, 2026-09-23: *"tighten the gate so :dev can't publish on red tests"*, then *"I don't want failing builds to publish anywhere going forward."* Run 7348 is the worked example. The backend unit lane went red on `2f8f0bc` and `build-web` pushed `:dev` in the same minute, because the lanes and the build were SEPARATE WORKFLOWS on the same push trigger. Neither could see the other's verdict. `:dev` was a "it built" signal, never a "it passed" one, and nothing about that was visible from either run. Two workflows cannot express the gate. A `needs:` edge only exists inside one graph. So `ci.yml`'s five lanes move into `build.yml` and `ci.yml` is deleted; `sign-extension`, `build-web` and `build-agent` now need all five. Nothing here is a new mechanism — it is the same edge that has gated `promote` since milestone 362 step 4, and it keeps that step's hardest-won property: **not running is not the same as passing.** `needs` treats a SKIPPED dependency as unsatisfied, so a lane that silently skips itself blocks the publish exactly as a failing one does. Run 5290 is why that is worth stating. Scope, said plainly rather than implied: - Gated: every image tag (`:dev`, `:latest`, `:c-<sha>`), the weekly base refresh, and the `ext-<version>` signed-XPI release asset — `sign-extension` publishes too, so it is gated with the rest. - Not gated, deliberately: `extension.yml` publishes nothing, and `release.yml` runs on a `v*` tag, generates notes rather than an artifact, and its commit already went through main's gated build. - `pull_request` (Renovate bumps into `dev`) comes across with the lanes. Its runs are the lanes and nothing else, via an `if:` on each publishing job rather than an inference from the `needs` chain. The cost, accepted knowingly: this workflow queues per branch and never cancels, so on two pushes in quick succession the second's lint feedback waits out the first's build. A slower red beats a fast red that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:15:42 -04:00
bvandeusenandClaude Opus 5	468ff71353	fix: two lane fixtures described a container that cannot exist (4295) ... `pools={"host-a": 0}` — a celery replica running an empty pool. billiard will not run one, and `gen_supervisord` starts every lane at one process for exactly that reason, so no container has ever looked like this. It passed for as long as the code agreed with it. The floor landed, the reconcile's target for a zero-slot lane became one, and the fixture's impossible zero read as a real difference: `changed: ['ml']` on a settled lane, and run 7346 red on the commit that fixed the live bug. Both now say one, which is what ml actually runs. The two tests the fix added that asserted the same properties from the other side are gone with them — one assertion per fact, in the test that already named it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:03:13 -04:00
bvandeusenandClaude Opus 5	2f8f0bc9eb	feat: worker lanes move to the System tab, in the roster's own idiom (4295) ... Operator, 2026-09-23: *"where is the setting surface to set the number of each kind of worker? we have a system view under settings but no options to be set here"* — then *"move it to system tab and improve the view to be more inline with other UI elements."* They went looking under System and found a read-only roster. The card's own first cut argued that a control had no place on a tab about things being broken; that was wrong twice over. Half these lanes are optional and off by DESIGN, so "not running" is the normal state someone comes here to change — and a tab that only reports is a tab you leave in order to act somewhere else. So it is the roster's second half now, and built out of the same parts: the pane above says what is running, this one says how hard. Dot, name, detail, numbers on the right; the controls take the place the roster gives to latency. It was a bordered card wrapping a v-table, which is what made it read as bolted on. - The row styles move from SystemHealthTab's scoped block into app.css, so the second pane IS the first one's idiom rather than imitating it. A copy would have drifted the first time either was touched. - Lane state uses the roster's four dots. A lane that is off reads grey, never red: colouring an intended state as a fault is how an indicator stops being read. - The card owns its own polling. On Activity it rode that tab's 3s queue timer; System has none, so moving without this would have shown an empty pane to anyone who opened System first. 15s and `document.hidden`-guarded — it is a dial, not a live meter, and v-window keeps a hidden tab mounted. - The table header carried the column names, so each control now labels itself: `slots · cap N`, `auto`, `on`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:01:54 -04:00
bvandeusenandClaude Opus 5	86d6509936	fix: a lane at zero slots tried to empty a pool billiard will not empty (4295) ... Found on the operator's live deploy, not in CI: [scheduler] worker_control: ml reconciled 1 -> 0 slots [ml] pidbox command error: ValueError("Can't shrink pool. All processes busy!") ML ships at 0 stored slots and disabled, and `gen_supervisord` starts every lane at one process so `add_consumer` has something to reach. So the stored value and the running pool disagreed by one, permanently: billiard will not remove the last worker, and `set_lane_slots_sync` returns True on SENDING the control message — the refusal happens later, on the worker. The reconcile logged a successful correction and reported `changed: ['ml']` every tick, forever, on the default configuration of every install. Lesson #4183 in production: an enforcer whose target is unreachable re-does its own work on every pass and says it worked. The floor is now one PROCESS, in one place — `effective_slots()` — applied wherever a target is COMPARED as well as wherever one is sent. Comparing against the unclamped 0 sees a difference no control message can ever close, which is the same non-convergence one layer up. Zero slots still means zero WORK: the lane's consumers are cancelled, and the idle process is what the enable switch lands on. A cross-file guard ties the generator's starting concurrency to the same function, so the two ends of the floor cannot drift apart again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:58:58 -04:00
bvandeusenandClaude Opus 5	a987ca41ca	perf: the lane read is one broadcast and three targeted, not four broadcasts (4295) ... Found while fixing the roster's budget (f23ab9f) and reported to the operator rather than changed mid-deploy; they asked for it. `inspect_lanes_sync` made FOUR broadcast inspect calls — active_queues, stats, active, reserved — at 2.0s each. A broadcast with no `destination` cannot know how many replies to expect, so each waits out its whole timeout rather than returning on the last one. About eight seconds, and `lane_view` sits on Settings -> Activity -> Worker lanes, so that was the load time of that card every time it was opened. The composite healthcheck paid it too, against a 15s timeout. Now the first read discovers the nodes and the other three name them, so celery stops as soon as those nodes have answered — milliseconds, for workers in this same container. The worst case is unchanged: a node that vanishes between the broadcast and the targeted reads still costs a full timeout waiting for a reply that is not coming, which is why the bound stays four. Nothing answering now costs ONE round trip instead of four. The three later reads exist only to describe what answered, so with an empty roster they described nothing at three full timeouts. That is the broker-down case — exactly when the healthcheck and the card need an answer rather than a wait. `lane_view` also gets a deadline. It awaited `to_thread` with no bound at all, which is rule 156's shape even though every inner call has its own timeout; on expiry it now reports every lane as not answering, because a page that renders "not answering" is a better answer than one that does not render. The budget is derived the same way the roster's now is — round trips times the timeout, plus slack — and a test asserts the relation rather than the number, plus one that reads the source so a fifth call cannot quietly put the deadline back under the work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:37:20 -04:00
bvandeusenandClaude Opus 5	f23ab9f50e	fix: the roster's inspect budget was exactly the work it waited for (4295) ... From the operator's first consolidated deploy, 2026-09-23. The app is serving — showcase, thumbnails, a Patreon ingest tick, all five lanes in one container — and this repeats in the log: WARNING service roster: celery inspect failed; roster not refreshed File "service_roster.py", line 138, in refresh_celery_roster grouped = await asyncio.wait_for(...) TimeoutError The inspect calls were working. The budget was wrong. `_inspect_celery_sync` makes TWO broadcasts — `active_queues()` and `active()` — and a broadcast with no `destination` cannot know how many replies to expect, so each waits out its full timeout rather than returning on the last reply. The sync call costs ~2 x INSPECT_TIMEOUT_SECONDS. The wrapper allowed `INSPECT_TIMEOUT_SECONDS * 2`. That reads like a safety factor and is precisely the worst case with nothing left over — and this runs on a web process that was serving ninety thumbnails a second at the time, so the thread handing off through `asyncio.to_thread` need not even be scheduled inside the budget. A budget equal to the work fails under any load at all. Now derived: `INSPECT_TIMEOUT_SECONDS * INSPECT_ROUND_TRIPS + slack`, with the round-trip count named beside the calls it counts. Both tests assert the RELATION rather than the numbers, and one reads the source to check the count still matches the calls actually made — a third inspect call added later is exactly how this comes back silently. Consequence while it was broken: the roster stopped advancing and the System tab's rows went stale, with a traceback per attempt. Never an outage — `refresh_celery_roster` catches and returns, `/api/system/health` kept answering 200 throughout, which the same log shows. ## Observed, not fixed here `worker_control.inspect_lanes_sync` makes FOUR of these broadcasts (active_queues, stats, active, reserved) at 2.0s each — roughly 8s — and `lane_view` awaits it with no deadline at all. That is the Settings -> Activity -> Worker lanes card, so that card likely takes ~8s to load, and the composite healthcheck carries the same cost against its 15s timeout. Reported to the operator rather than changed: they are mid-deploy, and the fix is to cut round trips rather than raise a number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:32:09 -04:00
bvandeusenandClaude Opus 5	b74a4c964b	fix: the image brings its own PID 1 instead of asking for init: true (4295) ... Operator, 2026-09-23: *"it is out of the norm to require this init call we need to fix this."* Correct, and it is the same mistake as declaring the healthcheck per service — the image needing a deployment to remember a flag before it behaves correctly. PID 1 carries a duty no other process has: every orphaned process in the container reparents to it and must be reaped or it stays a zombie holding a PID slot. This app makes orphans in normal operation — six service modules shell out (gallery_dl, thumbnailer, backup_service, external_fetch, download_service, download_backends) and celery's prefork pool forks children that spawn them. Whatever the role, something that is not an init ends up as PID 1: supervisord for `all`, hypercorn for `web`, celery for a worker. `init: true` covered that, and cost correctness the moment it was forgotten or silently dropped — an older Swarm, a plain `docker run`, a compose file someone copied. No signal either way. So tini goes in the image and is the ENTRYPOINT. `docker run <image>` is correct on its own now, `init: true` comes out of docker-compose.single.yml, and nothing downstream has to know. The smoke asserts /proc/1/comm is tini, read from /proc because the runtime stage installs no `ps`. ## A correction to what I told the operator I justified `init: true` by saying supervisord "has no idea about orphans it never started". That is very likely wrong: supervisord's reaper calls waitpid(-1) and logs "reaped unknown pid" for children it did not spawn, so it does reap orphans. I asserted the mechanism without checking it, and could not check it here — `supervisor` is not installed in this environment. It does not change this commit. tini is correct whichever way that lands, and it covers the single-role containers too, where celery or hypercorn is PID 1 and the subprocess-spawning is heaviest. But the reason I gave was not a verified one and should not have been stated as fact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:18:47 -04:00
bvandeusenandClaude Opus 5	22dcbcfb74	fix: the lane/program guard compared every section, not just programs (4295) ... `test_every_lane_gets_a_program` asserted `set(cp.sections()) - {"supervisord"} == expected`, so it failed the moment the generated config grew non-program sections. b09ee87 added three — the control socket that lets `supervisorctl` reach supervisord — and the property the test exists for had not moved at all: every lane still has a program and no program is not a lane. Now compared over `program:` sections only, both directions. A guard that fires on a correct change is one people learn to edit rather than read, which costs more than it catches. I reported b09ee87 as green off runs 7328 and 7330 and did not look at 7329, which was red on this. The image itself smoked clean — that part was true — but "green" was a claim about the commit and I checked two of its three runs. Earlier in this session I had been reading all three every time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 10:06:36 -04:00
bvandeusenandClaude Opus 5	b2da3acce9	feat: wait for Postgres and Redis before starting work (4295) ... Operator, 2026-09-23: *"it's a single container that need to connect successfully to redis and postgres before starting work shouldn't that simply be a check (with retries) at the start of the container."* Yes, and the consolidated layout makes it necessary rather than tidy. Swarm has no ordering primitive — it ignores `depends_on` outright — so every service in a stack starts at once and this container has always raced its own database on a cold deploy. The multi-service stack hid how sharp that is: a `web` task that failed `alembic upgrade head` against a still-initialising Postgres simply died, and Swarm restarted it until it worked. Nobody ever saw a problem worth naming. Consolidation removes that safety net. Each supervisord program gets `startretries=3`, so three quick failures put the program in FATAL and leave it there — supervisord keeps running, the container keeps running, and the application never starts. It would present as a permanently unhealthy container whose image was fine and whose database merely took twenty seconds to initialise, which is a miserable thing to debug on a first deploy. A TCP connect, not a query: the same probe ci.yml's integration lane and the build smoke already use. It answers the question actually being asked — is something listening — and cannot fail for a reason that retrying will never fix. A real query would be a stronger readiness signal and a worse gate, since a wrong password or a missing database is not transient, and a loop waiting for one to heal turns a five-second misconfiguration into a two-minute timeout with a misleading message. Those belong to alembic, which runs seconds later and says exactly what is wrong. Targets are derived from the same env the application reads, so the wait cannot drift from what the app will actually connect to — a gate checking a different host than the app uses is worse than no gate. Bounded at 120s (rule 156), reporting every few attempts so `docker logs` on a waiting container says what it is waiting for. Skipped for `shell`, which exists precisely for when something else is broken and you want a prompt rather than a gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:17:58 -04:00
bvandeusenandClaude Opus 5	b09ee87255	fix: supervisorctl could not reach supervisord inside the container (4295) ... The smoke's own diagnostic line printed this for a whole run and passed, because it was behind `|| true`: Error: .ini file does not include supervisorctl section supervisord was fine. `supervisorctl` simply could not talk to it — the generated config had no `[unix_http_server]`, `[supervisorctl]` or `[rpcinterface:supervisor]`. That is the first tool anyone reaches for when a lane misbehaves in the consolidated container. `docker exec <c> supervisorctl status` to see which processes are actually up; `restart ml` to bounce one without taking the whole application down with it. Consolidation took `docker ps` away as the way to see the lanes, and this is what replaces it — so shipping without it would have left an operator with one container, five processes inside it, and no way to ask about any of them. They are about to run this in production. The test asserts the three sections AGREE on one socket path rather than merely existing: a serverurl pointing where nothing listens fails in exactly the same way and reads as configured. The smoke's line loses its `|| true`. A diagnostic allowed to fail silently is one that stops being true without telling anyone — which is precisely what happened here. It still printed the evidence that something was wrong while nothing depended on it, which is the argument for printing it at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:09:11 -04:00
bvandeusenandClaude Opus 5	efde3b188f	refactor: the image carries its own healthcheck and picks it by role (4295) ... Operator, 2026-09-23: *"why isn't the healthcheck built into the image or base on what command runs if one is passed in. why is it manually declared in the stack here."* No good reason. The container is the only thing that knows what it was asked to run, and every compose file, stack file and README had to restate it: web -> urllib /api/health worker -> celery inspect ping -d celery@$HOSTNAME all -> both, for every lane Three checks written by hand, once per service, in every file anyone ever wrote — none of them wrong until a role changed, and all of them silently wrong after. The same duplication the lane table exists to remove one level down, and I built it without noticing. `entrypoint.sh` now records the role it started. The Dockerfile declares ONE `HEALTHCHECK` that reads it and asks the right question: HTTP for web, a self-addressed celery ping for a worker lane, both-for-every-lane for `all`, and nothing for shell/alembic, which are one-shot and have no liveness to probe. `docker-compose.single.yml` and the consolidated stack declare none. A service that wants something else can still declare its own; docker prefers it, so the escape hatch is the default docker behaviour rather than a flag. Two details that are load-bearing: * The role is written ONCE, by the outermost invocation. `all` starts the other roles through this same script under supervisord, and a child overwriting the container's role would turn the composite check into a web-only one — silently, and only on the consolidated path. FC_ROLE is exported so a child sees it set and skips. * The celery ping is addressed to THIS node, not a bare ping. A bare one is answered by any worker on the broker, so in a stack with replicas a dead container would report healthy for as long as a sibling lived — the check would be measuring the cluster rather than the container it is inside. `healthcheck_all.py` is deleted; its two probes moved into the dispatcher rather than being a second copy beside it. An unrecorded role PASSES. The entrypoint always writes the file, so the only way to miss it is bypassing the entrypoint — a debugging shape, where a check that cannot tell what it is looking at must not assert the thing is broken (snippet #3969). Said on stdout rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 09:07:39 -04:00
bvandeusenandClaude Opus 5	828c6a5ae3	fix: each lane needs its own celery node name, or three of four vanish (4295) ... Found by the all-role smoke on its very first execution (run 7319), which is the whole argument for having added it one commit ago. Celery's default node name is `celery@<hostname>`. In the single-container layout all four lanes share one hostname, so all four registered as the SAME node. Celery says so itself: DuplicateNodenameWarning: Received multiple replies from node name: celery@72adc5b706a7 `inspect` collapses four replies into one dict key and the last one wins, so three lanes read as absent — and WHICH three varies between calls: lanes not answering: maintenance_long, ml, worker lanes not answering: maintenance_long, scheduler, worker Fatal twice over: * The composite healthcheck can never pass. In Swarm that is a container that never goes healthy — restart loop, then an automatic rollback of a deploy whose image was fine. * `pool_grow`/`pool_shrink` take a `destination` of node names. The UI dial and the autoscaler would have resized whichever lane happened to answer rather than the one asked for — silently, and differently each time. Every celery role now starts with `-n "${CELERY_NODENAME:-celery}@%h"`, and the generated supervisord config sets that per lane. The lanes become worker@<cid>, scheduler@<cid>, maintenance_long@<cid>, ml@<cid> — distinct, so inspect keeps four entries and `destination` addresses what it names. `inspect_lanes_sync` maps hostname to lane by QUEUES, so nothing there changes; it just stops having three of its four entries overwritten. Unset, it falls back to `celery` — exactly celery's own default — so every service in the multi-service stack is byte-identical to before, including the `celery@$HOSTNAME` healthcheck in docker-compose.yml and in the operator's Swarm stack. The test asserts DISTINCTNESS across the whole lane table rather than a fixed string per lane. The property that broke is that no two collide, and stating it that way keeps holding when a lane is added. This is the bug I said a live deploy was needed to find, found in CI instead for the price of one `docker run` — and it would have met the operator as a rollback loop on their first consolidated deploy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:52:56 -04:00
bvandeusenandClaude Opus 5	43ac737516	feat: the whole application is what the image runs by default (4296) ... Operator, 2026-09-23: *"I also want to see that we remove the need for the command line of the configuration in the consolidated version."* `CMD` was `web`, so the single-container layout only worked if you knew to ask for it by name. A compose file that forgot `command: ["all"]` got a web server with nothing processing its queues — a gallery that loads, accepts an import, and never finishes one. Nothing errors; it just never progresses. Now `docker run fabledcurator` with no command starts hypercorn plus every lane under supervisord. `entrypoint.sh`'s own default moves with it, since the two are doors to the same decision and a disagreement would only show up as `--entrypoint` behaving differently from a plain run. `docker-compose.single.yml` drops its `command:` line; `["all"]` still works and still means the same thing. The multi-service stack is untouched — every service there names its role explicitly, which is what makes it the multi-service stack. ## And CI now actually boots it This is the gap I should have named when I reported milestone 422 at 7/7 and did not. Measured, not inferred: the smoke booted role `web` only (build.yml:1454), nothing in CI ran `all`, `docker-compose.single.yml` was read as TEXT by one test checking stop_grace_period and never run, and test_gen_supervisord asserts the generated config against the lane table without ever handing it to supervisord. So the shape this milestone is NAMED for had started nowhere. Steps 5-7 were marked done on evidence that did not cover it, and the operator is about to collapse their production stack onto exactly that. The smoke now boots the image with NO command — checking the Dockerfile CMD, the entrypoint default and the role together, the way an adopter gets it — and asserts `healthcheck_all`, which was itself never executed. That check passes only when hypercorn answers AND every lane in the table answers the broker; a web-only check goes green with every worker dead, which is the failure mode consolidation creates. It then prints `supervisorctl status`, so a lane that is merely restart-looping is visible rather than inferred. Cheap because ml ships at 0 slots and disabled: nothing loads a model, and the lane answers `inspect` with its consumers cancelled, which is what healthy means for a disabled lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:48:11 -04:00
bvandeusenandClaude Opus 5	2677ce020c	fix: the ml artifact was in a second list I never grepped for (4311) ... ac70f2a removed the `ml` image but CI went red on six tests: `tests/test_artifact_identity.py` parametrises over its own `ARTIFACTS = ("web", "ml", "agent", "extension")`, and every case now hits the dispatch guard that same commit added. My miss, and a specific one. I grepped for `fabledcurator-ml` and `ML_PATHS` and called the survey done — but the artifact is also named as a bare `"ml"`, which neither pattern finds. Rule 90 (grep pinned tests when changing a shared symbol) was surfaced to me while I was making the change and I ran a narrower sweep than it asks for. Lesson #4275 names the shape exactly: an absence claim is only as good as the search behind it, and a grep that matched nothing looks identical to a grep that asked the wrong question. The re-run was done by value, not by name: every occurrence of a bare `ml` in the repo, then filtering. That distinguishes the two things the token means — the celery LANE `ml` and the `backend/app/services/ml` package both stay and account for nearly every hit; only the IMAGE name went. Worth stating in the test, since the next person to grep will hit the same ambiguity. Also swept the prose the first pass left describing the old pipeline: "Four artifacts" (README, ci-requirements), "leaves web and ml alone" (×5 in build.yml, plus both docs), "CI publishes it alongside the web/ml images" (agent/README). The run 4896 build-time measurements keep their `ml` number — that was measured when ml was a real build — with a note saying the name has since gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:32:49 -04:00
bvandeusenandClaude Opus 5	ac70f2aadc	refactor: retire the fabledcurator-ml image name entirely (4311) ... Operator, 2026-09-23: *"we don't need to keep this as I'll be collapsing my stack to the same consolidated version that we're building here."* That was the only thing holding the name up. `fabledcurator-ml` has been the same bytes as `fabledcurator` since milestone 422 step 6, and 8152684 had already stopped rebuilding it — this removes the name. Gone: the whole `build-ml` job (137 more lines), its entry in promote's `needs` and in promote's tag loop, `ML_PATHS` and the `ml)` case in artifacts.sh, the `ml` artifact in test_artifact_paths, the image in release_notes.py and its test, and the README row. `docker-compose.yml`'s ml-worker service now runs `fabledcurator:latest` with `command: ["ml-worker"]` — the service still exists for anyone who wants lane separation, it just no longer needs a second image name. Prose that said "all three images" is now "both", except where it means three VALUES (revision/version/epoch) or records what happened on 2026-08-30, when there genuinely were three. Published tags are left alone. Nothing new goes to that name; what is already in the registry stays pullable, so a stack that has not moved yet keeps running rather than losing its image mid-flight. ## A latent trap this made reachable, found by running the script `artifacts.sh revision ml` did not fail. It printed the usage line to stderr and answered with **the newest commit in the whole repository** — a real-looking 12-char sha on stdout, exit 0. `newest()` inlined the path set as `git log ... -- $(cmd_paths "$1")`, and `usage` exits from the command SUBSHELL, so the substitution came back empty and `git log HEAD --` had no pathspec left to filter by. The reuse check would have compared that answer against a published label, missed, and rebuilt on every push forever with nothing going red. It could not be fixed inside `newest`, which was my first attempt: every caller wraps it in a substitution too, so its `exit` also died in a subshell — stdout went empty but the status stayed 0. The guard has to run in the main shell, so it validates the artifact name at dispatch. `newest` keeps a defensive `|| exit 2` for a future caller that reaches it another way, and the comment says which of the two is the real check. Latent while every name callers passed was valid. Removing `ml` from the set is what made a name that used to work start taking the silent path, so the test that pins it lands here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 08:30:47 -04:00
bvandeusenandClaude Opus 5	f0595e43e6	docs: record what the alias push actually costs — zero layer bytes (4311) ... The comment said the first run would answer whether the registry shares blobs across two repositories of one owner. Run 7300 answered it: every layer came back "Layer already exists" or "Mounted from bvandeusen/fabledcurator". The push moves a manifest and nothing else. 22 seconds, against the ~2 minutes the duplicate build cost (run 7282: build-ml 1m55s for output identical to build-web's). Also noted: the pushed manifest gets its own digest, because docker re-serialises it — same layers, same config, same fc.revision, different manifest bytes. Nothing compares the two, but the digests differing is the first thing someone will notice and misread as a second build still happening. Leaving a "we do not know yet" in the file once the run has said would make it the kind of stale note that misleads the next person to read it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 07:21:56 -04:00
bvandeusenandClaude Opus 5	815268418e	perf: stop building the same image twice under two names (4311) ... `fabledcurator-ml` has been the SAME IMAGE as `fabledcurator` since milestone 422 step 6 merged the ML layers into the one Dockerfile. `build-ml` was still running `file: Dockerfile, context: .` — byte for byte what `build-web` builds — and publishing the result under a second name. So CI did the whole build twice. Measured on run 7282, cold cache: build-web 1m54s, build-ml 1m55s, for identical output, plus a second push of a few hundred MB. Operator, 2026-09-22: *"please fix the CI so it doesn't do this superfluous work."* build-ml now publishes nothing of its own. It pulls the manifest build-web just pushed, re-tags it, and pushes that — so the second name goes on working for the operator's Swarm stack, which still references it, while CI stops compiling anything twice. 437 lines of duplicated build machinery go with it: its own buildx setup, its own reuse check, its own version derivation, its own tag repoint. It now `needs: [build-web]` rather than running in parallel. That is the one cost, and it is seconds: there is nothing to copy until build-web has decided what this run publishes, and the layers are already on the runner. Deliberately NOT `imagetools create`, which would have been the obvious one-liner. That wraps its source in an INDEX and `.Image.Config.Labels` does not resolve through one (#3183, run 4751) — and it would bite harder here than it did there, because `promote` asks the registry for IMAGE manifest media types ONLY, so an index at :refresh-candidate would fail its `test -n "$CT"` and break the weekly refresh for this name. Plain pull/tag/push keeps a plain manifest, which rule 40 also asks for. The step reads the tag back and fails if `fc.revision` stops resolving, so a future change that reintroduces index-wrapping says so instead of quietly costing the refresh. The tag list mirrors build-web's exactly, refresh included: a refresh writes :refresh-candidate alone and `promote` moves it to :latest after smoke-web passes. So this name keeps the gate it has today and `promote` needs no change — it already loops over all three image names. Source is build-web's `digest` when it built, else its `published_digest` on a reuse hit — the digest its channel tag already names, which is what a hit MEANS. With neither, the job fails with a reason rather than publishing something arbitrary. This does not retire the name; #4311 still does that, and is still gated on the operator's stack pointing ml-worker at `fabledcurator:latest` with `command: ["ml-worker"]`. Until then nothing of theirs changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 07:20:30 -04:00
bvandeusenandClaude Opus 5	187b6d2cdf	fix: the smoke always runs, so a change to the smoke can verify itself (4323) ... This workflow file is in no artifact's path set — correctly, since editing it changes no shipped byte. But smoke-web was gated on build-web having published something, so a commit touching ONLY the smoke moved no revision, hit reuse, emitted no digest, and skipped the smoke. The one commit whose purpose is changing this check was the one commit that could not run it. Twice already: 5ca1058 added the egress sandbox and went green three times with smoke-web SKIPPED; 7175ace fixed the bug hiding behind those greens (#4319) and needed a manual force_build to exercise. Both relied on someone remembering. It is also where the other historical skip lived — run 5290's `if:` read `env`, which a job condition cannot see, so it evaluated empty and skipped silently. Two skips, one expression. The expression goes. The job now smokes whichever manifest is current: the digest this run built, or — on a reuse hit — the one the channel tag already names, which the reuse step resolves anyway to read its fc.revision label and now exports as `published_digest`. Always a digest, never a tag (#4290). Kept separate from `digest`, which the :c-<sha> repoint reads and must go on meaning "what this run built". With neither available the job FAILS with a reason rather than passing quietly. Adding build.yml to WEB_PATHS would also make the smoke run, and would be wrong: fc.revision means "the commit this artifact's shipped files last changed in", so moving it for a CI edit makes the label lie and rebuilds three images for a change none of them ship. The problem was never the artifact's identity — it was when the guard runs. Two more defects in the same block, both from 5ca1058, both found by run 7288 — which only existed because the force_build above ran the check a second time: * `NET=smoke-noegress-$$` is not unique. The shell's pid is deterministic in this runner — every execution got 157 — so the second run died on "network with name smoke-noegress-157 already exists". A pid is unique among live processes, which is not unique over time. * The network's cleanup trap was destroyed before it could fire. `trap ... EXIT` REPLACES the previous handler, and the container's trap was installed further down, so every run leaked its network. Invisible in a passing run; it can only ever surface on the NEXT one. Now one EXIT handler does both, armed beside the network and tolerant of an empty CID so it covers failures before the container exists. Plus a sweep of any network earlier runs leaked, which fails harmlessly on one still in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 11:20:36 -04:00
bvandeusenandClaude Opus 5	7175ace67a	fix: the smoke's health probe has to run inside the sandbox it created (4319) ... The first real execution of the egress-blocked smoke (run 7282) failed with "web is running but never answered /api/health" — and the application was perfect. Its own log shows all four hypercorn workers serving three seconds after start and still up seven minutes later, with no internet: [entrypoint] Starting hypercorn on :8080 [14:08:58] [10] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [11] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [12] [INFO] Running on http://0.0.0.0:8080 [14:08:58] [13] [INFO] Running on http://0.0.0.0:8080 5ca1058 put the app container on an `--internal` network. Docker gives such a network isolation rules that DROP traffic entering it from any other interface, and this job's own container sits on the runner's default bridge — so its curl to the app was discarded before arrival. Dropped rather than refused, so every attempt burned the full --max-time and the loop read as a wedged app instead of an unroutable address. Steps 0-2 were right only by accident: each already runs its check inside a container on $NET. Step 3 was the one place that reached in from outside, and so the one place that could not work. It now probes from inside too, using the image's own python3 over `shell -c` — the same shape as the egress guard above it, and necessary because the runtime stage ships no curl. Attaching the job container to $NET would also work in one line. Rejected: it puts an internet-connected container on the network whose whole purpose is being offline, and it would rest on `hostname` equalling the container id. Same family as #3374 — a CI check that could never pass, failing in a way that accuses the thing it was meant to protect. Worth stating plainly: the egress property itself PASSED on 7282 ("smoke: no route out, as required") and the schema built from empty through 0104 with no network. Only the harness's last step was broken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 10:16:17 -04:00
bvandeusenandClaude Opus 5	a01165365b	feat: a saturated lane can grow itself, within the cap the operator set (4297) ... Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 10:05:43 -04:00
bvandeusenandClaude Opus 5	5ca1058fb5	test: the smoke runs with egress blocked, on every build, and proves the block (4296) ... Milestone 422 step 6's rule-164 half. THE PROPERTY WAS NEVER TESTED, not even weekly. smoke-web already booted the built image against real Postgres and Redis — but on the runner's default network, with the internet one hop away. It proved the image WORKS; it never proved it works OFFLINE, which is the thing rule 164 is about and the thing step 6 put at risk by moving download_models out of boot. Now on an `--internal` docker network, which is the mechanism rule 164's own verify_with names. `--network none` is explicitly the wrong check here: it would only prove the app fails without a database, which says nothing about egress. Internal blocks the default route while leaving container-to-container traffic and embedded DNS intact, so Postgres and Redis stay reachable and nothing else does. The service containers are runner-created siblings, so they are ATTACHED to the network rather than created on it, and their addresses are re-read on it — the bridge IPs discovered earlier are not routable from a container that is only on the internal network. A STEP THAT PROVES THE SANDBOX IS SEALED, before anything depends on it. It tries to reach 1.1.1.1:443 from inside the candidate and fails the job if it succeeds. Without it the rest is theatre: if `--internal` silently stopped working, or the container picked up a second network, every check below would pass with the internet available and report an offline boot that never happened. A guard that cannot fail is not a guard (rule 167). IT RUNS ON EVERY BUILD, not just the weekly refresh. The egress property is broken by a code or Dockerfile change — a push — so checking it only on the refresh would test it on the one trigger that changes no source. Addressed by the DIGEST build-web published rather than by a tag: a tag can move between the build and the smoke, and then the check reports on bytes nobody built here. A reuse hit is skipped, because those bytes were smoked when built. WHAT THIS STILL IS NOT, filed as #4310: on a push it runs AFTER build-web has written the channel tag, so it detects rather than gates. Rule 164 asks for the check BETWEEN build and push. Closing that means giving the push path the candidate-then-promote shape the refresh already has — per-channel candidate tags, promote learning its channel, and the :c-<sha> repoint moving after the gate. That is a redesign of the path that publishes production and it is not something to fold into a test change. Also filed #4311: retiring the fabledcurator-ml image NAME, gated on the operator moving their Swarm stack file. Same two-phase shape #406 used for pixiv, for the same reason — dropping it would not break their deploy, it would freeze it silently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 09:50:25 -04:00
bvandeusenandClaude Opus 5	0f98e46200	docs: the merged image's cost, measured — and it corrects my own estimate (4296) ... I wrote in ffcd130's Dockerfile comment that merging ML in means "everyone pulls it, including the many who will never turn tagging on", framed as a real cost the milestone accepted. My working estimate behind that was ~4GB. Measured from run 7273's build-web log: torch 2.12.1+cpu wheel 192.3 MB torchvision 0.27.1+cpu 1.8 MB transformers / onnxruntime / opencv / sklearn and friends 62.0, 35.3, 23.6, 16.7, 12.3, 9.2, 6.9 MB largest newly-pushed layer 222.07 MB The ML code adds a few HUNDRED MB, not gigabytes. The `--index-url` CPU resolution is what makes that true — the default PyPI torch wheel carries the CUDA runtime and is ~2GB by itself, and the log confirms 2.12.1+cpu resolved, so it is working as intended rather than as intended-but-unverified. Why this matters beyond a comment being wrong: it settles the trade this step was explicitly asked to weigh and could not, and it reverses how close the call looked. Baking the weights in adds ~3.5GB to every pull for a feature many adopters never enable; shipping the code and fetching on demand adds ~350MB. An order of magnitude, where the estimate had them within 15% of each other. Off-by-default is not a judgement call here, it is arithmetic. The gigabytes were always in the MODEL, and the model is not in the image. Two things NOT measured, still: the total image size (the push only transfers layers the registry lacks, so a push log cannot give it) and the per-slot resident RAM, which stays flagged `measured=False` in the lane table and renders as "about" in the UI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 09:02:47 -04:00
bvandeusenandClaude Opus 5	f174981b07	fix: ml and web are one image, so they are one path set (4296) ... Run 7271. `test_every_copied_path_is_in_the_artifacts_path_set` — a guard I did not know this repo had — caught the merge's real consequence within minutes, and it is precisely #3202's class: ml: Dockerfile copies 'frontend/package.json' into the ml image, but no include in scripts/artifacts.sh covers it. web: Dockerfile copies 'requirements-ml.txt' into the web image, but no include covers it. Both follow from step 6 and neither was visible by reading my own diff. The merged Dockerfile carries the frontend-builder stage, so the ml image now copies frontend files; and the web image now installs the ML requirements. Left as they were, each artifact's derived version would have stopped moving when those files changed, and a pinned build would serve stale bytes — silently, since nothing fails when a version simply does not advance. The fix is structural rather than two more entries. `fabledcurator-ml` IS web's image now: same Dockerfile, same context, same bytes, published under a second name only because the operator's Swarm stack still references it. So ML_PATHS is WEB_PATHS by assignment, and `cmd_paths` gives ml the same deriver and extension append — the XPI is in those bytes too. Two lists describing one image is the duplication this milestone has been collapsing all day. It existed for about an hour and the guard found it first, which is the argument for the guard. I also predicted this failure would be the image build. It was not; the build passed and the unit lane failed. Worth noting because the prediction was confident and wrong, and reading the log took one call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:57:09 -04:00
bvandeusenandClaude Opus 5	ffcd13096a	feat: one image for every lane, with the model fetch gated on enabling (4296) ... Milestone 422 step 6. Dockerfile.ml is gone; the main image carries torch, torchvision, transformers, onnxruntime and opencv, and serves every lane. WHY IT HAD TO MERGE: step 5 runs every lane in one process tree, so a second image would mean the `ml` lane could never be enabled from the UI — there would be no worker in that container to enable. The switch needs something to switch. THE MODEL NO LONGER DOWNLOADS AT BOOT. `entrypoint.sh`'s ml-worker role ran download_models before celery started, so every boot of that role reached HuggingFace for ~3.5GB — a startup dependency on a third party for a feature the operator may never use. Rule 164 permits a runtime fetch only for something "optional and clearly off", so the fetch is now a TASK, enqueued the moment the lane is ENABLED. Being a task is what makes it visible: it gets a TaskRun row, so the download shows in Activity with a duration and a status, and a failure is something an operator can see and retry rather than a container that quietly never became useful. Idempotent, so re-enabling a provisioned lane costs one no-op. Enqueued only when the lane actually came ON (`enabled is True`, not the resolved value) so re-saving slots does not re-fetch, and only when the consumer change landed — a task queued onto a queue nothing consumes would sit pending with no explanation. `fabledcurator-ml` KEEPS PUBLISHING, from the merged Dockerfile. The operator's Swarm stack references that name and lives outside this repo; dropping it would not break their deploy, it would freeze it silently at the last publish — the exact failure class this milestone keeps finding. Retiring the NAME is its own task, gated on that stack moving. Same two-phase shape #406 used for pixiv. THREE LIVE BREAKAGES from deleting the file, found by grepping for it rather than assuming the build was the only consumer: - `docker-compose.override.yml` built the ml service from it (contributor path would have failed at `docker compose build`). - `tests/test_artifact_paths.py` pins the ml path set. - `scripts/artifacts.sh` ML_PATHS named it. A path set naming a deleted file silently stops contributing to the derived revision — which the reuse check and the version string both read. That is #3202's recorded shape. The `--with-ml` flag is gone from the generator and the healthcheck rather than left defaulting to true. One image carries every lane now, so a flag that can only be passed one way is a branch pretending to be a choice. The advisory shipped in ecbd325 is what makes this honest to an adopter: the lane says it is optional, names the model, and gives its download and per-slot RAM before the switch is thrown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:51:44 -04:00