3 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	45bb7044f7	feat: the System tab reads a stored sample instead of inspecting per load (4295) ... Operator: "there is a repull every time this page loads is there a reason this info isn't being tracked in the background and stored in some way?" There was a reason and it had expired, and underneath it there was plain waste. The expired one: /api/system/workers was deliberately uncached because an operator dragging the stepper must not be shown a pre-change value. That stopped being true at 1353d34, when the UI began patching its row from the write's reply instead of refetching. The waste: size_worker_lanes already inspected the broker on a timer to decide pool sizes — computing the pool, active, reserved and queue depth the page shows, using them, and discarding them. The browser then asked the broker for the same numbers four times a minute, per open tab. So one inspect now feeds three things: the sizing decision, a stored sample (worker_lane_sample, alembic 0107), and the celery roster. No request path touches the broker at all — the roster refresh comes off /api/system/health too, where it had been rate-limited to 20s and so made worker liveness a function of whether anyone had a browser open. Consequences, stated rather than hidden: - The live figures are up to one sweep old. measured_at travels with each lane and the page says how old, because a stale number presented as current is how someone watches a queue "not move" that is moving. - The sweep is the roster's only writer now, so its period and the staleness thresholds are in a relationship. 60s against a 90s stale threshold left one missed tick between normal and all-yellow — the shape of lesson #4355 — so the period is 30s, named once in worker_lanes, and system_health asserts its headroom at import with a test stating the same thing in prose. - An idle lane therefore also gives a worker back twice as fast. That is the direction asked for: "idle instances quiet down when not running". Also bounds the inspect in push_lane_cap, which was an await with no deadline (rule 156) — harmless while it ran on a request, less so now that it runs in a background task where a hang would be silent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 18:52:08 -04:00
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	abe16aa382	feat: the System tab is one bounded table, and the dial is the switch (4295) ... Operator, 2026-09-23, on the screenshot: *"I feel that we can probably combine the two sections into a single table and to format it in such a way that it appears more bounded and less free-form or open. also there's nothing to describe what 'auto' means or why their needs to be or should be on/off toggles. almost all of it always needs to run there's only one optional piece and it is killed by moving the 'cap' to zero."* Three separate things, all correct. ## The four lanes were listed twice The roster (milestone 365) said "ML tagging is running", and four hundred pixels below it the lanes pane said "ML tagging · 1/1 busy". Two answers to one question from two endpoints, free to disagree on screen. I moved the second pane onto this tab yesterday and did not notice it duplicated the first. Now one row per part, with controls on the rows that have a lane and none on the rows that do not. The join is on the QUEUE SET, because that is what `service_roster` keys a celery part on — as a set, not as a string, so neither side has to agree about order. It lives in `utils/systemParts.js` rather than inline, and has a spec, because its failure is SILENT and is the exact thing it exists to prevent: a lane that stops matching its part does not throw, it grows a second row for the same worker. The duplication, returning through the code that removed it. ## Bounded, not free-form A real table — header, column rules, one bordered card — instead of dotted rows floating on the page background with nothing saying where the list began or what a column meant. ## The dial is the switch There was an `On` switch per lane beside the slots dial. Of four lanes, three must run for the application to work at all, so that switch offered a choice that was never real — and for the one lane that IS optional, "off" and "zero slots" were two ways of saying the same thing that could disagree with each other. So `enabled` is now DERIVED from the number: `set_lane` sets it from `slots > 0` when the caller did not say. It stays on the API and in the model — it is still the mechanism, and a drain-before-restart may still want a lane holding its process with consumers cancelled without destroying the operator's slot count to say so. Two things fell out that a test now pins: - The consumer command is sent on the CHANGE, not on the field being present. Otherwise every slots write re-sends a command that changes nothing — lesson #4183's churn, arriving through the new derivation. - The model fetch fires on the off→on TRANSITION. It used to test `enabled is True`, the field having been sent. The UI no longer sends it, so the download that makes the ML lane usable would simply never have fired and the lane would have come on to consume a queue it had no model for. ## And Auto now says what it is A legend under the table, in the operator's terms: what a slot is, that zero turns a lane off, that three of the four are not optional, what `of N` means, and that Auto lets a lane add slots by itself when its queue is backed up AND every slot is busy — with why it is off by default, since it is the only thing on the page that acts without being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:26:27 -04:00