4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	abe16aa382	feat: the System tab is one bounded table, and the dial is the switch (4295) ... Operator, 2026-09-23, on the screenshot: *"I feel that we can probably combine the two sections into a single table and to format it in such a way that it appears more bounded and less free-form or open. also there's nothing to describe what 'auto' means or why their needs to be or should be on/off toggles. almost all of it always needs to run there's only one optional piece and it is killed by moving the 'cap' to zero."* Three separate things, all correct. ## The four lanes were listed twice The roster (milestone 365) said "ML tagging is running", and four hundred pixels below it the lanes pane said "ML tagging · 1/1 busy". Two answers to one question from two endpoints, free to disagree on screen. I moved the second pane onto this tab yesterday and did not notice it duplicated the first. Now one row per part, with controls on the rows that have a lane and none on the rows that do not. The join is on the QUEUE SET, because that is what `service_roster` keys a celery part on — as a set, not as a string, so neither side has to agree about order. It lives in `utils/systemParts.js` rather than inline, and has a spec, because its failure is SILENT and is the exact thing it exists to prevent: a lane that stops matching its part does not throw, it grows a second row for the same worker. The duplication, returning through the code that removed it. ## Bounded, not free-form A real table — header, column rules, one bordered card — instead of dotted rows floating on the page background with nothing saying where the list began or what a column meant. ## The dial is the switch There was an `On` switch per lane beside the slots dial. Of four lanes, three must run for the application to work at all, so that switch offered a choice that was never real — and for the one lane that IS optional, "off" and "zero slots" were two ways of saying the same thing that could disagree with each other. So `enabled` is now DERIVED from the number: `set_lane` sets it from `slots > 0` when the caller did not say. It stays on the API and in the model — it is still the mechanism, and a drain-before-restart may still want a lane holding its process with consumers cancelled without destroying the operator's slot count to say so. Two things fell out that a test now pins: - The consumer command is sent on the CHANGE, not on the field being present. Otherwise every slots write re-sends a command that changes nothing — lesson #4183's churn, arriving through the new derivation. - The model fetch fires on the off→on TRANSITION. It used to test `enabled is True`, the field having been sent. The UI no longer sends it, so the download that makes the ML lane usable would simply never have fired and the lane would have come on to consume a queue it had no model for. ## And Auto now says what it is A legend under the table, in the operator's terms: what a slot is, that zero turns a lane off, that three of the four are not optional, what `of N` means, and that Auto lets a lane add slots by itself when its queue is backed up AND every slot is busy — with why it is off by default, since it is the only thing on the page that acts without being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:26:27 -04:00
bvandeusenandClaude Opus 5	a01165365b	feat: a saturated lane can grow itself, within the cap the operator set (4297) ... Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 10:05:43 -04:00
bvandeusenandClaude Opus 5	da48edf7da	feat: the worker-lanes card — see each lane, change its slots (4294) ... Milestone 422 step 4. Rule 27: no UI, no ship. This is where the previous three steps become usable. REACHABLE AT: Settings -> Activity -> Worker lanes, directly under the "Queues + workers" pane. Written from opening the view, not from memory of having built it — lesson #4282, and #3463 is the same trap landing inside milestone 365, where the System page shipped with no navigation to it. Under that pane deliberately, not in the System tab. System answers "is everything running", where every control would be about something broken. This is about something working that should work harder, and it belongs beside the backlog it reacts to: you watch a queue grow and give that lane another slot without leaving the pane. Per lane: queues, pending, busy, a stepper, an enable switch. - PENDING is depth + reserved, not LLEN. Celery prefetches, so LLEN alone reads 0 while a worker holds tasks in memory — the number that would make someone think a buried lane was idle. - NOT ANSWERING, never "stopped". present=false means nothing replied; saying stopped would send the operator looking for a crash that has not happened. - THE CEILING IS ON SCREEN, with "(memory)" on the ML lane. It is the one number here the operator cannot change, so it has to justify itself; a greyed stepper with no explanation reads as a bug. - `busy` is per lane, so adjusting one does not freeze the others. TWO OUTCOMES THAT MUST NOT COLLAPSE INTO ONE MESSAGE. A stored-but-unpushed change (applied:false — the lane is restarting) is information: the value is saved and the reconcile will carry it, so the card says so and invites waiting. A refused value (400) is an error and shows the endpoint's own sentence. Collapsing them would make one of the two invite the wrong action. A BUG CAUGHT BEFORE COMMIT: the card read `e.detail?.detail`, but ApiError puts the parsed body on `.body` and `.message` on the short `error` key. It would have shown the operator the bare word "refused" with no reason — the exact failure that line exists to prevent. The test would not have caught it either: `rejects.toThrow()` passes whether the sentence is reachable or not. It now asserts `err.body.detail` specifically. Also: queueOptions in SystemActivityTab was a fourth hand-kept copy of the queue list and had drifted — `maintenance_long` was missing, so activity on that lane could not be filtered for at all despite four task routes pointing there. Added, and DELIBERATELY left as a written-out list rather than derived like the other three were: this filters task_run HISTORY, so a derived list would hide the filter for any queue that has rows but no longer has a lane — precisely when someone is looking — and would be empty whenever the endpoint is down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:23:14 -04:00