3 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	a01165365b	feat: a saturated lane can grow itself, within the cap the operator set (4297) ... Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 10:05:43 -04:00
bvandeusenandClaude Opus 5	84f13135ce	feat: worker lanes become rows — slots, a settable cap, a derived ceiling (4291) ... Milestone 422 step 1. The data model the rest of the milestone reads. No behaviour change: nothing consumes these rows yet, and every lane still boots at its CELERY_CONCURRENCY env value. Three numbers, not two, per the operator's distinction — the derived value is a cap ON the cap: slots <= slots_cap <= derived_ceiling (live) (operator) (computed) They can always lower their own cap; they cannot raise it past what the container can hold. The ceiling is never stored, so a row written on a 32GB host and later run in a 4GB container is bounded by the 4GB. `services/worker_lanes.py` is the one place that knows the lane set. `models/worker_lane.py` holds only what an operator may change. Two deviations from the step as written, both deliberate: QUEUES ARE NOT A COLUMN. The step body said the row carries its `-Q` list, but a lane's queues are decided by celery_app's task_routes, not by preference — an operator cannot move a backup off maintenance_long. Storing them would create a row that can contradict the routing table, with nothing to notice until a queue had no consumer. So queues are code, slots are data. `test_every_routed_queue_has_a_lane_that_serves_it` reads the real routing table and fails if a route is ever added without a lane. ROLE_NAMES IS NOW DERIVED, not left alone. It was a hand-kept second copy of "queue set -> display name" and had already drifted: maintenance_long is a live lane with four task routes and a dedicated worker in the operator's stack, and the roster did not know its name — so the System tab labelled it `Worker (maintenance_long)`. Adding a lane table beside it would have made three copies. The ceiling honours cgroup limits rather than the host's. `os.cpu_count()` reports the HOST's cores from inside a container, so a 4-core quota on a 32-core host would otherwise offer 32 slots — and the operator's own stack sets `cpus: '4.0'` on ml-worker, so that is real configuration, not a hypothetical. Memory reads cgroup v2 then v1, and recognises v1's PAGE_SIZE-aligned LONG_MAX sentinel by magnitude rather than treating it as petabytes. Every uncertain case fails LOW. An unreadable limit yields UNKNOWN_CEILING, never unlimited — not knowing how much memory there is must not read as plenty. A box too small to hold one model beside the web process gets an ML ceiling of 0 rather than a floor of 1: offering a slot that OOMs the container the first time it is used is exactly what this exists to prevent. ML_BYTES_PER_SLOT is 4 GiB and is UNMEASURED — flagged as such in the code, with the method for replacing it with a real figure. It decides whether a stranger's server survives enabling tagging, so it errs toward refusing a slot that would have fitted. Seeded one-of-each with ml at 0 and disabled (alembic 0103). ML off is step 6's requirement arriving early: enabling the lane is what triggers the SigLIP download, and rule 164 permits a runtime fetch only for a feature that is optional and clearly off. The seed values are literals rather than an import of LANES — a migration is a statement about one moment, and importing the live defaults would silently change what this revision does on a fresh database in 2027. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 07:48:25 -04:00