8 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5.5	cc53d8db7b	feat: a GPU agent on the CPU shows as degraded, in the System view and on its own page (4410) ... torch and onnxruntime both fall back to the CPU without raising, so the agent that ran CPU-bound for weeks after a driver update leased and checked in like a healthy one. - The agent sends its startup accel report on every lease and heartbeat. - The server keeps a bounded copy on the roster row. A running agent with a runtime off the GPU becomes `degraded`, with a sentence naming the runtime and the reason. - The top nav shows it amber. - The agent page carries a banner, and its pill reads "CPU only". Also: the bandwidth field gets the page's − / + stepper, and both number fields drop the browser's spin arrows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-24 19:09:07 -04:00
bvandeusenandClaude Opus 5	45bb7044f7	feat: the System tab reads a stored sample instead of inspecting per load (4295) ... Operator: "there is a repull every time this page loads is there a reason this info isn't being tracked in the background and stored in some way?" There was a reason and it had expired, and underneath it there was plain waste. The expired one: /api/system/workers was deliberately uncached because an operator dragging the stepper must not be shown a pre-change value. That stopped being true at 1353d34, when the UI began patching its row from the write's reply instead of refetching. The waste: size_worker_lanes already inspected the broker on a timer to decide pool sizes — computing the pool, active, reserved and queue depth the page shows, using them, and discarding them. The browser then asked the broker for the same numbers four times a minute, per open tab. So one inspect now feeds three things: the sizing decision, a stored sample (worker_lane_sample, alembic 0107), and the celery roster. No request path touches the broker at all — the roster refresh comes off /api/system/health too, where it had been rate-limited to 20s and so made worker liveness a function of whether anyone had a browser open. Consequences, stated rather than hidden: - The live figures are up to one sweep old. measured_at travels with each lane and the page says how old, because a stale number presented as current is how someone watches a queue "not move" that is moving. - The sweep is the roster's only writer now, so its period and the staleness thresholds are in a relationship. 60s against a 90s stale threshold left one missed tick between normal and all-yellow — the shape of lesson #4355 — so the period is 30s, named once in worker_lanes, and system_health asserts its headroom at import with a test stating the same thing in prose. - An idle lane therefore also gives a worker back twice as fast. That is the direction asked for: "idle instances quiet down when not running". Also bounds the inspect in push_lane_cap, which was an await with no deadline (rule 156) — harmless while it ran on a request, less so now that it runs in a background task where a hang would be silent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 18:52:08 -04:00
bvandeusenandClaude Opus 5	7f1693a40d	fix: the ML dial offered slots the machine had no cores to feed (4295) ... Operator's 2026-09-23 log: embed_image taking 107-246s each, ~49 slots in flight by Little's law, and the daily CCIP sweep dying on its 1800s soft limit in a numpy matmul. The billiard/pool.py frame in that traceback is the soft-timeout signal handler, not a pool fault. Two causes, both mine. 1. `derived_ceiling` computed the ML lane from MEMORY ALONE. Meanwhile `embedder.py` carried `_INTRA_OP_THREADS = 4` beside a comment reading "keep N_replicas x this within the cores allotted to ML" — a constraint stated where nothing could act on it. A large-memory host offered ~49 slots, the operator took what the dial offered, and the lane asked the box for ~200 torch threads. The number moves onto the lane as `threads_per_slot`, the embedder reads it rather than restating it, and the ceiling is now the smaller of the two bounds. They fail differently on purpose: too little memory is honestly zero, because the first task would OOM the container; too few cores is merely slow, so it floors at one rather than making the lane unreachable on a small box. 2. `scheduled_ccip_auto_apply` scored one image per matmul, over every image in the library, on every daily run — ~119k products each too small to pay for its own BLAS setup. `char_maxima` does the same arithmetic in blocks bounded by elements, so its memory stays flat as either axis grows. Batching changes no arithmetic: a character's score for an image is a max over that image's figures AND that character's prototypes, and max does not care how it is grouped. Pinned against the old loop written out longhand, and against itself with the blocking forced to split every row. The UI copy said the ML ceiling came from memory; it says cores or memory, whichever runs out first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 16:46:22 -04:00
bvandeusenandClaude Opus 5	1353d346b3	fix: the cap dial waited out a broker round trip it did not need (4295) ... Operator: "when the number is changed the change should be queued so that it isn't blocking of the webui or the system itself. we shouldn't have to wait for the validation live." Two waits, and 5b6f2ba removed neither — it stopped a Postgres connection being HELD across them, which is what had been stalling the whole site, and left the press itself as slow as it was. 1. The store refetched after every write. GET /api/system/workers runs a celery inspect on an eleven-second budget, so the stepper stayed disabled through a round trip the press did not need. It now patches the row from the reply — cap, ceiling, enabled, the three fields that reply actually decides — and lets the 15s poll bring the live columns, which are measurements it must not invent. 2. The endpoint pushed to the broker before answering. Turning a lane off is four cancel_consumer messages; lowering a cap reads the live pool first. Now it stores the cap, answers `queued`, and hands the push to a Quart background task. Raising a cap was already free and stays free. Nothing is lost by not waiting: the stored cap is what the system obeys and the sizing pass re-reads it every minute. That sweep was already the backstop for a push that failed, which under `no_live_workers` is every push in the suite. Also closes a hole the move exposed: the model fetch was gated on the consumer change having landed, so raising ML off zero while the lane was restarting stored the cap, let the sizing pass start the consumers a minute later, and left the lane running with no model — nothing else ever asks for one. It now fires on the transition and waits in the ml queue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 15:41:32 -04:00
bvandeusenandClaude Opus 5	445164c852	feat: one number per lane — the cap — and the autoscaler is the mechanism (4295) ... Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle instances quiet down when not running. the number that is visible and something the user can tweak and manage should be the cap itself the number of running workers is handled by the autoscaling function which is always on."* They are right, and the reason it was not built this way is worth stating: the manual dial came first (steps 2-4) and the autoscaler came last (step 7), as an opt-in BESIDE a control that already existed. Nothing ever asked whether the dial should still exist once something could move it automatically. Each step was defensible; the result was three operator settings over one number. ## `slots`, `enabled` and `autoscale` are gone `slots` was a MEASUREMENT wearing a preference's clothes. How many workers a lane runs is read live and moved every minute; storing it meant the operator had to keep two numbers in agreement and the autoscaler had to be told it was allowed to touch one of them. `autoscale` gated the mechanism behind a choice, so a lane nobody opted in never gave its workers back — which is why an idle instance never quieted down. `enabled` is derived: a cap of zero means no consumers. "Off" and "may use no workers" were two spellings of one fact, stored separately, free to disagree. ## Two sweeps become one `reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_ sync` moved it away from that same number; and most of step 7's hardest reasoning — a stored value that is a FLOOR, a target of `max(stored, current)` — existed only to stop them fighting. Delete the stored number and the problem is not solved, it is absent. `size_lanes_sync` runs every minute and owns both consumers and pool size. It also subsumes what the reconcile was for: a worker restarted at its ENV concurrency is corrected on the next tick rather than after five. Growth is immediate, shrink is one worker per tick. Deliberately asymmetric — "always on" is only pleasant if the ramp keeps up, and +1/minute would take four minutes to answer a burst. Being one worker too large for a minute costs a sleeping process; being too small costs work not happening. For ML the asymmetry matters most: every new slot reloads a multi-GB model, so the slow shrink is what stops a quiet patch from paying that cost again a minute later. ## The caps ship at one, and zero for ML Per the operator. Conservative on purpose — and a conservative default nobody knows how to raise is just a slow product, which is the other half of what they asked for: "there needs to be something that tells the user to bump those numbers to improve processing rate or they'd never know the controls exist." So a lane running everything its cap allows while work piles up says so, in its own row, with the headroom named: *"4,060 waiting and all 1 worker busy. Raise the cap to run more at once — this machine allows up to 7."* It fires only when raising the cap would actually help. Not when the lane is keeping up, not when the sizing pass has room it has not taken, and not at the machine ceiling — where "raise the cap" is advice nobody can take. ## Migration 0105 rewrites the caps rather than carrying them The old defaults (4/2/2/1) bounded a manual control and were loose because moving within them was the ordinary act. The number now means "the most workers this lane may use", which is a different promise; carrying the old figure over would quadruple the worker lane on every existing install at the moment this deploys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 12:48:22 -04:00
bvandeusenandClaude Opus 5	abe16aa382	feat: the System tab is one bounded table, and the dial is the switch (4295) ... Operator, 2026-09-23, on the screenshot: *"I feel that we can probably combine the two sections into a single table and to format it in such a way that it appears more bounded and less free-form or open. also there's nothing to describe what 'auto' means or why their needs to be or should be on/off toggles. almost all of it always needs to run there's only one optional piece and it is killed by moving the 'cap' to zero."* Three separate things, all correct. ## The four lanes were listed twice The roster (milestone 365) said "ML tagging is running", and four hundred pixels below it the lanes pane said "ML tagging · 1/1 busy". Two answers to one question from two endpoints, free to disagree on screen. I moved the second pane onto this tab yesterday and did not notice it duplicated the first. Now one row per part, with controls on the rows that have a lane and none on the rows that do not. The join is on the QUEUE SET, because that is what `service_roster` keys a celery part on — as a set, not as a string, so neither side has to agree about order. It lives in `utils/systemParts.js` rather than inline, and has a spec, because its failure is SILENT and is the exact thing it exists to prevent: a lane that stops matching its part does not throw, it grows a second row for the same worker. The duplication, returning through the code that removed it. ## Bounded, not free-form A real table — header, column rules, one bordered card — instead of dotted rows floating on the page background with nothing saying where the list began or what a column meant. ## The dial is the switch There was an `On` switch per lane beside the slots dial. Of four lanes, three must run for the application to work at all, so that switch offered a choice that was never real — and for the one lane that IS optional, "off" and "zero slots" were two ways of saying the same thing that could disagree with each other. So `enabled` is now DERIVED from the number: `set_lane` sets it from `slots > 0` when the caller did not say. It stays on the API and in the model — it is still the mechanism, and a drain-before-restart may still want a lane holding its process with consumers cancelled without destroying the operator's slot count to say so. Two things fell out that a test now pins: - The consumer command is sent on the CHANGE, not on the field being present. Otherwise every slots write re-sends a command that changes nothing — lesson #4183's churn, arriving through the new derivation. - The model fetch fires on the off→on TRANSITION. It used to test `enabled is True`, the field having been sent. The UI no longer sends it, so the download that makes the ML lane usable would simply never have fired and the lane would have come on to consume a queue it had no model for. ## And Auto now says what it is A legend under the table, in the operator's terms: what a slot is, that zero turns a lane off, that three of the four are not optional, what `of N` means, and that Auto lets a lane add slots by itself when its queue is backed up AND every slot is busy — with why it is off by default, since it is the only thing on the page that acts without being asked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:26:27 -04:00
bvandeusenandClaude Opus 5	2f8f0bc9eb	feat: worker lanes move to the System tab, in the roster's own idiom (4295) ... Operator, 2026-09-23: *"where is the setting surface to set the number of each kind of worker? we have a system view under settings but no options to be set here"* — then *"move it to system tab and improve the view to be more inline with other UI elements."* They went looking under System and found a read-only roster. The card's own first cut argued that a control had no place on a tab about things being broken; that was wrong twice over. Half these lanes are optional and off by DESIGN, so "not running" is the normal state someone comes here to change — and a tab that only reports is a tab you leave in order to act somewhere else. So it is the roster's second half now, and built out of the same parts: the pane above says what is running, this one says how hard. Dot, name, detail, numbers on the right; the controls take the place the roster gives to latency. It was a bordered card wrapping a v-table, which is what made it read as bolted on. - The row styles move from SystemHealthTab's scoped block into app.css, so the second pane IS the first one's idiom rather than imitating it. A copy would have drifted the first time either was touched. - Lane state uses the roster's four dots. A lane that is off reads grey, never red: colouring an intended state as a fault is how an indicator stops being read. - The card owns its own polling. On Activity it rode that tab's 3s queue timer; System has none, so moving without this would have shown an empty pane to anyone who opened System first. 15s and `document.hidden`-guarded — it is a dial, not a live meter, and v-window keeps a hidden tab mounted. - The table header carried the column names, so each control now labels itself: `slots · cap N`, `auto`, `on`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 11:01:54 -04:00
bvandeusenandClaude Opus 5	ad8392b790	fix: system health is a Settings tab, not a page only the dot reached ... The surface shipped at /system with no nav entry, reachable only by clicking the health dot beside the brand — a target you have to already suspect something is wrong to go looking for. Operator-flagged: it needs a path someone can walk to. Settings is where you go to ask the instance about itself, so the view becomes a tab there, beside Activity — Activity answers "what is the queue doing", System answers "is anything left to do it". - SystemView.vue moves to components/settings/SystemHealthTab.vue; the content is unchanged apart from shedding its own container and h1. - SettingsView adopts useTabQuery (the composable Browse and Subscriptions already use) so a tab can be linked TO. The health dot now points at ?tab=system, and /system redirects there so the previous build's link and any bookmark still land. - The tab drops its own 10s poll. v-window keeps a visited item mounted rather than destroyed, so that timer would have gone on firing behind Maintenance — and TopNav already polls the same store every 15s for the dot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 19:45:54 -04:00