4 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	ecbd325437	feat: an optional lane says it is optional, and what enabling it costs (4296) ... Operator, 2026-09-22: "since the ml-worker is optional it should be shown as such in the UI and have a warning about what it does and that it pulls the models and what models and their projected size and ram requirements to run." The card previously said "a few GB, once" — a number sourced from nothing, which is exactly the hand-wave I had flagged in this step's own survey log as something that should be measured rather than asserted. ONE FACT CORRECTED WHILE WRITING THE COPY. I had named the lane "ML tagging". It downloads an EMBEDDER: google/siglip-so400m-patch14-384. WD14 tagging is the GPU agent's job — celery_app.py:5 still names both, but that has been stale since B3 (#1238), when the agent took over and this lane was left as the CPU embed fallback for stacks running no agent (see MLSettings.cpu_embed_enabled). Telling someone the lane "does tagging" would have been wrong in exactly the way this request exists to prevent. The facts are structured data on the lane, not prose in a component: ModelRequirement(repo, approx_download_bytes, approx_resident_bytes, measured). The API carries them; the card renders them. Numbers come from the system, wording from the UI. ML_BYTES_PER_SLOT IS NOW DERIVED from that requirement rather than stated separately. They have to be one number: the figure quoted to the operator before they enable the lane and the figure the cap enforces. Two copies could disagree, and the UI would promise a slot the cap then refuses. `measured=False` travels with the numbers and the card renders "about". They are estimates from the checkpoint's parameter count and dtype — ~877M params at fp32 is ~3.5GB of weights — not from a build. This decides whether someone's server survives, so it is labelled rather than rounded into something that reads like a fact. A test asserts the flag is false, to be flipped in the same commit that records a real measurement. The card now shows: an "optional" chip in the row itself (someone scanning the table should not have to enable a lane to learn it was never required), and before the switch, what the lane does, that you only need it if you are NOT running the GPU agent, the repo id, the download size, the per-slot RAM, and why the ceiling is what it is — including saying plainly when a box has too little memory to run it at all. Keyed on the lane's own `optional` flag, not on the name 'ml', so a second optional lane gets the same treatment without anyone remembering to add it. A test asserts no REQUIRED lane declares a model: if one ever needs a download, it stops being required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:45:22 -04:00
bvandeusenandClaude Opus 5	5f8c63f61b	feat: reconcile every running lane back to its stored slots (4293) ... Milestone 422 step 3. `pool_grow` is not durable: a worker restarted by its supervisor comes back at its ENV concurrency, silently below whatever the operator set, and nothing on step 2's write path would ever notice. Storing the value made it survivable; this makes it survive. A BEAT TASK, NOT A HOOK IN WEB — a deliberate deviation from the step as written, for a reason already recorded in this codebase. Step 3 said "web applies the stored values after it starts". It cannot: service_roster.py documents that hypercorn runs --workers 4, so anything in before_serving becomes four concurrent loops per container hammering the broker forever. service_roster's own answer — refresh on demand from whichever request arrives — was also rejected, because the two solve different problems. A stale ROSTER only misleads someone looking at it, so recomputing when they look is exactly right. A lane running at the wrong size is doing less work than it was told to whether or not anyone is watching, and the case that matters is a deploy at 3am followed by a backlog nobody is awake to see. So: unattended, every 5 minutes, on the quick `maintenance` lane beside the other recovery sweeps. Accepted cost — a dead scheduler stops reconciliation, but a dead scheduler already stops every other sweep and the roster reports it, so this adds no new blind spot. A BUG I WROTE AND CAUGHT BEFORE COMMITTING. The first version called set_lane_enabled_sync unconditionally, so a settled system re-sent add_consumer for every queue on every tick — forever. Harmless per call (add_consumer on an already-consumed queue does nothing), unbounded in aggregate, and completely invisible. That is lesson #4183's failure mode exactly, in the very function whose docstring cites it. Worse, my test would not have caught it: it asserted only on grew/shrank. LaneLiveState now carries `consuming` — which queues a lane is actually serving, distinct from the queues it was configured with — so the reconcile compares before acting. The test now asserts ALL FOUR control families are silent on a settled tick, plus a new case for an already-disabled lane, which is the other half of the same fixed point. An absent lane is SKIPPED, not corrected. present=False means nothing answered, not zero slots; correcting it would be a conclusion from an unswept read (snippet #3969), and there would be nothing to send the message to. One lane failing does not stop the others. One inspect serves every lane: the two setters now take an optional pre-fetched LaneLiveState, so a tick costs one broker round trip rather than one per lane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:08:38 -04:00
bvandeusenandClaude Opus 5	a9c1b421a7	feat: change a lane's slots on a running system, over the broker (4292) ... Milestone 422 step 2. `GET /api/system/workers` reports every lane joined to its live pool; `POST /api/system/workers/<name>` changes it. NO DOCKER SOCKET. Milestone 365 deferred "acting on the state" because restarting a dead worker needs a socket the web container deliberately does not have. That holds for restarting a CONTAINER; it does not hold for changing how much work a RUNNING worker does. celery's pool_grow / pool_shrink / add_consumer / cancel_consumer send a message over the Redis the app already uses, and the worker resizes itself. No new privilege, no new surface, and the security question that deferred this is never raised. PERSIST AND PUSH, in one call, in that order. pool_grow is not durable — a restart drops every lane to its env concurrency — so a UI that only pushed would lose the setting on the next deploy with nothing to show for it (lesson #4202). Storing alone would describe nothing until something restarted. A failed PUSH is not a failed setting: 200 with `applied: false` and a reason, so the UI says "saved, not yet live" rather than "that didn't work". Step 3's reconcile carries it when the lane answers again. PER-REPLICA DELTAS. `pool_grow(n, destination=[...])` adds n to EACH destination, so while `worker` runs `replicas: 2` a single delta from an aggregate is wrong for both. `slots` therefore means what CELERY_CONCURRENCY means — one process's pool — and each replica is driven to it from its OWN current size, so replicas that drifted apart converge rather than moving in lockstep. I wrote this wrong first: the docstring claimed per-replica while the code computed one delta from the max across replicas. LaneLiveState now carries `pools` per hostname and exposes `pool` as a property. A replica already at the target is sent nothing at all — the reachable fixed point step 3's periodic reconcile needs, or it re-issues a grow of zero every tick forever (lesson #4183). A replica that answered inspect but not stats is NAMED in the error rather than skipped silently, since otherwise it would run at a size the UI claims it does not. `present=False` is not "zero slots", it is "nothing answered" — kept distinct throughout, because step 3 skips an absent lane rather than correcting it. /workers now also reports pool size (from `insp.stats()`) and RESERVED count. Celery prefetches, so tasks that have left the Redis list but not started are invisible to LLEN: a lane can read depth 0 with thirty tasks held in worker memory. `pending` is depth + reserved. The UI is misleading without this and step 7's autoscaler would be simply wrong. Also kills the THIRD copy of the queue list: system_activity's _QUEUE_NAMES, whose own comment admitted the coupling ("must match celery_app.task_routes") and which sat alongside task_routes and the ROLE_NAMES copy step 1 collapsed. Now derived from LANES. The rendered order changes to lane grouping, which is the better shape for a lane-oriented UI. Separate blueprint rather than folding into system_activity, which states in its first line that it is read-only and answers a different question — its /workers is keyed on celery HOSTNAME and reports which nodes answered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 08:01:23 -04:00
bvandeusenandClaude Opus 5	5974a1bfbc	fix: two errors in the worker-lane tests (4291) ... Both mine, both in tests/test_worker_lanes.py, neither in the code under test. Run 7242. RUFF I001 — two blank lines between the import block and the first module-level comment. Rule 102 names this exact trap ("exactly ONE blank line between imports and a module-level constant/comment/pytestmark") and I was pointed at that rule repeatedly before opening it. SIX FAILURES in test_worker_lane_check_constraints — the test asserted bare constraint names, but Base.metadata's naming_convention has already applied the `ck_worker_lane_` prefix by the time __table__.constraints is read. The failure output is worth keeping: it shows the model emits exactly the three intended constraints, prefixed once — ck_worker_lane_slots_non_negative slots >= 0 ck_worker_lane_cap_non_negative slots_cap >= 0 ck_worker_lane_slots_within_cap slots <= slots_cap — which is the model behaving correctly, and confirms the migration's op.f() names match what the ORM produces. The assertion is now an equality against the prefixed names plus an explicit check for a doubled prefix. That is strictly more valuable than what I wrote: a bare-name assertion would have passed just as happily against `ck_worker_lane_ck_worker_lane_slots_within_cap`, which is the defect alembic 0088 had to rename four constraints for (#3275). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-22 07:53:30 -04:00