Commit Graph
3 Commits
Author SHA1 Message Date
bvandeusenandClaude Opus 5 abe16aa382 feat: the System tab is one bounded table, and the dial is the switch (4295)
CI and images / frontend-build (push) Successful in 20s
CI and images / backend-lint-and-test (push) Successful in 32s
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / integration (push) Successful in 2m10s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 2m10s
CI and images / smoke-web (push) Successful in 52s
CI and images / promote (push) Skipped
Operator, 2026-09-23, on the screenshot: *"I feel that we can probably combine
the two sections into a single table and to format it in such a way that it
appears more bounded and less free-form or open. also there's nothing to
describe what 'auto' means or why their needs to be or should be on/off
toggles. almost all of it always needs to run there's only one optional piece
and it is killed by moving the 'cap' to zero."*

Three separate things, all correct.

## The four lanes were listed twice

The roster (milestone 365) said "ML tagging is running", and four hundred
pixels below it the lanes pane said "ML tagging · 1/1 busy". Two answers to
one question from two endpoints, free to disagree on screen. I moved the
second pane onto this tab yesterday and did not notice it duplicated the
first.

Now one row per part, with controls on the rows that have a lane and none on
the rows that do not. The join is on the QUEUE SET, because that is what
`service_roster` keys a celery part on — as a set, not as a string, so neither
side has to agree about order.

It lives in `utils/systemParts.js` rather than inline, and has a spec, because
its failure is SILENT and is the exact thing it exists to prevent: a lane that
stops matching its part does not throw, it grows a second row for the same
worker. The duplication, returning through the code that removed it.

## Bounded, not free-form

A real table — header, column rules, one bordered card — instead of dotted
rows floating on the page background with nothing saying where the list began
or what a column meant.

## The dial is the switch

There was an `On` switch per lane beside the slots dial. Of four lanes, three
must run for the application to work at all, so that switch offered a choice
that was never real — and for the one lane that IS optional, "off" and "zero
slots" were two ways of saying the same thing that could disagree with each
other.

So `enabled` is now DERIVED from the number: `set_lane` sets it from
`slots > 0` when the caller did not say. It stays on the API and in the model
— it is still the mechanism, and a drain-before-restart may still want a lane
holding its process with consumers cancelled without destroying the operator's
slot count to say so.

Two things fell out that a test now pins:

- The consumer command is sent on the CHANGE, not on the field being present.
  Otherwise every slots write re-sends a command that changes nothing —
  lesson #4183's churn, arriving through the new derivation.
- The model fetch fires on the off→on TRANSITION. It used to test `enabled is
  True`, the field having been sent. The UI no longer sends it, so the
  download that makes the ML lane usable would simply never have fired and the
  lane would have come on to consume a queue it had no model for.

## And Auto now says what it is

A legend under the table, in the operator's terms: what a slot is, that zero
turns a lane off, that three of the four are not optional, what `of N` means,
and that Auto lets a lane add slots by itself when its queue is backed up AND
every slot is busy — with why it is off by default, since it is the only thing
on the page that acts without being asked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 11:26:27 -04:00
bvandeusenandClaude Opus 5 a01165365b feat: a saturated lane can grow itself, within the cap the operator set (4297)
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 36s
Build images / build-ml (push) Successful in 1m55s
Build images / build-web (push) Successful in 1m54s
CI / integration (push) Successful in 2m16s
Build images / smoke-web (push) Failing after 7m48s
Build images / promote (push) Skipped
Milestone 422 step 7 — the one sweep in this milestone that decides rather
than obeys, so it is off until a lane is opted in, bounded by the operator's
cap, floored at the operator's value, and it reports every decision including
the ones where it did nothing.

Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means
celery is about to pick those up and growing would add idle children (#1253
is that bug in the GPU agent); saturation alone means the lane is busy with
exactly as much work as exists. The backlog is depth PLUS reserved, because
celery prefetches and LLEN reads 0 while a worker holds thirty tasks in
memory — the case an LLEN-only autoscaler misses entirely, and the reason
step 2 plumbed `reserved` through.

The two sweeps had to be taught not to fight. The reconcile drives every
lane to its stored slots every five minutes, which would have reverted each
grow on the next tick: grow, revert, grow, revert, forever. For an
autoscaling lane the stored value is now a FLOOR — restored when a lane
falls below it, never taken back above it.

The operator's "a task that runs for x concurrent time" idea stays a UI
warning rather than a trigger: a long task does not finish sooner because
the lane gained a slot, so scaling on it would spend memory to change
nothing. Read from `task_run` on our own wall clock, not celery's
`time_start`, which is the WORKER's monotonic clock and would produce a
duration that is meaningless in the direction that matters — plausible.

Caught while reading it back: the first version read the stored slots as the
CURRENT pool. The autoscaler never writes that row, so every tick would have
proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica
already past the target is issued no message and reports success), and
capping the lane one slot above its floor forever while claiming otherwise.
It now reads the live pool and keeps the stored value purely as the floor,
and the tests fix the two to different numbers so an equal-fixture pass
cannot hide it again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-22 10:05:43 -04:00
bvandeusenandClaude Opus 5 da48edf7da feat: the worker-lanes card — see each lane, change its slots (4294)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 7s
Build images / build-ml (push) Successful in 7s
CI / frontend-build (push) Successful in 20s
CI / backend-lint-and-test (push) Successful in 34s
Build images / build-web (push) Successful in 59s
Build images / smoke-web (push) Skipped
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m19s
Milestone 422 step 4. Rule 27: no UI, no ship. This is where the previous
three steps become usable.

REACHABLE AT: Settings -> Activity -> Worker lanes, directly under the
"Queues + workers" pane. Written from opening the view, not from memory of
having built it — lesson #4282, and #3463 is the same trap landing inside
milestone 365, where the System page shipped with no navigation to it.

Under that pane deliberately, not in the System tab. System answers "is
everything running", where every control would be about something broken.
This is about something working that should work harder, and it belongs
beside the backlog it reacts to: you watch a queue grow and give that lane
another slot without leaving the pane.

Per lane: queues, pending, busy, a stepper, an enable switch.

- PENDING is depth + reserved, not LLEN. Celery prefetches, so LLEN alone
  reads 0 while a worker holds tasks in memory — the number that would make
  someone think a buried lane was idle.
- NOT ANSWERING, never "stopped". present=false means nothing replied; saying
  stopped would send the operator looking for a crash that has not happened.
- THE CEILING IS ON SCREEN, with "(memory)" on the ML lane. It is the one
  number here the operator cannot change, so it has to justify itself; a
  greyed stepper with no explanation reads as a bug.
- `busy` is per lane, so adjusting one does not freeze the others.

TWO OUTCOMES THAT MUST NOT COLLAPSE INTO ONE MESSAGE. A stored-but-unpushed
change (applied:false — the lane is restarting) is information: the value is
saved and the reconcile will carry it, so the card says so and invites
waiting. A refused value (400) is an error and shows the endpoint's own
sentence. Collapsing them would make one of the two invite the wrong action.

A BUG CAUGHT BEFORE COMMIT: the card read `e.detail?.detail`, but ApiError
puts the parsed body on `.body` and `.message` on the short `error` key. It
would have shown the operator the bare word "refused" with no reason — the
exact failure that line exists to prevent. The test would not have caught it
either: `rejects.toThrow()` passes whether the sentence is reachable or not.
It now asserts `err.body.detail` specifically.

Also: queueOptions in SystemActivityTab was a fourth hand-kept copy of the
queue list and had drifted — `maintenance_long` was missing, so activity on
that lane could not be filtered for at all despite four task routes pointing
there. Added, and DELIBERATELY left as a written-out list rather than derived
like the other three were: this filters task_run HISTORY, so a derived list
would hide the filter for any queue that has rows but no longer has a lane —
precisely when someone is looking — and would be empty whenever the endpoint
is down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-22 08:23:14 -04:00