Milestone 422 step 2. `GET /api/system/workers` reports every lane joined to
its live pool; `POST /api/system/workers/<name>` changes it.
NO DOCKER SOCKET. Milestone 365 deferred "acting on the state" because
restarting a dead worker needs a socket the web container deliberately does
not have. That holds for restarting a CONTAINER; it does not hold for
changing how much work a RUNNING worker does. celery's pool_grow /
pool_shrink / add_consumer / cancel_consumer send a message over the Redis
the app already uses, and the worker resizes itself. No new privilege, no new
surface, and the security question that deferred this is never raised.
PERSIST AND PUSH, in one call, in that order. pool_grow is not durable — a
restart drops every lane to its env concurrency — so a UI that only pushed
would lose the setting on the next deploy with nothing to show for it (lesson
#4202). Storing alone would describe nothing until something restarted. A
failed PUSH is not a failed setting: 200 with `applied: false` and a reason,
so the UI says "saved, not yet live" rather than "that didn't work". Step 3's
reconcile carries it when the lane answers again.
PER-REPLICA DELTAS. `pool_grow(n, destination=[...])` adds n to EACH
destination, so while `worker` runs `replicas: 2` a single delta from an
aggregate is wrong for both. `slots` therefore means what CELERY_CONCURRENCY
means — one process's pool — and each replica is driven to it from its OWN
current size, so replicas that drifted apart converge rather than moving in
lockstep. I wrote this wrong first: the docstring claimed per-replica while
the code computed one delta from the max across replicas. LaneLiveState now
carries `pools` per hostname and exposes `pool` as a property.
A replica already at the target is sent nothing at all — the reachable fixed
point step 3's periodic reconcile needs, or it re-issues a grow of zero every
tick forever (lesson #4183). A replica that answered inspect but not stats is
NAMED in the error rather than skipped silently, since otherwise it would run
at a size the UI claims it does not.
`present=False` is not "zero slots", it is "nothing answered" — kept distinct
throughout, because step 3 skips an absent lane rather than correcting it.
/workers now also reports pool size (from `insp.stats()`) and RESERVED count.
Celery prefetches, so tasks that have left the Redis list but not started are
invisible to LLEN: a lane can read depth 0 with thirty tasks held in worker
memory. `pending` is depth + reserved. The UI is misleading without this and
step 7's autoscaler would be simply wrong.
Also kills the THIRD copy of the queue list: system_activity's _QUEUE_NAMES,
whose own comment admitted the coupling ("must match celery_app.task_routes")
and which sat alongside task_routes and the ROLE_NAMES copy step 1 collapsed.
Now derived from LANES. The rendered order changes to lane grouping, which is
the better shape for a lane-oriented UI.
Separate blueprint rather than folding into system_activity, which states in
its first line that it is read-only and answers a different question — its
/workers is keyed on celery HOSTNAME and reports which nodes answered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
Recent failures gains a client-side search over the already-loaded 24h
rows (task/queue/target/error), shown as a filtered/total count alongside
the existing error-type chips. All recent activity gains a debounced
server-side task-name search (new `task` ILIKE param on /runs) so it
spans the full history, not just the loaded page. LIKE wildcards are
escaped so task names' literal underscores match literally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Even chunked, a single concurrency-1 maintenance lane is fragile — a 30-min DB
backup or a multi-chunk library audit holds the slot and delays the quick
self-healing recovery sweeps / vacuum (operator-flagged 2026-06-07: long runs
must never block quick maintenance).
Route the long one-shots — backup.*, admin.* (normalize/re-extract/cascade-
delete), library_audit.* — to a new `maintenance_long` queue served by a
dedicated worker (concurrency 1), added to docker-compose (+ dev override). The
scheduler keeps the quick `maintenance` lane (sweeps, vacuum, cleanup) for
itself, so a backup can no longer starve a 5-min vacuum. UI queue list +
routing tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New /api/system/activity/summary aggregates scheduler health + per-queue
pending depths + running count + 24h failure count in one cached call (safe
to poll app-wide). PipelineStatusChip lives in the TopNav on every page: a
compact running/queued/failing chip with a scheduler-health dot that expands
to a popover (scheduler, busy queues, counts, link to downloads). Polls the
summary every 8s, paused when backgrounded. Reuses the queue-cache read via
_queues_cached(). + API test for the summary shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bvandeusenandClaude Opus 5