bvandeusenandClaude Opus 5
48108a3569
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 36s
CI and images / integration (push) Successful in 2m26s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
Operator, 2026-09-23: *"clean up the stale service_seen rows"*.
**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.
A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:
1. The lanes table reported the lane **not answering** — the signal for a
crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
rendered as a display name — shown "running" beside the real lane's row
going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
lane present. ML ships at cap 0, so a fresh install was permanently
unhealthy and Swarm restarts an unhealthy task forever.
That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.
`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.
The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.
Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR