fix: a lane that is OFF was not attributable to itself (4295)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 36s
CI and images / integration (push) Successful in 2m26s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 36s
CI and images / integration (push) Successful in 2m26s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
Operator, 2026-09-23: *"clean up the stale service_seen rows"*.
**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.
A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:
1. The lanes table reported the lane **not answering** — the signal for a
crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
rendered as a display name — shown "running" beside the real lane's row
going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
lane present. ML ships at cap 0, so a fresh install was permanently
unhealthy and Swarm restarts an unhealthy task forever.
That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.
`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.
The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.
Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -62,6 +62,7 @@ from .worker_lanes import (
|
||||
MIN_POOL_SLOTS,
|
||||
Lane,
|
||||
derived_ceiling,
|
||||
lane_for_node,
|
||||
)
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -189,7 +190,13 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]:
|
||||
return out
|
||||
|
||||
for hostname, queues in active_queues.items():
|
||||
lane = _lane_for_queues(tuple(q["name"] for q in queues))
|
||||
# The NODE NAME first — see `lane_for_node`. A lane at cap 0 has its
|
||||
# consumers cancelled and answers with an empty queue list, which
|
||||
# matches no lane, so attributing by queues alone dropped every lane
|
||||
# the operator had turned off and reported it as "not answering".
|
||||
lane = lane_for_node(hostname) or _lane_for_queues(
|
||||
tuple(q["name"] for q in queues)
|
||||
)
|
||||
if lane is None:
|
||||
# A deployment slicing CELERY_QUEUES differently. Reported by the
|
||||
# roster under its raw queue list; it simply has no lane row to
|
||||
|
||||
Reference in New Issue
Block a user