fix: a lane that is OFF was not attributable to itself (4295)

Operator, 2026-09-23: *"clean up the stale service_seen rows"*.

**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.

A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:

1. The lanes table reported the lane **not answering** — the signal for a
   crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
   rendered as a display name — shown "running" beside the real lane's row
   going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
   lane present. ML ships at cap 0, so a fresh install was permanently
   unhealthy and Swarm restarts an unhealthy task forever.

That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.

`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.

The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.

Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/scripts/healthcheck.py

@@ -121,11 +121,19 @@ def _this_node_ok() -> tuple[bool, str]:
 def _lanes_ok() -> tuple[bool, str]:
     """Every lane in the table is answering.
 
-    Deliberately ignores whether a lane is ENABLED: a disabled lane still runs
-    its process with its consumers cancelled, so it answers `inspect` and is
+    Deliberately ignores whether a lane is ON: a lane at cap 0 still runs its
+    process with its consumers cancelled, so it answers `inspect` and is
     healthy. Health is "is the process alive"; whether it should be consuming
-    is a settings question the reconcile owns, and conflating them would make
-    turning a lane off in the UI mark the container unhealthy.
+    is a settings question the sizing pass owns, and conflating them would
+    make turning a lane off mark the container unhealthy.
+
+    That was not merely a risk — it was happening. Until 2026-09-23 a worker
+    was attributed to its lane by the queues it was CONSUMING, and a lane with
+    its consumers cancelled reports none, so it read as absent and this check
+    failed. ML ships at cap 0, so a fresh install was permanently unhealthy
+    and Swarm restarts an unhealthy task forever. The docstring above said the
+    right thing while the code did the opposite; `worker_lanes.lane_for_node`
+    is what makes it true.
     """
     from ..services.worker_control import inspect_lanes_sync
     from ..services.worker_lanes import LANES

backend/app/services/service_roster.py

@@ -36,7 +36,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from ..models import ServiceSeen
-from .worker_lanes import LANES
+from .worker_lanes import LANES, lane_for_node
 
 log = logging.getLogger(__name__)
 
@@ -118,7 +118,20 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
 
     grouped: dict[tuple[str, ...], dict] = {}
     for hostname, queues in active_queues.items():
-        key = tuple(sorted({q["name"] for q in queues}))
+        # Keyed on the LANE's queue set when the node name identifies one, so
+        # a lane keeps the same roster row whether or not it is consuming.
+        #
+        # Grouping on the ACTIVE queues alone meant a lane at cap 0 — which
+        # cancels its consumers — reported an empty set, landed under the key
+        # `celery:`, and rendered as a phantom row named `Worker ()` while its
+        # real row went stale beside it. Both symptoms on the operator's
+        # screen, 2026-09-23, from this one line.
+        #
+        # Deriving the key from `lane.queue_key` rather than inventing a new
+        # one keeps every existing row: it is the same string the lane already
+        # had while it was running.
+        lane = lane_for_node(hostname)
+        key = lane.queue_key if lane else tuple(sorted({q["name"] for q in queues}))
         entry = grouped.setdefault(key, {"hostnames": [], "active": 0})
         entry["hostnames"].append(hostname)
         entry["active"] += len(active_tasks.get(hostname, []))

backend/app/services/worker_control.py

@@ -62,6 +62,7 @@ from .worker_lanes import (
     MIN_POOL_SLOTS,
     Lane,
     derived_ceiling,
+    lane_for_node,
 )
 
 log = logging.getLogger(__name__)
@@ -189,7 +190,13 @@ def inspect_lanes_sync() -> dict[str, LaneLiveState]:
         return out
 
     for hostname, queues in active_queues.items():
-        lane = _lane_for_queues(tuple(q["name"] for q in queues))
+        # The NODE NAME first — see `lane_for_node`. A lane at cap 0 has its
+        # consumers cancelled and answers with an empty queue list, which
+        # matches no lane, so attributing by queues alone dropped every lane
+        # the operator had turned off and reported it as "not answering".
+        lane = lane_for_node(hostname) or _lane_for_queues(
+            tuple(q["name"] for q in queues)
+        )
         if lane is None:
             # A deployment slicing CELERY_QUEUES differently. Reported by the
             # roster under its raw queue list; it simply has no lane row to

backend/app/services/worker_lanes.py

@@ -332,6 +332,44 @@ def container_cpu_count() -> int | None:
     return os.cpu_count()
 
 
+def lane_for_node(hostname: str) -> Lane | None:
+    """`ml@7f3c9a1b` -> the ml lane. None for a node this build did not name.
+
+    ## Why the node name, and not the queues it is consuming
+
+    Because a lane that is OFF is consuming nothing, and "nothing" identifies
+    no lane at all.
+
+    Both the roster and `inspect_lanes_sync` used to map a worker to its lane
+    through `active_queues()`. That is exact while the lane is running and
+    useless the moment it is not: a lane at cap 0 has its consumers cancelled,
+    so it answers the broadcast with an EMPTY queue list, matches no lane, and
+    is dropped. Three things followed, and the operator saw all three at once
+    on 2026-09-23:
+
+    1. The lanes table showed the lane as **not answering** — which is the
+       signal for a crashed worker, not for one the operator turned off.
+    2. The roster grew a phantom row called **`Worker ()`**, the empty queue
+       set rendered as a display name, "running" beside the real lane's row
+       going stale.
+    3. **The container went unhealthy.** `healthcheck._lanes_ok` requires
+       every lane in the table to be present, and its docstring asserted the
+       opposite of what the code did — *"a disabled lane still runs its
+       process with its consumers cancelled, so it answers inspect and is
+       healthy"*. It answers; it is not attributed. ML ships at cap 0, so a
+       fresh install would have been permanently unhealthy, and Swarm
+       restarts an unhealthy task forever.
+
+    The node name survives all of that: `gen_supervisord` sets
+    `CELERY_NODENAME={lane.name}` per program and the entrypoint passes it to
+    `celery -n`, so the identity travels with the PROCESS rather than with
+    what it happens to be doing. Falls back to the queue set for a deployment
+    that sets no node name — the multi-service compose stack, where every node
+    is `celery@<host>`.
+    """
+    return LANES_BY_NAME.get(hostname.split("@", 1)[0])
+
+
 def derived_ceiling(lane: Lane) -> int:
     """The most slots `lane` may be given on this container.