fix: a lane that is OFF was not attributable to itself (4295)

Operator, 2026-09-23: *"clean up the stale service_seen rows"*.

**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.

A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:

1. The lanes table reported the lane **not answering** — the signal for a
   crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
   rendered as a display name — shown "running" beside the real lane's row
   going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
   lane present. ML ships at cap 0, so a fresh install was permanently
   unhealthy and Swarm restarts an unhealthy task forever.

That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.

`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.

The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.

Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

tests/test_service_roster.py

@@ -52,3 +52,80 @@ def test_the_round_trip_count_matches_the_calls_actually_made():
         f"_inspect_celery_sync makes {calls} inspect calls but "
         f"INSPECT_ROUND_TRIPS says {sr.INSPECT_ROUND_TRIPS}"
     )
+
+
+# --- a lane that is off keeps its own row ------------------------------------
+
+
+def _stub_inspect(monkeypatch, active_queues):
+    class _Insp:
+        def __init__(self, **_):
+            pass
+
+        def active_queues(self):
+            return active_queues
+
+        def active(self):
+            return {}
+
+    class _Control:
+        inspect = _Insp
+
+    import sys
+    import types
+    mod = types.ModuleType("backend.app.celery_app")
+
+    class _C:
+        pass
+
+    c = _C()
+    c.control = _Control()
+    mod.celery = c
+    monkeypatch.setitem(sys.modules, "backend.app.celery_app", mod)
+
+
+def test_a_lane_with_no_consumers_keeps_the_row_it_had_while_running(monkeypatch):
+    """The phantom, and the reason the operator had two wrong rows at once.
+
+    A lane at cap 0 cancels its consumers, so it answers with an EMPTY queue
+    list. Grouped on that, it landed under the key `celery:` and rendered as a
+    row called `Worker ()` — reported running — while the real lane's row went
+    stale beside it because nothing updated it any more.
+
+    Keyed on the LANE's queue set now, which is the same string the row
+    already had while the lane was consuming. So turning a lane off updates
+    its row instead of minting a second one.
+    """
+    from backend.app.services.worker_lanes import LANES_BY_NAME
+
+    _stub_inspect(monkeypatch, {"ml@abc123": []})
+
+    grouped = sr._inspect_celery_sync()
+
+    assert list(grouped) == [LANES_BY_NAME["ml"].queue_key]
+    assert () not in grouped, "the empty queue set is the phantom `Worker ()`"
+
+
+def test_the_row_is_the_same_one_whether_the_lane_is_consuming_or_not(monkeypatch):
+    """Stated as an identity rather than as two separate assertions: if these
+    keys ever differ, turning a lane off silently starts a second roster row
+    and the first goes stale — which is exactly what happened."""
+    ml_queues = [{"name": "ml"}]
+
+    _stub_inspect(monkeypatch, {"ml@abc123": ml_queues})
+    on = set(sr._inspect_celery_sync())
+    _stub_inspect(monkeypatch, {"ml@abc123": []})
+    off = set(sr._inspect_celery_sync())
+
+    assert on == off
+
+
+def test_a_worker_this_build_did_not_name_is_still_grouped_by_its_queues(
+    monkeypatch,
+):
+    """The fallback, and the case the roster exists to report honestly: a
+    deployment slicing CELERY_QUEUES differently gets its raw queue list
+    rather than a name this code invented for it."""
+    _stub_inspect(monkeypatch, {"celery@xyz": [{"name": "odd"}]})
+
+    assert list(sr._inspect_celery_sync()) == [("odd",)]

tests/test_worker_control.py

@@ -570,3 +570,104 @@ def test_the_sizing_task_is_registered_and_scheduled():
     tasks = {e["task"] for e in entries.values()}
     assert "backend.app.tasks.maintenance.reconcile_worker_lanes" not in tasks
     assert "backend.app.tasks.maintenance.autoscale_worker_lanes" not in tasks
+
+
+# --- a lane that is OFF is still its own lane ---------------------------------
+#
+# The operator's screen, 2026-09-23: "ML tagging has not checked in for 135
+# min" beside a phantom row called "Worker ()" reported as running. One cause:
+# a worker was attributed to its lane by the queues it was CONSUMING, and a
+# lane at cap 0 has its consumers cancelled, so it consumes none and matched
+# nothing.
+
+
+def _stub_active_queues(monkeypatch, by_host):
+    """Only `active_queues` — the read that decides which lane a node IS."""
+    class _Insp:
+        def __init__(self, **_):
+            pass
+
+        def active_queues(self):
+            return by_host
+
+        def stats(self):
+            return {h: {"pool": {"max-concurrency": 1}} for h in by_host}
+
+        def active(self):
+            return {}
+
+        def reserved(self):
+            return {}
+
+    control = _stub_control(monkeypatch)
+    control.inspect = _Insp
+    import sys
+    import types
+    mod = types.ModuleType("backend.app.celery_app")
+
+    class _C:
+        pass
+
+    c = _C()
+    c.control = control
+    mod.celery = c
+    monkeypatch.setitem(sys.modules, "backend.app.celery_app", mod)
+
+
+def test_a_lane_with_its_consumers_cancelled_is_still_found(monkeypatch):
+    """THE bug. `ml@host` consuming nothing must read as the ml lane, present,
+    rather than as no lane at all.
+
+    Attributed by the NODE NAME, which survives having no consumers —
+    `gen_supervisord` sets `CELERY_NODENAME={lane.name}` per program for
+    exactly this."""
+    _stub_active_queues(monkeypatch, {"ml@abc123": []})
+
+    live = wc.inspect_lanes_sync()
+
+    assert live["ml"].present is True
+    assert live["ml"].consuming == set(), "it is present AND consuming nothing"
+
+
+def test_that_is_what_keeps_the_container_healthy(monkeypatch):
+    """Why it mattered more than a cosmetic row.
+
+    `healthcheck._lanes_ok` requires every lane to be present, and ML ships at
+    cap 0 — so a fresh install reported a lane not answering, the container
+    went permanently unhealthy, and Swarm restarts an unhealthy task forever.
+    """
+    _stub_active_queues(monkeypatch, {
+        "worker@a": [{"name": q} for q in LANES_BY_NAME["worker"].queues],
+        "scheduler@a": [{"name": q} for q in LANES_BY_NAME["scheduler"].queues],
+        "maintenance_long@a": [
+            {"name": q} for q in LANES_BY_NAME["maintenance_long"].queues
+        ],
+        "ml@a": [],   # off, as it ships
+    })
+
+    live = wc.inspect_lanes_sync()
+
+    missing = sorted(name for name, s in live.items() if not s.present)
+    assert missing == [], f"the healthcheck would fail the container: {missing}"
+
+
+def test_a_node_this_build_did_not_name_still_matches_on_its_queues(monkeypatch):
+    """The fallback. The multi-service compose stack ran every worker as
+    `celery@<host>`, and a deployment that sets no node name must keep
+    working — the node name is an improvement, not a requirement."""
+    _stub_active_queues(monkeypatch, {
+        "celery@xyz": [{"name": q} for q in LANES_BY_NAME["worker"].queues],
+    })
+
+    assert wc.inspect_lanes_sync()["worker"].present is True
+
+
+def test_an_unknown_worker_is_still_ignored_rather_than_guessed_at(monkeypatch):
+    """A deployment slicing CELERY_QUEUES differently has no lane row to
+    control. It belongs to the roster, not here — and must not be attributed
+    to whichever lane happens to be first."""
+    _stub_active_queues(monkeypatch, {"celery@xyz": [{"name": "something-else"}]})
+
+    live = wc.inspect_lanes_sync()
+
+    assert all(not s.present for s in live.values())