fix: a lane that is OFF was not attributable to itself (4295)

Operator, 2026-09-23: *"clean up the stale service_seen rows"*.

**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.

A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:

1. The lanes table reported the lane **not answering** — the signal for a
   crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
   rendered as a display name — shown "running" beside the real lane's row
   going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
   lane present. ML ships at cap 0, so a fresh install was permanently
   unhealthy and Swarm restarts an unhealthy task forever.

That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.

`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.

The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.

Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

alembic/versions/0106_prune_phantom_roster_rows.py

@@ -0,0 +1,70 @@
+"""service_seen — delete the roster rows the fixed code can no longer write.
+
+Operator, 2026-09-23: *"clean up the stale service_seen rows"*. They were not
+stale. They were PHANTOMS, written on purpose by code that identified a celery
+worker from the queues it was consuming.
+
+A lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
+with an empty list. The roster grouped on that empty set, wrote it under the
+key `celery:` and rendered `role_display_name(())` as the display name — a row
+called **`Worker ()`**, reported as running, beside the real lane's row going
+stale because nothing updated it any more.
+
+`worker_lanes.lane_for_node` fixes the cause: a worker is attributed by its
+NODE NAME, which survives having no consumers. Nothing will write `celery:`
+again.
+
+## Why a migration and not a retention sweep
+
+Lesson #4202: a guard that refuses to produce a bad value does not undo the
+bad value already stored. The row is the thing that has to change.
+
+And it must be deleted rather than aged out, because the roster deliberately
+NEVER forgets — *"anything that has run at least once stays listed, that is
+what lets a stopped one be noticed rather than simply vanishing"*. A row that
+merely goes quiet is exactly what the roster is for. Only a row that cannot
+correspond to anything real is safe to remove, and `celery:` is precisely
+that: the empty queue set, which no correctly-attributed worker can produce.
+
+## What is deliberately NOT deleted
+
+**Celery rows with a real but unmatched queue set.** A deployment slicing
+`CELERY_QUEUES` differently is supported and its rows are true. It is not this
+migration's business to decide that somebody else's worker is obsolete.
+
+**Agent rows, including a possible `agent:agent` from a build that omitted
+`agent_id`.** Nothing here can tell an abandoned agent id from a second agent
+that is currently down, and deleting a real one would hide a genuinely dead
+GPU agent — the one thing the roster exists to show. If such a row is present
+it needs a person to look at it, not a migration guessing.
+
+Revision ID: 0106
+Revises: 0105
+Create Date: 2026-09-23
+
+"""
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0106"
+down_revision: Union[str, None] = "0105"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # Exactly the one key the empty queue set produced. Matched literally
+    # rather than by a LIKE or a prefix: `celery:` with nothing after it is
+    # the phantom, and `celery:ml` is a real lane.
+    op.execute(
+        sa.text("DELETE FROM service_seen WHERE key = :key").bindparams(key="celery:")
+    )
+
+
+def downgrade() -> None:
+    # Nothing. The row carried no information — an empty queue set and a
+    # timestamp — and the roster re-learns anything real on its next refresh.
+    # Re-creating it would put a phantom back.
+    pass