bvandeusenandClaude Opus 5
48108a3569
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 36s
CI and images / integration (push) Successful in 2m26s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
Operator, 2026-09-23: *"clean up the stale service_seen rows"*.
**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.
A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:
1. The lanes table reported the lane **not answering** — the signal for a
crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
rendered as a display name — shown "running" beside the real lane's row
going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
lane present. ML ships at cap 0, so a fresh install was permanently
unhealthy and Swarm restarts an unhealthy task forever.
That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.
`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.
The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.
Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
71 lines
2.9 KiB
Python
71 lines
2.9 KiB
Python
"""service_seen — delete the roster rows the fixed code can no longer write.
|
|
|
|
Operator, 2026-09-23: *"clean up the stale service_seen rows"*. They were not
|
|
stale. They were PHANTOMS, written on purpose by code that identified a celery
|
|
worker from the queues it was consuming.
|
|
|
|
A lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
|
|
with an empty list. The roster grouped on that empty set, wrote it under the
|
|
key `celery:` and rendered `role_display_name(())` as the display name — a row
|
|
called **`Worker ()`**, reported as running, beside the real lane's row going
|
|
stale because nothing updated it any more.
|
|
|
|
`worker_lanes.lane_for_node` fixes the cause: a worker is attributed by its
|
|
NODE NAME, which survives having no consumers. Nothing will write `celery:`
|
|
again.
|
|
|
|
## Why a migration and not a retention sweep
|
|
|
|
Lesson #4202: a guard that refuses to produce a bad value does not undo the
|
|
bad value already stored. The row is the thing that has to change.
|
|
|
|
And it must be deleted rather than aged out, because the roster deliberately
|
|
NEVER forgets — *"anything that has run at least once stays listed, that is
|
|
what lets a stopped one be noticed rather than simply vanishing"*. A row that
|
|
merely goes quiet is exactly what the roster is for. Only a row that cannot
|
|
correspond to anything real is safe to remove, and `celery:` is precisely
|
|
that: the empty queue set, which no correctly-attributed worker can produce.
|
|
|
|
## What is deliberately NOT deleted
|
|
|
|
**Celery rows with a real but unmatched queue set.** A deployment slicing
|
|
`CELERY_QUEUES` differently is supported and its rows are true. It is not this
|
|
migration's business to decide that somebody else's worker is obsolete.
|
|
|
|
**Agent rows, including a possible `agent:agent` from a build that omitted
|
|
`agent_id`.** Nothing here can tell an abandoned agent id from a second agent
|
|
that is currently down, and deleting a real one would hide a genuinely dead
|
|
GPU agent — the one thing the roster exists to show. If such a row is present
|
|
it needs a person to look at it, not a migration guessing.
|
|
|
|
Revision ID: 0106
|
|
Revises: 0105
|
|
Create Date: 2026-09-23
|
|
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0106"
|
|
down_revision: Union[str, None] = "0105"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
# Exactly the one key the empty queue set produced. Matched literally
|
|
# rather than by a LIKE or a prefix: `celery:` with nothing after it is
|
|
# the phantom, and `celery:ml` is a real lane.
|
|
op.execute(
|
|
sa.text("DELETE FROM service_seen WHERE key = :key").bindparams(key="celery:")
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
# Nothing. The row carried no information — an empty queue set and a
|
|
# timestamp — and the roster re-learns anything real on its next refresh.
|
|
# Re-creating it would put a phantom back.
|
|
pass
|