fix: a lane that is OFF was not attributable to itself (4295)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 36s
CI and images / integration (push) Successful in 2m26s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 36s
CI and images / integration (push) Successful in 2m26s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 56s
CI and images / promote (push) Skipped
Operator, 2026-09-23: *"clean up the stale service_seen rows"*.
**They were not stale.** They were phantoms, written on purpose, and they will
come back on every install that turns a lane off — so the rows are the smaller
half of this.
A celery worker was attributed to its lane by the queues it was CONSUMING. A
lane at cap 0 has its consumers cancelled, so it answers `active_queues()`
with an empty list, matches no lane, and is dropped. Three consequences, all
on the operator's screen at once:
1. The lanes table reported the lane **not answering** — the signal for a
crashed worker, not for one the operator turned off.
2. The roster grew a phantom row named **`Worker ()`** — the empty queue set
rendered as a display name — shown "running" beside the real lane's row
going stale, because nothing updated it any more.
3. **The container went unhealthy.** `healthcheck._lanes_ok` requires every
lane present. ML ships at cap 0, so a fresh install was permanently
unhealthy and Swarm restarts an unhealthy task forever.
That third one is the severe one, and its docstring asserted the opposite of
what the code did — *"a disabled lane still runs its process with its
consumers cancelled, so it answers inspect and is healthy"*. It answers. It
was not attributed. A comment can be right about the intent and wrong about
the program, and this one had been wrong since the consolidated container
shipped.
`worker_lanes.lane_for_node` attributes by NODE NAME instead: identity travels
with the process rather than with what it happens to be doing.
`gen_supervisord` already sets `CELERY_NODENAME={lane.name}` per program — the
information was there and nothing read it. Falls back to the queue set for a
deployment that names no node, and `docker-compose.yml` now sets one per
service so the multi-service stack gets it too.
The roster keys on the LANE's queue set when the node resolves, which is the
same string the row already had while it was consuming — so an existing row
keeps updating rather than a second one appearing.
Migration 0106 deletes the one key the bug produced, `celery:`. Deliberately
NOT a retention sweep: the roster never forgets on purpose, so a quiet row is
what it is FOR, and only a row that cannot correspond to anything real is safe
to remove. An `agent:agent` row, if one exists, is left alone — nothing here
can tell an abandoned agent id from a second agent that is genuinely down, and
hiding a dead GPU agent is the one thing the roster must not do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -180,6 +180,10 @@ services:
|
||||
environment:
|
||||
<<: *app_env
|
||||
CELERY_QUEUES: default,import,thumbnail,download
|
||||
# Names the celery node for the roster. A lane whose consumers are
|
||||
# cancelled reports no queues, so the NODE is the only thing left that
|
||||
# identifies it — see services/worker_lanes.lane_for_node.
|
||||
CELERY_NODENAME: worker
|
||||
CELERY_CONCURRENCY: "2"
|
||||
# /downloads dropped — nothing in the app references it (operator-flagged
|
||||
# 2026-06-07: it wasn't mapped in prod and everything worked).
|
||||
@@ -200,6 +204,10 @@ services:
|
||||
environment:
|
||||
<<: *app_env
|
||||
CELERY_QUEUES: maintenance,scan
|
||||
# Names the celery node for the roster. A lane whose consumers are
|
||||
# cancelled reports no queues, so the NODE is the only thing left that
|
||||
# identifies it — see services/worker_lanes.lane_for_node.
|
||||
CELERY_NODENAME: scheduler
|
||||
volumes:
|
||||
- ./images:/images
|
||||
- ./import:/import
|
||||
@@ -223,6 +231,10 @@ services:
|
||||
environment:
|
||||
<<: *app_env
|
||||
CELERY_QUEUES: maintenance_long
|
||||
# Names the celery node for the roster. A lane whose consumers are
|
||||
# cancelled reports no queues, so the NODE is the only thing left that
|
||||
# identifies it — see services/worker_lanes.lane_for_node.
|
||||
CELERY_NODENAME: maintenance_long
|
||||
CELERY_CONCURRENCY: "1"
|
||||
# Only /images: backups write to /images/_backups, audits read /images, and
|
||||
# the admin tasks (re-extract/cascade-delete/normalize) operate on /images.
|
||||
@@ -241,6 +253,9 @@ services:
|
||||
deploy: *deploy_policy
|
||||
environment:
|
||||
<<: *app_env
|
||||
# See the worker service — the node name is what identifies a lane
|
||||
# whose consumers are cancelled.
|
||||
CELERY_NODENAME: ml
|
||||
volumes:
|
||||
- ./images:/images:ro
|
||||
- ./models:/models
|
||||
|
||||
Reference in New Issue
Block a user