fix: each lane needs its own celery node name, or three of four vanish (4295)
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 2m6s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
Build images / build-web (push) Successful in 1m52s
Build images / smoke-web (push) Successful in 56s
Build images / promote (push) Skipped
CI / lint (push) Successful in 3s
CI / extension-version (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 31s
CI / integration (push) Successful in 2m6s
Build images / sign-extension (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
Build images / build-web (push) Successful in 1m52s
Build images / smoke-web (push) Successful in 56s
Build images / promote (push) Skipped
Found by the all-role smoke on its very first execution (run 7319), which is
the whole argument for having added it one commit ago.
Celery's default node name is `celery@<hostname>`. In the single-container
layout all four lanes share one hostname, so all four registered as the SAME
node. Celery says so itself:
DuplicateNodenameWarning: Received multiple replies from node name:
celery@72adc5b706a7
`inspect` collapses four replies into one dict key and the last one wins, so
three lanes read as absent — and WHICH three varies between calls:
lanes not answering: maintenance_long, ml, worker
lanes not answering: maintenance_long, scheduler, worker
Fatal twice over:
* The composite healthcheck can never pass. In Swarm that is a container
that never goes healthy — restart loop, then an automatic rollback of a
deploy whose image was fine.
* `pool_grow`/`pool_shrink` take a `destination` of node names. The UI dial
and the autoscaler would have resized whichever lane happened to answer
rather than the one asked for — silently, and differently each time.
Every celery role now starts with `-n "${CELERY_NODENAME:-celery}@%h"`, and
the generated supervisord config sets that per lane. The lanes become
worker@<cid>, scheduler@<cid>, maintenance_long@<cid>, ml@<cid> — distinct,
so inspect keeps four entries and `destination` addresses what it names.
`inspect_lanes_sync` maps hostname to lane by QUEUES, so nothing there
changes; it just stops having three of its four entries overwritten.
Unset, it falls back to `celery` — exactly celery's own default — so every
service in the multi-service stack is byte-identical to before, including the
`celery@$HOSTNAME` healthcheck in docker-compose.yml and in the operator's
Swarm stack.
The test asserts DISTINCTNESS across the whole lane table rather than a fixed
string per lane. The property that broke is that no two collide, and stating
it that way keeps holding when a lane is added.
This is the bug I said a live deploy was needed to find, found in CI instead
for the price of one `docker run` — and it would have met the operator as a
rollback loop on their first consolidated deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -184,3 +184,50 @@ def test_every_program_writes_to_the_container_stdout_with_its_lane_named():
|
||||
assert cp.getint(section, "stdout_logfile_maxbytes") == 0, section
|
||||
assert cp.get(section, "redirect_stderr") == "true", section
|
||||
assert f"[{name}] " in cp.get(section, "command"), section
|
||||
|
||||
|
||||
def test_every_lane_gets_its_own_celery_node_name():
|
||||
"""The bug that made the single-container layout unusable, found the first
|
||||
time CI actually booted it (run 7319).
|
||||
|
||||
Celery's default node name is `celery@<hostname>`, and these processes
|
||||
share one hostname — so all four lanes registered as the SAME node:
|
||||
|
||||
DuplicateNodenameWarning: Received multiple replies from node name:
|
||||
celery@72adc5b706a7
|
||||
|
||||
`inspect` then collapses four replies into one dict key, last one wins.
|
||||
Three lanes read as absent, and WHICH three varied between calls:
|
||||
|
||||
lanes not answering: maintenance_long, ml, worker
|
||||
lanes not answering: maintenance_long, scheduler, worker
|
||||
|
||||
Fatal twice: the composite healthcheck can never pass, so the container is
|
||||
permanently unhealthy and Swarm restart-loops it; and `pool_grow` takes a
|
||||
`destination` of node names, so the UI dial and the autoscaler would have
|
||||
resized whichever lane happened to answer rather than the one asked for.
|
||||
|
||||
Asserted as DISTINCTNESS across the whole table rather than as a fixed
|
||||
string per lane — the property is that no two lanes collide, which is what
|
||||
actually broke, and it keeps holding when a lane is added.
|
||||
"""
|
||||
cp = _parse()
|
||||
names = {}
|
||||
for section in cp.sections():
|
||||
if not section.startswith("program:"):
|
||||
continue
|
||||
lane = section.split(":", 1)[1]
|
||||
if lane == "web":
|
||||
continue # hypercorn, not a celery node
|
||||
env = cp.get(section, "environment")
|
||||
pairs = dict(
|
||||
p.split("=", 1) for p in env.split(",") if "=" in p and not p.startswith('"')
|
||||
)
|
||||
assert "CELERY_NODENAME" in pairs, f"{lane} has no CELERY_NODENAME: {env}"
|
||||
names[lane] = pairs["CELERY_NODENAME"]
|
||||
|
||||
assert len(set(names.values())) == len(names), (
|
||||
f"two lanes share a celery node name, which is the collapse itself: {names}"
|
||||
)
|
||||
for lane, node in names.items():
|
||||
assert node == lane, f"{lane} announces itself as {node!r}"
|
||||
|
||||
Reference in New Issue
Block a user