fix: a lane at zero slots tried to empty a pool billiard will not empty (4295)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Failing after 40s
Build images / build-web (push) Successful in 2m1s
CI / integration (push) Successful in 2m11s
Build images / smoke-web (push) Successful in 1m0s
Build images / promote (push) Skipped
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Failing after 40s
Build images / build-web (push) Successful in 2m1s
CI / integration (push) Successful in 2m11s
Build images / smoke-web (push) Successful in 1m0s
Build images / promote (push) Skipped
Found on the operator's live deploy, not in CI:
[scheduler] worker_control: ml reconciled 1 -> 0 slots
[ml] pidbox command error:
ValueError("Can't shrink pool. All processes busy!")
ML ships at 0 stored slots and disabled, and `gen_supervisord` starts every
lane at one process so `add_consumer` has something to reach. So the stored
value and the running pool disagreed by one, permanently: billiard will not
remove the last worker, and `set_lane_slots_sync` returns True on SENDING the
control message — the refusal happens later, on the worker. The reconcile
logged a successful correction and reported `changed: ['ml']` every tick,
forever, on the default configuration of every install.
Lesson #4183 in production: an enforcer whose target is unreachable re-does
its own work on every pass and says it worked.
The floor is now one PROCESS, in one place — `effective_slots()` — applied
wherever a target is COMPARED as well as wherever one is sent. Comparing
against the unclamped 0 sees a difference no control message can ever close,
which is the same non-convergence one layer up.
Zero slots still means zero WORK: the lane's consumers are cancelled, and the
idle process is what the enable switch lands on.
A cross-file guard ties the generator's starting concurrency to the same
function, so the two ends of the floor cannot drift apart again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -270,3 +270,34 @@ def test_supervisorctl_can_reach_supervisord():
|
||||
assert talking == f"unix://{listening}", (
|
||||
f"supervisorctl talks to {talking}, supervisord listens on {listening}"
|
||||
)
|
||||
|
||||
|
||||
def test_each_program_starts_at_the_smallest_pool_the_control_path_allows():
|
||||
"""The two ends of the same floor, asserted together.
|
||||
|
||||
`gen_supervisord` starts every lane at `max(1, default_slots)` because
|
||||
billiard will not run a pool of zero. `worker_control` has the same floor
|
||||
for the opposite reason: it cannot SHRINK to zero either —
|
||||
|
||||
[ml] pidbox command error:
|
||||
ValueError("Can't shrink pool. All processes busy!")
|
||||
|
||||
Live, 2026-09-23. ML starts at one process and stores zero, so the
|
||||
reconcile tried 1 -> 0 on every tick, billiard refused, and
|
||||
`set_lane_slots_sync` — which returns True on SENDING the message —
|
||||
reported the lane changed forever (lesson #4183, on the default
|
||||
configuration of every install).
|
||||
|
||||
Two constants, in two files, that must agree or the container cannot
|
||||
settle. Asserted through `effective_slots` rather than against a literal
|
||||
1, so raising the floor moves both ends at once.
|
||||
"""
|
||||
from backend.app.services.worker_control import effective_slots
|
||||
|
||||
cp = _parse()
|
||||
for lane in LANES:
|
||||
env = cp.get(f"program:{lane.name}", "environment")
|
||||
want = effective_slots(lane.default_slots)
|
||||
assert f"CELERY_CONCURRENCY={want}," in env, (
|
||||
f"{lane.name} starts at a size the control path cannot reach"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user