fix: a lane at zero slots tried to empty a pool billiard will not empty (4295)
CI / lint (push) Successful in 4s
CI / extension-version (push) Successful in 4s
Build images / sign-extension (push) Successful in 5s
Build images / build-agent (push) Successful in 7s
CI / frontend-build (push) Successful in 24s
CI / backend-lint-and-test (push) Failing after 40s
Build images / build-web (push) Successful in 2m1s
CI / integration (push) Successful in 2m11s
Build images / smoke-web (push) Successful in 1m0s
Build images / promote (push) Skipped

Found on the operator's live deploy, not in CI:

    [scheduler] worker_control: ml reconciled 1 -> 0 slots
    [ml] pidbox command error:
         ValueError("Can't shrink pool. All processes busy!")

ML ships at 0 stored slots and disabled, and `gen_supervisord` starts every
lane at one process so `add_consumer` has something to reach. So the stored
value and the running pool disagreed by one, permanently: billiard will not
remove the last worker, and `set_lane_slots_sync` returns True on SENDING the
control message — the refusal happens later, on the worker. The reconcile
logged a successful correction and reported `changed: ['ml']` every tick,
forever, on the default configuration of every install.

Lesson #4183 in production: an enforcer whose target is unreachable re-does
its own work on every pass and says it worked.

The floor is now one PROCESS, in one place — `effective_slots()` — applied
wherever a target is COMPARED as well as wherever one is sent. Comparing
against the unclamped 0 sees a difference no control message can ever close,
which is the same non-convergence one layer up.

Zero slots still means zero WORK: the lane's consumers are cancelled, and the
idle process is what the enable switch lands on.

A cross-file guard ties the generator's starting concurrency to the same
function, so the two ends of the floor cannot drift apart again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-23 10:58:58 -04:00
co-authored by Claude Opus 5
parent a987ca41ca
commit 86d6509936
3 changed files with 145 additions and 2 deletions
+31
View File
@@ -270,3 +270,34 @@ def test_supervisorctl_can_reach_supervisord():
assert talking == f"unix://{listening}", (
f"supervisorctl talks to {talking}, supervisord listens on {listening}"
)
def test_each_program_starts_at_the_smallest_pool_the_control_path_allows():
"""The two ends of the same floor, asserted together.
`gen_supervisord` starts every lane at `max(1, default_slots)` because
billiard will not run a pool of zero. `worker_control` has the same floor
for the opposite reason: it cannot SHRINK to zero either —
[ml] pidbox command error:
ValueError("Can't shrink pool. All processes busy!")
Live, 2026-09-23. ML starts at one process and stores zero, so the
reconcile tried 1 -> 0 on every tick, billiard refused, and
`set_lane_slots_sync` — which returns True on SENDING the message —
reported the lane changed forever (lesson #4183, on the default
configuration of every install).
Two constants, in two files, that must agree or the container cannot
settle. Asserted through `effective_slots` rather than against a literal
1, so raising the floor moves both ends at once.
"""
from backend.app.services.worker_control import effective_slots
cp = _parse()
for lane in LANES:
env = cp.get(f"program:{lane.name}", "environment")
want = effective_slots(lane.default_slots)
assert f"CELERY_CONCURRENCY={want}," in env, (
f"{lane.name} starts at a size the control path cannot reach"
)