fix: the lane sizer reads the live pool, so it stops growing a lane past its cap (4409)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 24s
CI and images / backend-lint-and-test (push) Successful in 31s
CI and images / integration (push) Successful in 2m16s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m39s
CI and images / smoke-web (push) Successful in 55s
CI and images / promote (push) Successful in 1s

It read celery's `max-concurrency` as the pool size, but prefork reports that
as the limit the pool booted with. pool_grow and pool_shrink never update it.
A lane booted at 1 therefore read 1 forever. The sweep sent `target - 1` on
every tick while work waited, and could never shrink, since 1 - 1 is 0. The
System tab showed Scheduler "6 / 1" at a cap of 2.

pool_size() counts `processes` and falls back to the limit only when there is
no process list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-24 18:21:11 -04:00
co-authored by Claude Opus 5.5
parent 7a09dc3cda
commit f127c50207
2 changed files with 58 additions and 5 deletions
+33
View File
@@ -671,3 +671,36 @@ def test_an_unknown_worker_is_still_ignored_rather_than_guessed_at(monkeypatch):
live = wc.inspect_lanes_sync()
assert all(not s.present for s in live.values())
# --- the pool size is the processes running, not the boot limit ---------------
#
# The operator's System tab, 2026-09-24: Scheduler "6 / 1" at a cap of 2. Celery
# reports `max-concurrency` as the limit the pool BOOTED with; pool_grow and
# pool_shrink never update it. The sweep read 1 forever, sent `target - 1` on
# every tick, and grew the lane past its cap with no way to shrink it back.
def test_pool_size_counts_the_live_processes_not_the_boot_limit():
"""After two pool_grow calls on a pool booted at 1."""
assert wc.pool_size({"max-concurrency": 1, "processes": [11, 12, 13]}) == 3
def test_pool_size_falls_back_to_the_limit_without_a_process_list():
assert wc.pool_size({"max-concurrency": 4}) == 4
assert wc.pool_size({}) is None
def test_a_grown_pool_reads_at_its_real_size(monkeypatch):
"""The read the sweep sizes from: a pool at its target must read AT its
target, or `target - current` never reaches zero and every tick grows it."""
_stub_active_queues(monkeypatch, {"scheduler@h": []})
import sys
insp = sys.modules["backend.app.celery_app"].celery.control.inspect
insp.stats = lambda self: {
"scheduler@h": {"pool": {"max-concurrency": 1, "processes": [1, 2]}},
}
live = wc.inspect_lanes_sync()
assert live["scheduler"].pool == 2