feat: one number per lane — the cap — and the autoscaler is the mechanism (4295)
CI and images / lint (push) Successful in 4s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 24s
CI and images / integration (push) Failing after 24s
CI and images / backend-lint-and-test (push) Failing after 34s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped

Operator, 2026-09-23: *"auto should be always on, not a setting, so that idle
instances quiet down when not running. the number that is visible and
something the user can tweak and manage should be the cap itself the number of
running workers is handled by the autoscaling function which is always on."*

They are right, and the reason it was not built this way is worth stating: the
manual dial came first (steps 2-4) and the autoscaler came last (step 7), as
an opt-in BESIDE a control that already existed. Nothing ever asked whether
the dial should still exist once something could move it automatically. Each
step was defensible; the result was three operator settings over one number.

## `slots`, `enabled` and `autoscale` are gone

`slots` was a MEASUREMENT wearing a preference's clothes. How many workers a
lane runs is read live and moved every minute; storing it meant the operator
had to keep two numbers in agreement and the autoscaler had to be told it was
allowed to touch one of them.

`autoscale` gated the mechanism behind a choice, so a lane nobody opted in
never gave its workers back — which is why an idle instance never quieted
down.

`enabled` is derived: a cap of zero means no consumers. "Off" and "may use no
workers" were two spellings of one fact, stored separately, free to disagree.

## Two sweeps become one

`reconcile_lanes_sync` drove the pool to the stored `slots`; `autoscale_lanes_
sync` moved it away from that same number; and most of step 7's hardest
reasoning — a stored value that is a FLOOR, a target of `max(stored, current)`
— existed only to stop them fighting. Delete the stored number and the problem
is not solved, it is absent.

`size_lanes_sync` runs every minute and owns both consumers and pool size. It
also subsumes what the reconcile was for: a worker restarted at its ENV
concurrency is corrected on the next tick rather than after five.

Growth is immediate, shrink is one worker per tick. Deliberately asymmetric —
"always on" is only pleasant if the ramp keeps up, and +1/minute would take
four minutes to answer a burst. Being one worker too large for a minute costs
a sleeping process; being too small costs work not happening. For ML the
asymmetry matters most: every new slot reloads a multi-GB model, so the slow
shrink is what stops a quiet patch from paying that cost again a minute later.

## The caps ship at one, and zero for ML

Per the operator. Conservative on purpose — and a conservative default nobody
knows how to raise is just a slow product, which is the other half of what
they asked for:

    "there needs to be something that tells the user to bump those numbers to
     improve processing rate or they'd never know the controls exist."

So a lane running everything its cap allows while work piles up says so, in
its own row, with the headroom named: *"4,060 waiting and all 1 worker busy.
Raise the cap to run more at once — this machine allows up to 7."*

It fires only when raising the cap would actually help. Not when the lane is
keeping up, not when the sizing pass has room it has not taken, and not at the
machine ceiling — where "raise the cap" is advice nobody can take.

## Migration 0105 rewrites the caps rather than carrying them

The old defaults (4/2/2/1) bounded a manual control and were loose because
moving within them was the ordinary act. The number now means "the most
workers this lane may use", which is a different promise; carrying the old
figure over would quadruple the worker lane on every existing install at the
moment this deploys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
2026-09-23 12:48:22 -04:00
co-authored by Claude Opus 5
parent abe16aa382
commit 445164c852
16 changed files with 1110 additions and 1271 deletions
+120 -171
View File
@@ -47,237 +47,186 @@ async def test_get_lists_every_lane_with_its_ceiling(client, no_live_workers):
assert set(by_name) == {"worker", "scheduler", "maintenance_long", "ml"}
for lane in body["lanes"]:
assert lane["ceiling"] >= 0
assert lane["slots"] <= lane["slots_cap"]
assert lane["slots_cap"] <= lane["ceiling"]
# Nothing is running, so live state must say so rather than report
# zeroes that read like a healthy idle lane.
assert lane["live"]["present"] is False
@pytest.mark.asyncio
async def test_enabled_is_derived_from_the_cap_and_never_stored(
client, no_live_workers,
):
"""The reshape of 2026-09-23. "Off" and "may use no workers" were two
spellings of one fact, stored separately and free to disagree."""
body = await (await client.get("/api/system/workers")).get_json()
for lane in body["lanes"]:
assert lane["enabled"] == (lane["slots_cap"] > 0), lane["name"]
@pytest.mark.asyncio
async def test_ml_ships_off(client, no_live_workers):
"""Rule 164's carve-out and the weak-hardware default in one row: enabling
the lane is what triggers the SigLIP download, so a fresh install must not
find it on."""
"""Rule 164's carve-out and the weak-hardware default in one row: raising
the cap is what triggers the SigLIP download, so a fresh install must not
find it above zero."""
body = await (await client.get("/api/system/workers")).get_json()
ml = next(lane for lane in body["lanes"] if lane["name"] == "ml")
assert ml["slots_cap"] == 0
assert ml["enabled"] is False
assert ml["slots"] == 0
@pytest.mark.asyncio
async def test_the_other_lanes_ship_at_one(client, no_live_workers):
"""Operator: *"the cap defaults should be 1 and 0 for the ml-worker."*"""
body = await (await client.get("/api/system/workers")).get_json()
caps = {lane["name"]: lane["slots_cap"] for lane in body["lanes"]}
assert caps == {
"worker": 1, "scheduler": 1, "maintenance_long": 1, "ml": 0,
}
# --- the persist / push split ------------------------------------------------
@pytest.mark.asyncio
async def test_a_value_is_stored_even_when_it_cannot_be_pushed(
client, db, no_live_workers
async def test_a_cap_is_stored_even_when_it_cannot_be_pushed(
client, db, no_live_workers,
):
"""THE test for this step. `pool_grow` is not durable and the lane is not
answering, so the push fails — and the setting must survive anyway, or a
UI that saved while a worker was restarting would silently lose it
(lesson #4202). 200 with `applied: false`, not an error.
"""
resp = await client.post("/api/system/workers/worker", json={"slots": 3})
"""Nothing is answering, so the live push fails. That is NOT a failed
setting: the value is saved and the sizing pass carries it within a
minute (lesson #4202 — a live change that does not survive, with nothing
saying so)."""
resp = await client.post("/api/system/workers/worker", json={"slots_cap": 3})
assert resp.status_code == 200
body = await resp.get_json()
assert body["slots"] == 3
assert body["slots_cap"] == 3
assert body["applied"] is False
assert "not running" in body["apply_error"]
assert (await _lane_row(db, "worker")).slots == 3
assert (await _lane_row(db, "worker")).slots_cap == 3
# --- the cap is the switch ---------------------------------------------------
@pytest.mark.asyncio
async def test_a_partial_update_leaves_the_other_fields_alone(
client, db, no_live_workers
async def test_a_cap_of_zero_turns_the_lane_off(client, db, no_live_workers):
await client.post("/api/system/workers/worker", json={"slots_cap": 0})
row = await _lane_row(db, "worker")
assert row.slots_cap == 0
body = await (await client.get("/api/system/workers")).get_json()
worker = next(lane for lane in body["lanes"] if lane["name"] == "worker")
assert worker["enabled"] is False
@pytest.mark.asyncio
async def test_raising_it_off_zero_turns_the_lane_on(client, db, no_live_workers):
await client.post("/api/system/workers/ml", json={"slots_cap": 1})
assert (await _lane_row(db, "ml")).slots_cap == 1
body = await (await client.get("/api/system/workers")).get_json()
ml = next(lane for lane in body["lanes"] if lane["name"] == "ml")
assert ml["enabled"] is True
@pytest.mark.asyncio
async def test_the_model_fetch_fires_on_the_transition_not_on_every_write(
client, db, no_live_workers, monkeypatch,
):
"""The stepper sends `{"slots": n}` without restating a cap it did not
touch."""
before = await _lane_row(db, "worker")
original_cap = before.slots_cap
"""Raising the cap off zero downloads SigLIP, once. A second nudge of the
same dial must not re-enqueue a multi-GB download — and the trigger must
be the TRANSITION rather than "a field was sent", which is what it tested
before the UI stopped sending `enabled` at all."""
monkeypatch.setattr(
wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None),
)
monkeypatch.setattr(
wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None),
)
fired = []
monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True)
await client.post("/api/system/workers/worker", json={"slots": 2})
first = await (await client.post(
"/api/system/workers/ml", json={"slots_cap": 1},
)).get_json()
second = await (await client.post(
"/api/system/workers/ml", json={"slots_cap": 2},
)).get_json()
await db.refresh(before)
assert before.slots == 2
assert before.slots_cap == original_cap
assert first["fetching_models"] is True
assert second["fetching_models"] is False
assert fired == [1]
# --- what is refused ---------------------------------------------------------
@pytest.mark.asyncio
async def test_slots_above_the_cap_are_refused_and_nothing_is_stored(
client, db, no_live_workers
):
row = await _lane_row(db, "worker")
resp = await client.post(
"/api/system/workers/worker", json={"slots": row.slots_cap + 1},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "refused"
# The sentence the UI shows. A greyed control with no reason reads as a bug.
assert "cap" in body["detail"]
await db.refresh(row)
assert row.slots <= row.slots_cap
@pytest.mark.asyncio
async def test_a_cap_above_the_derived_ceiling_is_refused(
client, db, no_live_workers
client, db, no_live_workers,
):
"""The operator's cap is theirs to set, but not past what the container
can hold — the whole point of deriving a ceiling rather than typing one."""
"""The ceiling is the machine's, not the operator's, and it is the one
bound they cannot lower themselves past. The detail is written to be read
by a person — a refused control with no reason reads as a bug."""
before = (await _lane_row(db, "ml")).slots_cap
resp = await client.post(
"/api/system/workers/ml", json={"slots_cap": 10_000},
)
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "refused"
assert "container can hold" in body["detail"]
await _refreshed(db, "ml", before)
@pytest.mark.asyncio
async def test_a_boolean_is_not_accepted_as_a_slot_count(client, no_live_workers):
"""`True` is an int in Python. Coerced, it would silently set one slot —
a control that appears to work and does something nobody asked for."""
resp = await client.post("/api/system/workers/worker", json={"slots": True})
async def test_a_negative_cap_is_refused(client, db, no_live_workers):
resp = await client.post("/api/system/workers/worker", json={"slots_cap": -1})
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_body"
@pytest.mark.asyncio
async def test_a_boolean_is_not_accepted_as_a_cap(client, no_live_workers):
"""`True` is an int in Python. Reading it as a cap of 1 would be a control
that appears to work and sets something nobody asked for."""
resp = await client.post("/api/system/workers/worker", json={"slots_cap": True})
assert resp.status_code == 400
@pytest.mark.asyncio
async def test_the_retired_fields_are_no_longer_accepted(client, no_live_workers):
"""`slots`, `enabled` and `autoscale` are gone. A client still sending one
must be told, not silently ignored — a POST that returns 200 having
changed nothing is the worst of the three outcomes."""
for field in ("slots", "enabled", "autoscale"):
resp = await client.post(
"/api/system/workers/worker", json={field: 2},
)
assert resp.status_code == 400, field
@pytest.mark.asyncio
async def test_an_unknown_lane_is_refused_and_names_the_known_ones(
client, no_live_workers
client, no_live_workers,
):
resp = await client.post("/api/system/workers/nonsense", json={"slots": 1})
resp = await client.post("/api/system/workers/nope", json={"slots_cap": 1})
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "unknown_lane"
assert "worker" in body["known"]
@pytest.mark.asyncio
async def test_an_empty_body_is_refused_rather_than_treated_as_a_no_op(
client, no_live_workers
client, no_live_workers,
):
"""A POST that changes nothing and returns 200 is indistinguishable from
one that worked, which is how a broken UI control goes unnoticed."""
resp = await client.post("/api/system/workers/worker", json={})
assert resp.status_code == 400
body = await resp.get_json()
assert body["error"] == "invalid_body"
# --- the dial is the switch --------------------------------------------------
#
# Operator, 2026-09-23: *"almost all of it always needs to run there's only one
# optional piece and it is killed by moving the 'cap' to zero."* So `enabled`
# is derived from the number rather than being a second control the operator
# has to keep in agreement with it. It stays on the API — these assert that it
# still does, because it is the mechanism the reconcile and the healthcheck
# read.
@pytest.mark.asyncio
async def test_dialling_a_lane_to_zero_turns_it_off(client, db, no_live_workers):
await client.post("/api/system/workers/worker", json={"slots": 0})
row = await _lane_row(db, "worker")
assert row.slots == 0
assert row.enabled is False
@pytest.mark.asyncio
async def test_dialling_it_back_up_turns_it_on(client, db, no_live_workers):
await client.post("/api/system/workers/ml", json={"slots": 1})
row = await _lane_row(db, "ml")
assert row.slots == 1
assert row.enabled is True, "the lane the operator just asked for work from"
@pytest.mark.asyncio
async def test_an_explicit_enabled_still_wins(client, db, no_live_workers):
"""The field is not removed, only derived when absent. Something that
genuinely wants a lane holding its process with consumers cancelled — a
drain before a restart — must still be able to say so without having to
destroy the operator's slot count to express it."""
await client.post(
"/api/system/workers/worker", json={"slots": 3, "enabled": False},
)
row = await _lane_row(db, "worker")
assert (row.slots, row.enabled) == (3, False)
@pytest.mark.asyncio
async def test_a_cap_only_write_does_not_decide_the_switch(
client, db, no_live_workers,
):
"""Only the SLOTS dial derives it. A cap is a ceiling, not a request for
work, and letting it flip the lane would make raising a ceiling start
something."""
before = await _lane_row(db, "ml")
assert before.enabled is False
await client.post("/api/system/workers/ml", json={"slots_cap": 1})
await db.refresh(before)
assert (before.slots, before.enabled) == (0, False)
@pytest.mark.asyncio
async def test_the_model_fetch_fires_on_the_transition_not_on_the_field(
client, db, no_live_workers, monkeypatch,
):
"""The trap the derivation set, caught here rather than in production.
The fetch used to be conditioned on `enabled is True` — the FIELD having
been sent. The UI no longer sends it at all, so the download that makes
the ML lane usable would simply never have fired, and the lane would have
come on and sat there consuming a queue it had no model for.
"""
fired = []
monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True)
monkeypatch.setattr(
wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None),
)
monkeypatch.setattr(
wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None),
)
body = await (await client.post(
"/api/system/workers/ml", json={"slots": 1},
)).get_json()
assert body["fetching_models"] is True
assert fired == [1]
@pytest.mark.asyncio
async def test_it_does_not_fire_again_on_a_lane_already_running(
client, db, no_live_workers, monkeypatch,
):
"""The other half. A second nudge of the dial on a lane that is already on
must not re-enqueue a multi-GB download."""
monkeypatch.setattr(
wc, "set_lane_enabled_sync", lambda lane, enabled, live=None: (True, None),
)
monkeypatch.setattr(
wc, "set_lane_slots_sync", lambda lane, target, live=None: (True, None),
)
await client.post("/api/system/workers/ml", json={"slots": 1})
fired = []
monkeypatch.setattr(wc, "_enqueue_model_fetch", lambda: fired.append(1) or True)
body = await (await client.post(
"/api/system/workers/ml", json={"slots": 1},
)).get_json()
assert body["fetching_models"] is False
assert fired == []
async def _refreshed(db, name: str, expected: int) -> None:
row = await _lane_row(db, name)
await db.refresh(row)
assert row.slots_cap == expected, "a refused write must store nothing"
+17 -18
View File
@@ -54,13 +54,13 @@ def test_every_lane_gets_a_program():
assert programs == expected
def test_the_ml_lane_runs_even_though_it_ships_disabled():
def test_the_ml_lane_runs_even_though_it_ships_off():
"""It holds a PROCESS and no model. `add_consumer` needs a running worker
to reach, so without this the UI switch would have nothing to switch —
to reach, so without this raising the cap would have nothing to reach —
and nothing is downloaded by starting it, which is what lets rule 164
permit the fetch at all."""
assert _parse().has_section("program:ml")
assert LANES_BY_NAME["ml"].default_enabled is False
assert LANES_BY_NAME["ml"].default_slots_cap == 0
# --- the coupling this generator exists to guarantee -------------------------
@@ -152,12 +152,12 @@ def test_no_program_waits_longer_than_the_compose_stop_grace_period():
# --- what runs, and how much ------------------------------------------------
def test_a_zero_slot_lane_still_gets_a_running_process():
"""ML ships at 0 slots and disabled — but `add_consumer` needs something to
reach. With no process there would be nothing for the UI switch to switch,
and enabling tagging could not work at all."""
def test_a_lane_capped_at_zero_still_gets_a_running_process():
"""ML ships at cap 0 — but `add_consumer` needs something to reach. With
no process there would be nothing for the cap to switch back on, and
enabling tagging could not work at all."""
cp = _parse()
assert LANES_BY_NAME["ml"].default_slots == 0
assert LANES_BY_NAME["ml"].default_slots_cap == 0
env = cp.get("program:ml", "environment")
assert "CELERY_CONCURRENCY=1" in env
@@ -275,29 +275,28 @@ def test_supervisorctl_can_reach_supervisord():
def test_each_program_starts_at_the_smallest_pool_the_control_path_allows():
"""The two ends of the same floor, asserted together.
`gen_supervisord` starts every lane at `max(1, default_slots)` because
billiard will not run a pool of zero. `worker_control` has the same floor
for the opposite reason: it cannot SHRINK to zero either —
`gen_supervisord` starts every lane at one process because billiard will
not run a pool of zero. The sizing pass has the same floor for the
opposite reason: it cannot SHRINK to zero either —
[ml] pidbox command error:
ValueError("Can't shrink pool. All processes busy!")
Live, 2026-09-23. ML starts at one process and stores zero, so the
Live, 2026-09-23. ML started at one process and stored zero, so the
reconcile tried 1 -> 0 on every tick, billiard refused, and
`set_lane_slots_sync` — which returns True on SENDING the message —
reported the lane changed forever (lesson #4183, on the default
configuration of every install).
Two constants, in two files, that must agree or the container cannot
settle. Asserted through `effective_slots` rather than against a literal
1, so raising the floor moves both ends at once.
One constant now, read from the same module by both, rather than two that
must agree. Asserted through it rather than against a literal 1, so
raising the floor moves both ends at once.
"""
from backend.app.services.worker_control import effective_slots
from backend.app.services.worker_lanes import MIN_POOL_SLOTS
cp = _parse()
for lane in LANES:
env = cp.get(f"program:{lane.name}", "environment")
want = effective_slots(lane.default_slots)
assert f"CELERY_CONCURRENCY={want}," in env, (
assert f"CELERY_CONCURRENCY={MIN_POOL_SLOTS}," in env, (
f"{lane.name} starts at a size the control path cannot reach"
)
+254 -410
View File
@@ -195,388 +195,6 @@ def test_an_unknown_queue_set_maps_to_no_lane():
assert wc._lane_for_queues(("something", "else")) is None
# --- the reconcile (step 3) --------------------------------------------------
def test_a_settled_system_sends_no_control_messages(monkeypatch):
"""THE property this sweep lives or dies on. It runs every 5 minutes
forever, so a converged tick must be silent — one broker round trip and
nothing else. An enforcer that re-issues a grow of zero churns forever and
buries a real correction in its own noise (lesson #4183).
"""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "worker", pools={"host-a": 4})
result = wc.reconcile_lanes_sync({"worker": (4, True)})
# EVERY control family, not just the pool ones. The first version of this
# test asserted only grew/shrank and would have passed while the reconcile
# re-sent add_consumer for all four queues on every tick — harmless per
# call, unbounded churn in aggregate, and invisible.
assert control.grew == []
assert control.shrank == []
assert control.added == []
assert control.cancelled == []
assert result["changed"] == []
def test_a_worker_back_at_its_env_concurrency_is_corrected(monkeypatch):
"""The failure this exists for: a restart drops the pool to CELERY_
CONCURRENCY, silently below what the operator set."""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "worker", pools={"host-a": 2})
result = wc.reconcile_lanes_sync({"worker": (6, True)})
assert control.grew == [(4, ["host-a"])]
assert result["changed"] == ["worker"]
def test_an_absent_lane_is_skipped_not_corrected(monkeypatch):
"""`present=False` is 'nothing answered', not 'zero slots'. Correcting it
would be a conclusion drawn from an unswept read (snippet #3969) — and
there is nothing to send the message to anyway."""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "worker", pools={}, present=False)
result = wc.reconcile_lanes_sync({"worker": (6, True)})
assert result["skipped"] == ["worker"]
assert result["changed"] == []
assert control.grew == []
assert control.shrank == []
def test_one_lane_failing_does_not_stop_the_others(monkeypatch):
"""A broker blip on one lane must not leave the rest un-reconciled for
another five minutes."""
control = _stub_control(monkeypatch)
present = wc.LaneLiveState(
present=True, replicas=1, hostnames=["host-a"], pools={"host-a": 1},
)
absent = wc.LaneLiveState()
monkeypatch.setattr(
wc, "inspect_lanes_sync",
lambda: {"worker": absent, "scheduler": present,
"maintenance_long": absent, "ml": absent},
)
result = wc.reconcile_lanes_sync({
"worker": (4, True), "scheduler": (3, True),
})
assert result["skipped"] == ["worker"]
assert result["changed"] == ["scheduler"]
assert control.grew == [(2, ["host-a"])]
def test_a_lane_disabled_in_settings_but_still_consuming_is_stopped(monkeypatch):
"""The case that made `consuming` necessary: a lane turned off while its
worker was down comes back consuming, and must be stopped when it
returns.
Its live pool is ONE — zero slots means zero WORK, not an empty pool.
billiard will not run one, and the process is what the enable switch lands
on.
"""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "ml", pools={"host-a": 1}, consuming={"ml"})
result = wc.reconcile_lanes_sync({"ml": (0, False)})
assert control.cancelled == [("ml", ["host-a"])]
assert result["changed"] == ["ml"]
def test_an_already_disabled_lane_is_not_cancelled_again(monkeypatch):
"""The other half of the fixed point. `cancel_consumer` on a queue that is
not being consumed succeeds and does nothing, so an unconditional
reconcile would churn here forever with no symptom.
The live pool is ONE, not zero. This fixture said zero until the floor
landed, and it was describing a container that cannot exist — billiard
will not run an empty pool, and the generator starts ml at one process for
exactly that reason. A fixture in an impossible state passes for the wrong
reason and then fails on the correct fix, which is what it did.
"""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "ml", pools={"host-a": 1}, consuming=set())
result = wc.reconcile_lanes_sync({"ml": (0, False)})
assert control.cancelled == []
assert control.added == []
assert result["changed"] == []
def test_a_lane_with_no_stored_row_is_left_alone(monkeypatch):
"""A lane in LANES but not in `desired` is one whose row has not been
seeded. Inventing a target here would let the sweep enforce a number that
disagrees with the migration it is supposed to be upholding."""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "worker", pools={"host-a": 2})
result = wc.reconcile_lanes_sync({})
assert result["changed"] == []
assert control.grew == []
def test_the_reconcile_task_is_registered_and_scheduled():
"""A task name only enters `celery.tasks` when its module is imported, and
a beat entry naming a task that is not registered fails at tick time
rather than at import — silently, every five minutes."""
import backend.app.tasks.maintenance # noqa: F401
from backend.app.celery_app import celery
name = "backend.app.tasks.maintenance.reconcile_worker_lanes"
assert name in celery.tasks
scheduled = {e["task"] for e in celery.conf.beat_schedule.values()}
assert name in scheduled
# --- the autoscaler (step 7) --------------------------------------------------
#
# Every case below fixes the LIVE pool and the stored floor to DIFFERENT
# numbers wherever it can. That is deliberate: the first version of this
# function read the stored value as the current one, and with the two equal
# — which is what a single tick of a settled system looks like — every test
# here still passed while the autoscaler could not grow past floor+1 or shrink
# at all. Equal fixtures cannot see that bug.
def _autoscale_live(
monkeypatch, *, pools, active, reserved, depth, present=True,
):
state = wc.LaneLiveState(
present=present,
replicas=len(pools),
hostnames=sorted(pools),
pools=dict(pools),
active=active,
reserved=reserved,
consuming=set(LANES_BY_NAME["worker"].queues),
)
monkeypatch.setattr(
wc, "inspect_lanes_sync",
lambda: {name: (state if name == "worker" else wc.LaneLiveState())
for name in LANES_BY_NAME},
)
monkeypatch.setattr(
wc, "_queue_depths_sync",
lambda: {q: depth for lane in LANES_BY_NAME.values() for q in lane.queues},
)
return state
def _worker(decisions):
return next(d for d in decisions if d.lane == "worker")
def test_a_prefetched_backlog_still_triggers_growth(monkeypatch):
"""THE case an LLEN-only implementation misses, and the reason `reserved`
was plumbed through in step 2. Celery prefetches, so a lane can read queue
depth 0 while holding thirty tasks in worker memory — an autoscaler
watching LLEN alone sees an idle system and never grows."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=30, depth=0)
d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)}))
assert d.action == "grew"
assert d.slots == 3
assert control.grew == [(1, ["host-a"])]
def test_it_keeps_climbing_past_the_floor_on_later_ticks(monkeypatch):
"""The regression test for reading the stored value as the current one.
The row still says 2 — the autoscaler never writes it — but the live pool
is already 5 from earlier ticks. The target must be 6. Computing from the
stored 2 would propose 3, which resizes nothing (the replica is past it),
reports success, and pins the lane one slot above its floor forever while
claiming to grow on every tick."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 5}, active=5, reserved=40, depth=0)
d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)}))
assert d.slots == 6
assert control.grew == [(1, ["host-a"])]
def test_backlog_with_free_slots_does_nothing(monkeypatch):
"""Depth alone is not a signal — celery is about to pick those up, and
growing would add children that idle."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 8}, active=1, reserved=0, depth=50)
assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held"
assert control.grew == []
def test_saturation_is_measured_against_every_replica(monkeypatch):
"""`active` is summed across replicas and `pool` is one replica's size, so
comparing them calls two half-busy replicas of 4 saturated at 4 active.
Capacity is 8 here and 4 tasks are running: half the lane is idle."""
control = _stub_control(monkeypatch)
_autoscale_live(
monkeypatch, pools={"host-a": 4, "host-b": 4},
active=4, reserved=40, depth=0,
)
assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held"
assert control.grew == []
def test_a_saturated_lane_with_no_backlog_does_nothing(monkeypatch):
"""The long-running-task case. One slow task holding every slot with an
empty queue needs no extra slots — growing does not make it finish sooner,
which is why the operator's 'runs for x time' idea became a UI warning
rather than a trigger."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=0, depth=0)
assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held"
assert control.grew == []
def test_growth_stops_at_the_cap_and_says_so(monkeypatch):
"""The autoscaler gets no authority the operator does not already have.
And it SAYS it is capped — that is the moment they would want to know they
set one."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 4}, active=4, reserved=99, depth=0)
d = _worker(wc.autoscale_lanes_sync({"worker": (4, 2, True)}))
assert d.action == "held"
assert "cap is 4" in d.reason
assert control.grew == []
def test_a_cleared_backlog_returns_the_lane_to_the_operator_s_value(monkeypatch):
"""Down toward `configured`, one step at a time."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 5}, active=0, reserved=0, depth=0)
d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)}))
assert d.action == "shrank"
assert d.slots == 4
assert control.shrank == [(1, ["host-a"])]
def test_it_never_shrinks_below_what_the_operator_set(monkeypatch):
"""The stored value is the operator's and the autoscaler must not eat it —
there would be nothing left to restore to."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 2}, active=0, reserved=0, depth=0)
assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held"
assert control.shrank == []
def test_hysteresis_keeps_a_middling_backlog_from_flapping(monkeypatch):
"""Between the two thresholds nothing happens. Equal thresholds would grow
and shrink the lane forever as one task arrives and leaves — lesson
#4183's churn arriving through a different door."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 5}, active=5, reserved=5, depth=0)
assert _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)})).action == "held"
assert control.grew == []
assert control.shrank == []
def test_a_lane_that_did_not_opt_in_is_never_touched(monkeypatch):
"""Off by default, per lane. This is the only sweep that decides rather
than obeys, so it acts only where someone said it may — and it does not
even report on a lane it was not given."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=99, depth=0)
assert wc.autoscale_lanes_sync({"worker": (8, 2, False)}) == []
assert control.grew == []
def test_an_absent_lane_holds_rather_than_guessing(monkeypatch):
"""`present=False` is 'nothing answered', not 'idle'. Deciding from an
unswept read is snippet #3969's shape."""
control = _stub_control(monkeypatch)
_autoscale_live(
monkeypatch, pools={}, active=0, reserved=0, depth=0, present=False,
)
d = _worker(wc.autoscale_lanes_sync({"worker": (8, 2, True)}))
assert d.action == "held"
assert "not answering" in d.reason
assert control.grew == []
def test_a_settled_lane_sends_no_control_messages(monkeypatch):
"""The fixed point, asserted as a property rather than inferred from the
reported action: this runs every minute forever, so a settled system has
to be silent or a real correction drowns in the heartbeat."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 2}, active=1, reserved=0, depth=1)
for _ in range(5):
assert _worker(
wc.autoscale_lanes_sync({"worker": (8, 2, True)})
).action == "held"
assert control.grew == []
assert control.shrank == []
# --- the two sweeps must not fight -------------------------------------------
def test_the_reconcile_does_not_undo_what_the_autoscaler_added(monkeypatch):
"""Otherwise the two would fight every five minutes: grow, revert, grow,
revert. For an autoscaling lane the stored slots are a FLOOR."""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "worker", pools={"host-a": 6})
wc.reconcile_lanes_sync({"worker": (2, True)}, frozenset({"worker"}))
assert control.shrank == []
def test_the_reconcile_still_restores_an_autoscaling_lane_that_fell_below(
monkeypatch,
):
"""A floor is still a floor. A restart drops the lane to its env value and
this must bring it back to what the operator set."""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "worker", pools={"host-a": 1})
wc.reconcile_lanes_sync({"worker": (4, True)}, frozenset({"worker"}))
assert control.grew == [(3, ["host-a"])]
def test_a_non_autoscaling_lane_is_still_driven_down_to_its_value(monkeypatch):
"""The floor applies only to lanes that opted in — otherwise turning
autoscale off would leave the lane stuck at whatever it had grown to."""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "worker", pools={"host-a": 6})
wc.reconcile_lanes_sync({"worker": (2, True)}, frozenset())
assert control.shrank == [(4, ["host-a"])]
def test_the_autoscale_task_is_registered_and_scheduled():
import backend.app.tasks.maintenance # noqa: F401
from backend.app.celery_app import celery
name = "backend.app.tasks.maintenance.autoscale_worker_lanes"
assert name in celery.tasks
assert name in {e["task"] for e in celery.conf.beat_schedule.values()}
# --- the inspect round trips -------------------------------------------------
@@ -680,49 +298,275 @@ def test_the_round_trip_bound_matches_the_reads_actually_made():
)
# --- a pool cannot be emptied ------------------------------------------------
def test_a_lane_at_zero_slots_is_never_shrunk_to_an_empty_pool(monkeypatch):
"""The error in the operator's live log, 2026-09-23:
# --- the sizing pass ---------------------------------------------------------
#
# One sweep replaced two on 2026-09-23. The reconcile drove the pool to a
# stored `slots`; the autoscaler moved it away from that same number; and most
# of the autoscaler's design existed to stop the reconcile undoing its work.
# The stored number is gone, so there is nothing left to disagree about — and
# these tests no longer have to assert that two sweeps get along.
#
# The fixtures deliberately set the live pool and the cap to DIFFERENT numbers
# wherever the distinction matters. Lesson #4318: equal fixtures cannot tell
# "reads live" from "reads stored", and that is exactly how the old autoscaler
# shipped unable to do its job with every test green.
[scheduler] worker_control: ml reconciled 1 -> 0 slots
[ml] pidbox command error:
ValueError("Can't shrink pool. All processes busy!")
billiard refuses to remove the last worker, so a lane at zero stored slots
could never reach its target — and `set_lane_slots_sync` returns True on
SENDING the message, so the reconcile logged success and reported the lane
as `changed` on every tick, forever. Lesson #4183 in production, on the
default configuration of every install.
"""
def _sizing_live(
monkeypatch, *, pools, active, reserved, depth, consuming=None, present=True,
):
state = wc.LaneLiveState(
present=present,
replicas=len(pools),
hostnames=sorted(pools),
pools=dict(pools),
active=active,
reserved=reserved,
consuming=set(LANES_BY_NAME["worker"].queues if consuming is None else consuming),
)
monkeypatch.setattr(
wc, "inspect_lanes_sync",
lambda: {name: (state if name == "worker" else wc.LaneLiveState())
for name in LANES_BY_NAME},
)
monkeypatch.setattr(
wc, "_queue_depths_sync",
lambda: {q: depth for lane in LANES_BY_NAME.values() for q in lane.queues},
)
return state
def _worker(sized):
return next(d for d in sized if d.lane == "worker")
# --- what a lane has work for ------------------------------------------------
def test_work_in_flight_and_waiting_each_justify_a_worker():
# Two running plus six queued is eight tasks, so eight workers — if the
# cap allowed it.
assert wc.wanted_slots(cap=10, active=2, pending=6) == 8
def test_it_never_exceeds_the_cap():
assert wc.wanted_slots(cap=2, active=2, pending=4000) == 2
def test_an_idle_lane_falls_to_one_and_not_to_zero():
"""billiard will not run an empty pool, and the parked process is what
`add_consumer` lands on when the cap goes back up."""
assert wc.wanted_slots(cap=8, active=0, pending=0) == 1
def test_a_lane_capped_at_zero_still_keeps_its_process():
"""`cap 0` is expressed by cancelling CONSUMERS, not by emptying the pool.
A lane with no process reads as absent, which is the same signal as a
crash — and the roster exists to keep those two apart."""
assert wc.wanted_slots(cap=0, active=0, pending=0) == wc.MIN_POOL_SLOTS
def test_an_unknown_backlog_contributes_nothing_rather_than_zero():
"""The broker did not answer for these queues. Reading that as "empty"
would shrink a lane on the strength of a failed read (snippet #3969) —
and reading it as "huge" would grow one. It contributes nothing, and the
work actually in flight still counts."""
assert wc.wanted_slots(cap=8, active=3, pending=None) == 3
# --- growing -----------------------------------------------------------------
def test_a_backlog_is_met_in_one_tick_not_one_slot_per_minute(monkeypatch):
"""The operator's condition for always-on autoscaling: *"always on"* is
only pleasant if the ramp keeps up. Growing +1 per minute would take four
minutes to answer a burst, and the old autoscaler did exactly that."""
control = _stub_control(monkeypatch)
_stub_live(monkeypatch, "ml", pools={"host-a": 1}, consuming=set())
_sizing_live(monkeypatch, pools={"host-a": 1}, active=1, reserved=0, depth=4000)
wc.reconcile_lanes_sync({"ml": (0, False)})
d = _worker(wc.size_lanes_sync({"worker": 4}))
assert control.shrank == [], "tried to empty a pool billiard will not empty"
assert (d.action, d.slots) == ("grew", 4)
assert control.grew == [(3, ["host-a"])]
def test_the_floor_applies_to_a_direct_resize_too(monkeypatch):
"""Not only the reconcile — the UI dial and the autoscaler go through the
same function, so the clamp belongs there rather than at each caller."""
def test_a_prefetched_backlog_counts(monkeypatch):
"""THE case an LLEN-only implementation misses. Celery prefetches, so a
lane can read queue depth 0 while holding thirty tasks in worker memory —
a sizing pass watching LLEN alone sees an idle system and shrinks."""
control = _stub_control(monkeypatch)
state = _stub_live(monkeypatch, "worker", pools={"host-a": 3})
_sizing_live(monkeypatch, pools={"host-a": 1}, active=1, reserved=30, depth=0)
wc.set_lane_slots_sync(LANES_BY_NAME["worker"], 0, live=state)
assert control.shrank == [(2, ["host-a"])], "should stop at one, not zero"
assert _worker(wc.size_lanes_sync({"worker": 4})).action == "grew"
assert control.grew == [(3, ["host-a"])]
def test_the_autoscaler_will_not_shrink_into_an_empty_pool(monkeypatch):
"""A lane whose operator value is 0 and whose live pool is the floor has
nowhere to shrink to — and saying `shrank` every minute is the same
non-convergence wearing a different hat."""
def test_a_restarted_worker_is_pulled_back_up(monkeypatch):
"""What the five-minute reconcile existed for, now done in one minute by
the pass that was already running. `pool_grow` is not durable: a worker
restarted by its supervisor comes back at its ENV concurrency, silently
below what the lane should be running."""
control = _stub_control(monkeypatch)
_autoscale_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0)
_sizing_live(monkeypatch, pools={"host-a": 1}, active=1, reserved=9, depth=0)
d = _worker(wc.autoscale_lanes_sync({"worker": (8, 0, True)}))
assert _worker(wc.size_lanes_sync({"worker": 4})).slots == 4
assert control.grew == [(3, ["host-a"])]
# --- shrinking ---------------------------------------------------------------
def test_an_idle_lane_gives_a_worker_back_one_at_a_time(monkeypatch):
"""Operator: *"so that idle instances quiet down when not running."*
One per tick on the way down, against immediate growth. Being one worker
too large for a minute costs a sleeping process; being too small costs
work not happening. For ML it matters most — every new slot reloads a
multi-GB model, so a slow shrink is what stops a quiet patch from paying
that cost again a minute later."""
control = _stub_control(monkeypatch)
_sizing_live(monkeypatch, pools={"host-a": 4}, active=0, reserved=0, depth=0)
d = _worker(wc.size_lanes_sync({"worker": 4}))
assert (d.action, d.slots) == ("shrank", 3)
assert control.shrank == [(1, ["host-a"])]
def test_it_stops_shrinking_at_one(monkeypatch):
control = _stub_control(monkeypatch)
_sizing_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0)
assert _worker(wc.size_lanes_sync({"worker": 4})).action == "held"
assert control.shrank == []
def test_lowering_the_cap_pulls_a_lane_down(monkeypatch):
"""The cap is a ceiling on the live pool, not only on future growth."""
control = _stub_control(monkeypatch)
_sizing_live(monkeypatch, pools={"host-a": 4}, active=4, reserved=99, depth=0)
assert _worker(wc.size_lanes_sync({"worker": 2})).slots == 3
assert control.shrank == [(1, ["host-a"])]
# --- the fixed point ---------------------------------------------------------
def test_a_settled_lane_sends_no_control_messages(monkeypatch):
"""THE property this pass lives or dies on. It runs every minute forever,
so a converged tick must be silent — one inspect, one LLEN sweep, nothing
else. An enforcer that re-issues a grow of zero churns forever and buries
a real correction in its own noise (lesson #4183)."""
control = _stub_control(monkeypatch)
_sizing_live(monkeypatch, pools={"host-a": 2}, active=2, reserved=0, depth=0)
d = _worker(wc.size_lanes_sync({"worker": 4}))
assert d.action == "held"
# EVERY control family, not just the pool ones: an unconditional
# add_consumer for four queues per tick is harmless per call and unbounded
# in aggregate, and invisible.
assert control.grew == []
assert control.shrank == []
assert control.added == []
assert control.cancelled == []
def test_an_absent_lane_is_skipped_not_corrected(monkeypatch):
"""`present=False` means nothing answered — a worker restarting, or an
unreachable broker. It is NOT zero workers, and there is nothing to send
a message to."""
control = _stub_control(monkeypatch)
_sizing_live(
monkeypatch, pools={}, active=0, reserved=0, depth=0, present=False,
)
d = _worker(wc.size_lanes_sync({"worker": 4}))
assert d.action == "skipped"
assert control.grew == [] and control.shrank == []
def test_a_lane_with_no_row_is_left_alone(monkeypatch):
control = _stub_control(monkeypatch)
_sizing_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=99)
assert wc.size_lanes_sync({}) == []
assert control.grew == []
# --- the cap is also the switch ----------------------------------------------
def test_a_cap_of_zero_stops_the_lane_consuming(monkeypatch):
control = _stub_control(monkeypatch)
_sizing_live(monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0)
wc.size_lanes_sync({"worker": 0})
assert sorted(q for q, _ in control.cancelled) == sorted(
LANES_BY_NAME["worker"].queues
)
def test_a_cap_above_zero_starts_it_consuming(monkeypatch):
control = _stub_control(monkeypatch)
_sizing_live(
monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0,
consuming=set(),
)
wc.size_lanes_sync({"worker": 2})
assert sorted(q for q, _ in control.added) == sorted(
LANES_BY_NAME["worker"].queues
)
def test_an_already_stopped_lane_is_not_cancelled_again(monkeypatch):
"""The other half of the fixed point. `cancel_consumer` on a queue that is
not being consumed succeeds and does nothing, so an unconditional pass
would churn here forever with no symptom."""
control = _stub_control(monkeypatch)
_sizing_live(
monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0,
consuming=set(),
)
wc.size_lanes_sync({"worker": 0})
assert control.cancelled == []
assert control.added == []
def test_a_stopped_lane_still_keeps_exactly_one_process(monkeypatch):
"""The live bug of 2026-09-23, from the other direction: the pass must not
try to empty a pool billiard will not empty, and must not report having
done so every tick."""
control = _stub_control(monkeypatch)
_sizing_live(
monkeypatch, pools={"host-a": 1}, active=0, reserved=0, depth=0,
consuming=set(),
)
assert _worker(wc.size_lanes_sync({"worker": 0})).action == "held"
assert control.shrank == []
def test_the_sizing_task_is_registered_and_scheduled():
"""A pass nothing schedules is a pass that never runs — and since this one
replaced two entries, a stale name in the beat schedule would leave the
lanes unmanaged with nothing red anywhere."""
from backend.app.celery_app import celery
name = "backend.app.tasks.maintenance.size_worker_lanes"
assert name in celery.tasks
entries = celery.conf.beat_schedule
assert any(e["task"] == name for e in entries.values())
# The two it replaced are gone, not merely unreferenced.
tasks = {e["task"] for e in entries.values()}
assert "backend.app.tasks.maintenance.reconcile_worker_lanes" not in tasks
assert "backend.app.tasks.maintenance.autoscale_worker_lanes" not in tasks
+37 -16
View File
@@ -46,29 +46,50 @@ def test_every_routed_queue_has_a_lane_that_serves_it():
)
def test_defaults_are_one_of_each_with_ml_off():
"""Operator, 2026-09-22: 'that starting value should be one of each.'
def test_the_shipped_caps_are_one_of_each_with_ml_at_zero():
"""Operator, 2026-09-22: *"that starting value should be one of each."*
Restated 2026-09-23 when the cap became the ONLY number: *"the cap
defaults should be 1 and 0 for the ml-worker."*
ML at zero AND disabled is milestone 422 step 6's requirement arriving
early: enabling the lane is what triggers the SigLIP download, and rule
164 allows a runtime fetch only for a feature that is optional and clearly
off. A default of 1 here would make every fresh install reach HuggingFace.
ML at zero is milestone 422 step 6's requirement: a cap of zero means no
consumers, and raising it is what triggers the SigLIP download. Rule 164
allows a runtime fetch only for a feature that is optional and clearly
off, so a default of 1 here would make every fresh install reach
HuggingFace.
"""
by_name = wl.LANES_BY_NAME
for name in ("worker", "scheduler", "maintenance_long"):
assert by_name[name].default_slots == 1
assert by_name[name].default_enabled is True
assert by_name["ml"].default_slots == 0
assert by_name["ml"].default_enabled is False
assert by_name[name].default_slots_cap == 1
assert by_name["ml"].default_slots_cap == 0
def test_default_caps_leave_room_but_are_not_the_ceiling():
"""A cap that starts at the ceiling is a rubber stamp. Each lane's cap
must be at least its default slots (or the CHECK constraint rejects the
seeded row) and must leave somewhere to grow."""
def test_a_lane_has_exactly_one_operator_setting():
"""The whole point of the 2026-09-23 reshape. `slots`, `enabled` and
`autoscale` are gone: how many workers run is a measurement the sizing
pass owns, and "off" is a cap of zero.
Asserted against the dataclass's own fields rather than by name, so a
second knob added later fails here instead of quietly reappearing in the
UI beside the cap — which is exactly how three settings accumulated the
first time."""
import dataclasses
settable = {
f.name for f in dataclasses.fields(wl.Lane) if f.name.startswith("default_")
}
assert settable == {"default_slots_cap"}, (
f"a lane has more than one operator default: {sorted(settable)}"
)
def test_the_shipped_cap_is_never_the_ceiling_on_a_real_machine():
"""A cap that starts at the ceiling is a rubber stamp: there is nowhere to
raise it to, so the nudge that tells an operator to raise it would have
nothing to say. One-of-each leaves room on anything bigger than a
single-core box."""
for lane in wl.LANES:
assert lane.default_slots_cap >= lane.default_slots
assert lane.default_slots_cap >= 1
assert lane.default_slots_cap >= 0
assert lane.default_slots_cap <= 1
def test_only_ml_is_memory_bound():