CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 2s
Build images / sign-extension (push) Successful in 4s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 25s
CI / backend-lint-and-test (push) Successful in 31s
Build images / build-web (push) Successful in 55s
Build images / smoke-web (push) Skipped
Build images / build-ml (push) Successful in 1m41s
Build images / promote (push) Skipped
CI / integration (push) Successful in 2m8s
Milestone 422 step 2. `GET /api/system/workers` reports every lane joined to its live pool; `POST /api/system/workers/<name>` changes it. NO DOCKER SOCKET. Milestone 365 deferred "acting on the state" because restarting a dead worker needs a socket the web container deliberately does not have. That holds for restarting a CONTAINER; it does not hold for changing how much work a RUNNING worker does. celery's pool_grow / pool_shrink / add_consumer / cancel_consumer send a message over the Redis the app already uses, and the worker resizes itself. No new privilege, no new surface, and the security question that deferred this is never raised. PERSIST AND PUSH, in one call, in that order. pool_grow is not durable — a restart drops every lane to its env concurrency — so a UI that only pushed would lose the setting on the next deploy with nothing to show for it (lesson #4202). Storing alone would describe nothing until something restarted. A failed PUSH is not a failed setting: 200 with `applied: false` and a reason, so the UI says "saved, not yet live" rather than "that didn't work". Step 3's reconcile carries it when the lane answers again. PER-REPLICA DELTAS. `pool_grow(n, destination=[...])` adds n to EACH destination, so while `worker` runs `replicas: 2` a single delta from an aggregate is wrong for both. `slots` therefore means what CELERY_CONCURRENCY means — one process's pool — and each replica is driven to it from its OWN current size, so replicas that drifted apart converge rather than moving in lockstep. I wrote this wrong first: the docstring claimed per-replica while the code computed one delta from the max across replicas. LaneLiveState now carries `pools` per hostname and exposes `pool` as a property. A replica already at the target is sent nothing at all — the reachable fixed point step 3's periodic reconcile needs, or it re-issues a grow of zero every tick forever (lesson #4183). A replica that answered inspect but not stats is NAMED in the error rather than skipped silently, since otherwise it would run at a size the UI claims it does not. `present=False` is not "zero slots", it is "nothing answered" — kept distinct throughout, because step 3 skips an absent lane rather than correcting it. /workers now also reports pool size (from `insp.stats()`) and RESERVED count. Celery prefetches, so tasks that have left the Redis list but not started are invisible to LLEN: a lane can read depth 0 with thirty tasks held in worker memory. `pending` is depth + reserved. The UI is misleading without this and step 7's autoscaler would be simply wrong. Also kills the THIRD copy of the queue list: system_activity's _QUEUE_NAMES, whose own comment admitted the coupling ("must match celery_app.task_routes") and which sat alongside task_routes and the ROLE_NAMES copy step 1 collapsed. Now derived from LANES. The rendered order changes to lane grouping, which is the better shape for a lane-oriented UI. Separate blueprint rather than folding into system_activity, which states in its first line that it is read-only and answers a different question — its /workers is keyed on celery HOSTNAME and reports which nodes answered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
173 lines
5.9 KiB
Python
173 lines
5.9 KiB
Python
"""/api/system/workers — the lane dial (milestone 422 step 2).
|
|
|
|
Exercises the real endpoint against the real database. Only `celery inspect`
|
|
is stubbed, and only to keep the suite fast: an unstubbed inspect blocks for
|
|
its full 2s timeout per call with no workers to answer, which several writes
|
|
would turn into most of the lane's runtime.
|
|
"""
|
|
|
|
import pytest
|
|
import pytest_asyncio
|
|
from sqlalchemy import select
|
|
|
|
from backend.app.models import WorkerLane
|
|
from backend.app.services import worker_control as wc
|
|
from backend.app.services.worker_lanes import LANES
|
|
|
|
pytestmark = pytest.mark.integration
|
|
|
|
|
|
@pytest_asyncio.fixture
|
|
async def no_live_workers(monkeypatch):
|
|
"""Nothing is running — which is the CI lane's actual truth, asserted
|
|
rather than waited for. Makes every push fail, which is the interesting
|
|
half: the setting must still be stored."""
|
|
monkeypatch.setattr(
|
|
wc, "inspect_lanes_sync",
|
|
lambda: {lane.name: wc.LaneLiveState() for lane in LANES},
|
|
)
|
|
|
|
|
|
async def _lane_row(db, name: str) -> WorkerLane:
|
|
return (await db.execute(
|
|
select(WorkerLane).where(WorkerLane.name == name)
|
|
)).scalar_one()
|
|
|
|
|
|
# --- reading -----------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_get_lists_every_lane_with_its_ceiling(client, no_live_workers):
|
|
resp = await client.get("/api/system/workers")
|
|
assert resp.status_code == 200
|
|
body = await resp.get_json()
|
|
|
|
by_name = {lane["name"]: lane for lane in body["lanes"]}
|
|
assert set(by_name) == {"worker", "scheduler", "maintenance_long", "ml"}
|
|
for lane in body["lanes"]:
|
|
assert lane["ceiling"] >= 0
|
|
assert lane["slots"] <= lane["slots_cap"]
|
|
# Nothing is running, so live state must say so rather than report
|
|
# zeroes that read like a healthy idle lane.
|
|
assert lane["live"]["present"] is False
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ml_ships_off(client, no_live_workers):
|
|
"""Rule 164's carve-out and the weak-hardware default in one row: enabling
|
|
the lane is what triggers the SigLIP download, so a fresh install must not
|
|
find it on."""
|
|
body = await (await client.get("/api/system/workers")).get_json()
|
|
ml = next(lane for lane in body["lanes"] if lane["name"] == "ml")
|
|
assert ml["enabled"] is False
|
|
assert ml["slots"] == 0
|
|
|
|
|
|
# --- the persist / push split ------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_value_is_stored_even_when_it_cannot_be_pushed(
|
|
client, db, no_live_workers
|
|
):
|
|
"""THE test for this step. `pool_grow` is not durable and the lane is not
|
|
answering, so the push fails — and the setting must survive anyway, or a
|
|
UI that saved while a worker was restarting would silently lose it
|
|
(lesson #4202). 200 with `applied: false`, not an error.
|
|
"""
|
|
resp = await client.post("/api/system/workers/worker", json={"slots": 3})
|
|
assert resp.status_code == 200
|
|
body = await resp.get_json()
|
|
|
|
assert body["slots"] == 3
|
|
assert body["applied"] is False
|
|
assert "not running" in body["apply_error"]
|
|
|
|
assert (await _lane_row(db, "worker")).slots == 3
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_partial_update_leaves_the_other_fields_alone(
|
|
client, db, no_live_workers
|
|
):
|
|
"""The stepper sends `{"slots": n}` without restating a cap it did not
|
|
touch."""
|
|
before = await _lane_row(db, "worker")
|
|
original_cap = before.slots_cap
|
|
|
|
await client.post("/api/system/workers/worker", json={"slots": 2})
|
|
|
|
await db.refresh(before)
|
|
assert before.slots == 2
|
|
assert before.slots_cap == original_cap
|
|
|
|
|
|
# --- what is refused ---------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_slots_above_the_cap_are_refused_and_nothing_is_stored(
|
|
client, db, no_live_workers
|
|
):
|
|
row = await _lane_row(db, "worker")
|
|
resp = await client.post(
|
|
"/api/system/workers/worker", json={"slots": row.slots_cap + 1},
|
|
)
|
|
assert resp.status_code == 400
|
|
body = await resp.get_json()
|
|
assert body["error"] == "refused"
|
|
# The sentence the UI shows. A greyed control with no reason reads as a bug.
|
|
assert "cap" in body["detail"]
|
|
|
|
await db.refresh(row)
|
|
assert row.slots <= row.slots_cap
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_cap_above_the_derived_ceiling_is_refused(
|
|
client, db, no_live_workers
|
|
):
|
|
"""The operator's cap is theirs to set, but not past what the container
|
|
can hold — the whole point of deriving a ceiling rather than typing one."""
|
|
resp = await client.post(
|
|
"/api/system/workers/ml", json={"slots_cap": 10_000},
|
|
)
|
|
assert resp.status_code == 400
|
|
body = await resp.get_json()
|
|
assert body["error"] == "refused"
|
|
assert "container can hold" in body["detail"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_boolean_is_not_accepted_as_a_slot_count(client, no_live_workers):
|
|
"""`True` is an int in Python. Coerced, it would silently set one slot —
|
|
a control that appears to work and does something nobody asked for."""
|
|
resp = await client.post("/api/system/workers/worker", json={"slots": True})
|
|
assert resp.status_code == 400
|
|
body = await resp.get_json()
|
|
assert body["error"] == "invalid_body"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unknown_lane_is_refused_and_names_the_known_ones(
|
|
client, no_live_workers
|
|
):
|
|
resp = await client.post("/api/system/workers/nonsense", json={"slots": 1})
|
|
assert resp.status_code == 400
|
|
body = await resp.get_json()
|
|
assert body["error"] == "unknown_lane"
|
|
assert "worker" in body["known"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_empty_body_is_refused_rather_than_treated_as_a_no_op(
|
|
client, no_live_workers
|
|
):
|
|
"""A POST that changes nothing and returns 200 is indistinguishable from
|
|
one that worked, which is how a broken UI control goes unnoticed."""
|
|
resp = await client.post("/api/system/workers/worker", json={})
|
|
assert resp.status_code == 400
|
|
body = await resp.get_json()
|
|
assert body["error"] == "invalid_body"
|