bvandeusenandClaude Opus 5
a01165365b
Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 36s
Build images / build-ml (push) Successful in 1m55s
Build images / build-web (push) Successful in 1m54s
CI / integration (push) Successful in 2m16s
Build images / smoke-web (push) Failing after 7m48s
Build images / promote (push) Skipped
Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
105 lines
4.0 KiB
Python
105 lines
4.0 KiB
Python
"""Worker lanes: what each is doing, and the dial that changes it.
|
|
|
|
Milestone 422 step 2. The write half of a surface `api/system_activity.py`
|
|
only reads.
|
|
|
|
## Why this is a separate blueprint
|
|
|
|
`system_activity` says in its own first line that it is read-only, and it
|
|
answers a different question: its `/workers` is keyed on celery HOSTNAME and
|
|
reports which nodes answered. That stays as it is — the existing
|
|
SystemActivityTab consumes it.
|
|
|
|
This is keyed on LANE, joins the stored settings to the live pool, and
|
|
accepts writes. Two endpoints answering "which celery processes exist" and
|
|
"how much work is each lane allowed to do" are not the same endpoint, and
|
|
folding the second into the first would make a read-only module a write one.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from datetime import UTC, datetime
|
|
|
|
from quart import Blueprint, jsonify, request
|
|
|
|
from ..extensions import get_session
|
|
from ..services.worker_control import LaneUpdateRefused, lane_view, set_lane
|
|
from ..services.worker_lanes import LANES_BY_NAME
|
|
from ._responses import error_response as _bad
|
|
|
|
workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers")
|
|
|
|
|
|
@workers_bp.route("", methods=["GET"])
|
|
async def list_lanes():
|
|
"""Every lane: configured slots, the cap, the ceiling, and live state.
|
|
|
|
Response: {lanes: [...], fetched_at: iso8601}
|
|
|
|
Deliberately NOT cached, unlike system_activity's 2s/5s caches. This is
|
|
the surface an operator watches while dragging a stepper, and a cached
|
|
reply would show them the value from before their own change and read as
|
|
the control having failed.
|
|
"""
|
|
async with get_session() as session:
|
|
lanes = await lane_view(session)
|
|
return jsonify({
|
|
"lanes": lanes,
|
|
"fetched_at": datetime.now(UTC).isoformat(),
|
|
})
|
|
|
|
|
|
@workers_bp.route("/<name>", methods=["POST"])
|
|
async def update_lane(name: str):
|
|
"""Set a lane's slots, cap and/or enabled flag. Stores, then pushes live.
|
|
|
|
Partial: only the keys present are changed, so the UI's stepper can send
|
|
`{"slots": 3}` without restating the cap it did not touch.
|
|
|
|
Two failure kinds, deliberately different statuses:
|
|
|
|
* **400** — the value is not allowed (above the cap, above the ceiling,
|
|
negative). Nothing was stored. The body carries `detail`, which is the
|
|
sentence the UI shows; a refused control with no reason reads as a bug.
|
|
* **200 with `applied: false`** — the value WAS stored but could not be
|
|
pushed, because the lane is not currently answering. That is not an
|
|
error: step 3's reconcile carries it when the lane comes back, and the
|
|
UI should say "saved, not yet live" rather than "that didn't work".
|
|
"""
|
|
lane = LANES_BY_NAME.get(name)
|
|
if lane is None:
|
|
return _bad("unknown_lane", detail=name, known=sorted(LANES_BY_NAME))
|
|
|
|
body = await request.get_json()
|
|
if not isinstance(body, dict):
|
|
return _bad("invalid_body", detail="body must be a JSON object")
|
|
|
|
fields: dict = {}
|
|
for key in ("slots", "slots_cap"):
|
|
if key in body:
|
|
value = body[key]
|
|
# Rejected rather than coerced: `True` is an int in Python, and
|
|
# silently reading it as 1 slot would be a control that appears to
|
|
# work and sets something nobody asked for.
|
|
if not isinstance(value, int) or isinstance(value, bool):
|
|
return _bad("invalid_body", detail=f"{key} must be an integer")
|
|
fields[key] = value
|
|
for key in ("enabled", "autoscale"):
|
|
if key in body:
|
|
if not isinstance(body[key], bool):
|
|
return _bad("invalid_body", detail=f"{key} must be a boolean")
|
|
fields[key] = body[key]
|
|
|
|
if not fields:
|
|
return _bad(
|
|
"invalid_body",
|
|
detail="give at least one of slots, slots_cap, enabled, autoscale",
|
|
)
|
|
|
|
async with get_session() as session:
|
|
try:
|
|
result = await set_lane(session, lane, **fields)
|
|
except LaneUpdateRefused as exc:
|
|
return _bad("refused", detail=str(exc))
|
|
return jsonify(result)
|