bvandeusenandClaude Opus 5
5b6f2ba526
Operator, 2026-09-23: *"something about changing the cap number is blocking to the website... it shouldn't be"*. Nothing here was slow in itself. A database connection was held across work that is slow, and that is why it surfaced as the whole site stalling rather than as one slow page. `lane_view` took the session and kept it open through a celery inspect whose budget is 11s. The System tab polls that endpoint every 15s — and with a lane not answering, every inspect runs to nearly its full budget, so each poll pinned a connection for most of the interval. SQLAlchemy's default pool is 5 plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing, and a cap change adding two more inspects exhausts it, and every OTHER request then waits for a connection. Split so the database work finishes before the broker work starts: - `lane_settings(session)` reads the caps and the oldest running task, then the session closes. `lane_view(settings)` does the inspect with none held. - `store_lane_cap(session, …)` validates and commits, then the session closes. `push_lane_cap(lane, …)` does the live push with none held. And a second finding while measuring it: **raising a cap now costs no broker round trip at all.** The first cut only knew on/off, so it inspected on every raise to find out whether the pool needed lowering — the control meant to be instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap so the push knows the direction; only a lowering needs to say anything. The guard is structural, not timed: `lane_view` and `push_lane_cap` must not ACCEPT a session. A timing test would be flaky, and a call-order test would pass against a version that took the session and merely used it early. `/api/system/health` has the same shape and is NOT fixed here — it is rate-limited by `refresh_if_stale` so it does not inspect on every request. Worth doing, separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
110 lines
4.3 KiB
Python
110 lines
4.3 KiB
Python
"""Worker lanes: what each is doing, and the dial that changes it.
|
|
|
|
Milestone 422 step 2. The write half of a surface `api/system_activity.py`
|
|
only reads.
|
|
|
|
## Why this is a separate blueprint
|
|
|
|
`system_activity` says in its own first line that it is read-only, and it
|
|
answers a different question: its `/workers` is keyed on celery HOSTNAME and
|
|
reports which nodes answered. That stays as it is — the existing
|
|
SystemActivityTab consumes it.
|
|
|
|
This is keyed on LANE, joins the stored cap to the live pool, and accepts
|
|
writes. Two endpoints answering "which celery processes exist" and "how much
|
|
work is each lane allowed to do" are not the same endpoint, and folding the
|
|
second into the first would make a read-only module a write one.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from datetime import UTC, datetime
|
|
|
|
from quart import Blueprint, jsonify, request
|
|
|
|
from ..extensions import get_session
|
|
from ..services.worker_control import (
|
|
LaneUpdateRefused,
|
|
lane_settings,
|
|
lane_view,
|
|
push_lane_cap,
|
|
store_lane_cap,
|
|
)
|
|
from ..services.worker_lanes import LANES_BY_NAME
|
|
from ._responses import error_response as _bad
|
|
|
|
workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers")
|
|
|
|
|
|
@workers_bp.route("", methods=["GET"])
|
|
async def list_lanes():
|
|
"""Every lane: its cap, the ceiling above it, and what is live.
|
|
|
|
Response: {lanes: [...], fetched_at: iso8601}
|
|
|
|
Deliberately NOT cached, unlike system_activity's 2s/5s caches. This is
|
|
the surface an operator watches while dragging a stepper, and a cached
|
|
reply would show them the value from before their own change and read as
|
|
the control having failed.
|
|
"""
|
|
# The session closes BEFORE the broker work. Holding a Postgres connection
|
|
# across a celery inspect is what made this page block the whole site —
|
|
# see `worker_control.LaneSettings`. This endpoint polls every 15s and the
|
|
# inspect budget is 11s, so each poll was pinning a connection for most of
|
|
# the interval.
|
|
async with get_session() as session:
|
|
settings = await lane_settings(session)
|
|
lanes = await lane_view(settings)
|
|
return jsonify({
|
|
"lanes": lanes,
|
|
"fetched_at": datetime.now(UTC).isoformat(),
|
|
})
|
|
|
|
|
|
@workers_bp.route("/<name>", methods=["POST"])
|
|
async def update_lane(name: str):
|
|
"""Set a lane's cap. Stores it, then makes the live lane obey it.
|
|
|
|
ONE field, since 2026-09-23. It used to take `slots`, `slots_cap`,
|
|
`enabled` and `autoscale`; how many workers are running is now a
|
|
measurement the sizing pass owns, and `enabled` is `cap > 0`.
|
|
|
|
Two failure kinds, deliberately different statuses:
|
|
|
|
* **400** — the value is not allowed (negative, or above what this
|
|
container can hold). Nothing was stored. The body carries `detail`,
|
|
which is the sentence the UI shows; a refused control with no reason
|
|
reads as a bug.
|
|
* **200 with `applied: false`** — the value WAS stored but could not be
|
|
pushed, because the lane is not currently answering. Not an error: the
|
|
sizing pass carries it within a minute, and the UI should say "saved,
|
|
not yet live" rather than "that didn't work".
|
|
"""
|
|
lane = LANES_BY_NAME.get(name)
|
|
if lane is None:
|
|
return _bad("unknown_lane", detail=name, known=sorted(LANES_BY_NAME))
|
|
|
|
body = await request.get_json()
|
|
if not isinstance(body, dict):
|
|
return _bad("invalid_body", detail="body must be a JSON object")
|
|
|
|
if "slots_cap" not in body:
|
|
return _bad("invalid_body", detail="give slots_cap")
|
|
value = body["slots_cap"]
|
|
# Rejected rather than coerced: `True` is an int in Python, and silently
|
|
# reading it as a cap of 1 would be a control that appears to work and
|
|
# sets something nobody asked for.
|
|
if not isinstance(value, int) or isinstance(value, bool):
|
|
return _bad("invalid_body", detail="slots_cap must be an integer")
|
|
|
|
# Store, close the session, THEN push. Same reason as the GET above, and
|
|
# more sharply here: a cap change could do three broker round trips, all
|
|
# of them previously with a connection held.
|
|
async with get_session() as session:
|
|
try:
|
|
was_cap = await store_lane_cap(session, lane, value)
|
|
except LaneUpdateRefused as exc:
|
|
return _bad("refused", detail=str(exc))
|
|
result = await push_lane_cap(lane, value, was_cap=was_cap)
|
|
return jsonify(result)
|