fix: the cap dial waited out a broker round trip it did not need (4295)
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m13s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m59s
CI and images / smoke-web (push) Successful in 58s
CI and images / promote (push) Skipped
CI and images / lint (push) Successful in 3s
CI and images / extension-version (push) Successful in 3s
CI and images / frontend-build (push) Successful in 25s
CI and images / backend-lint-and-test (push) Successful in 33s
CI and images / integration (push) Successful in 2m13s
CI and images / sign-extension (push) Successful in 3s
CI and images / build-agent (push) Successful in 6s
CI and images / build-web (push) Successful in 1m59s
CI and images / smoke-web (push) Successful in 58s
CI and images / promote (push) Skipped
Operator: "when the number is changed the change should be queued so that
it isn't blocking of the webui or the system itself. we shouldn't have to
wait for the validation live."
Two waits, and 5b6f2ba removed neither — it stopped a Postgres connection
being HELD across them, which is what had been stalling the whole site,
and left the press itself as slow as it was.
1. The store refetched after every write. GET /api/system/workers runs a
celery inspect on an eleven-second budget, so the stepper stayed
disabled through a round trip the press did not need. It now patches
the row from the reply — cap, ceiling, enabled, the three fields that
reply actually decides — and lets the 15s poll bring the live columns,
which are measurements it must not invent.
2. The endpoint pushed to the broker before answering. Turning a lane off
is four cancel_consumer messages; lowering a cap reads the live pool
first. Now it stores the cap, answers `queued`, and hands the push to a
Quart background task. Raising a cap was already free and stays free.
Nothing is lost by not waiting: the stored cap is what the system obeys
and the sizing pass re-reads it every minute. That sweep was already the
backstop for a push that failed, which under `no_live_workers` is every
push in the suite.
Also closes a hole the move exposed: the model fetch was gated on the
consumer change having landed, so raising ML off zero while the lane was
restarting stored the cap, let the sizing pass start the consumers a
minute later, and left the lane running with no model — nothing else ever
asks for one. It now fires on the transition and waits in the ml queue.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
This commit is contained in:
@@ -563,6 +563,13 @@ async def store_lane_cap(
|
||||
async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
|
||||
"""Make the running lane obey a cap that is already stored. NO database.
|
||||
|
||||
Runs OFF the request path since 2026-09-23 — the endpoint stores the cap,
|
||||
answers, and hands this to a background task (operator: *"the change
|
||||
should be queued so that it isn't blocking of the webui"*). Nothing here
|
||||
changed as a result except who waits for it: the return value is now read
|
||||
by the log rather than by a browser, and every branch below already
|
||||
treated failure as "the sizing pass will carry it".
|
||||
|
||||
## What is pushed, and what is not
|
||||
|
||||
Consumers follow the cap immediately in BOTH directions: zero means off,
|
||||
@@ -612,13 +619,29 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
|
||||
# feature that is optional and clearly OFF.
|
||||
#
|
||||
# On the TRANSITION, so re-saving a cap on a lane already running does not
|
||||
# re-enqueue. And only when the consumer change landed: enqueueing onto a
|
||||
# queue nothing is consuming would leave the task pending with no
|
||||
# explanation until the lane returns.
|
||||
# re-enqueue.
|
||||
#
|
||||
# NOT gated on the consumer change having landed, which it was until
|
||||
# 2026-09-23. The reasoning then was that enqueueing onto a queue nothing
|
||||
# consumes leaves the task pending — true, and it is the right place for
|
||||
# it to wait. Gated, a cap raised while the lane was restarting stored the
|
||||
# cap, let the sizing pass start the consumers a minute later, and left
|
||||
# the lane running with no model, because nothing else ever asks for one.
|
||||
# A task parked on the `ml` queue is picked up the moment that happens.
|
||||
fetching = False
|
||||
if now_on and not was_on and lane.models and applied:
|
||||
if now_on and not was_on and lane.models:
|
||||
fetching = _enqueue_model_fetch()
|
||||
|
||||
# Nobody is waiting on this any more, so the log is where a push that did
|
||||
# not land has to be visible. Not an error: the value is stored and the
|
||||
# sizing pass carries it within a minute.
|
||||
if not applied:
|
||||
log.info(
|
||||
"worker_control: %s cap %s stored, not pushed (%s); "
|
||||
"the sizing pass will carry it",
|
||||
lane.name, slots_cap, error,
|
||||
)
|
||||
|
||||
return {
|
||||
"name": lane.name,
|
||||
"slots_cap": slots_cap,
|
||||
|
||||
Reference in New Issue
Block a user