bvandeusenandClaude Opus 5
5b6f2ba526
Operator, 2026-09-23: *"something about changing the cap number is blocking to the website... it shouldn't be"*. Nothing here was slow in itself. A database connection was held across work that is slow, and that is why it surfaced as the whole site stalling rather than as one slow page. `lane_view` took the session and kept it open through a celery inspect whose budget is 11s. The System tab polls that endpoint every 15s — and with a lane not answering, every inspect runs to nearly its full budget, so each poll pinned a connection for most of the interval. SQLAlchemy's default pool is 5 plus 10 overflow. Two browser tabs, `/api/system/health` doing the same thing, and a cap change adding two more inspects exhausts it, and every OTHER request then waits for a connection. Split so the database work finishes before the broker work starts: - `lane_settings(session)` reads the caps and the oldest running task, then the session closes. `lane_view(settings)` does the inspect with none held. - `store_lane_cap(session, …)` validates and commits, then the session closes. `push_lane_cap(lane, …)` does the live push with none held. And a second finding while measuring it: **raising a cap now costs no broker round trip at all.** The first cut only knew on/off, so it inspected on every raise to find out whether the pool needed lowering — the control meant to be instant still waited out an inspect. `store_lane_cap` returns the PREVIOUS cap so the push knows the direction; only a lowering needs to say anything. The guard is structural, not timed: `lane_view` and `push_lane_cap` must not ACCEPT a session. A timing test would be flaky, and a call-order test would pass against a version that took the session and merely used it early. `/api/system/health` has the same shape and is NOT fixed here — it is rate-limited by `refresh_if_stale` so it does not inspect on every request. Worth doing, separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
813 lines
34 KiB
Python
813 lines
34 KiB
Python
"""Read and change a lane's live pool, over the broker.
|
|
|
|
Milestone 422 step 2. The half of the milestone that does something.
|
|
|
|
## No docker socket is involved, and that is the point
|
|
|
|
Milestone 365 put "acting on the state" out of scope because restarting a
|
|
dead worker needs a docker socket the web container deliberately does not
|
|
have. That is true of RESTARTING a container. It is not true of changing how
|
|
much work a RUNNING worker does: celery's remote control sends a message over
|
|
the broker and the worker resizes its own pool. Same Redis the app already
|
|
uses, no new privilege, no new surface.
|
|
|
|
pool_grow / pool_shrink how many slots a lane runs
|
|
add_consumer / cancel_consumer whether it consumes its queues at all
|
|
|
|
The operator ruled the socket out independently (2026-09-22: *"this feature
|
|
is a very invasive idea in my mind and I'd like to avoid it"*), and nothing
|
|
here raises the question.
|
|
|
|
## The setting is PER PROCESS, not per lane total
|
|
|
|
`pool_grow(n, destination=[...])` adds n slots to EACH destination it names.
|
|
While the stack still runs several containers per lane — the operator's
|
|
production `worker` is `replicas: 2` — a single delta applied to a lane's
|
|
total would be wrong for every replica.
|
|
|
|
So `slots` means what `CELERY_CONCURRENCY` means: the pool size of one
|
|
process. The reconcile below drives EACH replica to that number
|
|
independently, computing its own delta from that replica's current pool, so
|
|
replicas that have drifted apart (one restarted, one was grown) converge
|
|
rather than being moved in lockstep from a shared baseline.
|
|
|
|
After step 5 there is one process per lane and the distinction disappears.
|
|
It matters now, and getting it wrong now would be invisible — the totals
|
|
would simply be double what the UI claimed.
|
|
|
|
## Why reserved() is read alongside the queue depth
|
|
|
|
Celery PREFETCHES: a worker pulls more messages than it can run and holds
|
|
them in memory. Those have already left the Redis list, so `LLEN` — which is
|
|
what `/api/system/activity/queues` reports — can read 0 while thirty tasks
|
|
are waiting inside a worker. Any judgement about backlog that uses only LLEN
|
|
under-reports, which matters for the UI and is disqualifying for step 7's
|
|
autoscaler.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import logging
|
|
from dataclasses import dataclass, field
|
|
from datetime import UTC, datetime
|
|
|
|
from sqlalchemy import func, select
|
|
from sqlalchemy.ext.asyncio import AsyncSession
|
|
|
|
from ..models import TaskRun, WorkerLane
|
|
from .worker_lanes import (
|
|
LANES,
|
|
LANES_BY_QUEUE_KEY,
|
|
MIN_POOL_SLOTS,
|
|
Lane,
|
|
derived_ceiling,
|
|
)
|
|
|
|
log = logging.getLogger(__name__)
|
|
|
|
# celery control is a broker round trip on a request path, so it gets a
|
|
# deadline (rule 156) — the same reasoning and the same budget as
|
|
# service_roster's inspect. A broker that stopped answering must make this
|
|
# report "not present", which is true, rather than hang the page.
|
|
CONTROL_TIMEOUT_SECONDS = 2.0
|
|
|
|
|
|
# The WORST case of `inspect_lanes_sync`, for callers that need a deadline.
|
|
#
|
|
# One broadcast plus three targeted reads. The targeted three normally return
|
|
# as soon as the named nodes answer; each can still cost a full timeout if a
|
|
# node disappears mid-read, so the bound stays four.
|
|
CONTROL_ROUND_TRIPS = 4
|
|
|
|
# Slack for the `asyncio.to_thread` handoff. A budget equal to the work is a
|
|
# budget that fails under load — the roster carried exactly that bug into the
|
|
# operator's first consolidated deploy and logged a TimeoutError per refresh
|
|
# while the inspect calls underneath were working fine.
|
|
CONTROL_SLACK_SECONDS = 3.0
|
|
|
|
INSPECT_BUDGET_SECONDS = (
|
|
CONTROL_TIMEOUT_SECONDS * CONTROL_ROUND_TRIPS + CONTROL_SLACK_SECONDS
|
|
)
|
|
|
|
|
|
@dataclass
|
|
class LaneLiveState:
|
|
"""What `celery inspect` says about one lane right now.
|
|
|
|
`present=False` is NOT "zero slots" — it is "nothing answered". A lane
|
|
whose worker is restarting, or whose broker is unreachable, must read as
|
|
unknown rather than as stopped: an unswept absence is not a verdict
|
|
(snippet #3969). The reconcile in step 3 skips an absent lane rather than
|
|
correcting it, which is only safe because this distinction is kept.
|
|
"""
|
|
|
|
present: bool = False
|
|
replicas: int = 0
|
|
active: int = 0
|
|
reserved: int = 0
|
|
hostnames: list[str] = field(default_factory=list)
|
|
# The queues this lane is actually consuming right now, across replicas.
|
|
# Distinct from the lane's CONFIGURED queues: `cancel_consumer` stops a
|
|
# worker consuming one without changing what it was started with, which
|
|
# is how `enabled=false` is implemented. The reconcile needs this to tell
|
|
# "already disabled" from "needs disabling" — without it, it would re-send
|
|
# add_consumer for every queue on every tick forever (lesson #4183).
|
|
consuming: set[str] = field(default_factory=set)
|
|
# Pool size PER HOSTNAME, not aggregated. The resize below computes each
|
|
# replica's own delta from its own current pool, so replicas that have
|
|
# drifted apart converge instead of being moved in lockstep from a shared
|
|
# baseline — which is what an aggregate here would silently reintroduce.
|
|
pools: dict[str, int] = field(default_factory=dict)
|
|
|
|
@property
|
|
def pool(self) -> int | None:
|
|
"""One number for the UI. `max` rather than a sum: `slots` means the
|
|
pool size of ONE process (see the module docstring), so the largest
|
|
replica is the honest answer to "what is this lane set to". None when
|
|
no replica reported — unknown, never zero."""
|
|
return max(self.pools.values()) if self.pools else None
|
|
|
|
@property
|
|
def capacity(self) -> int:
|
|
"""Total slots across replicas — how many tasks this lane can run at
|
|
once. Distinct from `pool`, and the two must not be confused: `pool`
|
|
is the DIAL (one process's size, what grow/shrink move), `capacity` is
|
|
the CAPABILITY. Asking "is this lane saturated" compares `active`,
|
|
which is summed across replicas, against this — against `pool` it
|
|
would call two half-busy replicas of 4 saturated at 4 active."""
|
|
return sum(self.pools.values())
|
|
|
|
|
|
def _lane_for_queues(queues: tuple[str, ...]) -> Lane | None:
|
|
return LANES_BY_QUEUE_KEY.get(tuple(sorted(queues)))
|
|
|
|
|
|
def inspect_lanes_sync() -> dict[str, LaneLiveState]:
|
|
"""Live state per lane name. Sync — callers wrap in asyncio.to_thread.
|
|
|
|
Never raises. Every lane is present in the result; ones nothing answered
|
|
for carry `present=False`, so a caller cannot accidentally read a missing
|
|
lane as an empty one by iterating only what came back.
|
|
"""
|
|
out = {lane.name: LaneLiveState() for lane in LANES}
|
|
try:
|
|
from ..celery_app import celery as celery_app
|
|
|
|
# ONE broadcast, then three TARGETED reads.
|
|
#
|
|
# A broadcast with no `destination` cannot know how many replies to
|
|
# expect, so it waits out its whole timeout rather than returning on
|
|
# the last one. Four of those is four full timeouts — about eight
|
|
# seconds — and `lane_view` sits on the Settings card, so that was the
|
|
# load time of the Worker lanes page every time it was opened.
|
|
#
|
|
# Naming the destinations lets celery stop as soon as those nodes have
|
|
# answered, which for workers in this same container is milliseconds.
|
|
# The worst case is unchanged: a node that vanishes between the
|
|
# broadcast and the targeted reads costs a full timeout waiting for a
|
|
# reply that is not coming.
|
|
insp = celery_app.control.inspect(timeout=CONTROL_TIMEOUT_SECONDS)
|
|
active_queues = insp.active_queues() or {}
|
|
|
|
# Nothing answered — and the three reads below exist only to describe
|
|
# what did. Returning here also makes the broker-down case FAST
|
|
# (one timeout, not four), which is exactly when the healthcheck and
|
|
# the card need an answer rather than a long wait.
|
|
if not active_queues:
|
|
return out
|
|
|
|
targeted = celery_app.control.inspect(
|
|
destination=sorted(active_queues),
|
|
timeout=CONTROL_TIMEOUT_SECONDS,
|
|
)
|
|
stats = targeted.stats() or {}
|
|
active = targeted.active() or {}
|
|
reserved = targeted.reserved() or {}
|
|
except Exception:
|
|
log.warning("worker_control: celery inspect failed", exc_info=True)
|
|
return out
|
|
|
|
for hostname, queues in active_queues.items():
|
|
lane = _lane_for_queues(tuple(q["name"] for q in queues))
|
|
if lane is None:
|
|
# A deployment slicing CELERY_QUEUES differently. Reported by the
|
|
# roster under its raw queue list; it simply has no lane row to
|
|
# control, which is honest rather than an error.
|
|
continue
|
|
state = out[lane.name]
|
|
state.present = True
|
|
state.replicas += 1
|
|
state.hostnames.append(hostname)
|
|
state.active += len(active.get(hostname, []))
|
|
state.reserved += len(reserved.get(hostname, []))
|
|
state.consuming.update(q["name"] for q in queues)
|
|
|
|
# `pool.max-concurrency` is the number pool_grow/pool_shrink move and
|
|
# the number the UI shows. Absent on a worker whose stats did not
|
|
# answer, which leaves pool=None — unknown, not zero.
|
|
pool = (stats.get(hostname) or {}).get("pool", {}).get("max-concurrency")
|
|
if isinstance(pool, int):
|
|
state.pools[hostname] = pool
|
|
|
|
for state in out.values():
|
|
state.hostnames.sort()
|
|
return out
|
|
|
|
|
|
def effective_slots(target: int) -> int:
|
|
"""What a pool can actually be set to. Never below one process.
|
|
|
|
Used wherever a target is COMPARED as well as wherever one is sent: a
|
|
reconcile that compares against the unclamped number sees a difference
|
|
that no control message can ever close, and re-sends it every tick.
|
|
"""
|
|
return max(MIN_POOL_SLOTS, target)
|
|
|
|
|
|
def set_lane_slots_sync(
|
|
lane: Lane, target: int, live: LaneLiveState | None = None,
|
|
) -> tuple[bool, str | None]:
|
|
"""Drive every replica of `lane` to `target` slots. Returns (applied, err).
|
|
|
|
Per-replica deltas rather than one shared delta: see the module docstring.
|
|
A replica already at the target is issued nothing at all, which is what
|
|
makes step 3's periodic reconcile converge instead of re-sending a grow of
|
|
zero forever (lesson #4183 — an enforcer without a reachable fixed point
|
|
re-does its own work every tick).
|
|
|
|
`applied=False` is not a failure of the SETTING. The caller has already
|
|
stored the value; this says only that the live push did not land, and the
|
|
reconcile will carry it when the lane answers again.
|
|
"""
|
|
target = effective_slots(target)
|
|
try:
|
|
from ..celery_app import celery as celery_app
|
|
|
|
if live is None:
|
|
live = inspect_lanes_sync()[lane.name]
|
|
if not live.present:
|
|
return False, "lane is not running"
|
|
if not live.pools:
|
|
return False, "worker did not report its pool size"
|
|
|
|
control = celery_app.control
|
|
unreported = [h for h in live.hostnames if h not in live.pools]
|
|
for hostname, current in live.pools.items():
|
|
delta = target - current
|
|
if delta > 0:
|
|
control.pool_grow(delta, destination=[hostname])
|
|
elif delta < 0:
|
|
control.pool_shrink(-delta, destination=[hostname])
|
|
if unreported:
|
|
# Resized what could be resized, and said which could not. Silence
|
|
# here would leave a replica running at a size the UI claims it is
|
|
# not, with nothing anywhere recording the gap.
|
|
return False, f"no pool size reported by {', '.join(sorted(unreported))}"
|
|
return True, None
|
|
except Exception as exc: # noqa: BLE001 — reported, never raised at a caller
|
|
log.warning("worker_control: could not resize %s", lane.name, exc_info=True)
|
|
return False, str(exc)
|
|
|
|
|
|
def set_lane_enabled_sync(
|
|
lane: Lane, enabled: bool, live: LaneLiveState | None = None,
|
|
) -> tuple[bool, str | None]:
|
|
"""Start or stop `lane` consuming its queues, without killing the process.
|
|
|
|
`cancel_consumer` rather than a shutdown: a stopped consumer keeps its
|
|
worker alive and answering `inspect`, so a disabled lane stays visible and
|
|
can be turned back on. A killed worker would read as absent, which is the
|
|
same signal as a crash — and the whole point of the roster (#365) is that
|
|
those two must not look alike.
|
|
"""
|
|
try:
|
|
from ..celery_app import celery as celery_app
|
|
|
|
if live is None:
|
|
live = inspect_lanes_sync()[lane.name]
|
|
if not live.present:
|
|
return False, "lane is not running"
|
|
control = celery_app.control
|
|
for queue in lane.queues:
|
|
if enabled:
|
|
control.add_consumer(queue, destination=live.hostnames)
|
|
else:
|
|
control.cancel_consumer(queue, destination=live.hostnames)
|
|
return True, None
|
|
except Exception as exc: # noqa: BLE001
|
|
log.warning(
|
|
"worker_control: could not %s %s",
|
|
"enable" if enabled else "disable", lane.name, exc_info=True,
|
|
)
|
|
return False, str(exc)
|
|
|
|
|
|
# --- the settings half, which is async ----------------------------------------
|
|
#
|
|
# Sync celery control above, async DB below, in one module. Same split
|
|
# `service_roster` already runs (`_inspect_celery_sync` beside `touch_service`)
|
|
# — the boundary is the transport, not the concern, and "control the workers"
|
|
# is one concern.
|
|
|
|
|
|
async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]:
|
|
"""Every lane's row, creating any that are missing from its LANES defaults.
|
|
|
|
Self-heals rather than depending on a migration having run for a lane
|
|
added later: alembic 0103 seeded the four that existed on 2026-09-22, and
|
|
a fifth added to LANES afterwards gets its row the first time anything
|
|
asks. Without this, a new lane would read as absent and the UI would
|
|
simply not show it.
|
|
"""
|
|
rows = {
|
|
row.name: row
|
|
for row in (await session.execute(select(WorkerLane))).scalars()
|
|
}
|
|
missing = [lane for lane in LANES if lane.name not in rows]
|
|
for lane in missing:
|
|
row = WorkerLane(name=lane.name, slots_cap=lane.default_slots_cap)
|
|
session.add(row)
|
|
rows[lane.name] = row
|
|
if missing:
|
|
await session.commit()
|
|
return rows
|
|
|
|
|
|
@dataclass
|
|
class LaneSettings:
|
|
"""What the DATABASE knows about the lanes — read and finished with before
|
|
anything touches the broker.
|
|
|
|
This exists because holding a Postgres connection across a celery round
|
|
trip is what made the System tab block the whole site (operator,
|
|
2026-09-23: *"something about changing the cap number is blocking to the
|
|
website"*).
|
|
|
|
`lane_view` used to take the session and keep it open through an inspect
|
|
whose budget is eleven seconds — and that page polls every fifteen. With a
|
|
lane not answering, every inspect ran to nearly its full budget, so each
|
|
poll pinned a connection for ten seconds. SQLAlchemy's default pool is
|
|
five connections plus ten overflow; a couple of browser tabs, the health
|
|
endpoint doing the same thing, and a cap change adding two more inspects
|
|
exhausts that, and every OTHER request then waits on a connection.
|
|
|
|
Nothing was slow in itself. The slowness was a scarce resource held across
|
|
it, which is why it surfaced as the whole site stalling rather than as one
|
|
slow page.
|
|
"""
|
|
|
|
caps: dict[str, int]
|
|
oldest_by_queue: dict[str, datetime]
|
|
|
|
|
|
async def lane_settings(session: AsyncSession) -> LaneSettings:
|
|
"""Every DB read the lane view needs, in one short-lived session."""
|
|
rows = await _rows_by_name(session)
|
|
return LaneSettings(
|
|
caps={name: row.slots_cap for name, row in rows.items()},
|
|
oldest_by_queue=await _oldest_running_by_queue(session),
|
|
)
|
|
|
|
|
|
async def lane_view(settings: LaneSettings) -> list[dict]:
|
|
"""Every lane: what is configured, what is live, what it may grow to.
|
|
|
|
Takes the settings rather than a session ON PURPOSE — see `LaneSettings`.
|
|
Everything below this line is broker work, and no database connection is
|
|
held while it happens.
|
|
|
|
`pending` is the honest backlog — Redis depth PLUS reserved — because
|
|
celery prefetches and LLEN alone reads 0 while a worker holds tasks in
|
|
memory.
|
|
"""
|
|
# A deadline, because this is a request path and `to_thread` on its own is
|
|
# an await with no bound (rule 156). `inspect_lanes_sync` never raises and
|
|
# every inner call has its own timeout, so the only way past the budget is
|
|
# the thread not being scheduled — and a page that renders "not answering"
|
|
# is a better answer than one that does not render.
|
|
try:
|
|
live = await asyncio.wait_for(
|
|
asyncio.to_thread(inspect_lanes_sync),
|
|
timeout=INSPECT_BUDGET_SECONDS,
|
|
)
|
|
except TimeoutError:
|
|
log.warning(
|
|
"worker_control: inspect exceeded %ss; reporting every lane as "
|
|
"not answering", INSPECT_BUDGET_SECONDS,
|
|
)
|
|
live = {lane.name: LaneLiveState() for lane in LANES}
|
|
depths = await asyncio.to_thread(_queue_depths_sync)
|
|
oldest = settings.oldest_by_queue
|
|
|
|
now = datetime.now(UTC)
|
|
out = []
|
|
for lane in LANES:
|
|
cap = settings.caps[lane.name]
|
|
state = live[lane.name]
|
|
# None for a queue the broker did not answer for, which must not be
|
|
# silently summed as zero — an unknown depth is not an empty one.
|
|
known = [depths.get(q) for q in lane.queues]
|
|
depth = sum(d for d in known if d is not None) if any(
|
|
d is not None for d in known
|
|
) else None
|
|
out.append({
|
|
"name": lane.name,
|
|
"display_name": lane.display_name,
|
|
"queues": list(lane.queues),
|
|
"slots_cap": cap,
|
|
"ceiling": derived_ceiling(lane),
|
|
# DERIVED, never stored. A cap of zero means no consumers, so
|
|
# "off" and "may use no workers" cannot disagree.
|
|
"enabled": cap > 0,
|
|
"memory_bound": lane.memory_bound,
|
|
"optional": lane.optional,
|
|
# What raising this lane's cap will download, so the UI can say
|
|
# WHICH model and how big BEFORE the first slot is asked for
|
|
# rather than after a multi-GB fetch has started. `measured`
|
|
# travels with the numbers: the UI must not present an estimate
|
|
# as a fact.
|
|
"models": [
|
|
{
|
|
"repo": m.repo,
|
|
"download_bytes": m.approx_download_bytes,
|
|
"resident_bytes": m.approx_resident_bytes,
|
|
"measured": m.measured,
|
|
}
|
|
for m in lane.models
|
|
],
|
|
"live": {
|
|
"present": state.present,
|
|
"replicas": state.replicas,
|
|
"pool": state.pool,
|
|
"active": state.active,
|
|
"reserved": state.reserved,
|
|
},
|
|
"queue_depth": depth,
|
|
"pending": None if depth is None else depth + state.reserved,
|
|
# How long the oldest still-running task on this lane has been
|
|
# going, in minutes. The operator asked for a trigger here — grow
|
|
# a lane whose tasks run past some duration — and it stayed a
|
|
# REPORT: a long task does not finish sooner because the lane
|
|
# gained a slot, so scaling on it would spend memory to change
|
|
# nothing. Shown so they can see a lane wedged on one slow job,
|
|
# which is the genuinely useful half of the idea.
|
|
"oldest_running_minutes": _minutes_since(
|
|
min(
|
|
(oldest[q] for q in lane.queues if q in oldest),
|
|
default=None,
|
|
),
|
|
now,
|
|
),
|
|
})
|
|
return out
|
|
|
|
|
|
async def _oldest_running_by_queue(session: AsyncSession) -> dict[str, datetime]:
|
|
"""When the longest-running unfinished task on each queue started.
|
|
|
|
Read from `task_run`, which is OUR OWN table on OUR OWN wall clock, and
|
|
deliberately not from celery's `inspect active()`. Those entries carry a
|
|
`time_start` taken from the WORKER's `time.monotonic()` — a clock with an
|
|
arbitrary origin per process. Subtracting it from this process's wall
|
|
clock produces a number that looks like a duration and is meaningless, and
|
|
it would be meaningless in the direction that matters: plausible.
|
|
|
|
`task_run` also already carries the per-queue staleness thresholds the
|
|
recovery sweep uses, so a row still `running` here is one the system
|
|
itself considers legitimately in flight rather than abandoned.
|
|
"""
|
|
result = await session.execute(
|
|
select(TaskRun.queue, func.min(TaskRun.started_at))
|
|
.where(TaskRun.status == "running", TaskRun.finished_at.is_(None))
|
|
.group_by(TaskRun.queue)
|
|
)
|
|
return {queue: started for queue, started in result if started is not None}
|
|
|
|
|
|
def _minutes_since(started: datetime | None, now: datetime) -> int | None:
|
|
"""Whole minutes, or None when nothing is running. Never negative: a row
|
|
written by a container whose clock is a few seconds ahead must read as 0
|
|
rather than as a task that starts in the future."""
|
|
if started is None:
|
|
return None
|
|
return max(0, int((now - started).total_seconds() // 60))
|
|
|
|
|
|
def _queue_depths_sync() -> dict[str, int | None]:
|
|
"""Redis LLEN per queue. None for one that did not answer — see lane_view.
|
|
|
|
Sync; the caller threads it. A per-queue try/except so one bad queue does
|
|
not cost the whole report, matching `api/system_activity._read_queues_sync`.
|
|
"""
|
|
import redis
|
|
|
|
from ..config import get_config
|
|
|
|
out: dict[str, int | None] = {}
|
|
try:
|
|
client = redis.Redis.from_url(get_config().celery_broker_url)
|
|
except Exception:
|
|
log.warning("worker_control: no broker for queue depths", exc_info=True)
|
|
return {q: None for lane in LANES for q in lane.queues}
|
|
for lane in LANES:
|
|
for queue in lane.queues:
|
|
try:
|
|
out[queue] = int(client.llen(queue))
|
|
except Exception: # noqa: BLE001 — a hiccup must not break the UI
|
|
out[queue] = None
|
|
return out
|
|
|
|
|
|
class LaneUpdateRefused(ValueError):
|
|
"""A requested value is outside what the lane may hold. Carries the reason
|
|
the UI shows — a greyed control with no explanation reads as a bug."""
|
|
|
|
|
|
async def store_lane_cap(
|
|
session: AsyncSession, lane: Lane, slots_cap: int,
|
|
) -> int:
|
|
"""Validate and store the cap. Returns the PREVIOUS cap. DB only.
|
|
|
|
Split from the live push for the reason `LaneSettings` gives at length: a
|
|
Postgres connection must not be held across a celery round trip. Everything
|
|
here is fast and finished with before `push_lane_cap` starts.
|
|
"""
|
|
rows = await _rows_by_name(session)
|
|
row = rows[lane.name]
|
|
|
|
ceiling = derived_ceiling(lane)
|
|
if slots_cap < 0:
|
|
raise LaneUpdateRefused("a cap cannot be negative")
|
|
if slots_cap > ceiling:
|
|
raise LaneUpdateRefused(
|
|
f"a cap of {slots_cap} is above what this container can hold "
|
|
f"({ceiling} for {lane.display_name})"
|
|
)
|
|
|
|
was_cap = row.slots_cap
|
|
row.slots_cap = slots_cap
|
|
await session.commit()
|
|
# The previous value, because the push needs the DIRECTION: lowering a cap
|
|
# has to reach the running lane now, and raising one has nothing to say.
|
|
return was_cap
|
|
|
|
|
|
async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
|
|
"""Make the running lane obey a cap that is already stored. NO database.
|
|
|
|
## What is pushed, and what is not
|
|
|
|
Consumers follow the cap immediately in BOTH directions: zero means off,
|
|
and off must take effect when it is asked for rather than up to a minute
|
|
later.
|
|
|
|
The pool is only ever pushed DOWNWARD. Raising a cap is permission, not a
|
|
request — growing on permission would put workers on a lane with nothing
|
|
to do — so the sizing pass spends it on its next tick if there is work.
|
|
That also makes the common case (raising a cap) free: no broker round trip
|
|
AT ALL, which is the difference between a control that answers instantly
|
|
and one that takes ten seconds. Keyed on the previous cap rather than on
|
|
"is it on" — the first cut only knew on/off, so it inspected on every
|
|
raise to find out whether the pool needed lowering, and the control it was
|
|
meant to make instant still waited out an inspect.
|
|
|
|
A failed push is not a failed setting. The value is already stored and the
|
|
sizing pass carries it within a minute; `applied: false` with a reason
|
|
lets the UI say "saved, not yet live" rather than "that didn't work"
|
|
(lesson #4202 — a live change that does not survive, with nothing saying
|
|
so).
|
|
"""
|
|
was_on, now_on = was_cap > 0, slots_cap > 0
|
|
applied, error = True, None
|
|
if now_on != was_on:
|
|
applied, error = await asyncio.to_thread(set_lane_enabled_sync, lane, now_on)
|
|
if applied and not now_on:
|
|
# Down to the floor at once. The pool cannot be emptied, so "off" is
|
|
# one parked process with its consumers cancelled.
|
|
applied, error = await asyncio.to_thread(
|
|
set_lane_slots_sync, lane, MIN_POOL_SLOTS,
|
|
)
|
|
elif applied and now_on and slots_cap < was_cap:
|
|
# LOWERED on a running lane. Only this direction needs a message, and
|
|
# only when the pool is actually above the new cap — so it reads the
|
|
# live pool rather than resizing blind. A raise never reaches here.
|
|
live = await asyncio.to_thread(inspect_lanes_sync)
|
|
current = live[lane.name].pool
|
|
if current is not None and current > slots_cap:
|
|
applied, error = await asyncio.to_thread(
|
|
set_lane_slots_sync, lane, slots_cap, live=live[lane.name],
|
|
)
|
|
|
|
# Raising the cap off zero is what triggers the model download (milestone
|
|
# 422 step 6). Never at boot: that made every start of the ML role reach
|
|
# HuggingFace for ~3.5GB, and rule 164 permits a runtime fetch only for a
|
|
# feature that is optional and clearly OFF.
|
|
#
|
|
# On the TRANSITION, so re-saving a cap on a lane already running does not
|
|
# re-enqueue. And only when the consumer change landed: enqueueing onto a
|
|
# queue nothing is consuming would leave the task pending with no
|
|
# explanation until the lane returns.
|
|
fetching = False
|
|
if now_on and not was_on and lane.models and applied:
|
|
fetching = _enqueue_model_fetch()
|
|
|
|
return {
|
|
"name": lane.name,
|
|
"slots_cap": slots_cap,
|
|
"ceiling": derived_ceiling(lane),
|
|
"enabled": now_on,
|
|
"applied": applied,
|
|
"apply_error": error,
|
|
# Tells the UI to say a download has started rather than leaving the
|
|
# operator to wonder why a lane they just turned on is busy.
|
|
"fetching_models": fetching,
|
|
}
|
|
|
|
|
|
def _enqueue_model_fetch() -> bool:
|
|
"""Queue the model download. Returns whether it was accepted.
|
|
|
|
Import inside the function: `backend.app.tasks.ml` pulls in torch, and web
|
|
must not pay that import cost on a module that every settings request
|
|
touches.
|
|
|
|
Never raises. A broker that will not take the task is worth reporting, but
|
|
the SETTING has already been stored and the lane is already enabled — so
|
|
failing the whole request here would roll back nothing and tell the
|
|
operator their change did not happen when it did.
|
|
"""
|
|
try:
|
|
from ..tasks.ml import ensure_models
|
|
|
|
ensure_models.delay()
|
|
return True
|
|
except Exception: # noqa: BLE001 — reported, never raised at a caller
|
|
log.warning("worker_control: could not enqueue the model fetch", exc_info=True)
|
|
return False
|
|
|
|
|
|
|
|
# --- the sizing pass: one sweep, always on ------------------------------------
|
|
#
|
|
# This replaced BOTH `reconcile_lanes_sync` (step 3) and `autoscale_lanes_sync`
|
|
# (step 7) on 2026-09-23. They were two enforcers over one number, and the
|
|
# whole of step 7's hardest reasoning — a stored value that is a FLOOR, a
|
|
# target of `max(stored, current)` so the reconcile does not undo what the
|
|
# autoscaler added — existed only to stop them fighting. Delete one of them and
|
|
# the problem is not solved, it is absent.
|
|
#
|
|
# Operator: *"auto should be always on, not a setting, so that idle instances
|
|
# quiet down when not running. the number that is visible and something the
|
|
# user can tweak and manage should be the cap itself the number of running
|
|
# workers is handled by the autoscaling function which is always on."*
|
|
#
|
|
# So there is one pass, it runs every minute, it reads the live pool rather
|
|
# than any stored number, and the only thing it obeys is the cap.
|
|
#
|
|
# It also subsumes what the reconcile existed for. `pool_grow` is not durable:
|
|
# a worker restarted by its supervisor comes back at its ENV concurrency,
|
|
# silently below what the lane should be running. This pass reads the live
|
|
# pool every minute and sizes from the backlog, so that worker is corrected on
|
|
# the next tick — sooner than the five-minute reconcile managed, and without a
|
|
# second sweep that could disagree with this one.
|
|
|
|
# How much work justifies a slot. `pending` is depth + reserved, so it already
|
|
# counts what celery has prefetched into worker memory — one task, one slot.
|
|
#
|
|
# Growth is IMMEDIATE and shrink is one slot per tick, deliberately asymmetric.
|
|
# A backlog of four thousand should not take an hour to reach the cap, and a
|
|
# lane that idles for one minute should not drop every process it has: the
|
|
# cost of being one slot too large for a minute is a sleeping process, and the
|
|
# cost of being too small is work not happening. For ML the asymmetry matters
|
|
# most — every new slot reloads a multi-GB model, so the slow shrink is what
|
|
# stops a quiet patch from paying that cost again a minute later.
|
|
SHRINK_STEP = 1
|
|
|
|
|
|
@dataclass
|
|
class LaneSizing:
|
|
"""What the pass did to one lane, and why — in the operator's terms.
|
|
|
|
A reason on every outcome including "held", because a sizing pass that
|
|
only speaks when it acts is one nobody can debug when it does not.
|
|
"""
|
|
|
|
lane: str
|
|
action: str # "grew" | "shrank" | "held" | "skipped"
|
|
slots: int
|
|
reason: str
|
|
|
|
|
|
def wanted_slots(cap: int, active: int, pending: int | None) -> int:
|
|
"""How many workers this lane has work for right now, within its cap.
|
|
|
|
One slot per task in flight or waiting, floored at one process and
|
|
ceilinged by the cap. `pending` of None means the broker did not answer
|
|
for this lane's queues — an unknown backlog is not an empty one (snippet
|
|
#3969), so it contributes nothing rather than being read as zero.
|
|
|
|
A cap of zero still returns one: billiard cannot run an empty pool, and
|
|
the parked process is what `add_consumer` lands on when the cap goes back
|
|
up. "Off" is expressed by cancelling consumers, not by emptying the pool.
|
|
"""
|
|
if cap <= 0:
|
|
return MIN_POOL_SLOTS
|
|
return max(MIN_POOL_SLOTS, min(cap, active + (pending or 0)))
|
|
|
|
|
|
def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
|
|
"""Size every lane to its backlog, within the cap. The whole control loop.
|
|
|
|
`caps` is lane name -> slots_cap, read from the database by the caller.
|
|
This function touches no database: the celery task that schedules it owns
|
|
the session, and keeping the DB out of here is what lets it be called from
|
|
anywhere that already knows the caps.
|
|
|
|
## It must converge and then go quiet
|
|
|
|
One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing to a
|
|
replica already at its target. A settled system therefore performs one
|
|
broker round trip plus one LLEN sweep per tick and sends no control
|
|
messages at all — the reachable fixed point lesson #4183 is about. An
|
|
enforcer that re-sent a grow of zero every tick would churn forever and
|
|
bury a real correction in its own noise.
|
|
|
|
## An absent lane is SKIPPED, not corrected
|
|
|
|
`present=False` means nothing answered — a worker restarting, or an
|
|
unreachable broker. It does NOT mean zero slots. Deciding from that would
|
|
be a verdict drawn from an unswept read, and here it is worse than
|
|
useless: there is nothing to send the message to.
|
|
"""
|
|
live = inspect_lanes_sync()
|
|
depths = _queue_depths_sync()
|
|
out: list[LaneSizing] = []
|
|
|
|
for lane in LANES:
|
|
cap = caps.get(lane.name)
|
|
if cap is None:
|
|
continue
|
|
state = live[lane.name]
|
|
if not state.present:
|
|
out.append(LaneSizing(lane.name, "skipped", 0, "lane is not answering"))
|
|
continue
|
|
|
|
# Consumers first, and only when they DISAGREE. Sending add_consumer
|
|
# for every queue on every tick of a settled system is the exact churn
|
|
# above, and invisible: add_consumer on a queue already consumed is
|
|
# harmless and reports success.
|
|
should_consume = cap > 0
|
|
if should_consume != state.consuming.issuperset(lane.queues):
|
|
ok, err = set_lane_enabled_sync(lane, should_consume, live=state)
|
|
if not ok:
|
|
out.append(LaneSizing(
|
|
lane.name, "held", state.pool or 0,
|
|
f"could not {'start' if should_consume else 'stop'} "
|
|
f"consuming: {err}",
|
|
))
|
|
continue
|
|
|
|
current = state.pool
|
|
if current is None:
|
|
out.append(LaneSizing(
|
|
lane.name, "held", 0, "worker did not report its pool size",
|
|
))
|
|
continue
|
|
|
|
known = [depths.get(q) for q in lane.queues]
|
|
depth = (
|
|
sum(d for d in known if d is not None)
|
|
if any(d is not None for d in known) else None
|
|
)
|
|
pending = None if depth is None else depth + state.reserved
|
|
want = wanted_slots(cap, state.active, pending)
|
|
|
|
if want > current:
|
|
new = want
|
|
verb = "grew"
|
|
elif want < current:
|
|
# One at a time on the way down. See SHRINK_STEP.
|
|
new = max(want, current - SHRINK_STEP)
|
|
verb = "shrank"
|
|
else:
|
|
out.append(LaneSizing(
|
|
lane.name, "held", current,
|
|
f"{pending if pending is not None else '?'} waiting, "
|
|
f"{state.active} busy, cap {cap}",
|
|
))
|
|
continue
|
|
|
|
ok, err = set_lane_slots_sync(lane, new, live=state)
|
|
if not ok:
|
|
out.append(LaneSizing(
|
|
lane.name, "held", current, f"could not resize: {err}",
|
|
))
|
|
continue
|
|
out.append(LaneSizing(
|
|
lane.name, verb, new,
|
|
f"{pending if pending is not None else '?'} waiting, "
|
|
f"{state.active} busy, cap {cap}",
|
|
))
|
|
return out
|