FabledCurator/backend/app/services/worker_control.py

"""Read and change a lane's live pool, over the broker.

Milestone 422 step 2. The half of the milestone that does something.

## No docker socket is involved, and that is the point

Milestone 365 put "acting on the state" out of scope because restarting a
dead worker needs a docker socket the web container deliberately does not
have. That is true of RESTARTING a container. It is not true of changing how
much work a RUNNING worker does: celery's remote control sends a message over
the broker and the worker resizes its own pool. Same Redis the app already
uses, no new privilege, no new surface.

    pool_grow / pool_shrink   how many slots a lane runs
    add_consumer / cancel_consumer   whether it consumes its queues at all

The operator ruled the socket out independently (2026-09-22: *"this feature
is a very invasive idea in my mind and I'd like to avoid it"*), and nothing
here raises the question.

## The setting is PER PROCESS, not per lane total

`pool_grow(n, destination=[...])` adds n slots to EACH destination it names.
While the stack still runs several containers per lane — the operator's
production `worker` is `replicas: 2` — a single delta applied to a lane's
total would be wrong for every replica.

So `slots` means what `CELERY_CONCURRENCY` means: the pool size of one
process. The reconcile below drives EACH replica to that number
independently, computing its own delta from that replica's current pool, so
replicas that have drifted apart (one restarted, one was grown) converge
rather than being moved in lockstep from a shared baseline.

After step 5 there is one process per lane and the distinction disappears.
It matters now, and getting it wrong now would be invisible — the totals
would simply be double what the UI claimed.

## Why reserved() is read alongside the queue depth

Celery PREFETCHES: a worker pulls more messages than it can run and holds
them in memory. Those have already left the Redis list, so `LLEN` — which is
what `/api/system/activity/queues` reports — can read 0 while thirty tasks
are waiting inside a worker. Any judgement about backlog that uses only LLEN
under-reports, which matters for the UI and is disqualifying for step 7's
autoscaler.
"""

from __future__ import annotations

import asyncio
import logging
from dataclasses import dataclass, field
from datetime import UTC, datetime

from sqlalchemy import func, select
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.ext.asyncio import AsyncSession

from ..models import TaskRun, WorkerLane, WorkerLaneSample
from .worker_lanes import (
    LANES,
    LANES_BY_QUEUE_KEY,
    MIN_POOL_SLOTS,
    Lane,
    derived_ceiling,
    lane_for_node,
)

log = logging.getLogger(__name__)

# celery control is a broker round trip on a request path, so it gets a
# deadline (rule 156) — the same reasoning and the same budget as
# service_roster's inspect. A broker that stopped answering must make this
# report "not present", which is true, rather than hang the page.
CONTROL_TIMEOUT_SECONDS = 2.0


# The WORST case of `inspect_lanes_sync`, for callers that need a deadline.
#
# One broadcast plus three targeted reads. The targeted three normally return
# as soon as the named nodes answer; each can still cost a full timeout if a
# node disappears mid-read, so the bound stays four.
CONTROL_ROUND_TRIPS = 4

# Slack for the `asyncio.to_thread` handoff. A budget equal to the work is a
# budget that fails under load — the roster carried exactly that bug into the
# operator's first consolidated deploy and logged a TimeoutError per refresh
# while the inspect calls underneath were working fine.
CONTROL_SLACK_SECONDS = 3.0

INSPECT_BUDGET_SECONDS = (
    CONTROL_TIMEOUT_SECONDS * CONTROL_ROUND_TRIPS + CONTROL_SLACK_SECONDS
)


@dataclass
class LaneLiveState:
    """What `celery inspect` says about one lane right now.

    `present=False` is NOT "zero slots" — it is "nothing answered". A lane
    whose worker is restarting, or whose broker is unreachable, must read as
    unknown rather than as stopped: an unswept absence is not a verdict
    (snippet #3969). The reconcile in step 3 skips an absent lane rather than
    correcting it, which is only safe because this distinction is kept.
    """

    present: bool = False
    replicas: int = 0
    active: int = 0
    reserved: int = 0
    hostnames: list[str] = field(default_factory=list)
    # The queues this lane is actually consuming right now, across replicas.
    # Distinct from the lane's CONFIGURED queues: `cancel_consumer` stops a
    # worker consuming one without changing what it was started with, which
    # is how `enabled=false` is implemented. The reconcile needs this to tell
    # "already disabled" from "needs disabling" — without it, it would re-send
    # add_consumer for every queue on every tick forever (lesson #4183).
    consuming: set[str] = field(default_factory=set)
    # Pool size PER HOSTNAME, not aggregated. The resize below computes each
    # replica's own delta from its own current pool, so replicas that have
    # drifted apart converge instead of being moved in lockstep from a shared
    # baseline — which is what an aggregate here would silently reintroduce.
    pools: dict[str, int] = field(default_factory=dict)

    @property
    def pool(self) -> int | None:
        """One number for the UI. `max` rather than a sum: `slots` means the
        pool size of ONE process (see the module docstring), so the largest
        replica is the honest answer to "what is this lane set to". None when
        no replica reported — unknown, never zero."""
        return max(self.pools.values()) if self.pools else None

    @property
    def capacity(self) -> int:
        """Total slots across replicas — how many tasks this lane can run at
        once. Distinct from `pool`, and the two must not be confused: `pool`
        is the DIAL (one process's size, what grow/shrink move), `capacity` is
        the CAPABILITY. Asking "is this lane saturated" compares `active`,
        which is summed across replicas, against this — against `pool` it
        would call two half-busy replicas of 4 saturated at 4 active."""
        return sum(self.pools.values())


def _lane_for_queues(queues: tuple[str, ...]) -> Lane | None:
    return LANES_BY_QUEUE_KEY.get(tuple(sorted(queues)))


def inspect_lanes_sync() -> dict[str, LaneLiveState]:
    """Live state per lane name. Sync — callers wrap in asyncio.to_thread.

    Never raises. Every lane is present in the result; ones nothing answered
    for carry `present=False`, so a caller cannot accidentally read a missing
    lane as an empty one by iterating only what came back.
    """
    out = {lane.name: LaneLiveState() for lane in LANES}
    try:
        from ..celery_app import celery as celery_app

        # ONE broadcast, then three TARGETED reads.
        #
        # A broadcast with no `destination` cannot know how many replies to
        # expect, so it waits out its whole timeout rather than returning on
        # the last one. Four of those is four full timeouts — about eight
        # seconds — and `lane_view` sits on the Settings card, so that was the
        # load time of the Worker lanes page every time it was opened.
        #
        # Naming the destinations lets celery stop as soon as those nodes have
        # answered, which for workers in this same container is milliseconds.
        # The worst case is unchanged: a node that vanishes between the
        # broadcast and the targeted reads costs a full timeout waiting for a
        # reply that is not coming.
        insp = celery_app.control.inspect(timeout=CONTROL_TIMEOUT_SECONDS)
        active_queues = insp.active_queues() or {}

        # Nothing answered — and the three reads below exist only to describe
        # what did. Returning here also makes the broker-down case FAST
        # (one timeout, not four), which is exactly when the healthcheck and
        # the card need an answer rather than a long wait.
        if not active_queues:
            return out

        targeted = celery_app.control.inspect(
            destination=sorted(active_queues),
            timeout=CONTROL_TIMEOUT_SECONDS,
        )
        stats = targeted.stats() or {}
        active = targeted.active() or {}
        reserved = targeted.reserved() or {}
    except Exception:
        log.warning("worker_control: celery inspect failed", exc_info=True)
        return out

    for hostname, queues in active_queues.items():
        # The NODE NAME first — see `lane_for_node`. A lane at cap 0 has its
        # consumers cancelled and answers with an empty queue list, which
        # matches no lane, so attributing by queues alone dropped every lane
        # the operator had turned off and reported it as "not answering".
        lane = lane_for_node(hostname) or _lane_for_queues(
            tuple(q["name"] for q in queues)
        )
        if lane is None:
            # A deployment slicing CELERY_QUEUES differently. Reported by the
            # roster under its raw queue list; it simply has no lane row to
            # control, which is honest rather than an error.
            continue
        state = out[lane.name]
        state.present = True
        state.replicas += 1
        state.hostnames.append(hostname)
        state.active += len(active.get(hostname, []))
        state.reserved += len(reserved.get(hostname, []))
        state.consuming.update(q["name"] for q in queues)

        # `pool.max-concurrency` is the number pool_grow/pool_shrink move and
        # the number the UI shows. Absent on a worker whose stats did not
        # answer, which leaves pool=None — unknown, not zero.
        pool = (stats.get(hostname) or {}).get("pool", {}).get("max-concurrency")
        if isinstance(pool, int):
            state.pools[hostname] = pool

    for state in out.values():
        state.hostnames.sort()
    return out


def effective_slots(target: int) -> int:
    """What a pool can actually be set to. Never below one process.

    Used wherever a target is COMPARED as well as wherever one is sent: a
    reconcile that compares against the unclamped number sees a difference
    that no control message can ever close, and re-sends it every tick.
    """
    return max(MIN_POOL_SLOTS, target)


def set_lane_slots_sync(
    lane: Lane, target: int, live: LaneLiveState | None = None,
) -> tuple[bool, str | None]:
    """Drive every replica of `lane` to `target` slots. Returns (applied, err).

    Per-replica deltas rather than one shared delta: see the module docstring.
    A replica already at the target is issued nothing at all, which is what
    makes step 3's periodic reconcile converge instead of re-sending a grow of
    zero forever (lesson #4183 — an enforcer without a reachable fixed point
    re-does its own work every tick).

    `applied=False` is not a failure of the SETTING. The caller has already
    stored the value; this says only that the live push did not land, and the
    reconcile will carry it when the lane answers again.
    """
    target = effective_slots(target)
    try:
        from ..celery_app import celery as celery_app

        if live is None:
            live = inspect_lanes_sync()[lane.name]
        if not live.present:
            return False, "lane is not running"
        if not live.pools:
            return False, "worker did not report its pool size"

        control = celery_app.control
        unreported = [h for h in live.hostnames if h not in live.pools]
        for hostname, current in live.pools.items():
            delta = target - current
            if delta > 0:
                control.pool_grow(delta, destination=[hostname])
            elif delta < 0:
                control.pool_shrink(-delta, destination=[hostname])
        if unreported:
            # Resized what could be resized, and said which could not. Silence
            # here would leave a replica running at a size the UI claims it is
            # not, with nothing anywhere recording the gap.
            return False, f"no pool size reported by {', '.join(sorted(unreported))}"
        return True, None
    except Exception as exc:  # noqa: BLE001 — reported, never raised at a caller
        log.warning("worker_control: could not resize %s", lane.name, exc_info=True)
        return False, str(exc)


def set_lane_enabled_sync(
    lane: Lane, enabled: bool, live: LaneLiveState | None = None,
) -> tuple[bool, str | None]:
    """Start or stop `lane` consuming its queues, without killing the process.

    `cancel_consumer` rather than a shutdown: a stopped consumer keeps its
    worker alive and answering `inspect`, so a disabled lane stays visible and
    can be turned back on. A killed worker would read as absent, which is the
    same signal as a crash — and the whole point of the roster (#365) is that
    those two must not look alike.
    """
    try:
        from ..celery_app import celery as celery_app

        if live is None:
            live = inspect_lanes_sync()[lane.name]
        if not live.present:
            return False, "lane is not running"
        control = celery_app.control
        for queue in lane.queues:
            if enabled:
                control.add_consumer(queue, destination=live.hostnames)
            else:
                control.cancel_consumer(queue, destination=live.hostnames)
        return True, None
    except Exception as exc:  # noqa: BLE001
        log.warning(
            "worker_control: could not %s %s",
            "enable" if enabled else "disable", lane.name, exc_info=True,
        )
        return False, str(exc)


# --- the settings half, which is async ----------------------------------------
#
# Sync celery control above, async DB below, in one module. Same split
# `service_roster` already runs (`_inspect_celery_sync` beside `touch_service`)
# — the boundary is the transport, not the concern, and "control the workers"
# is one concern.


async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]:
    """Every lane's row, creating any that are missing from its LANES defaults.

    Self-heals rather than depending on a migration having run for a lane
    added later: alembic 0103 seeded the four that existed on 2026-09-22, and
    a fifth added to LANES afterwards gets its row the first time anything
    asks. Without this, a new lane would read as absent and the UI would
    simply not show it.
    """
    rows = {
        row.name: row
        for row in (await session.execute(select(WorkerLane))).scalars()
    }
    missing = [lane for lane in LANES if lane.name not in rows]
    for lane in missing:
        row = WorkerLane(name=lane.name, slots_cap=lane.default_slots_cap)
        session.add(row)
        rows[lane.name] = row
    if missing:
        await session.commit()
    return rows


@dataclass(frozen=True)
class LaneSample:
    """What the sizing sweep last measured about one lane.

    The same fields `LaneLiveState` carries, plus the queue depth and WHEN —
    because this one is read from a table rather than from the broker, and a
    reading with no timestamp invites being presented as current.

    `measured_at=None` means no sweep has written this lane yet: a fresh
    install inside its first period, or a stack whose beat is not running.
    Distinct from `present=False` (something asked, nothing answered), and the
    UI says different things about the two.
    """

    present: bool = False
    replicas: int = 0
    pool: int | None = None
    active: int = 0
    reserved: int = 0
    queue_depth: int | None = None
    measured_at: datetime | None = None


def _lane_depth(lane: Lane, depths: dict[str, int | None]) -> int | None:
    """A lane's backlog across its queues — None when NOTHING answered.

    A queue the broker did not answer for must not be summed as zero: an
    unknown depth is not an empty one, and reporting a buried lane as idle is
    the direction that matters.
    """
    known = [depths.get(q) for q in lane.queues]
    if not any(d is not None for d in known):
        return None
    return sum(d for d in known if d is not None)


def store_lane_samples_sync(session, live: dict[str, LaneLiveState], depths) -> None:
    """Write what the sweep just measured. SYNC — the celery task owns a sync
    session, and this is the only place these rows are written.

    Upsert per lane, last writer wins, same shape as `service_roster`'s
    `touch_service`: two processes sweeping at once is a benign race that
    needs no coordination, because both are recording what they actually saw.

    A lane that did not answer is STILL written, with `present=False`. Skipping
    it would leave the previous reading in place and let the page go on showing
    a pool that is no longer there — the stale row would read as a current one
    (lesson #4202: the row is the thing that has to change).
    """
    now = datetime.now(UTC)
    for lane in LANES:
        state = live.get(lane.name) or LaneLiveState()
        values = {
            "lane": lane.name,
            "present": state.present,
            "replicas": state.replicas,
            "pool": state.pool,
            "active": state.active,
            "reserved": state.reserved,
            "queue_depth": _lane_depth(lane, depths),
            "measured_at": now,
        }
        stmt = pg_insert(WorkerLaneSample).values(**values)
        session.execute(stmt.on_conflict_do_update(
            index_elements=[WorkerLaneSample.lane],
            set_={k: v for k, v in values.items() if k != "lane"},
        ))
    session.commit()


@dataclass
class LaneSettings:
    """What the DATABASE knows about the lanes — read and finished with before
    anything touches the broker.

    This exists because holding a Postgres connection across a celery round
    trip is what made the System tab block the whole site (operator,
    2026-09-23: *"something about changing the cap number is blocking to the
    website"*).

    `lane_view` used to take the session and keep it open through an inspect
    whose budget is eleven seconds — and that page polls every fifteen. With a
    lane not answering, every inspect ran to nearly its full budget, so each
    poll pinned a connection for ten seconds. SQLAlchemy's default pool is
    five connections plus ten overflow; a couple of browser tabs, the health
    endpoint doing the same thing, and a cap change adding two more inspects
    exhausts that, and every OTHER request then waits on a connection.

    Nothing was slow in itself. The slowness was a scarce resource held across
    it, which is why it surfaced as the whole site stalling rather than as one
    slow page.
    """

    caps: dict[str, int]
    oldest_by_queue: dict[str, datetime]
    # The sizing sweep's last reading per lane. Since 2026-09-23 this is where
    # the live numbers come from: the endpoint no longer inspects at all.
    samples: dict[str, LaneSample] = field(default_factory=dict)


async def lane_settings(session: AsyncSession) -> LaneSettings:
    """Every DB read the lane view needs, in one short-lived session.

    Which is now ALL of them. `lane_view` below takes what this returns and
    talks to nothing.
    """
    rows = await _rows_by_name(session)
    samples = {
        row.lane: LaneSample(
            present=row.present,
            replicas=row.replicas,
            pool=row.pool,
            active=row.active,
            reserved=row.reserved,
            queue_depth=row.queue_depth,
            measured_at=row.measured_at,
        )
        for row in (
            await session.execute(select(WorkerLaneSample))
        ).scalars()
    }
    return LaneSettings(
        caps={name: row.slots_cap for name, row in rows.items()},
        oldest_by_queue=await _oldest_running_by_queue(session),
        samples=samples,
    )


def lane_view(settings: LaneSettings) -> list[dict]:
    """Every lane: what is configured, what was last measured, what it may
    grow to. NO broker call, and no database — `settings` is the whole input.

    ## It used to inspect, on every request

    Four broadcast round trips on an eleven-second budget, on a page that
    polls every fifteen seconds. Operator, 2026-09-23: *"there is a repull
    every time this page loads — is there a reason this info isn't being
    tracked in the background and stored in some way?"*

    There was one, and it had expired. The docstring here used to say the
    endpoint was deliberately uncached because *"this is the surface an
    operator watches while dragging a stepper, and a cached reply would show
    them the value from before their own change"*. True while a cap change
    refetched the table — and that refetch is exactly what was removed in
    `1353d34`, so the UI now patches its own row from the write's reply and
    nothing depends on this being live.

    Meanwhile `size_worker_lanes` was already inspecting on a timer to decide
    pool sizes: the same numbers, computed, used, and discarded, while the
    browser asked the broker for them again four times a minute.

    So the sweep writes `worker_lane_sample` and this reads it. The reading is
    up to `SWEEP_PERIOD_SECONDS` old, and `measured_at` travels with it so the
    UI can say so rather than implying it is current.

    `pending` is still the honest backlog — depth PLUS reserved — because
    celery prefetches and LLEN alone reads 0 while a worker holds tasks in
    memory.
    """
    oldest = settings.oldest_by_queue
    now = datetime.now(UTC)
    out = []
    for lane in LANES:
        cap = settings.caps[lane.name]
        # A lane with no row yet is not-measured, which is distinct from
        # measured-as-absent. The default carries `measured_at=None`, and the
        # UI says "not measured yet" rather than "not answering".
        sample = settings.samples.get(lane.name) or LaneSample()
        depth = sample.queue_depth
        out.append({
            "name": lane.name,
            "display_name": lane.display_name,
            "queues": list(lane.queues),
            "slots_cap": cap,
            "ceiling": derived_ceiling(lane),
            # DERIVED, never stored. A cap of zero means no consumers, so
            # "off" and "may use no workers" cannot disagree.
            "enabled": cap > 0,
            "memory_bound": lane.memory_bound,
            "optional": lane.optional,
            # What raising this lane's cap will download, so the UI can say
            # WHICH model and how big BEFORE the first slot is asked for
            # rather than after a multi-GB fetch has started. `measured`
            # travels with the numbers: the UI must not present an estimate
            # as a fact.
            "models": [
                {
                    "repo": m.repo,
                    "download_bytes": m.approx_download_bytes,
                    "resident_bytes": m.approx_resident_bytes,
                    "measured": m.measured,
                }
                for m in lane.models
            ],
            "live": {
                "present": sample.present,
                "replicas": sample.replicas,
                "pool": sample.pool,
                "active": sample.active,
                "reserved": sample.reserved,
            },
            "queue_depth": depth,
            "pending": None if depth is None else depth + sample.reserved,
            # When the numbers above were read. Per lane rather than one for
            # the response, because a lane whose row has never been written
            # has no reading at all and must not borrow another lane's.
            "measured_at": (
                sample.measured_at.isoformat() if sample.measured_at else None
            ),
            # How long the oldest still-running task on this lane has been
            # going, in minutes. Read from `task_run`, not from the sweep, so
            # this one IS current. The operator asked for a trigger here —
            # grow a lane whose tasks run past some duration — and it stayed a
            # REPORT: a long task does not finish sooner because the lane
            # gained a slot, so scaling on it would spend memory to change
            # nothing. Shown so they can see a lane wedged on one slow job,
            # which is the genuinely useful half of the idea.
            "oldest_running_minutes": _minutes_since(
                min(
                    (oldest[q] for q in lane.queues if q in oldest),
                    default=None,
                ),
                now,
            ),
        })
    return out


async def _oldest_running_by_queue(session: AsyncSession) -> dict[str, datetime]:
    """When the longest-running unfinished task on each queue started.

    Read from `task_run`, which is OUR OWN table on OUR OWN wall clock, and
    deliberately not from celery's `inspect active()`. Those entries carry a
    `time_start` taken from the WORKER's `time.monotonic()` — a clock with an
    arbitrary origin per process. Subtracting it from this process's wall
    clock produces a number that looks like a duration and is meaningless, and
    it would be meaningless in the direction that matters: plausible.

    `task_run` also already carries the per-queue staleness thresholds the
    recovery sweep uses, so a row still `running` here is one the system
    itself considers legitimately in flight rather than abandoned.
    """
    result = await session.execute(
        select(TaskRun.queue, func.min(TaskRun.started_at))
        .where(TaskRun.status == "running", TaskRun.finished_at.is_(None))
        .group_by(TaskRun.queue)
    )
    return {queue: started for queue, started in result if started is not None}


def _minutes_since(started: datetime | None, now: datetime) -> int | None:
    """Whole minutes, or None when nothing is running. Never negative: a row
    written by a container whose clock is a few seconds ahead must read as 0
    rather than as a task that starts in the future."""
    if started is None:
        return None
    return max(0, int((now - started).total_seconds() // 60))


def _queue_depths_sync() -> dict[str, int | None]:
    """Redis LLEN per queue. None for one that did not answer — see lane_view.

    Sync; the caller threads it. A per-queue try/except so one bad queue does
    not cost the whole report, matching `api/system_activity._read_queues_sync`.
    """
    import redis

    from ..config import get_config

    out: dict[str, int | None] = {}
    try:
        client = redis.Redis.from_url(get_config().celery_broker_url)
    except Exception:
        log.warning("worker_control: no broker for queue depths", exc_info=True)
        return {q: None for lane in LANES for q in lane.queues}
    for lane in LANES:
        for queue in lane.queues:
            try:
                out[queue] = int(client.llen(queue))
            except Exception:  # noqa: BLE001 — a hiccup must not break the UI
                out[queue] = None
    return out


class LaneUpdateRefused(ValueError):
    """A requested value is outside what the lane may hold. Carries the reason
    the UI shows — a greyed control with no explanation reads as a bug."""


async def store_lane_cap(
    session: AsyncSession, lane: Lane, slots_cap: int,
) -> int:
    """Validate and store the cap. Returns the PREVIOUS cap. DB only.

    Split from the live push for the reason `LaneSettings` gives at length: a
    Postgres connection must not be held across a celery round trip. Everything
    here is fast and finished with before `push_lane_cap` starts.
    """
    rows = await _rows_by_name(session)
    row = rows[lane.name]

    ceiling = derived_ceiling(lane)
    if slots_cap < 0:
        raise LaneUpdateRefused("a cap cannot be negative")
    if slots_cap > ceiling:
        raise LaneUpdateRefused(
            f"a cap of {slots_cap} is above what this container can hold "
            f"({ceiling} for {lane.display_name})"
        )

    was_cap = row.slots_cap
    row.slots_cap = slots_cap
    await session.commit()
    # The previous value, because the push needs the DIRECTION: lowering a cap
    # has to reach the running lane now, and raising one has nothing to say.
    return was_cap


async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
    """Make the running lane obey a cap that is already stored. NO database.

    Runs OFF the request path since 2026-09-23 — the endpoint stores the cap,
    answers, and hands this to a background task (operator: *"the change
    should be queued so that it isn't blocking of the webui"*). Nothing here
    changed as a result except who waits for it: the return value is now read
    by the log rather than by a browser, and every branch below already
    treated failure as "the sizing pass will carry it".

    ## What is pushed, and what is not

    Consumers follow the cap immediately in BOTH directions: zero means off,
    and off must take effect when it is asked for rather than up to a minute
    later.

    The pool is only ever pushed DOWNWARD. Raising a cap is permission, not a
    request — growing on permission would put workers on a lane with nothing
    to do — so the sizing pass spends it on its next tick if there is work.
    That also makes the common case (raising a cap) free: no broker round trip
    AT ALL, which is the difference between a control that answers instantly
    and one that takes ten seconds. Keyed on the previous cap rather than on
    "is it on" — the first cut only knew on/off, so it inspected on every
    raise to find out whether the pool needed lowering, and the control it was
    meant to make instant still waited out an inspect.

    A failed push is not a failed setting. The value is already stored and the
    sizing pass carries it within a minute; `applied: false` with a reason
    lets the UI say "saved, not yet live" rather than "that didn't work"
    (lesson #4202 — a live change that does not survive, with nothing saying
    so).
    """
    was_on, now_on = was_cap > 0, slots_cap > 0
    applied, error = True, None
    if now_on != was_on:
        applied, error = await asyncio.to_thread(set_lane_enabled_sync, lane, now_on)
    if applied and not now_on:
        # Down to the floor at once. The pool cannot be emptied, so "off" is
        # one parked process with its consumers cancelled.
        applied, error = await asyncio.to_thread(
            set_lane_slots_sync, lane, MIN_POOL_SLOTS,
        )
    elif applied and now_on and slots_cap < was_cap:
        # LOWERED on a running lane. Only this direction needs a message, and
        # only when the pool is actually above the new cap — so it reads the
        # live pool rather than resizing blind. A raise never reaches here.
        #
        # Bounded (rule 156): `to_thread` on its own is an await with no
        # deadline, and this runs in a background task where a hang would be
        # silent rather than visible as a slow page. On a timeout the lane is
        # simply not resized here and the sizing sweep carries it.
        try:
            live = await asyncio.wait_for(
                asyncio.to_thread(inspect_lanes_sync),
                timeout=INSPECT_BUDGET_SECONDS,
            )
        except TimeoutError:
            log.warning(
                "worker_control: inspect exceeded %ss lowering %s; leaving the "
                "pool to the sizing pass", INSPECT_BUDGET_SECONDS, lane.name,
            )
            return _cap_result(lane, slots_cap, now_on, applied, error, False)
        current = live[lane.name].pool
        if current is not None and current > slots_cap:
            applied, error = await asyncio.to_thread(
                set_lane_slots_sync, lane, slots_cap, live=live[lane.name],
            )

    # Raising the cap off zero is what triggers the model download (milestone
    # 422 step 6). Never at boot: that made every start of the ML role reach
    # HuggingFace for ~3.5GB, and rule 164 permits a runtime fetch only for a
    # feature that is optional and clearly OFF.
    #
    # On the TRANSITION, so re-saving a cap on a lane already running does not
    # re-enqueue.
    #
    # NOT gated on the consumer change having landed, which it was until
    # 2026-09-23. The reasoning then was that enqueueing onto a queue nothing
    # consumes leaves the task pending — true, and it is the right place for
    # it to wait. Gated, a cap raised while the lane was restarting stored the
    # cap, let the sizing pass start the consumers a minute later, and left
    # the lane running with no model, because nothing else ever asks for one.
    # A task parked on the `ml` queue is picked up the moment that happens.
    fetching = False
    if now_on and not was_on and lane.models:
        fetching = _enqueue_model_fetch()

    # Nobody is waiting on this any more, so the log is where a push that did
    # not land has to be visible. Not an error: the value is stored and the
    # sizing pass carries it within a minute.
    if not applied:
        log.info(
            "worker_control: %s cap %s stored, not pushed (%s); "
            "the sizing pass will carry it",
            lane.name, slots_cap, error,
        )

    return _cap_result(lane, slots_cap, now_on, applied, error, fetching)


def _cap_result(
    lane: Lane, slots_cap: int, now_on: bool, applied: bool,
    error: str | None, fetching: bool,
) -> dict:
    """The push's outcome. One builder, because `push_lane_cap` has two exits
    and a second literal would be free to disagree with the first."""
    return {
        "name": lane.name,
        "slots_cap": slots_cap,
        "ceiling": derived_ceiling(lane),
        "enabled": now_on,
        "applied": applied,
        "apply_error": error,
        # Tells the UI to say a download has started rather than leaving the
        # operator to wonder why a lane they just turned on is busy.
        "fetching_models": fetching,
    }


def _enqueue_model_fetch() -> bool:
    """Queue the model download. Returns whether it was accepted.

    Import inside the function: `backend.app.tasks.ml` pulls in torch, and web
    must not pay that import cost on a module that every settings request
    touches.

    Never raises. A broker that will not take the task is worth reporting, but
    the SETTING has already been stored and the lane is already enabled — so
    failing the whole request here would roll back nothing and tell the
    operator their change did not happen when it did.
    """
    try:
        from ..tasks.ml import ensure_models

        ensure_models.delay()
        return True
    except Exception:  # noqa: BLE001 — reported, never raised at a caller
        log.warning("worker_control: could not enqueue the model fetch", exc_info=True)
        return False



# --- the sizing pass: one sweep, always on ------------------------------------
#
# This replaced BOTH `reconcile_lanes_sync` (step 3) and `autoscale_lanes_sync`
# (step 7) on 2026-09-23. They were two enforcers over one number, and the
# whole of step 7's hardest reasoning — a stored value that is a FLOOR, a
# target of `max(stored, current)` so the reconcile does not undo what the
# autoscaler added — existed only to stop them fighting. Delete one of them and
# the problem is not solved, it is absent.
#
# Operator: *"auto should be always on, not a setting, so that idle instances
# quiet down when not running. the number that is visible and something the
# user can tweak and manage should be the cap itself the number of running
# workers is handled by the autoscaling function which is always on."*
#
# So there is one pass, it runs every minute, it reads the live pool rather
# than any stored number, and the only thing it obeys is the cap.
#
# It also subsumes what the reconcile existed for. `pool_grow` is not durable:
# a worker restarted by its supervisor comes back at its ENV concurrency,
# silently below what the lane should be running. This pass reads the live
# pool every minute and sizes from the backlog, so that worker is corrected on
# the next tick — sooner than the five-minute reconcile managed, and without a
# second sweep that could disagree with this one.

# How much work justifies a slot. `pending` is depth + reserved, so it already
# counts what celery has prefetched into worker memory — one task, one slot.
#
# Growth is IMMEDIATE and shrink is one slot per tick, deliberately asymmetric.
# A backlog of four thousand should not take an hour to reach the cap, and a
# lane that idles for one minute should not drop every process it has: the
# cost of being one slot too large for a minute is a sleeping process, and the
# cost of being too small is work not happening. For ML the asymmetry matters
# most — every new slot reloads a multi-GB model, so the slow shrink is what
# stops a quiet patch from paying that cost again a minute later.
SHRINK_STEP = 1


@dataclass
class LaneSizing:
    """What the pass did to one lane, and why — in the operator's terms.

    A reason on every outcome including "held", because a sizing pass that
    only speaks when it acts is one nobody can debug when it does not.
    """

    lane: str
    action: str  # "grew" | "shrank" | "held" | "skipped"
    slots: int
    reason: str


def wanted_slots(cap: int, active: int, pending: int | None) -> int:
    """How many workers this lane has work for right now, within its cap.

    One slot per task in flight or waiting, floored at one process and
    ceilinged by the cap. `pending` of None means the broker did not answer
    for this lane's queues — an unknown backlog is not an empty one (snippet
    #3969), so it contributes nothing rather than being read as zero.

    A cap of zero still returns one: billiard cannot run an empty pool, and
    the parked process is what `add_consumer` lands on when the cap goes back
    up. "Off" is expressed by cancelling consumers, not by emptying the pool.
    """
    if cap <= 0:
        return MIN_POOL_SLOTS
    return max(MIN_POOL_SLOTS, min(cap, active + (pending or 0)))


def size_lanes_sync(
    caps: dict[str, int],
    *,
    live: dict[str, LaneLiveState] | None = None,
    depths: dict[str, int | None] | None = None,
) -> list[LaneSizing]:
    """Size every lane to its backlog, within the cap. The whole control loop.

    `caps` is lane name -> slots_cap, read from the database by the caller.
    This function touches no database: the celery task that schedules it owns
    the session, and keeping the DB out of here is what lets it be called from
    anywhere that already knows the caps.

    `live` and `depths` are the measurements. Passing them in is not an
    optimisation — it is how the caller gets to KEEP them. The sweep now
    stores what it measured (`worker_lane_sample`) so the System tab reads a
    table instead of inspecting on every page load, and that is only possible
    if the same reading serves both purposes. Measured here when not given, so
    every existing caller and test is unaffected.

    ## It must converge and then go quiet

    One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing to a
    replica already at its target. A settled system therefore performs one
    broker round trip plus one LLEN sweep per tick and sends no control
    messages at all — the reachable fixed point lesson #4183 is about. An
    enforcer that re-sent a grow of zero every tick would churn forever and
    bury a real correction in its own noise.

    ## An absent lane is SKIPPED, not corrected

    `present=False` means nothing answered — a worker restarting, or an
    unreachable broker. It does NOT mean zero slots. Deciding from that would
    be a verdict drawn from an unswept read, and here it is worse than
    useless: there is nothing to send the message to.
    """
    if live is None:
        live = inspect_lanes_sync()
    if depths is None:
        depths = _queue_depths_sync()
    out: list[LaneSizing] = []

    for lane in LANES:
        cap = caps.get(lane.name)
        if cap is None:
            continue
        state = live[lane.name]
        if not state.present:
            out.append(LaneSizing(lane.name, "skipped", 0, "lane is not answering"))
            continue

        # Consumers first, and only when they DISAGREE. Sending add_consumer
        # for every queue on every tick of a settled system is the exact churn
        # above, and invisible: add_consumer on a queue already consumed is
        # harmless and reports success.
        should_consume = cap > 0
        if should_consume != state.consuming.issuperset(lane.queues):
            ok, err = set_lane_enabled_sync(lane, should_consume, live=state)
            if not ok:
                out.append(LaneSizing(
                    lane.name, "held", state.pool or 0,
                    f"could not {'start' if should_consume else 'stop'} "
                    f"consuming: {err}",
                ))
                continue

        current = state.pool
        if current is None:
            out.append(LaneSizing(
                lane.name, "held", 0, "worker did not report its pool size",
            ))
            continue

        depth = _lane_depth(lane, depths)
        pending = None if depth is None else depth + state.reserved
        want = wanted_slots(cap, state.active, pending)

        if want > current:
            new = want
            verb = "grew"
        elif want < current:
            # One at a time on the way down. See SHRINK_STEP.
            new = max(want, current - SHRINK_STEP)
            verb = "shrank"
        else:
            out.append(LaneSizing(
                lane.name, "held", current,
                f"{pending if pending is not None else '?'} waiting, "
                f"{state.active} busy, cap {cap}",
            ))
            continue

        ok, err = set_lane_slots_sync(lane, new, live=state)
        if not ok:
            out.append(LaneSizing(
                lane.name, "held", current, f"could not resize: {err}",
            ))
            continue
        out.append(LaneSizing(
            lane.name, verb, new,
            f"{pending if pending is not None else '?'} waiting, "
            f"{state.active} busy, cap {cap}",
        ))
    return out