feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/services/worker_control.py

@@ -53,9 +53,10 @@ from dataclasses import dataclass, field
 from datetime import UTC, datetime
 
 from sqlalchemy import func, select
+from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from ..models import TaskRun, WorkerLane
+from ..models import TaskRun, WorkerLane, WorkerLaneSample
 from .worker_lanes import (
     LANES,
     LANES_BY_QUEUE_KEY,
@@ -341,6 +342,76 @@ async def _rows_by_name(session: AsyncSession) -> dict[str, WorkerLane]:
     return rows
 
 
+@dataclass(frozen=True)
+class LaneSample:
+    """What the sizing sweep last measured about one lane.
+
+    The same fields `LaneLiveState` carries, plus the queue depth and WHEN —
+    because this one is read from a table rather than from the broker, and a
+    reading with no timestamp invites being presented as current.
+
+    `measured_at=None` means no sweep has written this lane yet: a fresh
+    install inside its first period, or a stack whose beat is not running.
+    Distinct from `present=False` (something asked, nothing answered), and the
+    UI says different things about the two.
+    """
+
+    present: bool = False
+    replicas: int = 0
+    pool: int | None = None
+    active: int = 0
+    reserved: int = 0
+    queue_depth: int | None = None
+    measured_at: datetime | None = None
+
+
+def _lane_depth(lane: Lane, depths: dict[str, int | None]) -> int | None:
+    """A lane's backlog across its queues — None when NOTHING answered.
+
+    A queue the broker did not answer for must not be summed as zero: an
+    unknown depth is not an empty one, and reporting a buried lane as idle is
+    the direction that matters.
+    """
+    known = [depths.get(q) for q in lane.queues]
+    if not any(d is not None for d in known):
+        return None
+    return sum(d for d in known if d is not None)
+
+
+def store_lane_samples_sync(session, live: dict[str, LaneLiveState], depths) -> None:
+    """Write what the sweep just measured. SYNC — the celery task owns a sync
+    session, and this is the only place these rows are written.
+
+    Upsert per lane, last writer wins, same shape as `service_roster`'s
+    `touch_service`: two processes sweeping at once is a benign race that
+    needs no coordination, because both are recording what they actually saw.
+
+    A lane that did not answer is STILL written, with `present=False`. Skipping
+    it would leave the previous reading in place and let the page go on showing
+    a pool that is no longer there — the stale row would read as a current one
+    (lesson #4202: the row is the thing that has to change).
+    """
+    now = datetime.now(UTC)
+    for lane in LANES:
+        state = live.get(lane.name) or LaneLiveState()
+        values = {
+            "lane": lane.name,
+            "present": state.present,
+            "replicas": state.replicas,
+            "pool": state.pool,
+            "active": state.active,
+            "reserved": state.reserved,
+            "queue_depth": _lane_depth(lane, depths),
+            "measured_at": now,
+        }
+        stmt = pg_insert(WorkerLaneSample).values(**values)
+        session.execute(stmt.on_conflict_do_update(
+            index_elements=[WorkerLaneSample.lane],
+            set_={k: v for k, v in values.items() if k != "lane"},
+        ))
+    session.commit()
+
+
 @dataclass
 class LaneSettings:
     """What the DATABASE knows about the lanes — read and finished with before
@@ -366,58 +437,80 @@ class LaneSettings:
 
     caps: dict[str, int]
     oldest_by_queue: dict[str, datetime]
+    # The sizing sweep's last reading per lane. Since 2026-09-23 this is where
+    # the live numbers come from: the endpoint no longer inspects at all.
+    samples: dict[str, LaneSample] = field(default_factory=dict)
 
 
 async def lane_settings(session: AsyncSession) -> LaneSettings:
-    """Every DB read the lane view needs, in one short-lived session."""
+    """Every DB read the lane view needs, in one short-lived session.
+
+    Which is now ALL of them. `lane_view` below takes what this returns and
+    talks to nothing.
+    """
     rows = await _rows_by_name(session)
+    samples = {
+        row.lane: LaneSample(
+            present=row.present,
+            replicas=row.replicas,
+            pool=row.pool,
+            active=row.active,
+            reserved=row.reserved,
+            queue_depth=row.queue_depth,
+            measured_at=row.measured_at,
+        )
+        for row in (
+            await session.execute(select(WorkerLaneSample))
+        ).scalars()
+    }
     return LaneSettings(
         caps={name: row.slots_cap for name, row in rows.items()},
         oldest_by_queue=await _oldest_running_by_queue(session),
+        samples=samples,
     )
 
 
-async def lane_view(settings: LaneSettings) -> list[dict]:
-    """Every lane: what is configured, what is live, what it may grow to.
+def lane_view(settings: LaneSettings) -> list[dict]:
+    """Every lane: what is configured, what was last measured, what it may
+    grow to. NO broker call, and no database — `settings` is the whole input.
 
-    Takes the settings rather than a session ON PURPOSE — see `LaneSettings`.
-    Everything below this line is broker work, and no database connection is
-    held while it happens.
+    ## It used to inspect, on every request
 
-    `pending` is the honest backlog — Redis depth PLUS reserved — because
+    Four broadcast round trips on an eleven-second budget, on a page that
+    polls every fifteen seconds. Operator, 2026-09-23: *"there is a repull
+    every time this page loads — is there a reason this info isn't being
+    tracked in the background and stored in some way?"*
+
+    There was one, and it had expired. The docstring here used to say the
+    endpoint was deliberately uncached because *"this is the surface an
+    operator watches while dragging a stepper, and a cached reply would show
+    them the value from before their own change"*. True while a cap change
+    refetched the table — and that refetch is exactly what was removed in
+    `1353d34`, so the UI now patches its own row from the write's reply and
+    nothing depends on this being live.
+
+    Meanwhile `size_worker_lanes` was already inspecting on a timer to decide
+    pool sizes: the same numbers, computed, used, and discarded, while the
+    browser asked the broker for them again four times a minute.
+
+    So the sweep writes `worker_lane_sample` and this reads it. The reading is
+    up to `SWEEP_PERIOD_SECONDS` old, and `measured_at` travels with it so the
+    UI can say so rather than implying it is current.
+
+    `pending` is still the honest backlog — depth PLUS reserved — because
     celery prefetches and LLEN alone reads 0 while a worker holds tasks in
     memory.
     """
-    # A deadline, because this is a request path and `to_thread` on its own is
-    # an await with no bound (rule 156). `inspect_lanes_sync` never raises and
-    # every inner call has its own timeout, so the only way past the budget is
-    # the thread not being scheduled — and a page that renders "not answering"
-    # is a better answer than one that does not render.
-    try:
-        live = await asyncio.wait_for(
-            asyncio.to_thread(inspect_lanes_sync),
-            timeout=INSPECT_BUDGET_SECONDS,
-        )
-    except TimeoutError:
-        log.warning(
-            "worker_control: inspect exceeded %ss; reporting every lane as "
-            "not answering", INSPECT_BUDGET_SECONDS,
-        )
-        live = {lane.name: LaneLiveState() for lane in LANES}
-    depths = await asyncio.to_thread(_queue_depths_sync)
     oldest = settings.oldest_by_queue
-
     now = datetime.now(UTC)
     out = []
     for lane in LANES:
         cap = settings.caps[lane.name]
-        state = live[lane.name]
-        # None for a queue the broker did not answer for, which must not be
-        # silently summed as zero — an unknown depth is not an empty one.
-        known = [depths.get(q) for q in lane.queues]
-        depth = sum(d for d in known if d is not None) if any(
-            d is not None for d in known
-        ) else None
+        # A lane with no row yet is not-measured, which is distinct from
+        # measured-as-absent. The default carries `measured_at=None`, and the
+        # UI says "not measured yet" rather than "not answering".
+        sample = settings.samples.get(lane.name) or LaneSample()
+        depth = sample.queue_depth
         out.append({
             "name": lane.name,
             "display_name": lane.display_name,
@@ -444,17 +537,24 @@ async def lane_view(settings: LaneSettings) -> list[dict]:
                 for m in lane.models
             ],
             "live": {
-                "present": state.present,
-                "replicas": state.replicas,
-                "pool": state.pool,
-                "active": state.active,
-                "reserved": state.reserved,
+                "present": sample.present,
+                "replicas": sample.replicas,
+                "pool": sample.pool,
+                "active": sample.active,
+                "reserved": sample.reserved,
             },
             "queue_depth": depth,
-            "pending": None if depth is None else depth + state.reserved,
+            "pending": None if depth is None else depth + sample.reserved,
+            # When the numbers above were read. Per lane rather than one for
+            # the response, because a lane whose row has never been written
+            # has no reading at all and must not borrow another lane's.
+            "measured_at": (
+                sample.measured_at.isoformat() if sample.measured_at else None
+            ),
             # How long the oldest still-running task on this lane has been
-            # going, in minutes. The operator asked for a trigger here — grow
-            # a lane whose tasks run past some duration — and it stayed a
+            # going, in minutes. Read from `task_run`, not from the sweep, so
+            # this one IS current. The operator asked for a trigger here —
+            # grow a lane whose tasks run past some duration — and it stayed a
             # REPORT: a long task does not finish sooner because the lane
             # gained a slot, so scaling on it would spend memory to change
             # nothing. Shown so they can see a lane wedged on one slow job,
@@ -606,7 +706,22 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
         # LOWERED on a running lane. Only this direction needs a message, and
         # only when the pool is actually above the new cap — so it reads the
         # live pool rather than resizing blind. A raise never reaches here.
-        live = await asyncio.to_thread(inspect_lanes_sync)
+        #
+        # Bounded (rule 156): `to_thread` on its own is an await with no
+        # deadline, and this runs in a background task where a hang would be
+        # silent rather than visible as a slow page. On a timeout the lane is
+        # simply not resized here and the sizing sweep carries it.
+        try:
+            live = await asyncio.wait_for(
+                asyncio.to_thread(inspect_lanes_sync),
+                timeout=INSPECT_BUDGET_SECONDS,
+            )
+        except TimeoutError:
+            log.warning(
+                "worker_control: inspect exceeded %ss lowering %s; leaving the "
+                "pool to the sizing pass", INSPECT_BUDGET_SECONDS, lane.name,
+            )
+            return _cap_result(lane, slots_cap, now_on, applied, error, False)
         current = live[lane.name].pool
         if current is not None and current > slots_cap:
             applied, error = await asyncio.to_thread(
@@ -642,6 +757,15 @@ async def push_lane_cap(lane: Lane, slots_cap: int, *, was_cap: int) -> dict:
             lane.name, slots_cap, error,
         )
 
+    return _cap_result(lane, slots_cap, now_on, applied, error, fetching)
+
+
+def _cap_result(
+    lane: Lane, slots_cap: int, now_on: bool, applied: bool,
+    error: str | None, fetching: bool,
+) -> dict:
+    """The push's outcome. One builder, because `push_lane_cap` has two exits
+    and a second literal would be free to disagree with the first."""
     return {
         "name": lane.name,
         "slots_cap": slots_cap,
@@ -746,7 +870,12 @@ def wanted_slots(cap: int, active: int, pending: int | None) -> int:
     return max(MIN_POOL_SLOTS, min(cap, active + (pending or 0)))
 
 
-def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
+def size_lanes_sync(
+    caps: dict[str, int],
+    *,
+    live: dict[str, LaneLiveState] | None = None,
+    depths: dict[str, int | None] | None = None,
+) -> list[LaneSizing]:
     """Size every lane to its backlog, within the cap. The whole control loop.
 
     `caps` is lane name -> slots_cap, read from the database by the caller.
@@ -754,6 +883,13 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
     the session, and keeping the DB out of here is what lets it be called from
     anywhere that already knows the caps.
 
+    `live` and `depths` are the measurements. Passing them in is not an
+    optimisation — it is how the caller gets to KEEP them. The sweep now
+    stores what it measured (`worker_lane_sample`) so the System tab reads a
+    table instead of inspecting on every page load, and that is only possible
+    if the same reading serves both purposes. Measured here when not given, so
+    every existing caller and test is unaffected.
+
     ## It must converge and then go quiet
 
     One `inspect` for all lanes, and `set_lane_slots_sync` issues nothing to a
@@ -770,8 +906,10 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
     be a verdict drawn from an unswept read, and here it is worse than
     useless: there is nothing to send the message to.
     """
-    live = inspect_lanes_sync()
-    depths = _queue_depths_sync()
+    if live is None:
+        live = inspect_lanes_sync()
+    if depths is None:
+        depths = _queue_depths_sync()
     out: list[LaneSizing] = []
 
     for lane in LANES:
@@ -805,11 +943,7 @@ def size_lanes_sync(caps: dict[str, int]) -> list[LaneSizing]:
             ))
             continue
 
-        known = [depths.get(q) for q in lane.queues]
-        depth = (
-            sum(d for d in known if d is not None)
-            if any(d is not None for d in known) else None
-        )
+        depth = _lane_depth(lane, depths)
         pending = None if depth is None else depth + state.reserved
         want = wanted_slots(cap, state.active, pending)