feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/api/system_health.py

@@ -26,7 +26,6 @@ a true statement.
 from __future__ import annotations
 
 import asyncio
-import logging
 import time
 from datetime import UTC, datetime
 
@@ -36,9 +35,7 @@ from sqlalchemy import select, text
 from ..config import get_config
 from ..extensions import get_session
 from ..models import ServiceSeen
-from ..services.service_roster import refresh_if_stale
-
-log = logging.getLogger(__name__)
+from ..services.worker_lanes import SWEEP_PERIOD_SECONDS
 
 system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system")
 
@@ -53,6 +50,23 @@ system_health_bp = Blueprint("system_health", __name__, url_prefix="/api/system"
 STALE_AFTER_SECONDS = 90
 DOWN_AFTER_SECONDS = 300
 
+# The celery roster is written by `size_worker_lanes` and by nothing else, so
+# these thresholds are only meaningful against ITS cadence. Asserted at import
+# rather than left to a reader, because this is precisely the comparison that
+# was never made for the GPU agent: its lease poll backed off to 900s while
+# the roster called it stopped at 300s, and both numbers were individually
+# correct, in different directions, in different files (lesson #4355).
+#
+# Two clear sweeps before a part is even called STALE. One missed tick is
+# routine — the sweep rides the maintenance queue and does an inspect that can
+# take eleven seconds — and must not turn the page yellow.
+_SWEEPS_BEFORE_STALE = 2
+assert STALE_AFTER_SECONDS >= SWEEP_PERIOD_SECONDS * _SWEEPS_BEFORE_STALE, (
+    f"a {SWEEP_PERIOD_SECONDS}s sweep cannot keep a roster fresh against a "
+    f"{STALE_AFTER_SECONDS}s stale threshold: raise the threshold or shorten "
+    f"the sweep"
+)
+
 # Probes cross a process boundary, so they carry deadlines. A hung Postgres
 # must make this endpoint say "postgres: down", not hang alongside it.
 PROBE_TIMEOUT_SECONDS = 2.0
@@ -151,14 +165,11 @@ async def system_health():
         parts.append(pg)
 
         if pg["state"] == _OK:
-            # Rate-limited inside; see service_roster on why the web process
-            # is the right observer.
-            try:
-                await refresh_if_stale(session)
-                await session.commit()
-            except Exception:  # noqa: BLE001
-                log.warning("system health: roster refresh failed", exc_info=True)
-
+            # A PURE READ since 2026-09-23. This used to refresh the celery
+            # roster here, rate-limited to once per 20s — so the roster only
+            # advanced while somebody had a browser open, and a broadcast rode
+            # on a request. `size_worker_lanes` writes it now, on a timer, and
+            # the assertion below is what keeps that cadence honest.
             rows = (
                 await session.execute(select(ServiceSeen).order_by(ServiceSeen.display_name))
             ).scalars().all()

backend/app/api/workers.py

@@ -31,7 +31,12 @@ from ..services.worker_control import (
     push_lane_cap,
     store_lane_cap,
 )
-from ..services.worker_lanes import LANES_BY_NAME, Lane, derived_ceiling
+from ..services.worker_lanes import (
+    LANES_BY_NAME,
+    SWEEP_PERIOD_SECONDS,
+    Lane,
+    derived_ceiling,
+)
 from ._responses import error_response as _bad
 
 workers_bp = Blueprint("workers", __name__, url_prefix="/api/system/workers")
@@ -43,22 +48,26 @@ async def list_lanes():
 
     Response: {lanes: [...], fetched_at: iso8601}
 
-    Deliberately NOT cached, unlike system_activity's 2s/5s caches. This is
-    the surface an operator watches while dragging a stepper, and a cached
-    reply would show them the value from before their own change and read as
-    the control having failed.
+    One database read, and NO broker call. Operator, 2026-09-23: *"there is a
+    repull every time this page loads — is there a reason this info isn't
+    being tracked in the background and stored in some way?"*
+
+    It used to inspect the broker here, four broadcasts on an eleven-second
+    budget, four times a minute per open tab — while `size_worker_lanes` was
+    already inspecting on a timer and discarding the same numbers. The sweep
+    stores them now (`worker_lane_sample`) and this reads them.
+
+    So the live figures are up to `SWEEP_PERIOD_SECONDS` old, and each lane
+    carries the `measured_at` that says so. `sweep_period_seconds` is returned
+    alongside, so the UI can explain the age without hard-coding the cadence
+    in a second place.
     """
-    # The session closes BEFORE the broker work. Holding a Postgres connection
-    # across a celery inspect is what made this page block the whole site —
-    # see `worker_control.LaneSettings`. This endpoint polls every 15s and the
-    # inspect budget is 11s, so each poll was pinning a connection for most of
-    # the interval.
     async with get_session() as session:
         settings = await lane_settings(session)
-    lanes = await lane_view(settings)
     return jsonify({
-        "lanes": lanes,
+        "lanes": lane_view(settings),
         "fetched_at": datetime.now(UTC).isoformat(),
+        "sweep_period_seconds": SWEEP_PERIOD_SECONDS,
     })