feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/tasks/maintenance.py

@@ -1368,9 +1368,31 @@ def size_worker_lanes() -> dict:
     Returns every lane's outcome INCLUDING the ones it held, each with a
     reason. A pass that only speaks when it acts cannot be debugged on the day
     it does not.
+
+    ## It is also the only thing that MEASURES, since 2026-09-23
+
+    It always inspected the broker to decide pool sizes, and then threw the
+    reading away — while `/api/system/workers` ran the same inspect on every
+    page load and the System tab polls it four times a minute. Operator:
+    *"there is a repull every time this page loads — is there a reason this
+    info isn't being tracked in the background and stored in some way?"*
+
+    So one inspect now feeds three things: the sizing decision, the stored
+    sample the System tab reads, and the celery roster. No request path
+    touches the broker any more.
+
+    Order matters. The sample is stored BEFORE the roster refresh, because
+    that refresh does its own broadcast and a broker that has just started
+    failing must not cost us the reading we already have.
     """
     from ..models import WorkerLane
-    from ..services.worker_control import size_lanes_sync
+    from ..services.service_roster import refresh_celery_roster_sync
+    from ..services.worker_control import (
+        _queue_depths_sync,
+        inspect_lanes_sync,
+        size_lanes_sync,
+        store_lane_samples_sync,
+    )
 
     # Read INSIDE the session. Reading a column off a detached instance
     # happens to work while the attribute is still loaded and stops working
@@ -1387,7 +1409,18 @@ def size_worker_lanes() -> dict:
         # let this task disagree with the seed it is meant to be enforcing.
         return {"sized": []}
 
-    sized = size_lanes_sync(caps)
+    # Measured ONCE, here, and then used three times. Passing them down is
+    # what makes the reading keepable rather than an implementation detail of
+    # a function that returns decisions.
+    live = inspect_lanes_sync()
+    depths = _queue_depths_sync()
+
+    sized = size_lanes_sync(caps, live=live, depths=depths)
+
+    with _sync_session_factory()() as session:
+        store_lane_samples_sync(session, live, depths)
+        refresh_celery_roster_sync(session)
+
     for d in sized:
         if d.action not in ("held", "skipped"):
             log.info(