feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

tests/test_worker_lanes.py

@@ -364,3 +364,39 @@ def test_estimated_numbers_are_flagged_as_estimates():
     fact. Flip this to True in the same commit that records a real
     measurement."""
     assert wl.SIGLIP_MODEL.measured is False
+
+
+# --- the sweep's cadence, against the thresholds that read it ----------------
+
+
+def test_the_sweep_runs_often_enough_to_keep_the_roster_fresh():
+    """The comparison that was never made for the GPU agent.
+
+    Its idle lease poll backed off to a 900s ceiling while the roster called it
+    stopped at 300s. Both numbers were right on their own, in different files,
+    written ten weeks apart — and an idle agent was structurally guaranteed to
+    read as stopped (lesson #4355).
+
+    `size_worker_lanes` is now the ONLY writer of the celery roster, so its
+    period and the staleness thresholds are in exactly that relationship. Two
+    clear sweeps before a part is even doubted: one missed tick is routine,
+    because the sweep rides the maintenance queue and does an inspect that can
+    take eleven seconds.
+    """
+    from backend.app.api.system_health import (
+        DOWN_AFTER_SECONDS,
+        STALE_AFTER_SECONDS,
+    )
+
+    assert wl.SWEEP_PERIOD_SECONDS * 2 <= STALE_AFTER_SECONDS
+    assert wl.SWEEP_PERIOD_SECONDS * 2 <= DOWN_AFTER_SECONDS
+
+
+def test_the_beat_schedule_is_the_same_number_and_not_a_copy_of_it():
+    """A schedule that merely happens to equal the constant is one edit away
+    from disagreeing with the test above, which would then be asserting
+    headroom the running system does not have."""
+    from backend.app.celery_app import celery
+
+    entry = celery.conf.beat_schedule["size-worker-lanes"]
+    assert entry["schedule"] == wl.SWEEP_PERIOD_SECONDS