feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/services/worker_lanes.py

@@ -282,6 +282,31 @@ MIN_POOL_SLOTS = 1
 # not to make a small machine unusable.
 MIN_CEILING = 1
 
+# How often `size_worker_lanes` runs — the beat schedule, and the freshness of
+# everything the System tab shows.
+#
+# It is here, in the import-light module, because three places have to agree
+# about it and they are in different packages: the beat entry in `celery_app`,
+# the sample the sweep writes (`worker_lane_sample`), and the roster's
+# staleness thresholds in `api/system_health`, which now depend on this sweep
+# rather than on a browser being open.
+#
+# 30s, down from 60s, because the sweep became the ONLY writer of the celery
+# roster on 2026-09-23. A part is called stale after 90s of silence, so a
+# 60-second sweep left one missed tick between "normal" and "everything is
+# yellow". That is the shape of lesson #4355 — a reader's threshold and an
+# emitter's cadence chosen in different files and never compared — and the
+# fix is headroom plus a test that asserts it, not a number that happens to
+# work today.
+#
+# The cost is one inspect every 30s instead of every 60s; the saving is every
+# inspect that used to run on a request path, which with a single tab open
+# was roughly four a minute against this two. Consequence worth knowing: the
+# pass also SHRINKS an idle lane by one slot per tick, so an idle lane now
+# gives its workers back twice as fast. That is the direction the operator
+# asked for — *"idle instances quiet down when not running"*.
+SWEEP_PERIOD_SECONDS = 30.0
+
 # What an unreadable limit yields. Low rather than unlimited, on purpose: not
 # knowing how much memory there is must never read as "plenty". An unswept
 # absence is not a verdict.