feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/models/__init__.py

@@ -45,6 +45,7 @@ from .tag_positive_confirmation import TagPositiveConfirmation
 from .tag_suggestion_rejection import TagSuggestionRejection
 from .task_run import TaskRun
 from .worker_lane import WorkerLane
+from .worker_lane_sample import WorkerLaneSample
 
 __all__ = [
     "Base",
@@ -96,4 +97,5 @@ __all__ = [
     "TagSuggestionRejection",
     "TaskRun",
     "WorkerLane",
+    "WorkerLaneSample",
 ]

backend/app/models/worker_lane_sample.py

@@ -0,0 +1,83 @@
+"""worker_lane_sample — the last thing the sizing sweep measured about a lane.
+
+Milestone 422, 2026-09-23. A MEASUREMENT table, deliberately separate from
+`worker_lane`, which holds the one number an operator sets.
+
+## Why this exists
+
+Operator, 2026-09-23, looking at the System tab: *"there is a repull every
+time this page loads — is there a reason this info isn't being tracked in the
+background and stored in some way?"*
+
+There was not a good one. `/api/system/workers` ran a full celery inspect —
+four broadcast round trips on an eleven-second budget — on every call, and
+the page polls it every fifteen seconds. Meanwhile `size_worker_lanes` was
+already inspecting on a timer to decide pool sizes, computing exactly these
+numbers, using them, and throwing them away. The browser then asked the
+broker for them again.
+
+So the sweep writes what it saw here, and the endpoint reads this table. The
+request path makes no broker call at all any more.
+
+## Why NOT columns on `worker_lane`
+
+Because that is the mistake this milestone already made once and undid. That
+table used to carry `slots` — how many workers were running — beside
+`slots_cap`, and a measurement sitting next to a preference reads as a second
+preference: the operator had to keep two numbers in agreement, and the
+autoscaler had to be granted permission to move one of them.
+
+The distinction is the whole design, so it is a table boundary. Nothing an
+operator sets lives here; nothing here is ever an input to a decision about
+what they wanted.
+
+## Freshness is a value, not an assumption
+
+`measured_at` is returned to the UI, which says how old the reading is rather
+than implying it is live. A sample is a fact about a moment, and a page that
+presents a one-minute-old number as current is how an operator ends up
+mistrusting the whole surface.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import Boolean, DateTime, Integer, String, func
+from sqlalchemy.orm import Mapped, mapped_column
+
+from .base import Base
+
+
+class WorkerLaneSample(Base):
+    __tablename__ = "worker_lane_sample"
+
+    # The lane name from services/worker_lanes.LANES. One row per lane,
+    # overwritten in place: this is the LATEST reading, not a history. A time
+    # series would be a different table with a different retention problem,
+    # and nothing has asked for one.
+    lane: Mapped[str] = mapped_column(String(32), primary_key=True)
+
+    # Whether anything answered for this lane. NOT the same as "zero workers"
+    # — an unswept absence is not a verdict (snippet #3969). False here means
+    # the inspect came back without this lane, so every count below is
+    # meaningless and the UI must say "not answering" rather than "0".
+    present: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
+
+    replicas: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+
+    # Pool size of ONE process, nullable because a worker that answered
+    # without reporting its pool is unknown rather than empty.
+    pool: Mapped[int | None] = mapped_column(Integer, nullable=True)
+
+    active: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+    reserved: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
+
+    # Redis LLEN across the lane's queues. Nullable for the same reason as
+    # `pool`: a queue the broker did not answer for is unknown, and summing it
+    # as zero would report a buried lane as idle.
+    queue_depth: Mapped[int | None] = mapped_column(Integer, nullable=True)
+
+    measured_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True),
+        nullable=False,
+        server_default=func.now(),
+    )