feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

backend/app/services/service_roster.py

@@ -31,7 +31,7 @@ from __future__ import annotations
 import asyncio
 import logging
 
-from sqlalchemy import func, select
+from sqlalchemy import func
 from sqlalchemy.dialects.postgresql import insert as pg_insert
 from sqlalchemy.ext.asyncio import AsyncSession
 
@@ -40,10 +40,11 @@ from .worker_lanes import LANES, lane_for_node
 
 log = logging.getLogger(__name__)
 
-# How stale the roster may be before a health request refreshes it. Comfortably
-# under the staleness thresholds that decide a service is missing, so the
-# verdict is never limited by how often anyone looked.
-REFRESH_TTL_SECONDS = 20.0
+# There is no refresh TTL any more. It existed because the HEALTH REQUEST
+# refreshed the roster, rate-limited to 20s so that a page open in two tabs
+# did not inspect twice as often. `size_worker_lanes` owns the refresh now, on
+# `SWEEP_PERIOD_SECONDS`, so the cadence is a schedule rather than a side
+# effect of someone looking.
 
 # celery inspect is a broker round trip and this sits on a request path, so it
 # gets a deadline (rule 156). A broker that has stopped answering must make the
@@ -140,10 +141,13 @@ def _inspect_celery_sync() -> dict[tuple[str, ...], dict]:
     return grouped
 
 
-async def touch_service(
-    session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict
-) -> None:
-    """Record that a part checked in just now.
+def touch_service_stmt(*, key: str, kind: str, display_name: str, details: dict):
+    """The upsert that records a check-in, as a statement.
+
+    Built here rather than inline so the async caller (an agent lease, over
+    the API) and the sync one (the sizing sweep, in a celery task) run the
+    SAME write. Two spellings of one upsert is the kind of duplication that
+    stays correct right up until one of them gains a column.
 
     Upsert rather than read-modify-write: several web processes and several
     agents can be doing this at once, and the last writer is simply the most
@@ -154,7 +158,7 @@ async def touch_service(
     stmt = pg_insert(ServiceSeen).values(
         key=key, kind=kind, display_name=display_name, details=details,
     )
-    stmt = stmt.on_conflict_do_update(
+    return stmt.on_conflict_do_update(
         index_elements=[ServiceSeen.key],
         set_={
             "kind": stmt.excluded.kind,
@@ -163,7 +167,75 @@ async def touch_service(
             "last_seen_at": func.now(),
         },
     )
-    await session.execute(stmt)
+
+
+async def touch_service(
+    session: AsyncSession, *, key: str, kind: str, display_name: str, details: dict
+) -> None:
+    """Record that a part checked in just now."""
+    await session.execute(touch_service_stmt(
+        key=key, kind=kind, display_name=display_name, details=details,
+    ))
+
+
+def _roster_rows(grouped: dict[tuple[str, ...], dict]) -> list[dict]:
+    """The `touch_service` arguments for everything that answered.
+
+    Split from the write so the async and sync refreshes below share the
+    mapping as well as the statement — what a roster row IS should not depend
+    on which kind of session is writing it.
+    """
+    return [
+        {
+            "key": "celery:" + ",".join(queues),
+            "kind": "celery",
+            "display_name": role_display_name(queues),
+            "details": {
+                "queues": list(queues),
+                "hostnames": entry["hostnames"],
+                "replicas": len(entry["hostnames"]),
+                "active": entry["active"],
+            },
+        }
+        for queues, entry in grouped.items()
+    ]
+
+
+def refresh_celery_roster_sync(session) -> None:
+    """The roster refresh, from the sizing sweep's sync session.
+
+    ## Why the sweep owns this now
+
+    It used to run on the request path, rate-limited to once every 20s by the
+    newest celery row. So the roster only advanced while somebody had a
+    browser open — the liveness of the workers was a function of whether
+    anyone was looking at them, which is the observer-effect version of the
+    bug this roster exists to prevent.
+
+    Operator, 2026-09-23: *"there is a repull every time this page loads — is
+    there a reason this info isn't being tracked in the background and stored
+    in some way?"*
+
+    Now a timer writes it and the page only reads. The cadence is
+    `SWEEP_PERIOD_SECONDS`, and `api/system_health` asserts it leaves headroom
+    under the staleness thresholds — because a sweep period and a stale
+    threshold chosen in different files and never compared is exactly how the
+    idle GPU agent came to read as stopped (lesson #4355).
+
+    Never raises. A failure means the roster does not advance, and the rows
+    going stale is then a TRUE report about a broker nobody can reach.
+    """
+    try:
+        grouped = _inspect_celery_sync()
+    except Exception:
+        log.warning(
+            "service roster: celery inspect failed; roster not refreshed",
+            exc_info=True,
+        )
+        return
+    for row in _roster_rows(grouped):
+        session.execute(touch_service_stmt(**row))
+    session.commit()
 
 
 async def refresh_celery_roster(session: AsyncSession) -> None:
@@ -186,39 +258,5 @@ async def refresh_celery_roster(session: AsyncSession) -> None:
         log.warning("service roster: celery inspect failed; roster not refreshed", exc_info=True)
         return
 
-    for queues, entry in grouped.items():
-        await touch_service(
-            session,
-            key="celery:" + ",".join(queues),
-            kind="celery",
-            display_name=role_display_name(queues),
-            details={
-                "queues": list(queues),
-                "hostnames": entry["hostnames"],
-                "replicas": len(entry["hostnames"]),
-                "active": entry["active"],
-            },
-        )
-
-
-async def refresh_if_stale(session: AsyncSession) -> None:
-    """Refresh the celery roster if nobody has for REFRESH_TTL_SECONDS.
-
-    Rate-limited by the data rather than by a lock: the gate is the newest
-    last_seen_at across the celery rows, which every web process can see. Two
-    processes racing through the gate costs one redundant inspect and writes
-    the same values twice, so the benign outcome needs no coordination to
-    prevent.
-    """
-    newest = (
-        await session.execute(
-            select(func.max(ServiceSeen.last_seen_at)).where(ServiceSeen.kind == "celery")
-        )
-    ).scalar_one_or_none()
-
-    if newest is not None:
-        age = (await session.execute(select(func.now()))).scalar_one() - newest
-        if age.total_seconds() < REFRESH_TTL_SECONDS:
-            return
-
-    await refresh_celery_roster(session)
+    for row in _roster_rows(grouped):
+        await touch_service(session, **row)