feat: the System tab reads a stored sample instead of inspecting per load (4295)

Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR

frontend/src/components/settings/SystemHealthTab.vue

@@ -150,6 +150,24 @@
       </v-table>
     </v-card>
 
+    <!-- Where the numbers came from, and when.
+         The worker counts are read by a background sweep and stored, not
+         fetched when this page loads (operator, 2026-09-23: *"is there a
+         reason this info isn't being tracked in the background and stored in
+         some way?"*). They are therefore up to one sweep old, and saying so
+         is the difference between a lagging number and a wrong one. -->
+    <p v-if="!store.lastError" class="fc-parts__measured mt-2 mb-0">
+      <template v-if="measured">
+        Waiting and worker counts were measured {{ formatRelative(measured) }}<span
+          v-if="sweepSeconds"
+        >, and are re-read every {{ Math.round(sweepSeconds) }}s</span>.
+      </template>
+      <template v-else>
+        Waiting and worker counts have not been measured yet — the first
+        reading lands within a minute of startup.
+      </template>
+    </p>
+
     <!-- What the one control actually means. Operator, 2026-09-23: *"there's
          nothing to describe what 'auto' means or why their needs to be or
          should be on/off toggles."* There were three controls and no sentence
@@ -235,7 +253,7 @@ import { computed, onMounted, onUnmounted, ref } from 'vue'
 import { laneStuckFor, useSystemActivityStore } from '../../stores/systemActivity.js'
 import { useSystemHealthStore } from '../../stores/systemHealth.js'
 import { formatRelative } from '../../utils/date.js'
-import { laneAdvice, mergeParts } from '../../utils/systemParts.js'
+import { laneAdvice, measuredAt, mergeParts } from '../../utils/systemParts.js'
 
 const store = useSystemHealthStore()
 const lanesStore = useSystemActivityStore()
@@ -273,6 +291,13 @@ const rows = computed(() => mergeParts(
   store.parts, lanesStore.lanes?.lanes ?? [], laneStuckFor,
 ).map((row) => ({ ...row, advice: laneAdvice(row.lane) })))
 
+// When the stored worker numbers were last read, and how often that happens.
+// Both come from the endpoint rather than being restated here — the cadence
+// is `SWEEP_PERIOD_SECONDS`, and a second copy of it in the UI would be free
+// to drift from the schedule it is describing.
+const measured = computed(() => measuredAt(lanesStore.lanes?.lanes ?? []))
+const sweepSeconds = computed(() => lanesStore.lanes?.sweep_period_seconds ?? null)
+
 const offOptionalLanes = computed(() =>
   (lanesStore.lanes?.lanes ?? []).filter(
     (l) => l.optional && l.slots_cap === 0 && l.models?.length,
@@ -389,6 +414,9 @@ function step(lane, delta) {
   font-size: 0.78rem; color: rgb(var(--v-theme-on-surface) / 0.6);
 }
 .fc-parts__num { font-variant-numeric: tabular-nums; }
+.fc-parts__measured {
+  font-size: 0.78rem; color: rgb(var(--v-theme-on-surface) / 0.55);
+}
 .fc-parts__sub {
   font-size: 0.68rem; text-transform: uppercase; letter-spacing: 0.04em;
   color: rgb(var(--v-theme-on-surface) / 0.5); white-space: nowrap;

frontend/src/utils/systemParts.js

@@ -142,3 +142,25 @@ export function laneAdvice(lane) {
     + `Raise the cap to run more at once — this machine allows up to `
     + `${lane.ceiling}.`
 }
+
+
+// When the worker numbers in the table were actually read.
+//
+// They used to be fetched live on every page load — a celery broadcast per
+// request, on a page that polls every 15s. Operator, 2026-09-23: *"there is a
+// repull every time this page loads — is there a reason this info isn't being
+// tracked in the background and stored in some way?"* It is now: a sweep
+// measures on a timer and stores it, and the page reads the table.
+//
+// So the age has to be ON SCREEN. A number that is up to half a minute old,
+// presented as current, is how someone ends up watching a queue "not move"
+// that is in fact moving. The NEWEST across lanes, because they are written
+// by one sweep in one pass — a lane lagging the others means its row has
+// never been written, and that lane says so in its own detail line.
+export function measuredAt(lanes) {
+  const stamps = (lanes || [])
+    .map((l) => l.measured_at)
+    .filter(Boolean)
+    .sort()
+  return stamps.length ? stamps[stamps.length - 1] : null
+}

frontend/test/systemParts.spec.js

@@ -1,6 +1,6 @@
 import { describe, expect, it } from 'vitest'
 
-import { laneAdvice, mergeParts, queueKey } from '../src/utils/systemParts.js'
+import { laneAdvice, measuredAt, mergeParts, queueKey } from '../src/utils/systemParts.js'
 
 // The System tab's one table (operator 2026-09-23: "combine the two sections
 // into a single table"). What is pinned here is the JOIN, because its failure
@@ -190,3 +190,37 @@ describe('laneAdvice', () => {
     expect(laneAdvice(undefined)).toBeNull()
   })
 })
+
+
+// When the worker numbers were read. They stopped being fetched per page load
+// on 2026-09-23 — a sweep measures them on a timer and stores them — so the
+// age belongs on screen. Operator: *"is there a reason this info isn't being
+// tracked in the background and stored in some way?"*
+
+describe('measuredAt', () => {
+  it('reports the newest reading across the lanes', () => {
+    expect(measuredAt([
+      { measured_at: '2026-09-23T20:00:00Z' },
+      { measured_at: '2026-09-23T20:00:30Z' },
+      { measured_at: '2026-09-23T19:59:30Z' },
+    ])).toBe('2026-09-23T20:00:30Z')
+  })
+
+  it('ignores a lane the sweep has never written', () => {
+    // A lane added by a newer build, or a first boot mid-sweep. It must not
+    // drag the reported age back to null while other lanes have real ones.
+    expect(measuredAt([
+      { measured_at: null },
+      { measured_at: '2026-09-23T20:00:00Z' },
+    ])).toBe('2026-09-23T20:00:00Z')
+  })
+
+  it('says null when nothing has been measured, rather than guessing now', () => {
+    // The first period of a fresh install. "Not measured yet" and "measured
+    // just now" are opposite claims, and defaulting to the clock would make
+    // the page assert the wrong one at exactly the moment it knows least.
+    expect(measuredAt([{ measured_at: null }])).toBeNull()
+    expect(measuredAt([])).toBeNull()
+    expect(measuredAt(undefined)).toBeNull()
+  })
+})