Files
FabledCurator/frontend/test/systemParts.spec.js
T
bvandeusenandClaude Opus 5 45bb7044f7
CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 31s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m44s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
feat: the System tab reads a stored sample instead of inspecting per load (4295)
Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"

There was a reason and it had expired, and underneath it there was plain
waste.

The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.

The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.

So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.

Consequences, stated rather than hidden:

- The live figures are up to one sweep old. measured_at travels with each
  lane and the page says how old, because a stale number presented as
  current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
  staleness thresholds are in a relationship. 60s against a 90s stale
  threshold left one missed tick between normal and all-yellow — the
  shape of lesson #4355 — so the period is 30s, named once in
  worker_lanes, and system_health asserts its headroom at import with a
  test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
  the direction asked for: "idle instances quiet down when not running".

Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
2026-09-23 18:52:08 -04:00

227 lines
8.4 KiB
JavaScript

import { describe, expect, it } from 'vitest'
import { laneAdvice, measuredAt, mergeParts, queueKey } from '../src/utils/systemParts.js'
// The System tab's one table (operator 2026-09-23: "combine the two sections
// into a single table"). What is pinned here is the JOIN, because its failure
// is silent and is precisely the thing the merge exists to fix: a lane that
// stops matching its roster part does not throw — the table grows a SECOND row
// for the same worker, one with controls and one without.
const PART = {
key: 'celery:default,download,import,thumbnail',
kind: 'celery',
name: 'Worker',
state: 'ok',
detail: 'Worker is running',
// The roster SORTS a celery part's queues into its key.
queues: ['default', 'download', 'import', 'thumbnail'],
}
const LANE = {
name: 'worker',
display_name: 'Worker',
// The lane table lists them in the order the role reads them, which is NOT
// sorted. If the join ever compares these two lists directly rather than as
// sets, this fixture is what catches it.
queues: ['default', 'import', 'thumbnail', 'download'],
slots_cap: 4,
ceiling: 8,
live: { present: true, replicas: 1, pool: 2, active: 0 },
pending: 0,
}
const POSTGRES = {
key: 'postgres', kind: 'datastore', name: 'PostgreSQL',
state: 'ok', detail: 'answering', latency_ms: 2.5,
}
describe('queueKey', () => {
it('does not care what order either side lists its queues in', () => {
expect(queueKey(LANE.queues)).toBe(queueKey(PART.queues))
})
it('survives a part that has no queues at all', () => {
// A datastore, and also the stale `Worker ()` row a previous deployment
// left in the roster with an empty queue set. Neither must match a lane.
expect(queueKey(undefined)).toBe('')
expect(queueKey([])).toBe('')
})
})
describe('mergeParts', () => {
it('gives a worker ONE row, carrying its controls', () => {
const rows = mergeParts([PART, POSTGRES], [LANE])
expect(rows).toHaveLength(2)
const worker = rows.find((r) => r.name === 'Worker')
expect(worker.lane).toBe(LANE)
expect(rows.filter((r) => r.name === 'Worker')).toHaveLength(1)
})
it('leaves a datastore without a lane rather than guessing one', () => {
const pg = mergeParts([PART, POSTGRES], [LANE]).find((r) => r.key === 'postgres')
expect(pg.lane).toBeUndefined()
})
it('still lists a lane the roster has never seen', () => {
// Parts are learned as they appear; the lane table is known up front. The
// lane an operator most needs to find — an optional one, never started —
// is exactly the one with no roster entry.
const ml = {
...LANE, name: 'ml', display_name: 'ML tagging', queues: ['ml'],
slots_cap: 0, optional: true,
live: { present: false, replicas: 0, pool: null, active: 0 },
}
const rows = mergeParts([POSTGRES], [ml])
const row = rows.find((r) => r.name === 'ML tagging')
expect(row.lane).toBe(ml)
expect(row.kindLabel).toBe('optional lane')
})
it('does not call a lane capped at zero broken', () => {
// The roster only knows a heartbeat age, so it goes on saying "is running"
// for a lane the operator deliberately dialled to nothing. The lane knows
// the difference; reporting the operator's own choice as a fault is how an
// indicator stops being read.
const off = { ...LANE, slots_cap: 0 }
const row = mergeParts([PART], [off])[0]
expect(row.detail).toBe('off — cap is zero')
})
it('puts the broken thing first, whatever it is', () => {
const down = { ...POSTGRES, state: 'down', detail: 'not answering' }
const rows = mergeParts([PART, down], [LANE])
expect(rows[0].name).toBe('PostgreSQL')
})
it('otherwise puts the rows you can act on first', () => {
const rows = mergeParts([POSTGRES, PART], [LANE])
expect(rows.map((r) => r.name)).toEqual(['Worker', 'PostgreSQL'])
})
it('reports a wedged lane, and only through the reporter it was given', () => {
// `laneStuckFor` is passed in rather than imported, so this file does not
// re-test the store's rule — it tests that the merge asks.
const asked = []
const rows = mergeParts([PART], [LANE], (lane) => {
asked.push(lane.name)
return '40 minutes'
})
expect(asked).toEqual(['worker'])
expect(rows[0].stuckFor).toBe('40 minutes')
// The roster still owns a matched row's state — `stuckFor` is a note
// beside it, not a verdict that overrides the heartbeat.
expect(rows[0].state).toBe('ok')
})
it('handles an empty everything without inventing rows', () => {
expect(mergeParts([], [])).toEqual([])
expect(mergeParts(undefined, undefined)).toEqual([])
})
})
// The nudge. Operator, 2026-09-23: *"there needs to be something that tells
// you user to bump those numbers to improve processing rate or they'd never
// know the controls exist."*
//
// The caps ship at one of each, so on a busy instance the SHIPPED
// CONFIGURATION is the bottleneck. What is pinned here is that it fires only
// when raising the cap would actually help — a notice on a lane that is
// keeping up, or one already at the machine's limit, is a notice people learn
// to scroll past, and at that point it is worse than not having it.
describe('laneAdvice', () => {
const busyAtCap = {
slots_cap: 1, ceiling: 7, pending: 4060,
live: { present: true, pool: 1, active: 1 },
}
it('speaks when the cap is the limiting factor', () => {
const advice = laneAdvice(busyAtCap)
expect(advice).toContain('4,060 waiting')
expect(advice).toContain('Raise the cap')
// The headroom, so it is an instruction rather than a complaint.
expect(advice).toContain('7')
})
it('says something different about a lane that is switched off', () => {
const advice = laneAdvice({ ...busyAtCap, slots_cap: 0 })
expect(advice).toContain('this lane is off')
})
it('stays quiet when the lane is keeping up', () => {
expect(laneAdvice({ ...busyAtCap, pending: 2 })).toBeNull()
})
it('stays quiet when the cap is not the thing holding it back', () => {
// Four allowed, two running — the sizing pass has room it has not taken,
// so the queue is not the cap's fault.
expect(laneAdvice({
...busyAtCap, slots_cap: 4, live: { present: true, pool: 2, active: 2 },
})).toBeNull()
})
it('stays quiet at the machine ceiling, where the advice is untakeable', () => {
// The one case where saying "raise the cap" would send someone to a
// control that will refuse them.
expect(laneAdvice({ ...busyAtCap, ceiling: 1 })).toBeNull()
})
it('stays quiet about a lane that is not answering', () => {
// An unswept read is not a verdict: nothing replied, so "all workers
// busy" is a claim nobody made.
expect(laneAdvice({
...busyAtCap, live: { present: false, pool: null, active: 0 },
})).toBeNull()
})
it('stays quiet when the backlog is unknown rather than reading it as huge', () => {
expect(laneAdvice({ ...busyAtCap, pending: null })).toBeNull()
})
it('says nothing about a row that is not a lane at all', () => {
expect(laneAdvice(undefined)).toBeNull()
})
})
// When the worker numbers were read. They stopped being fetched per page load
// on 2026-09-23 — a sweep measures them on a timer and stores them — so the
// age belongs on screen. Operator: *"is there a reason this info isn't being
// tracked in the background and stored in some way?"*
describe('measuredAt', () => {
it('reports the newest reading across the lanes', () => {
expect(measuredAt([
{ measured_at: '2026-09-23T20:00:00Z' },
{ measured_at: '2026-09-23T20:00:30Z' },
{ measured_at: '2026-09-23T19:59:30Z' },
])).toBe('2026-09-23T20:00:30Z')
})
it('ignores a lane the sweep has never written', () => {
// A lane added by a newer build, or a first boot mid-sweep. It must not
// drag the reported age back to null while other lanes have real ones.
expect(measuredAt([
{ measured_at: null },
{ measured_at: '2026-09-23T20:00:00Z' },
])).toBe('2026-09-23T20:00:00Z')
})
it('says null when nothing has been measured, rather than guessing now', () => {
// The first period of a fresh install. "Not measured yet" and "measured
// just now" are opposite claims, and defaulting to the clock would make
// the page assert the wrong one at exactly the moment it knows least.
expect(measuredAt([{ measured_at: null }])).toBeNull()
expect(measuredAt([])).toBeNull()
expect(measuredAt(undefined)).toBeNull()
})
})