CI and images / lint (push) Failing after 3s
CI and images / extension-version (push) Successful in 4s
CI and images / frontend-build (push) Successful in 31s
CI and images / backend-lint-and-test (push) Successful in 35s
CI and images / integration (push) Successful in 2m44s
CI and images / sign-extension (push) Skipped
CI and images / build-web (push) Skipped
CI and images / smoke-web (push) Skipped
CI and images / promote (push) Skipped
CI and images / build-agent (push) Skipped
Operator: "there is a repull every time this page loads is there a reason
this info isn't being tracked in the background and stored in some way?"
There was a reason and it had expired, and underneath it there was plain
waste.
The expired one: /api/system/workers was deliberately uncached because an
operator dragging the stepper must not be shown a pre-change value. That
stopped being true at 1353d34, when the UI began patching its row from the
write's reply instead of refetching.
The waste: size_worker_lanes already inspected the broker on a timer to
decide pool sizes — computing the pool, active, reserved and queue depth
the page shows, using them, and discarding them. The browser then asked
the broker for the same numbers four times a minute, per open tab.
So one inspect now feeds three things: the sizing decision, a stored
sample (worker_lane_sample, alembic 0107), and the celery roster. No
request path touches the broker at all — the roster refresh comes off
/api/system/health too, where it had been rate-limited to 20s and so made
worker liveness a function of whether anyone had a browser open.
Consequences, stated rather than hidden:
- The live figures are up to one sweep old. measured_at travels with each
lane and the page says how old, because a stale number presented as
current is how someone watches a queue "not move" that is moving.
- The sweep is the roster's only writer now, so its period and the
staleness thresholds are in a relationship. 60s against a 90s stale
threshold left one missed tick between normal and all-yellow — the
shape of lesson #4355 — so the period is 30s, named once in
worker_lanes, and system_health asserts its headroom at import with a
test stating the same thing in prose.
- An idle lane therefore also gives a worker back twice as fast. That is
the direction asked for: "idle instances quiet down when not running".
Also bounds the inspect in push_lane_cap, which was an await with no
deadline (rule 156) — harmless while it ran on a request, less so now
that it runs in a background task where a hang would be silent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
227 lines
8.4 KiB
JavaScript
227 lines
8.4 KiB
JavaScript
import { describe, expect, it } from 'vitest'
|
|
|
|
import { laneAdvice, measuredAt, mergeParts, queueKey } from '../src/utils/systemParts.js'
|
|
|
|
// The System tab's one table (operator 2026-09-23: "combine the two sections
|
|
// into a single table"). What is pinned here is the JOIN, because its failure
|
|
// is silent and is precisely the thing the merge exists to fix: a lane that
|
|
// stops matching its roster part does not throw — the table grows a SECOND row
|
|
// for the same worker, one with controls and one without.
|
|
|
|
const PART = {
|
|
key: 'celery:default,download,import,thumbnail',
|
|
kind: 'celery',
|
|
name: 'Worker',
|
|
state: 'ok',
|
|
detail: 'Worker is running',
|
|
// The roster SORTS a celery part's queues into its key.
|
|
queues: ['default', 'download', 'import', 'thumbnail'],
|
|
}
|
|
|
|
const LANE = {
|
|
name: 'worker',
|
|
display_name: 'Worker',
|
|
// The lane table lists them in the order the role reads them, which is NOT
|
|
// sorted. If the join ever compares these two lists directly rather than as
|
|
// sets, this fixture is what catches it.
|
|
queues: ['default', 'import', 'thumbnail', 'download'],
|
|
slots_cap: 4,
|
|
ceiling: 8,
|
|
live: { present: true, replicas: 1, pool: 2, active: 0 },
|
|
pending: 0,
|
|
}
|
|
|
|
const POSTGRES = {
|
|
key: 'postgres', kind: 'datastore', name: 'PostgreSQL',
|
|
state: 'ok', detail: 'answering', latency_ms: 2.5,
|
|
}
|
|
|
|
describe('queueKey', () => {
|
|
it('does not care what order either side lists its queues in', () => {
|
|
expect(queueKey(LANE.queues)).toBe(queueKey(PART.queues))
|
|
})
|
|
|
|
it('survives a part that has no queues at all', () => {
|
|
// A datastore, and also the stale `Worker ()` row a previous deployment
|
|
// left in the roster with an empty queue set. Neither must match a lane.
|
|
expect(queueKey(undefined)).toBe('')
|
|
expect(queueKey([])).toBe('')
|
|
})
|
|
})
|
|
|
|
describe('mergeParts', () => {
|
|
it('gives a worker ONE row, carrying its controls', () => {
|
|
const rows = mergeParts([PART, POSTGRES], [LANE])
|
|
|
|
expect(rows).toHaveLength(2)
|
|
const worker = rows.find((r) => r.name === 'Worker')
|
|
expect(worker.lane).toBe(LANE)
|
|
expect(rows.filter((r) => r.name === 'Worker')).toHaveLength(1)
|
|
})
|
|
|
|
it('leaves a datastore without a lane rather than guessing one', () => {
|
|
const pg = mergeParts([PART, POSTGRES], [LANE]).find((r) => r.key === 'postgres')
|
|
expect(pg.lane).toBeUndefined()
|
|
})
|
|
|
|
it('still lists a lane the roster has never seen', () => {
|
|
// Parts are learned as they appear; the lane table is known up front. The
|
|
// lane an operator most needs to find — an optional one, never started —
|
|
// is exactly the one with no roster entry.
|
|
const ml = {
|
|
...LANE, name: 'ml', display_name: 'ML tagging', queues: ['ml'],
|
|
slots_cap: 0, optional: true,
|
|
live: { present: false, replicas: 0, pool: null, active: 0 },
|
|
}
|
|
const rows = mergeParts([POSTGRES], [ml])
|
|
|
|
const row = rows.find((r) => r.name === 'ML tagging')
|
|
expect(row.lane).toBe(ml)
|
|
expect(row.kindLabel).toBe('optional lane')
|
|
})
|
|
|
|
it('does not call a lane capped at zero broken', () => {
|
|
// The roster only knows a heartbeat age, so it goes on saying "is running"
|
|
// for a lane the operator deliberately dialled to nothing. The lane knows
|
|
// the difference; reporting the operator's own choice as a fault is how an
|
|
// indicator stops being read.
|
|
const off = { ...LANE, slots_cap: 0 }
|
|
const row = mergeParts([PART], [off])[0]
|
|
|
|
expect(row.detail).toBe('off — cap is zero')
|
|
})
|
|
|
|
it('puts the broken thing first, whatever it is', () => {
|
|
const down = { ...POSTGRES, state: 'down', detail: 'not answering' }
|
|
const rows = mergeParts([PART, down], [LANE])
|
|
|
|
expect(rows[0].name).toBe('PostgreSQL')
|
|
})
|
|
|
|
it('otherwise puts the rows you can act on first', () => {
|
|
const rows = mergeParts([POSTGRES, PART], [LANE])
|
|
|
|
expect(rows.map((r) => r.name)).toEqual(['Worker', 'PostgreSQL'])
|
|
})
|
|
|
|
it('reports a wedged lane, and only through the reporter it was given', () => {
|
|
// `laneStuckFor` is passed in rather than imported, so this file does not
|
|
// re-test the store's rule — it tests that the merge asks.
|
|
const asked = []
|
|
const rows = mergeParts([PART], [LANE], (lane) => {
|
|
asked.push(lane.name)
|
|
return '40 minutes'
|
|
})
|
|
|
|
expect(asked).toEqual(['worker'])
|
|
expect(rows[0].stuckFor).toBe('40 minutes')
|
|
// The roster still owns a matched row's state — `stuckFor` is a note
|
|
// beside it, not a verdict that overrides the heartbeat.
|
|
expect(rows[0].state).toBe('ok')
|
|
})
|
|
|
|
it('handles an empty everything without inventing rows', () => {
|
|
expect(mergeParts([], [])).toEqual([])
|
|
expect(mergeParts(undefined, undefined)).toEqual([])
|
|
})
|
|
})
|
|
|
|
|
|
// The nudge. Operator, 2026-09-23: *"there needs to be something that tells
|
|
// you user to bump those numbers to improve processing rate or they'd never
|
|
// know the controls exist."*
|
|
//
|
|
// The caps ship at one of each, so on a busy instance the SHIPPED
|
|
// CONFIGURATION is the bottleneck. What is pinned here is that it fires only
|
|
// when raising the cap would actually help — a notice on a lane that is
|
|
// keeping up, or one already at the machine's limit, is a notice people learn
|
|
// to scroll past, and at that point it is worse than not having it.
|
|
|
|
describe('laneAdvice', () => {
|
|
const busyAtCap = {
|
|
slots_cap: 1, ceiling: 7, pending: 4060,
|
|
live: { present: true, pool: 1, active: 1 },
|
|
}
|
|
|
|
it('speaks when the cap is the limiting factor', () => {
|
|
const advice = laneAdvice(busyAtCap)
|
|
expect(advice).toContain('4,060 waiting')
|
|
expect(advice).toContain('Raise the cap')
|
|
// The headroom, so it is an instruction rather than a complaint.
|
|
expect(advice).toContain('7')
|
|
})
|
|
|
|
it('says something different about a lane that is switched off', () => {
|
|
const advice = laneAdvice({ ...busyAtCap, slots_cap: 0 })
|
|
expect(advice).toContain('this lane is off')
|
|
})
|
|
|
|
it('stays quiet when the lane is keeping up', () => {
|
|
expect(laneAdvice({ ...busyAtCap, pending: 2 })).toBeNull()
|
|
})
|
|
|
|
it('stays quiet when the cap is not the thing holding it back', () => {
|
|
// Four allowed, two running — the sizing pass has room it has not taken,
|
|
// so the queue is not the cap's fault.
|
|
expect(laneAdvice({
|
|
...busyAtCap, slots_cap: 4, live: { present: true, pool: 2, active: 2 },
|
|
})).toBeNull()
|
|
})
|
|
|
|
it('stays quiet at the machine ceiling, where the advice is untakeable', () => {
|
|
// The one case where saying "raise the cap" would send someone to a
|
|
// control that will refuse them.
|
|
expect(laneAdvice({ ...busyAtCap, ceiling: 1 })).toBeNull()
|
|
})
|
|
|
|
it('stays quiet about a lane that is not answering', () => {
|
|
// An unswept read is not a verdict: nothing replied, so "all workers
|
|
// busy" is a claim nobody made.
|
|
expect(laneAdvice({
|
|
...busyAtCap, live: { present: false, pool: null, active: 0 },
|
|
})).toBeNull()
|
|
})
|
|
|
|
it('stays quiet when the backlog is unknown rather than reading it as huge', () => {
|
|
expect(laneAdvice({ ...busyAtCap, pending: null })).toBeNull()
|
|
})
|
|
|
|
it('says nothing about a row that is not a lane at all', () => {
|
|
expect(laneAdvice(undefined)).toBeNull()
|
|
})
|
|
})
|
|
|
|
|
|
// When the worker numbers were read. They stopped being fetched per page load
|
|
// on 2026-09-23 — a sweep measures them on a timer and stores them — so the
|
|
// age belongs on screen. Operator: *"is there a reason this info isn't being
|
|
// tracked in the background and stored in some way?"*
|
|
|
|
describe('measuredAt', () => {
|
|
it('reports the newest reading across the lanes', () => {
|
|
expect(measuredAt([
|
|
{ measured_at: '2026-09-23T20:00:00Z' },
|
|
{ measured_at: '2026-09-23T20:00:30Z' },
|
|
{ measured_at: '2026-09-23T19:59:30Z' },
|
|
])).toBe('2026-09-23T20:00:30Z')
|
|
})
|
|
|
|
it('ignores a lane the sweep has never written', () => {
|
|
// A lane added by a newer build, or a first boot mid-sweep. It must not
|
|
// drag the reported age back to null while other lanes have real ones.
|
|
expect(measuredAt([
|
|
{ measured_at: null },
|
|
{ measured_at: '2026-09-23T20:00:00Z' },
|
|
])).toBe('2026-09-23T20:00:00Z')
|
|
})
|
|
|
|
it('says null when nothing has been measured, rather than guessing now', () => {
|
|
// The first period of a fresh install. "Not measured yet" and "measured
|
|
// just now" are opposite claims, and defaulting to the clock would make
|
|
// the page assert the wrong one at exactly the moment it knows least.
|
|
expect(measuredAt([{ measured_at: null }])).toBeNull()
|
|
expect(measuredAt([])).toBeNull()
|
|
expect(measuredAt(undefined)).toBeNull()
|
|
})
|
|
})
|