Build images / sign-extension (push) Successful in 3s
CI / lint (push) Successful in 2s
CI / extension-version (push) Successful in 3s
Build images / build-agent (push) Successful in 6s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 36s
Build images / build-ml (push) Successful in 1m55s
Build images / build-web (push) Successful in 1m54s
CI / integration (push) Successful in 2m16s
Build images / smoke-web (push) Failing after 7m48s
Build images / promote (push) Skipped
Milestone 422 step 7 — the one sweep in this milestone that decides rather than obeys, so it is off until a lane is opted in, bounded by the operator's cap, floored at the operator's value, and it reports every decision including the ones where it did nothing. Growth needs BOTH halves: all slots busy AND a backlog. Depth alone means celery is about to pick those up and growing would add idle children (#1253 is that bug in the GPU agent); saturation alone means the lane is busy with exactly as much work as exists. The backlog is depth PLUS reserved, because celery prefetches and LLEN reads 0 while a worker holds thirty tasks in memory — the case an LLEN-only autoscaler misses entirely, and the reason step 2 plumbed `reserved` through. The two sweeps had to be taught not to fight. The reconcile drives every lane to its stored slots every five minutes, which would have reverted each grow on the next tick: grow, revert, grow, revert, forever. For an autoscaling lane the stored value is now a FLOOR — restored when a lane falls below it, never taken back above it. The operator's "a task that runs for x concurrent time" idea stays a UI warning rather than a trigger: a long task does not finish sooner because the lane gained a slot, so scaling on it would spend memory to change nothing. Read from `task_run` on our own wall clock, not celery's `time_start`, which is the WORKER's monotonic clock and would produce a duration that is meaningless in the direction that matters — plausible. Caught while reading it back: the first version read the stored slots as the CURRENT pool. The autoscaler never writes that row, so every tick would have proposed floor+1 — resizing nothing, reporting `grew` anyway (a replica already past the target is issued no message and reports success), and capping the lane one slot above its floor forever while claiming otherwise. It now reads the live pool and keeps the stored value purely as the floor, and the tests fix the two to different numbers so an equal-fixture pass cannot hide it again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR
193 lines
7.5 KiB
JavaScript
193 lines
7.5 KiB
JavaScript
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
|
|
import { setActivePinia, createPinia } from 'pinia'
|
|
import { laneStuckFor, useSystemActivityStore } from '../src/stores/systemActivity.js'
|
|
|
|
// Milestone 422 step 4. Covers the store half of the worker-lane dial — the
|
|
// part that decides what the card can tell the operator.
|
|
//
|
|
// The distinction being protected: a change that was STORED but not pushed
|
|
// (`applied: false`, because the lane is restarting) is not a failure, and a
|
|
// REFUSED value (400) is. Collapsing those two into one message is how a
|
|
// control stops being trustworthy — one invites waiting, the other invites
|
|
// changing what you asked for.
|
|
|
|
function stubFetch(handler) {
|
|
globalThis.fetch = vi.fn(async (url, init) => {
|
|
const { status, body } = handler(url, init)
|
|
return {
|
|
ok: status >= 200 && status < 300,
|
|
status,
|
|
statusText: String(status),
|
|
text: async () => (body == null ? '' : JSON.stringify(body)),
|
|
}
|
|
})
|
|
}
|
|
|
|
const LANES_BODY = {
|
|
lanes: [
|
|
{
|
|
name: 'worker', display_name: 'Worker',
|
|
queues: ['default', 'import', 'thumbnail', 'download'],
|
|
slots: 1, slots_cap: 4, ceiling: 8, enabled: true, memory_bound: false,
|
|
live: { present: true, replicas: 1, pool: 1, active: 0, reserved: 3 },
|
|
queue_depth: 5, pending: 8,
|
|
},
|
|
{
|
|
name: 'ml', display_name: 'ML tagging', queues: ['ml'],
|
|
slots: 0, slots_cap: 1, ceiling: 2, enabled: false, memory_bound: true,
|
|
live: { present: false, replicas: 0, pool: null, active: 0, reserved: 0 },
|
|
queue_depth: null, pending: null,
|
|
},
|
|
],
|
|
fetched_at: '2026-09-22T12:00:00Z',
|
|
}
|
|
|
|
describe('worker lanes store', () => {
|
|
beforeEach(() => setActivePinia(createPinia()))
|
|
afterEach(() => vi.restoreAllMocks())
|
|
|
|
it('loads the lanes', async () => {
|
|
stubFetch(() => ({ status: 200, body: LANES_BODY }))
|
|
const s = useSystemActivityStore()
|
|
await s.loadLanes()
|
|
expect(s.lanes.lanes.map((l) => l.name)).toEqual(['worker', 'ml'])
|
|
})
|
|
|
|
it('a load failure records the error rather than throwing at the caller', async () => {
|
|
// The card polls this every 3s. An unhandled rejection per tick would
|
|
// drown the console and stop the other pollers in the same function.
|
|
stubFetch(() => ({ status: 500, body: { error: 'boom' } }))
|
|
const s = useSystemActivityStore()
|
|
await expect(s.loadLanes()).resolves.toBeUndefined()
|
|
expect(s.lastError).toBeTruthy()
|
|
})
|
|
|
|
it('setLane posts only the fields it was given', async () => {
|
|
// Partial update: the stepper sends slots without restating a cap it did
|
|
// not touch. Sending the whole row back would make two operators editing
|
|
// different fields clobber each other.
|
|
const calls = []
|
|
stubFetch((url, init) => {
|
|
calls.push({ url, init })
|
|
if (init?.method === 'POST') {
|
|
return { status: 200, body: { name: 'worker', slots: 2, applied: true } }
|
|
}
|
|
return { status: 200, body: LANES_BODY }
|
|
})
|
|
const s = useSystemActivityStore()
|
|
await s.setLane('worker', { slots: 2 })
|
|
|
|
const post = calls.find((c) => c.init?.method === 'POST')
|
|
expect(post.url).toContain('/api/system/workers/worker')
|
|
expect(JSON.parse(post.init.body)).toEqual({ slots: 2 })
|
|
})
|
|
|
|
it('setLane refetches so the card shows the server truth, not the guess', async () => {
|
|
// The reply is one lane; the card renders all of them plus live pool and
|
|
// pending. Patching the local row from the reply would leave every other
|
|
// column stale and eventually wrong.
|
|
let gets = 0
|
|
stubFetch((url, init) => {
|
|
if (init?.method === 'POST') return { status: 200, body: { applied: true } }
|
|
gets += 1
|
|
return { status: 200, body: LANES_BODY }
|
|
})
|
|
const s = useSystemActivityStore()
|
|
await s.setLane('worker', { slots: 2 })
|
|
expect(gets).toBe(1)
|
|
})
|
|
|
|
it('a stored-but-unapplied change comes back as applied:false, not an error', async () => {
|
|
// The lane is restarting. The value IS saved and the reconcile will carry
|
|
// it — so this must reach the card as information, not as a failure that
|
|
// invites the operator to set it again.
|
|
stubFetch((url, init) => {
|
|
if (init?.method === 'POST') {
|
|
return {
|
|
status: 200,
|
|
body: { applied: false, apply_error: 'lane is not running', slots: 2 },
|
|
}
|
|
}
|
|
return { status: 200, body: LANES_BODY }
|
|
})
|
|
const s = useSystemActivityStore()
|
|
const reply = await s.setLane('worker', { slots: 2 })
|
|
expect(reply.applied).toBe(false)
|
|
expect(reply.apply_error).toContain('not running')
|
|
})
|
|
|
|
it('a refused value throws so the card can show the reason', async () => {
|
|
// Deliberately NOT swallowed. The detail is written to be read by a person
|
|
// ("above what this container can hold"), and a control that silently does
|
|
// nothing is worse than one that refuses out loud.
|
|
stubFetch((url, init) => {
|
|
if (init?.method === 'POST') {
|
|
return {
|
|
status: 400,
|
|
body: { error: 'refused', detail: 'cap 10000 is above what this container can hold (2 for ML tagging)' },
|
|
}
|
|
}
|
|
return { status: 200, body: LANES_BODY }
|
|
})
|
|
const s = useSystemActivityStore()
|
|
// Assert the REASON is reachable, not merely that it threw. `toThrow()`
|
|
// alone passes whether the card can read the sentence or not — which is
|
|
// how the first version of the card shipped reading `e.detail` (always
|
|
// undefined) and would have shown the operator the bare word "refused".
|
|
const err = await s.setLane('ml', { slots_cap: 10000 }).catch((e) => e)
|
|
expect(err.status).toBe(400)
|
|
expect(err.body.detail).toContain('container can hold')
|
|
})
|
|
})
|
|
|
|
// --- the long-running-task warning (step 7) ----------------------------------
|
|
//
|
|
// The operator asked for "a task runs for x concurrent time" to TRIGGER
|
|
// growth. It became a warning: extra slots do not make a running task finish
|
|
// sooner. What is pinned here is the AND — a warning that fires on half its
|
|
// condition is one people learn to ignore, and at that point it is worse than
|
|
// not having it.
|
|
|
|
describe('laneStuckFor', () => {
|
|
const lane = (over = {}) => ({
|
|
oldest_running_minutes: 40,
|
|
live: { present: true, pool: 4, active: 4 },
|
|
...over,
|
|
})
|
|
|
|
it('warns when every slot is busy and the oldest task is old', () => {
|
|
expect(laneStuckFor(lane())).toBe('40 minutes')
|
|
})
|
|
|
|
it('says nothing when the lane has a free slot', () => {
|
|
expect(laneStuckFor(lane({ live: { present: true, pool: 4, active: 3 } })))
|
|
.toBeNull()
|
|
})
|
|
|
|
it('says nothing for a saturated lane whose tasks are young', () => {
|
|
expect(laneStuckFor(lane({ oldest_running_minutes: 2 }))).toBeNull()
|
|
})
|
|
|
|
it('says nothing when nothing is running', () => {
|
|
expect(laneStuckFor(lane({ oldest_running_minutes: null }))).toBeNull()
|
|
})
|
|
|
|
it('says nothing about a lane that is not answering', () => {
|
|
// Unknown is not busy. A lane nothing answered for has no live reading to
|
|
// call saturated, and asserting one would be a verdict from an unswept
|
|
// read — the same distinction `present` exists for everywhere else here.
|
|
expect(laneStuckFor(lane({ live: { present: false, pool: null, active: 0 } })))
|
|
.toBeNull()
|
|
})
|
|
|
|
it('says nothing when the pool is zero', () => {
|
|
// A lane sized to zero is not "fully busy at zero" — it is off.
|
|
expect(laneStuckFor(lane({ live: { present: true, pool: 0, active: 0 } })))
|
|
.toBeNull()
|
|
})
|
|
|
|
it('switches to hours once minutes stop being readable', () => {
|
|
expect(laneStuckFor(lane({ oldest_running_minutes: 195 }))).toBe('3 hours')
|
|
})
|
|
})
|