2 Commits

Include renames

Author	SHA1	Message	Date
bvandeusenandClaude Opus 5	45bb7044f7	feat: the System tab reads a stored sample instead of inspecting per load (4295) ... Operator: "there is a repull every time this page loads is there a reason this info isn't being tracked in the background and stored in some way?" There was a reason and it had expired, and underneath it there was plain waste. The expired one: /api/system/workers was deliberately uncached because an operator dragging the stepper must not be shown a pre-change value. That stopped being true at 1353d34, when the UI began patching its row from the write's reply instead of refetching. The waste: size_worker_lanes already inspected the broker on a timer to decide pool sizes — computing the pool, active, reserved and queue depth the page shows, using them, and discarding them. The browser then asked the broker for the same numbers four times a minute, per open tab. So one inspect now feeds three things: the sizing decision, a stored sample (worker_lane_sample, alembic 0107), and the celery roster. No request path touches the broker at all — the roster refresh comes off /api/system/health too, where it had been rate-limited to 20s and so made worker liveness a function of whether anyone had a browser open. Consequences, stated rather than hidden: - The live figures are up to one sweep old. measured_at travels with each lane and the page says how old, because a stale number presented as current is how someone watches a queue "not move" that is moving. - The sweep is the roster's only writer now, so its period and the staleness thresholds are in a relationship. 60s against a 90s stale threshold left one missed tick between normal and all-yellow — the shape of lesson #4355 — so the period is 30s, named once in worker_lanes, and system_health asserts its headroom at import with a test stating the same thing in prose. - An idle lane therefore also gives a worker back twice as fast. That is the direction asked for: "idle instances quiet down when not running". Also bounds the inspect in push_lane_cap, which was an await with no deadline (rule 156) — harmless while it ran on a request, less so now that it runs in a background task where a hang would be silent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjrnpQjRgHdvq95rASoiR	2026-09-23 18:52:08 -04:00
bvandeusenandClaude Opus 5	fe4e0f2b71	feat: /api/system/health — one verdict for the whole stack (milestone 365 step 3) ... The single endpoint the nav indicator and the System page will both read. Composing a verdict is this module's job, not the UI's. Two kinds of part, answered differently. LEARNED — celery roles and the GPU agent, out of service_seen, where the question is "how long since it checked in" and the answer can be "it has not". PROBED — Postgres and Redis, always expected, never learned, because a last-seen for them would be actively misleading: that Redis answered thirty seconds ago says nothing about now. **The endpoint must never fail because something it checks has failed.** That inversion is easy to write by accident and it destroys the feature exactly when it is needed — a 500 when Redis is down instead of `redis: down`. Every probe is wrapped, every wait carries a deadline (rule 156), and the roster refresh swallows its own errors. The worst case is a part reported `unknown`, which is a true statement about the system. Postgres is probed first and gates the rest, because if it is unreachable nothing else can be read — and "the database is down" is the most useful single thing this can ever say. The staleness thresholds are the design risk, not the code, and they are deliberately generous: 90s to doubt, 300s to disbelieve. The constraint is a deploy rather than a crash — `docker compose up -d` rolls start-first, so a role is briefly served by two containers and then by neither while the old one drains. Thresholds tight enough to catch a crash in seconds would paint the page red on every update, and an alarm that cries wolf on every deploy is one nobody reads. Tune down only after watching a real deploy pass through. The numbers ship in the response so the UI can explain a `stale` without keeping a second copy of them. States are described in sentences rather than left as chips: "Scheduler has not checked in for 6 min — treat it as stopped" is what someone needs at the moment they are deciding whether to go and open Portainer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TTjbZZ6JirCMSaJzQV1RhA	2026-09-02 17:17:22 -04:00