fix(agent): stable util-band autoscaler + live GPU meters
CI / lint (push) Successful in 4s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 27s
CI / integration (push) Successful in 3m40s

Two operator-reported issues with the GPU agent:

1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb
   probed +1, judged it over a too-short noisy throughput window, saw no clear
   gain and reverted -1 — every tick. Replace it with a GPU-utilization-band
   controller: HOLD while smoothed util sits in a healthy band, grow only on
   clear spare capacity (util below the low mark + VRAM headroom), shrink under
   saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced
   (DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool.
   Load stays consistent instead of probe/reverting.

2. GPU util/VRAM bars only updated on manual refresh. They rode the /status
   poll, which blocks on the curator queue call (slow when curator is busy), so
   the meters froze between refreshes. Give them a dedicated /gpu endpoint
   (local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop
   the curator queue-status timeout 15s -> 5s so /status itself stays snappy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-30 19:16:17 -04:00
parent c259d03618
commit 3b34230fbd
3 changed files with 70 additions and 56 deletions
+3 -1
View File
@@ -93,6 +93,8 @@ class FcClient:
return r.content
def queue_status(self) -> dict:
r = self.s.get(f"{self.base}/api/gpu/status", timeout=15)
# Short timeout: this backs the UI /status poll, so a busy curator must
# not hang the page for long (the GPU meters poll /gpu separately).
r = self.s.get(f"{self.base}/api/gpu/status", timeout=5)
r.raise_for_status()
return r.json()