fix(agent): stable util-band autoscaler + live GPU meters
Two operator-reported issues with the GPU agent: 1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb probed +1, judged it over a too-short noisy throughput window, saw no clear gain and reverted -1 — every tick. Replace it with a GPU-utilization-band controller: HOLD while smoothed util sits in a healthy band, grow only on clear spare capacity (util below the low mark + VRAM headroom), shrink under saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced (DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool. Load stays consistent instead of probe/reverting. 2. GPU util/VRAM bars only updated on manual refresh. They rode the /status poll, which blocks on the curator queue call (slow when curator is busy), so the meters froze between refreshes. Give them a dedicated /gpu endpoint (local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop the curator queue-status timeout 15s -> 5s so /status itself stays snappy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -93,6 +93,8 @@ class FcClient:
|
||||
return r.content
|
||||
|
||||
def queue_status(self) -> dict:
|
||||
r = self.s.get(f"{self.base}/api/gpu/status", timeout=15)
|
||||
# Short timeout: this backs the UI /status poll, so a busy curator must
|
||||
# not hang the page for long (the GPU meters poll /gpu separately).
|
||||
r = self.s.get(f"{self.base}/api/gpu/status", timeout=5)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
Reference in New Issue
Block a user