fix(agent): stable util-band autoscaler + live GPU meters

Two operator-reported issues with the GPU agent: 1. Worker count flopped almost every cycle, spiking the GPU. The hill-climb probed +1, judged it over a too-short noisy throughput window, saw no clear gain and reverted -1 — every tick. Replace it with a GPU-utilization-band controller: HOLD while smoothed util sits in a healthy band, grow only on clear spare capacity (util below the low mark + VRAM headroom), shrink under saturation or memory pressure. Util is EWMA-smoothed and decisions are spaced (DECIDE_EVERY samples), so a noisy nvidia-smi reading can't move the pool. Load stays consistent instead of probe/reverting. 2. GPU util/VRAM bars only updated on manual refresh. They rode the /status poll, which blocks on the curator queue call (slow when curator is busy), so the meters froze between refreshes. Give them a dedicated /gpu endpoint (local nvidia-smi only, no curator round-trip) polled every 1.5s, and drop the curator queue-status timeout 15s -> 5s so /status itself stays snappy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
2026-06-30 19:16:17 -04:00
parent c259d03618
commit 3b34230fbd
3 changed files with 70 additions and 56 deletions
@@ -59,6 +59,14 @@ async def auto(request: Request):
    return JSONResponse(worker.status())


+@app.get("/gpu")
+def gpu():
+    # GPU meters poll this on their own fast cadence. It only reads local
+    # nvidia-smi — no curator round-trip — so the util/VRAM bars stay live even
+    # when /status is slow waiting on the (sometimes busy) curator queue call.
+    return JSONResponse(read_gpu() or {})
+
+
@app.get("/logs")
 def logs():
    return JSONResponse({"lines": list(logbuf.LINES)})
@@ -233,12 +241,17 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
  conchint.textContent=s.auto?('auto-tuning to fill the GPU · max '+CAP):('manual · max '+CAP)
  if(document.activeElement!==conc) conc.value=s.concurrency
  conc.max=CAP
-  if(s.gpu){
-   const u=s.gpu.util_pct, used=s.gpu.mem_used_mb, tot=s.gpu.mem_total_mb||1
-   utillbl.textContent=u+'% · '+s.gpu.temp_c+'°C'; utilbar.style.width=u+'%'
+  queue.textContent=s.queue?('queue · pending '+s.queue.pending+' · in flight '+s.queue.leased+' · done '+s.queue.done+' · errored '+s.queue.error):'queue · unreachable'
+ }
+ // GPU meters poll their OWN endpoint on a fast cadence — kept off /status so a
+ // slow curator queue call can't freeze the bars (they only stale on refresh).
+ async function refreshGpu(){
+  let g; try{ g=await (await fetch('/gpu')).json() }catch{ return }
+  if(g && g.util_pct!=null){
+   const u=g.util_pct, used=g.mem_used_mb, tot=g.mem_total_mb||1
+   utillbl.textContent=u+'% · '+g.temp_c+'°C'; utilbar.style.width=u+'%'
   vramlbl.textContent=used+' / '+tot+' MB'; gpubar.style.width=Math.round(100*used/tot)+'%'
  } else { utillbl.textContent='n/a'; vramlbl.textContent='n/a (CPU?)'; utilbar.style.width='0%'; gpubar.style.width='0%' }
-  queue.textContent=s.queue?('queue · pending '+s.queue.pending+' · in flight '+s.queue.leased+' · done '+s.queue.done+' · errored '+s.queue.error):'queue · unreachable'
 }
 async function refreshLogs(){
  try{
@@ -255,6 +268,6 @@ _PAGE = """<!doctype html><html><head><meta charset=utf-8>
   t.select(); try{document.execCommand('copy')}catch{}; t.remove() }
  copybtn.textContent='Copied'; setTimeout(()=>{copybtn.textContent='Copy'},1200)
 }
- refresh(); refreshLogs()
- setInterval(refresh,3000); setInterval(refreshLogs,2500)
+ refresh(); refreshGpu(); refreshLogs()
+ setInterval(refresh,3000); setInterval(refreshGpu,1500); setInterval(refreshLogs,2500)
 </script></body></html>"""