feat(agent): graceful Start/Stop with starting/stopping states + instant status
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 19s
CI / backend-lint-and-test (push) Successful in 26s
CI / integration (push) Successful in 3m22s

Operator: the buttons fire but the status view doesn't reflect the change. Cause:
act() ignored the POST's own status response and waited on the separate /status
poll (which lags behind the curator queue call). Now:

- act() applies the POST's returned status immediately for instant feedback, and
  shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled)
  the moment it's clicked.
- A stop that still has in-flight jobs draining shows "stopping" until active
  hits 0, then resolves to "stopped" on its own.
- applyStatus() guards the /status-only fields (connection pill + queue) so the
  lean action response can't blank them — the Start/Stop path deliberately skips
  the slow curator call to stay snappy.

Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a
time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll +
autoscaler + /status share a single subprocess instead of piling up in the
server thread pool (which was what made clicks feel dead under load).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-30 19:38:37 -04:00
parent 3b34230fbd
commit c2e9157822
2 changed files with 79 additions and 15 deletions
+37 -2
View File
@@ -1,10 +1,24 @@
"""GPU load readout via nvidia-smi (present in the container thanks to the
NVIDIA Container Toolkit's `utility` capability). Returns None if unavailable —
the UI just shows n/a (e.g. CPU-fallback run)."""
the UI just shows n/a (e.g. CPU-fallback run).
Reads are CACHED and de-duplicated: the UI meter polls fast, /status reads it,
and the autoscaler samples it — if each spawned its own `nvidia-smi` (slow on a
busy GPU) those blocking subprocesses would pile up in the server's thread pool
and make the Start/Stop buttons feel dead. So a short TTL serves recent callers
from cache, and only ONE probe runs at a time (others get the last value)."""
import subprocess
import threading
import time
_TTL = 1.0 # seconds a sample is reused before re-probing
_lock = threading.Lock()
_cache: dict | None = None
_cache_t = 0.0
_probing = False
def read_gpu() -> dict | None:
def _probe() -> dict | None:
try:
out = subprocess.run(
[
@@ -28,3 +42,24 @@ def read_gpu() -> dict | None:
}
except (ValueError, IndexError):
return None
def read_gpu(max_age: float = _TTL) -> dict | None:
"""Latest GPU reading, cached. Serves from cache when fresh; when stale,
exactly one caller re-probes while the rest get the last value — so request
threads never block behind more than one `nvidia-smi`."""
global _cache, _cache_t, _probing
now = time.monotonic()
with _lock:
fresh = _cache is not None and (now - _cache_t) < max_age
if fresh or _probing: # fresh, or a probe is already running
return _cache
_probing = True
try:
val = _probe()
finally:
with _lock:
_cache = val
_cache_t = time.monotonic()
_probing = False
return val