feat(agent): graceful Start/Stop with starting/stopping states + instant status
Operator: the buttons fire but the status view doesn't reflect the change. Cause: act() ignored the POST's own status response and waited on the separate /status poll (which lags behind the curator queue call). Now: - act() applies the POST's returned status immediately for instant feedback, and shows an optimistic "starting"/"stopping" state (pulsing, buttons disabled) the moment it's clicked. - A stop that still has in-flight jobs draining shows "stopping" until active hits 0, then resolves to "stopped" on its own. - applyStatus() guards the /status-only fields (connection pill + queue) so the lean action response can't blank them — the Start/Stop path deliberately skips the slow curator call to stay snappy. Also de-duplicate GPU reads: read_gpu() now caches (1s TTL) with one probe at a time, and /status no longer spawns its own nvidia-smi — so the fast /gpu poll + autoscaler + /status share a single subprocess instead of piling up in the server thread pool (which was what made clicks feel dead under load). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
+37
-2
@@ -1,10 +1,24 @@
|
||||
"""GPU load readout via nvidia-smi (present in the container thanks to the
|
||||
NVIDIA Container Toolkit's `utility` capability). Returns None if unavailable —
|
||||
the UI just shows n/a (e.g. CPU-fallback run)."""
|
||||
the UI just shows n/a (e.g. CPU-fallback run).
|
||||
|
||||
Reads are CACHED and de-duplicated: the UI meter polls fast, /status reads it,
|
||||
and the autoscaler samples it — if each spawned its own `nvidia-smi` (slow on a
|
||||
busy GPU) those blocking subprocesses would pile up in the server's thread pool
|
||||
and make the Start/Stop buttons feel dead. So a short TTL serves recent callers
|
||||
from cache, and only ONE probe runs at a time (others get the last value)."""
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
|
||||
_TTL = 1.0 # seconds a sample is reused before re-probing
|
||||
_lock = threading.Lock()
|
||||
_cache: dict | None = None
|
||||
_cache_t = 0.0
|
||||
_probing = False
|
||||
|
||||
|
||||
def read_gpu() -> dict | None:
|
||||
def _probe() -> dict | None:
|
||||
try:
|
||||
out = subprocess.run(
|
||||
[
|
||||
@@ -28,3 +42,24 @@ def read_gpu() -> dict | None:
|
||||
}
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
|
||||
|
||||
def read_gpu(max_age: float = _TTL) -> dict | None:
|
||||
"""Latest GPU reading, cached. Serves from cache when fresh; when stale,
|
||||
exactly one caller re-probes while the rest get the last value — so request
|
||||
threads never block behind more than one `nvidia-smi`."""
|
||||
global _cache, _cache_t, _probing
|
||||
now = time.monotonic()
|
||||
with _lock:
|
||||
fresh = _cache is not None and (now - _cache_t) < max_age
|
||||
if fresh or _probing: # fresh, or a probe is already running
|
||||
return _cache
|
||||
_probing = True
|
||||
try:
|
||||
val = _probe()
|
||||
finally:
|
||||
with _lock:
|
||||
_cache = val
|
||||
_cache_t = time.monotonic()
|
||||
_probing = False
|
||||
return val
|
||||
|
||||
Reference in New Issue
Block a user