feat(agent): idle-unload GPU models to free VRAM when the queue is idle
The SigLIP embedder + YOLO proposers load lazily then stay resident for the container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded. Add a monitor thread that unloads the torch-owned models after cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with the GPU genuinely idle (active==0, buffer drained, no job completed in the window), then torch.cuda.empty_cache() to hand the blocks back to the driver. They reload lazily on the next job via the existing _ensure_embedder / _proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in /status (models_loaded) and the agent UI pipe line; the VRAM meter drops too. Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident (no clean unload API) — idle VRAM drops substantially, not to zero. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa
This commit is contained in:
@@ -51,6 +51,12 @@ class Config:
|
||||
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
|
||||
# all downloaders + video streams (0 = unlimited);
|
||||
# tunable live from the agent UI
|
||||
idle_unload_seconds: float # after this long with the GPU idle (nothing in
|
||||
# flight, queue empty or Stopped), unload the
|
||||
# SigLIP embedder + YOLO proposers to free their
|
||||
# VRAM; they reload lazily on the next job. A
|
||||
# 24/7 agent otherwise squats on ~5GB doing
|
||||
# nothing. 0 disables (keep models warm forever).
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> Config:
|
||||
@@ -87,4 +93,8 @@ class Config:
|
||||
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
|
||||
# from the agent UI on wired/faster networks.
|
||||
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
|
||||
# 5 min: long enough that a lull between job bursts doesn't thrash the
|
||||
# (few-second) reload, short enough that an agent left running with an
|
||||
# empty queue hands its VRAM back promptly.
|
||||
idle_unload_seconds=float(os.environ.get("IDLE_UNLOAD_SECONDS", "300")),
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user