feat(agent): idle-unload GPU models to free VRAM when the queue is idle
CI / lint (push) Successful in 3s
CI / frontend-build (push) Successful in 22s
CI / backend-lint-and-test (push) Successful in 38s
CI / integration (push) Successful in 4m3s

The SigLIP embedder + YOLO proposers load lazily then stay resident for the
container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of
VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep
mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded.

Add a monitor thread that unloads the torch-owned models after
cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with
the GPU genuinely idle (active==0, buffer drained, no job completed in the
window), then torch.cuda.empty_cache() to hand the blocks back to the driver.
They reload lazily on the next job via the existing _ensure_embedder /
_proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in
/status (models_loaded) and the agent UI pipe line; the VRAM meter drops too.

Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident
(no clean unload API) — idle VRAM drops substantially, not to zero.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa
This commit is contained in:
2026-07-17 12:57:31 -04:00
parent ec66ea5f83
commit 57e52433d0
5 changed files with 119 additions and 1 deletions
+10
View File
@@ -51,6 +51,12 @@ class Config:
bandwidth_limit_mb_s: float # aggregate download cap in MEGABYTES/s across
# all downloaders + video streams (0 = unlimited);
# tunable live from the agent UI
idle_unload_seconds: float # after this long with the GPU idle (nothing in
# flight, queue empty or Stopped), unload the
# SigLIP embedder + YOLO proposers to free their
# VRAM; they reload lazily on the next job. A
# 24/7 agent otherwise squats on ~5GB doing
# nothing. 0 disables (keep models warm forever).
@classmethod
def from_env(cls) -> Config:
@@ -87,4 +93,8 @@ class Config:
# link to ~1-1.5 MB/s per stream, browser included). Raise it (or 0)
# from the agent UI on wired/faster networks.
bandwidth_limit_mb_s=float(os.environ.get("BANDWIDTH_LIMIT_MB_S", "8")),
# 5 min: long enough that a lull between job bursts doesn't thrash the
# (few-second) reload, short enough that an agent left running with an
# empty queue hands its VRAM back promptly.
idle_unload_seconds=float(os.environ.get("IDLE_UNLOAD_SECONDS", "300")),
)