feat(agent): idle-unload GPU models to free VRAM when the queue is idle
The SigLIP embedder + YOLO proposers load lazily then stay resident for the container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded. Add a monitor thread that unloads the torch-owned models after cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with the GPU genuinely idle (active==0, buffer drained, no job completed in the window), then torch.cuda.empty_cache() to hand the blocks back to the driver. They reload lazily on the next job via the existing _ensure_embedder / _proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in /status (models_loaded) and the agent UI pipe line; the VRAM meter drops too. Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident (no clean unload API) — idle VRAM drops substantially, not to zero. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa
This commit is contained in:
@@ -75,3 +75,18 @@ class CropEmbedder:
|
||||
pooled = out.pooler_output if hasattr(out, "pooler_output") else out
|
||||
arr = pooled.float().cpu().numpy().astype(np.float32)
|
||||
return [row.reshape(-1).tolist() for row in arr]
|
||||
|
||||
def unload(self) -> bool:
|
||||
"""Drop the loaded model so its VRAM can be reclaimed — the idle monitor
|
||||
calls this after a spell with no work so an idle agent doesn't squat on
|
||||
the card; the next embed() reloads it lazily (a few seconds). Held under
|
||||
BOTH the load and inference locks so it can never race a concurrent load
|
||||
or an in-flight forward pass. Returns True if a model was actually
|
||||
released (the caller then runs one empty_cache() to hand the freed blocks
|
||||
back to the driver)."""
|
||||
with self._load_lock, self._infer_lock:
|
||||
if self._model is None:
|
||||
return False
|
||||
self._model = None
|
||||
self._processor = None
|
||||
return True
|
||||
|
||||
Reference in New Issue
Block a user