feat(agent): idle-unload GPU models to free VRAM when the queue is idle
The SigLIP embedder + YOLO proposers load lazily then stay resident for the container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded. Add a monitor thread that unloads the torch-owned models after cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with the GPU genuinely idle (active==0, buffer drained, no job completed in the window), then torch.cuda.empty_cache() to hand the blocks back to the driver. They reload lazily on the next job via the existing _ensure_embedder / _proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in /status (models_loaded) and the agent UI pipe line; the VRAM meter drops too. Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident (no clean unload API) — idle VRAM drops substantially, not to zero. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa
This commit is contained in:
@@ -170,6 +170,13 @@ class YoloProposer:
|
||||
))
|
||||
return out
|
||||
|
||||
def unload(self) -> None:
|
||||
"""Drop the loaded YOLO so its VRAM can be reclaimed; detect() reloads it
|
||||
lazily on the next job. Leaves _ok untouched — a healthy proposer comes
|
||||
back, but one that self-disabled on a fault stays off."""
|
||||
with self._lock:
|
||||
self._model = None
|
||||
|
||||
|
||||
class Proposers:
|
||||
"""The agent's proposer set, built from config. Each detector is optional —
|
||||
@@ -216,3 +223,11 @@ class Proposers:
|
||||
|
||||
def panels(self, image):
|
||||
return self._top(self._panel, image, self.cfg.max_panels)
|
||||
|
||||
def unload(self) -> None:
|
||||
"""Release every loaded proposer's YOLO (idle VRAM reclaim). The worker
|
||||
also drops its reference to this Proposers and rebuilds a fresh one via
|
||||
_proposers_for on the next job, so this is belt-and-braces."""
|
||||
for p in (self._person, self._anatomy, self._panel):
|
||||
if p is not None:
|
||||
p.unload()
|
||||
|
||||
Reference in New Issue
Block a user