feat(agent): idle-unload GPU models to free VRAM when the queue is idle #233

2026-07-17T13:07:03-04:00

bvandeusen commented

2026-07-17 13:07:03 -04:00

The GPU agent kept the SigLIP embedder + YOLO proposers resident for the container's whole lifetime, so a 24/7 agent with an empty queue squatted on ~5GB of VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep mode only sheds downloaders; even a UI Stop left the models loaded.

Adds a monitor thread that unloads the torch-owned models after cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with the GPU genuinely idle (active==0, buffer drained, no job completed in the window), then torch.cuda.empty_cache(). They reload lazily on the next job. Covers both sleep-mode idle and a full Stop. Surfaced in /status (models_loaded) + the agent UI pipe line.

Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident (no clean unload API) — idle VRAM drops substantially, not to zero.

CI green on dev head 57e5243 (run #2472). Merging publishes fabledcurator-agent:latest with idle-unload.

Scribe issue #1682.

The GPU agent kept the SigLIP embedder + YOLO proposers resident for the container's whole lifetime, so a 24/7 agent with an empty queue squatted on ~5GB of VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep mode only sheds downloaders; even a UI Stop left the models loaded. Adds a monitor thread that unloads the torch-owned models after `cfg.idle_unload_seconds` (env `IDLE_UNLOAD_SECONDS`, default 300; 0 disables) with the GPU genuinely idle (active==0, buffer drained, no job completed in the window), then `torch.cuda.empty_cache()`. They reload lazily on the next job. Covers both sleep-mode idle and a full Stop. Surfaced in `/status` (`models_loaded`) + the agent UI pipe line. Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident (no clean unload API) — idle VRAM drops substantially, not to zero. CI green on dev head 57e5243 (run #2472). Merging publishes `fabledcurator-agent:latest` with idle-unload. Scribe issue #1682.

bvandeusen added 1 commit 2026-07-17 13:07:03 -04:00

feat(agent): idle-unload GPU models to free VRAM when the queue is idle

CI / lint (push) Successful in 3s

Details

CI / frontend-build (push) Successful in 22s

Details

CI / backend-lint-and-test (push) Successful in 38s

Details

CI / integration (push) Successful in 4m3s

Details

57e52433d0

The SigLIP embedder + YOLO proposers load lazily then stay resident for the
container's whole lifetime — a 24/7 agent with an empty queue squats on ~5GB of
VRAM doing nothing (operator-observed: 4900MiB held at GPU-util 8% / P8). Sleep
mode only sheds downloaders + poll cadence; even a UI Stop left the models loaded.

Add a monitor thread that unloads the torch-owned models after
cfg.idle_unload_seconds (env IDLE_UNLOAD_SECONDS, default 300; 0 disables) with
the GPU genuinely idle (active==0, buffer drained, no job completed in the
window), then torch.cuda.empty_cache() to hand the blocks back to the driver.
They reload lazily on the next job via the existing _ensure_embedder /
_proposers_for. Covers both sleep-mode idle and a full Stop. Surfaced in
/status (models_loaded) and the agent UI pipe line; the VRAM meter drops too.

Residual: imgutils CCIP/person ONNX sessions + the CUDA context stay resident
(no clean unload API) — idle VRAM drops substantially, not to zero.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TbrA36zNczjVhrM6cWThQa

bvandeusen merged commit b5b437ca80 into main

2026-07-17 13:07:15 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#233