Agent: GPU load + live worker tuning + fast orphan recovery #147

Merged

bvandeusen merged 3 commits from dev into main 2026-06-29 19:11:22 -04:00

Conversation 0 Commits 3 Files Changed 12

+328 -88

bvandeusen commented 2026-06-29 19:11:16 -04:00

Owner

Copy Link

Copy Source

Agent control UI

• GPU load readout (nvidia-smi): util %, VRAM used/total + bar, temp.
• Live worker-count tuning — a − / + control (POST /concurrency), not just an env. The worker is now a pool of slots (shared model → slots add concurrent inference, not N× VRAM). Start/Stop + the dial replace pause/resume.

Fast orphan recovery (three layers)

• Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back to pending instantly (/api/gpu/jobs/release).
• 60s recovery sweep (recover_orphaned_gpu_jobs beat): resets expired leases (hard-crashed agent), keeps queue counts honest.
• Lease TTL 300→180s — faster hard-crash recovery, still above any single job.

Backend (release/recover/TTL/sweep) is CI-tested (1009 passed); the agent half is verified by running. Triggers a build-agent republish (includes the cuDNN base).

🤖 Generated with Claude Code

## Agent control UI - **GPU load readout** (nvidia-smi): util %, VRAM used/total + bar, temp. - **Live worker-count tuning** — a − / + control (`POST /concurrency`), not just an env. The worker is now a pool of slots (shared model → slots add concurrent inference, not N× VRAM). Start/Stop + the dial replace pause/resume. ## Fast orphan recovery (three layers) - **Graceful release** on stop / pool-shrink: a slot hands its still-leased jobs back to `pending` instantly (`/api/gpu/jobs/release`). - **60s recovery sweep** (`recover_orphaned_gpu_jobs` beat): resets expired leases (hard-crashed agent), keeps queue counts honest. - **Lease TTL 300→180s** — faster hard-crash recovery, still above any single job. Backend (release/recover/TTL/sweep) is CI-tested (1009 passed); the agent half is verified by running. Triggers a `build-agent` republish (includes the cuDNN base). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bvandeusen added 3 commits 2026-06-29 19:11:17 -04:00

docs(agent): note the NVIDIA Container Toolkit host prereq ... 614b6bc52a

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

feat(gpu): fast orphan recovery — graceful release + 60s sweep (#114) ... 2cb0427868

So work an agent orphaned gets picked back up quickly, three layers:
- GpuJobService.release(): a graceful agent stop hands its still-leased jobs back
  to pending instantly (POST /api/gpu/jobs/release), no waiting out the lease.
- GpuJobService.recover_orphaned() + recover_orphaned_gpu_jobs Celery task on a
  60s beat: resets expired leases (a hard-crashed agent) to pending and keeps the
  queue counts honest even when nothing is leasing.
- Lease TTL 300→180s: still well above any single job (a capped-frame video embed
  is tens of seconds, and a live worker heartbeats), but a hard crash recovers
  faster once the sweep fires.

Tests: release returns-to-pending (token-scoped), recover_orphaned resets only
expired leases, release API round-trip.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

feat(agent): GPU load readout + live worker-count tuning (#114) ... 4a1a9ec5a7

Control UI gains what the operator asked for:
- GPU load (nvidia-smi): util %, VRAM used/total + bar, temp — so you can see how
  hard the card is working while you're at the desktop.
- Worker count is now a live − / + control (POST /concurrency), not just an env:
  the worker is a pool of independent slots (shared model, so slots add concurrent
  inference, not N× VRAM). Dial up for speed, down to free the card. Replaces
  pause/resume with Start/Stop + the worker dial.
- Graceful release on stop / pool-shrink: a slot hands its still-leased jobs back
  via client.release() so they're re-picked immediately (pairs with the server
  recovery sweep).

Not CI-tested (agent/ outside CI) — verified by running.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa

bvandeusen merged commit 18300e1f8a into main 2026-06-29 19:11:22 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

bvandeusen

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bvandeusen/FabledCurator#147