feat(gpu): fast orphan recovery — graceful release + 60s sweep (#114)

So work an agent orphaned gets picked back up quickly, three layers:
- GpuJobService.release(): a graceful agent stop hands its still-leased jobs back
  to pending instantly (POST /api/gpu/jobs/release), no waiting out the lease.
- GpuJobService.recover_orphaned() + recover_orphaned_gpu_jobs Celery task on a
  60s beat: resets expired leases (a hard-crashed agent) to pending and keeps the
  queue counts honest even when nothing is leasing.
- Lease TTL 300→180s: still well above any single job (a capped-frame video embed
  is tens of seconds, and a live worker heartbeats), but a hard crash recovers
  faster once the sweep fires.

Tests: release returns-to-pending (token-scoped), recover_orphaned resets only
expired leases, release API round-trip.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
2026-06-29 19:07:40 -04:00
parent 614b6bc52a
commit 2cb0427868
6 changed files with 149 additions and 6 deletions
+4
View File
@@ -117,6 +117,10 @@ def make_celery() -> Celery:
"task": "backend.app.tasks.ml.scheduled_apply_head_tags",
"schedule": 86400.0, # no-op unless head_auto_apply_enabled
},
"recover-orphaned-gpu-jobs": {
"task": "backend.app.tasks.ml.recover_orphaned_gpu_jobs",
"schedule": 60.0, # quick pickup of work a dead agent orphaned
},
"snapshot-head-metrics-daily": {
"task": "backend.app.tasks.maintenance.snapshot_head_metrics",
"schedule": 86400.0,