feat(gpu): fast orphan recovery — graceful release + 60s sweep (#114)
So work an agent orphaned gets picked back up quickly, three layers: - GpuJobService.release(): a graceful agent stop hands its still-leased jobs back to pending instantly (POST /api/gpu/jobs/release), no waiting out the lease. - GpuJobService.recover_orphaned() + recover_orphaned_gpu_jobs Celery task on a 60s beat: resets expired leases (a hard-crashed agent) to pending and keeps the queue counts honest even when nothing is leasing. - Lease TTL 300→180s: still well above any single job (a capped-frame video embed is tens of seconds, and a live worker heartbeats), but a hard crash recovers faster once the sweep fires. Tests: release returns-to-pending (token-scoped), recover_orphaned resets only expired leases, release API round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -768,3 +768,30 @@ def enqueue_gpu_backfill(task_name: str) -> int:
|
||||
).fetchall()
|
||||
session.commit()
|
||||
return len(rows)
|
||||
|
||||
|
||||
@celery.task(name="backend.app.tasks.ml.recover_orphaned_gpu_jobs")
|
||||
def recover_orphaned_gpu_jobs() -> int:
|
||||
"""Reset expired GPU-job leases back to pending — recovers work orphaned by an
|
||||
agent that died mid-job (no graceful release). Short beat cadence so orphans
|
||||
get picked back up quickly + the queue counts read honestly. Returns the
|
||||
number recovered."""
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import update
|
||||
|
||||
from ..models import GpuJob
|
||||
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
now = datetime.now(UTC)
|
||||
res = session.execute(
|
||||
update(GpuJob)
|
||||
.where(GpuJob.status == "leased", GpuJob.lease_expires_at < now)
|
||||
.values(
|
||||
status="pending", lease_token=None, leased_at=None,
|
||||
lease_expires_at=None, updated_at=now,
|
||||
)
|
||||
)
|
||||
session.commit()
|
||||
return res.rowcount or 0
|
||||
|
||||
Reference in New Issue
Block a user