feat(gpu): fast orphan recovery — graceful release + 60s sweep (#114)
So work an agent orphaned gets picked back up quickly, three layers: - GpuJobService.release(): a graceful agent stop hands its still-leased jobs back to pending instantly (POST /api/gpu/jobs/release), no waiting out the lease. - GpuJobService.recover_orphaned() + recover_orphaned_gpu_jobs Celery task on a 60s beat: resets expired leases (a hard-crashed agent) to pending and keeps the queue counts honest even when nothing is leasing. - Lease TTL 300→180s: still well above any single job (a capped-frame video embed is tens of seconds, and a live worker heartbeats), but a hard crash recovers faster once the sweep fires. Tests: release returns-to-pending (token-scoped), recover_orphaned resets only expired leases, release API round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttrj5P7upUTueSfoJcxEqa
This commit is contained in:
@@ -196,3 +196,18 @@ async def fail():
|
||||
)
|
||||
await session.commit()
|
||||
return jsonify({"ok": ok})
|
||||
|
||||
|
||||
@gpu_bp.route("/jobs/release", methods=["POST"])
|
||||
async def release():
|
||||
"""Graceful stop: the agent hands its still-leased jobs back to pending so
|
||||
they're picked up immediately instead of waiting out the lease."""
|
||||
body = await request.get_json(silent=True) or {}
|
||||
agent_id = str(body.get("agent_id") or "agent")
|
||||
job_ids = [int(x) for x in (body.get("job_ids") or [])]
|
||||
async with get_session() as session:
|
||||
if not await _agent_authed(session):
|
||||
return jsonify({"error": "unauthorized"}), 401
|
||||
n = await GpuJobService(session).release(agent_id, job_ids)
|
||||
await session.commit()
|
||||
return jsonify({"released": n})
|
||||
|
||||
Reference in New Issue
Block a user