feat(gpu): "Retry errored jobs" — scoped requeue of errors only
After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.
POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.
UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.
Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+22
-1
@@ -11,7 +11,7 @@ homelab admin.
|
||||
import secrets
|
||||
|
||||
from quart import Blueprint, jsonify, request
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
|
||||
from ..extensions import get_session
|
||||
@@ -109,6 +109,27 @@ async def reprocess():
|
||||
return jsonify({"celery_task_id": r.id, "task": task}), 202
|
||||
|
||||
|
||||
@gpu_bp.route("/retry_errors", methods=["POST"])
|
||||
async def retry_errors():
|
||||
"""Requeue every ERRORED job (all task types) back to pending — the scoped
|
||||
recovery after an agent-side fix (e.g. the short-video sampler), where
|
||||
/reprocess would needlessly re-run the whole done library too. Attempts and
|
||||
the stored error reset so each job gets its full retry budget under the
|
||||
fixed pipeline. Small row count (errors only) → inline UPDATE, and the
|
||||
response carries the number requeued for the UI toast."""
|
||||
async with get_session() as session:
|
||||
res = await session.execute(
|
||||
update(GpuJob)
|
||||
.where(GpuJob.status == "error")
|
||||
.values(
|
||||
status="pending", attempts=0, error=None, lease_token=None,
|
||||
leased_at=None, lease_expires_at=None, updated_at=func.now(),
|
||||
)
|
||||
)
|
||||
await session.commit()
|
||||
return jsonify({"requeued": res.rowcount or 0})
|
||||
|
||||
|
||||
# --- Agent (bearer token): lease / submit / heartbeat / fail ------------
|
||||
|
||||
@gpu_bp.route("/jobs/lease", methods=["POST"])
|
||||
|
||||
Reference in New Issue
Block a user