feat(gpu): "Retry errored jobs" — scoped requeue of errors only
CI / backend-lint-and-test (push) Successful in 25s
CI / integration (push) Successful in 3m26s
CI / lint (push) Successful in 2s
CI / frontend-build (push) Successful in 20s

After an agent-side fix (e.g. the short-video sampler), the errored jobs
(~2.8k) have exhausted their 3 attempts and stay parked: backfill skips
images that already have a job, and /reprocess is the nuclear option (it
resets the 179k DONE jobs too). There was no way to re-run just the errors.

POST /api/gpu/retry_errors resets every status='error' job (all task types)
to pending with attempts=0 and the stored error cleared — a small inline
UPDATE that returns {requeued: n} so the UI toast can show the count.

UI: a "Retry errored jobs" button on the GPU-agent card, right under the
queue tiles; disabled when errored==0. With the agent now logging ffmpeg's
stderr on failure, retrying also reveals which errors were real vs victims
of the fps-filter bug.

Test: retry_errors requeues the errored job (fresh attempts, error cleared)
and leaves done work untouched; asserts via column selects (Core-DML
gotcha), not ORM refresh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-01 21:09:07 -04:00
parent 3a683d7feb
commit 686808d3f3
4 changed files with 92 additions and 3 deletions
+22 -1
View File
@@ -11,7 +11,7 @@ homelab admin.
import secrets
from quart import Blueprint, jsonify, request
from sqlalchemy import func, select
from sqlalchemy import func, select, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from ..extensions import get_session
@@ -109,6 +109,27 @@ async def reprocess():
return jsonify({"celery_task_id": r.id, "task": task}), 202
@gpu_bp.route("/retry_errors", methods=["POST"])
async def retry_errors():
"""Requeue every ERRORED job (all task types) back to pending — the scoped
recovery after an agent-side fix (e.g. the short-video sampler), where
/reprocess would needlessly re-run the whole done library too. Attempts and
the stored error reset so each job gets its full retry budget under the
fixed pipeline. Small row count (errors only) → inline UPDATE, and the
response carries the number requeued for the UI toast."""
async with get_session() as session:
res = await session.execute(
update(GpuJob)
.where(GpuJob.status == "error")
.values(
status="pending", attempts=0, error=None, lease_token=None,
leased_at=None, lease_expires_at=None, updated_at=func.now(),
)
)
await session.commit()
return jsonify({"requeued": res.rowcount or 0})
# --- Agent (bearer token): lease / submit / heartbeat / fail ------------
@gpu_bp.route("/jobs/lease", methods=["POST"])