fix(gpu-jobs): end the error-tombstone loop — deliberate retry semantics + poison-job guards

The hourly ccip backfill's skip-list lacked 'error' (and the daily
siglip/embed variants re-gated failures on their missing results), so every
permanently-bad file got a fresh doomed job each run — ~24 duplicate error
rows/day per file, the perpetual 'unprocessable' flood. An errored job is now
a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via
/retry_errors (an errored back-catalogue needs one button press after a
model swap).

One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements)
runs before every backfill and inside /retry_errors: error rows made moot by a
later pending/leased/done row go first, then older duplicates (newest reason
survives) — so the error count reads as distinct failing files and a retry
can't fan one file out into duplicate pending jobs. /retry_errors now returns
{requeued, pruned} and the toast shows both.

Poison-loop guards (release and lease-expiry burn no attempts, so a job that
stalls its transfer or crashes the agent every time cycled forever —
operator-observed jobs 99044/125288/131594/143131):
- agent: 3 in-session transient bounces (fetch or submit) → fail with the real
  reason instead of another release; strikes never count while stopping, and
  clear on submit success. Agent build 2026-07-02.3.
- server: the 60s orphan sweep (statements shared between the beat task and
  GpuJobService so they can't drift) converts expired leases with >=5 lease
  grants and pending jobs with >=10 to 'error', preserving the last stored
  failure reason. Backstops old agent builds.

Tests: tombstone rule across all three backfill variants, moot-row pruning,
poison conversions, and the extended /retry_errors dedupe contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

backend/app/api/gpu.py

@@ -17,7 +17,7 @@ from sqlalchemy.dialects.postgresql import insert as pg_insert
 from ..extensions import get_session
 from ..models import AppSetting, GpuJob, ImageRecord, MLSettings
 from ..services.gallery_service import image_url
-from ..services.ml.gpu_jobs import GpuJobService
+from ..services.ml.gpu_jobs import GpuJobService, error_dedupe_statements
 from ..services.ml.regions import RegionService
 
 gpu_bp = Blueprint("gpu", __name__, url_prefix="/api/gpu")
@@ -115,9 +115,15 @@ async def retry_errors():
     recovery after an agent-side fix (e.g. the short-video sampler), where
     /reprocess would needlessly re-run the whole done library too. Attempts and
     the stored error reset so each job gets its full retry budget under the
-    fixed pipeline. Small row count (errors only) → inline UPDATE, and the
-    response carries the number requeued for the UI toast."""
+    fixed pipeline. Stale tombstones are pruned FIRST (loop-era duplicates and
+    rows a later success made moot — the same statements the backfills run), so
+    one failing file requeues as ONE job, never a fan-out of duplicates. Small
+    row count (errors only) → inline statements; the response carries both
+    counts for the UI toast."""
     async with get_session() as session:
+        pruned = 0
+        for stmt in error_dedupe_statements():
+            pruned += (await session.execute(stmt)).rowcount or 0
         res = await session.execute(
             update(GpuJob)
             .where(GpuJob.status == "error")
@@ -127,7 +133,7 @@ async def retry_errors():
             )
         )
         await session.commit()
-    return jsonify({"requeued": res.rowcount or 0})
+    return jsonify({"requeued": res.rowcount or 0, "pruned": pruned})
 
 
 # --- Agent (bearer token): lease / submit / heartbeat / fail ------------