fix(gpu-jobs): end the error-tombstone loop — deliberate retry semantics + poison-job guards

The hourly ccip backfill's skip-list lacked 'error' (and the daily
siglip/embed variants re-gated failures on their missing results), so every
permanently-bad file got a fresh doomed job each run — ~24 duplicate error
rows/day per file, the perpetual 'unprocessable' flood. An errored job is now
a TOMBSTONE: no backfill re-enqueues it; retry is deliberate-only via
/retry_errors (an errored back-catalogue needs one button press after a
model swap).

One shared set of dedupe DELETEs (services/ml/gpu_jobs.error_dedupe_statements)
runs before every backfill and inside /retry_errors: error rows made moot by a
later pending/leased/done row go first, then older duplicates (newest reason
survives) — so the error count reads as distinct failing files and a retry
can't fan one file out into duplicate pending jobs. /retry_errors now returns
{requeued, pruned} and the toast shows both.

Poison-loop guards (release and lease-expiry burn no attempts, so a job that
stalls its transfer or crashes the agent every time cycled forever —
operator-observed jobs 99044/125288/131594/143131):
- agent: 3 in-session transient bounces (fetch or submit) → fail with the real
  reason instead of another release; strikes never count while stopping, and
  clear on submit success. Agent build 2026-07-02.3.
- server: the 60s orphan sweep (statements shared between the beat task and
  GpuJobService so they can't drift) converts expired leases with >=5 lease
  grants and pending jobs with >=10 to 'error', preserving the last stored
  failure reason. Backstops old agent builds.

Tests: tombstone rule across all three backfill variants, moot-row pruning,
poison conversions, and the extended /retry_errors dedupe contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM

agent/fc_agent/app.py

@@ -21,7 +21,7 @@ log = logging.getLogger("fc_agent.app")
 # Bump on every agent change. The page embeds this and /status reports it; the UI
 # warns to reload when they differ — so a stale browser-cached page can't be
 # mistaken for "the new image didn't deploy". (Belt-and-braces with no-store.)
-VERSION = "2026-07-02.2 · job.error now carries ffmpeg's actual failure reason"
+VERSION = "2026-07-02.3 · poison guard: repeated transient failures fail the job (with reason) instead of releasing it forever"
 
 logbuf.install()
 cfg = Config.from_env()

agent/fc_agent/worker.py

@@ -49,6 +49,16 @@ from .detectors import dedupe_crops
 # restart needed.
 MAX_BACKOFF_SECONDS = 60.0
 
+# A job whose fetch dies transiently this many times IN ONE SESSION stops being
+# handed back and is failed instead. Transient handbacks (release) burn no
+# attempts on the server, so a poisoned transfer — an original that stalls the
+# download every single time — would otherwise release→re-lease forever, churning
+# bandwidth without ever landing in the error queue (operator-observed jobs
+# 99044/125288/131594/143131, 2026-07-01). Failing it lets curator's attempt cap
+# tombstone it WITH the real reason. A genuine curator outage is unaffected:
+# every job takes at most one strike per outage, and strikes clear on success.
+TRANSIENT_JOB_CAP = 3
+
 
 def _is_transient(exc: requests.RequestException) -> bool:
     """A server/transport problem (wait it out) vs a job-specific fault (fail it).
@@ -192,6 +202,11 @@ class Worker:
         # all at once. Add on lease, discard on every terminal client call.
         self._held: set[int] = set()
         self._held_lock = threading.Lock()
+        # job_id → in-session transient-handback count (guarded by _held_lock).
+        # Cleared on success or terminal fail; KEPT across releases — persisting
+        # through the release→re-lease cycle is what lets TRANSIENT_JOB_CAP
+        # catch a poisoned transfer.
+        self._transient_seen: dict[int, int] = {}
         self.processed = 0
         self.downloaded = 0       # jobs fetched+decoded into the buffer (monotonic);
         #                           feeds the server-side downloads/min rate below.
@@ -235,6 +250,21 @@ class Worker:
         with self._held_lock:
             self._held.discard(job_id)
 
+    def _strike(self, jid: int) -> int:
+        """Record a transient bounce for this job; returns the in-session total
+        (compared against TRANSIENT_JOB_CAP by both the fetch and submit paths)."""
+        with self._held_lock:
+            n = self._transient_seen.get(jid, 0) + 1
+            self._transient_seen[jid] = n
+            return n
+
+    def _forget_strikes(self, jid: int) -> None:
+        """Terminal outcome (submitted or failed) → this job's strike count no
+        longer matters. Deliberately NOT called on release: strikes surviving
+        the release→re-lease cycle is what makes TRANSIENT_JOB_CAP work."""
+        with self._held_lock:
+            self._transient_seen.pop(jid, None)
+
     def _release(self, job_ids: list[int]) -> None:
         """Hand still-held leases back to curator and drop them from the held set —
         the single hand-back path for both a downloader exiting (stop/shrink) with
@@ -253,6 +283,7 @@ class Worker:
         log.warning("job %s (image %s) %s: %s", jid, image_id, verb, str(exc)[:200])
         self.client.fail(jid, str(exc)[:500])
         self._unhold(jid)
+        self._forget_strikes(jid)
 
     def _stopped(self, stop_evt: threading.Event) -> bool:
         """The shared 'should I bail now?' check — the worker is stopping (global
@@ -580,6 +611,20 @@ class Worker:
                 except requests.RequestException as exc:
                     owned.remove(jid)
                     if _is_transient(exc):
+                        # A Stop mid-fetch lands here too (a killed ffmpeg is
+                        # not the job's fault) — no strike for that; only count
+                        # bounces taken while genuinely running.
+                        if (not self._stopped(stop_evt)
+                                and self._strike(jid) >= TRANSIENT_JOB_CAP):
+                            # THIS job keeps dying while others move: a poisoned
+                            # transfer, not a curator outage. Fail it so the
+                            # server's attempt cap tombstones it with the real
+                            # reason instead of cycling it forever.
+                            self._fail(
+                                jid, job.get("image_id"), exc,
+                                verb="gave up after repeated transient failures",
+                            )
+                            continue
                         # curator down/redeploying or our lease was reclaimed —
                         # NOT the job's fault. Hand back this job + the rest of the
                         # batch and back the whole loop off.
@@ -742,6 +787,7 @@ class Worker:
                 with self._timed("submit"):
                     self.client.submit_embedding(jid, vec, embed_version)
                 self._unhold(jid)
+                self._forget_strikes(jid)
                 return True
 
             # task picks what to produce per crop:
@@ -854,9 +900,18 @@ class Worker:
             with self._timed("submit"):
                 self.client.submit(jid, regions, replace_kinds)
             self._unhold(jid)
+            self._forget_strikes(jid)
             return True
         except requests.RequestException as exc:
             if _is_transient(exc):
+                if (not self._stopped(stop_evt)
+                        and self._strike(jid) >= TRANSIENT_JOB_CAP):
+                    # Same poison rationale as the fetch path: a job whose
+                    # submit keeps dying transiently would otherwise re-lease →
+                    # re-download → re-GPU forever.
+                    self._fail(jid, job.get("image_id"), exc,
+                               verb="gave up after repeated transient failures")
+                    return False
                 # curator down/redeploying, a 5xx, or our lease was reclaimed
                 # while we worked. NOT the job's fault — hand it back (best
                 # effort; then the server's orphan-recovery reclaims it if down).