feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3)

An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.

- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
  burns agent time re-minting the tombstone); response now reports
  defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
  probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
  delete the defective copy + record (full cascade takes the tombstones too)
  and re-poll its subscription Source so a fresh copy re-imports and re-enters
  the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
  reason summary, probe-now, defect list with thumbnails + per-image Recover.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
2026-07-02 12:36:02 -04:00
parent 1f27189b8f
commit a7abcc41ca
12 changed files with 763 additions and 10 deletions
+22
View File
@@ -582,6 +582,28 @@ def verify_integrity() -> int:
return total
@celery.task(
name="backend.app.tasks.maintenance.triage_gpu_errors",
# Bounded small-set probe (only errored images, once each), but a single
# large original's sha256 over NFS can run tens of seconds — same quick-lane
# tolerance rationale as verify_integrity above.
soft_time_limit=600, time_limit=900,
)
def triage_gpu_errors() -> dict:
"""Failure triage (#125): probe each errored GPU job's file once and write
the verdicts (ImageRecord.integrity_status + GpuJob.triage_status) — see
services/ml/gpu_triage.py. Time-boxed + resumable; no-op when every errored
job is already triaged."""
from ..services.ml.gpu_triage import triage_errored_jobs
SessionLocal = _sync_session_factory()
with SessionLocal() as session:
summary = triage_errored_jobs(session, time_budget_seconds=300.0)
if summary["probed"]:
log.info("triage_gpu_errors: %s", summary)
return summary
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_download_events")
def recover_stalled_download_events() -> int:
"""Recover DownloadEvent rows stuck pending/running past the worker hard kill.