feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3)
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
@@ -582,6 +582,28 @@ def verify_integrity() -> int:
|
||||
return total
|
||||
|
||||
|
||||
@celery.task(
|
||||
name="backend.app.tasks.maintenance.triage_gpu_errors",
|
||||
# Bounded small-set probe (only errored images, once each), but a single
|
||||
# large original's sha256 over NFS can run tens of seconds — same quick-lane
|
||||
# tolerance rationale as verify_integrity above.
|
||||
soft_time_limit=600, time_limit=900,
|
||||
)
|
||||
def triage_gpu_errors() -> dict:
|
||||
"""Failure triage (#125): probe each errored GPU job's file once and write
|
||||
the verdicts (ImageRecord.integrity_status + GpuJob.triage_status) — see
|
||||
services/ml/gpu_triage.py. Time-boxed + resumable; no-op when every errored
|
||||
job is already triaged."""
|
||||
from ..services.ml.gpu_triage import triage_errored_jobs
|
||||
|
||||
SessionLocal = _sync_session_factory()
|
||||
with SessionLocal() as session:
|
||||
summary = triage_errored_jobs(session, time_budget_seconds=300.0)
|
||||
if summary["probed"]:
|
||||
log.info("triage_gpu_errors: %s", summary)
|
||||
return summary
|
||||
|
||||
|
||||
@celery.task(name="backend.app.tasks.maintenance.recover_stalled_download_events")
|
||||
def recover_stalled_download_events() -> int:
|
||||
"""Recover DownloadEvent rows stuck pending/running past the worker hard kill.
|
||||
|
||||
Reference in New Issue
Block a user