feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3)

An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.

- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
  burns agent time re-minting the tombstone); response now reports
  defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
  probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
  delete the defective copy + record (full cascade takes the tombstones too)
  and re-poll its subscription Source so a fresh copy re-imports and re-enters
  the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
  reason summary, probe-now, defect list with thumbnails + per-image Recover.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
2026-07-02 12:36:02 -04:00
parent 1f27189b8f
commit a7abcc41ca
12 changed files with 763 additions and 10 deletions
@@ -0,0 +1,32 @@
"""gpu_job.triage_status — the probe's verdict on an errored job's FILE
Failure triage (#125): a periodic sweep probes each errored image's file
(sha256 + decode, verify_integrity's machinery) exactly once and stores the
verdict here — 'defect' (the file is bad: recovery material, excluded from
/retry_errors) or 'file_ok' (failure was operational, safe to retry). NULL
means not yet probed; selecting on NULL is what makes the sweep resumable.
No index: the errored slice the sweep scans is tiny by design (tombstones).
Revision ID: 0072
Revises: 0071
Create Date: 2026-07-02
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "0072"
down_revision: Union[str, None] = "0071"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"gpu_job", sa.Column("triage_status", sa.String(16), nullable=True)
)
def downgrade() -> None:
op.drop_column("gpu_job", "triage_status")