feat(triage): failed-processing triage — probe errored files, flag defects, recover (#125 C1-C3)
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
This commit is contained in:
@@ -0,0 +1,32 @@
|
||||
"""gpu_job.triage_status — the probe's verdict on an errored job's FILE
|
||||
|
||||
Failure triage (#125): a periodic sweep probes each errored image's file
|
||||
(sha256 + decode, verify_integrity's machinery) exactly once and stores the
|
||||
verdict here — 'defect' (the file is bad: recovery material, excluded from
|
||||
/retry_errors) or 'file_ok' (failure was operational, safe to retry). NULL
|
||||
means not yet probed; selecting on NULL is what makes the sweep resumable.
|
||||
No index: the errored slice the sweep scans is tiny by design (tombstones).
|
||||
|
||||
Revision ID: 0072
|
||||
Revises: 0071
|
||||
Create Date: 2026-07-02
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0072"
|
||||
down_revision: Union[str, None] = "0071"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"gpu_job", sa.Column("triage_status", sa.String(16), nullable=True)
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("gpu_job", "triage_status")
|
||||
Reference in New Issue
Block a user