bvandeusen
a7abcc41ca
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
33 lines
1.0 KiB
Python
33 lines
1.0 KiB
Python
"""gpu_job.triage_status — the probe's verdict on an errored job's FILE
|
|
|
|
Failure triage (#125): a periodic sweep probes each errored image's file
|
|
(sha256 + decode, verify_integrity's machinery) exactly once and stores the
|
|
verdict here — 'defect' (the file is bad: recovery material, excluded from
|
|
/retry_errors) or 'file_ok' (failure was operational, safe to retry). NULL
|
|
means not yet probed; selecting on NULL is what makes the sweep resumable.
|
|
No index: the errored slice the sweep scans is tiny by design (tombstones).
|
|
|
|
Revision ID: 0072
|
|
Revises: 0071
|
|
Create Date: 2026-07-02
|
|
"""
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0072"
|
|
down_revision: Union[str, None] = "0071"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.add_column(
|
|
"gpu_job", sa.Column("triage_status", sa.String(16), nullable=True)
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_column("gpu_job", "triage_status")
|