bvandeusen
a7abcc41ca
An errored GPU job's stored reason is a suspicion; the file probe is the
verdict. A 15-min beat sweep (triage_gpu_errors) runs verify_integrity's own
probe (sha256 + decode) on each errored image ONCE and writes both verdicts:
ImageRecord.integrity_status and the new GpuJob.triage_status ('defect' |
'file_ok', migration 0072). Every classification logs at WARNING so it
surfaces in Logs/System Activity.
- 'defect' rows are excluded from /retry_errors (re-running a known-bad file
burns agent time re-minting the tombstone); response now reports
defects_kept and the GpuAgentCard toast says so.
- GET /api/gpu/errors: triage view — reason buckets (classify_reason),
probe verdicts, per-job detail. POST /errors/triage runs the sweep now.
- POST /api/gpu/errors/<id>/recover: reuses the Layer-2 refetch pattern —
delete the defective copy + record (full cascade takes the tombstones too)
and re-poll its subscription Source so a fresh copy re-imports and re-enters
the pipeline; 'no_source' when nothing pollable resolves.
- New 'Failed processing' card (GpuTriageCard) in Maintenance: verdict counts,
reason summary, probe-now, defect list with thumbnails + per-image Recover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDgx8bQS5YrGRK76v8HUnM
76 lines
3.0 KiB
Python
76 lines
3.0 KiB
Python
"""GpuJob — a unit of GPU work the desktop agent pulls over HTTP (#114).
|
|
|
|
The durable work list that lets the agent stay HTTP-only: the server enqueues a
|
|
job per (image, task) — e.g. detect figures + CCIP-embed — and the agent LEASES a
|
|
batch, computes on its GPU, then SUBMITS results, all over the already-exposed web
|
|
API. Redis/Postgres stay private. A lease has an expiry; the lease query itself
|
|
re-claims expired leases (agent died / stopped mid-batch), so the queue is
|
|
self-healing without a separate sweep. One job is per ITEM; the agent fans a
|
|
VIDEO out into per-frame instances internally (see image_region.frame_time).
|
|
|
|
State: pending → leased → done | error (a failure under the attempt cap returns to
|
|
pending for another agent).
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import (
|
|
DateTime,
|
|
ForeignKey,
|
|
Index,
|
|
Integer,
|
|
String,
|
|
Text,
|
|
func,
|
|
text,
|
|
)
|
|
from sqlalchemy.orm import Mapped, mapped_column
|
|
|
|
from .base import Base
|
|
|
|
|
|
class GpuJob(Base):
|
|
__tablename__ = "gpu_job"
|
|
|
|
# Partial indexes over just the live slice (see migration 0070): the lease
|
|
# reads the lowest-id pending jobs on the hot path, and reclaims expired
|
|
# leases as a backstop — both stay O(batch) as done/error history grows.
|
|
__table_args__ = (
|
|
Index("ix_gpu_job_pending", "id", postgresql_where=text("status = 'pending'")),
|
|
Index(
|
|
"ix_gpu_job_leased_expires", "lease_expires_at",
|
|
postgresql_where=text("status = 'leased'"),
|
|
),
|
|
)
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
image_record_id: Mapped[int] = mapped_column(
|
|
ForeignKey("image_record.id", ondelete="CASCADE"), index=True
|
|
)
|
|
# What to compute, e.g. 'ccip' (detect figures + CCIP-embed) or 'siglip_region'.
|
|
task: Mapped[str] = mapped_column(String(32), nullable=False)
|
|
status: Mapped[str] = mapped_column(
|
|
String(16), nullable=False, default="pending", index=True
|
|
)
|
|
# pending | leased | done | error
|
|
lease_token: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
|
leased_at: Mapped[datetime | None] = mapped_column(
|
|
DateTime(timezone=True), nullable=True
|
|
)
|
|
lease_expires_at: Mapped[datetime | None] = mapped_column(
|
|
DateTime(timezone=True), nullable=True
|
|
)
|
|
attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
# Triage verdict for an ERRORED job (#125): NULL = not yet probed;
|
|
# 'defect' = the integrity probe says the FILE itself is bad (surfaced for
|
|
# recovery, excluded from /retry_errors); 'file_ok' = the file passes —
|
|
# the failure was operational (timeout/transient), safe to retry.
|
|
triage_status: Mapped[str | None] = mapped_column(String(16), nullable=True)
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|
|
updated_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|