bvandeusen
e77afe8295
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
58 lines
2.2 KiB
Python
58 lines
2.2 KiB
Python
"""ImportTask — a single source file to import as part of a batch.
|
|
|
|
State machine: pending -> queued -> processing -> complete | skipped | failed.
|
|
Workers crashing mid-task leave rows in 'processing'; the recovery sweep
|
|
(tasks/maintenance.py::recover_interrupted_tasks) re-queues rows that have
|
|
been processing longer than the stuck-task threshold.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from sqlalchemy import (
|
|
BigInteger,
|
|
Boolean,
|
|
DateTime,
|
|
ForeignKey,
|
|
Integer,
|
|
String,
|
|
Text,
|
|
func,
|
|
)
|
|
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
|
|
|
from .base import Base
|
|
|
|
|
|
class ImportTask(Base):
|
|
__tablename__ = "import_task"
|
|
|
|
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
batch_id: Mapped[int] = mapped_column(
|
|
ForeignKey("import_batch.id", ondelete="CASCADE"), nullable=False, index=True
|
|
)
|
|
|
|
source_path: Mapped[str] = mapped_column(Text, nullable=False)
|
|
task_type: Mapped[str] = mapped_column(String(16), nullable=False) # media|archive
|
|
status: Mapped[str] = mapped_column(String(16), nullable=False, default="pending", index=True)
|
|
|
|
# Poison-pill circuit breaker (alembic 0026). recovery_count tracks
|
|
# how many times the stuck-task sweep has re-queued this row; after
|
|
# the cap it's failed with a diagnostic instead of looping. refetched
|
|
# bounds the one-shot re-download remediation to a single attempt.
|
|
recovery_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
refetched: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
|
|
|
|
result_image_id: Mapped[int | None] = mapped_column(
|
|
ForeignKey("image_record.id", ondelete="SET NULL"), nullable=True
|
|
)
|
|
error: Mapped[str | None] = mapped_column(Text, nullable=True)
|
|
size_bytes: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
|
|
|
|
created_at: Mapped[datetime] = mapped_column(
|
|
DateTime(timezone=True), nullable=False, server_default=func.now()
|
|
)
|
|
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
|
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
|
|
|
batch = relationship("ImportBatch", back_populates="tasks")
|