feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
This commit is contained in:
@@ -164,6 +164,52 @@ def test_recover_interrupted_handles_both_stuck_and_orphans(db_sync, monkeypatch
|
||||
assert dispatched == [stuck.id] # stuck rows re-enqueue; orphans don't
|
||||
|
||||
|
||||
def test_recover_interrupted_poison_pill_caps_at_max(db_sync, monkeypatch):
|
||||
"""A stuck row that's already been recovered MAX_RECOVERY_ATTEMPTS-1
|
||||
times is marked 'failed' (with a diagnostic) instead of re-queued —
|
||||
the circuit breaker against an input that hard-crashes the worker
|
||||
every run. Operator-flagged 2026-05-28."""
|
||||
from backend.app.tasks import import_file
|
||||
from backend.app.tasks.maintenance import (
|
||||
MAX_RECOVERY_ATTEMPTS,
|
||||
recover_interrupted_tasks,
|
||||
)
|
||||
dispatched: list[int] = []
|
||||
monkeypatch.setattr(
|
||||
import_file.import_media_file, "delay", dispatched.append
|
||||
)
|
||||
|
||||
batch_id = _make_batch(db_sync)
|
||||
now = datetime.now(UTC)
|
||||
|
||||
# At the cap already (recovered MAX-1 times) → fail, don't re-queue.
|
||||
poison = ImportTask(
|
||||
batch_id=batch_id, source_path="/import/poison.jpg", task_type="media",
|
||||
status="processing", started_at=now - timedelta(hours=2),
|
||||
recovery_count=MAX_RECOVERY_ATTEMPTS - 1,
|
||||
)
|
||||
# One recovery short of the cap → re-queue + increment.
|
||||
recoverable = ImportTask(
|
||||
batch_id=batch_id, source_path="/import/ok.jpg", task_type="media",
|
||||
status="processing", started_at=now - timedelta(hours=2),
|
||||
recovery_count=MAX_RECOVERY_ATTEMPTS - 2,
|
||||
)
|
||||
db_sync.add_all([poison, recoverable])
|
||||
db_sync.commit()
|
||||
|
||||
touched = recover_interrupted_tasks.apply().get()
|
||||
assert touched == 2 # one failed + one re-queued
|
||||
|
||||
db_sync.refresh(poison)
|
||||
db_sync.refresh(recoverable)
|
||||
assert poison.status == "failed"
|
||||
assert "corrupt or" in (poison.error or "")
|
||||
assert recoverable.status == "queued"
|
||||
assert recoverable.recovery_count == MAX_RECOVERY_ATTEMPTS - 1
|
||||
# Only the recoverable row re-enqueues; the poison pill does not.
|
||||
assert dispatched == [recoverable.id]
|
||||
|
||||
|
||||
def test_cleanup_old_deletes_finished_old(db_sync):
|
||||
batch_id = _make_batch(db_sync)
|
||||
now = datetime.now(UTC)
|
||||
|
||||
Reference in New Issue
Block a user