feat(import-resilience L1): poison-pill circuit breaker — cap stuck-task re-queues
Layer 1 of the import-task resilience work (operator-requested
2026-05-28). The recover_interrupted_tasks sweep re-queues rows stuck
in 'processing' — correct for a worker crash, but without a cap a row
that RELIABLY hard-crashes the worker (OOM/segfault/SIGKILL on a
corrupt or oversized input) loops forever: re-queue → crash → re-queue,
burning a worker slot every 5 min. A caught exception flips to terminal
'failed' and never enters this loop; only process-killing inputs do.
- alembic 0026: import_task.recovery_count (int, default 0) +
import_task.refetched (bool, default false — backs Layer 2).
- recover_interrupted_tasks now runs a poison-pill UPDATE FIRST: stuck
rows whose recovery_count has already reached MAX_RECOVERY_ATTEMPTS-1
are marked 'failed' with a diagnostic ("crashed or stalled the worker
N times … likely a corrupt or oversized input … inspect/replace the
file, then retry via /api/import/retry-failed") instead of re-queued.
The re-queue pass then handles the remaining stuck rows and bumps
recovery_count. Shared stuck_predicate (and_/or_) keeps the
media-5min / archive-40min split.
- MAX_RECOVERY_ATTEMPTS=3 (two recoveries then give up).
The failed poison pill surfaces in the existing import-failures view
with its file path, directly answering "help me identify them."
Test test_recover_interrupted_poison_pill_caps_at_max pins both
branches: a row at the cap is failed (not re-enqueued, diagnostic
present), a row one short is re-queued + incremented.
This commit is contained in:
@@ -0,0 +1,53 @@
|
||||
"""import_task.recovery_count + refetched — poison-pill circuit breaker
|
||||
|
||||
Revision ID: 0026
|
||||
Revises: 0025
|
||||
Create Date: 2026-05-28
|
||||
|
||||
Backs the import-task resilience work (operator-flagged 2026-05-28):
|
||||
|
||||
- recovery_count: how many times recover_interrupted_tasks has
|
||||
re-queued this row from a stuck 'processing' state. A row that
|
||||
hard-crashes the worker (OOM / segfault on a corrupt or oversized
|
||||
input) leaves no terminal flip, so the sweep re-queues it — and
|
||||
without a cap it would loop forever, re-crashing the worker each
|
||||
time. After MAX_RECOVERY_ATTEMPTS the sweep marks it 'failed' with a
|
||||
diagnostic instead.
|
||||
|
||||
- refetched: whether a one-shot re-download has already been attempted
|
||||
for this task's file. Bounds the Layer-2 re-fetch remediation to a
|
||||
single attempt so source-side corruption doesn't loop.
|
||||
|
||||
Both default to 0 / false; additive, no backfill needed.
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0026"
|
||||
down_revision: Union[str, None] = "0025"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"import_task",
|
||||
sa.Column(
|
||||
"recovery_count", sa.Integer(), nullable=False,
|
||||
server_default="0",
|
||||
),
|
||||
)
|
||||
op.add_column(
|
||||
"import_task",
|
||||
sa.Column(
|
||||
"refetched", sa.Boolean(), nullable=False,
|
||||
server_default=sa.false(),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("import_task", "refetched")
|
||||
op.drop_column("import_task", "recovery_count")
|
||||
Reference in New Issue
Block a user